How avoid unwanted systemd-journald?

[Franta Hanzlík]

For one thing I'm in the conviction that binary logs are hazardous
bullshit, for another on my two F19 machines systemd-journald occupy
significant part of resources (after several days it is often 1.5 to
2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).

Then, how I can avoid this crap and log only to standard rsyslogd?

I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
/etc/systemd/{system.conf,user.conf}, but this not helped; setting
"Storage=none" in "[Journal]" section of /etc/systemd/journald.conf
is better, but I want ideally completely cut out systemd-journald
from my systems.

TIA, Franta Hanzlik

[Tom H]

On Mon, Nov 25, 2013 at 12:58 AM, Frantisek Hanzlik <franta(a)hanzlici.cz&gt; wrote:
>  Frank Murphy wrote:
> > On Sun, 17 Nov 2013 12:37:03 +0100
> > Patrick Lists <fedora-list(a)puzzled.xs4all.nl&gt; wrote:
> >>
> >> I meant the transmission of the log to another log server. Not the
> >> log itself. Anyway, here is Lennart Poettering's rationale behind
> >> journald:
> >
> > One of the things you will see after a hard reset is:
> > Journal corrupted, journal deleted and being restarted.
> > ie.. no useful info.
>
>  This is perhaps most significant shortcoming. And beside it I see in
>  log messages as:
>
>  systemd-journald[295]: Failed to set ACL on
> /var/log/journal/70ed4dbe694041d484165719514f195e/user-1045.journal, ignoring: Invalid
> argument
>  systemd-journald[2948]: Failed to write entry, ignoring: Cannot allocate memory
>  systemd-journal[388]: Suppressed 966 messages from /system/dbus.service
>  systemd-journal[299]: Forwarding to syslog missed 92 messages.
>
>  which induce sense of journald logging futility. 
Googling shows that it was fixed in May and applied in OpenSUSE in
June and in Debian in September:

http://lists.opensuse.org/opensuse-commit/2013-06/msg01102.html

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717863

Given that there have been F19 systemd uploads since May, I assume
that it's also fixed in Fedora but the changelogs don't mention it.

[Joonas Sarajärvi]

2013/11/17 Frantisek Hanzlik <franta(a)hanzlici.cz&gt;:
>  Steven Stern wrote:
> > On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
> >> For one thing I'm in the conviction that binary logs are hazardous
> >> bullshit,
> >
> > In what way might the logs be hazardous?
>
>  It was mean mainly from administrator view. When things go bad,
>  machine HW/SW fail or any other disasters occurs, logs are very
>  valuable. And I'm confident that binary logs are too weak in this
>  situation. Text logs are useful even if log file is damaged or ends
>  with fragments and can be easy readable with lot of tools. Binary
>  logs, by contrast, may be useless when log file is damaged or I
>  haven't this one unique utility for reading them. And my experiences
>  with systems where binary logs are implemented says clearly that
>  binary logs is bad idea.
>  Second, it is question when tight integration of systemd and logging
>  services has any benefits - there is number of situation (logging
>  over network, for example) which speaks for separate logging service. 
The journald log format is documented at least to some extent [1], and
there exists free software for reading the log. To me, it sounds like
way more accessible than if it was a binary data format of a typical
proprietary tool. For example, booting any Fedora live image should
suffice if you need to read the journal of a system that uses journald
and happens to become unbootable.

Typically journalctl will generate you a representation of the data in
syslog log file format. This loses some infromation that the journal
stores, because there is no way to represent it all while keeping the
output syslog-like and easily human-readable. This output can be
easily fed to traditional unix tools like grep, if that is your
preferred way of extracting information from logs.

Finally, since you can also run a normal syslog daemon which maintains
a text-format logs for you, I do not really see why having some log
data in journald format under /run/log/journal/ should be considered
hazardous. You can pretty much just ignore it if you feel like using
other kinds of tools for storing and managing logs.

>  And possibilities with e.g. rsyslogd are better than with journald
>  - why Fedora must again replace verified, reliable, Unix standard
>  things with some crappy solutions? For nightmares of its users? 
At least for my
needs, the journal has been way more convenient to use
than rsyslog. It is much nicer to read logs when journalctl e.g.
combines the older rotated logs with the latest ones. Also, it easily
allows me to easily specify that I want just the logs of this month or
of just this one boot, or of just some specific service.

If I was writing tools that'd automatically handle the logs, I think I
would also benefit from the additional data that journal stores that
is usually not available in a syslog formatted log. Having to use e.g.
the journal API would of course be some burden, but I can imagine it
being nicer than having to e.g. parse all the different date formats
that a text-formatted log could have. Or having to handle all of the
other things that may or may not be in the syslog lines, in various
formats.

Of course there are still bugs and the other issues in the new tools,
but they certainly aren't there in order to cause nightmares. I am
hopeful that the issues can be and are fixed. For my (relatively
simple) setups, there aren't any major showshoppers, though.

[1] http://www.freedesktop.org/wiki/Software/systemd/journal-files/

-Joonas

[Suvayu Ali]

On Mon, Nov 18, 2013 at 07:31:21PM +0200, Joonas Sarajärvi wrote:
>  2013/11/18 Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;:
>  > Hi Jonas,
>  >
>  > I have a comment, and a question.
>  >
>  > On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
>  >> The journald log format is documented at least to some extent [1], and
>  >> there exists free software for reading the log. To me, it sounds like
>  >> way more accessible than if it was a binary data format of a typical
>  >> proprietary tool. For example, booting any Fedora live image should
>  >> suffice if you need to read the journal of a system that uses journald
>  >> and happens to become unbootable.
>  >
>  > I am sure you (and everyone else on the list) will agree, tools for
>  > viewing plain text (cat, more, less, <favourite editor>) outnumber and
>  > are available on every platform compared to any specialised tool.  Even
>  > if a format is open, it still needs tools that support it.  I would not
>  > count on having a gui when my system crashes.  Sometimes the only access
>  > you have to a crashed system is the recovery prompt.
>  I wonder where this notion that you'd need a GUI to access the journal
>  content came from. You do need a working journalctl, but it is not a
>  GUI tool. Normally it outputs syslog-like output that you could feed
>  to your favourite syslog-output-expecting log analysis tool. 
Well the gui was mentioned in the thread, hence my mention.  And as you
see from later in my message, I have used journalctl, but
unsuccessfully.  I looked at the dependencies for journalctl, compared
to less or cat, it is rather long.  On a crashed system I wouldn't trust
to have access to all these linked libraries.

>  > I have been very frustrated with journalctl.  The manual page is
> very
>  > unhelpful in that regard.  For example the other day, I wanted to
>  > investigate why my laptop shutdown suddenly (I think it was
>  > overheating), but there was no reasonable way for me to filter the cpu
>  > specific messages.  Could you give some pointers how I can do that?  I
>  > would be very grateful.
>  I am not sure where that message in particular would show up. My first
>  instinct in such a situation would be to just run journalctl without
>  arguments. If the output is not piped anywhere, journalctl opens less
>  and puts the syslog-like output there. Now it is pretty same as if I
>  had run less /var/log/messages in a system that uses a traditional
>  syslog. I can for example press / to access the search function of
>  less and search for the "cpu" string.
>  
>  Since a hardware related error would likely be logged by the kernel, I
>  might invoke journalctl like this:
>  journalctl _TRANSPORT=kernel 
This helped, thank you!

>  This would limit the things I get so that only kernel debug messages
>  are shown. Different filter criteria you could use are described in
>  the systemd.journal-fields man page. 
I looked at that.  But was not clear what are valid values for the
fields.  Your and Tom's responses have given me enough pointers now to
investigate more.

>  
>  Another useful parameter would be the --since parameter, e.g. like this:
>  journalctl --since 2013-11-15 _TRANSPORT=kernel 
This is also very helpful, thank you.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

[Suvayu Ali]

On Mon, Nov 18, 2013 at 07:44:55PM +0200, Joonas Sarajärvi wrote:
>  Sorry for posting again so soon and replying to myself, but I just
>  noticed one very useful thing that might help quite much in the
>  specific problem you described:
>  
>  2013/11/18 Joonas Sarajärvi <muep(a)iki.fi&gt;:
>  > 2013/11/18 Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;:
>  >> the other day, I wanted to
>  >> investigate why my laptop shutdown suddenly (I think it was
>  >> overheating), but there was no reasonable way for me to filter the cpu
>  >> specific messages.
>  > I am not sure where that message in particular would show up.
>  
>  Likely you would not want to look for a specific subsystem anyway
>  Often there are many things in the system interacting and it is nicer
>  to look at the whole log in the correct point in time. One nice filter
>  criteria is to filter by boot id with the --boot option of journalctl
>  
>  journalctl --boot
>  
>  The above would show just the messages that were gathered during the
>  currently ongoing boot cycle. Now if you want messages of the previous
>  boot, --boot allows you to specify an offset. So if you pass -1 there
>  like this, it would show you messages from the previous boot cycle:
>  journalctl --boot=-1 
I had seen that option, but could not figure out how do I get a boot id.
Now that I know it is an offset, it will be very useful indeed.

>  In some cases like a kernel panic, you will likely not have the
> kernel
>  panic message in the log. This is because usually at the point where
>  the kernel has the message, it will not write anything to disk
>  anymore. If there is nothing interesting in the end of that boot, you
>  should also check if the following boot has a note about the journal
>  being rotated due to not being properly closed. This would at least
>  let you know that you miss some messages. 
Again, very good suggestion!  Thanks a lot.

I have to say something though.  Somehow I feel most of Lennart
Poettering's projects (systemd, pulseaudio) have terribly impenetrable
documentation.  When reading them it feels like it is meant for other
developers littered with developer jargon.  And I'm no stranger to *nix
docs; I know my way around `man bash', for example.

-- 
Suvayu

Open source is the future. It sets us free.

[Suvayu Ali]

On Mon, Nov 18, 2013 at 05:29:27PM +0000, Tom H wrote:
>  On Mon, Nov 18, 2013 at 4:41 PM, Suvayu Ali
> <fatkasuvayu+linux(a)gmail.com&gt; wrote:
>  > On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
>  >> On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;
> wrote:
>  >>>
>  >>> I have been very frustrated with journalctl. The manual page is very
>  >>> unhelpful in that regard. For example the other day, I wanted to
>  >>> investigate why my laptop shutdown suddenly (I think it was
>  >>> overheating), but there was no reasonable way for me to filter the cpu
>  >>> specific messages. Could you give some pointers how I can do that? I
>  >>> would be very grateful.
>  >>
>  >> Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
>  >>
>  >> acpi for power?
>  >
>  > That does filter the output, but not the messages I was looking for.
>  > Looking at /var/log/messages tells me the lines should be something like
>  > this:
>  >
>  > kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled
> (total events = 38)
>  >
>  > But I still can't find where it says shutting down. Anyway, what
>  > subsystem is the above? Also how can I limit it to kernel messages from
>  > all subsystems?
>  >
>  > Thanks for your response,
>  
>  You're welcome.
>  
>  "acpi" had been a guess!
>  
>  I scratched my head, read the journalctl man page, and found "-F":
>  
>  # journalctl -F _KERNEL_SUBSYSTEM
>  platform
>  scsi
>  pnp
>  pci
>  pci_bus
>  acpi 
On my laptop I have a few more:

  $ journalctl -F _KERNEL_SUBSYSTEM
  ieee80211
  serio
  scsi
  pnp
  pci
  pci_bus
  acpi
  hid
  usb
  thermal

`thermal' does the trick :-p.

  $ journalctl -r _KERNEL_SUBSYSTEM=thermal
  -- Logs begin at Fri 2013-09-20 23:52:53 CEST, end at Tue 2013-11-19 00:45:01 CET. --
  Nov 14 10:08:54 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical
temperature reached(100 C),shutting down
  Nov 14 10:08:54 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical
temperature reached(100 C),shutting down
  -- Reboot --
  Nov 14 09:56:17 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical
temperature reached(100 C),shutting down
  Nov 14 09:56:17 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical
temperature reached(100 C),shutting down

Wallah!

-- 
Suvayu

Open source is the future. It sets us free.

[Olav Vitters]

On Mon, Nov 18, 2013 at 04:48:32PM +1030, Tim wrote:
>  Tim:
>  >> And if logs are in a format that you cannot read, you cannot safely
>  >> submit them to an outside server.  You don't know what they contain.
>  >> Logon credentials, confidential data that you're working on, etc.
>  
>  Patrick Lists:
>  > IIRC that's the reason why journald supports encryption. I don't
>  > recall the link but there's a blog post somewhere (on redhat.com?)
>  > where the reasons for moving to journald were outlined. Might be worth
>  > a search if you want to know more. 
>  
>  If my logs might contain confidential data, there's no way I'm
>  submitting them to something else to assess (e.g. bug reports).
>  Encrypting isn't the issue, it's (me) being unable to tell what's in
>  them, nor edit the confidential stuff out of it beforehand. 
You can read it with journalctl. You can have syslog running beside it.
You can still inspect it with strings (because there isn't any
compression). You can forward it to another server (like syslog) or just
use syslog itself.

AFAIK there is no encryption in the journal, only Forward Secure Sealing
(FSS), where things are hashed. See man journalctl for more information
on that.

-- 
Regards,
Olav

[Alexander Aguilar Torrico]

Oxl
El nov 15, 2013 6:46 a.m., "Frantisek Hanzlik" <franta(a)hanzlici.cz&gt;
escribió:

>  For one thing I'm in the conviction that binary logs are
> hazardous
>  bullshit, for another on my two F19 machines systemd-journald occupy
>  significant part of resources (after several days it is often 1.5 to
>  2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
>
>  Then, how I can avoid this crap and log only to standard rsyslogd?
>
>  I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
>  /etc/systemd/{system.conf,user.conf}, but this not helped; setting
>  "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf
>  is better, but I want ideally completely cut out systemd-journald
>  from my systems.
>
>  TIA, Franta Hanzlik
>  --
>  users mailing list
>  users(a)lists.fedoraproject.org
>  To unsubscribe or change subscription options:
>  https://admin.fedoraproject.org/mailman/listinfo/users
>  Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
>  Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
>  Have a question? Ask away: http://ask.fedoraproject.org
>

[Joonas Sarajärvi]

Sorry for posting again so soon and replying to myself, but I just
noticed one very useful thing that might help quite much in the
specific problem you described:

2013/11/18 Joonas Sarajärvi <muep(a)iki.fi&gt;:
>  2013/11/18 Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;:
> > the other day, I wanted to
> > investigate why my laptop shutdown suddenly (I think it was
> > overheating), but there was no reasonable way for me to filter the cpu
> > specific messages.
>  I am not sure where that message in particular would show up. 
Likely you would not want to look for a specific subsystem anyway
Often there are many things in the system interacting and it is nicer
to look at the whole log in the correct point in time. One nice filter
criteria is to filter by boot id with the --boot option of journalctl

journalctl --boot

The above would show just the messages that were gathered during the
currently ongoing boot cycle. Now if you want messages of the previous
boot, --boot allows you to specify an offset. So if you pass -1 there
like this, it would show you messages from the previous boot cycle:
journalctl --boot=-1

If you already booted many times, just grow the number after the minus
sign to find the one that ended before you intended it to. Then just
scroll to the end (the > key usually does this in less) to see what
were the last messages that were recorded before shutdown.

In some cases like a kernel panic, you will likely not have the kernel
panic message in the log. This is because usually at the point where
the kernel has the message, it will not write anything to disk
anymore. If there is nothing interesting in the end of that boot, you
should also check if the following boot has a note about the journal
being rotated due to not being properly closed. This would at least
let you know that you miss some messages.

-Joonas

[Tom H]

On Mon, Nov 18, 2013 at 4:41 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt; wrote:
>  On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
> > On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;
> wrote:
> >>
> >> I have been very frustrated with journalctl. The manual page is very
> >> unhelpful in that regard. For example the other day, I wanted to
> >> investigate why my laptop shutdown suddenly (I think it was
> >> overheating), but there was no reasonable way for me to filter the cpu
> >> specific messages. Could you give some pointers how I can do that? I
> >> would be very grateful.
> >
> > Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
> >
> > acpi for power?
>
>  That does filter the output, but not the messages I was looking for.
>  Looking at /var/log/messages tells me the lines should be something like
>  this:
>
>  kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled
> (total events = 38)
>
>  But I still can't find where it says shutting down. Anyway, what
>  subsystem is the above? Also how can I limit it to kernel messages from
>  all subsystems?
>
>  Thanks for your response, 
You're welcome.

"acpi" had been a guess!

I scratched my head, read the journalctl man page, and found "-F":

# journalctl -F _KERNEL_SUBSYSTEM
platform
scsi
pnp
pci
pci_bus
acpi

[Suvayu Ali]

On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
>  On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali
> <fatkasuvayu+linux(a)gmail.com&gt; wrote:
>  >
>  > I have been very frustrated with journalctl. The manual page is very
>  > unhelpful in that regard. For example the other day, I wanted to
>  > investigate why my laptop shutdown suddenly (I think it was
>  > overheating), but there was no reasonable way for me to filter the cpu
>  > specific messages. Could you give some pointers how I can do that? I
>  > would be very grateful.
>  
>  Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
>  
>  acpi for power? 
That does filter the output, but not the messages I was looking for.
Looking at /var/log/messages tells me the lines should be something like
this:

 kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled
(total events = 38)

But I still can't find where it says shutting down.  Anyway, what
subsystem is the above?  Also how can I limit it to kernel messages from
all subsystems?

Thanks for your response,

-- 
Suvayu

Open source is the future. It sets us free.

[Tom H]

On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt; wrote:
>  I have been very frustrated with journalctl. The manual page is very
>  unhelpful in that regard. For example the other day, I wanted to
>  investigate why my laptop shutdown suddenly (I think it was
>  overheating), but there was no reasonable way for me to filter the cpu
>  specific messages. Could you give some pointers how I can do that? I
>  would be very grateful. 
Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"

acpi for power?

[Rahul Sundaram]

Hi


On Sun, Nov 17, 2013 at 5:39 PM, Joe Zeff  wrote:


>  That's nice, although I'd like to have a CLI reader as well
> if there isn't
>  one already.
>  
There is and it has been mentioned before.  Use journalctl.

Rahul

[Olav Vitters]

On Sat, Nov 16, 2013 at 10:44:21PM +0100, Frantisek Hanzlik wrote:
>  I agree, all systemd stuff seems very crude for me.
>  And 'enough time' to debug or help to fix systemd monster I really not
>  have, I even don't have time to report all bugs which happens to me.
>  Thus I must only wish to systemd developers to finish their work to
>  some usable state earlier than these 10 years. 
Why not at least report one bug if you encounter so many of them?

-- 
Regards,
Olav

[Olav Vitters]

On Sun, Nov 17, 2013 at 12:46:40PM -0800, Joe Zeff wrote:
>  On 11/17/2013 12:32 PM, Frantisek Hanzlik wrote:
>  >I'm convinced it is as similar things on eg. ms windows (binary logs,
>  >registry,...) - when is all working fine, then it look fine too.
>  
>  Yes.  When there's a major crash, you may not have access to the
>  fancy programs for reading binary logs, especially if they're GUI
>  only, and you're stuck in CLI mode.  And once you have a binary
>  database of all of your system info, the temptation to let
>  non-system applications use it gets to be irresistible, and we all
>  know what that leads to.  There are some things Windows does that
>  are worth copying; the Registry isn't one of them. 
A GUI for the journal is already being developed.
https://wiki.gnome.org/Apps/Logs

Anyway, seems good to have easy access to show the various log and error
output of programs in an easy way. More discoverable for users.

-- 
Regards,
Olav

[Natxo Asenjo]

On Sun, Nov 17, 2013 at 12:12 PM, Patrick Lists
<fedora-list(a)puzzled.xs4all.nl&gt; wrote:
>  On 11/17/2013 11:21 AM, Tim wrote:
> > Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
> >> Binary logs, by contrast, may be useless when log file is damaged or I
> >> haven't this one unique utility for reading them. And my experiences
> >> with systems where binary logs are implemented says clearly that
> >> binary logs is bad idea.
> >
> > And if logs are in a format that you cannot read, you cannot safely
> > submit them to an outside server.  You don't know what they contain.
> > Logon credentials, confidential data that you're working on, etc.
>
>  IIRC that's the reason why journald supports encryption. I don't recall
>  the link but there's a blog post somewhere (on redhat.com?) where the
>  reasons for moving to journald were outlined. Might be worth a search if
>  you want to know more. 
syslog (using rsyslog) already supports encryption of you require that
(even kerberos if that is what you want, we do). Journald is a
solution in search of a problem ;-)

[Frank Murphy]

On Sun, 17 Nov 2013 12:37:03 +0100
Patrick Lists <fedora-list(a)puzzled.xs4all.nl&gt; wrote:

>  
>  I meant the transmission of the log to another log server. Not the
>  log itself. Anyway, here is Lennart Poettering's rationale behind
>  journald:
>   
One of the things you will see after a hard reset is:
Journal corrupted, journal deleted and being restarted.
ie.. no useful info.


-- 
Regards,
Frank 
www.frankly3d.com

[Patrick Lists]

On 11/17/2013 12:16 PM, Frank Murphy wrote:
>  On Sun, 17 Nov 2013 12:12:08 +0100
>  Patrick Lists <fedora-list(a)puzzled.xs4all.nl&gt; wrote:
>  
> > IIRC that's the reason why journald supports encryption. 
>  
>  It's not encrypted it's binary, 
>  similar to any compiled app or virus. 
I meant the transmission of the log to another log server. Not the log
itself. Anyway, here is Lennart Poettering's rationale behind journald:

https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgj...

Regards,
Patrick

[Tim]

Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
>  Binary logs, by contrast, may be useless when log file is damaged or
> I
>  haven't this one unique utility for reading them. And my experiences
>  with systems where binary logs are implemented says clearly that
>  binary logs is bad idea.  
And if logs are in a format that you cannot read, you cannot safely
submit them to an outside server.  You don't know what they contain.
Logon credentials, confidential data that you're working on, etc.

-- 
[tim@localhost ~]$ uname -rsvp
Linux 3.9.10-100.fc17.x86_64 #1 SMP Sun Jul 14 01:31:27 UTC 2013 x86_64

All mail to my mailbox is automatically deleted, there is no point
trying to privately email me, I will only read messages posted to the
public lists.

George Orwell's '1984' was supposed to be a warning against tyranny, not
a set of instructions for supposedly democratic governments.

[Franta Hanzlík]

Steven Stern wrote:
>  On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
> > For one thing I'm in the conviction that binary logs are hazardous
> > bullshit,
>  
>  In what way might the logs be hazardous? 
It was mean mainly from administrator view. When things go bad,
machine HW/SW fail or any other disasters occurs, logs are very
valuable. And I'm confident that binary logs are too weak in this
situation. Text logs are useful even if log file is damaged or ends
with fragments and can be easy readable with lot of tools. Binary
logs, by contrast, may be useless when log file is damaged or I
haven't this one unique utility for reading them. And my experiences
with systems where binary logs are implemented says clearly that
binary logs is bad idea.
Second, it is question when tight integration of systemd and logging
services has any benefits - there is number of situation (logging
over network, for example) which speaks for separate logging service.
And possibilities with e.g. rsyslogd are better than with journald
- why Fedora must again replace verified, reliable, Unix standard
things with some crappy solutions? For nightmares of its users?

Franta Hanzlik

[T.C. Hollingsworth]

On Sat, Nov 16, 2013 at 11:18 AM, Frantisek Hanzlik <franta(a)hanzlici.cz&gt; wrote:
>  I tried delete '/usr/lib/systemd/systemd-journald', but this
> may not be
>  called clean solution. And after that, several services (e.g. sendmail,
>  dovecot, sshd - what I shortly browse) stop logging via rsyslogd. On
>  other hand, other services log to rsyslogd well. 
The journal service is responsible for consuming stdout/stderr from
daemons stared with systemd and forwarding it to syslog.  If you
disable it, you lose logs from daemons that rely on this
functionality.  If you insist on disabling the journal service, you'll
need to configure such daemons to log to syslog themselves instead of
relying on the journal to take care of it for them.

Additionally, some daemons speak the journal protocol natively.  This
obviously doesn't work if there is no journal service.  If you insist
on disabling the journal service, you must configure such daemons to
log to syslog instead of logging to the journal.

-T.C.

[SternData]

On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
>  For one thing I'm in the conviction that binary logs are
> hazardous
>  bullshit, 
In what way might the logs be hazardous?

-- 
-- Steve

[Franta Hanzlík]

Mateusz Marzantowicz wrote:
>  On 15.11.2013 11:46, Frantisek Hanzlik wrote:
> > For one thing I'm in the conviction that binary logs are hazardous
> > bullshit, for another on my two F19 machines systemd-journald occupy
> > significant part of resources (after several days it is often 1.5 to
> > 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
> >
> > Then, how I can avoid this crap and log only to standard rsyslogd?
> >
> > I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
> > /etc/systemd/{system.conf,user.conf}, but this not helped; setting
> > "Storage=none" in "[Journal]" section of
> /etc/systemd/journald.conf
> > is better, but I want ideally completely cut out systemd-journald
> > from my systems.
> >
> > TIA, Franta Hanzlik
> >
>  
>  You can set restrictions on resources used by journald. Reducing disk
>  space is one of them. Take a look at /etc/systemd/journald.conf, but I
>  suppose you already know that.
>  
>  High memory usage might be caused by leaks and it's nothing unexpected
>  in such immature application. You can help to fix it if you have enough
>  time. After 10 years of testing in production maybe journald would reach
>  quality of other syslogd implementations. By that time it should be
>  considered not reliable. It has great potential but needs more testing.
>  
>  Of course this doesn't solve your first issue - hazardous binary logs :-)
>  
>  Mateusz Marzantowicz 
I agree, all systemd stuff seems very crude for me.
And 'enough time' to debug or help to fix systemd monster I really not
have, I even don't have time to report all bugs which happens to me.
Thus I must only wish to systemd developers to finish their work to
some usable state earlier than these 10 years.

Thanks, Franta Hanzlik

[Franta Hanzlík]

Rick Stevens wrote:
>  On 11/15/2013 02:46 AM, Frantisek Hanzlik issued this missive:
> > For one thing I'm in the conviction that binary logs are hazardous
> > bullshit, for another on my two F19 machines systemd-journald occupy
> > significant part of resources (after several days it is often 1.5 to
> > 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
> >
> > Then, how I can avoid this crap and log only to standard rsyslogd?
> >
> > I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
> > /etc/systemd/{system.conf,user.conf}, but this not helped; setting
> > "Storage=none" in "[Journal]" section of
> /etc/systemd/journald.conf
> > is better, but I want ideally completely cut out systemd-journald
> > from my systems.
>  
>  Uhm,
>  
>      # systemctl stop systemd-journald.service
>      # systemctl disable systemd-journald.service
>   
'systemctl list-unit-files|grep journald\.service' list systemd-journald
service as static, and it seems as these cannot be disabled or masked.

I tried delete '/usr/lib/systemd/systemd-journald', but this may not be
called clean solution. And after that, several services (e.g. sendmail,
dovecot, sshd - what I shortly browse) stop logging via rsyslogd. On
other hand, other services log to rsyslogd well.

Franta Hanzlik

[Mateusz Marzantowicz]

On 15.11.2013 11:46, Frantisek Hanzlik wrote:
>  For one thing I'm in the conviction that binary logs are
> hazardous
>  bullshit, for another on my two F19 machines systemd-journald occupy
>  significant part of resources (after several days it is often 1.5 to
>  2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
>  
>  Then, how I can avoid this crap and log only to standard rsyslogd?
>  
>  I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
>  /etc/systemd/{system.conf,user.conf}, but this not helped; setting
>  "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf
>  is better, but I want ideally completely cut out systemd-journald
>  from my systems.
>  
>  TIA, Franta Hanzlik
>   
You can set restrictions on resources used by journald. Reducing disk
space is one of them. Take a look at /etc/systemd/journald.conf, but I
suppose you already know that.

High memory usage might be caused by leaks and it's nothing unexpected
in such immature application. You can help to fix it if you have enough
time. After 10 years of testing in production maybe journald would reach
quality of other syslogd implementations. By that time it should be
considered not reliable. It has great potential but needs more testing.

Of course this doesn't solve your first issue - hazardous binary logs :-)


Mateusz Marzantowicz

[Rick Stevens]

On 11/15/2013 02:46 AM, Frantisek Hanzlik issued this missive:
>  For one thing I'm in the conviction that binary logs are
> hazardous
>  bullshit, for another on my two F19 machines systemd-journald occupy
>  significant part of resources (after several days it is often 1.5 to
>  2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
>
>  Then, how I can avoid this crap and log only to standard rsyslogd?
>
>  I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of
>  /etc/systemd/{system.conf,user.conf}, but this not helped; setting
>  "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf
>  is better, but I want ideally completely cut out systemd-journald
>  from my systems. 
Uhm,

	# systemctl stop systemd-journald.service
	# systemctl disable systemd-journald.service

----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-      Batteries not included.  Offer not valid in some states.      -
-           Your mileage may vary.  Void where prohibited.           -
----------------------------------------------------------------------

[Frank Murphy]

On Sun, 17 Nov 2013 12:12:08 +0100
Patrick Lists <fedora-list(a)puzzled.xs4all.nl&gt; wrote:

>  IIRC that's the reason why journald supports encryption.  
It's not encrypted it's binary, 
similar to any compiled app or virus.

-- 
Regards,
Frank 
www.frankly3d.com

[Joe Zeff]

On 11/17/2013 02:18 PM, Olav Vitters wrote:
>  A GUI for the journal is already being developed.
>  https://wiki.gnome.org/Apps/Logs 
That's nice, although I'd like to have a CLI reader as well if there 
isn't one already.

[Franta Hanzlík]

Frank Murphy wrote:
>  On Sun, 17 Nov 2013 12:37:03 +0100
>  Patrick Lists <fedora-list(a)puzzled.xs4all.nl&gt; wrote:
>  
> >
> > I meant the transmission of the log to another log server. Not the
> > log itself. Anyway, here is Lennart Poettering's rationale behind
> > journald:
> >
>  
>  One of the things you will see after a hard reset is:
>  Journal corrupted, journal deleted and being restarted.
>  ie.. no useful info. 
This is perhaps most significant shortcoming. And beside it I see in
log messages as:

systemd-journald[295]: Failed to set ACL on
/var/log/journal/70ed4dbe694041d484165719514f195e/user-1045.journal, ignoring: Invalid
argument
systemd-journald[2948]: Failed to write entry, ignoring: Cannot allocate memory
systemd-journal[388]: Suppressed 966 messages from /system/dbus.service
systemd-journal[299]: Forwarding to syslog missed 92 messages.

which induce sense of journald logging futility.

[Tim]

Tim:
> > And if logs are in a format that you cannot read, you cannot
> safely
> > submit them to an outside server.  You don't know what they contain.
> > Logon credentials, confidential data that you're working on, etc. 
Patrick Lists:
>  IIRC that's the reason why journald supports encryption. I
> don't
>  recall the link but there's a blog post somewhere (on redhat.com?)
>  where the reasons for moving to journald were outlined. Might be worth
>  a search if you want to know more.  
If my logs might contain confidential data, there's no way I'm
submitting them to something else to assess (e.g. bug reports).
Encrypting isn't the issue, it's (me) being unable to tell what's in
them, nor edit the confidential stuff out of it beforehand.

-- 
[tim@localhost ~]$ uname -rsvp
Linux 3.9.10-100.fc17.x86_64 #1 SMP Sun Jul 14 01:31:27 UTC 2013 x86_64

All mail to my mailbox is automatically deleted, there is no point
trying to privately email me, I will only read messages posted to the
public lists.

George Orwell's '1984' was supposed to be a warning against tyranny, not
a set of instructions for supposedly democratic governments.

[Joe Zeff]

On 11/17/2013 12:32 PM, Frantisek Hanzlik wrote:
>  I'm convinced it is as similar things on eg. ms windows (binary
> logs,
>  registry,...) - when is all working fine, then it look fine too. 
Yes.  When there's a major crash, you may not have access to the fancy 
programs for reading binary logs, especially if they're GUI only, and 
you're stuck in CLI mode.  And once you have a binary database of all of 
your system info, the temptation to let non-system applications use it 
gets to be irresistible, and we all know what that leads to.  There are 
some things Windows does that are worth copying; the Registry isn't one 
of them.

[Franta Hanzlík]

Joonas Sarajärvi wrote:
>  2013/11/17 Frantisek Hanzlik <franta(a)hanzlici.cz&gt;:
> > Steven Stern wrote:
> >> On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
> >>> For one thing I'm in the conviction that binary logs are hazardous
> >>> bullshit,
> >>
> >> In what way might the logs be hazardous?
> >
> > It was mean mainly from administrator view. When things go bad,
> > machine HW/SW fail or any other disasters occurs, logs are very
> > valuable. And I'm confident that binary logs are too weak in this
> > situation. Text logs are useful even if log file is damaged or ends
> > with fragments and can be easy readable with lot of tools. Binary
> > logs, by contrast, may be useless when log file is damaged or I
> > haven't this one unique utility for reading them. And my experiences
> > with systems where binary logs are implemented says clearly that
> > binary logs is bad idea.
> > Second, it is question when tight integration of systemd and logging
> > services has any benefits - there is number of situation (logging
> > over network, for example) which speaks for separate logging service.
>  
>  The journald log format is documented at least to some extent [1], and
>  there exists free software for reading the log. To me, it sounds like
>  way more accessible than if it was a binary data format of a typical
>  proprietary tool. For example, booting any Fedora live image should
>  suffice if you need to read the journal of a system that uses journald
>  and happens to become unbootable.
>  
>  Typically journalctl will generate you a representation of the data in
>  syslog log file format. This loses some infromation that the journal
>  stores, because there is no way to represent it all while keeping the
>  output syslog-like and easily human-readable. This output can be
>  easily fed to traditional unix tools like grep, if that is your
>  preferred way of extracting information from logs.
>  
>  Finally, since you can also run a normal syslog daemon which maintains
>  a text-format logs for you, I do not really see why having some log
>  data in journald format under /run/log/journal/ should be considered
>  hazardous. You can pretty much just ignore it if you feel like using
>  other kinds of tools for storing and managing logs.
>  
> > And possibilities with e.g. rsyslogd are better than with journald
> > - why Fedora must again replace verified, reliable, Unix standard
> > things with some crappy solutions? For nightmares of its users?
>  At least for my needs, the journal has been way more convenient to use
>  than rsyslog. It is much nicer to read logs when journalctl e.g.
>  combines the older rotated logs with the latest ones. Also, it easily
>  allows me to easily specify that I want just the logs of this month or
>  of just this one boot, or of just some specific service.
>  
>  If I was writing tools that'd automatically handle the logs, I think I
>  would also benefit from the additional data that journal stores that
>  is usually not available in a syslog formatted log. Having to use e.g.
>  the journal API would of course be some burden, but I can imagine it
>  being nicer than having to e.g. parse all the different date formats
>  that a text-formatted log could have. Or having to handle all of the
>  other things that may or may not be in the syslog lines, in various
>  formats.
>  
>  Of course there are still bugs and the other issues in the new tools,
>  but they certainly aren't there in order to cause nightmares. I am
>  hopeful that the issues can be and are fixed. For my (relatively
>  simple) setups, there aren't any major showshoppers, though.
>  
>  [1] http://www.freedesktop.org/wiki/Software/systemd/journal-files/
>  
>  -Joonas 
I'm convinced it is as similar things on eg. ms windows (binary logs,
registry,...) - when is all working fine, then it look fine too.
But when machine somehow fail, then this binary mud is serious problem.
And I do not want to wait to time, when for some similar failed crap
I have to reinstall Linux box. Until now I never needed it, no matter
how significant Linux box crashes were...
Franta Hanzlik

[Patrick Lists]

On 11/17/2013 11:21 AM, Tim wrote:
>  Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
> > Binary logs, by contrast, may be useless when log file is damaged or I
> > haven't this one unique utility for reading them. And my experiences
> > with systems where binary logs are implemented says clearly that
> > binary logs is bad idea. 
>  
>  And if logs are in a format that you cannot read, you cannot safely
>  submit them to an outside server.  You don't know what they contain.
>  Logon credentials, confidential data that you're working on, etc. 
IIRC that's the reason why journald supports encryption. I don't recall
the link but there's a blog post somewhere (on redhat.com?) where the
reasons for moving to journald were outlined. Might be worth a search if
you want to know more.

Regards,
Patrick

[Suvayu Ali]

Hi Jonas,

I have a comment, and a question.

On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
>  2013/11/17 Frantisek Hanzlik <franta(a)hanzlici.cz&gt;:
>  >
>  > It was mean mainly from administrator view. When things go bad,
>  > machine HW/SW fail or any other disasters occurs, logs are very
>  > valuable. And I'm confident that binary logs are too weak in this
>  > situation. Text logs are useful even if log file is damaged or ends
>  > with fragments and can be easy readable with lot of tools. Binary
>  > logs, by contrast, may be useless when log file is damaged or I
>  > haven't this one unique utility for reading them. And my experiences
>  > with systems where binary logs are implemented says clearly that
>  > binary logs is bad idea.
>  > Second, it is question when tight integration of systemd and logging
>  > services has any benefits - there is number of situation (logging
>  > over network, for example) which speaks for separate logging service.
>  
>  The journald log format is documented at least to some extent [1], and
>  there exists free software for reading the log. To me, it sounds like
>  way more accessible than if it was a binary data format of a typical
>  proprietary tool. For example, booting any Fedora live image should
>  suffice if you need to read the journal of a system that uses journald
>  and happens to become unbootable. 
I am sure you (and everyone else on the list) will agree, tools for
viewing plain text (cat, more, less, <favourite editor>) outnumber and
are available on every platform compared to any specialised tool.  Even
if a format is open, it still needs tools that support it.  I would not
count on having a gui when my system crashes.  Sometimes the only access
you have to a crashed system is the recovery prompt.

 [...chomp...chomp...chomp...]

>  At least for my needs, the journal has been way more convenient to
> use
>  than rsyslog. It is much nicer to read logs when journalctl e.g.
>  combines the older rotated logs with the latest ones. Also, it easily
>  allows me to easily specify that I want just the logs of this month or
>  of just this one boot, or of just some specific service. 
I have been very frustrated with journalctl.  The manual page is very
unhelpful in that regard.  For example the other day, I wanted to
investigate why my laptop shutdown suddenly (I think it was
overheating), but there was no reasonable way for me to filter the cpu
specific messages.  Could you give some pointers how I can do that?  I
would be very grateful.

Thanks,

PS: I'm no sysadmin, just a regular *nix user who prefers the command line.

-- 
Suvayu

Open source is the future. It sets us free.

[Joonas Sarajärvi]

2013/11/18 Suvayu Ali <fatkasuvayu+linux(a)gmail.com&gt;:
>  Hi Jonas,
>
>  I have a comment, and a question.
>
>  On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
> > The journald log format is documented at least to some extent [1], and
> > there exists free software for reading the log. To me, it sounds like
> > way more accessible than if it was a binary data format of a typical
> > proprietary tool. For example, booting any Fedora live image should
> > suffice if you need to read the journal of a system that uses journald
> > and happens to become unbootable.
>
>  I am sure you (and everyone else on the list) will agree, tools for
>  viewing plain text (cat, more, less, <favourite editor>) outnumber and
>  are available on every platform compared to any specialised tool.  Even
>  if a format is open, it still needs tools that support it.  I would not
>  count on having a gui when my system crashes.  Sometimes the only access
>  you have to a crashed system is the recovery prompt. 
I wonder where this notion
that you'd need a GUI to access the journal
content came from. You do need a working journalctl, but it is not a
GUI tool. Normally it outputs syslog-like output that you could feed
to your favourite syslog-output-expecting log analysis tool.

> > At least for my needs, the journal has been way more convenient
> to use
> > than rsyslog. It is much nicer to read logs when journalctl e.g.
> > combines the older rotated logs with the latest ones. Also, it easily
> > allows me to easily specify that I want just the logs of this month or
> > of just this one boot, or of just some specific service.
>
>  I have been very frustrated with journalctl.  The manual page is very
>  unhelpful in that regard.  For example the other day, I wanted to
>  investigate why my laptop shutdown suddenly (I think it was
>  overheating), but there was no reasonable way for me to filter the cpu
>  specific messages.  Could you give some pointers how I can do that?  I
>  would be very grateful. 
I am not sure where that message in particular would show
up. My first
instinct in such a situation would be to just run journalctl without
arguments. If the output is not piped anywhere, journalctl opens less
and puts the syslog-like output there. Now it is pretty same as if I
had run less /var/log/messages in a system that uses a traditional
syslog. I can for example press / to access the search function of
less and search for the "cpu" string.

Since a hardware related error would likely be logged by the kernel, I
might invoke journalctl like this:
journalctl _TRANSPORT=kernel

This would limit the things I get so that only kernel debug messages
are shown. Different filter criteria you could use are described in
the systemd.journal-fields man page.

Another useful parameter would be the --since parameter, e.g. like this:
journalctl --since 2013-11-15 _TRANSPORT=kernel

Now journalctl would omit old stuff that likely would not be
interesting in this case.

But maybe the message is not in the kernel log, and you'd wish to use
grep to search for the message like you could with a text-based
syslog? You can like this:
journalctl --since 2013-11-15 | grep -i cpu

The --since parameter is still useful, in case you happen to have many
months of log data stored in the journal. With the --since and --until
parameters, you can limit the search to any arbitrary interval that is
covered by your stored log data, regardless of how those times relate
to the times at which log rotation has happened.

Hopefully this helps. At least you should know that you can still use
your knowledge of the traditional unixy text-processing tools.

-Joonas