For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
TIA, Franta Hanzlik
On 11/15/2013 02:46 AM, Frantisek Hanzlik issued this missive:
For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
Uhm,
# systemctl stop systemd-journald.service # systemctl disable systemd-journald.service
---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Batteries not included. Offer not valid in some states. - - Your mileage may vary. Void where prohibited. - ----------------------------------------------------------------------
Rick Stevens wrote:
On 11/15/2013 02:46 AM, Frantisek Hanzlik issued this missive:
For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
Uhm,
# systemctl stop systemd-journald.service # systemctl disable systemd-journald.service
'systemctl list-unit-files|grep journald.service' list systemd-journald service as static, and it seems as these cannot be disabled or masked.
I tried delete '/usr/lib/systemd/systemd-journald', but this may not be called clean solution. And after that, several services (e.g. sendmail, dovecot, sshd - what I shortly browse) stop logging via rsyslogd. On other hand, other services log to rsyslogd well.
Franta Hanzlik
On Sat, Nov 16, 2013 at 11:18 AM, Frantisek Hanzlik franta@hanzlici.cz wrote:
I tried delete '/usr/lib/systemd/systemd-journald', but this may not be called clean solution. And after that, several services (e.g. sendmail, dovecot, sshd - what I shortly browse) stop logging via rsyslogd. On other hand, other services log to rsyslogd well.
The journal service is responsible for consuming stdout/stderr from daemons stared with systemd and forwarding it to syslog. If you disable it, you lose logs from daemons that rely on this functionality. If you insist on disabling the journal service, you'll need to configure such daemons to log to syslog themselves instead of relying on the journal to take care of it for them.
Additionally, some daemons speak the journal protocol natively. This obviously doesn't work if there is no journal service. If you insist on disabling the journal service, you must configure such daemons to log to syslog instead of logging to the journal.
-T.C.
On 15.11.2013 11:46, Frantisek Hanzlik wrote:
For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
TIA, Franta Hanzlik
You can set restrictions on resources used by journald. Reducing disk space is one of them. Take a look at /etc/systemd/journald.conf, but I suppose you already know that.
High memory usage might be caused by leaks and it's nothing unexpected in such immature application. You can help to fix it if you have enough time. After 10 years of testing in production maybe journald would reach quality of other syslogd implementations. By that time it should be considered not reliable. It has great potential but needs more testing.
Of course this doesn't solve your first issue - hazardous binary logs :-)
Mateusz Marzantowicz
Mateusz Marzantowicz wrote:
On 15.11.2013 11:46, Frantisek Hanzlik wrote:
For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
TIA, Franta Hanzlik
You can set restrictions on resources used by journald. Reducing disk space is one of them. Take a look at /etc/systemd/journald.conf, but I suppose you already know that.
High memory usage might be caused by leaks and it's nothing unexpected in such immature application. You can help to fix it if you have enough time. After 10 years of testing in production maybe journald would reach quality of other syslogd implementations. By that time it should be considered not reliable. It has great potential but needs more testing.
Of course this doesn't solve your first issue - hazardous binary logs :-)
Mateusz Marzantowicz
I agree, all systemd stuff seems very crude for me. And 'enough time' to debug or help to fix systemd monster I really not have, I even don't have time to report all bugs which happens to me. Thus I must only wish to systemd developers to finish their work to some usable state earlier than these 10 years.
Thanks, Franta Hanzlik
On Sat, Nov 16, 2013 at 10:44:21PM +0100, Frantisek Hanzlik wrote:
I agree, all systemd stuff seems very crude for me. And 'enough time' to debug or help to fix systemd monster I really not have, I even don't have time to report all bugs which happens to me. Thus I must only wish to systemd developers to finish their work to some usable state earlier than these 10 years.
Why not at least report one bug if you encounter so many of them?
Steven Stern wrote:
On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
For one thing I'm in the conviction that binary logs are hazardous bullshit,
In what way might the logs be hazardous?
It was mean mainly from administrator view. When things go bad, machine HW/SW fail or any other disasters occurs, logs are very valuable. And I'm confident that binary logs are too weak in this situation. Text logs are useful even if log file is damaged or ends with fragments and can be easy readable with lot of tools. Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea. Second, it is question when tight integration of systemd and logging services has any benefits - there is number of situation (logging over network, for example) which speaks for separate logging service. And possibilities with e.g. rsyslogd are better than with journald - why Fedora must again replace verified, reliable, Unix standard things with some crappy solutions? For nightmares of its users?
Franta Hanzlik
Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea.
And if logs are in a format that you cannot read, you cannot safely submit them to an outside server. You don't know what they contain. Logon credentials, confidential data that you're working on, etc.
On 11/17/2013 11:21 AM, Tim wrote:
Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea.
And if logs are in a format that you cannot read, you cannot safely submit them to an outside server. You don't know what they contain. Logon credentials, confidential data that you're working on, etc.
IIRC that's the reason why journald supports encryption. I don't recall the link but there's a blog post somewhere (on redhat.com?) where the reasons for moving to journald were outlined. Might be worth a search if you want to know more.
Regards, Patrick
On Sun, 17 Nov 2013 12:12:08 +0100 Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
IIRC that's the reason why journald supports encryption.
It's not encrypted it's binary, similar to any compiled app or virus.
On 11/17/2013 12:16 PM, Frank Murphy wrote:
On Sun, 17 Nov 2013 12:12:08 +0100 Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
IIRC that's the reason why journald supports encryption.
It's not encrypted it's binary, similar to any compiled app or virus.
I meant the transmission of the log to another log server. Not the log itself. Anyway, here is Lennart Poettering's rationale behind journald:
https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUE...
Regards, Patrick
On Sun, 17 Nov 2013 12:37:03 +0100 Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
I meant the transmission of the log to another log server. Not the log itself. Anyway, here is Lennart Poettering's rationale behind journald:
One of the things you will see after a hard reset is: Journal corrupted, journal deleted and being restarted. ie.. no useful info.
Frank Murphy wrote:
On Sun, 17 Nov 2013 12:37:03 +0100 Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
I meant the transmission of the log to another log server. Not the log itself. Anyway, here is Lennart Poettering's rationale behind journald:
One of the things you will see after a hard reset is: Journal corrupted, journal deleted and being restarted. ie.. no useful info.
This is perhaps most significant shortcoming. And beside it I see in log messages as:
systemd-journald[295]: Failed to set ACL on /var/log/journal/70ed4dbe694041d484165719514f195e/user-1045.journal, ignoring: Invalid argument systemd-journald[2948]: Failed to write entry, ignoring: Cannot allocate memory systemd-journal[388]: Suppressed 966 messages from /system/dbus.service systemd-journal[299]: Forwarding to syslog missed 92 messages.
which induce sense of journald logging futility.
On Mon, Nov 25, 2013 at 12:58 AM, Frantisek Hanzlik franta@hanzlici.cz wrote:
Frank Murphy wrote:
On Sun, 17 Nov 2013 12:37:03 +0100 Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
I meant the transmission of the log to another log server. Not the log itself. Anyway, here is Lennart Poettering's rationale behind journald:
One of the things you will see after a hard reset is: Journal corrupted, journal deleted and being restarted. ie.. no useful info.
This is perhaps most significant shortcoming. And beside it I see in log messages as:
systemd-journald[295]: Failed to set ACL on /var/log/journal/70ed4dbe694041d484165719514f195e/user-1045.journal, ignoring: Invalid argument systemd-journald[2948]: Failed to write entry, ignoring: Cannot allocate memory systemd-journal[388]: Suppressed 966 messages from /system/dbus.service systemd-journal[299]: Forwarding to syslog missed 92 messages.
which induce sense of journald logging futility.
Googling shows that it was fixed in May and applied in OpenSUSE in June and in Debian in September:
http://lists.opensuse.org/opensuse-commit/2013-06/msg01102.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717863
Given that there have been F19 systemd uploads since May, I assume that it's also fixed in Fedora but the changelogs don't mention it.
On Sun, Nov 17, 2013 at 12:12 PM, Patrick Lists fedora-list@puzzled.xs4all.nl wrote:
On 11/17/2013 11:21 AM, Tim wrote:
Allegedly, on or about 17 November 2013, Frantisek Hanzlik sent:
Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea.
And if logs are in a format that you cannot read, you cannot safely submit them to an outside server. You don't know what they contain. Logon credentials, confidential data that you're working on, etc.
IIRC that's the reason why journald supports encryption. I don't recall the link but there's a blog post somewhere (on redhat.com?) where the reasons for moving to journald were outlined. Might be worth a search if you want to know more.
syslog (using rsyslog) already supports encryption of you require that (even kerberos if that is what you want, we do). Journald is a solution in search of a problem ;-)
Tim:
And if logs are in a format that you cannot read, you cannot safely submit them to an outside server. You don't know what they contain. Logon credentials, confidential data that you're working on, etc.
Patrick Lists:
IIRC that's the reason why journald supports encryption. I don't recall the link but there's a blog post somewhere (on redhat.com?) where the reasons for moving to journald were outlined. Might be worth a search if you want to know more.
If my logs might contain confidential data, there's no way I'm submitting them to something else to assess (e.g. bug reports). Encrypting isn't the issue, it's (me) being unable to tell what's in them, nor edit the confidential stuff out of it beforehand.
On Mon, Nov 18, 2013 at 04:48:32PM +1030, Tim wrote:
Tim:
And if logs are in a format that you cannot read, you cannot safely submit them to an outside server. You don't know what they contain. Logon credentials, confidential data that you're working on, etc.
Patrick Lists:
IIRC that's the reason why journald supports encryption. I don't recall the link but there's a blog post somewhere (on redhat.com?) where the reasons for moving to journald were outlined. Might be worth a search if you want to know more.
If my logs might contain confidential data, there's no way I'm submitting them to something else to assess (e.g. bug reports). Encrypting isn't the issue, it's (me) being unable to tell what's in them, nor edit the confidential stuff out of it beforehand.
You can read it with journalctl. You can have syslog running beside it. You can still inspect it with strings (because there isn't any compression). You can forward it to another server (like syslog) or just use syslog itself.
AFAIK there is no encryption in the journal, only Forward Secure Sealing (FSS), where things are hashed. See man journalctl for more information on that.
2013/11/17 Frantisek Hanzlik franta@hanzlici.cz:
Steven Stern wrote:
On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
For one thing I'm in the conviction that binary logs are hazardous bullshit,
In what way might the logs be hazardous?
It was mean mainly from administrator view. When things go bad, machine HW/SW fail or any other disasters occurs, logs are very valuable. And I'm confident that binary logs are too weak in this situation. Text logs are useful even if log file is damaged or ends with fragments and can be easy readable with lot of tools. Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea. Second, it is question when tight integration of systemd and logging services has any benefits - there is number of situation (logging over network, for example) which speaks for separate logging service.
The journald log format is documented at least to some extent [1], and there exists free software for reading the log. To me, it sounds like way more accessible than if it was a binary data format of a typical proprietary tool. For example, booting any Fedora live image should suffice if you need to read the journal of a system that uses journald and happens to become unbootable.
Typically journalctl will generate you a representation of the data in syslog log file format. This loses some infromation that the journal stores, because there is no way to represent it all while keeping the output syslog-like and easily human-readable. This output can be easily fed to traditional unix tools like grep, if that is your preferred way of extracting information from logs.
Finally, since you can also run a normal syslog daemon which maintains a text-format logs for you, I do not really see why having some log data in journald format under /run/log/journal/ should be considered hazardous. You can pretty much just ignore it if you feel like using other kinds of tools for storing and managing logs.
And possibilities with e.g. rsyslogd are better than with journald
- why Fedora must again replace verified, reliable, Unix standard
things with some crappy solutions? For nightmares of its users?
At least for my needs, the journal has been way more convenient to use than rsyslog. It is much nicer to read logs when journalctl e.g. combines the older rotated logs with the latest ones. Also, it easily allows me to easily specify that I want just the logs of this month or of just this one boot, or of just some specific service.
If I was writing tools that'd automatically handle the logs, I think I would also benefit from the additional data that journal stores that is usually not available in a syslog formatted log. Having to use e.g. the journal API would of course be some burden, but I can imagine it being nicer than having to e.g. parse all the different date formats that a text-formatted log could have. Or having to handle all of the other things that may or may not be in the syslog lines, in various formats.
Of course there are still bugs and the other issues in the new tools, but they certainly aren't there in order to cause nightmares. I am hopeful that the issues can be and are fixed. For my (relatively simple) setups, there aren't any major showshoppers, though.
[1] http://www.freedesktop.org/wiki/Software/systemd/journal-files/
-Joonas
Joonas Sarajärvi wrote:
2013/11/17 Frantisek Hanzlik franta@hanzlici.cz:
Steven Stern wrote:
On 11/15/2013 04:46 AM, Frantisek Hanzlik wrote:
For one thing I'm in the conviction that binary logs are hazardous bullshit,
In what way might the logs be hazardous?
It was mean mainly from administrator view. When things go bad, machine HW/SW fail or any other disasters occurs, logs are very valuable. And I'm confident that binary logs are too weak in this situation. Text logs are useful even if log file is damaged or ends with fragments and can be easy readable with lot of tools. Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea. Second, it is question when tight integration of systemd and logging services has any benefits - there is number of situation (logging over network, for example) which speaks for separate logging service.
The journald log format is documented at least to some extent [1], and there exists free software for reading the log. To me, it sounds like way more accessible than if it was a binary data format of a typical proprietary tool. For example, booting any Fedora live image should suffice if you need to read the journal of a system that uses journald and happens to become unbootable.
Typically journalctl will generate you a representation of the data in syslog log file format. This loses some infromation that the journal stores, because there is no way to represent it all while keeping the output syslog-like and easily human-readable. This output can be easily fed to traditional unix tools like grep, if that is your preferred way of extracting information from logs.
Finally, since you can also run a normal syslog daemon which maintains a text-format logs for you, I do not really see why having some log data in journald format under /run/log/journal/ should be considered hazardous. You can pretty much just ignore it if you feel like using other kinds of tools for storing and managing logs.
And possibilities with e.g. rsyslogd are better than with journald
- why Fedora must again replace verified, reliable, Unix standard
things with some crappy solutions? For nightmares of its users?
At least for my needs, the journal has been way more convenient to use than rsyslog. It is much nicer to read logs when journalctl e.g. combines the older rotated logs with the latest ones. Also, it easily allows me to easily specify that I want just the logs of this month or of just this one boot, or of just some specific service.
If I was writing tools that'd automatically handle the logs, I think I would also benefit from the additional data that journal stores that is usually not available in a syslog formatted log. Having to use e.g. the journal API would of course be some burden, but I can imagine it being nicer than having to e.g. parse all the different date formats that a text-formatted log could have. Or having to handle all of the other things that may or may not be in the syslog lines, in various formats.
Of course there are still bugs and the other issues in the new tools, but they certainly aren't there in order to cause nightmares. I am hopeful that the issues can be and are fixed. For my (relatively simple) setups, there aren't any major showshoppers, though.
[1] http://www.freedesktop.org/wiki/Software/systemd/journal-files/
-Joonas
I'm convinced it is as similar things on eg. ms windows (binary logs, registry,...) - when is all working fine, then it look fine too. But when machine somehow fail, then this binary mud is serious problem. And I do not want to wait to time, when for some similar failed crap I have to reinstall Linux box. Until now I never needed it, no matter how significant Linux box crashes were... Franta Hanzlik
On 11/17/2013 12:32 PM, Frantisek Hanzlik wrote:
I'm convinced it is as similar things on eg. ms windows (binary logs, registry,...) - when is all working fine, then it look fine too.
Yes. When there's a major crash, you may not have access to the fancy programs for reading binary logs, especially if they're GUI only, and you're stuck in CLI mode. And once you have a binary database of all of your system info, the temptation to let non-system applications use it gets to be irresistible, and we all know what that leads to. There are some things Windows does that are worth copying; the Registry isn't one of them.
On Sun, Nov 17, 2013 at 12:46:40PM -0800, Joe Zeff wrote:
On 11/17/2013 12:32 PM, Frantisek Hanzlik wrote:
I'm convinced it is as similar things on eg. ms windows (binary logs, registry,...) - when is all working fine, then it look fine too.
Yes. When there's a major crash, you may not have access to the fancy programs for reading binary logs, especially if they're GUI only, and you're stuck in CLI mode. And once you have a binary database of all of your system info, the temptation to let non-system applications use it gets to be irresistible, and we all know what that leads to. There are some things Windows does that are worth copying; the Registry isn't one of them.
A GUI for the journal is already being developed. https://wiki.gnome.org/Apps/Logs
Anyway, seems good to have easy access to show the various log and error output of programs in an easy way. More discoverable for users.
On 11/17/2013 02:18 PM, Olav Vitters wrote:
A GUI for the journal is already being developed. https://wiki.gnome.org/Apps/Logs
That's nice, although I'd like to have a CLI reader as well if there isn't one already.
Hi Jonas,
I have a comment, and a question.
On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
2013/11/17 Frantisek Hanzlik franta@hanzlici.cz:
It was mean mainly from administrator view. When things go bad, machine HW/SW fail or any other disasters occurs, logs are very valuable. And I'm confident that binary logs are too weak in this situation. Text logs are useful even if log file is damaged or ends with fragments and can be easy readable with lot of tools. Binary logs, by contrast, may be useless when log file is damaged or I haven't this one unique utility for reading them. And my experiences with systems where binary logs are implemented says clearly that binary logs is bad idea. Second, it is question when tight integration of systemd and logging services has any benefits - there is number of situation (logging over network, for example) which speaks for separate logging service.
The journald log format is documented at least to some extent [1], and there exists free software for reading the log. To me, it sounds like way more accessible than if it was a binary data format of a typical proprietary tool. For example, booting any Fedora live image should suffice if you need to read the journal of a system that uses journald and happens to become unbootable.
I am sure you (and everyone else on the list) will agree, tools for viewing plain text (cat, more, less, <favourite editor>) outnumber and are available on every platform compared to any specialised tool. Even if a format is open, it still needs tools that support it. I would not count on having a gui when my system crashes. Sometimes the only access you have to a crashed system is the recovery prompt.
[...chomp...chomp...chomp...]
At least for my needs, the journal has been way more convenient to use than rsyslog. It is much nicer to read logs when journalctl e.g. combines the older rotated logs with the latest ones. Also, it easily allows me to easily specify that I want just the logs of this month or of just this one boot, or of just some specific service.
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
Thanks,
PS: I'm no sysadmin, just a regular *nix user who prefers the command line.
On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
acpi for power?
On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
acpi for power?
That does filter the output, but not the messages I was looking for. Looking at /var/log/messages tells me the lines should be something like this:
kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled (total events = 38)
But I still can't find where it says shutting down. Anyway, what subsystem is the above? Also how can I limit it to kernel messages from all subsystems?
Thanks for your response,
On Mon, Nov 18, 2013 at 4:41 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
acpi for power?
That does filter the output, but not the messages I was looking for. Looking at /var/log/messages tells me the lines should be something like this:
kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled (total events = 38)
But I still can't find where it says shutting down. Anyway, what subsystem is the above? Also how can I limit it to kernel messages from all subsystems?
Thanks for your response,
You're welcome.
"acpi" had been a guess!
I scratched my head, read the journalctl man page, and found "-F":
# journalctl -F _KERNEL_SUBSYSTEM platform scsi pnp pci pci_bus acpi
On Mon, Nov 18, 2013 at 05:29:27PM +0000, Tom H wrote:
On Mon, Nov 18, 2013 at 4:41 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
On Mon, Nov 18, 2013 at 03:43:57PM +0000, Tom H wrote:
On Mon, Nov 18, 2013 at 3:14 PM, Suvayu Ali fatkasuvayu+linux@gmail.com wrote:
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
Perhaps "journalctl _KERNEL_SUBSYSTEM=<subsystem>"
acpi for power?
That does filter the output, but not the messages I was looking for. Looking at /var/log/messages tells me the lines should be something like this:
kernel: [381990.959785] CPU3: Core temperature above threshold, cpu clock throttled (total events = 38)
But I still can't find where it says shutting down. Anyway, what subsystem is the above? Also how can I limit it to kernel messages from all subsystems?
Thanks for your response,
You're welcome.
"acpi" had been a guess!
I scratched my head, read the journalctl man page, and found "-F":
# journalctl -F _KERNEL_SUBSYSTEM platform scsi pnp pci pci_bus acpi
On my laptop I have a few more:
$ journalctl -F _KERNEL_SUBSYSTEM ieee80211 serio scsi pnp pci pci_bus acpi hid usb thermal
`thermal' does the trick :-p.
$ journalctl -r _KERNEL_SUBSYSTEM=thermal -- Logs begin at Fri 2013-09-20 23:52:53 CEST, end at Tue 2013-11-19 00:45:01 CET. -- Nov 14 10:08:54 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical temperature reached(100 C),shutting down Nov 14 10:08:54 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical temperature reached(100 C),shutting down -- Reboot -- Nov 14 09:56:17 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical temperature reached(100 C),shutting down Nov 14 09:56:17 kuru.dyndns-at-home.com kernel: thermal thermal_zone0: critical temperature reached(100 C),shutting down
Wallah!
2013/11/18 Suvayu Ali fatkasuvayu+linux@gmail.com:
Hi Jonas,
I have a comment, and a question.
On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
The journald log format is documented at least to some extent [1], and there exists free software for reading the log. To me, it sounds like way more accessible than if it was a binary data format of a typical proprietary tool. For example, booting any Fedora live image should suffice if you need to read the journal of a system that uses journald and happens to become unbootable.
I am sure you (and everyone else on the list) will agree, tools for viewing plain text (cat, more, less, <favourite editor>) outnumber and are available on every platform compared to any specialised tool. Even if a format is open, it still needs tools that support it. I would not count on having a gui when my system crashes. Sometimes the only access you have to a crashed system is the recovery prompt.
I wonder where this notion that you'd need a GUI to access the journal content came from. You do need a working journalctl, but it is not a GUI tool. Normally it outputs syslog-like output that you could feed to your favourite syslog-output-expecting log analysis tool.
At least for my needs, the journal has been way more convenient to use than rsyslog. It is much nicer to read logs when journalctl e.g. combines the older rotated logs with the latest ones. Also, it easily allows me to easily specify that I want just the logs of this month or of just this one boot, or of just some specific service.
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
I am not sure where that message in particular would show up. My first instinct in such a situation would be to just run journalctl without arguments. If the output is not piped anywhere, journalctl opens less and puts the syslog-like output there. Now it is pretty same as if I had run less /var/log/messages in a system that uses a traditional syslog. I can for example press / to access the search function of less and search for the "cpu" string.
Since a hardware related error would likely be logged by the kernel, I might invoke journalctl like this: journalctl _TRANSPORT=kernel
This would limit the things I get so that only kernel debug messages are shown. Different filter criteria you could use are described in the systemd.journal-fields man page.
Another useful parameter would be the --since parameter, e.g. like this: journalctl --since 2013-11-15 _TRANSPORT=kernel
Now journalctl would omit old stuff that likely would not be interesting in this case.
But maybe the message is not in the kernel log, and you'd wish to use grep to search for the message like you could with a text-based syslog? You can like this: journalctl --since 2013-11-15 | grep -i cpu
The --since parameter is still useful, in case you happen to have many months of log data stored in the journal. With the --since and --until parameters, you can limit the search to any arbitrary interval that is covered by your stored log data, regardless of how those times relate to the times at which log rotation has happened.
Hopefully this helps. At least you should know that you can still use your knowledge of the traditional unixy text-processing tools.
-Joonas
Sorry for posting again so soon and replying to myself, but I just noticed one very useful thing that might help quite much in the specific problem you described:
2013/11/18 Joonas Sarajärvi muep@iki.fi:
2013/11/18 Suvayu Ali fatkasuvayu+linux@gmail.com:
the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages.
I am not sure where that message in particular would show up.
Likely you would not want to look for a specific subsystem anyway Often there are many things in the system interacting and it is nicer to look at the whole log in the correct point in time. One nice filter criteria is to filter by boot id with the --boot option of journalctl
journalctl --boot
The above would show just the messages that were gathered during the currently ongoing boot cycle. Now if you want messages of the previous boot, --boot allows you to specify an offset. So if you pass -1 there like this, it would show you messages from the previous boot cycle: journalctl --boot=-1
If you already booted many times, just grow the number after the minus sign to find the one that ended before you intended it to. Then just scroll to the end (the > key usually does this in less) to see what were the last messages that were recorded before shutdown.
In some cases like a kernel panic, you will likely not have the kernel panic message in the log. This is because usually at the point where the kernel has the message, it will not write anything to disk anymore. If there is nothing interesting in the end of that boot, you should also check if the following boot has a note about the journal being rotated due to not being properly closed. This would at least let you know that you miss some messages.
-Joonas
On Mon, Nov 18, 2013 at 07:44:55PM +0200, Joonas Sarajärvi wrote:
Sorry for posting again so soon and replying to myself, but I just noticed one very useful thing that might help quite much in the specific problem you described:
2013/11/18 Joonas Sarajärvi muep@iki.fi:
2013/11/18 Suvayu Ali fatkasuvayu+linux@gmail.com:
the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages.
I am not sure where that message in particular would show up.
Likely you would not want to look for a specific subsystem anyway Often there are many things in the system interacting and it is nicer to look at the whole log in the correct point in time. One nice filter criteria is to filter by boot id with the --boot option of journalctl
journalctl --boot
The above would show just the messages that were gathered during the currently ongoing boot cycle. Now if you want messages of the previous boot, --boot allows you to specify an offset. So if you pass -1 there like this, it would show you messages from the previous boot cycle: journalctl --boot=-1
I had seen that option, but could not figure out how do I get a boot id. Now that I know it is an offset, it will be very useful indeed.
In some cases like a kernel panic, you will likely not have the kernel panic message in the log. This is because usually at the point where the kernel has the message, it will not write anything to disk anymore. If there is nothing interesting in the end of that boot, you should also check if the following boot has a note about the journal being rotated due to not being properly closed. This would at least let you know that you miss some messages.
Again, very good suggestion! Thanks a lot.
I have to say something though. Somehow I feel most of Lennart Poettering's projects (systemd, pulseaudio) have terribly impenetrable documentation. When reading them it feels like it is meant for other developers littered with developer jargon. And I'm no stranger to *nix docs; I know my way around `man bash', for example.
On Mon, Nov 18, 2013 at 07:31:21PM +0200, Joonas Sarajärvi wrote:
2013/11/18 Suvayu Ali fatkasuvayu+linux@gmail.com:
Hi Jonas,
I have a comment, and a question.
On Sun, Nov 17, 2013 at 03:46:55PM +0200, Joonas Sarajärvi wrote:
The journald log format is documented at least to some extent [1], and there exists free software for reading the log. To me, it sounds like way more accessible than if it was a binary data format of a typical proprietary tool. For example, booting any Fedora live image should suffice if you need to read the journal of a system that uses journald and happens to become unbootable.
I am sure you (and everyone else on the list) will agree, tools for viewing plain text (cat, more, less, <favourite editor>) outnumber and are available on every platform compared to any specialised tool. Even if a format is open, it still needs tools that support it. I would not count on having a gui when my system crashes. Sometimes the only access you have to a crashed system is the recovery prompt.
I wonder where this notion that you'd need a GUI to access the journal content came from. You do need a working journalctl, but it is not a GUI tool. Normally it outputs syslog-like output that you could feed to your favourite syslog-output-expecting log analysis tool.
Well the gui was mentioned in the thread, hence my mention. And as you see from later in my message, I have used journalctl, but unsuccessfully. I looked at the dependencies for journalctl, compared to less or cat, it is rather long. On a crashed system I wouldn't trust to have access to all these linked libraries.
I have been very frustrated with journalctl. The manual page is very unhelpful in that regard. For example the other day, I wanted to investigate why my laptop shutdown suddenly (I think it was overheating), but there was no reasonable way for me to filter the cpu specific messages. Could you give some pointers how I can do that? I would be very grateful.
I am not sure where that message in particular would show up. My first instinct in such a situation would be to just run journalctl without arguments. If the output is not piped anywhere, journalctl opens less and puts the syslog-like output there. Now it is pretty same as if I had run less /var/log/messages in a system that uses a traditional syslog. I can for example press / to access the search function of less and search for the "cpu" string.
Since a hardware related error would likely be logged by the kernel, I might invoke journalctl like this: journalctl _TRANSPORT=kernel
This helped, thank you!
This would limit the things I get so that only kernel debug messages are shown. Different filter criteria you could use are described in the systemd.journal-fields man page.
I looked at that. But was not clear what are valid values for the fields. Your and Tom's responses have given me enough pointers now to investigate more.
Another useful parameter would be the --since parameter, e.g. like this: journalctl --since 2013-11-15 _TRANSPORT=kernel
This is also very helpful, thank you.
Cheers,
Oxl El nov 15, 2013 6:46 a.m., "Frantisek Hanzlik" franta@hanzlici.cz escribió:
For one thing I'm in the conviction that binary logs are hazardous bullshit, for another on my two F19 machines systemd-journald occupy significant part of resources (after several days it is often 1.5 to 2.5 GB RAM and several GB on /var/log/journal/*/* filesystem).
Then, how I can avoid this crap and log only to standard rsyslogd?
I tried set "LogTarget=syslog-or-kmsg" in "[Manager]" section of /etc/systemd/{system.conf,user.conf}, but this not helped; setting "Storage=none" in "[Journal]" section of /etc/systemd/journald.conf is better, but I want ideally completely cut out systemd-journald from my systems.
TIA, Franta Hanzlik
users mailing list users@lists.fedoraproject.org To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines Have a question? Ask away: http://ask.fedoraproject.org
-
Alexander Aguilar Torrico -
Frank Murphy -
Franta Hanzlík -
Joe Zeff -
Joonas Sarajärvi -
Mateusz Marzantowicz -
Natxo Asenjo -
Olav Vitters -
Patrick Lists -
Rahul Sundaram -
Rick Stevens -
SternData -
Suvayu Ali -
T.C. Hollingsworth -
Tim -
Tom H