Manipulating journalctl output

[Suvayu Ali]

Hi,

Lately I have been facing a lot of difficulty trying to get the
information I want easily using journalctl.  I find the manpage of
limited use; as in, it has the basic information but the more advanced
information is scattered in several manpages and the text is littered
with jargon more appropriate for developers (IMO this last point holds
true for most of systemd documentation)!

So I thought it might be useful to share a few methods I know, followed
by some questions.  Maybe others can share their tricks too.  This
thread could then serve as a more accessible documentation for users
(which could then be ported to a wiki page).

Some basic comments:

1. If your journal size is large, piping to grep is quite a bit slow.
2. To run journalctl as a regular user, you need to add yourself to the
   group systemd-journal, logout, and login again.

Useful commandline switches I'm familiar with:

1. Most recent entries first, `-r/--reverse'.

2. To follow, `-f/--follow'.

3. To limit logs by timestamp, `--since/--until'; it takes absolute
   timestamps (2013-12-31) as well as relative time stamps (-2d, -10m).
   BTW, the manpage does not say anything about the units for relative
   times.  I had to find out by trial-and-error m stands for minutes,
   not months.

4. Limit output from this boot, `-b/--this-boot'.

     $ journalctl -b 'bootid'.

   I find the more general interface to filter by _BOOT_ID most
   ridiculous.  How is the user supposed to know what was the boot-id
   for any of the previous sessions?

5. Filter by unit files, `-u/--unit'.

     $ journalctl -u <unit_file> # .service extension optional

6. To filter by journal fields, just pass FIELD=<value>.

7. The actual useful documentation for (4), (5) & (6) is really in
   systemd.journal-fields(7).  No mention is made of this other than the
   `SEE ALSO' section at the very end.  The fields manpage is also the
   perfect example of documentation written for developers instead of
   users (another one would be journald.conf(5)).

8. You can list valid values for the fields in (6) with `-F/--field'.
   Example, for all known boot-ids do this:

     $ journalctl -F _BOOT_ID


Now my questions:

1. How can I filter messages printed to the logs from my cron jobs?  I
   will try to explain by example:

     $ journalctl -ru crond --since=-3d
     -- Logs begin at Sun 2013-11-17 02:48:46 CET, end at Wed 2014-01-01 20:31:27 CET. --

     $ journalctl -r --since -3d | grep rsnapshot
     Jan 01 04:34:08 <hostname> rsnapshot[15294]: /usr/bin/rsnapshot daily: completed successfully
     Jan 01 04:30:01 <hostname> CROND[15270]: (root) CMD (/usr/bin/rsnapshot daily)
     Jan 01 04:00:07 <hostname> rsnapshot[15198]: /usr/bin/rsnapshot monthly: completed successfully
     Jan 01 04:00:01 <hostname> CROND[15196]: (root) CMD (/usr/bin/rsnapshot monthly)
     Dec 31 04:35:45 <hostname> rsnapshot[11360]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsnapshot daily: completed, but with some errors
     Dec 31 04:35:45 <hostname> rsnapshot[11359]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 255 while processing user at host:/etc/
     Dec 31 04:33:37 <hostname> rsnapshot[11353]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 255 while processing user at host:/home/user/
     Dec 31 04:30:01 <hostname> CROND[11334]: (root) CMD (/usr/bin/rsnapshot daily)
     Dec 30 04:36:05 <hostname> rsnapshot[8265]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsnapshot daily: completed, but with some errors
     Dec 30 04:36:05 <hostname> rsnapshot[8264]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 255 while processing user at host:/etc/
     Dec 30 04:33:58 <hostname> rsnapshot[8254]: /usr/bin/rsnapshot daily: ERROR: /usr/bin/rsync returned 255 while processing user at host:/home/user/
     Dec 30 04:30:02 <hostname> CROND[8237]: (root) CMD (/usr/bin/rsnapshot daily)

   What I understand from this is `-u <unit>' only tells me what happened
   when the unit file was (un)loaded, not what the process prints to the
   log files.  How do I get this information?  Is cron special in this
   regard?

2. I would like to filter logs that typically go into /var/log/secure
   (or other similar files); how do I do that?  Is grep my only option
   for cases like these?

Thanks in advance for any answers.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.