F21 Self Contained Change: Remote Journal Logging

[Jaroslav Reznik]

= Proposed Self Contained Change: Remote Journal Logging = 
https://fedoraproject.org/wiki/Changes/Remote_Journal_Logging

Change owner(s): Zbigniew Jędrzejewski-Szmek <zbyszek at in.waw.pl>

Systemd journal can be configured to forward events to a remote server. 
Entries are forwarded including full metadata, and are stored in normal 
journal files, identically to locally generated logs. 

== Detailed Description ==
Systemd's journal currently provides a replacement for most functionality 
offered by traditional syslog daemons,
with two notable exceptions: arbitrary filtering of messages and forwarding of 
messages over the network. This Change targets the latter.

The high-level goal is to have a mechanism where journal logging can be 
extended
to keep a copy of logs on a remote server, without requiring any maintenance, 
done
fairly efficiently and in a secure way.

Two new daemons are added as part of the systemd package:
* on the receiver side systemd-journal-remote accepts messages in the Journal 
Export Format [1]. The export format is a simple serialization of journal 
entries, supporting both text and binary fields. This means that the messages 
are transferred intact, apart from the "cursors", which specify the location 
in the journal file. Received entries are stored in local journal files 
underneath /var/log/journal. Those files are subject to normal journald rules 
for rotation, and the older ones will be removed as necessary to stay within 
disk usage limits. Once entries have been written to the journal file, they 
can be read using journalctl and the journal APIs, and are available to all 
clients, e.g. Gnome Logs [2].

* on the sender side systemd-journal-upload is a journal client, which exports 
all available journal messages and uploads them over the network. The (local) 
cursor of the last message successfully forwarded is stored on disk, so when 
systemd-journal-upload is restarted (possibly after a reboot of the machine), 
it will send all recent messages found in the journal and then new ones as 
they arrive.

The communication between the two daemons is done over standard HTTPS, 
following rather simple rules, so it is possible to create alternate 
implementations without much work. For example, curl can be easily used to 
upload journal entries from a text file containing entries in the export 
format. Basically, the data are sent in an HTTP POST to /upload with Content-
Type: application/vnd.fdo.journal. When doing "live" forwarding, the size of 
the transfer cannot be known in advance, so Transfer-Encoding: chunked is 
used. All communication is encrypted, and the identity of both sides is 
verified by checking for appropriate signatures on the certificates.

== Scope ==
* Proposal owners:
The code in systemd for systemd-journal-remote and systemd-journal-upload will 
have to be added. The first is done, and has already been released in the 
latest Rawhide systemd package. The second is mostly done, and will be 
submitted upstream soon. Necessary units will have to be created, and a 
location with suitable permissions will have to be created so that systemd-
journal-remote can run unprivileged. This means that systemd-journal-remote 
should probably be packaged as a separate subpackage, similarly to systemd-
journal-gatewayd. systemd-journal-upload uses libcurl, and thus should be also 
packaged as a separate subpackage to avoid introducing a new dependency for 
systemd.

Suitable defaults for timeouts for transfers and maximum accepted entry sizes 
have to be chosen.

A port will have to be picked: systemd-journal-gatewayd uses 19531, so 
systemd-journal-remote should probably use 19532.

Two new users will have to be created when the packages are installed.

* Other developers: N/A (not a System Wide Change)
* Release engineering: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change) 

[1] http://www.freedesktop.org/wiki/Software/systemd/export/
[2] https://wiki.gnome.org/Apps/Logs
_______________________________________________
devel-announce mailing list
devel-announce at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce