Machine-readable text format for log and config files

[Miloslav Trmač]

Hello,
On Fri, Feb 17, 2012 at 9:05 AM, Alexander Sauta <demosito at gmail.com> wrote:
> I'd like to propose possibly less destructive but yet efficient
> solution for this problem: machine-readable text logs. The most famous
> such format is, definitely, JSON. I will use it to illustrate my
> ideas, but it's absolutely not the best one for this purpose.
>
> Long story short, here is possible single log entry for apache:
>
> { "date": "2011-11-23 23:25:36.0545 +0400", "pid": 2104,
> "name":"apache2", "severity": 1, "sha1":
> 2a5162d0e83756bd559e13d13a5a7651fbe0d068, ...
> "msg": { "ip": "127.0.0.1", "req": "GET", "file": "index.html",
> "http_version": 1.0, "result": 200 }  }

An amazing coincidence.  Yesterday various people working on logging
and event processing, including the syslog-ng and rsyslog maintainers,
met in Brno for a "logging minisummit", and the main outcome was an
agreement to support and encourage JSON carried over the syslog
transport.

The primary advantages over other structured logging alternatives, as
we see them, are:
* Zero additional investment is necessary in the infrastructure -
existing logging setups, both local and company-wide, will continue
working
* It is possible to migrate/enhance applications one log message at a
time because both structured and unstructured log records will be
transferred via the same transport, in the proper order.

I'm not a logging expert, but I believe the agreed-upon format is
@cee:{ JSON here }
and at least some syslog implementations support this format already!

I'm sure that there will be details that need to be fleshed out,
libraries and tools written, and so on.  If you are interested in
helping, please do talk to us!

> But we can extend this even further and use such formats not only for
> log files: since it can describe arbitary data structures, it can be
> used for configuration files and in commands output.
The challenge for local file formats is how to migrate, or even how to
persuade anyone to migrate.  There is a good reason to push for a
global standard in logging messages, because messages from many
different applications (or even different platforms) often end up in a
single log storage; this is not really the case for local file
formats.
    Mirek