Machine-readable text format for log and config files

[Alexander Sauta]

Hello All.

I'm just a Linux Russian GNU/Linux user and I would like to propose
one simple idea. I'll try to be as brief as possible.

A few months ago a proposal of new binary log system coupled with
systemd has been made by Lennart Poettering. It rose numerous
emotional discussions in Russian Linux community, a lot of users have
seen severe unix-way violation there and thus wouldn't like to adopt
it.

The new log system addresses the issues outlined by Lennart in his
intro on
https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgj...
At first I wanted to make a comparison against the proposed approach,
but now I don't want this message to be that long. However, one can
state that all the goals, mentioned there, can be achieved within my
method combined with some external tools if necessary.

I'd like to propose possibly less destructive but yet efficient
solution for this problem: machine-readable text logs. The most famous
such format is, definitely, JSON. I will use it to illustrate my
ideas, but it's absolutely not the best one for this purpose.

Long story short, here is possible single log entry for apache:

{ "date": "2011-11-23 23:25:36.0545 +0400", "pid": 2104,
"name":"apache2", "severity": 1, "sha1":
2a5162d0e83756bd559e13d13a5a7651fbe0d068, ...
"msg": { "ip": "127.0.0.1", "req":
"GET", "file": "index.html",
"http_version": 1.0, "result": 200 }  }

Note that this is complete JSON object consisting of two lines: first
is syslog header, second is application-defined message enclosed in
"msg" object. This allows to use traditional tools like grep for log
processing - not much changes here. And you can still read log with a
naked eye.

What makes difference is that we've created some structured objects
within simple text file and can now create automated tools to analyze
log of any program. Something like this:

jparser "SELECT date, msg.file FROM apache.log WHERE msg.code=200 AND
date >2011.11.22"

So we can make queries directly to log files with a tool similar to
XPath for JSON (JSONPath). But probably a better solution would be to
import log into a NoSQL database and use all its enormous power and
performance for further log processing. Note that this requires almost
zero effort since all such tools already exist.

Advantages over plain-text log files:
- Easy and unambigous parsing
- Traditional processing can be used more effectively combined with
some preprocessing
- Strict and formal structure of most important fields
- Unified format fot logs, config files, utilities output etc.
- Queries can be made directly to log file
- Applications can easily add their own fields with mere printf()
without breaking any processing tools
- Log can be directly imported into powerfull NoSQL database
- Binary blobs can be unobtrusively added as bas64-encoded fields
- Multiple logs can be processed at once with something similar to SQL
JOIN operator
- Index can be easily built separately for fast searching

Advantages over proposed binary log files:
- Backward compatibility (mostly) with existing tools
- Much simpler to emit
- More reliable and can be fixed with just VIM
- Existing syslogs doesn't even need to be heavily modified
- Direct import into existing fast and powerfull NoSQL databases
(almost no conversion required)
- Format is easiliy extendable withou breaking any compatibility
- Applications can make logs without any external libraries with mere printf()
- More advanced goals like clustering still can be achieved with
external methods without bloating the core subsystem

But we can extend this even further and use such formats not only for
log files: since it can describe arbitary data structures, it can be
used for configuration files and in commands output. I was once
looking through the code of linux utils, which take their data from
procfs and the implementation made me cry: each utility has to write
its own buggy and redundant parser for arbitary file format. While if
we had unified text format for data representation, the parsing would
be something like

int free_mem = jparser_get_num ("/proc/meminfo", "FreeMem");
int free_pages = jparser_get_num ("/proc/vmstat", "nr_free_pages" );

and so on. The same applies to grep'ing numerous ulitities (like
ifconfig and others) output. Current scripts are barely readable,
insecure and highly fragile - simple addition of a new parameter or
indentation change will break them. While we could just something
write

ifconfig --json | jparser "interfaces[*].inet"

to get addresses of all interfaces regardless of formatting,
indentation and presence of other params (note that I'm using
different syntax to show different possibilities).

I understand that some work has to be done anyway. For example, the
format definitely needs to be extended with at least variables for
config files, which will make the parser more complicated. I think
that this is all possible to resolve, though. And I would like to
emphasize once again that I don't think that JSON is the best choice
here.

So, what do you think? Are there any chances that something like this
will be introduced in unix configs, utilities and log files? And if
not - why?

PS
Another maybe more readable lisp-style format alternative:
(((:date 2011-11-23 23:25:36.0545 +0400) :pid 2104 :name apache2 ...)
(:msg :ip 127.0.0.1 :req GET :file index.html ...))

--
Alexander Sauta

[Miloslav Trmač]

Hello,
On Fri, Feb 17, 2012 at 9:05 AM, Alexander Sauta <demosito(a)gmail.com&gt; wrote:
>  I'd like to propose possibly less destructive but yet efficient
>  solution for this problem: machine-readable text logs. The most famous
>  such format is, definitely, JSON. I will use it to illustrate my
>  ideas, but it's absolutely not the best one for this purpose.
>
>  Long story short, here is possible single log entry for apache:
>
>  { "date": "2011-11-23 23:25:36.0545 +0400", "pid": 2104,
>  "name":"apache2", "severity": 1, "sha1":
>  2a5162d0e83756bd559e13d13a5a7651fbe0d068, ...
>  "msg": { "ip": "127.0.0.1", "req":
> "GET", "file": "index.html",
>  "http_version": 1.0, "result": 200 }  } 
An amazing coincidence.  Yesterday various people working on logging
and event processing, including the syslog-ng and rsyslog maintainers,
met in Brno for a "logging minisummit", and the main outcome was an
agreement to support and encourage JSON carried over the syslog
transport.

The primary advantages over other structured logging alternatives, as
we see them, are:
* Zero additional investment is necessary in the infrastructure -
existing logging setups, both local and company-wide, will continue
working
* It is possible to migrate/enhance applications one log message at a
time because both structured and unstructured log records will be
transferred via the same transport, in the proper order.

I'm not a logging expert, but I believe the agreed-upon format is
@cee:{ JSON here }
and at least some syslog implementations support this format already!

I'm sure that there will be details that need to be fleshed out,
libraries and tools written, and so on.  If you are interested in
helping, please do talk to us!

>  But we can extend this even further and use such formats not only
> for
>  log files: since it can describe arbitary data structures, it can be
>  used for configuration files and in commands output. 
The challenge for local file
formats is how to migrate, or even how to
persuade anyone to migrate.  There is a good reason to push for a
global standard in logging messages, because messages from many
different applications (or even different platforms) often end up in a
single log storage; this is not really the case for local file
formats.
    Mirek