Log management

Maxim Burgerhout maxim at wzzrd.com
Tue Jan 12 11:23:36 UTC 2010


After last weeks meeting, Mike asked me to look into a solution for
log management, reporting and alerting. As the new guy, I happily
jumped in and so here's a short status report on what I found and what
some of the options are. I would appreciate any input.

First of all, there's a lot of different projects trying to deliver
this functionality. A big part of them though, are either dead,
commercial and / or mainly aimed at parsing Apache logfiles.

I am assuming we want an open source tool, so, skipping the commercial
products like Splunk and skipping the dead projects, it boils down to
these as most promising. If I turn out to be missing some great
project, please tell me and I'll look into it.

1. octopussy (http://www.8pussy.org/dokuwiki/doku.php)
Log aggregation tool, providing a one stop web UI to view all logs for
a bunch of servers, complete with alerting to email, jabber or nagios,
creation of graphs etc. Imports data into a MySQL database. The
downsides are that it's developer base is pretty narrow, it is
relatively complex to configure and it is Debian centric in both
documentation and available packaging. On the other hand, octopussy is
what comes closest to Splunk in the open source world that I know of.
Sadly, all other Splunk-like apps are commercial. Last release:
December 2009, last commit a couple of days ago.

2. logreport / lire (http://www.logreport.org/)
Log parser that runs from cron or manually; parses logs from many
different applications and generates html, txt or pdf that can
optionally be mailed to people. Slightly odd curses configuration
frontend. Works by importing log files into what is called the 'dlf
store' and renders periodic reports from that data. Can receive
logfiles over mail. Downside is that development is pretty slow, even
if it is backed by a foundation (about one release per year; last one
was in March 2009). Low mailinglist traffic. Does not do alerting and
real time parsing. DLF has sqlite as a backend, afaict, and I'm not
sure how well that scales in the long run. Last release: March 2009,
no activity on mailinglist and CVS since then.

3. epylog (https://fedorahosted.org/epylog/)
Has some fans in the Infra group already, it seems. Modular log
parser, run from cron. Stores offsets for already parsed files, to the
next run can start where the previous one left off. Generates reports
in html in a configurable location or sends them out per mail. Custom
report delivery methods can be configured. Does not do alerting and
real time parsing. Hosted on Fedorahosted, written in Python (the
other two apps are mainly in Perl). Very easy to write custom modules
for. Last release: none recently, but subversion is active.

Of the projects, only epylog is packaged for Fedora already and
Octopussy is the only one that does realtime log parsing and alerting.

What should be the next step in this? I probably need to do some more
testing, but before I do that, let me know what you think is important
in a log monitoring and reporting application? Any specific tests you
would like to see done for these apps?

Let me know what you think.

Maxim Burgerhout
maxim at wzzrd.com
----------------
GPG Fingerprint
EB11 5E56 E648 9D99 E8EF 05FB C513 6FD4 1302 B48A


More information about the infrastructure mailing list