rsyslog/epylog reporting

[seth vidal]

Hi folks, 
On friday, due to personal irritation, I went ahead and fixed our
rsyslog config and setup epylog on our central loghost. 
 epylog: http://fedorahosted.org/epylog/


sample output: http://fedorapeople.org/~icon/epylog/sample-report.html


I setup a merged-log location in /var/log/merged on log02. I also setup
a logrotate job which will rotate those files once a day and only keep
one additional day's worth of those logs. These are just a duplicate
output of the logging we have in the per-host directories currently, so
there is no need to keep them for any amount of time.

Right now epylog runs once a day but that may change to a couple of
times a day or more, just until we get a handle on what cruft needs to
be removed.

I merged all logs from all hosts into one merged log file, however, I
think it might be worthwhile to consider breaking these out a bit more
into sets of hosts. Suggestions on this are welcome.


If you're not familiar with epylog, it takes a set of logs and parses
them and collates results across multiple hosts to present a smaller set
of output of important events and remove all the noise and cruft. Then
it also includes a set of items which were not ignored and were not in
the parsed set. Right now, we still have a lot of crap in the unparsed
logs section but we've been working diligently on reducing that noise.

In the future we'll be working on some additional epylog modules to
clean up more of our noise and provide better results. Items that need
some more love:

- the mail/postfix module in epylog needs some more cleanup -
alternatively we could not do mail log parsing and let pflogsumm handle
it.

- an rsync module I wrote years ago would be useful for our epylog
instance

- a sudo-watching module to  dump out in a nice layout all sudo commands
run anywhere

- updating spamd module to get the newer spamassassin outputs

- fixing the login/sshd module to see pam_unix(sshd:*) properly

- nagios overview-module:
   X alerts today
   X failures today, etc, etc


other ideas?

-sv