Package suggestion: Epylog

[Konstantin Ryabitsev]

Hello, all:

I would like to humbly suggest that Epylog be included into Fedora Core
-- not in the current test, of course, but some time in the future.

What it is:
Epylog is a syslog parser which runs periodically, looks at your logs,
processes some of the entries in order to present them in a more
comprehensible format, and then mails you the output. It is written
specifically for large network clusters where a lot of machines (around
50 and upwards) log to the same loghost using syslog or syslog-ng. It is
an alternative to a similar package called LogWatch.
See more: http://linux.duke.edu/projects/epylog/

Pros:

1. Written in Python (2.2 and above)
2. Written specifically for Red Hat Linux 7.x and above.
3. GPL
4. Actively maintained
5. Sends log reports in both HTML and plaintext.
6. Can publish reports to a location instead of mailing them, with
optional notification via email about new reports available.
7. Memory footprint is kept small at all times, despite the size of logs
(which may be hundreds of megabytes in size)
8. Robust and easily extensible using pluggable parsing modules.
9. Parsing modules can be written in Python, or in fact any other
language, using the external module API.
10. Been used extensively to do log reporting for large cluster
installations at Duke and several other EDUs.
11. Graphical front-ends to it do not exist at the moment, but should be
simple to write. Theoretically, it could be a replacement/extension for
redhat-logviewer.

Cons:

1. It's a relatively obscure application largely not known outside Duke
University.
2. Though it's been in use for over a year at Duke, it's not well tested
in a large variety of situations.
3. Parsing modules do not exist for other platforms where syslog strings
may differ from the ones in Linux/Fedora.
4. Due to threading and speed optimizations, Python parsing module API
is not very easy to grok. :/

Please see the website for more information. I would be willing to be
the Fedora package maintainer for it.

http://linux.duke.edu/projects/epylog/

Best regards,
-- 
Konstantin ("Icon") Riabitsev
Duke Physics Sysadmin, RHCE

[Alexandre Oliva]

On Sep 26, 2003, Konstantin Riabitsev <icon(a)linux.duke.edu&gt; wrote:

>  Epylog is a syslog parser which runs periodically, looks at your
> logs,
>  processes some of the entries in order to present them in a more
>  comprehensible format, and then mails you the output. 
Fedora already has LogWatch, and it would suck to get *two* e-mails a
day for each host :-)

How do they compare?

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva(a){redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva(a){lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer

[Bill Anderson]

On Fri, 2003-09-26 at 15:42, Alexandre Oliva wrote:
>  On Sep 26, 2003, Konstantin Riabitsev <icon(a)linux.duke.edu&gt;
> wrote:
>  
>  > Epylog is a syslog parser which runs periodically, looks at your logs,
>  > processes some of the entries in order to present them in a more
>  > comprehensible format, and then mails you the output.
>  
>  Fedora already has LogWatch, and it would suck to get *two* e-mails a
>  day for each host :-)
>  
>  How do they compare? 
I've neer seen Epylog before today but LoWatch doesn't compare when
looking at output.

Of course, having both available doesn't mean two emails, just select
the one you prefer. The output of Epylog looks tres sweet. Time to dig
in to the code. :)

I say +1


-- 
Bill Anderson
RHCE #807302597505773
bill(a)noreboots.com

[Randall J. Parr]

Is there / has there been any consideration of syslog-ng or one of the 
other available syslod extension/replacements that allows redirection to 
logfiles, et.al. via reg expr match?

I, for one, would really like an easily configured method for 
redirectoring some of those iptables messages, etc.

R.Parr
Temporal Arts

[seth vidal]

On Fri, 2003-09-26 at 17:02, Konstantin Riabitsev wrote:
>  Hello, all:
>  
>  I would like to humbly suggest that Epylog be included into Fedora Core
>  -- not in the current test, of course, but some time in the future.
>  
>  What it is:
>  Epylog is a syslog parser which runs periodically, looks at your logs,
>  processes some of the entries in order to present them in a more
>  comprehensible format, and then mails you the output. It is written
>  specifically for large network clusters where a lot of machines (around
>  50 and upwards) log to the same loghost using syslog or syslog-ng. It is
>  an alternative to a similar package called LogWatch.
>  See more: http://linux.duke.edu/projects/epylog/ 
I think one of the suggestions for packages not already in core is that
they need to do 'time' and be vetted in at least one release cycle in
fedora extras. This is a place where packages can be pruned or cleaned
or debugged - much like gnome fifth toe.


Does that sound like a good, workable, preliminary policy?

-sv

[Konstantin Ryabitsev]

On Fri, 2003-09-26 at 17:42, Alexandre Oliva wrote:
>  > Epylog is a syslog parser which runs periodically, looks at your
> logs,
>  > processes some of the entries in order to present them in a more
>  > comprehensible format, and then mails you the output.
>  
>  Fedora already has LogWatch, and it would suck to get *two* e-mails a
>  day for each host :-) 
Yes, that would suck. :) Which is why Epylog was written with the idea
that instead of doing that, you log everything to one loghost and when
Epylog runs on that machine, it presents the data about your entire
cluster in one report, instead of a report-per-machine. When you have
over 200 machines, this quickly becomes a priority. :)

>  How do they compare? 
Here are a few things that Epylog can do, which LogWatch can't:

1. By default it generates HTML-formatted reports, so they are better
formatted and are easier to follow (tables, colors, etc). Those who
don't like HTML email can ask for it to be rendered in plaintext, of
course. :) HTML reports can be published to a protected directory and
accessed via the web.
2. Epylog is threaded, meaning that network lookups take a fraction of
the time they do in LogWatch. Makes a lot of difference when your
firewall gets hit by half the world when the next big worm comes
through.
3. Adding a graphical front-end to Epylog should be relatively easy --
the backend is completely abstracted from front-end. A graphical
interface would allow doing some neat things -- like selecting date
ranges, checking which parsing modules to enable, or which logs to
process -- pretty simple. A pygtk interface would be pretty
straightforward to write, though I don't know if I'll ever get around to
that -- GUIs aren't my stroung suit. :)
4. It's written in Python, so maintaining it is much easier. :) </troll>

The largest win, in my opinion, is its usefulness in large cluster
installations, where logwatch simply doesn't "cut it." Other things I
consider just bonuses.

Regards,
-- 
Konstantin Riabitsev <icon(a)linux.duke.edu&gt;
Linux@DUKE

[seth vidal]

>  
>  I've neer seen Epylog before today but LoWatch doesn't compare when
>  looking at output.
>  
>  Of course, having both available doesn't mean two emails, just select
>  the one you prefer. The output of Epylog looks tres sweet. Time to dig
>  in to the code. :)
>  
>  I say +1 
The iptables scan consolidation is phenomenal. IT makes reading your
iptables log reports about 1000 times easier.

Give it a shot and see if you like it - for the majority of rhl logs it
does a beautiful job.

-sv

[Wil Cooley]

On Fri, 2003-09-26 at 17:17, Konstantin Riabitsev wrote:

>  The largest win, in my opinion, is its usefulness in large cluster
>  installations, where logwatch simply doesn't "cut it." Other things I
>  consider just bonuses. 
I'm going to have to look at this, since this is one of the most
bothersome missing features of LogWatch, logcheck, and pflogsumm.

Wil
-- 
Wil Cooley                                 wcooley(a)nakedape.cc
Naked Ape Consulting                        http://nakedape.cc
* * * * * *  Linux Services for Small Businesses  * * * * * *
*       Easy, reliable solutions for small businesses       *
*    Naked Ape Business Server http://nakedape.cc/r/sms     *

[seth vidal]

On Sat, 2003-09-27 at 01:21, Wil Cooley wrote:
>  On Fri, 2003-09-26 at 17:17, Konstantin Riabitsev wrote:
>  
>  > The largest win, in my opinion, is its usefulness in large cluster
>  > installations, where logwatch simply doesn't "cut it." Other things I
>  > consider just bonuses.
>  
>  I'm going to have to look at this, since this is one of the most
>  bothersome missing features of LogWatch, logcheck, and pflogsumm. 
To give you some idea of how much it reduces.

We have 230 systems logging to one loghost. The loghost runs syslog-ng.
nothing special being done with syslog-ng, really. epylog parses logs
once an hour b/t 9am and 9pm and once at 4am.

Our average log report is about 19-30K

it's tidy, it summarizes the info you want to see, and shows you the
aberrations at the end of the report.

We've caught more weirdness b/c it has reduced the crap we don't need to
see.

-sv

[Nicolas Mailhot]

Alexandre Oliva wrote:

> On Sep 26, 2003, Konstantin Riabitsev <icon(a)linux.duke.edu&gt;
> wrote:
>
>   
>
> >Epylog is a syslog parser which runs periodically, looks at your logs,
> >processes some of the entries in order to present them in a more
> >comprehensible format, and then mails you the output.
> >    
> >
>
> Fedora already has LogWatch, and it would suck to get *two* e-mails a
> day for each host :-)
>   
>  
Sure. However Logwatch is rather unsatisfying at times.
And while I do not approve of html mail in general, for the kind of 
summary report this kind of application is supposed to output a nice 
html table with colors would be nice.

-- 
Nicolas Mailhot