rsyslog/epylog reporting

[seth vidal]

On Mon, 2011-06-20 at 09:12 -0600, Kevin Fenzi wrote:

> Possibly: 
> 
> Staging
> 	*stg*

   I've been thinking *Stg* should just go to /dev/null considering all
the crap they spewed this weekend. :)

> Build/Rel-eng
> 	x86* ppc* releng* relepel* sign* compose* koji* 
> rest of everything

+1


> > - the mail/postfix module in epylog needs some more cleanup -
> > alternatively we could not do mail log parsing and let pflogsumm
> > handle it.
> 
> Perhaps pflogsumm on the machines that normally process lots of emails
> and have epylog do something with the logs on machines that do not
> normally process lots of emails. 

smtp*, collab*, bastion* would cover it for maillogs - but in terms of
the parser - pflogsumm shouldn't have any issue dealing with our daily
mail logs globally.

Mainly what we're looking for there is anomalies and outliers.



> > - an rsync module I wrote years ago would be useful for our epylog
> > instance
> 
> Yes. Stats from download* and hosted would be nice. Would show if one
> of the machines was processing less, etc. 

I need to go find that rsync module. I think I know where it is.



> I wonder if it would be worthwhile to expand on the Total messages
> weeded thing. Ie, have a number of lines weeded by each regex? 
> Or would that get too big? That way we could more easily work to reduce
> the number of lines of logs that we really don't care about. ;) 

I don't think that information is collected in epylog - it would take
some structural changes to store it.


> I'm not sure the "SSH scan from" lines are of use. Are we intending to
> take any action from them? Perhaps a total # so we could see if it was
> increasing/decreasing? Related: perhaps a denyhosts module?

the ssh scan is particular to the env that epylog was first developed
in. Mainly we were on the lookout for systems within a certain ip range
that were scanning. It meant they had been cracked. 


> Kernel oopses might be nice to capture in their own section somehow. 
> That way we can note and investigate them. 

I was thinking about the oopses some. What we really need is some way of
saying:

box1: 12 oopses - same module(s) - see unparsed logs for complete pastes
box2: 4 oopses - different modules(s) - ...

not sure how much pain that will be in the modules or not. I just don't
think having a huge dump of the oops/trace in the middle of the
formatted log report is going to help much. Then again, in theory, the
oopses shouldn't happen often I would hope.



> 
> Rkhunter logs to it's own log, worth adding a module for it and adding
> it in here? 

afaict rkhunter doesn't send anythign to syslog or the central log host
at all. Not sure how I can get to the data from epylog.


> Would it be possible to expand to httpd logs? I know we have awstats
> (which needs work), but I was thinking of things like tracebacks from
> our wsgi applications or other error conditions, not stats related. 

Not sure - mainly getting the logs into one place so epylog can parse
them will be the tricky part. the modules make some assumptions about
file paths which can be a challenge.


-sv