rsyslog/epylog reporting

[Kevin Fenzi]

On Mon, 20 Jun 2011 11:31:00 -0400
seth vidal <skvidal at fedoraproject.org> wrote:

> On Mon, 2011-06-20 at 09:12 -0600, Kevin Fenzi wrote:
> 
> > Possibly: 
> > 
> > Staging
> > 	*stg*
> 
>    I've been thinking *Stg* should just go to /dev/null considering
> all the crap they spewed this weekend. :)

well, but it did make us see problems in stg and fix (some) of them. 
I think it's still helpful or else stg will continue to not be
worthwhile. 

> smtp*, collab*, bastion* would cover it for maillogs - but in terms of
> the parser - pflogsumm shouldn't have any issue dealing with our daily
> mail logs globally.
> 
> Mainly what we're looking for there is anomalies and outliers.

Yep. 

> > I wonder if it would be worthwhile to expand on the Total messages
> > weeded thing. Ie, have a number of lines weeded by each regex? 
> > Or would that get too big? That way we could more easily work to
> > reduce the number of lines of logs that we really don't care
> > about. ;) 
> 
> I don't think that information is collected in epylog - it would take
> some structural changes to store it.

Yeah, likely so. I'm just thinking that once we have the report down to
a small amount, we might say "hey, it's all good", when in fact we
should really look at reducing the weed list. If it's a log we don't
care about, why not get it to not log and take up resources in the
first place. ;) 

> > I'm not sure the "SSH scan from" lines are of use. Are we intending
> > to take any action from them? Perhaps a total # so we could see if
> > it was increasing/decreasing? Related: perhaps a denyhosts module?
> 
> the ssh scan is particular to the env that epylog was first developed
> in. Mainly we were on the lookout for systems within a certain ip
> range that were scanning. It meant they had been cracked. 

yeah. I think all our outside machines are going to see this
activity. ;( 

At my old workplace we had a nagios check/firewall log rule for
OUTGOING irc connections. None of our servers did irc or were used by
clients who irc'ed, but it sure was a good way to see when a machine
was compromised. ;) 

> > Kernel oopses might be nice to capture in their own section
> > somehow. That way we can note and investigate them. 
> 
> I was thinking about the oopses some. What we really need is some way
> of saying:
> 
> box1: 12 oopses - same module(s) - see unparsed logs for complete
> pastes box2: 4 oopses - different modules(s) - ...
> 
> not sure how much pain that will be in the modules or not. I just
> don't think having a huge dump of the oops/trace in the middle of the
> formatted log report is going to help much. Then again, in theory, the
> oopses shouldn't happen often I would hope.

yeah. They seem to happen on builders, likely due to the stress they
are under. 

> > Rkhunter logs to it's own log, worth adding a module for it and
> > adding it in here? 
> 
> afaict rkhunter doesn't send anythign to syslog or the central log
> host at all. Not sure how I can get to the data from epylog.

Yeah, it doesn't. ;( It logs locally... aside from 'started' and 'took
x minutes to run' 

> > Would it be possible to expand to httpd logs? I know we have awstats
> > (which needs work), but I was thinking of things like tracebacks
> > from our wsgi applications or other error conditions, not stats
> > related. 
> 
> Not sure - mainly getting the logs into one place so epylog can parse
> them will be the tricky part. the modules make some assumptions about
> file paths which can be a challenge.

yeah, to be clear, I don't want stats gathering, I want error
gathering. things we as admins want to know about and take action on. 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110621/7776f5f9/attachment.bin