rsyslog/epylog reporting

[Kevin Fenzi]

On Mon, 20 Jun 2011 10:31:02 -0400
seth vidal <skvidal at fedoraproject.org> wrote:

> 
> Hi folks, 
> On friday, due to personal irritation, I went ahead and fixed our
> rsyslog config and setup epylog on our central loghost. 
>  epylog: http://fedorahosted.org/epylog/
> 
> 
> sample output: http://fedorapeople.org/~icon/epylog/sample-report.html
> 
> 
> I setup a merged-log location in /var/log/merged on log02. I also
> setup a logrotate job which will rotate those files once a day and
> only keep one additional day's worth of those logs. These are just a
> duplicate output of the logging we have in the per-host directories
> currently, so there is no need to keep them for any amount of time.
> 
> Right now epylog runs once a day but that may change to a couple of
> times a day or more, just until we get a handle on what cruft needs to
> be removed.
> 
> I merged all logs from all hosts into one merged log file, however, I
> think it might be worthwhile to consider breaking these out a bit more
> into sets of hosts. Suggestions on this are welcome.

Possibly: 

Staging
	*stg*
Build/Rel-eng
	x86* ppc* releng* relepel* sign* compose* koji* 
rest of everything

That way if fires are burning we can ignore/not pay as much immediate
attention to the staging ones. 

> - the mail/postfix module in epylog needs some more cleanup -
> alternatively we could not do mail log parsing and let pflogsumm
> handle it.

Perhaps pflogsumm on the machines that normally process lots of emails
and have epylog do something with the logs on machines that do not
normally process lots of emails. 

> - an rsync module I wrote years ago would be useful for our epylog
> instance

Yes. Stats from download* and hosted would be nice. Would show if one
of the machines was processing less, etc. 

> - a sudo-watching module to  dump out in a nice layout all sudo
> commands run anywhere

I look forward to my typos being laughed at. ;) 

> - updating spamd module to get the newer spamassassin outputs
> 
> - fixing the login/sshd module to see pam_unix(sshd:*) properly
> 
> - nagios overview-module:
>    X alerts today
>    X failures today, etc, etc
> 
> 
> other ideas?

I wonder if it would be worthwhile to expand on the Total messages
weeded thing. Ie, have a number of lines weeded by each regex? 
Or would that get too big? That way we could more easily work to reduce
the number of lines of logs that we really don't care about. ;) 

I'm not sure the "SSH scan from" lines are of use. Are we intending to
take any action from them? Perhaps a total # so we could see if it was
increasing/decreasing? Related: perhaps a denyhosts module?

Kernel oopses might be nice to capture in their own section somehow. 
That way we can note and investigate them. 

Rkhunter logs to it's own log, worth adding a module for it and adding
it in here? 

Would it be possible to expand to httpd logs? I know we have awstats
(which needs work), but I was thinking of things like tracebacks from
our wsgi applications or other error conditions, not stats related. 

Just some random thoughts on it. :) Thanks for working on it. 

kevin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110620/534cd886/attachment.bin