rsyslog/epylog reporting

[Luke Macken]

Excerpts from seth vidal's message of Mon Jun 20 13:03:50 -0400 2011:
> On Mon, 2011-06-20 at 12:51 -0400, Luke Macken wrote:
> > Excerpts from seth vidal's message of Mon Jun 20 10:31:02 -0400 2011:
> > > other ideas?
> > 
> > It would be interesting if we could extract Python tracebacks from our
> > mod_wsgi logs. For TG2, we've got WebError middleware, which can email
> > us tracebacks, but our TG1 apps have plenty of bugs that go unnoticed
> > until someone complains. Another potential solution to this problem
> > would be to setup an app-wide catch-all CherryPy filter[0] that we could
> > use in all of our TG1 apps.  This would also be a solution for "handling
> > FAS outages gracefully"[1].
> > 
> > Also, maybe a mod_security epylog module would be useful as well?
> > 
> 
> I assume the traceback from mod_wsgi logs are just stored on the *app*
> servers locally. If so then we need some way of getting them to a common
> place so we can get a single report.

At the moment we're not very consistent with our app logs. Some throw
their tracebacks in /var/log/httpd/error_log, and some apps, like bodhi,
have their own logs (/var/log/bodhi/server.log).

> Additionally, if we are pushing things over via rsyslog we can setup a
> tool I've used in the past called 'sec' which can trigger events based
> on what shows up in the logs.
> 
> Instead of waiting for an epylog run it checks the incoming log as it
> happens.

This sounds like a good solution.

luke