apache and app logs retrieval

[seth vidal]

Last week when we were talking about spawning rdiff-backup to backup
our systems, we diverged into discussing app/apache logs and the
somewhat complicated system we currently have for grabbing those logs.

Right now we have a list of hosts on log02 that it should grab logs
from. Those hosts need to have rsyncd running on them to allow access
from log02 to fetch the /var/log/httpd/ path from them.

That requires 2 things to be coupled and it is a bit awkward if you set
up a host that is tricky to access from log02 or isn't on the vpn.

In general I also am not in love with having to have rsyncd listening
on systems - even if it is ip-restricted.

So the thought was we could do something like this on log02:

1. setup an ssh key on log02 that can run rsync to /var/log/httpd on
all hosts
2. make any host that needs to have its logs retrieved be marked in
the ansible inventory host/group vars
3.  git clone public-ansible-repo onto log02
4. use group_by to construct a group of the hosts which can then be
retrieved using rsync. 

The sole reason for using ansible here is so we can keep the log sync
info in our inventory and to parallelize the retrieval of logs.

This is more or less identical to what we talked about for backups
using rdiff-backup.

When we were discussing this Luke mentioned then using
tbgrep(https://pypi.python.org/pypi/tbgrep) to search the resulting
files and compile a set of tracebacks our apps are dumping out.

If we have all the logs on log02 generating a report like this would be
pleasantly kept away from the rest of our hosts and could give us
reasonably useful reports of brokenness.


I'd love some feed back on if this is all crazy or not :)
-sv