Re: apache and app logs retrieval

[Kevin Fenzi]

On Tue, 25 Jun 2013 15:16:04 -0400
seth vidal <skvidal(a)fedoraproject.org&gt; wrote:

>  Last week when we were talking about spawning rdiff-backup to backup
>  our systems, we diverged into discussing app/apache logs and the
>  somewhat complicated system we currently have for grabbing those logs.
>  
>  Right now we have a list of hosts on log02 that it should grab logs
>  from. Those hosts need to have rsyncd running on them to allow access
>  from log02 to fetch the /var/log/httpd/ path from them.
>  
>  That requires 2 things to be coupled and it is a bit awkward if you
>  set up a host that is tricky to access from log02 or isn't on the vpn.
>  
>  In general I also am not in love with having to have rsyncd listening
>  on systems - even if it is ip-restricted.
>  
>  So the thought was we could do something like this on log02:
>  
>  1. setup an ssh key on log02 that can run rsync to /var/log/httpd on
>  all hosts
>  2. make any host that needs to have its logs retrieved be marked in
>  the ansible inventory host/group vars
>  3.  git clone public-ansible-repo onto log02
>  4. use group_by to construct a group of the hosts which can then be
>  retrieved using rsync. 
>  
>  The sole reason for using ansible here is so we can keep the log sync
>  info in our inventory and to parallelize the retrieval of logs.
>  
>  This is more or less identical to what we talked about for backups
>  using rdiff-backup. 
Yeah, I think it's worth trying out... we may need to wait until we
have more stuff moved over to ansible just to try and minimize
confusion. 

>  When we were discussing this Luke mentioned then using
>  tbgrep(https://pypi.python.org/pypi/tbgrep) to search the resulting
>  files and compile a set of tracebacks our apps are dumping out. 
Sounds good to me. We might also look at any stats we want to pull out
or other errors we want to note. We could do these similar to the
epylog runs? generate some kind of report link and mail it to
interested parties?
 
>  If we have all the logs on log02 generating a report like this would
>  be pleasantly kept away from the rest of our hosts and could give us
>  reasonably useful reports of brokenness. 
Yeah, I agree... 
>  
>  I'd love some feed back on if this is all crazy or not :) 
I don't think it's crazy at all. 

kevin