logging infrastructure and notes
I did a little spelunking around our system and I have some suggestions
for the logging infrastructure. We have enough hosts and complexity that
log analysis will help us know when something is misconfigured or flapping
in a weird way.
1. logs in /var/log/hosts on log1 are not consistently named - sometimes
they are being reported with ips, sometimes with short hostname, sometimes
with fqdn. It needs to be made consistent
2. we need to make sure we cleanup old logs from the above, too.
3. the structure of the log dir doesn't seem to match what we'd normally
see in /var/log on any host. They are being logged as a different dir per
day, which is great, but it'd be good if rsyslog was putting in the same
file structure as a normal set of logs so normal log analysis tools will
work on it
4. I installed pflogsumm on log1 so I could do a little postfix mail log
analysis - found some issues that way too. Regularly generating these
reports, especially the error reports would help us figure out what we
need to improve. We are clearly sending/redelivering A LOT more mail than
we're receiving so bumping our smtp process count would help.
5. Grouping the logs by type of service would also help look at
group/service trending and issues. especially if an issue is only popping
up on one box.
Just some initial thoughts.
-sv