Logging.

[Stephen John Smoogen]

Currently logs are 'shipped' to log01 which uses some syntactic magic
to put various logs into hostname appropriate directories.. eg logs
from xen10 go into /var/log/hosts/xen10/. However sometimes this does
not work correctly. hostnames are found via reverse lookups and if no
hostname is found then the IP address is found. So anytime there is a
DNS outage or problem, logs get shoved into directories like
/var/log/hosts/10.5.126.10/

In order to clean up the various spazes, I have moved all the files
into appropriate hostnames and made symbolic links so that IP address
points to hostname.

10.5.126.110 -> xen10

This fixes most of the problems except for some odd directories left
over: exiting, last, gconfd, Mailman, scratch, ServeRAID, syslogd, and
my favorite Rootkit

These are from malformed syslog packets where the ip address got
mangled. I am hoping by moving the boxes to tcp logging these will go
away.

The final directory that is filled with various 'junk' is
/var/log/hosts/unused. Red Hat used a hosting convention that every
reverse IP address not set up is called unused. This means that if DNS
was not set up correctly for a host, it and every other server that
isn't set correctly is called unused. Depending on the month this
might have been 1-4 hosts. I think we are down to 2. After the freeze,
I am going to change the reverse names on 'unused' hosts to better
track this down in the future.

Other things left to do. We need a puppet scriptlet that links forward
and reverse DNS directories and keeps up with them somehow so that
various 1-10 minute poops don't spew logs everywhere again. Second.. I
start concatenating logs but found that htis was error prone as the
logs need to be date sorted as you might have logs going to xen10
direcotires, then 10.5.126.110, then xen10 and back again as whatever
problem fixed itself.




-- 
Stephen J Smoogen.

Ah, but a man's reach should exceed his grasp. Or what's a heaven for?
-- Robert Browning