Log analyzer improvements, ticket #226

[Michael Yingbull]

Hi all,

I'm following up from ticket #226, which is tracking improvements to the log
analyzer system.
This would be what analyzers the logs on lockbox, which is the syslog host
for infrastructure machines:
https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/226

I wanted to capture what we wanted the new analyzer to do.
Main feedback I had from discussion in #fedora-admin was a need for more
signal, less noise:
the current 'analyzed' logs were too verbose and had too much cruft.

Did I capture that requirement?
Are there other requirements besides improving the presentation?
Anything else that people feel they need from the log analyzer that they
aren't getting?

Currently Epylog is used - I did some looking around, and I'm not seeing
something that looks like its any better.
If someone knows another open source log analyzer they think would be much
better, I'd like to hear.
Else, my plan is to continue with Epylog, reconfigure it... and if really
needed to get what we need, patch it and contribute upstream.

Thanks all, hope everyone is having a good weekend.

Cheers,
Michael

[Mike McGrath]

Michael Yingbull wrote:
>  Hi all,
>
>  I'm following up from ticket #226, which is tracking improvements to the log
>  analyzer system.
>  This would be what analyzers the logs on lockbox, which is the syslog host
>  for infrastructure machines:
>  https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/226
>
>  I wanted to capture what we wanted the new analyzer to do.
>  Main feedback I had from discussion in #fedora-admin was a need for more
>  signal, less noise:
>  the current 'analyzed' logs were too verbose and had too much cruft.
>
>  Did I capture that requirement?
>     
I think this is the biggest thing.  Obviously we don't want to /dev/null 
log lines but at the same time the current format is pretty useless to 
us.  I guess it might be best to do as much cleanup as possible and then 
see where things are.

    -Mike

[Seth Vidal]

On Mon, 2007-11-26 at 08:33 -0600, Mike McGrath wrote:
>  Michael Yingbull wrote:
>  > Hi all,
>  >
>  > I'm following up from ticket #226, which is tracking improvements to the log
>  > analyzer system.
>  > This would be what analyzers the logs on lockbox, which is the syslog host
>  > for infrastructure machines:
>  > https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/226
>  >
>  > I wanted to capture what we wanted the new analyzer to do.
>  > Main feedback I had from discussion in #fedora-admin was a need for more
>  > signal, less noise:
>  > the current 'analyzed' logs were too verbose and had too much cruft.
>  >
>  > Did I capture that requirement?
>  >   
>  
>  I think this is the biggest thing.  Obviously we don't want to /dev/null 
>  log lines but at the same time the current format is pretty useless to 
>  us.  I guess it might be best to do as much cleanup as possible and then 
>  see where things are. 
Actually, there's a huge portion of what is in the current logs that
needs to either:
1. be dumped out by  epylog's weeder
2. be stopped from occurring on the system generating the message.

Michael, if you need any assistance with this, let me know, I have a
fair bit of experience adding weedlists to epylog.

-sv

[Mike McGrath]

seth vidal wrote:
>  On Mon, 2007-11-26 at 08:33 -0600, Mike McGrath wrote:
>    
> > Michael Yingbull wrote:
> >     
> >> Hi all,
> >>
> >> I'm following up from ticket #226, which is tracking improvements to the log
> >> analyzer system.
> >> This would be what analyzers the logs on lockbox, which is the syslog host
> >> for infrastructure machines:
> >> https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/226
> >>
> >> I wanted to capture what we wanted the new analyzer to do.
> >> Main feedback I had from discussion in #fedora-admin was a need for more
> >> signal, less noise:
> >> the current 'analyzed' logs were too verbose and had too much cruft.
> >>
> >> Did I capture that requirement?
> >>   
> >>       
> > I think this is the biggest thing.  Obviously we don't want to /dev/null 
> > log lines but at the same time the current format is pretty useless to 
> > us.  I guess it might be best to do as much cleanup as possible and then 
> > see where things are.
> >     
>
>  Actually, there's a huge portion of what is in the current logs that
>  needs to either:
>  1. be dumped out by  epylog's weeder
>  2. be stopped from occurring on the system generating the message.
>
>  Michael, if you need any assistance with this, let me know, I have a
>  fair bit of experience adding weedlists to epylog.
>     
2) would be more favored by me where possible.

    -Mike

[Seth Vidal]

On Mon, 2007-11-26 at 08:42 -0600, Mike McGrath wrote:

>  2) would be more favored by me where possible.
>   
No problem. From todays' report a couple of things we can do:

1. remove all user failure reports. They don't do us any good and
they're always ssh bruteforce attacks. Denyhosts will do its thing, or
not, but we can't be told about them all the time.

2. weed out pretty much everything beginning with:
rsyncd - informational messages about rsync processes - not useful
puppetd - notices on what is or is not done - not useful, either
        - if we can turn off the syslog component of this and only have 
          this in the local puppet logs that'd be fine
ntpd - garbage noise - not useful for a log report 
git-daemon - do I really need to explain why we can nuke this?


3. all of these lines: 
crond[19403]: pam_unix(crond:session): session closed for user root 

iirc, there is a new login module which handles these

4. puppetmasterd* - these appear to be errors/warnings from
puppetmasterd - these need to be fixed.

pruning out the items in 2 alone will nuke the better part of this
logreport.

-sv