Minor bug report - epylock missing, cron fails
by Jon Peck
Hi,
Thanks for all the recent development, epylog's a great resource!
I think there's a minor oversight in the installer or an OS specific
assumption. Full disclosure, I had been using epylog 1.03 on Ubuntu 10.04
LTS, did purge remove and installed 1.07 from source. With that said, I
don't think that was the cause of the issue.
xxx@yyy:~/tmp/epylog-1.0.7$ sudo /usr/sbin/epylog --cron
Traceback (most recent call last):
File "/usr/sbin/epylog", line 300, in <module>
main(sys.argv)
File "/usr/sbin/epylog", line 257, in main
epylock()
File "/usr/sbin/epylog", line 144, in epylock
if not msg.strerror == "File exists": raise msg
OSError: [Errno 2] No such file or directory: '/usr/var/run/epylog.pid'
xxx@yyy:~/tmp/epylog-1.0.7$ ls -la /usr/var
total 12
drwxr-xr-x 3 root root 4096 2011-09-06 13:25 .
drwxr-xr-x 13 root root 4096 2011-09-06 13:25 ..
drwxr-xr-x 3 root root 4096 2012-02-20 20:20 lib
That explains why there's no file.
xxx@yyy:~/tmp/epylog-1.0.7$ sudo mkdir -p /usr/var/run
Then it executed normally.
Best regards,
Jon Peck
12 years, 4 months
- 1
- 0
[epylog] Created tag v1.0.7
by Konstantin Ryabitsev
The signed tag 'v1.0.7' was created.
Tagger: Konstantin Ryabitsev <mricon(a)kernel.org>
Date: Fri Feb 10 22:22:35 2012 -0500
Really tag for 1.0.7
Changes since the last tag 'v1.0.6':
Konstantin Ryabitsev (2):
Do not return an error on 0-length log files.
Quickfix for 1.0.7 (I hate autoconf)
12 years, 4 months
- 1
- 0
[epylog] Deleted tag v1.0.7
by Konstantin Ryabitsev
The signed tag 'v1.0.7' was deleted. It previously pointed to:
e2f8780... Do not return an error on 0-length log files.
12 years, 4 months
- 1
- 0
[epylog/stable-1.0.x] Quickfix for 1.0.7 (I hate autoconf)
by Konstantin Ryabitsev
commit bf6188bbb334522a91f0626f6c95c9a854977269
Author: Konstantin Ryabitsev <mricon(a)kernel.org>
Date: Fri Feb 10 22:21:38 2012 -0500
Quickfix for 1.0.7 (I hate autoconf)
configure.in | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
---
diff --git a/configure.in b/configure.in
index b003f4e..fce006f 100644
--- a/configure.in
+++ b/configure.in
@@ -1,6 +1,6 @@
dnl Process this file with autoconf to produce a configure script
-AC_INIT(epylog.spec)
+AC_INIT(epylog.in)
PACKAGE_TARNAME=epylog
PACKAGE_VERSION=1.0.7
12 years, 4 months
- 1
- 0
[epylog] Created tag v1.0.7
by Konstantin Ryabitsev
The signed tag 'v1.0.7' was created.
Tagger: Konstantin Ryabitsev <mricon(a)kernel.org>
Date: Fri Feb 10 22:11:28 2012 -0500
Tag for Epylog-1.0.7
Changes since the last tag 'v1.0.6':
Konstantin Ryabitsev (1):
Do not return an error on 0-length log files.
12 years, 4 months
- 1
- 0
[epylog/stable-1.0.x] Do not return an error on 0-length log files.
by Konstantin Ryabitsev
commit e2f8780f9b26316a67e14cdafe2ba06815cb0dbf
Author: Konstantin Ryabitsev <mricon(a)kernel.org>
Date: Fri Feb 10 22:07:29 2012 -0500
Do not return an error on 0-length log files.
ChangeLog | 10 ++-
configure.in | 2 +-
epylog.spec | 199 -------------------------------------------------
py/epylog/__init__.py | 11 +++-
py/epylog/log.py | 32 +++++---
py/epylog/module.py | 7 ++
6 files changed, 45 insertions(+), 216 deletions(-)
---
diff --git a/ChangeLog b/ChangeLog
index a855c09..aca45a9 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,7 @@
+Epylog-1.0.7
+ * Re-apply a fix to not return an error with a 0-length log file.
+ * Remove unmaintained spec file.
+
Epylog-1.0.6
* Back out the unfinished work to support timestamped rotation
(will be implemented fully in 1.1). Fixes the cron-run problem
@@ -30,14 +34,14 @@ Epylog-1.0.1
not work.
* Automatically detect python version during .spec building.
* Cleaned up trojans.list so it's less ugly
-
+
Epylog-1.0
* Out with 1.0 already!
* Handle ::ffff: fake ipv6 addresses for hostname resolution
* Do not depend on elinks in RPM.
* Default setting is to send html-only (so we don't depend on lynx)
* Packets module can now sort by port, system, and source.
-
+
Epylog-0.9.7
* Accepted Makefile patches from Will Newton
* Accepted patches for missing logs from Will Newton (#135)
@@ -57,7 +61,7 @@ Epylog-0.9.5
* Fix for bug #57
* Fix for bug #53
* Cron mode of operation added -- checks for a lockfile (bug #79)
-
+
Epylog-0.9.4
* Fix for bug #38 (incorrect offsets were causing backtrace)
* Normalized logger calls (bug #9)
diff --git a/configure.in b/configure.in
index 50c5d4b..b003f4e 100644
--- a/configure.in
+++ b/configure.in
@@ -2,7 +2,7 @@ dnl Process this file with autoconf to produce a configure script
AC_INIT(epylog.spec)
PACKAGE_TARNAME=epylog
-PACKAGE_VERSION=1.0.6
+PACKAGE_VERSION=1.0.7
dnl Package information.
PACKAGE=$PACKAGE_TARNAME
diff --git a/py/epylog/__init__.py b/py/epylog/__init__.py
index a8cc68c..ab762de 100644
--- a/py/epylog/__init__.py
+++ b/py/epylog/__init__.py
@@ -47,7 +47,7 @@ from report import Report
from module import Module
from log import LogTracker
-VERSION = 'Epylog-1.0.6'
+VERSION = 'Epylog-1.0.7'
CHUNK_SIZE = 8192
GREP_LINES = 10000
QUEUE_LIMIT = 500
@@ -125,6 +125,15 @@ class NoSuchLogError(exceptions.Exception):
logger.put(5, '!NoSuchLogError: %s' % message)
self.args = message
+class EmptyLogError(exceptions.Exception):
+ """
+ This exception is raised when Epylog finds an empty logfile.
+ """
+ def __init__(self, message, logger):
+ exceptions.Exception.__init__(self)
+ logger.put(5, '!EmptyLogError: %s' % message)
+ self.args = message
+
class GenericError(exceptions.Exception):
"""
This exception is raised for all other Epylog conditions.
diff --git a/py/epylog/log.py b/py/epylog/log.py
index 30b2e39..f0750a8 100644
--- a/py/epylog/log.py
+++ b/py/epylog/log.py
@@ -436,10 +436,19 @@ class Log:
self.entry = entry
filename = self._get_filename()
logger.puthang(3, 'Initializing the logfile "%s"' % filename)
- logfile = LogFile(filename, tmpprefix, monthmap, logger)
+ self.loglist = []
+ self.cur_rot_ix = 0
+ try:
+ logfile = LogFile(filename, tmpprefix, monthmap, logger)
+ logger.put(3, 'Appending logfile to the loglist')
+ self.loglist.append(logfile)
+ except epylog.EmptyLogError:
+ logger.endhang(3)
+ logger.puthang(3, '%s is empty, using the previous rotated log'
+ % filename)
+ self._init_next_rotfile()
+ logfile = self.loglist[0]
logger.endhang(3)
- logger.put(3, 'Appending logfile to the loglist')
- self.loglist = [logfile]
self.orange = OffsetRange(0, 0, 0, logfile.end_offset, logger)
logger.endhang(3)
self.lp = None
@@ -805,24 +814,24 @@ class Log:
"""
logger = self.logger
logger.put(5, '>Log._init_next_rotfile')
- ix = len(self.loglist)
- rotname = self._get_rotname_by_ix(ix)
+ self.cur_rot_ix += 1
+ rotname = self._get_rotname_by_ix(self.cur_rot_ix)
try:
logger.put(3, 'Initializing log for rotated file "%s"' % rotname)
rotlog = LogFile(rotname, self.tmpprefix, self.monthmap, logger)
+ self.loglist.append(rotlog)
except epylog.AccessError:
msg = 'No further rotated files for entry "%s"' % self.entry
raise epylog.NoSuchLogError(msg, logger)
- self.loglist.append(rotlog)
+ except epylog.EmptyLogError:
+ msg = 'Found an empty rotated log, ignoring it.'
+ rotlog = self._init_next_rotfile()
logger.put(5, '<Log._init_next_rotfile')
return rotlog
def _get_rotname_by_ix(self, ix):
"""
- The good thing about rotated files is that they are exactly at the same
- position in the log list, as the identifier appended to them by
- logrotate. E.g. messages.1 will be at position 1, messages.2 at
- position 2, and just messages at position 0.
+ Figure out the rotated file name by index passed.
"""
logger = self.logger
logger.put(5, '>Log._get_rotname_by_ix')
@@ -1300,8 +1309,7 @@ class LogFile:
logger.put(3, 'Making it 0')
stamp = 0
else:
- logger.put(5, 'Nothing in the range')
- stamp = 0
+ raise epylog.EmptyLogError('%s is empty' % self.filename, logger)
logger.put(5, '<LogFile._get_stamp')
return stamp
diff --git a/py/epylog/module.py b/py/epylog/module.py
index 7eaded2..600c5a1 100644
--- a/py/epylog/module.py
+++ b/py/epylog/module.py
@@ -123,6 +123,13 @@ class Module:
#
logger.put(0, 'Could not init logfile for entry "%s"' % entry)
continue
+ except epylog.NoSuchLogError:
+ ##
+ # Looks like all logfiles for this log entry are empty.
+ # Ignore this log entry.
+ logger.put(1, ('No logs found for %s, or they are all empty, '+
+ 'ignoring.') % entry)
+ continue
logger.put(5, 'Appending the log object to self.logs[]')
self.logs.append(log)
if len(self.logs) == 0:
12 years, 4 months
- 1
- 0
parsing Apache logs?
by Warren Myers
Maybe my google-fu is failing me, but is there a way for Epylog to handle
Apache logs, too?
--
Warren Myers
http://warrenmyers.com
http://twitter.com/warrenmyers
http://www.linkedin.com/in/warrenmyers
12 years, 4 months
- 2
- 1
[epylog] Big changes for 1.1.
by Konstantin Ryabitsev
commit 5a4dfaf114bab217c1196da530c1b2f3566bfe00
Author: Konstantin Ryabitsev <mricon(a)kernel.org>
Date: Thu Feb 9 10:31:24 2012 -0500
Big changes for 1.1.
- Ripped out autoconf
- Dropped any support for external modules
- Removed perl bits
- Switched to using python's own logging module
- Define logs in the logsources.conf file
- Switch notices from xml to yaml (not working at all yet)
- Move stuff around a bit for simplicity's sake
TODO:
- Modify modules to work with the new logging changes
- Fix notices module to work with yaml and the new format
- Support for pre-matching based on tag
- Fix support for storing and retrieving offsets
(running epylog from cron is broken for now, only with --last)
- Whatever else requires fixing
ChangeLog | 11 +-
Makefile.in | 92 --
compiledir.in | 12 -
configure.in | 159 ---
cron/Makefile.in | 66 -
cron/epylog.cron | 2 +
cron/epylog.cron.in | 6 -
epylog.in => epylog.py | 267 +++--
epylog.spec | 199 ---
{py/epylog => epylog}/__init__.py | 690 +++++-----
{py/epylog => epylog}/helpers.py | 109 +-
epylog/log.py | 1393 +++++++++++++++++++
epylog/module.py | 296 +++++
{py/epylog => epylog}/mytempfile.py | 0
epylog/publishers.py | 657 +++++++++
epylog/report.py | 232 ++++
etc/Makefile.in | 81 --
etc/{epylog.conf.in => epylog.conf} | 43 +-
etc/logsources.conf | 19 +
etc/modules.d/Makefile.in | 67 -
...ommon_unparsed.conf.in => common_unparsed.conf} | 9 +-
etc/modules.d/{logins.conf.in => logins.conf} | 18 +-
etc/modules.d/{mail.conf.in => mail.conf} | 9 +-
etc/modules.d/{notices.conf.in => notices.conf} | 13 +-
etc/modules.d/ntp.conf | 7 +
etc/modules.d/ntp.conf.in | 8 -
etc/modules.d/{packets.conf.in => packets.conf} | 11 +-
etc/modules.d/{rsyncd.conf.in => rsyncd.conf} | 7 +-
etc/modules.d/selinux.conf | 10 +
etc/modules.d/selinux.conf.in | 11 -
etc/modules.d/smart.conf | 7 +
etc/modules.d/smart.conf.in | 8 -
etc/modules.d/{spamd.conf.in => spamd.conf} | 7 +-
etc/modules.d/sudo.conf | 9 +
etc/modules.d/sudo.conf.in | 11 -
etc/modules.d/weeder.conf | 10 +
etc/modules.d/weeder.conf.in | 30 -
etc/modules.d/yum.conf | 9 +
etc/modules.d/yum.conf.in | 11 -
etc/notice_dist.xml | 105 --
etc/notice_dist.yaml | 85 ++
etc/notice_local.xml | 19 -
etc/notice_local.yaml | 18 +
etc/report_template.html | 14 +-
etc/trojans.list | 410 ------
install-sh | 251 ----
man/Makefile.in | 59 -
mkinstalldirs | 40 -
modules/Makefile.in | 62 -
modules/sudo_mod.py | 51 +-
modules/weeder_mod.py | 85 +-
modules/yum_mod.py | 53 +-
perl/Makefile.in | 56 -
perl/epylog.pm | 428 ------
py/Makefile.in | 58 -
py/epylog/log.py | 1401 --------------------
py/epylog/module.py | 403 ------
py/epylog/publishers.py | 608 ---------
py/epylog/report.py | 355 -----
59 files changed, 3505 insertions(+), 5662 deletions(-)
---
diff --git a/ChangeLog b/ChangeLog
index 9e0b3c7..9ffeec8 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,12 @@
+Epylog-1.1
+ * Rip out autoconf
+ * Rework to use python's internal logging module
+ * Use .format() wherever possible
+ * Define log sources in logsources.conf
+ * Use python's template language for most things
+
+-------------------------------------------------------------------------------
+
Epylog-1.0.4
* Be more lenient about syslog format (FC7 changes) (ticket #4)
* Add a "save_rawlogs" option to file publisher, and don't save them
@@ -28,7 +37,7 @@ Epylog-1.0
* Do not depend on elinks in RPM.
* Default setting is to send html-only (so we don't depend on lynx)
* Packets module can now sort by port, system, and source.
-
+
Epylog-0.9.7
* Accepted Makefile patches from Will Newton
* Accepted patches for missing logs from Will Newton (#135)
diff --git a/cron/epylog.cron b/cron/epylog.cron
new file mode 100644
index 0000000..9555571
--- /dev/null
+++ b/cron/epylog.cron
@@ -0,0 +1,2 @@
+#!/bin/sh
+/usr/sbin/epylog --cron
diff --git a/epylog.in b/epylog.py
similarity index 55%
rename from epylog.in
rename to epylog.py
index 68abbb5..40ceedf 100644
--- a/epylog.in
+++ b/epylog.py
@@ -1,6 +1,7 @@
-#!%%PYTHON_BIN%%
+#!/usr/bin/python -tt
##
-# Copyright (C) 2003 by Duke University
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
@@ -17,23 +18,19 @@
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
# 02111-1307, USA.
#
-# $Id$
-#
-# @Author Konstantin Ryabitsev <icon(a)linux.duke.edu>
-# @version $Date$
-#
import os
import sys
-import getopt
import time
-import libxml2
-sys.path.insert(0, '%%PY_MODULE_DIR%%')
-from epylog import *
+from optparse import OptionParser
+
+from epylog import Epylog, VERSION, ConsoleUi, ConfigError, ModuleError
-DEFAULT_EPYLOG_CONFIG = '%%pkgconfdir%%/epylog.conf'
-EPYLOG_PIDFILE = '%%localstatedir%%/run/epylog.pid'
+import logging
+
+DEFAULT_EPYLOG_CONFIG = '/etc/epylog/epylog.conf'
+EPYLOG_PIDFILE = '/var/run/epylog.pid'
def unxmlify_offsets(ofile, logger):
"""
@@ -173,133 +170,191 @@ def parselast(last):
"""
Make sense of the --last value
"""
- msg = "Unknown setting for --last: %s. See --help" % last
- if last == 'hour': last = '1h'
- elif last == 'day': last = '1d'
- elif last == 'week': last = '1w'
- elif last == 'month': last = '1m'
- cat = last[-1:].lower()
- try: num = int(last[:-1])
- except: sys.exit(msg)
- if cat == 'h': mult = 1
- elif cat == 'd': mult = 24
- elif cat == 'w': mult = 24*7
- elif cat == 'm': mult = 24*30
- else: sys.exit(msg)
- now = int(time.time())
- then = now - (num * mult * 60 * 60)
- return then
+ msg = "Unknown setting for --last: {}. Seek --help.".format(last)
-def usage():
- print """
- Usage: epylog [--quiet] [--store-offsets] [--last] [-c] [-d]
+ wordmap = {
+ 'hour' : '1h',
+ 'day' : '1d',
+ 'week' : '1w',
+ 'month' : '1m'
+ }
- -c config-file
- read a custom config file instead of /etc/epylog/epylog.conf
+ # seconds
+ multmap = {
+ 'h' : 3600,
+ 'd' : 86400,
+ 'w' : 604800,
+ 'm' : 2592000
+ }
- -d debug-level
- a number from 0 to 5. 0 means only critical output, while 5
- means lots and lots of debugging info.
+ if last in wordmap.keys():
+ last = wordmap[last]
- --store-offsets
- this will store an offset.xml file in /var/lib/epylog. This
- is useful when running epylog from cron, since then it relies
- on actual offsets as opposed to timestamps, which do not have to
- be accurate.
+ cat = last[-1:].lower()
- --quiet
- completely identical to -d 0
+ if cat not in multmap.keys():
+ sys.exit(msg)
- --cron
- Equivalent of --quiet --store-offsets, plus it will create a
- lock file that will not allow more than one cron instance of
- epylog to run.
+ try:
+ num = int(last[:-1])
+ except ValueError:
+ sys.exit(msg)
- --last [hour|day|week|month|Nh|Nd|Nw|Nm]
- will analyze strings from the past [time period] specified.
+ now = int(time.time())
+ then = now - (num * multmap[cat])
- If no command-line options are provided, then the logs will be
- processed in their entirety (WARNING: this can mean a LOT of logs).
- A useful way to init a system would be to run:
- epylog --last [hour|day] --store-offsets
+ return then
- Example:
- epylog --last day
+def legacyput(level, message):
+ """
+ Legacy wrapper for the old-style logger.
"""
- sys.exit(1)
+ logger = logging.getLogger('epylog')
+
+ message = 'LEGACY PUT: ' + message
+
+ if level < 3:
+ logger.warning(message)
+ return
+
+ if level == 3:
+ logger.info(message)
+ return
+
+ logger.debug(message)
+ return
+
+def legacyendhang(level, message='done'):
+ return
def main(args):
- debuglvl = 1
- o_stor = 0
- o_stamp = 0
- o_cron = 0
- config_file = DEFAULT_EPYLOG_CONFIG
- cmdargs = args[1:]
- try:
- gopts, cmds = getopt.getopt(cmdargs, 'd:c:h',
- ['quiet', 'store-offsets', 'last=',
- 'help', 'cron'])
- for o,a in gopts:
- if o == '-d': debuglvl = int(a)
- elif o == '--quiet': debuglvl = 0
- elif o == '--store-offsets': o_stor = 1
- elif o == '--cron': o_cron = 1
- elif o == '--last': o_stamp = parselast(a)
- elif o == '-c': config_file = a
- elif o == '-h' or o == '--help': usage()
- except getopt.error, e:
- print 'Error: %s' % e
- usage()
- if o_cron:
- ##
- # Cron mode. Try to lock, and set --quiet and --store-offsets
- #
+
+ usage = ''' Usage: %prog [options]
+ Analyze logs and produce a log report.'''
+
+ parser = OptionParser(usage=usage, version=VERSION)
+
+ parser.add_option('--debuglevel', '-d', dest='o_debug',
+ default=1,
+ help=('a number from 0 to 5. 0 means only critical output, while '
+ '5 means lots and lots of debugging info.'))
+ parser.add_option('--quiet', '-q', dest='o_quiet', action='store_true',
+ default=False,
+ help='Completely identical to -d 0')
+ parser.add_option('--store-offsets', '-s', dest='o_stor',
+ action='store_true', default=False,
+ help=('After processing the logs, record the position of the '
+ 'latest location of each logfile. This is useful when running '
+ 'epylog from cron, since then it relies on the latest-processed '
+ 'position within the logfile and not on timestamps, which do '
+ 'not have to be accurate.'))
+ parser.add_option('--cron', '-n', dest='o_cron', action='store_true',
+ default=False,
+ help=('Equivalent of --quiet --store-offsets, plus it will create a '
+ 'lock file that will not allow more than one cron instance of '
+ 'epylog to run.'))
+ parser.add_option('--last', '-l', dest='o_last',
+ default=None,
+ help=('Will analyze entries from the past '
+ '[hour|day|week|month|Nh|Nd|Nw|Nm] specified. A useful '
+ 'way to init epylog is to run:\n'
+ 'epylog --last day --store-offsets'))
+ parser.add_option('--config-file', '-c', dest='o_config',
+ default=DEFAULT_EPYLOG_CONFIG,
+ help=('read a custom config file instead of %default'))
+
+ (opts, args) = parser.parse_args()
+
+ if args:
+ parser.error('Epylog takes no arguments.')
+
+ if opts.o_quiet:
+ debuglvl = logging.CRITICAL
+ else:
+ debuglvl = 50 - (10*int(opts.o_debug))
+ if debuglvl < 10:
+ debuglvl = 10
+ elif debuglvl > 50:
+ debuglvl = 50
+
+ if opts.o_cron:
epylock()
- o_stor = 1
- debuglvl = 0
- logger = Logger(debuglvl)
- logger.puthang(1, 'Initializing epylog')
+
+ o_stor = True
+ debuglvl = logging.CRITICAL
+ opts.o_quiet = True
+
+ logger = logging.getLogger('epylog')
+ logger.setLevel(debuglvl)
+
+ ch = logging.StreamHandler()
+ ch.setLevel(debuglvl)
+ formatter = logging.Formatter("[%(levelname)s:%(module)s:%(funcName)s:"
+ "%(lineno)s] %(message)s")
+ ch.setFormatter(formatter)
+ logger.addHandler(ch)
+
+ o_stor = opts.o_stor
+ o_cron = opts.o_cron
+ config_file = opts.o_config
+
+ #logger.put = legacyput
+ #logger.puthang = legacyput
+ #logger.endhang = legacyendhang
+
+ logger.debug('Config file set to {}'.format(config_file))
+
+ if opts.o_last is not None:
+ o_stamp = parselast(opts.o_last)
+
+ ui = ConsoleUi(opts.o_quiet)
+ ui.puthang('Initializing {}'.format(VERSION))
+
try:
- epylog = Epylog(config_file, logger)
+ epylog = Epylog(config_file, ui)
except (ConfigError, ModuleError), e:
- logger.put(0, "Error returned: %s" % e)
+ logger.critical("Error returned: {}".format(e))
sys.exit(1)
- logger.endhang(1, 'done')
- if o_stamp == 0:
+
+ ui.endhang('done')
+
+ if not opts.o_last:
+ # TODO: Totally broken for now
logger.puthang(1, 'Restoring log offsets')
restore_offsets(epylog)
logger.endhang(1, 'done')
else:
- logger.puthang(1, 'Setting the offsets by timestamp')
+ ui.puthang('Setting the offsets by timestamp')
epylog.logtracker.set_range_by_timestamps(o_stamp, int(time.time()))
- logger.endhang(1)
- logger.put(1, 'Invoking the module execution routines:')
+ ui.endhang('done')
+
+ ui.put('Invoking the module execution routines:')
epylog.process_modules()
- logger.put(1, 'Finished processing modules')
- logger.puthang(1, 'Making the report')
+ ui.put('Finished processing modules')
+
+ ui.puthang('Making the report')
useful = epylog.make_report()
- logger.endhang(1, 'done')
+ ui.endhang('done')
+
if useful:
- logger.puthang(1, 'Publishing the report')
+ ui.puthang('Publishing the report')
epylog.publish_report()
- logger.endhang(1, 'done')
+ ui.endhang('done')
+
if o_stor:
logger.puthang(1, 'Storing the offsets')
store_offsets(epylog)
logger.endhang(1, 'done')
else:
- logger.put(1, 'Report is empty. Exiting.')
+ ui.put('Report is empty. Exiting.')
- logger.puthang(1, 'Cleaning up')
+ ui.puthang('Cleaning up')
epylog.cleanup()
- logger.endhang(1, 'done')
- if o_cron: epyunlock()
+ ui.endhang('done')
+ if o_cron:
+ epyunlock()
if __name__ == '__main__':
main(sys.argv)
-##
-# local variables:
-# mode: python
-# end:
diff --git a/py/epylog/__init__.py b/epylog/__init__.py
similarity index 50%
rename from py/epylog/__init__.py
rename to epylog/__init__.py
index ea1827a..be284a7 100644
--- a/py/epylog/__init__.py
+++ b/epylog/__init__.py
@@ -3,6 +3,7 @@ This module contains the main classes and methods for Epylog.
"""
##
# Copyright (C) 2003 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
@@ -19,10 +20,7 @@ This module contains the main classes and methods for Epylog.
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
# 02111-1307, USA.
#
-# $Id$
-#
-# @Author Konstantin Ryabitsev <icon(a)linux.duke.edu>
-# @version $Date$
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
#
import ConfigParser
@@ -36,73 +34,69 @@ import pwd
import socket
import sys
-if 'mkdtemp' not in dir(tempfile):
- ##
- # Must be python < 2.3
- #
- del tempfile
- import mytempfile as tempfile
+from string import Template
from report import Report
from module import Module
-from log import LogTracker
+from log import LogTracker
+
+import logging
-VERSION = 'Epylog-1.0.3'
-CHUNK_SIZE = 8192
-GREP_LINES = 10000
+VERSION = 'Epylog-1.0.3'
+CHUNK_SIZE = 8192
+GREP_LINES = 10000
QUEUE_LIMIT = 500
-LOG_SPLIT_RE = re.compile(r'(.{15,15})\s+(\S+)\s+(.*)$')
-SYSLOG_NG_STRIP = re.compile(r'.*[@/]')
+
+# TODO: support other timestamp formats
+LOG_SPLIT_RE = re.compile(r'(.{15,15})\s+(\S+)\s+(.*)$')
+SYSLOG_NG_STRIP = re.compile(r'.*[@/]')
MESSAGE_REPEATED_RE = re.compile(r'last message repeated (\S+) times')
+logger = logging.getLogger('epylog')
+
class FormatError(exceptions.Exception):
"""
This exception is raised when there are problems with the syslog
line processed.
"""
- def __init__(self, message, logger):
+ def __init__(self, message):
exceptions.Exception.__init__(self)
- logger.put(5, '!FormatError: %s' % message)
- self.args = message
+ logger.debug('!FormatError: {}'.format(message))
class ConfigError(exceptions.Exception):
"""
This exception is raised when there are misconfiguration problems.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!ConfigError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!ConfigError: {}'.format(message))
class AccessError(exceptions.Exception):
"""
This exception is raised when there are errors accessing certain
components of Epylog, log files, or temporary writing spaces.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!AccessError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!AccessError: {}'.format(message))
class OutOfRangeError(exceptions.Exception):
"""
This happens when Epylog tries to access a line in a logfile that is
outside the specified range.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!OutOfRangeError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!OutOfRangeError: {}'.format(message))
class ModuleError(exceptions.Exception):
"""
This exception is raised when an Epylog module crashes or otherwise
creates a problem.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!ModuleError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!ModuleError: {}'.format(message))
class SysCallError(exceptions.Exception):
"""
@@ -110,272 +104,252 @@ class SysCallError(exceptions.Exception):
successful. Most notable ones are grep (only used with external modules)
and lynx/links/w3m.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!SysCallError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!SysCallError: {}'.format(message))
class NoSuchLogError(exceptions.Exception):
"""
This exception is raised when Epylog tries to access or initialize a
logfile that does not exist.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!NoSuchLogError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!NoSuchLogError: {}'.format(message))
class GenericError(exceptions.Exception):
"""
This exception is raised for all other Epylog conditions.
"""
- def __init__(self, message, logger):
- exceptions.Exception.__init__(self)
- logger.put(5, '!GenericError: %s' % message)
- self.args = message
+ def __init__(self, message):
+ exceptions.Exception.__init__(self, message)
+ logger.debug('!GenericError: {}'.format(message))
class Epylog:
"""
This is the core class of Epylog. A UI would usually communicate
with it an it only.
"""
- def __init__(self, cfgfile, logger):
+ def __init__(self, cfgfile, ui):
"""
- UIs may override the included logger, which would be useful for
+ UIs may override the included ui, which would be useful for
things like a possible GTK interface, a web interface, etc.
"""
- self.logger = logger
- logger.put(5, '>Epylog.__init__')
+ self.ui = ui
config = ConfigParser.ConfigParser()
- logger.puthang(3, 'Reading the config file "%s"' % cfgfile)
- try: config.read(cfgfile)
+ logger.info('Reading the config file "{}"'.format(cfgfile))
+
+ try:
+ config.read(cfgfile)
except:
- msg = 'Could not read/parse config file "%s"' % cfgfile
- raise ConfigError(msg, logger)
- logger.endhang(3)
- ##
- # Read in the main configuration
- #
- logger.puthang(3, "Reading in main entries")
+ msg = 'Could not read/parse config file "{}"'.format(cfgfile)
+ raise ConfigError(msg)
+
+ logger.debug('Reading in main entries')
+
try:
self.cfgdir = config.get('main', 'cfgdir')
self.vardir = config.get('main', 'vardir')
except:
- msg = 'Could not parse the main config file "%s"' % cfgfile
- raise ConfigError(msg, logger)
- logger.put(5, 'cfgdir=%s' % self.cfgdir)
- logger.put(5, 'vardir=%s' % self.vardir)
- logger.endhang(3)
-
- logger.put(3, 'Checking if we can write to vardir')
+ msg = 'Could not parse the main config file "{}"'.format(cfgfile)
+ raise ConfigError(msg)
+
+ # use this to perform template operations
+ config.paths = {
+ 'cfgdir': self.cfgdir,
+ 'vardir': self.vardir
+ }
+
+ moduledir = config.get('main', 'moduledir')
+
+ self.moduledir = Template(moduledir).safe_substitute(config.paths)
+ config.paths['moduledir'] = self.moduledir
+
+ logger.debug(('cfgdir={self.cfgdir}, vardir={self.vardir}, '
+ 'moduledir={self.moduledir}').format(self=self))
+
if not os.access(self.vardir, os.W_OK):
- msg = 'Write access required for vardir "%s"' % self.vardir
- raise ConfigError(msg, logger)
+ msg = 'Write access required for vardir "{}"'.format(self.vardir)
+ raise ConfigError(msg)
##
# Set up a safe temp dir
#
- logger.put(3, 'Setting up a temporary directory')
+ logger.info('Setting up a temporary directory')
try:
tmpdir = config.get('main', 'tmpdir')
+ tmpdir = Template(tmpdir).safe_substitute(config.paths)
+ logger.debug('tmpdir={}'.format(tmpdir))
tempfile.tempdir = tmpdir
- except: pass
- logger.put(3, 'Creating a safe temporary directory')
- try: tmpprefix = tempfile.mkdtemp('EPYLOG')
+ except:
+ # Will use OS-default temp dir
+ pass
+
+ logger.info('Creating a safe temporary directory')
+ try:
+ tmpprefix = tempfile.mkdtemp('.EPYLOG')
except:
- msg = 'Could not create a safe temp directory in "%s"' % tmpprefix
- raise ConfigError(msg, logger)
- self.tmpprefix = tmpprefix
+ msg = 'Could not create a temp directory in "{}"'.format(tmpprefix)
+ raise ConfigError(msg)
+
+ self.tmpprefix = tmpprefix
tempfile.tempdir = tmpprefix
- logger.put(3, 'Temporary directory created in "%s"' % tmpprefix)
- logger.put(3, 'Sticking tmpprefix into config to pass to other objs')
- config.tmpprefix = self.tmpprefix
+
+ logger.info('Temporary directory created in "{}"'.format(tmpprefix))
+ config.paths['tmpprefix'] = tmpprefix
+ config.tmpprefix = tmpprefix
+
##
# Create a file for unparsed strings.
#
- self.unparsed = tempfile.mktemp('UNPARSED')
- logger.put(3, 'Unparsed strings will go into %s' % self.unparsed)
+ self.unparsed = tempfile.mktemp('.UNPARSED')
+ logger.info('Unparsed strings will go into {}'.format(self.unparsed))
+
##
# Get multimatch pref
#
- try: self.multimatch = config.getboolean('main', 'multimatch')
- except: self.multimatch = 0
- logger.put(5, 'multimatch=%d' % self.multimatch)
+ try:
+ self.multimatch = config.getboolean('main', 'multimatch')
+ except:
+ self.multimatch = False
+
+ logger.debug('multimatch={}'.format(self.multimatch))
+
##
# Get threading pref
#
try:
threads = config.getint('main', 'threads')
if threads < 2:
- logger.put(0, 'Threads set to less than 2, fixing')
+ logger.error('Threads set to less than 2, fixing')
threads = 2
self.threads = threads
except:
self.threads = 50
- logger.put(5, 'threads=%d' % self.threads)
+
+ logger.debug('threads={}'.format(self.threads))
+
##
# Initialize the Report object
#
- logger.puthang(3, 'Initializing the Report')
- self.report = Report(config, logger)
- logger.endhang(3)
+ logger.info('Initializing the Report')
+ self.report = Report(config, ui)
##
# Initialize the LogTracker object
#
- logger.puthang(3, 'Initializing the log tracker object')
- logtracker = LogTracker(config, logger)
+ logger.info('Initializing the log tracker object')
+ logtracker = LogTracker(config, ui)
+
self.logtracker = logtracker
- logger.endhang(3)
- ##
- # Process module configurations
- #
+
+ logger.info('Processing module configurations')
+
self.modules = []
- priorities = []
+ priorities = []
+
modcfgdir = os.path.join(self.cfgdir, 'modules.d')
- logger.put(3, 'Checking if module config dir "%s" exists' % modcfgdir)
+ logger.debug('modcfgdir={}'.format(modcfgdir))
+
if not os.path.isdir(modcfgdir):
- msg = 'Module configuration directory "%s" not found' % modcfgdir
- raise ConfigError(msg, logger)
- logger.put(3, 'Looking for module configs in %s' % modcfgdir)
+ msg = 'modules.d not found in "{}"'.format(modcfgdir)
+ raise ConfigError(msg)
+
for file in os.listdir(modcfgdir):
cfgfile = os.path.join(modcfgdir, file)
- if os.path.isfile(cfgfile):
- logger.put(3, 'Found file: %s' % cfgfile)
- if not re.compile('\.conf$').search(cfgfile, 1):
- logger.put(3, 'Not a module config file, skipping.')
- continue
- logger.puthang(3, 'Calling the Module init routines')
- try:
- module = Module(cfgfile, logtracker, tmpprefix, logger)
- except (ConfigError, ModuleError), e:
- msg = 'Module Error: %s' % e
- logger.put(0, msg)
- continue
- logger.endhang(3)
- if module.enabled:
- logger.put(3, 'Module "%s" is enabled' % module.name)
- module.sanity_check()
- self.modules.append(module)
- priorities.append(module.priority)
- else:
- logger.put(3, 'Module "%s" is not enabled, ignoring'
- % module.name)
+
+ if not os.path.isfile(cfgfile) or cfgfile[-5:] != '.conf':
+ logger.debug('Not a config file: {}'.format(cfgfile))
+ continue
+
+ logger.info('Found module config file: {}'.format(cfgfile))
+
+ logger.info('Calling the Module init routines')
+ try:
+ module = Module(cfgfile, logtracker, config, ui)
+ except (ConfigError, ModuleError), e:
+ logger.error('Module error: {}'.format(e))
+ continue
+
+ if module.enabled:
+ logger.info('Module "{}" is enabled'.format(module.name))
+ self.modules.append(module)
+ priorities.append(module.priority)
else:
- logger.put(3, '%s is not a regular file, ignoring' % cfgfile)
- logger.put(3, 'Total of %d modules initialized' % len(self.modules))
+ logger.info('Module "{}" is not enabled, ignoring'.format(
+ module.name))
+
+ logger.info('{} modules initialized'.format(len(self.modules)))
+
if len(self.modules) == 0:
- raise ConfigError('No modules are enabled. Exiting.', logger)
+ raise ConfigError('No modules are enabled. Exiting.')
+
##
# Sort modules by priority
#
- logger.put(3, 'sorting modules by priority')
priorities.sort()
for module in self.modules:
- logger.put(3, 'analyzing module: %s' % module.name)
+ logger.debug('analyzing module pritory: {}'.format(module.name))
for i in range(0, len(priorities)):
- try:
- logger.put(5, 'module.priority=%d, priorities[i]=%d'
- % (module.priority, priorities[i]))
- except:
- logger.put(5, 'priorities[i] is module: %s'
- % priorities[i].name)
if module.priority == priorities[i]:
priorities[i] = module
- logger.put(5, 'priorities[i] is now: %s' % module.name)
break
+
self.modules = priorities
- self.imodules = []
- self.emodules = []
- for module in self.modules:
- logger.put(5, 'module: %s, priority: %d'
- % (module.name, module.priority))
- if module.is_internal(): self.imodules.append(module)
- else: self.emodules.append(module)
- logger.put(5, '<Epylog.__init__')
def process_modules(self):
"""
Invoke the modules to process the logfile entries.
"""
- logger = self.logger
- logger.put(5, '>Epylog.process_modules')
- logger.put(3, 'Finding internal modules')
- if len(self.imodules):
+ if len(self.modules):
self._process_internal_modules()
- if len(self.emodules):
- logger.puthang(3, 'Processing external modules')
- for module in self.emodules:
- logger.puthang(1, 'Processing module "%s"' % module.name)
- try:
- module.invoke_external_module(self.cfgdir)
- except ModuleError, e:
- ##
- # Module execution error!
- # Do not die, but provide a visible warning.
- #
- logger.put(0, str(e))
- logger.endhang(1, 'done')
- logger.endhang(3)
- logger.put(5, '<Epylog.process_modules')
def make_report(self):
"""
Create the report based on the result of the Epylog run.
"""
- logger = self.logger
- logger.put(5, '>Epylog.make_report')
for module in self.modules:
- logger.put(3, 'Analyzing reports from module "%s"' % module.name)
- logger.put(5, 'logerport=%s' % module.logreport)
- logger.put(5, 'logfilter=%s' % module.logfilter)
+ logger.info('Analyzing reports from "{}"'.format(module.name))
+ logger.debug('logerport={}'.format(module.logreport))
+ logger.debug('logfilter={}'.format(module.logfilter))
+
if module.logreport is None and module.logfilter is None:
- logger.put(3, 'No output from module "%s"' % module.name)
- logger.put(3, 'Skipping module "%s"' % module.name)
+ logger.info('No output from "{}"'.format(module.name))
+ logger.info('Skipping module "{}"'.format(module.name))
continue
- logger.put(3, 'Preparing a report for module "%s"' % module.name)
+
+ logger.info('Preparing a report for "{}"'.format(module.name))
module_report = module.get_html_report()
+
if module_report is not None:
self.report.append_module_report(module.name, module_report)
- if self.emodules:
- ##
- # We only need filtered strings if we have external modules
- #
- fsfh = module.get_filtered_strings_fh()
- self.report.append_filtered_strings(module.name, fsfh)
- fsfh.close()
+
self.report.set_stamps(self.logtracker.get_stamps())
- logger.put(5, '<Epylog.make_report')
return self.report.is_report_useful()
def publish_report(self):
"""
Publish the report.
"""
- logger = self.logger
- logger.put(5, '>Epylog.publish_report')
- logger.put(3, 'Dumping all log strings into a temp file')
+ logger.info('Dumping all log strings into a temp file')
+
tempfile.tempdir = self.tmpprefix
- rawfh = open(tempfile.mktemp('RAW'), 'w+')
- logger.put(3, 'RAW strings file created in "%s"' % rawfh.name)
+ rawfh = open(tempfile.mktemp('.RAW'), 'w+')
+
+ logger.info('RAW strings file created in "{}"'.format(rawfh.name))
+
self.logtracker.dump_all_strings(rawfh)
- if not self.emodules:
- ##
- # All modules were internal, meaning we have all unparsed
- # strings in the self.unparsed file.
- #
- unparsed = self._get_unparsed()
- else: unparsed = None
+
+ unparsed = self._get_unparsed()
+
self.report.publish(rawfh, unparsed)
- logger.put(5, '<Epylog.publish_report')
def cleanup(self):
"""
Clean up after ourselves.
"""
- logger = self.logger
- logger.put(3, 'Cleanup routine called')
- logger.put(3, 'Removing the temp dir "%s"' % self.tmpprefix)
+ logger.info('Cleanup routine called')
+ logger.info('Removing the temp dir "{}"'.format(self.tmpprefix))
shutil.rmtree(self.tmpprefix)
def _get_unparsed(self):
@@ -391,133 +365,148 @@ class Epylog:
"""
Invoke and process internal (python) modules.
"""
- logger = self.logger
- logger.put(5, '>Epylog._process_internal_modules')
- logger.puthang(1, 'Processing internal modules')
- logger.put(3, 'Collecting logfiles used by internal modules')
upfh = open(self.unparsed, 'w')
- logger.put(3, 'Opened unparsed strings file in "%s"' % self.unparsed)
+ logger.info('Opened unparsed strings file in {}'.format(self.unparsed))
+
logmap = {}
- for module in self.imodules:
+ for module in self.modules:
for log in module.logs:
- try: logmap[log.entry].append(module)
- except KeyError: logmap[log.entry] = [module]
- logger.put(5, 'logmap follows')
- logger.put(5, logmap)
- pq = ProcessingQueue(QUEUE_LIMIT, logger)
- logger.put(3, 'Starting the processing threads')
+ try:
+ logmap[log.stubname].append(module)
+ except KeyError:
+ logmap[log.stubname] = [module]
+
+ logger.debug('logmap={}'.format(logmap))
+
+ # Vive le quebec libre!
+ pq = ProcessingQueue(QUEUE_LIMIT)
+
+ logger.debug('Starting the processing threads')
threads = []
+
try:
- while 1:
- t = ConsumerThread(pq, logger)
+ while True:
+ t = ConsumerThread(pq)
t.start()
threads.append(t)
- if len(threads) > self.threads: break
- for entry in logmap.keys():
- log = self.logtracker.getlog(entry)
- if log.is_range_empty(): continue
+ if len(threads) > self.threads:
+ break
+
+ for stubname in logmap.keys():
+ log = self.logtracker.getlog(stubname)
+
+ if log.is_range_empty():
+ continue
+
matched = 0
- lines = 0
- while 1:
- logger.put(3, 'Getting next line from "%s"' % entry)
+ lines = 0
+
+ while True:
+ logger.info('Getting next line from {}'.format(stubname))
+
try:
linemap = log.nextline()
except FormatError, e:
- logger.put(5, 'Writing the line to unparsed')
+ logger.debug('Writing the line to unparsed')
upfh.write(str(e))
continue
- except OutOfRangeError: break
+ except OutOfRangeError:
+ break
+
lines += 1
- logger.put(5, 'We have the following:')
- logger.put(5, 'line=%s' % linemap['line'])
- logger.put(5, 'stamp=%d' % linemap['stamp'])
- logger.put(5, 'system=%s' % linemap['system'])
- logger.put(5, 'message=%s' % linemap['message'])
- logger.put(5, 'multiplier=%d' % linemap['multiplier'])
+
+ logger.debug('We have the following:')
+ logger.debug('line={}'.format(linemap['line']))
+ logger.debug('stamp={}'.format(linemap['stamp']))
+ logger.debug('system={}'.format(linemap['system']))
+ logger.debug('message={}'.format(linemap['message']))
+ logger.debug('multiplier={}'.format(linemap['multiplier']))
+
match = 0
- for module in logmap[entry]:
- logger.put(5, 'Matching module "%s"' % module.name)
+
+ for module in logmap[stubname]:
+ logger.debug('Matching module: {}'.format(module.name))
+
message = linemap['message']
- handler, regex = module.message_match(message)
+ (handler, regex) = module.message_match(message)
+
linemap['regex'] = regex
+
if handler is not None:
match = 1
pq.put_linemap(linemap, handler, module)
if not self.multimatch:
- logger.put(5, 'multimatch is not set')
- logger.put(5, 'Not matching other modules')
break
+
matched += match
+
if not match:
- logger.put(5, 'Writing the line to unparsed')
+ logger.debug('Writing the line to unparsed')
upfh.write(linemap['line'])
- bartitle = log.entry
- message = '%d of %d lines parsed' % (matched, lines)
- logger.endbar(1, bartitle, message)
+
+ bartitle = log.loglist[0].filename
+ message = '{} of {} lines parsed'.format(matched, lines)
+ self.ui.endbar(bartitle, message)
+
finally:
- logger.put(3, 'Notifying the threads that they may die now')
+ logger.info('Notifying the threads that they may die now')
pq.tell_threads_to_quit(threads)
bartitle = 'Waiting for threads to finish'
bartotal = len(threads)
bardone = 1
for t in threads:
- logger.progressbar(1, bartitle, bardone, bartotal)
+ self.ui.progressbar(bartitle, bardone, bartotal)
t.join()
bardone += 1
- logger.endbar(1, bartitle, 'all threads done')
+ self.ui.endbar(bartitle, 'all threads done')
upfh.close()
- logger.puthang(1, 'Finished all matching, now finalizing')
- for module in self.imodules:
- logger.puthang(1, 'Finalizing "%s"' % module.name)
+
+ self.ui.puthang('Finished all matching, now finalizing')
+ for module in self.modules:
+ self.ui.puthang('Finalizing "{}"'.format(module.name))
try:
rs = pq.get_resultset(module)
try:
module.finalize_processing(rs)
except Exception, e:
- msg = ('Module %s crashed in finalize stage: %s' %
- (module.name, e))
- logger.put(0, msg)
+ msg = ('Module {} crashed in finalize stage: {}'.format(
+ module.name, e))
+ logger.error(msg)
module.no_report()
except KeyError:
module.no_report()
- logger.endhang(1)
- logger.endhang(1)
- logger.endhang(1)
- logger.put(5, '<Epylog._process_internal_modules')
+ self.ui.endhang()
+ self.ui.endhang()
class ProcessingQueue:
"""
This is a standard cookie-cutter helper class for using threads in a
Python application.
"""
- def __init__(self, limit, logger):
- self.logger = logger
- logger.put(5, '>ProcessingQueue.__init__')
- logger.put(3, 'Initializing ProcessingQueue')
+ def __init__(self, limit):
+ logger.info('Initializing ProcessingQueue')
+
self.mon = threading.RLock()
- self.iw = threading.Condition(self.mon)
- self.ow = threading.Condition(self.mon)
- self.lineq = []
+ self.iw = threading.Condition(self.mon)
+ self.ow = threading.Condition(self.mon)
+
+ self.lineq = []
self.resultsets = {}
- self.limit = limit
- self.working = 1
- logger.put(5, '<ProcessingQueue.__init__')
+ self.limit = limit
+ self.working = True
def put_linemap(self, linemap, handler, module):
"""
Accepts a linemap and stores it to be picked up by a thread.
"""
self.mon.acquire()
- logger = self.logger
- logger.put(5, '>ProcessingQueue.put_linemap')
while len(self.lineq) >= self.limit:
- logger.put(5, 'Line queue is full, waiting...')
+ logger.debug('Line queue is full, waiting...')
self.ow.wait()
- self.lineq.append([linemap, handler, module])
- logger.put(3, 'Added a new line in lineq')
- logger.put(5, 'items in lineq: %d' % len(self.lineq))
+ self.lineq.append((linemap, handler, module))
+ logger.debug('Added a new line in lineq')
+ logger.debug('items in lineq: {}'.format(len(self.lineq)))
self.iw.notify()
- logger.put(5, '<ProcessingQueue.put_linemap')
self.mon.release()
def get_linemap(self):
@@ -526,18 +515,18 @@ class ProcessingQueue:
and processes it.
"""
self.mon.acquire()
- logger = self.logger
- logger.put(5, '>ProcessingQueue.get_linemap')
while not self.lineq and self.working:
- logger.put(5, 'Line queue is empty, waiting...')
+ logger.debug('Line queue is empty, waiting...')
self.iw.wait()
if self.working:
item = self.lineq.pop(0)
- logger.put(3, 'Got new linemap for the thread.')
- logger.put(5, 'items in lineq: %d' % len(self.lineq))
+ logger.debug('Got new linemap for the thread.')
+ logger.debug('items in lineq: {}'.format(len(self.lineq)))
self.ow.notify()
- else: item = None
- logger.put(5, '<ProcessingQueue.get_linemap')
+
+ else:
+ item = None
+
self.mon.release()
return item
@@ -547,18 +536,18 @@ class ProcessingQueue:
result and places it here.
"""
self.mon.acquire()
- logger = self.logger
- logger.put(5, '>ProcessingQueue.put_result')
+
if result is not None:
- try: self.resultsets[module].add_result(result)
+ try:
+ self.resultsets[module].add_result(result)
except KeyError:
self.resultsets[module] = Result()
self.resultsets[module].add_result(result)
module.put_filtered(line)
- logger.put(3, 'Added result from module "%s"' % module.name)
+ logger.debug('Added result from module {}'.format(module.name))
else:
- logger.put(3, '"%s" returned result None. Skipping.' % module.name)
- logger.put(5, '<ProcessingQueue.put_result')
+ logger.debug('{} returned None. Skipping.'.format(module.name))
+
self.mon.release()
def get_resultset(self, module):
@@ -566,9 +555,7 @@ class ProcessingQueue:
When all threads are done, the resultset is returned to anyone
interested.
"""
- self.logger.put(5, '>ProcessingQueue.get_resultset')
rs = self.resultsets[module]
- self.logger.put(5, '<ProcessingQueue.get_resultset')
return rs
def tell_threads_to_quit(self, threads):
@@ -576,18 +563,18 @@ class ProcessingQueue:
Tell all threads that they should exit as soon as possible.
"""
self.mon.acquire()
- logger = self.logger
- logger.put(5, '>ProcessingQueue.tell_threads_to_quit')
- logger.put(1, 'Telling all threads to quit')
- logger.put(5, 'Waiting till queue is empty')
+ logger.info('Telling all threads to quit')
+
while self.lineq:
- logger.put(5, 'items in lineq: %d' % len(self.lineq))
+ logger.debug('items in lineq: {}'.format(len(self.lineq)))
self.ow.wait()
- self.logger.put(5, 'working=0')
- self.working = 0
- logger.put(3, 'Sending %d semaphore notifications' % len(threads))
- for t in threads: self.iw.notify()
- logger.put(5, '<ProcessingQueue.tell_threads_to_quit')
+
+ self.working = False
+
+ logger.info('Sending {} semaphore notifications'.format(len(threads)))
+ for t in threads:
+ self.iw.notify()
+
self.mon.release()
class ConsumerThread(threading.Thread):
@@ -595,30 +582,30 @@ class ConsumerThread(threading.Thread):
This class extends Thread, and is used to thread up the internal
module invocation.
"""
- def __init__(self, queue, logger):
+ def __init__(self, queue):
threading.Thread.__init__(self)
- logger.put(5, '>ConsumerThread.__init__')
- self.logger = logger
self.queue = queue
- logger.put(5, '<ConsumerThread.__init__')
def run(self):
- logger = self.logger
- logger.put(5, '>ConsumerThread.run')
while self.queue.working:
- logger.put(3, '%s: getting a new linemap' % self.getName())
+ logger.debug('{}: getting a new linemap'.format(self.getName()))
item = self.queue.get_linemap()
+
if item is not None:
linemap, handler, module = item
- logger.put(3, '%s: calling the handler' % self.getName())
+ logger.debug('{}: calling the handler'.format(self.getName()))
+
try:
result = handler(linemap)
if result is not None:
line = linemap['line']
- logger.put(5, '%s: returning result' % self.getName())
+ logger.debug('{}: returning result'.format(
+ self.getName()))
self.queue.put_result(line, result, module)
else:
- logger.put(5, '%s: Result is None.' % self.getName())
+ logger.debug('{}: Result is None.'.format(
+ self.getName()))
+
except Exception, e:
erep = 'Handler crash. Dump follows:\n'
erep += ' Thread : %s\n' % self.getName()
@@ -627,11 +614,11 @@ class ConsumerThread(threading.Thread):
erep += ' Error : %s\n' % e
erep += ' Line : %s\n' % linemap['line'].strip()
erep += 'End Dump'
- logger.put(0, erep)
+ logger.error(erep)
else:
- logger.put(5, '%s: Item is none.' % self.getName())
- logger.put(3, '%s: I am now dying' % self.getName())
- logger.put(5, '<ConsumerThread.run')
+ logger.debug('{}: Item is none.'.format(self.getName()))
+
+ logger.debug('{}: I am now dying'.format(self.getName()))
class Result(dict):
"""
@@ -793,88 +780,103 @@ class InternalModule:
return (ksize, 'KB')
return (size, 'Bytes')
-class Logger:
+class ConsoleUi:
"""
- A default command-line logger class. Other GUIs should use their own,
+ A default command-line UI class. Other GUIs should use their own,
but fully implement the API.
"""
- indent = ' '
+ indent = ' '
hangmsg = []
- hanging = 0
+ hanging = False
+ quiet = False
- def __init__(self, loglevel):
- self.loglevel = loglevel
+ def __init__(self, quiet=False):
+ self.quiet = quiet
def is_quiet(self):
"""Check if we should be quiet"""
- if self.loglevel == 0:
- return 1
- else:
- return 0
+ return self.quiet
- def debuglevel(self):
- """Return the current debug level"""
- return str(self.loglevel)
+ def put(self, message):
+ """Output a message"""
+ if self.quiet:
+ return
- def put(self, level, message):
- """Log a message, but only if debug levels are lesser or match"""
- if (level <= self.loglevel):
- if self.hanging:
- self.hanging = 0
- print '%s%s' % (self._getindent(), message)
+ if self.hanging:
+ self.hanging = False
+ print '{}{}'.format(self._getindent(), message)
- def puthang(self, level, message):
+ def puthang(self, message):
"""
- This indents the output, create an easier-to-read debug data.
+ This indents the output, creating an easier-to read flow.
"""
- if (level <= self.loglevel):
- print '%sInvoking: "%s"...' % (self._getindent(), message)
- self.hanging = 1
- self.hangmsg.append(message)
+ if self.quiet:
+ return
+
+ print '{}Invoking: "{}"...'.format(self._getindent(), message)
+ self.hanging = True
+ self.hangmsg.append(message)
- def endhang(self, level, message='done'):
+ def endhang(self, message='done'):
"""Must be called after puthang has been put in effect"""
- if (level <= self.loglevel):
- hangmsg = self.hangmsg.pop()
- if self.hanging:
- self.hanging = 0
- print '%s%s...%s' % (self._getindent(), hangmsg, message)
- else:
- print '%s(Hanging from "%s")....%s' % (self._getindent(),
+ if self.quiet:
+ return
+
+ hangmsg = self.hangmsg.pop()
+ if self.hanging:
+ self.hanging = False
+ print '{}{}...{}'.format(self._getindent(), hangmsg, message)
+ else:
+ print '{}(Hanging from "{}")....{}'.format(self._getindent(),
hangmsg, message)
- def progressbar(self, level, title, done, total):
+ def progressbar(self, title, done, total):
"""
A simple command-line progress bar.
"""
- if level != self.loglevel: return
+ if self.quiet:
+ return
+
##
# Do some nifty calculations to present the bar
#
- if len(title) > 40: title = title[:40]
+ if len(title) > 40:
+ title = title[:40]
+
barwidth = 60 - len(title) - 2 - len(self._getindent())
- barmask = "[%-" + str(barwidth) + "s]"
- if total != 0: bardown = int(barwidth*(float(done)/float(total)))
- else: bardown = 0
+ barmask = "[%-" + str(barwidth) + "s]"
+
+ if total != 0:
+ bardown = int(barwidth*(float(done)/float(total)))
+ else:
+ bardown = 0
+
+ # XXX: Format
bar = barmask % ("=" * bardown)
- sys.stdout.write("\r%s%s: %s\r" % (self._getindent(), title, bar))
+ sys.stdout.write("\r{}{}: {}\r".format(self._getindent(), title, bar))
- def endbar(self, level, title, message):
+ def endbar(self, title, message):
"""
After the progress bar is no longer useful, let's replace it with
something useful.
"""
- if level != self.loglevel: return
+ if self.quiet:
+ return
+
if not message:
print
return
##
# Do some nifty calculations to present the bar
#
- if len(title) > 40: title = title[:40]
+ if len(title) > 40:
+ title = title[:40]
+
barwidth = 60 - len(title) - len(self._getindent()) - 2
- message = '[%s]' % message.center(barwidth)
- sys.stdout.write("\r%s%s: %s\n" % (self._getindent(), title, message))
+ message = '[%s]' % message.center(barwidth)
+
+ sys.stdout.write("\r{}{}: {}\n".format(self._getindent(), title,
+ message))
def _getindent(self):
"""
diff --git a/py/epylog/helpers.py b/epylog/helpers.py
similarity index 65%
rename from py/epylog/helpers.py
rename to epylog/helpers.py
index 32550e6..f279e1a 100644
--- a/py/epylog/helpers.py
+++ b/epylog/helpers.py
@@ -4,7 +4,8 @@ It provides several useful methods for running the modules standalone
without having to invoke them as part of Epylog.
"""
##
-# Copyright (C) 2003 by Duke University
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
@@ -21,17 +22,27 @@ without having to invoke them as part of Epylog.
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
# 02111-1307, USA.
#
-# $Id$
-#
-# @Author Konstantin Ryabitsev <icon(a)linux.duke.edu>
-# @version $Date$
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
#
import sys
-sys.path.insert(0, './py/')
import epylog
import getopt
+import logging
+
+logger = logging.getLogger('epylog')
+logger.setLevel(logging.DEBUG)
+
+ch = logging.StreamHandler()
+ch.setLevel(logging.DEBUG)
+
+formatter = logging.Formatter("[%(levelname)s:%(module)s:%(funcName)s:"
+ "%(lineno)s] %(message)s")
+ch.setFormatter(formatter)
+logger.addHandler(ch)
+
+monthmap = epylog.log.mkmonthmap()
class ModuleTest:
"""
@@ -40,7 +51,6 @@ class ModuleTest:
effective.
"""
def __init__(self, epyclass, args):
- logger = epylog.Logger(5)
cmdargs = args[1:]
if not cmdargs: self._usage(args[0])
infile = None
@@ -63,65 +73,81 @@ class ModuleTest:
opts[key] = value
except getopt.error, e: self._usage(args[0])
if opts:
- logger.put(5, 'Additional opts follow')
- logger.put(5, opts)
- logger.put(5, 'Instantiating the module')
- epymod = epyclass(opts, logger)
- if input is None: self._usage(args[0])
- logger.put(5, 'Trying to open file %s for reading' % infile)
- try: infh = open(infile)
+ logger.debug('Additional opts follow')
+ logger.debug(opts)
+ logger.debug('Instantiating the module')
+ epymod = epyclass(opts)
+
+ if input is None:
+ self._usage(args[0])
+
+ logger.debug('Trying to open file {} for reading'.format(infile))
+
+ try:
+ infh = open(infile)
except Exception, e:
- msg = "ERROR trying to open file %s: %s" % (infile, e)
+ msg = "ERROR trying to open file {}: {}".format(infile, e)
self._die(msg)
+
if filtfile is not None:
- logger.put(5, 'Trying to open %s for writing' % filtfile)
- try: filtfh = open(filtfile, 'w')
+ logger.debug('Trying to open {} for writing'.format(filtfile))
+ try:
+ filtfh = open(filtfile, 'w')
except Exception, e:
- msg = "ERROR trying to open file %s: %s" % (filtfile, e)
+ msg = "ERROR trying to open file {}: {}".format(filtfile, e)
self._die(msg)
- monthmap = epylog.log.mkmonthmap()
+
rs = epylog.Result()
- while 1:
+
+ while True:
line = infh.readline()
- if not line: break
- line = line.strip()
- linemap = self._mk_linemap(line, monthmap)
- msg = linemap['message']
+
+ if not line:
+ break
+
+ line = line.strip()
+ linemap = self._mk_linemap(line)
+ msg = linemap['message']
+
for regex in epymod.regex_map.keys():
if regex.search(msg):
handler = epymod.regex_map[regex]
linemap['regex'] = regex
- logger.put(5, '%s -> %s' % (handler.__name__, msg))
+ logger.debug('{} -> {}'.format(handler.__name__, msg))
result = handler(linemap)
+
if result is not None:
rs.add_result(result)
if filtfile is not None:
- filtfh.write('%s\n' % line)
+ filtfh.write('{}\n'.format(line))
break
infh.close()
- if filtfile is not None: filtfh.close()
+
+ if filtfile is not None:
+ filtfh.close()
+
if not rs.is_empty():
- logger.put(5, 'Finalizing')
+ logger.debug('Finalizing')
report = epymod.finalize(rs)
+
if repfile is not None:
- logger.put(5, 'Trying to write report to %s' % repfile)
+ logger.debug('Trying to write report to {}'.format(repfile))
repfh = open(repfile, 'w')
repfh.write(report)
repfh.close()
- logger.put(5, 'Report written to %s' % repfile)
+ logger.debug('Report written to {}'.format(repfile))
else:
- logger.put(5, 'Report follows:')
- print report
+ logger.debug('----Report follows----')
+ logger.debug(report)
else:
- logger.put(5, 'No results for this run')
- logger.put(5, 'Done')
+ logger.debug('No results for this run')
- def _mk_linemap(self, line, monthmap):
+ def _mk_linemap(self, line):
"""
Create a linemap out of a line entry.
"""
try:
- stamp, sys, msg = epylog.log.get_stamp_sys_msg(line, monthmap)
+ (stamp, sys, msg) = epylog.log.get_stamp_sys_msg(line)
except ValueError:
# If we got an empty log line just go on parsing with nothing
# instead of dying
@@ -133,8 +159,9 @@ class ModuleTest:
'multiplier': 1}
return linemap
else:
- msg = 'Invalid syslog line: %s' % line
+ msg = 'Invalid syslog line: {}'.format(line)
self._die(msg)
+
linemap = {'line': line,
'stamp': stamp,
'system': sys,
@@ -146,19 +173,19 @@ class ModuleTest:
"""
Hot Grits Death!
"""
- print 'FATAL ERROR: %s' % message
+ logger.critical('FATAL ERROR: {}'.format(message))
sys.exit(1)
def _usage(self, name):
- print '''Usage:
- %s -i testcase [-r report] [-f filter] [-o EXTRAOPTS]
+ print('''Usage:
+ {} -i testcase [-r report] [-f filter] [-o EXTRAOPTS]
If -r is omitted, the report is printed to stdout
If -f is omitted, filtered lines are not shown
EXTRAOPTS:
Extra options should be submitted in this matter:
-o "option=value; option2=value; option3=value"
- ''' % name
+ '''.format(name))
sys.exit(1)
if __name__ == '__main__':
diff --git a/epylog/log.py b/epylog/log.py
new file mode 100644
index 0000000..f278991
--- /dev/null
+++ b/epylog/log.py
@@ -0,0 +1,1393 @@
+"""
+This operates on logfiles, including looking up strings, repeated data,
+handling rotated logs, figuring out the dates, etc.
+"""
+##
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+#
+# @Author Konstantin Riabitsev <icon(a)mricon.com>
+#
+
+import epylog
+import os
+import re
+import string
+import time
+import tempfile
+import ConfigParser
+import glob
+
+from string import Template
+
+import logging
+
+logger = logging.getLogger('epylog')
+
+def mkmonthmap():
+ """
+ The problem with syslog is that it does not log the year when the
+ event has taken place. This makes certain things difficult, including
+ looking up entries based on a timestamp. This function creates a mapping
+ for months to a year. Pad sets how many months ahead of the current one
+ should be considered in this year, and how many in the last year.
+ This function was largely contributed by Michael Stenner.
+ """
+ pad = 2
+ months = []
+
+ for i in range(0, 12):
+ months.append(time.strftime("%b", (1, i+1, 1, 1,
+ 1, 1, 1, 1, 1)))
+ basetime = time.localtime(time.time())
+ now_year = basetime[0]
+ now_month = basetime[1]
+ pad_month = now_month + pad
+ monthmap = {}
+
+ for m in range(pad_month - 12, pad_month):
+ monthname = months[m % 12]
+ year = now_year + (m / 12)
+
+ monthmap[monthname] = year
+
+ return monthmap
+
+monthmap = mkmonthmap()
+
+def mkstamp_from_syslog_datestr(datestr):
+ """
+ Takes a syslog date string and makes a timestamp out of it.
+ """
+ try:
+ (m, d, t) = datestr.split()[:3]
+ y = str(monthmap[m])
+
+ datestr = string.join([y, m, d, t], ' ')
+ tuptime = time.strptime(datestr, '%Y %b %d %H:%M:%S')
+
+ ##
+ # Python 2.2.2 (at least) breaks with DST.
+ # Work around.
+ #
+ localtime = time.localtime(time.mktime(tuptime))
+ ltime = list(tuptime)
+ ltime[8] = localtime[8]
+ tuptime = tuple(ltime)
+ timestamp = int(time.mktime(tuptime))
+
+ except:
+ # No idea, we give up
+ timestamp = -1
+
+ return timestamp
+
+def get_stamp_sys_msg(line):
+ """
+ This function takes a syslog line and returns the timestamp of the event,
+ the system where it occured, and the message.
+ """
+ mo = epylog.LOG_SPLIT_RE.match(line)
+ if not mo:
+ raise ValueError('Unknown line format: {}'.format(line))
+
+ (time, sys, msg) = mo.groups()
+
+ stamp = mkstamp_from_syslog_datestr(time)
+ sys = re.sub(epylog.SYSLOG_NG_STRIP, '', sys)
+
+ return stamp, sys, msg
+
+class LogTracker:
+ """
+ This is a helper class to track the logfiles as requested by the modules,
+ so no logs are opened more often than needed. It also does tracking of
+ rotating logfiles, opening and initializing them as necessary.
+ """
+ def __init__(self, config, ui):
+ """
+ Initializer code. Passing in config so we can use the variables
+ set by the admin.
+ """
+ self.ui = ui
+
+ sourcescfg = os.path.join(config.paths['cfgdir'], 'logsources.conf')
+
+ if not os.access(sourcescfg, os.R_OK):
+ msg = 'Log definition file "{}" not found'.format(sourcescfg)
+ raise epylog.ConfigError(msg)
+
+ self.logs = {}
+
+ logger.info('Reading log sources config from "{}"'.format(sourcescfg))
+
+ logcfg = ConfigParser.ConfigParser()
+ logcfg.read(sourcescfg)
+
+ for stubname in logcfg.sections():
+ logger.info('Found log definition: {}'.format(stubname))
+ source = logcfg.get(stubname, 'source')
+ rotated = logcfg.get(stubname, 'rotated')
+ # TODO: tsformat
+
+ log = Log(config, stubname, source, rotated, ui)
+
+ self.logs[stubname] = log
+
+ def getlog(self, stubname):
+ """
+ Return a log object based on the stub name provided by the module
+ config file.
+ """
+ if stubname in self.logs.keys():
+ logger.debug('Returning {}'.format(stubname))
+ return self.logs[stubname]
+
+ msg = 'No logs found matching {}'.format(stubname)
+ raise epylog.NoSuchLogError(msg)
+
+ def get_offset_map(self):
+ """
+ Offset map is the virtual boundary that defines which log entries
+ out of the entire scope of a logfile/logfiles we are interested in.
+ It can span multiple rotated logfiles.
+ """
+ # TODO: BROKEN
+ logger = self.logger
+ logger.put(5, '>LogTracker.get_offset_map')
+ omap = []
+ for log in self.logs:
+ entry = log.entry
+ inode = log.getinode()
+ if log.orange.endix != 0:
+ offset = 0
+ else:
+ offset = log.orange.end_offset
+ omap.append([entry, inode, offset])
+ logger.put(5, 'omap follows')
+ logger.put(5, omap)
+ logger.put(5, '<LogTracker.get_offset_map')
+ return omap
+
+ def dump_all_strings(self, fh):
+ """
+ Dumps all strings in the internal omap into a specified fh.
+ """
+ dumped = 0
+ for log in self.logs.values():
+ logger.info('Dumping strings for log {}'.format(log.stubname))
+ dumped = dumped + log.dump_strings(fh)
+ logger.info('Total of {} bytes dumped into {}'.format(dumped, fh.name))
+ return dumped
+
+ def get_stamps(self):
+ """
+ Returns a tuple with the earliest and the latest time stamp
+ from all the logs.
+ """
+ start_stamps = []
+ end_stamps = []
+
+ for log in self.logs.values():
+ if log.is_range_empty():
+ logger.info('The range for this log is empty')
+ continue
+
+ (start_stamp, end_stamp) = log.get_stamps()
+
+ if start_stamp != 0:
+ start_stamps.append(start_stamp)
+ if end_stamp != 0:
+ end_stamps.append(end_stamp)
+
+ if len(start_stamps):
+ start_stamps.sort()
+ start_stamp = start_stamps.pop(0)
+ else:
+ start_stamp = 0
+
+ if len(end_stamps):
+ end_stamps.sort()
+ end_stamp = end_stamps.pop(-1)
+ else:
+ end_stamp = 0
+
+ logger.debug('start_stamp={}'.format(start_stamp))
+ logger.debug('end_stamp={}'.format(end_stamp))
+
+ return (start_stamp, end_stamp)
+
+ def set_range_by_timestamps(self, start_stamp, end_stamp):
+ """
+ Sets offsets in the omap based on the timestamps passed in as
+ arguments.
+ """
+ for stubname in self.logs.keys():
+ logger.debug('Setting ranges for "{}"'.format(stubname))
+ log = self.logs[stubname]
+
+ try:
+ log.set_range_by_timestamps(start_stamp, end_stamp)
+ except epylog.OutOfRangeError:
+ msg = 'Timestamps not found for log "{}"'.format(stubname)
+ logger.error(msg)
+
+
+class OffsetRange:
+ """
+ This is a helper class that handles offset ranges. Since there can be
+ more than one logfile in the chains of rotated logs, specifying
+ offsets can be tricky. This, effectively, keeps four coordinates
+ internally -- the starting index, pointing at an index in the list of
+ log object, then an offset in that log object, then the end index, and
+ the end offset.
+ """
+ def __init__(self, startix, start_offset, endix, end_offset):
+ self.startix = startix
+ self.endix = endix
+ self.start_offset = start_offset
+ self.end_offset = end_offset
+
+ self.total_size = 0
+
+ logger.debug('startix={}'.format(self.startix))
+ logger.debug('start_offset={}'.format(self.start_offset))
+ logger.debug('endix={}'.format(self.endix))
+ logger.debug('end_offset={}'.format(self.end_offset))
+
+ def setstart(self, ix, offset, loglist):
+ """
+ Set the start offset -- takes two coordinates
+ """
+ self.startix = ix
+ self.start_offset = offset
+
+ logger.debug('new startix={}'.format(self.startix))
+ logger.debug('new start_offset={}'.format(self.start_offset))
+
+ self._recalc_total_size(loglist)
+
+ def setend(self, ix, offset, loglist):
+ """
+ Set the end offset -- takes two coordinates.
+ """
+ self.endix = ix
+ self.end_offset = offset
+
+ logger.debug('new endix={}'.format(self.endix))
+ logger.debug('new end_offset={}'.format(self.end_offset))
+
+ self._recalc_total_size(loglist)
+
+ def start_is_end(self):
+ """
+ Check whether the coordinates for start and end are the same
+ and return true if so, otherwise return false.
+ """
+ empty = False
+ if self.startix == self.endix:
+ if self.start_offset == self.end_offset:
+ empty = True
+ logger.debug('This range points to same location')
+
+ return empty
+
+ def done_size(self, curix, offset, loglist):
+ """
+ This is a helper function to help calculate the percentage of
+ offset that has been processed already.
+ """
+ if curix == self.startix:
+ done = offset - self.start_offset
+ else:
+ done = 0
+ for ix in range(self.startix, curix, -1):
+ if ix == self.startix:
+ done = loglist[ix].end_offset - self.start_offset
+ else: done += loglist[ix].end_offset
+ done += offset
+ return done
+
+ def is_inside(self, ix, offset):
+ """
+ Check if a proposed index is inside this offset.
+ """
+ if ix > self.startix:
+ logger.debug('ix > self.startix')
+ return False
+
+ if ix < self.endix:
+ logger.debug('ix > self.endix')
+ return False
+
+ if ix == self.startix and offset < self.start_offset:
+ logger.debug('ix = self.startix and offset < self.start_offset')
+ return False
+
+ if ix == self.endix and offset > self.end_offset:
+ logger.debug('ix = self.endix and offset > self.end_offset')
+ return False
+
+ logger.debug('ix={}, offset={} is inside'.format(ix, offset))
+ return True
+
+ def _recalc_total_size(self, loglist):
+ """
+ If the offsets change, recalculate total size of the range.
+ Useful for figuring out how much is left to do and how much is
+ done already.
+ """
+ logger.debug('startix={}, endix={}'.format(self.startix, self.endix))
+
+ total = 0
+ for ix in range(self.startix, self.endix - 1, -1):
+ if ix == self.startix:
+ total = loglist[ix].end_offset - self.start_offset
+ elif ix == self.endix:
+ total += self.end_offset
+ else:
+ total += loglist[ix].end_offset
+
+ logger.debug('total={}'.format(total))
+
+ self.total_size = total
+
+class LinePointer:
+ """
+ LinePointer is a two-dimensional coordinate that contains the index
+ and an offset of a certain line. This is like a half of an offset range.
+ """
+ def __init__(self, ix, offset):
+ self.ix = ix
+ self.offset = offset
+
+ logger.debug('ix={}'.format(self.ix))
+ logger.debug('offset={}'.format(self.offset))
+
+ def pos(self, ix, offset):
+ self.ix = ix
+ self.offset = offset
+
+ logger.debug('ix={}'.format(self.ix))
+ logger.debug('offset={}'.format(self.offset))
+
+class Log:
+ """
+ This class is the collection of LogFile objects all belonging to the same
+ entry.. It handles things like reading from files, looking up lines, etc.
+ """
+ def __init__(self, config, stubname, source, rotated, ui):
+ self.tmpprefix = config.tmpprefix
+ self.stubname = stubname
+ self.ui = ui
+
+ # initialize the source and rotated files
+ logger.info('Initializing main log file {}'.format(source))
+
+ logfile = LogFile(source, config)
+ self.loglist = [logfile]
+
+ # We'll use the start_stamp in each rotfile to figure out
+ # the order in which they should be listed
+ timestamps = []
+ stampmap = {}
+
+ for rotfile in glob.glob(rotated):
+ logger.info('Initializing rotated logfile {}'.format(rotfile))
+ logfile = LogFile(rotfile, config)
+
+ timestamps.append(logfile.start_stamp)
+ stampmap[logfile.start_stamp] = logfile
+
+ timestamps.sort(reverse=True)
+ for timestamp in timestamps:
+ logfile = stampmap[timestamp]
+ logger.debug('Adding to loglist: {}'.format(logfile.filename))
+ self.loglist.append(logfile)
+
+ # current logfile is the oldest rotated file
+ startix = len(self.loglist) - 1
+
+ self.orange = OffsetRange(startix, 0, 0, logfile.end_offset)
+
+ # used to track linepointer
+ self.lp = None
+
+ def set_range_param(self, ix, offset, whence=False):
+ """
+ Sets an offset parameter. If whence is False, then the start offset
+ is assumed. If it's True, then the end offset is assumed.
+ """
+ logger.debug('ix={}'.format(ix))
+ logger.debug('offset={}'.format(offset))
+ logger.debug('whence={}'.format(whence))
+
+ logger.info('Checking if the offset makes sense')
+
+ if self.loglist[ix].end_offset < offset:
+ msg = 'Offset {} is past the end of {}: {}! Correcting.'.format(
+ offset, self.loglist[ix].filename, self.loglist[ix].end_offset)
+ logger.error(msg)
+
+ self.orange.setstart(ix, self.loglist[ix].end_offset, self.loglist)
+
+ else:
+ if whence:
+ self.orange.setend(ix, offset, self.loglist)
+ else:
+ self.orange.setstart(ix, offset, self.loglist)
+
+ def get_logfile_by_start_stamp(self, start_stamp):
+ ix = 0
+
+ # this loop will either return or exit via a NoSuchLogError
+ while ix < len(self.loglist):
+ logfile = self.loglist[ix]
+
+ logger.debug('Looking at: {}'.format(logfile.filename))
+ if logfile.start_stamp == start_stamp:
+ logger.debug('Found the match at ix={}'.format(ix))
+ return (logfile, ix)
+
+ ix += 1
+
+ raise epylog.NoSuchLogError('No matching entries')
+
+ def nextline(self):
+ """
+ Fetch the next line in the log, based on the internally stored line
+ pointer. If the line pointer is not set, then the start offset is
+ used.
+ """
+ if self.lp is None:
+ ix = self.orange.startix
+ offset = self.orange.start_offset
+
+ logger.debug('init linepointer with ix={}, offset={}'.format(
+ ix, offset))
+ self.lp = LinePointer(ix, offset)
+
+ ix = self.lp.ix
+ offset = self.lp.offset
+
+ logger.debug('Checking if we are past the orange end')
+
+ if not self.orange.is_inside(ix, offset):
+ msg = 'Moved past the end of the range'
+ raise epylog.OutOfRangeError(msg)
+
+ log = self.loglist[ix]
+ (line, offset) = log.get_line_at_offset(offset)
+
+ done = self.orange.done_size(ix, offset, self.loglist)
+ total = self.orange.total_size
+ title = log.filename
+
+ self.ui.progressbar(title, done, total)
+
+ if offset >= log.end_offset:
+ logger.debug('End of log "{}" reached'.format(log.filename))
+
+ ix -= 1
+ offset = 0
+
+ self.lp.pos(ix, offset)
+
+ try:
+ (stamp, system, message) = get_stamp_sys_msg(line)
+ multiplier = 1
+ mo = epylog.MESSAGE_REPEATED_RE.search(message)
+
+ if mo:
+ try:
+ message = self._lookup_repeated(system)
+ multiplier = int(mo.group(1))
+ except epylog.FormatError:
+ pass
+ except epylog.GenericError:
+ pass
+
+ log.repeated_cache[system] = message
+ linemap = {'line': line,
+ 'stamp': stamp,
+ 'system': system,
+ 'message': message,
+ 'multiplier': multiplier}
+
+ except ValueError:
+ logger.error('Invalid syslog format string in {}: {}'.format(
+ log.filename, line))
+ # Pass it on
+ raise epylog.FormatError(line, logger)
+
+ return linemap
+
+ def _lookup_repeated(self, system):
+ """
+ A helper method to resolve the pesky 'last message repeated' lines.
+ It takes a system name and tries to figure out the original line.
+ """
+ log = self.loglist[self.lp.ix]
+
+ try:
+ message = log.repeated_cache[system]
+ logger.debug('Found in repeated_cache by system')
+ return message
+ except KeyError:
+ pass
+
+ host_re = re.compile('.{15,15} .*[@/]*{}'.format(system))
+ offset = self.lp.offset
+
+ logger.debug('Looking in {} for the previous report from {}'.format(
+ log.filename, system))
+ offset_orig = offset
+
+ line = None
+
+ while True:
+ try:
+ (cline, offset) = log.find_previous_entry_by_re(offset, host_re)
+ except (IOError, epylog.OutOfRangeError), e:
+ break
+
+ if epylog.MESSAGE_REPEATED_RE.search(cline):
+ try:
+ rep_offset = log.repeated_cache[offset]
+ logger.debug('Found in cached values')
+ (line, offset) = log.get_line_at_offset(rep_offset)
+ logger.debug('line={}'.format(line))
+ log.repeated_cache[offset_orig] = rep_offset
+ break
+
+ except KeyError:
+ logger.debug('Not in cached values')
+ pass
+
+ else:
+ logger.debug('Found by backstepping')
+ line = cline
+ logger.debug('line={}'.format(line))
+ log.repeated_cache[offset_orig] = offset
+ break
+
+ if not line:
+ msg = 'Could not find the original message'
+ raise epylog.GenericError(msg)
+
+ try:
+ (stamp, system, message) = get_stamp_sys_msg(line)
+ except epylog.FormatError:
+ logger.error('Invalid syslog format string in {}: {}'.format(
+ log.filename, line))
+
+ return message
+
+ def dump_strings(self, fh):
+ """
+ Dump all strings in the offset into the specified fh.
+ """
+ logger.info('Dumping strings for log {}'.format(self.stubname))
+ ologs = self._get_orange_logs()
+ if len(ologs) == 1:
+ # All strings in the same file. Easy.
+ starto = self.orange.start_offset
+ endo = self.orange.end_offset
+
+ log = ologs[0]
+ log.set_offset_range(starto, endo)
+
+ buflen = log.dump_strings(fh)
+
+ logger.info('{} bytes dumped from {} into {}'.format(
+ buflen, log.filename, fh.name))
+ else:
+ # Strings are in different rotfiles. Hard.
+ buflen = 0
+
+ flog = ologs.pop(0)
+ elog = ologs.pop(-1)
+
+ logger.debug('Processing the earliest logfile')
+
+ starto = self.orange.start_offset
+ endo = flog.end_offset
+
+ flog.set_offset_range(starto, endo)
+
+ buflen = buflen + flog.dump_strings(fh)
+
+ if len(ologs):
+ logger.info('There are logfiles between the first and last')
+ for mlog in ologs:
+ mlog.set_offset_range(0, mlog.end_offset)
+ buflen = buflen + mlog.dump_strings(fh)
+
+ logger.info('Processing the latest logfile')
+
+ starto = 0
+ endo = self.orange.end_offset
+
+ elog.set_offset_range(starto, endo)
+ buflen = buflen + elog.dump_strings(fh)
+
+ logger.info('{} bytes dumped from multiple files into {}'.format(
+ buflen, fh.name))
+
+ return buflen
+
+ def get_stamps(self):
+ """
+ Get the stamps in the offset. Start stamp and end stamp are returned.
+ """
+ ##
+ # Returns a list with the earliest and the latest stamp in the
+ # current log range.
+ #
+ logs = self._get_orange_logs()
+ flog = logs.pop(0)
+
+ flog.range_start = self.orange.start_offset
+ (start_stamp, end_stamp) = flog.get_range_stamps()
+
+ if len(logs):
+ elog = logs.pop(-1)
+ elog.range_end = self.orange.end_offset
+ (junk, end_stamp) = elog.get_range_stamps()
+
+ logger.debug('start_stamp={}'.format(start_stamp))
+ logger.debug('end_stamp={}'.format(end_stamp))
+
+ return (start_stamp, end_stamp)
+
+ def set_range_by_timestamps(self, start_stamp, end_stamp):
+ """
+ Set the range by timestamps provided.
+ """
+ if start_stamp > end_stamp:
+ msg = 'Start stamp must be before end stamp'
+ raise epylog.OutOfRangeError(msg)
+
+ logger.debug('looking for start_stamp={}'.format(start_stamp))
+ logger.debug('looking for end_stamp={}'.format(end_stamp))
+
+ ix = 0
+ start_offset = None
+ end_offset = None
+
+ for ix in range(0, len(self.loglist)-1):
+ logger.debug('ix={}'.format(ix))
+
+ curlog = self.loglist[ix]
+ logger.info('Analyzing log file "{}"'.format(curlog.filename))
+
+ try:
+ pos_start = curlog.stamp_in_log(start_stamp)
+ pos_end = curlog.stamp_in_log(end_stamp)
+ except epylog.OutOfRangeError:
+ logger.info('No useful entries in this log, ignoring')
+ continue
+
+ if pos_start == 0:
+ # start stamp is in current log
+ logger.debug('start_stamp is in "{}"'.format(curlog.filename))
+
+ start_ix = ix
+ start_offset = curlog.find_offset_by_timestamp(start_stamp)
+
+ elif pos_start > 0:
+ # Past this log. This means that we have missed the start
+ # of this stamp. Set by the end_offset of the current log.
+ logger.debug('start_stamp is past {}'.format(curlog.filename))
+ logger.debug('setting to end_offset of this log')
+
+ start_ix = ix
+ start_offset = curlog.end_offset
+
+ if pos_end == 0:
+ # end stamp is in current log
+ logger.debug('end_stamp is in {}'.format(curlog.filename))
+
+ end_ix = ix
+ end_offset = curlog.find_offset_by_timestamp(end_stamp)
+
+ elif pos_end > 0 and end_offset is None:
+ # Means that end of the search is past the end of the last
+ # log.
+ logger.debug('end_stamp is past the most current entry')
+ logger.debug('setting to end_offset of this ix')
+
+ end_ix = ix
+ end_offset = curlog.end_offset
+
+ if start_offset is not None and end_offset is not None:
+ logger.debug('Found both the start and the end')
+ break
+
+ if start_offset is None:
+ if end_offset is not None:
+ logger.debug('setting start_offset to 0, last ix')
+ start_offset = 0
+ start_ix = len(self.loglist) - 1
+ else:
+ msg = 'Range not found when searching for timestamps'
+ raise epylog.OutOfRangeError(msg)
+
+ logger.debug('start_ix={}'.format(start_ix))
+ logger.debug('start_offset={}'.format(start_offset))
+ logger.debug('end_ix={}'.format(end_ix))
+ logger.debug('end_offset={}'.format(end_offset))
+
+ self.orange.setstart(start_ix, start_offset, self.loglist)
+ self.orange.setend(end_ix, end_offset, self.loglist)
+
+ def is_range_empty(self):
+ """
+ Check if the range is empty and return an appropriate true or false.
+ """
+ empty = False
+ if self.orange.start_is_end():
+ logger.debug('Yes, range is empty')
+ return True
+
+ startlog = self.loglist[self.orange.startix]
+ if (startlog.end_offset == self.orange.end_offset and
+ self.orange.endix == self.orange.startix + 1 and
+ self.orange.end_offset == 0):
+ # This means that start is at the end of the last rotlog
+ # and end is at the start of next rotlog, meaning that the
+ # range is really empty.
+ logger.debug('Yes, range is empty')
+ return True
+
+ return False
+
+ def _get_orange_logs(self):
+ """
+ Get the logs in the offset range. Returns a list.
+ """
+ ologs = []
+ for ix in range(self.orange.startix, self.orange.endix - 1, -1):
+ logger.debug('appending {}'.format(self.loglist[ix].filename))
+ ologs.append(self.loglist[ix])
+
+ return ologs
+
+class LogFile:
+ """
+ This class handles the log files themselves -- things like opening,
+ rewinding, reading, etc.
+ """
+ def __init__(self, filename, config):
+ self.tmpprefix = config.tmpprefix
+ self.filename = filename
+
+ # Use this indicator to minimize seeks to the line start
+ self.at_line_start = True
+
+ ##
+ # start_stamp: the timestamp at the start of the log
+ # end_stamp: the timestamp at the end of the log
+ # end_offset: this is where the end of the log is
+ #
+ self.start_stamp = None
+ self.end_stamp = None
+ self.end_offset = None
+
+ ##
+ # range_start: the start offset of the range
+ # range_end: the end offset of the range
+ #
+ self.range_start = 0
+ self.range_end = None
+
+ ##
+ # repeated_cache: map of offsets to repeated lines for
+ # unwrapping those pesky "last message repeated"
+ # entries
+ # also a map of last lines for systems.
+ #
+ self.repeated_cache = {}
+
+ logger.info('Running sanity checks on the logfile')
+ self._accesscheck()
+ logger.info('Initializing the file')
+ self._initfile()
+
+ def _initfile(self):
+ """
+ Initialize the logfile. This usually consitutes opening it,
+ figuring out if it's gzipped or not, and recording where the log
+ ends. That is important, as logs are usually being appended during
+ epylog runs.
+ """
+ if self.filename[-3:] == '.gz':
+ import gzip
+
+ logger.info('Using GzipFile to open {}'.format(self.filename))
+
+ tempfile.tmpdir = self.tmpprefix
+
+ ungzfile = tempfile.mktemp('.UNGZ')
+ ungzfh = open(tempfile.mktemp('UNGZ'), 'w+')
+
+ logger.debug('ungzfile={}'.format(ungzfile))
+
+ try:
+ gzfh = gzip.open(self.filename)
+ except:
+ msg = 'Could not open "{}" with gzip.'.format(self.filename)
+ raise epylog.ConfigError(msg)
+
+ logger.info('Putting the contents of the gzlog into ungzlog')
+
+ while True:
+ chunk = gzfh.read(epylog.CHUNK_SIZE)
+
+ if chunk:
+ ungzfh.write(chunk)
+ logger.debug('Read {} bytes from gzfh'.format(len(chunk)))
+ else:
+ logger.debug('Reached EOF')
+ break
+
+ gzfh.close()
+ self.fh = ungzfh
+
+ else:
+ logger.info('Does not end in .gz, assuming plain text')
+ self.fh = open(self.filename)
+
+ logger.info('Finding the start_stamp')
+ self.fh.seek(0)
+ self.at_line_start = True
+ self.start_stamp = self._get_stamp()
+ logger.debug('start_stamp={}'.format(self.start_stamp))
+
+ logger.info('Finding the end offset')
+ self.fh.seek(0, 2)
+ self.at_line_start = False
+ self._set_at_line_start()
+
+ self.end_offset = self.fh.tell()
+ self.range_end = self.fh.tell()
+
+ logger.info('Finding the end_stamp')
+ self.end_stamp = self._get_stamp()
+ logger.debug('end_stamp={}'.format(self.end_stamp))
+
+ def set_offset_range(self, start, end):
+ """
+ A two-dimensional coordinate is accepted that points to which
+ entries we are interested in.
+ """
+ logger.debug('start={}'.format(start))
+ logger.debug('end={}'.format(end))
+
+ if start < 0:
+ msg = 'Start of range cannot be less than zero'
+ raise epylog.OutOfRangeError(msg)
+
+ if end > self.end_offset:
+ msg = 'End of range {} is past the end of log'.format(end)
+ raise epylog.OutOfRangeError(msg)
+
+ if start > end:
+ msg = 'Start of range cannot be greater than end'
+ raise epylog.OutOfRangeError(msg)
+
+ self.fh.seek(start)
+ self.at_line_start = False
+
+ self._set_at_line_start()
+ self.range_start = self.fh.tell()
+
+ self.fh.seek(end)
+ self.at_line_start = False
+
+ self._set_at_line_start()
+ self.range_end = self.fh.tell()
+
+ logger.debug('range_start={}'.format(self.range_start))
+ logger.debug('range_end={}'.format(self.range_end))
+
+ def stamp_in_log(self, searchstamp):
+ """
+ Check if the timestamp specified is inside this logfile.
+ Return values:
+ -1 = before this log
+ 0 = in this log
+ 1 = after this log
+ """
+ logger.debug('searchstamp={}'.format(searchstamp))
+ logger.debug('start_stamp={}'.format(self.start_stamp))
+ logger.debug('end_stamp={}'.format(self.end_stamp))
+
+ if self.start_stamp == 0 or self.end_stamp == 0:
+ msg = 'No stampable entries in this log'
+ raise epylog.OutOfRangeError(msg)
+
+ if searchstamp > self.end_stamp:
+ logger.debug('past the end of this log')
+ return 1
+
+ if searchstamp < self.start_stamp:
+ logger.debug('before the start of this log')
+ return -1
+
+ logger.debug('IN this log')
+ return 0
+
+
+ def find_offset_by_timestamp(self, searchstamp):
+ """
+ Find an offset by timestamp specified.
+ """
+ if self.start_stamp == 0 or self.end_stamp == 0:
+ logger.debug('Does not seem like anything useful is in this file')
+ raise epylog.OutOfRangeError('Nothing useful in this log')
+
+ if self.stamp_in_log(searchstamp) != 0:
+ msg = 'This stamp does not appear to be in this log'
+ raise epylog.OutOfRangeError(msg)
+
+ self._crude_locate(searchstamp)
+ self._fine_locate(searchstamp)
+
+ offset = self.fh.tell()
+ logger.debug('offset={}'.format(offset))
+
+ return offset
+
+ def dump_strings(self, fh):
+ """
+ Dump all strings from this logfile into a provided fh. Only the
+ offset entries are used.
+ """
+ if self.range_end is None:
+ msg = 'No range defined for logfile {}'.format(self.filename)
+ raise epylog.OutOfRangeError(msg, logger)
+
+ chunklen = self.range_end - self.range_start
+
+ logger.debug('range_start={}'.format(self.range_start))
+ logger.debug('range_end={}'.format(self.range_end))
+ logger.debug('chunklen={}'.format(chunklen))
+
+ self.fh.seek(self.range_start)
+ self.at_line_start = False
+
+ if chunklen > 0:
+ iternum = int(chunklen/epylog.CHUNK_SIZE)
+ lastchunk = chunklen%epylog.CHUNK_SIZE
+
+ logger.debug('iternum={}'.format(iternum))
+ logger.debug('lastchunk={}'.format(lastchunk))
+
+ if iternum > 0:
+ for i in range(iternum):
+ chunk = self.fh.read(epylog.CHUNK_SIZE)
+ self.at_line_start = False
+
+ fh.write(chunk)
+ logger.debug('wrote {} bytes from {} to {}'.format(
+ len(chunk), self.filename, fh.name))
+ if lastchunk > 0:
+ chunk = self.fh.read(lastchunk)
+ self.at_line_start = False
+
+ fh.write(chunk)
+ logger.debug('wrote {} bytes from {} to {}'.format(
+ len(chunk), self.filename, fh.name))
+ return chunklen
+
+ def get_range_stamps(self):
+ """
+ Get the timestamps at the beginning of the range offset, and at the
+ end.
+ """
+ logger.debug('range_start={}'.format(self.range_start))
+ self.fh.seek(self.range_start)
+ self.at_line_start = False
+
+ start_stamp = self._get_stamp()
+
+ self.fh.seek(self.range_end)
+ self.at_line_start = False
+
+ end_stamp = self._get_stamp()
+
+ logger.debug('start_stamp={}'.format(start_stamp))
+ logger.debug('end_stamp={}'.format(end_stamp))
+
+ return (start_stamp, end_stamp)
+
+ def get_line_at_offset(self, offset):
+ """
+ Get and return the line at a specified offset.
+ """
+ self.fh.seek(offset)
+ self.at_line_start = False
+
+ line = self.fh.readline()
+ offset = self.fh.tell()
+
+ return (line, offset)
+
+ def find_previous_entry_by_re(self, offset, regex, limit=1000):
+ """
+ Back up one line at a time and try to locate the one that
+ matches the provided regex.
+ """
+ self.fh.seek(offset)
+ self.at_line_start = False
+
+ count = 0
+
+ while True:
+ line = self._lineback()
+ if regex.search(line):
+ logger.debug('Found line: {}'.format(line))
+ break
+
+ count += 1
+
+ logger.debug('No match, going back more (count={})'.format(count))
+
+ if count > limit:
+ logger.debug('Reached backstepping limit')
+ msg = 'Out of sane range looking for line'
+ raise epylog.OutOfRangeError(msg)
+
+ return (line, self.fh.tell())
+
+ def _crude_locate(self, stamp):
+ """
+ This is a binary search that would look for a line matching the
+ provided timestamp.
+ """
+ logger.info('Looking for {} in {}'.format(stamp, self.filename))
+
+ increment = int(self.end_offset/2)
+ relative = increment
+
+ logger.debug('rewinding the logfile')
+ self.fh.seek(0)
+ self.at_line_start = True
+
+ logger.debug('initial increment={}'.format(increment))
+ logger.debug('initial relative={}'.format(relative))
+
+ ostamp = None
+
+ while True:
+ old_ostamp = ostamp
+
+ self._rel_position(relative)
+
+ ostamp = self._get_stamp()
+ if ostamp == 0:
+ logger.debug('Bogus timestamp! Breaking.')
+ break
+
+ logger.debug('ostamp={}'.format(ostamp))
+
+ if old_ostamp == ostamp:
+ logger.debug('ostamp and old_ostamp the same. Breaking')
+ break
+
+ increment = int(increment/2)
+ logger.debug('increment={}'.format(increment))
+
+ if ostamp < stamp:
+ logger.debug('<<<<<<<')
+ relative = increment
+ logger.debug('Jumping forward by {}'.format(relative))
+
+ elif ostamp > stamp:
+ logger.debug('>>>>>>>')
+ relative = -increment
+ logger.debug('Jumping backward by {}'.format(relative))
+
+ elif ostamp == stamp:
+ logger.debug('=======')
+ break
+
+ logger.debug('Crude search done at offset {}'.format(self.fh.tell()))
+
+ def _fine_locate(self, stamp):
+ """
+ This search algorithm will locate the best match line by line.
+ It's best used after _crude_locate, otherwise it'll take forever.
+ """
+ lineloc = 0
+ oldlineloc = 0
+
+ before_stamp = None
+ after_stamp = None
+ current_stamp = None
+
+ while True:
+ try:
+ if lineloc > 0:
+ logger.debug('Going forward one line')
+
+ before_stamp = current_stamp
+ current_stamp = after_stamp
+ after_stamp = None
+
+ self._lineover()
+
+ elif lineloc < 0:
+ logger.debug('Going back one line')
+
+ before_stamp = None
+ current_stamp = before_stamp
+ after_stamp = current_stamp
+
+ self._lineback()
+
+ offset = self.fh.tell()
+
+ if offset >= self.end_offset:
+ # We have reached the end of the initialized log.
+ # There are possibly entries past this point, but
+ # we can't trust them, as they are appended after the
+ # init and can screw us up.
+ logger.debug('End of initialized log reached, breaking')
+ self.fh.seek(self.end_offset)
+ self.at_line_start = False
+
+ break
+
+ if current_stamp is None:
+ current_stamp = self._get_stamp()
+ self.fh.seek(offset)
+ self.at_line_start = False
+
+ if before_stamp is None:
+ self._lineback()
+ before_stamp = self._get_stamp()
+ self.fh.seek(offset)
+ self.at_line_start = False
+
+ if after_stamp is None:
+ self._lineover()
+ after_stamp = self._get_stamp()
+ self.fh.seek(offset)
+ self.at_line_start = False
+
+ except IOError:
+ logger.debug('Either end or start of file reached, breaking')
+ break
+
+ logger.debug('before_stamp={}'.format(before_stamp))
+ logger.debug('current_stamp={}'.format(current_stamp))
+ logger.debug('after_stamp={}'.format(after_stamp))
+ logger.debug('searching for {}'.format(stamp))
+
+ if before_stamp == 0 or current_stamp == 0 or after_stamp == 0:
+ logger.debug('Bogus stamps found. Breaking.')
+ break
+
+ oldlineloc = lineloc
+
+ if before_stamp >= stamp:
+ logger.debug('>>>>>')
+ lineloc = -1
+
+ elif before_stamp < stamp and after_stamp <= stamp:
+ logger.debug('<<<<<')
+ lineloc = 1
+
+ elif current_stamp < stamp and after_stamp >= stamp:
+ logger.debug('<<<<<')
+ lineloc = 1
+
+ elif before_stamp < stamp and current_stamp >= stamp:
+ logger.debug('=====')
+ break
+
+ if oldlineloc == -lineloc:
+ # fine_locate cannot reverse direction.
+ # If it does, that means that entries are not in order,
+ # which may happen quite frequently on poorly ntpd'd
+ # machines. Get out and hope this is good enough.
+ logger.warning('Entries in {} going back in time!'.format(
+ self.filename))
+ break
+
+ logger.debug('fine locate done at offset {}'.format(self.fh.tell()))
+
+ def _lineover(self):
+ """
+ Go forward one line and return it.
+ """
+ offset = self.fh.tell()
+ entry = self.fh.readline()
+
+ self.at_line_start = True
+
+ if self.fh.tell() == offset:
+ logger.debug('End of file reached!')
+ raise IOError
+
+ logger.debug('New offset at {}'.format(self.fh.tell()))
+
+ return entry
+
+ def _lineback(self):
+ """
+ Go back one line and return it.
+ """
+ self._set_at_line_start()
+
+ if self.fh.tell() <= 1:
+ logger.debug('Start of file reached')
+ raise IOError
+
+ entry = self._rel_position(-2)
+ logger.debug('New offset at {}'.format(self.fh.tell()))
+
+ return entry
+
+ def _get_stamp(self):
+ """
+ Get the timestamp at current offset.
+ """
+ self._set_at_line_start()
+
+ offset = self.fh.tell()
+ curline = self.fh.readline()
+
+ # rewind back where we were
+ self.fh.seek(offset)
+
+ if len(curline):
+ try:
+ stamp = self._mkstamp_from_syslog_datestr(curline)
+ except epylog.FormatError:
+ stamp = 0
+
+ else:
+ logger.debug('Nothing in the range')
+ stamp = 0
+
+ return stamp
+
+ def _rel_position(self, relative):
+ """
+ Position the offset within a file based on the "relative" variable
+ passed in as argument. It can be positive or negative. Then it will
+ position itself at the start of the line.
+ """
+ offset = self.fh.tell()
+ new_offset = offset + relative
+
+ logger.debug('offset={}'.format(offset))
+ logger.debug('relative={}'.format(relative))
+ logger.debug('new_offset={}'.format(new_offset))
+
+ if new_offset < 0:
+ logger.debug('new_offset less than 0. Setting to 0')
+ new_offset = 0
+
+ self.fh.seek(new_offset)
+ self.at_line_start = False
+
+ entry = self._set_at_line_start()
+
+ logger.debug('after _set_at_line_start: {}'.format(self.fh.tell()))
+
+ return entry
+
+ def _mkstamp_from_syslog_datestr(self, datestr):
+ """
+ Make a timestamp out of the syslog-format date entry.
+ """
+ logger.debug('datestr={}'.format(datestr))
+ timestamp = mkstamp_from_syslog_datestr(datestr)
+
+ if timestamp == -1:
+ msg = 'Odd date format in entry: {}'.format(datestr)
+ raise epylog.FormatError(msg)
+
+ logger.debug('timestamp={}'.format(timestamp))
+
+ return timestamp
+
+ def _accesscheck(self):
+ """
+ Quick sanity checks on the logfile.
+ """
+ logfile = self.filename
+
+ if not os.access(logfile, os.F_OK):
+ msg = 'Log file "{}" does not exist'.format(logfile)
+ raise epylog.AccessError(msg)
+
+ if not os.access(logfile, os.R_OK):
+ msg = 'Logfile "{}" is not readable'.format(logfile)
+ raise epylog.AccessError(msg)
+
+ def _set_at_line_start(self):
+ """
+ Position ourselves at the beginning of the line.
+ """
+ if self.at_line_start:
+ logger.debug('self.at_line_start says already there')
+ return ''
+
+ orig_offset = self.fh.tell()
+
+ if orig_offset == 0:
+ logger.debug('Already at file start')
+ return
+
+ logger.debug('starting the backstepping loop')
+
+ entry = ''
+ while True:
+ curchar = self.fh.read(1)
+ if curchar == '\n':
+ logger.debug('Found newline at offset %d' % self.fh.tell())
+ break
+
+ entry = curchar + entry
+ offset = self.fh.tell() - 1
+ self.fh.seek(offset)
+
+ if offset == 0:
+ logger.debug('Beginning of file reached!')
+ break
+
+ offset = offset - 1
+ self.fh.seek(offset)
+
+ logger.debug('Exited the backstepping loop')
+
+ now_offset = self.fh.tell()
+ rewound = orig_offset - now_offset
+
+ logger.debug('Line start found at offset "{}"'.format(now_offset))
+ logger.debug('rewound by {} characters'.format(rewound))
+
+ self.at_line_start = True
+
+ return entry
+
diff --git a/epylog/module.py b/epylog/module.py
new file mode 100644
index 0000000..c3e1a15
--- /dev/null
+++ b/epylog/module.py
@@ -0,0 +1,296 @@
+"""
+This module handles the... er... modules for epylog, both internal and
+external.
+"""
+##
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+#
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
+#
+
+import ConfigParser
+import epylog
+import os
+import tempfile
+import string
+import re
+
+from string import Template
+
+from ihooks import BasicModuleLoader
+_loader = BasicModuleLoader()
+
+import logging
+
+logger = logging.getLogger('epylog')
+
+class Module:
+ """epylog Module class"""
+
+ def __init__(self, cfgfile, logtracker, config, ui):
+ self.ui = ui
+
+ self.tmpprefix = config.tmpprefix
+ self.paths = config.paths
+
+ logger.info('Initializing module for cfgfile {}'.format(cfgfile))
+
+ modcfg = ConfigParser.ConfigParser()
+ modcfg.read(cfgfile)
+
+ try:
+ self.name = modcfg.get('module', 'desc')
+ except:
+ self.name = 'Unnamed Module'
+ try:
+ self.enabled = modcfg.getboolean('module', 'enabled')
+ except:
+ self.enabled = False
+
+ if not self.enabled:
+ logger.info('Module "{}" is not enabled.'.format(self.name))
+ return
+
+ try:
+ executable = modcfg.get('module', 'exec')
+ self.executable = Template(executable).safe_substitute(self.paths)
+ except:
+ msg = 'Did not find executable name in "{}"'.format(cfgfile)
+ raise epylog.ConfigError(msg)
+
+ try:
+ self.priority = modcfg.getint('module', 'priority')
+ except:
+ self.priority = 10
+
+ try:
+ logentries = modcfg.get('module', 'files')
+ except:
+ msg = 'Log sources not found in "{}"'.format(cfgfile)
+ raise epylog.ConfigError(msg)
+
+ self.extraopts = {}
+
+ if modcfg.has_section('conf'):
+ logger.info('Found extra options')
+
+ for option in modcfg.options('conf'):
+ value = modcfg.get('conf', option)
+ value = Template(value).safe_substitute(self.paths)
+ self.extraopts[option] = value
+
+ logger.debug('extra opt {}={}'.format(option, value))
+
+ modname = os.path.basename(self.executable)
+
+ tempfile.tempdir = self.tmpprefix
+
+ self.logdump = tempfile.mktemp('%s.DUMP' % modname)
+ self.logreport = tempfile.mktemp('%s.REPORT' % modname)
+ self.logfilter = tempfile.mktemp('%s.FILTER' % modname)
+
+ logger.debug('name={}'.format(self.name))
+ logger.debug('executable={}'.format(self.executable))
+ logger.debug('enabled={}'.format(self.enabled))
+ logger.debug('priority={}'.format(self.priority))
+ logger.debug('logentries={}'.format(logentries))
+ logger.debug('logdump={}'.format(self.logdump))
+ logger.debug('logreport={}'.format(self.logreport))
+ logger.debug('logfilter={}'.format(self.logfilter))
+
+ self._init_module()
+
+ stubnames = logentries.split(',')
+ self.logs = []
+
+ for stubname in stubnames:
+ stubname = stubname.strip()
+
+ if stubname[0] != '$':
+ msg = 'Log definitions must start with "$"'
+ raise epylog.ConfigError(msg)
+
+ stubname = stubname[1:]
+
+ logger.debug('stubname={}'.format(stubname))
+
+ if stubname == 'ALL':
+ logger.info('Getting ALL logs')
+ self.logs = logtracker.logs.values()
+
+ else:
+ logger.info('Getting a log matching "{}"'.format(stubname))
+
+ try:
+ log = logtracker.getlog(stubname)
+ except epylog.NoSuchLogError:
+ # Do not die, but disable this module and complain loudly
+ logger.error('Could not find log matching "{}"'.format(
+ stubname))
+ self.enabled = False
+ continue
+
+ self.logs.append(log)
+
+ if len(self.logs) == 0:
+ self.enabled = False
+ logger.warning('Module "{}" disabled'.format(self.name))
+ return
+
+ def _init_module(self):
+ """
+ Initializes an internal module by importing it and running
+ the __init__.
+ """
+ dirname = os.path.dirname(self.executable)
+ modname = os.path.basename(self.executable)
+ modname = modname[:-3]
+
+ logger.info('Importing module "{}"'.format(modname))
+
+ stuff = _loader.find_module_in_dir(modname, dirname)
+
+ if stuff:
+ try:
+ module = _loader.load_module(modname, stuff)
+ except Exception, e:
+ msg = 'Failure trying to import module "{}" ({}): {}'.format(
+ self.name, self.executable, e)
+ raise epylog.ModuleError(msg)
+
+ else:
+ msg = 'Could not find module "{}" in dir "{}"'.format(
+ modname, dirname)
+ raise epylog.ModuleError(msg)
+
+ try:
+ modclass = getattr(module, modname)
+ self.epymod = modclass(self.extraopts)
+ except AttributeError:
+ msg = 'Could not instantiate class "{}" in module "{}"'.format(
+ modname, self.executable)
+ raise epylog.ModuleError(msg)
+
+ logger.info('Opening "{}" for writing'.format(self.logfilter))
+ self.filtfh = open(self.logfilter, 'w+')
+
+ def message_match(self, message):
+ """
+ Used by internal modules to match the message of a syslog entry
+ against the list of regexes in the .regex_map.
+ """
+ handler = None
+ match_regex = None
+
+ for regex in self.epymod.regex_map.keys():
+ if regex.search(message):
+ logger.debug('match: {}'.format(message))
+ logger.debug('matching module: {}'.format(self.name))
+
+ match_regex = regex
+ handler = self.epymod.regex_map[regex]
+ break
+
+ return (handler, match_regex)
+
+ def put_filtered(self, line):
+ """
+ Puts a filtered line into the file with all filtered lines.
+ """
+ self.filtfh.write(line)
+ logger.debug('Wrote "{}" into filtfh'.format(line))
+
+ def no_report(self):
+ """
+ Cleanup routine in case there is no report for this module.
+ """
+ self.logreport = None
+ self.logfilter = None
+ self.close_filtered()
+
+ def close_filtered(self):
+ """
+ Closes the file with filtered messages.
+ """
+ self.filtfh.close()
+
+ def finalize_processing(self, rs):
+ """
+ Called at the end of all processing to generate the report,
+ return it, and delete the imported internal module.
+ """
+ logger.info('Finalizing for module "{}"'.format(self.name))
+
+ if self.filtfh.tell():
+ if not rs.is_empty():
+ report = self.epymod.finalize(rs)
+ if report:
+ logger.debug('----Report begins----')
+ logger.debug(report)
+ logger.debug('----Report ends-----')
+
+ repfh = open(self.logreport, 'w')
+ repfh.write(report)
+ repfh.close()
+ else:
+ self.logreport = None
+ self.logfilter = None
+ else:
+ logger.info('No filtered strings for this module')
+ self.logreport = None
+ self.logfilter = None
+
+ self.close_filtered()
+
+ logger.info('Done with this module, deleting')
+ del self.epymod
+
+ def get_html_report(self):
+ """
+ Get the report from a module, and if it's not HTML, making it HTML
+ first.
+ """
+ if self.logreport is None:
+ logger.info('No report from this module')
+ return None
+
+ if not os.access(self.logreport, os.R_OK):
+ msg = 'Log report from module "{}" is missing'.format(self.name)
+ raise epylog.ModuleError(msg)
+
+ logger.info('Reading the report from file "{}"'.format(self.logreport))
+
+ fh = open(self.logreport)
+ report = fh.read()
+ fh.close()
+
+ return report
+
+ def _make_into_html(self, report):
+ """
+ Utility function that turns plaintext into HTML by essentially
+ wrapping it into "<pre></pre>" and escaping the control chars.
+ """
+ report = report.replace('&', '&')
+ report = report.replace('<', '<')
+ report = report.replace('>', '>')
+
+ report = '<pre>\n{}\n</pre>'.format(report)
+
+ return report
diff --git a/py/epylog/mytempfile.py b/epylog/mytempfile.py
similarity index 100%
rename from py/epylog/mytempfile.py
rename to epylog/mytempfile.py
diff --git a/epylog/publishers.py b/epylog/publishers.py
new file mode 100644
index 0000000..c619fcd
--- /dev/null
+++ b/epylog/publishers.py
@@ -0,0 +1,657 @@
+"""
+This module is used to publish the report into a set of predefined
+publisher classes. You can write your own, as long as they contain the
+__init__ and publish methods.
+"""
+##
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+#
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
+#
+
+import epylog
+import os
+import re
+import socket
+import time
+import shutil
+import gzip
+import tempfile
+
+import logging
+
+from string import Template
+
+logger = logging.getLogger('epylog')
+
+def make_html_page(template, starttime, endtime, title, module_reports,
+ unparsed):
+ """
+ Make a html page out of a set of parameters, which include
+ module reports. Used by most, if not all, publishers.
+ """
+ logger.info('Making a standard report page')
+
+ valumap = {
+ 'starttime' : starttime,
+ 'endtime' : endtime,
+ 'title' : title,
+ 'hostname' : socket.gethostname()
+ }
+
+ logger.info('Concatenating the module reports together')
+
+ allrep = ''
+ for modrep in module_reports:
+ logger.info('Processing report for "{}"'.format(modrep.name))
+ allrep = '{}\n<h2>{}</h2>\n{}'.format(allrep, modrep.name,
+ modrep.htmlreport)
+ if allrep == '':
+ allrep = 'No module reports'
+
+ valumap['module_reports'] = allrep
+
+ if unparsed is not None:
+ unparsed = unparsed.replace('&', '&')
+ unparsed = unparsed.replace('<', '<')
+ unparsed = unparsed.replace('>', '>')
+
+ unparsed = '<pre>\n{}</pre>'.format(unparsed)
+
+ else:
+ unparsed = 'No unparsed strings'
+
+ valumap['unparsed_strings'] = unparsed
+ valumap['version'] = epylog.VERSION
+
+ endpage = Template(template).safe_substitute(valumap)
+
+ logger.debug('----htmlreport starts----')
+ logger.debug(endpage)
+ logger.debug('----htmlreport ends-----')
+
+ return endpage
+
+def do_chunked_gzip(infh, outfh, filename, ui):
+ """
+ A memory-friendly way of compressing the data.
+ """
+ gzfh = gzip.GzipFile('rawlogs', fileobj=outfh)
+
+ bartotal = infh.tell()
+ bardone = 0
+ bartitle = 'Gzipping ' + filename
+
+ infh.seek(0)
+
+ while True:
+ chunk = infh.read(epylog.CHUNK_SIZE)
+ if not chunk:
+ break
+
+ gzfh.write(chunk)
+ bardone += len(chunk)
+ ui.progressbar(bartitle, bardone, bartotal)
+ logger.debug('Wrote {} bytes'.format(len(chunk)))
+
+ gzfh.close()
+ ui.endbar(bartitle, 'gzipped down to {} bytes'.format(outfh.tell()))
+
+def mail_smtp(smtpserv, fromaddr, toaddr, msg):
+ """
+ Send mail using smtp.
+ """
+ import smtplib
+
+ logger.info('Mailing via SMTP server {}'.format(smtpserv))
+
+ server = smtplib.SMTP(smtpserv)
+ server.sendmail(fromaddr, toaddr, msg)
+ server.quit()
+
+def mail_sendmail(sendmail, msg):
+ """
+ Send mail using sendmail.
+ """
+ logger.info('Mailing the message via sendmail')
+
+ p = os.popen(sendmail, 'w')
+ p.write(msg)
+ p.close()
+
+class MailPublisher:
+ """
+ This publisher sends the results of an epylog run as an email message.
+ """
+
+ name = 'Mail Publisher'
+
+ def __init__(self, sec, config, ui):
+ self.ui = ui
+ self.tmpprefix = config.tmpprefix
+ self.section = sec
+
+ try:
+ mailto = config.get(self.section, 'mailto')
+ addrs = mailto.split(',')
+ self.mailto = []
+
+ for addr in addrs:
+ addr = addr.strip()
+ logger.debug('adding mailto=' + addr)
+ self.mailto.append(addr)
+ except:
+ self.mailto = ['root']
+
+ try:
+ mailfmt = config.get(self.section, 'format')
+ except:
+ mailfmt = 'both'
+
+ if mailfmt not in ('html', 'plain', 'both'):
+ msg = ('Format for Mail Publisher must be either "html", "plain",'
+ ' or "both." Format "%s" is unknown').format(mailfmt)
+ raise epylog.ConfigError(msg)
+
+ self.mailfmt = mailfmt
+
+ if mailfmt != 'html':
+ try:
+ lynx = config.get(self.section, 'lynx')
+ except:
+ lynx = '/usr/bin/lynx'
+ if not os.access(lynx, os.X_OK):
+ msg = 'Could not find "{}"'.format(lynx)
+ raise epylog.ConfigError(msg)
+
+ self.lynx = lynx
+ logger.info('Usable "lynx" found in "{}"'.format(self.lynx))
+
+ try:
+ include_rawlogs = config.getboolean(self.section, 'include_rawlogs')
+ except:
+ include_rawlogs = False
+
+ if include_rawlogs:
+ try:
+ rawlogs = int(config.get(self.section, 'rawlogs_limit'))
+ except:
+ rawlogs = 200
+ self.rawlogs = rawlogs * 1024
+
+ else:
+ self.rawlogs = 0
+
+ try:
+ self.smtpserv = config.get(self.section, 'smtpserv')
+ except:
+ self.smtpserv = 'localhost'
+
+ logger.debug('mailfmt={}'.format(self.mailfmt))
+ logger.debug('rawlogs={}'.format(self.rawlogs))
+ logger.debug('smtpserv={}'.format(self.smtpserv))
+
+ try:
+ self.gpg_encrypt = config.getboolean(self.section, 'gpg_encrypt')
+
+ try:
+ self.gpg_keyringdir = config.get(self.section, 'gpg_keyringdir')
+ except:
+ self.gpg_keyringdir = None
+
+ try:
+ gpg_recipients = config.get(self.section, 'gpg_recipients')
+ keyids = gpg_recipients.split(',')
+ self.gpg_recipients = []
+ for keyid in keyids:
+ keyid = keyid.strip()
+ logger.debug('adding gpg_recipient=' + keyid)
+ self.gpg_recipients.append(keyid)
+ except:
+ # Will use all recipients found in the keyring
+ self.gpg_recipients = None
+
+ try:
+ gpg_signers = config.get(self.section, 'gpg_signers')
+ keyids = gpg_signers.split(',')
+ self.gpg_signers = []
+ for keyid in keyids:
+ keyid = keyid.strip()
+ logger.debug('adding gpg_signer=' + keyid)
+ self.gpg_signers.append(keyid)
+ except:
+ self.gpg_signers = None
+
+ except:
+ self.gpg_encrypt = False
+
+ logger.debug('gpg_encrypt={}'.format(self.gpg_encrypt))
+
+
+ def publish(self, template, starttime, endtime, title, module_reports,
+ unparsed_strings, rawfh):
+ logger.info('Creating a standard html page report')
+
+ html_report = make_html_page(template, starttime, endtime, title,
+ module_reports, unparsed_strings)
+
+ self.htmlrep = html_report
+
+ self.plainrep = None
+
+ if self.mailfmt != 'html':
+ logger.info('Creating a plaintext format of the report')
+
+ tempfile.tempdir = self.tmpprefix
+ htmlfile = tempfile.mktemp('.html')
+
+ tfh = open(htmlfile, 'w')
+ tfh.write(html_report)
+ tfh.close()
+
+ logger.info('HTML report is in "{}"'.format(htmlfile))
+
+ plainfile = tempfile.mktemp('PLAIN')
+
+ logger.info('PLAIN report will go into "{}"'.format(plainfile))
+
+ logger.info('Making a syscall to "{}"'.format(self.lynx))
+
+ exitcode = os.system('{} -dump {} > {} 2>/dev/null'.format(
+ self.lynx, htmlfile, plainfile))
+
+ if exitcode or not os.access(plainfile, os.R_OK):
+ msg = 'Error making a call to "{}"'.format(self.lynx)
+ raise epylog.SysCallError(msg)
+
+ logger.info('Reading in the plain version')
+
+ tfh = open(plainfile)
+ self.plainrep = tfh.read()
+ tfh.close()
+
+ logger.debug('----plainrep follows----')
+ logger.debug(self.plainrep)
+ logger.debug('----plainrep ends----')
+
+ if self.rawlogs:
+ # GzipFile doesn't work with StringIO. :/ Bleh.
+ tempfile.tempdir = self.tmpprefix
+ outfh = open(tempfile.mktemp('GZIP'), 'w+')
+ do_chunked_gzip(rawfh, outfh, 'rawlogs', self.ui)
+ size = outfh.tell()
+
+ if size > self.rawlogs:
+ logger.warning('{} is over the defined max of "{}"'.format(
+ size, self.rawlogs))
+ logger.warning('Not attaching the raw logs')
+ self.rawlogs = 0
+ else:
+ logger.debug('Reading in the gzipped logs')
+ outfh.seek(0)
+ self.gzlogs = outfh.read()
+
+ outfh.close()
+
+ logger.info('Creating an email message')
+
+ try:
+ from email.mime.base import MIMEBase
+ from email.mime.text import MIMEText
+ from email.mime.multipart import MIMEMultipart
+ except ImportError:
+ from email.MIMEBase import MIMEBase
+ from email.MIMEText import MIMEText
+ from email.MIMEMultipart import MIMEMultipart
+
+ logger.debug('Creating a main header')
+
+ root_part = MIMEMultipart('mixed')
+ root_part.preamble = 'This is a multi-part message in MIME format.'
+
+ logger.debug('Creating the text/plain part')
+ text_part = MIMEText(self.plainrep, 'plain', 'utf-8')
+ logger.debug('Creating the text/html part')
+ html_part = MIMEText(self.htmlrep, 'html', 'utf-8')
+
+ if self.rawlogs > 0:
+ logger.debug('Creating the application/x-gzip part')
+ attach_part = MIMEBase('application', 'x-gzip')
+ attach_part.set_payload(self.gzlogs)
+
+ from email.encoders import encode_base64
+
+ logger.debug('Encoding the gzipped raw logs with base64')
+ encode_base64(attach_part)
+ attach_part.add_header('Content-Disposition', 'attachment',
+ filename='raw.log.gz')
+
+ if self.mailfmt == 'both':
+ # create another multipart for text+html
+ alt_part = MIMEMultipart('alternative')
+ alt_part.attach(text_part)
+ alt_part.attach(html_part)
+ root_part.attach(alt_part)
+ elif self.mailfmt == 'html':
+ root_part.attach(html_part)
+ elif self.mailfmt == 'plain':
+ root_part.attach(text_part)
+
+ if self.rawlogs > 0:
+ root_part.attach(attach_part)
+
+ if self.gpg_encrypt:
+ logger.info('Encrypting the message')
+
+ from StringIO import StringIO
+ try:
+ import gpgme
+
+ if self.gpg_keyringdir and os.path.exists(self.gpg_keyringdir):
+ logger.debug('Setting keyring dir to {}'.format(
+ self.gpg_keyringdir))
+ os.environ['GNUPGHOME'] = self.gpg_keyringdir
+
+ msg = root_part.as_string()
+ logger.debug('----Cleartext follows----')
+ logger.debug(msg)
+ logger.debug('----Cleartext ends----')
+
+ cleartext = StringIO(msg)
+ ciphertext = StringIO()
+
+ ctx = gpgme.Context()
+
+ ctx.armor = True
+
+ recipients = []
+ signers = []
+
+ logger.debug('gpg_recipients={}'.format(self.gpg_recipients))
+ logger.debug('gpg_signers={}'.format(self.gpg_signers))
+
+ if self.gpg_recipients is not None:
+ for recipient in self.gpg_recipients:
+ logger.debug('Looking for an encryption key for {}'.format(
+ recipient))
+ recipients.append(ctx.get_key(recipient))
+ else:
+ for key in ctx.keylist():
+ for subkey in key.subkeys:
+ if subkey.can_encrypt:
+ logger.debug('Found can_encrypt key={}'.format(
+ subkey.keyid))
+ recipients.append(key)
+ break
+
+ if self.gpg_signers is not None:
+ for signer in self.gpg_signers:
+ logger.debug('Looking for a signing key for {}'.format(
+ signer))
+ signers.append(ctx.get_key(signer))
+
+ if len(signers) > 0:
+ logger.info('Encrypting and signing the report')
+ ctx.signers = signers
+ ctx.encrypt_sign(recipients, gpgme.ENCRYPT_ALWAYS_TRUST,
+ cleartext, ciphertext)
+
+ else:
+ logger.info('Encrypting the report')
+ ctx.encrypt(recipients, gpgme.ENCRYPT_ALWAYS_TRUST,
+ cleartext, ciphertext)
+
+ logger.debug('Creating the MIME envelope for PGP')
+
+ gpg_envelope_part = MIMEMultipart('encrypted')
+ gpg_envelope_part.set_param('protocol',
+ 'application/pgp-encrypted', header='Content-Type')
+ gpg_envelope_part.preamble = ('This is an OpenPGP/MIME '
+ 'encrypted message (RFC 2440 and 3156)')
+
+ gpg_mime_version_part = MIMEBase('application', 'pgp-encrypted')
+ gpg_mime_version_part.add_header('Content-Disposition',
+ 'PGP/MIME version identification')
+ gpg_mime_version_part.set_payload('Version: 1')
+
+ gpg_payload_part = MIMEBase('application', 'octet-stream',
+ name='encrypted.asc')
+ gpg_payload_part.add_header('Content-Disposition',
+ 'OpenPGP encrypted message')
+ gpg_payload_part.add_header('Content-Disposition', 'inline',
+ filename='encrypted.asc')
+ gpg_payload_part.set_payload(ciphertext.getvalue())
+
+ gpg_envelope_part.attach(gpg_mime_version_part)
+ gpg_envelope_part.attach(gpg_payload_part)
+
+ # envelope becomes the new root part
+ root_part = gpg_envelope_part
+
+
+ except ImportError:
+ logger.error('Need crypto libraries for gpg_encrypt.')
+ logger.error('Install pygpgme for GPG encryption support.')
+ logger.error('Not mailing the report out of caution.')
+ return
+
+
+ root_part['Subject'] = title
+ root_part['To'] = ', '.join(self.mailto)
+ root_part['X-Mailer'] = epylog.VERSION
+
+ logger.debug('Creating the message as string')
+ msg = root_part.as_string()
+
+ logger.debug('----Message follows----')
+ logger.debug(msg)
+ logger.debug('----Message ends----')
+
+ logger.info('Figuring out if we are using sendmail or smtplib')
+
+ if re.compile('^/').search(self.smtpserv):
+ mail_sendmail(self.smtpserv, msg)
+ else:
+ fromaddr = 'root@%s' % socket.gethostname()
+ mail_smtp(self.smtpserv, fromaddr, self.mailto, msg)
+
+ self.ui.put('Mailed the report to: {}'.format(','.join(self.mailto)))
+
+
+class FilePublisher:
+ """
+ FilePublisher publishes the results of an Epylog run into a set of files
+ and directories on the hard drive.
+ """
+ name = 'File Publisher'
+ def __init__(self, sec, config, ui):
+ self.ui = ui
+ self.tmpprefix = config.tmpprefix
+
+ msg = 'Required attribute "{}" not found'
+ try:
+ expire = int(config.get(sec, 'expire_in'))
+ except:
+ raise epylog.ConfigError(msg.format('expire_in'))
+
+ try:
+ dirmask = config.get(sec, 'dirmask')
+ except:
+ raise epylog.ConfigError(msg.format('dirmask'))
+
+ try:
+ filemask = config.get(sec, 'filemask')
+ except:
+ raise epylog.ConfigError(msg.format('filemask'))
+
+ maskmsg = 'Invalid mask for {}: {}'
+ try:
+ self.dirname = time.strftime(dirmask, time.localtime())
+ except:
+ raise epylog.ConfigError(maskmsg.format('dirmask', dirmask))
+
+ try:
+ path = config.get(sec, 'path')
+ path = Template(path).safe_substitute(config.paths)
+ except:
+ raise epylog.ConfigError(msg.format('path'))
+
+ try:
+ self.filename = time.strftime(filemask, time.localtime())
+ except:
+ epylog.ConfigError(maskmsg.format('filemask', filemask))
+
+ self._prune_old(path, dirmask, expire)
+
+ self.path = os.path.join(path, self.dirname)
+
+ try:
+ self.save_rawlogs = config.getboolean(sec, 'save_rawlogs')
+ except:
+ self.save_rawlogs = 0
+
+ if self.save_rawlogs:
+ logger.info('Will save raw logs in the reports directory')
+
+ self.notify = []
+
+ try:
+ notify = config.get(sec, 'notify')
+ for addy in notify.split(','):
+ addy = addy.strip()
+ logger.info('Will notify: {}'.format(addy))
+ self.notify.append(addy)
+
+ except:
+ pass
+
+ try:
+ self.smtpserv = config.get(sec, 'smtpserv')
+ except:
+ self.smtpserv = '/usr/sbin/sendmail -t'
+
+ if self.notify:
+ try:
+ self.pubroot = config.get(sec, 'pubroot')
+ logger.debug('pubroot={}'.format(self.pubroot))
+ except:
+ msg = 'File publisher requires a pubroot when notify is set'
+ raise epylog.ConfigError(msg)
+
+ logger.debug('path={}'.format(self.path))
+ logger.debug('filename={}'.format(self.filename))
+
+ def _prune_old(self, path, dirmask, expire):
+ """
+ Removes the directories that are older than a certain date.
+ """
+ logger.info('Pruning directories older than {} days'.format(expire))
+
+ expire_limit = int(time.time()) - (86400 * expire)
+
+ logger.debug('expire_limit={}'.format(expire_limit))
+
+ if not os.path.isdir(path):
+ logger.info('Dir {} not found -- skipping pruning'.format(path))
+ return
+
+ for entry in os.listdir(path):
+ logger.debug('Found: {}'.format(entry))
+ if os.path.isdir(os.path.join(path, entry)):
+ try:
+ stamp = time.mktime(time.strptime(entry, dirmask))
+ except ValueError, e:
+ logger.info('Dir {} did not match dirmask {}: {}'.format(
+ entry, dirmask, e))
+ logger.info('Skipping {}'.format(entry))
+ continue
+
+ if stamp < expire_limit:
+ shutil.rmtree(os.path.join(path, entry))
+ self.ui.put('File Publisher: Pruned old dir: {}'.format(
+ entry))
+ else:
+ logger.info('{} is still active'.format(entry))
+ else:
+ logger.info('{} is not a directory. Skipping.'.format(entry))
+
+ logger.info('Finished with pruning')
+
+ def publish(self, template, starttime, endtime, title, module_reports,
+ unparsed_strings, rawfh):
+ logger.info('Checking and creating the report directories')
+
+ if not os.path.isdir(self.path):
+ try:
+ os.makedirs(self.path)
+ except OSError, e:
+ logger.error('Error creating directory "{}": {}'.format(
+ self.path, e))
+ logger.error('File publisher exiting.')
+ return
+
+ logger.info('Creating a standard html page report')
+ html_report = make_html_page(template, starttime, endtime, title,
+ module_reports, unparsed_strings)
+
+ filename = '{}.html'.format(self.filename)
+ repfile = os.path.join(self.path, filename)
+
+ logger.info('Dumping the report into {}'.format(repfile))
+
+ fh = open(repfile, 'w')
+ fh.write(html_report)
+ fh.close()
+
+ self.ui.put('Report saved in: {}'.format(self.path))
+
+ if self.notify:
+ logger.info('Creating an email message')
+ publoc = '{}/{}/{}'.format(self.pubroot, self.dirname, filename)
+
+ from email.mime.text import MIMEText
+ eml = MIMEText('New Epylog report is available at:\r\n{}'.format(
+ publoc))
+
+ eml['Subject'] = '{} (report notification)'.format(title)
+ eml['To'] = ', '.join(self.notify)
+ eml['X-Mailer'] = epylog.VERSION
+
+ msg = eml.as_string()
+
+ logger.info('Figuring out if we are using sendmail or smtplib')
+ if self.smtpserv[0] == '/':
+ mail_sendmail(self.smtpserv, msg)
+ else:
+ fromaddr = 'root@{}'.format(socket.gethostname())
+ mail_smtp(self.smtpserv, fromaddr, self.notify, msg)
+
+ self.ui.put('Notification mailed to: {}'.format(
+ ','.join(self.notify)))
+
+ if self.save_rawlogs:
+ logfilen = '{}.log'.format(self.filename)
+ logfile = os.path.join(self.path, '{}.gz'.format(logfilen))
+
+ logger.info('Gzipping logs and writing them to {}'.format(logfilen))
+ outfh = open(logfile, 'w+')
+ do_chunked_gzip(rawfh, outfh, logfilen, logger)
+ outfh.close()
+ self.ui.put('Gzipped logs saved in: {}'.format(self.path))
+
diff --git a/epylog/report.py b/epylog/report.py
new file mode 100644
index 0000000..a2676fc
--- /dev/null
+++ b/epylog/report.py
@@ -0,0 +1,232 @@
+"""
+This module handles generating the reports.
+"""
+##
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+#
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
+#
+
+import epylog
+import os
+import re
+import time
+import tempfile
+import socket
+
+from string import Template
+
+from publishers import *
+
+import logging
+
+logger = logging.getLogger('epylog')
+
+class ModuleReport:
+ """
+ A small helper class to hold Module HTML reports.
+ """
+ def __init__(self, name, htmlreport):
+ self.name = name
+ self.htmlreport = htmlreport
+
+class Report:
+ """
+ This helper class holds the contents of a report before it is
+ published using publisher classes.
+ """
+ def __init__(self, config, ui):
+ self.ui = ui
+ logger.info('Starting Report object intialization')
+
+ ##
+ # publishers: a tuple of publisher objects
+ # filt_fh: where the filtered strings from modules will go
+ # useful: tells epylog if the report is of any use or not.
+ # module_reports: module reports will be put here eventually
+ #
+ self.publishers = []
+ self.filt_fh = None
+ self.useful = False
+ self.module_reports = []
+
+ self.tmpprefix = config.tmpprefix
+ self.runtime = time.localtime()
+
+ sec = 'report'
+
+ try:
+ title = config.get(sec, 'title')
+ except:
+ title = '$hostname system events: $localtime'
+
+ try:
+ template = config.get(sec, 'template').strip()
+ except:
+ template = '$cfgdir/report_template.html'
+
+ self.template = Template(template).safe_substitute(config.paths)
+
+ try:
+ self.unparsed = config.getboolean(sec, 'include_unparsed')
+ except:
+ self.unparsed = True
+
+ try:
+ publishers = config.get(sec, 'publishers')
+ except:
+ msg = 'No publishers defined in "{}"'.format(sec)
+ raise epylog.ConfigError(msg)
+
+ titlevars = {
+ 'hostname' : socket.gethostname(),
+ 'localtime' : time.strftime('%c', self.runtime)
+ }
+
+ logger.debug('Before title={}'.format(title))
+ self.title = Template(title).safe_substitute(titlevars)
+ logger.debug('After title={}'.format(self.title))
+
+ logger.debug('template={}'.format(self.template))
+ logger.debug('unparsed={}'.format(self.unparsed))
+
+ if self.unparsed:
+ tempfile.tmpdir = self.tmpprefix
+ filen = tempfile.mktemp('.FILT')
+ self.filt_fh = open(filen, 'w+')
+ logger.info('Filtered strings file created in {}'.format(filen))
+
+ logger.debug('publishers={}'.format(publishers))
+ logger.info('Initializing publishers')
+
+ for sec in publishers.split(','):
+ sec = sec.strip()
+
+ if sec not in config.sections():
+ message = 'Publisher section "{}" not found'.format(sec)
+ raise epylog.ConfigError(message)
+
+ try:
+ method = config.get(sec, 'method')
+ except:
+ msg = 'Publishing method not defined in "{}"'.format(sec)
+ raise epylog.ConfigError(msg)
+
+ logger.debug('method={}'.format(method))
+
+ if method == 'file':
+ publisher = FilePublisher(sec, config, ui)
+ elif method == 'mail':
+ publisher = MailPublisher(sec, config, ui)
+ else:
+ msg = 'Publishing method "{}" not supported'.format(method)
+ raise epylog.ConfigError(msg )
+
+ self.publishers.append(publisher)
+
+ def append_module_report(self, module_name, module_report):
+ """
+ Appends a module report.
+ """
+ if len(module_report) > 0:
+ modrep = ModuleReport(module_name, module_report)
+ logger.info('Appending report for "{}"'.format(module_name))
+ logger.debug('----report follows----')
+ logger.debug(module_report)
+ logger.debug('----report ends----')
+ self.module_reports.append(modrep)
+ self.useful = True
+ else:
+ logger.info('Module report is empty, ignoring')
+
+ def append_filtered_strings(self, module_name, fsfh):
+ """
+ Adds filtered strings to the report.
+ """
+ if self.filt_fh is None:
+ logger.info('No open filt_fh, ignoring')
+ return
+
+ fsfh.seek(0, 2)
+
+ if fsfh.tell() != 0:
+ logger.info('Appending filtered strings from module "{}"'.format(
+ module_name))
+ logger.debug('Doing chunked read from {} to {}'.format(
+ fsfh.name, self.filt_fh.name))
+ fsfh.seek(0)
+
+ while True:
+ chunk = fsfh.read(epylog.CHUNK_SIZE)
+ if len(chunk):
+ self.filt_fh.write(chunk)
+ logger.debug('wrote {} bytes'.format(len(chunk)))
+ else:
+ logger.debug('EOF reached')
+ break
+ self.useful = True
+
+ else:
+ logger.info('Filtered Strings are empty, ignoring')
+
+ def set_stamps(self, stamps):
+ """
+ Set the timestamps of the report -- the starting and the ending one.
+ """
+ (self.start_stamp, self.end_stamp) = stamps
+ logger.debug('start_stamp={}'.format(self.start_stamp))
+ logger.debug('end_stamp={}'.format(self.end_stamp))
+
+ def publish(self, rawfh, unparsed):
+ """
+ Publishes the report using all enabled publishers.
+ """
+ if self.filt_fh is not None:
+ if unparsed is None:
+ unparsed = self.mk_unparsed_from_raw(rawfh)
+ else:
+ unparsed = ''
+
+ logger.info('Reading in the template file "{}"'.format(self.template))
+
+ fh = open(self.template)
+ template = fh.read()
+ fh.close()
+
+ starttime = time.strftime('%c', time.localtime(self.start_stamp))
+ endtime = time.strftime('%c', time.localtime(self.end_stamp))
+
+ for publisher in self.publishers:
+ logger.info('Invoking publisher "{}"'.format(publisher.name))
+
+ publisher.publish(template,
+ starttime,
+ endtime,
+ self.title,
+ self.module_reports,
+ unparsed,
+ rawfh)
+
+
+ def is_report_useful(self):
+ """
+ Returns False if the report is not useful (no new strings in logs).
+ """
+ return self.useful
+
diff --git a/etc/epylog.conf.in b/etc/epylog.conf
similarity index 51%
rename from etc/epylog.conf.in
rename to etc/epylog.conf
index 0495214..1bbbaa3 100644
--- a/etc/epylog.conf.in
+++ b/etc/epylog.conf
@@ -2,31 +2,32 @@
# Main Epylog configuration file. See epylog.conf(5) for more info.
#
[main]
-cfgdir = %%pkgconfdir%%
-tmpdir = %%TEMP_DIR%%
-vardir = %%pkgvardir%%
+cfgdir = /etc/epylog
+tmpdir = /var/tmp
+vardir = /var/lib/epylog
+moduledir = /usr/share/epylog/modules
[report]
-title = @@HOSTNAME@@ system events: @@LOCALTIME@@
-template = %%pkgconfdir%%/report_template.html
+title = $hostname system events: $localtime
+template = $cfgdir/report_template.html
+publishers = mail
include_unparsed = yes
-publishers = mail
[mail]
-method = mail
-smtpserv = /usr/sbin/sendmail -t
-mailto = root
-format = html
-lynx = %%LYNX_BIN%%
+method = mail
+smtpserv = /usr/sbin/sendmail -t
+mailto = root
+format = html
+lynx = /usr/bin/w3m
include_rawlogs = no
-rawlogs_limit = 200
+rawlogs_limit = 200
##
# GPG encryption requires pygpgme installed
#
gpg_encrypt = no
# If gpg_keyringdir is omitted, we'll use the default ~/.gnupg for the
# user running epylog (/root/.gnupg, usually).
-#gpg_keyringdir = %%pkgconfdir%%/gpg/
+#gpg_keyringdir = {vardir}/gpg/
# List key ids, can be emails or fingerprints. If omitted, we'll
# encrypt to all keys found in the pubring.
#gpg_recipients = admin1(a)example.com, admin2(a)example.com
@@ -35,12 +36,12 @@ gpg_encrypt = no
#gpg_signers = epylog(a)logserv.example.com
[file]
-method = file
-path = /var/www/html/epylog
-dirmask = %Y-%b-%d_%a
-filemask = %H%M
+method = file
+path = /var/www/epylog
+dirmask = %Y-%b-%d_%a
+filemask = %H%M
save_rawlogs = no
-expire_in = 7
-notify = root@localhost
-smtpserv = /usr/sbin/sendmail -t
-pubroot = http://localhost/epylog
+expire_in = 7
+notify = root
+smtpserv = /usr/sbin/sendmail -t
+pubroot = http://localhost/epylog
diff --git a/etc/logsources.conf b/etc/logsources.conf
new file mode 100644
index 0000000..4aa47a6
--- /dev/null
+++ b/etc/logsources.conf
@@ -0,0 +1,19 @@
+[messages]
+source = /home/mricon/work/git/epylog/log/messages
+rotated = /home/mricon/work/git/epylog/log/messages-*
+tsformat = traditional
+
+[secure]
+source = /home/mricon/work/git/epylog/log/secure
+rotated = /home/mricon/work/git/epylog/log/secure-*
+tsformat = traditional
+
+[mail]
+source = /home/mricon/work/git/epylog/log/maillog
+rotated = /home/mricon/work/git/epylog/log/maillog-*
+tsformat = traditional
+
+[cron]
+source = /home/mricon/work/git/epylog/log/cron
+rotated = /home/mricon/work/git/epylog/log/cron-*
+tsformat = traditional
diff --git a/etc/modules.d/common_unparsed.conf.in b/etc/modules.d/common_unparsed.conf
similarity index 58%
rename from etc/modules.d/common_unparsed.conf.in
rename to etc/modules.d/common_unparsed.conf
index 71a0620..00cf3fa 100644
--- a/etc/modules.d/common_unparsed.conf.in
+++ b/etc/modules.d/common_unparsed.conf
@@ -1,10 +1,9 @@
[module]
desc = Common Unparsed Similar Strings Module
-exec = %%MODULES_DIR%%/common_unparsed_mod.py
-files = /var/log/messages[.#] /var/log/secure[.#]
-enabled = yes
-internal = yes
-outhtml = yes
+exec = $moduledir/common_unparsed_mod.py
+files = $messages, $secure
+# Needs adjusting to match 1.1
+enabled = no
priority = 10
[conf]
diff --git a/etc/modules.d/logins.conf.in b/etc/modules.d/logins.conf
similarity index 88%
rename from etc/modules.d/logins.conf.in
rename to etc/modules.d/logins.conf
index f4f7b26..c3b8c5d 100644
--- a/etc/modules.d/logins.conf.in
+++ b/etc/modules.d/logins.conf
@@ -1,10 +1,10 @@
[module]
-desc = Logins
-exec = %%MODULES_DIR%%/logins_mod.py
-files = /var/log/messages[.#], /var/log/secure[.#]
-enabled = yes
+desc = Logins
+exec = $moduledir/logins_mod.py
+files = $messages, $secure
+enabled = no
internal = yes
-outhtml = yes
+outhtml = yes
priority = 0
[conf]
@@ -12,13 +12,13 @@ priority = 0
# Only enable things useful for your configuration to speed things
# up. The more stuff you enable, the slower matching will be.
#
-enable_pam = 1
-enable_xinetd = 1
-enable_sshd = 1
+enable_pam = 1
+enable_xinetd = 1
+enable_sshd = 1
enable_uw_imap = 0
enable_dovecot = 0
enable_courier = 0
-enable_imp = 0
+enable_imp = 0
enable_proftpd = 0
##
diff --git a/etc/modules.d/mail.conf.in b/etc/modules.d/mail.conf
similarity index 69%
rename from etc/modules.d/mail.conf.in
rename to etc/modules.d/mail.conf
index fc98745..57e4579 100644
--- a/etc/modules.d/mail.conf.in
+++ b/etc/modules.d/mail.conf
@@ -1,10 +1,9 @@
[module]
desc = Mail Report
-exec = %%MODULES_DIR%%/mail_mod.py
-files = /var/log/maillog[.#]
-enabled = yes
-internal = yes
-outhtml = yes
+exec = $moduledir/mail_mod.py
+files = $maillog
+# Need fixing to match 1.1
+enabled = no
priority = 3
[conf]
diff --git a/etc/modules.d/notices.conf.in b/etc/modules.d/notices.conf
similarity index 65%
rename from etc/modules.d/notices.conf.in
rename to etc/modules.d/notices.conf
index 93fccd3..8f8f0fe 100644
--- a/etc/modules.d/notices.conf.in
+++ b/etc/modules.d/notices.conf
@@ -1,22 +1,21 @@
[module]
desc = Notices
-exec = %%MODULES_DIR%%/notices_mod.py
-files = /var/log/messages[.#], /var/log/secure[.#], /var/log/maillog[.#]
-enabled = yes
-internal = yes
-outhtml = yes
+exec = $moduledir/notices_mod.py
+files = $ALL
+# Needs fixing in order to work at all
+enabled = no
priority = 9
[conf]
##
# Where is your notice_dist.xml file?
#
-notice_dist = %%pkgconfdir%%/notice_dist.xml
+notice_dist = $cfgdir/notice_dist.xml
##
# Add your own notices into notice_local.xml, not into notice_dist.xml!
# This way you don't risk missing future revisions to notice_dist.xml
#
-notice_local = %%pkgconfdir%%/notice_local.xml
+notice_local = $cfgdir/notice_local.xml
##
# You can list the ids of <notice> members from notice_dist.xml here
# namely, or you can use ALL to enable all of them. There is no need
diff --git a/etc/modules.d/ntp.conf b/etc/modules.d/ntp.conf
new file mode 100644
index 0000000..6ed5267
--- /dev/null
+++ b/etc/modules.d/ntp.conf
@@ -0,0 +1,7 @@
+[module]
+desc = NTP
+exec = $moduledir/sntp_mod.py
+files = $messages
+# Needs review and fixing
+enabled = no
+priority = 8
diff --git a/etc/modules.d/packets.conf.in b/etc/modules.d/packets.conf
similarity index 79%
rename from etc/modules.d/packets.conf.in
rename to etc/modules.d/packets.conf
index 24f89a7..673a79f 100644
--- a/etc/modules.d/packets.conf.in
+++ b/etc/modules.d/packets.conf
@@ -1,17 +1,16 @@
[module]
desc = Packet Filter
-exec = %%MODULES_DIR%%/packets_mod.py
-files = /var/log/messages[.#]
-enabled = yes
-internal = yes
-outhtml = yes
+exec = $moduledir/packets_mod.py
+files = $messages
+# Needs fixing in order to match 1.1
+enabled = no
priority = 1
[conf]
##
# Where to look for the trojans list.
#
-trojan_list = %%pkgconfdir%%/trojans.list
+trojan_list = $cfgdir/trojans.list
##
# If a remote host hits this many systems, then don't list them namely,
# but collapse them into a nice report, e.g.: [50 hosts]
diff --git a/etc/modules.d/rsyncd.conf.in b/etc/modules.d/rsyncd.conf
similarity index 67%
rename from etc/modules.d/rsyncd.conf.in
rename to etc/modules.d/rsyncd.conf
index a271f13..0e45f27 100644
--- a/etc/modules.d/rsyncd.conf.in
+++ b/etc/modules.d/rsyncd.conf
@@ -1,10 +1,9 @@
[module]
desc = Rsyncd
-exec = %%MODULES_DIR%%/rsyncd_mod.py
-files = /var/log/messages[.#]
+exec = $moduledir/rsyncd_mod.py
+files = $messages
+# Needs fixing and review for 1.1
enabled = no
-internal = yes
-outhtml = yes
priority = 7
[conf]
diff --git a/etc/modules.d/selinux.conf b/etc/modules.d/selinux.conf
new file mode 100644
index 0000000..212934b
--- /dev/null
+++ b/etc/modules.d/selinux.conf
@@ -0,0 +1,10 @@
+[module]
+desc = SELinux Report
+exec = $moduledir/selinux_mod.py
+files = $messages
+# Needs review and fixing for 1.1
+enabled = no
+priority = 5
+
+[conf]
+enable_selinux = 1
diff --git a/etc/modules.d/smart.conf b/etc/modules.d/smart.conf
new file mode 100644
index 0000000..e15983e
--- /dev/null
+++ b/etc/modules.d/smart.conf
@@ -0,0 +1,7 @@
+[module]
+desc = S.M.A.R.T.
+exec = $moduledir/smart_mod.py
+files = $messages
+# Needs review and fixing for 1.1
+enabled = no
+priority = 7
diff --git a/etc/modules.d/spamd.conf.in b/etc/modules.d/spamd.conf
similarity index 87%
rename from etc/modules.d/spamd.conf.in
rename to etc/modules.d/spamd.conf
index 107d87a..c72ae91 100644
--- a/etc/modules.d/spamd.conf.in
+++ b/etc/modules.d/spamd.conf
@@ -1,10 +1,9 @@
[module]
desc = Spamassassin
-exec = %%MODULES_DIR%%/spamd_mod.py
-files = /var/log/maillog[.#]
+exec = $moduledir/spamd_mod.py
+files = $maillog
+# Needs review and fixing for 1.1
enabled = no
-internal = yes
-outhtml = yes
priority = 4
[conf]
diff --git a/etc/modules.d/sudo.conf b/etc/modules.d/sudo.conf
new file mode 100644
index 0000000..e1c80f0
--- /dev/null
+++ b/etc/modules.d/sudo.conf
@@ -0,0 +1,9 @@
+[module]
+desc = Sudo Report
+exec = $moduledir/sudo_mod.py
+files = $secure
+enabled = yes
+priority = 5
+
+[conf]
+enable_sudo = 1
diff --git a/etc/modules.d/weeder.conf b/etc/modules.d/weeder.conf
new file mode 100644
index 0000000..8f8e134
--- /dev/null
+++ b/etc/modules.d/weeder.conf
@@ -0,0 +1,10 @@
+[module]
+desc = Weedeater
+exec = $moduledir/weeder_mod.py
+files = $ALL
+priority = 9
+enabled = yes
+
+[conf]
+weed_dist = $cfgdir/weed_dist.cf
+weed_local = $cfgdir/weed_local.cf
diff --git a/etc/modules.d/yum.conf b/etc/modules.d/yum.conf
new file mode 100644
index 0000000..f4c59cf
--- /dev/null
+++ b/etc/modules.d/yum.conf
@@ -0,0 +1,9 @@
+[module]
+desc = Yum Report
+exec = $moduledir/yum_mod.py
+files = $messages
+enabled = yes
+priority = 5
+
+[conf]
+enable_yum = 1
diff --git a/etc/notice_dist.yaml b/etc/notice_dist.yaml
new file mode 100644
index 0000000..e346a5e
--- /dev/null
+++ b/etc/notice_dist.yaml
@@ -0,0 +1,85 @@
+# CAUTION:
+# It is not advised to edit this file! You may miss any future
+# revisions made to it. Instead, create/edit notice_local.yaml and
+# add your rules to it following the same yaml format as presented in
+# this file.
+
+Gconfd locking errors:
+ tags: 'gconfd'
+ strings:
+ - 'Failed to get lock'
+ - 'Failed to create'
+ - 'Error releasing lockfile'
+ - 'Could not lock temporary file'
+ - 'another process has the lock'
+
+SFTP Activity:
+ tags: 'sshd'
+ strings:
+ - 'subsystem request for sftp'
+
+Misc floppy errors:
+ regexes:
+ - 'floppy0:|\(floppy\)'
+
+YPServ denied:
+ tags: 'ypserv'
+ regexes:
+ - 'refused\sconnect\sfrom\s(\S+):\d+\sto\sprocedure\s(\S+)'
+ report: 'ypserv: \1 denied for \2'
+
+Linux reboot:
+ critical: true
+ tags: 'kernel'
+ regexes:
+ - 'Linux\sversion\s(\S*)'
+ report: 'Rebooted with Linux kernel \1'
+
+SSH Scan:
+ tags: 'sshd'
+ regexes:
+ - 'Did not receive identification string from (\S*)'
+ report: 'SSH scan from \1'
+
+Dirty CDROM mount:
+ strings:
+ - 'VFS: busy inodes on changed media'
+
+Misc CDROM errors:
+ tags: 'kernel'
+ strings:
+ - 'cdrom: This disc doesn'
+ - 'Make sure there is a disc in the drive.'
+
+Dirty media mounts:
+ strings:
+ - 'attempt to access beyond end of device'
+ - 'kernel: bread in fat_access failed'
+ regexes:
+ - 'rw=\d+, want=\d+, limit=\d+'
+ - 'Directory sread .* failed'
+
+NFS Timeouts:
+ critical: true
+ tags: 'nfs'
+ regexes:
+ - 'server (\S+) not responding'
+ - 'server (\S+) OK'
+ report: 'NFS timeouts to server \1'
+
+insmod errors:
+ tags: 'insmod'
+ strings:
+ - 'Hint: insmod errors'
+
+Cron runs:
+ regexes:
+ - 'CROND\[\d+\]: \((\S+)\) CMD \(([^\)]+)\)'
+ - 'crond\[\d+\]: \((\S+)\) CMD \(([^\)]+)\)'
+ report: 'Crond: \1 => \2'
+
+Promiscuous mode:
+ tags: 'kernel'
+ regexes:
+ - 'device (\S+) entered promiscuous mode'
+ report: 'device \1 entered promiscuous mode'
diff --git a/etc/notice_local.yaml b/etc/notice_local.yaml
new file mode 100644
index 0000000..4802cd6
--- /dev/null
+++ b/etc/notice_local.yaml
@@ -0,0 +1,18 @@
+#This is where you should put your own notice rules. The format is
+#simple:
+#
+#Description:
+# critical: true/false
+# tags: 'syslog tag of the string ("sshd" for "sshd[55512]:")'
+# strings:
+# - 'simple substring matches'
+# - 'one or many, depending on the need'
+# regexes:
+# - '.* regex (\S+) matches (\d+)'
+# - 'if more than one, (\S+) must have the same (\d+) number of groups'
+# - 'in the same (\S+) order (\d+)'
+# report:
+# - 'Include \1 group references \2.'
+# - 'If omitted, Description will be used'
+#
+#See notice_dist.yaml for examples.
diff --git a/etc/report_template.html b/etc/report_template.html
index 4d60b38..243cd55 100644
--- a/etc/report_template.html
+++ b/etc/report_template.html
@@ -1,22 +1,22 @@
<html>
<head>
- <title>@@TITLE@@</title>
+ <title>$title</title>
<style type="text/css">
h1 {color: gray; border-bottom: 3px double silver}
h2,h3 {color: gray; border-bottom: 1px solid silver}
</style>
</head>
<body>
- <h1>@@HOSTNAME@@</h1>
- <p>First event: <strong>@@STARTTIME@@</strong><br />
- Last event: <strong>@@ENDTIME@@</strong></p>
+ <h1>$hostname<h1>
+ <p>First event: <strong>$starttime</strong><br />
+ Last event: <strong>$endtime</strong></p>
<hr />
- @@MODULE_REPORTS@@
+ $module_reports
<hr />
<h2>Unparsed Strings:</h2>
- @@UNPARSED_STRINGS@@
+ $unparsed_strings
<hr />
<p align="right">Brought to you by
- <a href="http://linux.duke.edu/projects/epylog/">@@VERSION@@</a></p>
+ <a href="http://fedorahosted.org/epylog/">$version</a></p>
</body>
</html>
diff --git a/modules/sudo_mod.py b/modules/sudo_mod.py
index d411b9f..d7c9776 100644
--- a/modules/sudo_mod.py
+++ b/modules/sudo_mod.py
@@ -8,23 +8,26 @@ Jeremy Kindy (kindyjd at wfu.edu), Wake Forest University
import sys
import re
-##
+import logging
+
+logger = logging.getLogger('epylog')
+
# This is for testing purposes, so you can invoke this from the
# modules directory. See also the testing notes at the end of the
# file.
-#
-sys.path.insert(0, '../py/')
+if __name__ == '__main__':
+ sys.path.insert(0, '../')
+
from epylog import Result, InternalModule
class sudo_mod(InternalModule):
- def __init__(self, opts, logger):
+ def __init__(self, opts):
InternalModule.__init__(self)
- self.logger = logger
- self.logger.put(2, 'initializing sudo')
+ logger.debug('initializing sudo')
rc = re.compile
- self.ignore = 0
- self.open = 1
+ self.ignore = 0
+ self.open = 1
self.not_allowed = 2
sudo_map = {
@@ -62,38 +65,37 @@ class sudo_mod(InternalModule):
#
def sudo(self, linemap):
action = self.open
- self.logger.put(2, 'sudo invoked')
sys, msg, mult = self.get_smm(linemap)
- self.logger.put(3, 'test sudo %d' % mult)
user = self._get_sudo_user(msg)
- self.logger.put(3, 'sudo user: %s' % user)
+ logger.debug('sudo user: {}'.format(user))
+
asuser = self._get_sudo_as_user(msg)
- self.logger.put(3, 'sudo asuser: %s' % asuser)
+ logger.debug('sudo asuser: {}'.format(asuser))
+
command_name = self._get_sudo_command_name(msg)
- self.logger.put(3, 'sudo command: %s' % command_name)
+ logger.debug('sudo command: {}'.format(command_name))
restuple = self._mk_restuple(sys, action, user, asuser, command_name, None)
- self.logger.put(2, 'sudo finished')
return {restuple: mult}
def sudo_na(self, linemap):
action = self.not_allowed
- self.logger.put(2, 'sudo_na invoked')
sys, msg, mult = self.get_smm(linemap)
- self.logger.put(3, 'test sudo %d' % mult)
user = self._get_sudo_user(msg)
- self.logger.put(3, 'sudo user: %s' % user)
+ logger.debug('sudo user: {}'.format(user))
+
asuser = self._get_sudo_as_user(msg)
- self.logger.put(3, 'sudo asuser: %s' % asuser)
+ logger.debug('sudo asuser: {}'.format(asuser))
+
command_name = self._get_sudo_command_name(msg)
- self.logger.put(3, 'sudo command: %s' % command_name)
+ logger.debug('sudo command: {}'.format(command_name))
+
error_message = self._get_sudo_error_message(msg)
- self.logger.put(3, 'sudo error_message: %s' % error_message)
+ logger.debug('sudo error_message: {}'.format(error_message))
restuple = self._mk_restuple(sys, action, user, asuser, command_name, error_message)
- self.logger.put(2, 'sudo finished')
return {restuple: mult}
def sudo_ignore(self, linemap):
@@ -135,7 +137,6 @@ class sudo_mod(InternalModule):
####
# Finalize the report
def finalize(self, rs):
- logger = self.logger
##
# Prepare report
#
@@ -147,7 +148,6 @@ class sudo_mod(InternalModule):
rep[action] = ''
flipper = ''
for user in rs.get_distinct((action,)):
- #logger.put(2, 'sudo user: %s' % user)
if flipper: flipper = ''
else: flipper = self.flip
service_rep = []
@@ -156,16 +156,13 @@ class sudo_mod(InternalModule):
for asuser in rs.get_distinct((action, user, command_name)):
for error_message in rs.get_distinct((action, user, command_name, asuser)):
mymap = rs.get_submap((action, user, command_name, asuser, error_message))
- #logger.put(2, 'sudo command_name: %s' % command_name)
key2s = []
for key2 in mymap.keys():
hostname = key2[0]
key2s.append('%s(%d)' % (hostname, mymap[key2]))
hostnames = ', '.join(key2s)
- #logger.put(2, 'sudo hostnames: %s' % hostnames)
service_rep.append([command_name, hostnames, asuser, error_message])
for svcrep in service_rep:
- #logger.put(2, 'sudo svcrep: %s' % svcrep)
if blank: user = ' '
else: blank = 1
if (action == self.open):
@@ -176,11 +173,9 @@ class sudo_mod(InternalModule):
if rep[self.open]:
report += self.subreport_wrap % (self.sudo_open_title, rep[self.open])
- logger.put(2, 'sudo report: self.open added')
if rep[self.not_allowed]:
report += self.subreport_na_wrap % (self.sudo_not_allowed_title, rep[self.not_allowed])
- logger.put(2, 'sudo report: self.not_allowed added')
report = self.report_wrap % report
return report
diff --git a/modules/weeder_mod.py b/modules/weeder_mod.py
index 3461c37..1ed73b4 100644
--- a/modules/weeder_mod.py
+++ b/modules/weeder_mod.py
@@ -3,7 +3,8 @@
Description will eventually go here.
"""
##
-# Copyright (C) 2003 by Duke University
+# Copyright (C) 2003-2005 by Duke University
+# Copyright (C) 2005-2012 by Konstantin Ryabitsev and contributors
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
@@ -20,10 +21,7 @@ Description will eventually go here.
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
# 02111-1307, USA.
#
-# $Id$
-#
-# @Author Konstantin Ryabitsev <icon(a)linux.duke.edu>
-# @version $Date$
+# @Author Konstantin Ryabitsev <icon(a)mricon.com>
#
import sys
@@ -34,33 +32,43 @@ import re
# modules directory. See also the testing notes at the end of the
# file.
#
-sys.path.insert(0, '../py/')
+if __name__ == '__main__':
+ sys.path.insert(0, '../')
+
from epylog import InternalModule
+import logging
+
+logger = logging.getLogger('epylog')
+
class weeder_mod(InternalModule):
- def __init__(self, opts, logger):
+ def __init__(self, opts):
InternalModule.__init__(self)
- self.logger = logger
rc = re.compile
- weed_dist = opts.get('weed_dist', '/etc/epylog/weed_dist.cf')
- weed_local = opts.get('weed_local', '/etc/epylog/weed.local.cf')
+ weed_dist = opts.get('weed_dist', '/etc/epylog/weed_dist.cf')
+ weed_local = opts.get('weed_local', '/etc/epylog/weed.local.cf')
- weed_dist = weed_dist.strip()
+ weed_dist = weed_dist.strip()
weed_local = weed_local.strip()
- logger.put(5, 'weed_dist=%s' % weed_dist)
- logger.put(5, 'weed_local=%s' % weed_local)
+ logger.debug('weed_dist={}'.format(weed_dist))
+ logger.debug('weed_local={}'.format(weed_local))
+
weed = {}
- self.regex_map = {}
+
+ self.regex_map = {}
self.section_re = rc('^\s*\[(.*)\]\s*$')
self.comment_re = rc('^\s*#')
- self.empty_re = rc('^\s*$')
+ self.empty_re = rc('^\s*$')
for weedfile in [weed_dist, weed_local]:
- try: weed = self._read_weed(open(weedfile), weed)
- except: logger.put(5, 'Error reading %s' % weedfile)
- if not weed: return
+ try:
+ weed = self._read_weed(open(weedfile), weed)
+ except:
+ logger.debug('Error reading {}'.format(weedfile))
+ if not weed:
+ return
if 'REMOVE' in weed:
removes = weed['REMOVE']
@@ -71,34 +79,51 @@ class weeder_mod(InternalModule):
regexes = weed[key]
weed[key] = []
for regex in regexes:
- if regex != remove: weed[key].append(regex)
+ if regex != remove:
+ weed[key].append(regex)
enable = opts.get('enable', 'ALL').split(',')
- if 'ADD' in weed: enable.append('ADD')
- if enable[0] == 'ALL': enable = weed.keys()
+ if 'ADD' in weed:
+ enable.append('ADD')
+
+ if enable[0] == 'ALL':
+ enable = weed.keys()
+
for key in enable:
key = key.strip()
regexes = weed.get(key, [])
for regex in regexes:
- try: regex_re = rc(regex)
+ try:
+ regex_re = rc(regex)
except:
- logger.put(5, 'Error compiling regex "%s"' % regex)
+ logger.debug('Error compiling regex "{}"'.format(regex))
continue
+
self.regex_map[regex_re] = self.do_weed
def _read_weed(self, fh, weed):
section = 'default'
- while 1:
+
+ while True:
line = fh.readline()
- if not line: break
- if self.comment_re.search(line): continue
- if self.empty_re.search(line): continue
+ if not line:
+ break
+
+ if self.comment_re.search(line):
+ continue
+ if self.empty_re.search(line):
+ continue
+
mo = self.section_re.search(line)
- if mo: section = mo.group(1)
+ if mo:
+ section = mo.group(1)
else:
- try: weed[section].append(line.strip())
- except KeyError: weed[section] = [line.strip()]
+ try:
+ weed[section].append(line.strip())
+ except KeyError:
+ weed[section] = [line.strip()]
+
return weed
##
diff --git a/modules/yum_mod.py b/modules/yum_mod.py
index c6409fc..ca9ac20 100644
--- a/modules/yum_mod.py
+++ b/modules/yum_mod.py
@@ -11,40 +11,44 @@ Jeremy Kindy (kindyjd at wfu.edu), Wake Forest University
import sys
import re
-##
+import logging
+
+logger = logging.getLogger('epylog')
+
# This is for testing purposes, so you can invoke this from the
# modules directory. See also the testing notes at the end of the
# file.
-#
-sys.path.insert(0, '../py/')
+if __name__ == '__main__':
+ sys.path.insert(0, '../')
+
from epylog import Result, InternalModule
class yum_mod(InternalModule):
- def __init__(self, opts, logger):
+ def __init__(self, opts):
InternalModule.__init__(self)
- self.logger = logger
rc = re.compile
- self.ignore = 0
+ self.ignore = 0
self.installed = 1
- self.updated = 2
- self.erased = 3
+ self.updated = 2
+ self.erased = 3
yum_map = {
- rc('yum\: Installed\:'): self.pkg_installed,
- rc('yum\: Updated\:'): self.pkg_updated,
- rc('yum\: Erased\:'): self.pkg_erased
- }
+ rc('yum\S+: Installed:'): self.pkg_installed,
+ rc('yum\S+: Updated:'): self.pkg_updated,
+ rc('yum\S+: Erased:'): self.pkg_erased
+ }
do_yum = int(opts.get('enable_yum', '1'))
self.regex_map = {}
- if do_yum: self.regex_map.update(yum_map)
+ if do_yum:
+ self.regex_map.update(yum_map)
- self.name_re = rc('profile\: \[(.*)\]')
+ self.name_re = rc('profile\: \[(.*)\]')
self.installed_name_re = rc('Installed\: (.*)')
- self.updated_name_re = rc('Updated\: (.*)')
- self.erased_name_re = rc('Erased\: (.*)')
+ self.updated_name_re = rc('Updated\: (.*)')
+ self.erased_name_re = rc('Erased\: (.*)')
self.yum_title = '<font color="blue">Yum Changes Report</font>'
self.yum_installed_title = '<font color="blue">Packages Installed</font>'
@@ -63,37 +67,31 @@ class yum_mod(InternalModule):
# Line-matching routines
#
def pkg_updated(self, linemap):
- self.logger.put(3, 'entered pkg_updated...')
action = self.updated
sys, msg, mult = self.get_smm(linemap)
- self.logger.put(3, 'test yum %d' % mult)
name = self._get_updated_name(msg)
- self.logger.put(3, 'name: %s' % name)
+ logger.debug('name: {}'.format(name))
restuple = self._mk_restuple(action, sys, name)
return {restuple: mult}
def pkg_installed(self, linemap):
- self.logger.put(3, 'entered pkg_installed...')
action = self.installed
sys, msg, mult = self.get_smm(linemap)
- self.logger.put(3, 'test yum %d' % mult)
name = self._get_installed_name(msg)
- self.logger.put(3, 'name: %s' % name)
+ logger.debug('name: {}'.format(name))
restuple = self._mk_restuple(action, sys, name)
return {restuple: mult}
def pkg_erased(self, linemap):
- self.logger.put(3, 'entered pkg_erased...')
action = self.erased
sys, msg, mult = self.get_smm(linemap)
- self.logger.put(3, 'test yum %d' % mult)
name = self._get_erased_name(msg)
- self.logger.put(3, 'name: %s' % name)
+ logger.debug('name: {}'.format(name))
restuple = self._mk_restuple(action, sys, name)
return {restuple: mult}
@@ -130,7 +128,6 @@ class yum_mod(InternalModule):
return name
def finalize(self, rs):
- logger = self.logger
##
# Prepare report
#
@@ -141,7 +138,7 @@ class yum_mod(InternalModule):
rep[action] = ''
flipper = ''
for system in rs.get_distinct((action,)):
- self.logger.put(3, 'system: %s' % system)
+ logger.debug('system: {}'.format(system))
if flipper: flipper = ''
else: flipper = self.flip
service_rep = []
@@ -167,4 +164,4 @@ class yum_mod(InternalModule):
if __name__ == '__main__':
from epylog.helpers import ModuleTest
- ModuleTest(users_mod, sys.argv)
+ ModuleTest(yum_mod, sys.argv)
12 years, 4 months
- 1
- 0
- ← Newer
- 1
- Older →