Zhou Zheng Sheng has uploaded a new change for review.
Change subject: logUtils: a timed log throttle filter for stats thread ......................................................................
logUtils: a timed log throttle filter for stats thread
The exception log of stats thread some times contains a lot of the same messages. The normal stats log is very verbose as well.
This patch adds a timed throttle filter to the stats log. If the a message in the same file same line (and same exception) is raised for above threshold times in a certain interval, the noise will be cuts down.
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=853725
Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858 Signed-off-by: Zhou Zheng Sheng zhshzhou@linux.vnet.ibm.com --- M vdsm.spec.in M vdsm/logUtils.py 2 files changed, 40 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/89/9689/1
diff --git a/vdsm.spec.in b/vdsm.spec.in index 45cd87a..c8b1281 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -137,6 +137,7 @@ Requires: %{name}-python = %{version}-%{release} Requires: pyparted Requires: mom >= 0.3.0 +Requires: python-repoze-lru
Requires(pre,preun): policycoreutils-python Requires(post): /usr/sbin/saslpasswd2 diff --git a/vdsm/logUtils.py b/vdsm/logUtils.py index 28c9e0a..ef082f8 100644 --- a/vdsm/logUtils.py +++ b/vdsm/logUtils.py @@ -20,10 +20,13 @@
import logging import sys +import time from functools import wraps from logging.handlers import WatchedFileHandler from logging.handlers import RotatingFileHandler from inspect import ismethod +from collections import deque +from repoze.lru import LRUCache
def funcName(func): @@ -112,6 +115,41 @@ return 1
+class TimedThrottleFilter(logging.Filter): + """ + Throttle duplicated log messages if the same message is emitted too fast. + """ + + def __init__(self, threshold=3, interval=10, cacheSize=1024): + """ + Throttle the same message if it is emitted threshold times in + interval seconds. + """ + self._threshold = threshold + self._interval = interval + self._cache = LRUCache(cacheSize) + + def _makeKey(self, record): + key = record.name + record.pathname + str(record.lineno) + if record.exc_info: + key += str(record.exc_info[0]) # class of the exception raised + return key + + def filter(self, record): + key = self._makeKey(record) + stamps = self._cache.get(key, deque(maxlen=self._threshold)) + if not stamps: + self._cache.put(key, stamps) + + ctime = int(time.time()) + if (len(stamps) < self._threshold or + ctime - stamps[0] > self._interval): + stamps.append(ctime) + return 1 + + return 0 + + class QueueHandler(logging.Handler): """ This handler sends events to a queue. Typically, it would be used together @@ -178,4 +216,4 @@ WatchedFileHandler, TracebackRepeatFilter())
FilteredRotatingFileHandler = getFilteredClass( - RotatingFileHandler, TracebackRepeatFilter()) + RotatingFileHandler, TracebackRepeatFilter(), TimedThrottleFilter(3, 10))
-- To view, visit http://gerrit.ovirt.org/9689 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange Gerrit-Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Zhou Zheng Sheng zhshzhou@linux.vnet.ibm.com