Change in vdsm[master]: logUtils: a timed log throttle filter for stats thread

Tuesday, 4 December 2012

Zhou Zheng Sheng has uploaded a new change for review.

Change subject: logUtils: a timed log throttle filter for stats thread
......................................................................

logUtils: a timed log throttle filter for stats thread

The exception log of stats thread some times contains a lot of the same
messages. The normal stats log is very verbose as well.

This patch adds a timed throttle filter to the stats log. If the a
message in the same file same line (and same exception) is raised for
above threshold times in a certain interval, the noise will be cuts down.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=853725

Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858
Signed-off-by: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt;
---
M vdsm.spec.in
M vdsm/logUtils.py
2 files changed, 40 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/89/9689/1

diff --git a/vdsm.spec.in b/vdsm.spec.in
index 45cd87a..c8b1281 100644
--- a/vdsm.spec.in
+++ b/vdsm.spec.in
@@ -137,6 +137,7 @@
 Requires: %{name}-python = %{version}-%{release}
 Requires: pyparted
 Requires: mom >= 0.3.0
+Requires: python-repoze-lru
 
 Requires(pre,preun): policycoreutils-python
 Requires(post): /usr/sbin/saslpasswd2
diff --git a/vdsm/logUtils.py b/vdsm/logUtils.py
index 28c9e0a..ef082f8 100644
--- a/vdsm/logUtils.py
+++ b/vdsm/logUtils.py
@@ -20,10 +20,13 @@
 
 import logging
 import sys
+import time
 from functools import wraps
 from logging.handlers import WatchedFileHandler
 from logging.handlers import RotatingFileHandler
 from inspect import ismethod
+from collections import deque
+from repoze.lru import LRUCache
 
 
 def funcName(func):
@@ -112,6 +115,41 @@
         return 1
 
 
+class TimedThrottleFilter(logging.Filter):
+    """
+    Throttle duplicated log messages if the same message is emitted too fast.
+    """
+
+    def __init__(self, threshold=3, interval=10, cacheSize=1024):
+        """
+        Throttle the same message if it is emitted threshold times in
+        interval seconds.
+        """
+        self._threshold = threshold
+        self._interval = interval
+        self._cache = LRUCache(cacheSize)
+
+    def _makeKey(self, record):
+        key = record.name + record.pathname + str(record.lineno)
+        if record.exc_info:
+            key += str(record.exc_info[0])  # class of the exception raised
+        return key
+
+    def filter(self, record):
+        key = self._makeKey(record)
+        stamps = self._cache.get(key, deque(maxlen=self._threshold))
+        if not stamps:
+            self._cache.put(key, stamps)
+
+        ctime = int(time.time())
+        if (len(stamps) < self._threshold or
+                ctime - stamps[0] > self._interval):
+            stamps.append(ctime)
+            return 1
+
+        return 0
+
+
 class QueueHandler(logging.Handler):
     """
     This handler sends events to a queue. Typically, it would be used together
@@ -178,4 +216,4 @@
     WatchedFileHandler, TracebackRepeatFilter())
 
 FilteredRotatingFileHandler = getFilteredClass(
-    RotatingFileHandler, TracebackRepeatFilter())
+    RotatingFileHandler, TracebackRepeatFilter(), TimedThrottleFilter(3, 10))


--
To view, visit http://gerrit.ovirt.org/9689
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I60fe36ce6b324941d2cbb60ac9b40ed85dcdb858
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Zhou Zheng Sheng <zhshzhou(a)linux.vnet.ibm.com&gt;

    

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011