Change in vdsm[master]: HACK: run GC in domain monitor

Adam Litke has uploaded a new change for review.

Change subject: HACK: run GC in domain monitor
......................................................................

HACK: run GC in domain monitor

Change-Id: I3c560e6fbccdf50b135cc9c90b23824ae04b0376
Signed-off-by: Adam Litke 
---
M vdsm/storage/monitor.py
M vdsm/storage/sdm/__init__.py
2 files changed, 22 insertions(+), 0 deletions(-)

git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/80/40380/1

diff --git a/vdsm/storage/monitor.py b/vdsm/storage/monitor.py
index ef032a5..a2a1bd7 100644
--- a/vdsm/storage/monitor.py
+++ b/vdsm/storage/monitor.py
@@ -28,6 +28,7 @@

from . import clusterlock
 from . import misc
+from . import sdm
 from .sdc import sdCache

@@ -249,6 +250,7 @@
             self._performDomainSelftest()
             self._checkReadDelay()
             self._collectStatistics()
+            self._garbageCollect()
         except Exception as e:
             self.log.exception("Error monitoring domain %s", self.sdUUID)
             self.nextStatus.error = e
@@ -340,6 +342,14 @@
         self.nextStatus.isoPrefix = self.isoPrefix
         self.nextStatus.version = self.domain.getVersion()

+    def _garbageCollect(self):
+        if True:  # XXX: limit this to domain ver 4 or later
+            try:
+                sdm.garbageCollectStorageDomain(self.domain)
+            except:
+                self.log.exception("Garbage collection failed for domain %s",
+                                   self.domain.sdUUID)
+
     # Managing host id

def _shouldAcquireHostId(self):
diff --git a/vdsm/storage/sdm/__init__.py b/vdsm/storage/sdm/__init__.py
index 1636e63..4a31332 100644
--- a/vdsm/storage/sdm/__init__.py
+++ b/vdsm/storage/sdm/__init__.py
@@ -226,3 +226,15 @@
             cls.isolateVolumes(domain, srcImgUUID, dstImgUUID, volumeList)
     finally:
         domain.releaseClusterLock()
+
+
+def garbageCollectStorageDomain(domain):
+    if domain.isISO():
+        return
+    cls = __getStoreClass(domain)
+    hostId = getDomainHostId(domain.sdUUID)
+    domain.acquireClusterLock(hostId)
+    try:
+        cls.garbageCollectStorageDomain(domain)
+    finally:
+        domain.releaseClusterLock()

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 1:

Build Started (1/2) -> http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/18316/

[e-mail] has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 1:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 1:

Build Started (2/2) -> http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/1546/

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 1: Verified-1

Build Failed

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/18316/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/1546/ : FAILURE

[Nir Soffer]

Nir Soffer has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 1:

(2 comments)

https://gerrit.ovirt.org/#/c/40380/1//COMMIT_MSG
Commit Message:

Line 4: Commit:     Adam Litke 
Line 5: CommitDate: 2015-04-28 16:47:13 -0400
Line 6: 
Line 7: HACK: run GC in domain monitor
Line 8: 
I like this - this is not a hack! We have a thread dedicated to each storage domain, but we do not use it for most storage domain operations.
Line 9: Change-Id: I3c560e6fbccdf50b135cc9c90b23824ae04b0376

https://gerrit.ovirt.org/#/c/40380/1/vdsm/storage/monitor.py
File vdsm/storage/monitor.py:

Line 249: 
Line 250:             self._performDomainSelftest()
Line 251:             self._checkReadDelay()
Line 252:             self._collectStatistics()
Line 253:             self._garbageCollect()
The domain monitor must wake up every 10 seconds for monitoring the storage, so this must be short operation that never blocks for long time, unless the blocking means that the domain is not accessible.

Collecting garbage here means also that notifying state changes, acquiring host id and setting next status will be delayed.

I think garbage collections should be integrated in the the monitor loop, after _monitorDomain returns, something like this:

deadline = monotonic_time()

while True:
        deadline += interval
        monitor domain
        collect garbage before deadline
        sleep until deadline

So gc is performed in the time the monitor used to sleep before, and cannot disturb reporting, unless the storage is stuck, which is a fine reason to disturb reporting.
Line 254:         except Exception as e:
Line 255:             self.log.exception("Error monitoring domain %s", self.sdUUID)
Line 256:             self.nextStatus.error = e
Line 257:

[e-mail] has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 2:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])

[e-mail] has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 3:

* Update tracker::IGNORE, no Bug-Url found
* Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url.
* Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3'])

[Jenkins CI RO]

Jenkins CI RO has abandoned this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Abandoned

Abandoned due to no activity - please restore if still relevant

gerrit-hooks has posted comments on this change.

Change subject: HACK: run GC in domain monitor
......................................................................

Patch Set 3:

* Update tracker: IGNORE, no Bug-Url found