Francesco Romani has uploaded a new change for review.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
virt: migration: decouple monitoring from thread
make the monitor logic decoupled from the monitor thread/scheduler in order to improve the testability.
Change-Id: I364a9eeb72e3b4213278adff352f3eade19548a3 Signed-off-by: Francesco Romani fromani@redhat.com --- M vdsm/virt/migration.py 1 file changed, 38 insertions(+), 26 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/26279/1
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py index 5d00120..8ac25e0 100644 --- a/vdsm/virt/migration.py +++ b/vdsm/virt/migration.py @@ -343,6 +343,32 @@
class MonitorThread(threading.Thread): + def __init__(self, vm, startTime, downTime): + super(MonitorThread, self).__init__() + self._stop = threading.Event() + self.daemon = True + self._mon = Monitor(vm, startTime, downTime) + + def run(self): + self._vm.log.debug('starting migration monitor thread') + + step = 1 + done = False + + while not self._stop.isSet() and not done: + self._stop.wait(1.0) + done = (self._mon.monitor_migration(step) and + self._mon.monitor_downtime(step)) + step += 1 + + self._vm.log.debug('migration monitor thread exiting') + + def stop(self): + self._vm.log.debug('stopping migration monitor thread') + self._stop.set() + + +class Monitor(object): _MONITOR_INTERVAL = config.getint( 'vars', 'migration_monitor_interval') # seconds _MAX_TIME_PER_GIB = config.getint( @@ -355,11 +381,8 @@ 'vars', 'migration_downtime_steps')
def __init__(self, vm, startTime, downTime): - super(MonitorThread, self).__init__() - self._stop = threading.Event() self._vm = vm self._startTime = startTime - self.daemon = True self.progress = 0
memSize = int(self._vm.conf['memSize']) @@ -371,28 +394,20 @@ self._DELAY_PER_GIB * max(memSize, 2048) + 1023) / 1024 self._downtimeInterval = self._wait / self._DOWNTIME_STEPS self._downtimeStep = 0 + self._lowmark = None + self._lastProgressTime = None
@property def enabled(self): return MonitorThread._MIGRATION_MONITOR_INTERVAL > 0
- def run(self): - self._vm.log.debug('starting migration monitor thread') - - step = 1 - self._lastProgressTime = time.time() - self._lowmark = None - - while not self._stop.isSet(): - self._stop.wait(1.0) - if self.enabled: - self.monitor_migration(step) - self.monitor_downtime(step) - step += 1 - - self._vm.log.debug('migration monitor thread exiting') - def monitor_migration(self, step): + if not self.enabled: + return False + + if self._lastProgressTime is None: + self._lastProgressTime = time.time() + def calculateProgress(remaining, total): if remaining == 0: return 100 @@ -428,8 +443,7 @@
if abort: self._vm._dom.abortJob() - self.stop() - return + return abort
if dataRemaining > self._lowmark: self._vm.log.warn( @@ -439,13 +453,14 @@ dataRemaining / Mbytes, self._lowmark / Mbytes)
if jobType == 0: - return + return False
self.progress = calculateProgress(dataRemaining, dataTotal)
self._vm.log.info('Migration Progress: %s seconds elapsed, %s%% of' ' data processed' % (timeElapsed / 1000, self.progress)) + return False
def update_downtime(self, i): return self._downtime * (i + 1) / self._DOWNTIME_STEPS @@ -457,7 +472,4 @@ self._vm.log.debug('setting migration downtime to %d', downtime) self._vm._dom.migrateSetMaxDowntime(downtime, 0) self._downtimeStep += 1 - - def stop(self): - self._vm.log.debug('stopping migration monitor thread') - self._stop.set() + return False
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 1:
Build Successful
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/6970/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/7872/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7761/ : SUCCESS
Vinzenz Feenstra has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 3: Code-Review+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 2:
Build Successful
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/6977/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/7879/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7768/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 3:
Build Successful
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/6984/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/7886/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7775/ : SUCCESS
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 4:
Patch set 4: rebased
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 4:
Build Failed
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit_el/7009/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_unit_tests_gerrit/7912/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/7801/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 5:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8042/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8155/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/7252/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 6:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/8279/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/7489/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/8397/ : SUCCESS
Antoni Segura Puimedon has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 6: Code-Review-1
(1 comment)
http://gerrit.ovirt.org/#/c/26279/6/vdsm/virt/migration.py File vdsm/virt/migration.py:
Line 474: downtime = self.update_downtime(self._downtimeStep) Line 475: self._vm.log.debug('setting migration downtime to %d', downtime) Line 476: self._vm._dom.migrateSetMaxDowntime(downtime, 0) Line 477: self._downtimeStep += 1 Line 478: return False if monitor_downtime always returns False, 'done' at line 363 will always be false.
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 6:
(1 comment)
http://gerrit.ovirt.org/#/c/26279/6/vdsm/virt/migration.py File vdsm/virt/migration.py:
Line 474: downtime = self.update_downtime(self._downtimeStep) Line 475: self._vm.log.debug('setting migration downtime to %d', downtime) Line 476: self._vm._dom.migrateSetMaxDowntime(downtime, 0) Line 477: self._downtimeStep += 1 Line 478: return False
if monitor_downtime always returns False, 'done' at line 363 will always be
yep, this is what is supposed to be.
In the expected, regular case, the migration monitor thread should live long at least as much as the underlying migration takes. The 'done' flag it is used to signal abnormal termination, as in the case of aborted migration.
This is the reason why monitor_migration can actually return True, and why monitor_downtime always return False.
However, this should be better expressed in the code. I'll fix this.
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 6:
(1 comment)
http://gerrit.ovirt.org/#/c/26279/6/vdsm/virt/migration.py File vdsm/virt/migration.py:
Line 474: downtime = self.update_downtime(self._downtimeStep) Line 475: self._vm.log.debug('setting migration downtime to %d', downtime) Line 476: self._vm._dom.migrateSetMaxDowntime(downtime, 0) Line 477: self._downtimeStep += 1 Line 478: return False
yep, this is what is supposed to be.
And I actually read your comment in reverse, so my comment is out of place. Anyway, the bulk of my comment remains, my code is clumsy (and buggy)
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 7: Verified+1
Verified with 25976, 25977, 25978 running migrations a few times and observing the migration progress (logs, UI) and the downtime (logs) being updated.
Vinzenz Feenstra has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 7: Code-Review+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 7:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9694/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8761/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9547/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 8:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/8920/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/9704/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/687/ : There was an infra issue, please contact infra@ovirt.org
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/9859/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 9:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9239/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10023/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/850/ : There was an infra issue, please contact infra@ovirt.org
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10178/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5105/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3262/ : SUCCESS
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 10:
rebased
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 10:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9303/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10087/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/870/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10243/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5169/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3327/ : SUCCESS
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 10:
rebased
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 11:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9354/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10138/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/893/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10294/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5220/ : FAILURE
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3378/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 12:
Build Successful
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9526/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10310/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit/10466/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_verify-error-codes_merged/5392/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_merged/3550/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/949/ : SUCCESS
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 13:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9716/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10501/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1050/ : There was an infra issue, please contact infra@ovirt.org
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/10658/ : SUCCESS
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 14:
Verified by doing a few migrations back and forth Tested-With: 25976, 25977, 25978, 26279
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 14: Verified+1
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 14:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/9739/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10524/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1066/ : There was an infra issue, please contact infra@ovirt.org
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/10681/ : SUCCESS
Francesco Romani has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 15: Verified+1
score lost to not-trivial rebase. Copied.
oVirt Jenkins CI Server has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 15:
Build Failed
http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/10134/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/10919/ : SUCCESS
http://jenkins.ovirt.org/job/vdsm_master_virt_functional_tests_gerrit/1221/ : There was an infra issue, please contact infra@ovirt.org
http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/11076/ : SUCCESS
Dan Kenigsberg has posted comments on this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Patch Set 15: Code-Review-1
(1 comment)
http://gerrit.ovirt.org/#/c/26279/15/vdsm/virt/migration.py File vdsm/virt/migration.py:
Line 353: """ Line 354: Line 355: Line 356: class MonitorThread(threading.Thread): Line 357: _MONITOR_TICK = 1 # unit: seconds. must be integer > 0. I thought we agreed that "step" must count steps, not seconds or "ticks". Line 358: Line 359: def __init__(self, vm, startTime, downTime): Line 360: super(MonitorThread, self).__init__() Line 361: self._stop = threading.Event()
Francesco Romani has abandoned this change.
Change subject: virt: migration: decouple monitoring from thread ......................................................................
Abandoned
further thoughts on this topic, for example during the discussion of http://gerrit.ovirt.org/#/c/25820/ , proved that this merger may not be so much beneficial after all. So, will take another route to make migration nicer. This patchset will be resumed if more compelling proof arise.
vdsm-patches@lists.fedorahosted.org