Change in vdsm[master]: virt: Send migration status events
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: Send migration status events
......................................................................
virt: Send migration status events
Migration progress is currently available in stats. Engine fetches the stats
from the host where a migrating VM is running, which is the source host during
pre-copy migration and the destination host during post-copy migration.
However the progress information is always available only on the source host,
so it's not available to Engine during the post-copy phase.
For this and other reasons Engine would like to receive events about migration
progress and the downtime of a completed migration. We introduce a new event
for that purpose containing that information and additionally information about
the pre-copy or post-copy mode of the migration (since downtime value
interpretation is inherently different in the two cases).
Change-Id: I6e392150e0470b99cccdf07f50345bb9e4d02c44
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
---
M lib/api/vdsm-events.yml
M lib/vdsm/virt/migration.py
M vdsm/virt/vm.py
3 files changed, 52 insertions(+), 0 deletions(-)
Approvals:
Jenkins CI: Passed CI tests
Francesco Romani: Looks good to me, approved
Milan Zamazal: Verified
--
To view, visit https://gerrit.ovirt.org/64257
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I6e392150e0470b99cccdf07f50345bb9e4d02c44
Gerrit-PatchSet: 27
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Arik Hadas <ahadas(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Report downtime after a post-copy migration
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: Report downtime after a post-copy migration
......................................................................
virt: Report downtime after a post-copy migration
There are two migration downtime length values provided by libvirt after
a completed migration: `downtime' and `downtime_net'. Ideally,
`downtime' provides the complete downtime length, but only under the
assumption that the clocks on the hosts are perfectly synchronized. If
they are not, the value may be screwed by the clock differences.
For this reason, we report `downtime_net' for pre-copy migrations, which
includes only the time needed to transfer the data (and not other
overheads), but is correct even when the clocks are not perfectly
synchronized.
`downtime_net' is not reported for post-copy migrations as it is always
zero. So only `downtime' can be reported for post-copy migrations,
regardless of the potential problem with the clocks.
Engine and other tools reporting the downtime must take into account the
differences of the downtime value in pre-copy and post-copy.
Change-Id: I5f34e1ff577db325cc976431b3570f91833ff815
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M lib/vdsm/libvirtconnection.py
M vdsm/clientIF.py
M vdsm/virt/vm.py
3 files changed, 18 insertions(+), 2 deletions(-)
Approvals:
Jenkins CI: Passed CI tests
Francesco Romani: Looks good to me, approved
Milan Zamazal: Verified
--
To view, visit https://gerrit.ovirt.org/64144
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I5f34e1ff577db325cc976431b3570f91833ff815
Gerrit-PatchSet: 22
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Arik Hadas <ahadas(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Inform Engine about switching to post-copy
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Inform Engine about switching to post-copy
......................................................................
virt: Inform Engine about switching to post-copy
Engine must be informed when a migration switches to post-copy mode (to
know where to get further events and stats from). We use status event
for that purpose. We report paused state for VMs in post-copy migration
mode on the source, so we must distinguish between paused state in
post-copy migration and paused state for other reasons. We use
pauseCode for that purpose.
Change-Id: I281ec36a6bd65b816afbdeef79205e6241d4a27c
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M vdsm/virt/vm.py
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/49/64149/8
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 195453f..abf2d75 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -4200,6 +4200,9 @@
elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY:
self._in_post_copy = True
self.log.debug("Migration entered post-copy mode")
+ with self._confLock:
+ self.conf['pauseCode'] = 'POSTCOPY'
+ self.send_status_event(pauseCode='POSTCOPY')
elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED:
# This can happen on both the ends of the migration.
# After a failed post-copy migration, the VM remains in a
--
To view, visit https://gerrit.ovirt.org/64149
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I281ec36a6bd65b816afbdeef79205e6241d4a27c
Gerrit-PatchSet: 8
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Arik Hadas <ahadas(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Prevent monitoring of VMs in post-copy
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Prevent monitoring of VMs in post-copy
......................................................................
virt: Prevent monitoring of VMs in post-copy
Migrating VMs in post-copy mode on the source end are paused. It makes
no sense to monitor them and we don't want to get error messages in the
log or experience other problems. So we skip VMs in that state in
periodic operations.
We also skip the VMs that just successfully initiated switching to
post-copy mode. Under normal circumstances, they should enter post-copy
mode shortly afterwards, so it's better to skip them immediately.
We still can't guarantee (without excessive locking) that VM doesn't
switch to post-copy (and gets paused), so we handle the corresponding
contingent failure as well.
Note that we still gather stats, since we need to know the migration
progress.
Change-Id: I010d3311ce497081720a871e2a3c715f53dcb299
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M lib/vdsm/virt/periodic.py
M vdsm/virt/vm.py
2 files changed, 10 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/46/64146/7
diff --git a/lib/vdsm/virt/periodic.py b/lib/vdsm/virt/periodic.py
index 132fe93..959ff1b 100644
--- a/lib/vdsm/virt/periodic.py
+++ b/lib/vdsm/virt/periodic.py
@@ -290,7 +290,8 @@
def required(self):
# Disable everything until the migration destination VM
# is fully started, to avoid false positives log spam.
- return not self._vm.incomingMigrationPending()
+ return (not self._vm.incomingMigrationPending() and
+ not self._vm.post_copy_initiated)
@property
def runnable(self):
@@ -307,6 +308,9 @@
self._vm.log.warning('could not run on %s: domain not connected',
self._vm.id)
except libvirt.libvirtError as e:
+ if self._vm.in_post_copy:
+ # race on entering post-copy, VM paused now
+ return
if e.get_error_code() in (
# race on shutdown/migration completion
libvirt.VIR_ERR_NO_DOMAIN,
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index f806c45..2fb498a 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -323,6 +323,11 @@
def in_post_copy(self):
return self._in_post_copy
+ @property
+ def post_copy_initiated(self):
+ return (self._in_post_copy or
+ self._migrationSourceThread.post_copy_initiated)
+
def _get_lastStatus(self):
# note that we don't use _statusLock here. One of the reasons is the
# non-obvious recursive locking in the following flow:
--
To view, visit https://gerrit.ovirt.org/64146
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I010d3311ce497081720a871e2a3c715f53dcb299
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Start all VM related stuff after switching to post-cop...
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Start all VM related stuff after switching to post-copy migration
......................................................................
virt: Start all VM related stuff after switching to post-copy migration
On the migration destination, we perform running-VM actions only after
the migration completes. But when the migration switches to post-copy
mode, the VM starts fully running on the destination. So switching to
post-copy should be considered as the actual start of the VM and
appropriate actions should be performed.
This patch implements unblocking the start up actions after switching to
post-copy mode.
The only exception is VM status, which must still signal migration until
the migration completes.
Change-Id: Ic8b17e58f63cbd9db09e4420871a562eaa0b3f3d
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M vdsm/virt/vm.py
1 file changed, 25 insertions(+), 6 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/51/63551/8
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index e24a90a..ed60354 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -264,6 +264,7 @@
self._migrationSourceThread = migration.SourceThread(self)
self._kvmEnable = self.conf.get('kvmEnable', 'true')
self._incomingMigrationFinished = threading.Event()
+ self._incoming_migration_vm_running = threading.Event()
self.id = self.conf['vmId']
self._volPrepareLock = threading.Lock()
self._initTimePauseCode = None
@@ -540,6 +541,11 @@
if ('migrationDest' in self.conf or 'restoreState' in self.conf) \
and self.lastStatus != vmstatus.DOWN:
self._completeIncomingMigration()
+ if self.lastStatus == vmstatus.MIGRATION_DESTINATION:
+ # Waiting for post-copy migration to finish before we can
+ # change status to UP.
+ # Should we set a timeout here?
+ self._incomingMigrationFinished.wait()
self.lastStatus = vmstatus.UP
if self._initTimePauseCode:
@@ -2921,7 +2927,8 @@
if self._needToWaitForMigrationToComplete():
usedTimeout = self._waitForUnderlyingMigration()
self._attachLibvirtDomainAfterMigration(
- self._incomingMigrationFinished.isSet(), usedTimeout)
+ self._incoming_migration_vm_running.is_set(),
+ usedTimeout)
# else domain connection already established earlier
self._domDependentInit()
del self.conf['migrationDest']
@@ -2973,7 +2980,7 @@
def _waitForUnderlyingMigration(self):
timeout = config.getint('vars', 'migration_destination_timeout')
self.log.debug("Waiting %s seconds for end of migration", timeout)
- self._incomingMigrationFinished.wait(timeout)
+ self._incoming_migration_vm_running.wait(timeout)
return timeout
def _attachLibvirtDomainAfterMigration(self, migrationFinished, timeout):
@@ -2991,7 +2998,7 @@
raise MigrationError("Migration Error - Timed out "
"(did not receive success "
"event)")
- self.log.debug("NOTE: incomingMigrationFinished event has "
+ self.log.debug("NOTE: incoming_migration_vm_running event has "
"not been set and wait timed out after %d "
"seconds. Current VM state: %d, reason %d. "
"Continuing with VM initialization anyway.",
@@ -3920,7 +3927,7 @@
self.log.info('Release VM resources')
self.lastStatus = vmstatus.POWERING_DOWN
# Terminate the VM's creation thread.
- self._incomingMigrationFinished.set()
+ self._incoming_migration_vm_running.set()
self.guestAgent.stop()
if self._dom.connected:
result = self._destroyVm(gracefulAttempts)
@@ -4155,6 +4162,8 @@
pass
else:
hooks.after_vm_pause(domxml, self.conf)
+ elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED:
+ pass # will be handled in a followup patch
elif event == libvirt.VIR_DOMAIN_EVENT_RESUMED:
self._setGuestCpuRunning(True)
@@ -4171,9 +4180,19 @@
pass
else:
hooks.after_vm_cont(domxml, self.conf)
- elif (detail == libvirt.VIR_DOMAIN_EVENT_RESUMED_MIGRATED and
- self.lastStatus == vmstatus.MIGRATION_DESTINATION):
+ elif (self.lastStatus == vmstatus.MIGRATION_DESTINATION and
+ detail == libvirt.VIR_DOMAIN_EVENT_RESUMED_MIGRATED):
+ self._incoming_migration_vm_running.set()
self._incomingMigrationFinished.set()
+ elif (self.lastStatus == vmstatus.MIGRATION_DESTINATION and
+ detail == libvirt.VIR_DOMAIN_EVENT_RESUMED_POSTCOPY):
+ # When we enter post-copy mode, the VM starts actually
+ # running on the destination, so we should unblock the
+ # start up processing here. The only exception is status,
+ # which must still signal incoming migration to not confuse
+ # Engine.
+ self._incoming_migration_vm_running.set()
+ self.log.info("Migration switched to post-copy mode")
def _updateDevicesDomxmlCache(self, xml):
"""
--
To view, visit https://gerrit.ovirt.org/63551
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic8b17e58f63cbd9db09e4420871a562eaa0b3f3d
Gerrit-PatchSet: 8
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Fetch only minimum VM stats during post-copy migration
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Fetch only minimum VM stats during post-copy migration
......................................................................
virt: Fetch only minimum VM stats during post-copy migration
When a migration switches to post-copy mode, the VM starts running on
the destination. Libvirt no longer provides actual stats on the source,
with the exception of progress, which is still available on the source
and only on the source.
This patch stops reporting stats other than progress during post-copy
migration.
Change-Id: Iac3da08b0816e7a9a247969147c17b15518c7624
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M vdsm/virt/migration.py
M vdsm/virt/vm.py
2 files changed, 21 insertions(+), 4 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/52/63552/8
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py
index 13f2155..63cdf8c 100644
--- a/vdsm/virt/migration.py
+++ b/vdsm/virt/migration.py
@@ -149,6 +149,11 @@
def hibernating(self):
return self._mode == MODE_FILE
+ @property
+ def in_post_copy(self):
+ return (self._monitorThread is not None and
+ self._monitorThread.in_post_copy)
+
def getStat(self):
"""
Get the status of the migration.
@@ -653,6 +658,10 @@
def enabled(self):
return MonitorThread._MIGRATION_MONITOR_INTERVAL > 0
+ @property
+ def in_post_copy(self):
+ return self._in_post_copy
+
@utils.traceback()
def run(self):
if self.enabled:
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index e274388..f32d119 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -1198,10 +1198,15 @@
if self.lastStatus == vmstatus.DOWN:
stats.update(self._getDownVmStats())
else:
- stats.update(self._getConfigVmStats())
- stats.update(self._getRunningVmStats())
+ if self.isMigrating() and self._migrationSourceThread.in_post_copy:
+ # Stats are on the destination during post-copy migration,
+ # except for migration progress, which is always on the source.
+ stats['migrationProgress'] = self._get_vm_migration_progress()
+ else:
+ stats.update(self._getConfigVmStats())
+ stats.update(self._getRunningVmStats())
+ stats.update(self._getGuestStats())
stats['status'] = self._getVmStatus()
- stats.update(self._getGuestStats())
return stats
def _getDownVmStats(self):
@@ -1255,7 +1260,7 @@
if 'pauseCode' in self.conf:
stats['pauseCode'] = self.conf['pauseCode']
if self.isMigrating():
- stats['migrationProgress'] = self.migrateStatus()['progress']
+ stats['migrationProgress'] = self._get_vm_migration_progress()
try:
vm_sample = sampling.stats_cache.get(self.id)
@@ -1346,6 +1351,9 @@
else:
return self.lastStatus
+ def _get_vm_migration_progress(self):
+ return self.migrateStatus()['progress']
+
def _getGraphicsStats(self):
def getInfo(dev):
return {
--
To view, visit https://gerrit.ovirt.org/63552
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iac3da08b0816e7a9a247969147c17b15518c7624
Gerrit-PatchSet: 8
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Distinguish between switching to and entering post-cop...
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Distinguish between switching to and entering post-copy migration
......................................................................
virt: Distinguish between switching to and entering post-copy migration
Switching to migration post-copy mode has two phases: Asking for
switching to post-copy and actually switching to post-copy. The
transition period should be short (if it is not then it is a
QEMU/libvirt bug), so we needn't care about it much.
But on various checks we should distinguish between the post-copy
request phase and the post-copy switch phase. For instance, we should
block canceling migrations very early to avoid timing issues and
possibly canceling a post-copy migration. On the other hand we should
report a true VM state to Engine, so we can report post-copy only after
the VM actually enters it.
Change-Id: I88dddeab84a609ebd3d5d9139724adff9cdb1352
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M vdsm/virt/migration.py
M vdsm/virt/vm.py
2 files changed, 29 insertions(+), 13 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/43/64143/7
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py
index df478cd..2c73375 100644
--- a/vdsm/virt/migration.py
+++ b/vdsm/virt/migration.py
@@ -150,9 +150,9 @@
return self._mode == MODE_FILE
@property
- def in_post_copy(self):
+ def post_copy_initiated(self):
return (self._monitorThread is not None and
- self._monitorThread.in_post_copy)
+ self._monitorThread.post_copy_initiated)
def getStat(self):
"""
@@ -644,7 +644,7 @@
self.progress = None
self._conv_schedule = conv_schedule
self._use_conv_schedule = use_conv_schedule
- self._in_post_copy = False
+ self._post_copy_initiated = False
self.downtime_thread = _FakeThreadInterface()
self._thread = concurrent.thread(self.run)
@@ -659,8 +659,8 @@
return MonitorThread._MIGRATION_MONITOR_INTERVAL > 0
@property
- def in_post_copy(self):
- return self._in_post_copy
+ def post_copy_initiated(self):
+ return self._post_copy_initiated
@utils.traceback()
def run(self):
@@ -709,7 +709,7 @@
progress = Progress.from_job_stats(job_stats)
now = time.time()
- if self._in_post_copy:
+ if self._post_copy_initiated:
# Post-copy mode is a final state of a migration -- it either
# completes or fails and stops the VM, there is no way to
# continue with the migration in either case. So we won't
@@ -718,8 +718,15 @@
# abort action after the post-copy action in the schedule, for
# the case when it's not possible to switch to the post-copy
# mode for some reason.
- self._vm.log.debug('Post-copy migration still in progress: %d',
- progress.data_remaining)
+ if self._vm.in_post_copy:
+ # If in_post-copy flag is false then we are in the interim
+ # phase (which should be short) between initiating the
+ # post-copy migration and the actual start of the post-copy
+ # migration. Nothing needs to be done in that case.
+ self._vm.log.debug(
+ 'Post-copy migration still in progress: %d',
+ progress.data_remaining
+ )
elif not self._use_conv_schedule and\
(0 < migrationMaxTime < now - self._startTime):
self._vm.log.warn('The migration took %d seconds which is '
@@ -741,7 +748,7 @@
' Refer to RHBZ#919201.',
progress.data_remaining / Mbytes, lowmark / Mbytes)
- if not self._in_post_copy and\
+ if not self._post_copy_initiated and\
lastDataRemaining is not None and\
lastDataRemaining < progress.data_remaining:
iterationCount += 1
@@ -802,8 +809,8 @@
vm.log.info('Switching to post-copy migration')
ret = vm.switch_migration_to_post_copy()
if ret >= 0:
- self._in_post_copy = True
- if not self._in_post_copy:
+ self._post_copy_initiated = True
+ if not self._post_copy_initiated:
# Do nothing for now; the next action will be invoked after
# a while
vm.log.warn('Failed to switch to post-copy migration')
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 0f2ece8..f806c45 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -309,6 +309,7 @@
self._numaInfo = {}
self._vmJobs = None
self._clientPort = ''
+ self._in_post_copy = False
@property
def start_time(self):
@@ -317,6 +318,10 @@
@property
def domain(self):
return self._domain
+
+ @property
+ def in_post_copy(self):
+ return self._in_post_copy
def _get_lastStatus(self):
# note that we don't use _statusLock here. One of the reasons is the
@@ -1199,7 +1204,7 @@
if self.lastStatus == vmstatus.DOWN:
stats.update(self._getDownVmStats())
else:
- if self.isMigrating() and self._migrationSourceThread.in_post_copy:
+ if self.isMigrating() and self._in_post_copy:
# Stats are on the destination during post-copy migration,
# except for migration progress, which is always on the source.
stats['migrationProgress'] = self._get_vm_migration_progress()
@@ -1428,7 +1433,8 @@
def migrateCancel(self):
with self._post_copy_lock:
- if self._migrationSourceThread.in_post_copy:
+ if self._migrationSourceThread.post_copy_initiated or \
+ self._in_post_copy:
return response.error(
'migCancelErr',
message=("Can't cancel migration in post-copy mode")
@@ -4181,6 +4187,9 @@
pass
else:
hooks.after_vm_pause(domxml, self.conf)
+ elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY:
+ self._in_post_copy = True
+ self.log.debug("Migration entered post-copy mode")
elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED:
# This can happen on both the ends of the migration.
# After a failed post-copy migration, the VM remains in a
--
To view, visit https://gerrit.ovirt.org/64143
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I88dddeab84a609ebd3d5d9139724adff9cdb1352
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Stop guest agent before switching to post-copy
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Stop guest agent before switching to post-copy
......................................................................
virt: Stop guest agent before switching to post-copy
Post-copy migration pauses the VM on the source. We don't want to get
spurious log messages from the guest agent about not being able to talk
to the VM or being disconnected from the communication socket. So we
stop the guest agent before switching to post-copy mode.
If switching to post-copy fails (which should rarely happen), we start
the agent again. Starting the agent may fail, e.g. in case the VM got
destroyed in the meantime. So we catch the contingent exception.
Change-Id: Ibd855ee36847b293009ac35d8d03b01521aaace8
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
Bug-Url: https://bugzilla.redhat.com/1354343
---
M vdsm/virt/migration.py
1 file changed, 7 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/45/64145/7
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py
index 2c73375..522a4fc 100644
--- a/vdsm/virt/migration.py
+++ b/vdsm/virt/migration.py
@@ -807,6 +807,7 @@
vm._dom.migrateSetMaxDowntime(downtime, 0)
elif action == CONVERGENCE_SCHEDULE_POST_COPY:
vm.log.info('Switching to post-copy migration')
+ vm.guestAgent.stop()
ret = vm.switch_migration_to_post_copy()
if ret >= 0:
self._post_copy_initiated = True
@@ -814,6 +815,12 @@
# Do nothing for now; the next action will be invoked after
# a while
vm.log.warn('Failed to switch to post-copy migration')
+ try:
+ vm.guestAgent.start()
+ except:
+ self.log.exception("Failed to start guest agent after "
+ "unsuccessful switch to post-copy "
+ "migration")
elif action == CONVERGENCE_SCHEDULE_SET_ABORT:
vm.log.warn('Aborting migration')
vm._dom.abortJob()
--
To view, visit https://gerrit.ovirt.org/64145
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd855ee36847b293009ac35d8d03b01521aaace8
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Use a variable for self._vm in MonitorThread action ha...
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Use a variable for self._vm in MonitorThread action handling
......................................................................
virt: Use a variable for self._vm in MonitorThread action handling
This is a reformatting patch to prevent further line breaking in a
followup patch.
Change-Id: I3e41f8b3ef27319c51e18ad2c75bbb3c61b61c76
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
---
M vdsm/virt/migration.py
1 file changed, 6 insertions(+), 6 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/63553/9
diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py
index 4633d47..6cd88f4 100644
--- a/vdsm/virt/migration.py
+++ b/vdsm/virt/migration.py
@@ -783,19 +783,19 @@
def _execute_action_with_params(self, action_with_params):
action = str(action_with_params['name'])
+ vm = self._vm
if action == CONVERGENCE_SCHEDULE_SET_DOWNTIME:
downtime = int(action_with_params['params'][0])
- self._vm.log.debug('Setting downtime to %d',
- downtime)
- self._vm._dom.migrateSetMaxDowntime(downtime, 0)
+ vm.log.debug('Setting downtime to %d', downtime)
+ vm._dom.migrateSetMaxDowntime(downtime, 0)
elif action == CONVERGENCE_SCHEDULE_POST_COPY:
if not self._vm.switch_migration_to_post_copy():
# Do nothing for now; the next action will be invoked after a
# while
- self._vm.log.warn('Failed to switch to post-copy migration')
+ vm.log.warn('Failed to switch to post-copy migration')
elif action == CONVERGENCE_SCHEDULE_SET_ABORT:
- self._vm.log.warn('Aborting migration')
- self._vm._dom.abortJob()
+ vm.log.warn('Aborting migration')
+ vm._dom.abortJob()
self.stop()
--
To view, visit https://gerrit.ovirt.org/63553
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3e41f8b3ef27319c51e18ad2c75bbb3c61b61c76
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months
Change in vdsm[master]: virt: Refuse to cancel post-copy migrations
by mzamazal＠redhat.com
Milan Zamazal has uploaded a new change for review.
Change subject: virt: Refuse to cancel post-copy migrations
......................................................................
virt: Refuse to cancel post-copy migrations
A VM in post-copy migration is split between the source and the
destination, with inconsistent memory state on both the ends of the
migration. Stopping the migration in that phase is not a good idea, it
loses the VM and leaves it in the pause state on the source and may lead
to data loss.
Since it's not possible to stop a post-copy migration safely and the
migration is guaranteed to finish sooner or later, we prevent its
cancellation with this patch.
Change-Id: I670ab48ed3fe69582646675d8b683bcdb3cfa038
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
---
M vdsm/virt/vm.py
1 file changed, 25 insertions(+), 17 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/54/63554/10
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index b5a92cb..11cacc9 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -274,6 +274,7 @@
self._guestEventTime = 0
self._guestCpuRunning = False
self._guestCpuLock = threading.Lock()
+ self._post_copy_lock = threading.Lock()
self._startTime = time.time() - \
float(self.conf.pop('elapsedTimeOffset', 0))
@@ -1447,20 +1448,26 @@
return self._migrationSourceThread.getStat()
def migrateCancel(self):
- self._acquireCpuLockWithTimeout()
- try:
- self._migrationSourceThread.stop()
- self._migrationSourceThread.status['status']['message'] = \
- 'Migration process cancelled'
- return self._migrationSourceThread.status
- except libvirt.libvirtError as e:
- if e.get_error_code() == libvirt.VIR_ERR_OPERATION_INVALID:
+ with self._post_copy_lock:
+ if self.post_copy:
+ return response.error(
+ 'migCancelErr',
+ message=("Can't cancel migration in post-copy mode")
+ )
+ self._acquireCpuLockWithTimeout()
+ try:
+ self._migrationSourceThread.stop()
+ self._migrationSourceThread.status['status']['message'] = \
+ 'Migration process cancelled'
+ return self._migrationSourceThread.status
+ except libvirt.libvirtError as e:
+ if e.get_error_code() == libvirt.VIR_ERR_OPERATION_INVALID:
+ return response.error('migCancelErr')
+ raise
+ except virdomain.NotConnectedError:
return response.error('migCancelErr')
- raise
- except virdomain.NotConnectedError:
- return response.error('migCancelErr')
- finally:
- self._guestCpuLock.release()
+ finally:
+ self._guestCpuLock.release()
def _getSerialConsole(self):
"""
@@ -1512,10 +1519,11 @@
corresponding libvirt event.
"""
self.log.info('Switching to post-copy migration')
- result = self._dom.migrateStartPostCopy(0)
- success = result >= 0
- if success:
- self._post_copy = True
+ with self._post_copy_lock:
+ result = self._dom.migrateStartPostCopy(0)
+ success = result >= 0
+ if success:
+ self._post_copy = True
return success
def _customDevices(self):
--
To view, visit https://gerrit.ovirt.org/63554
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I670ab48ed3fe69582646675d8b683bcdb3cfa038
Gerrit-PatchSet: 10
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years, 6 months