vdsm-patches September 2015

vdsm-patches@lists.fedorahosted.org

46 participants
667 discussions

Change in vdsm[master]: VDSM: implement nodeDeviceMapper
by Martin Polednik 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: VDSM: implement nodeDeviceMapper ...................................................................... VDSM: implement nodeDeviceMapper NodeDeviceMapper is structure to keep track of available node devices and provide easy access to querying and managing them. The mapper is needed in order to allow VDSM to meaningfully report the usage of these devices, along with managing their availability. Change-Id: I6f21d465a90cfe2eb16ba70943e5fcf1683f1656 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M client/vdsClient.py M debian/vdsm.install M vdsm.spec.in M vdsm/API.py M vdsm/clientIF.py M vdsm/rpc/BindingXMLRPC.py M vdsm/virt/Makefile.am 7 files changed, 44 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/30209/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 2c09b28..89dceab 100644 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -461,6 +461,14 @@ def do_getAllVmStats(self, args): return self.ExecAndExit(self.s.getAllVmStats()) + def do_getDevicesByCaps(self, args): + caps = list(args) + return self.ExecAndExit(self.s.getDevicesByCaps(caps)) + + def do_getDevicesByDomain(self, args): + vmId = args[0] + return self.ExecAndExit(self.s.getDevicesByDomain(vmId)) + def desktopLogin(self, args): vmId, domain, user, password = tuple(args) response = self.s.desktopLogin(vmId, domain, user, password) @@ -2077,6 +2085,16 @@ ('', 'Get Statistics info for all existing VMs' )), + 'getDevicesByCaps': (serv.do_getDevicesByCaps, + ('[<caps>]', + 'Get available node devices on the host with ' + 'given capability' + )), + 'getDevicesByDomain': (serv.do_getDevicesByDomain, + ('<vmId>', + 'Get available node devices attached to specified ' + 'domain' + )), 'getVGList': (serv.getVGList, ('storageType', 'List of all VGs.' diff --git a/debian/vdsm.install b/debian/vdsm.install index 3af1100..e1d8652 100644 --- a/debian/vdsm.install +++ b/debian/vdsm.install @@ -142,6 +142,7 @@ ./usr/share/vdsm/virt/__init__.py ./usr/share/vdsm/virt/guestagent.py ./usr/share/vdsm/virt/migration.py +./usr/share/vdsm/virt/nodedev.py ./usr/share/vdsm/virt/sampling.py ./usr/share/vdsm/virt/vm.py ./usr/share/vdsm/virt/vmchannels.py diff --git a/vdsm.spec.in b/vdsm.spec.in index 16b7834..0ddacbe 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -939,6 +939,7 @@ %{_datadir}/%{vdsm_name}/virt/__init__.py* %{_datadir}/%{vdsm_name}/virt/guestagent.py* %{_datadir}/%{vdsm_name}/virt/migration.py* +%{_datadir}/%{vdsm_name}/virt/nodedev.py* %{_datadir}/%{vdsm_name}/virt/vmchannels.py* %{_datadir}/%{vdsm_name}/virt/vmstatus.py* %{_datadir}/%{vdsm_name}/virt/vmtune.py* diff --git a/vdsm/API.py b/vdsm/API.py index 6feccf6..e891b80 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1244,6 +1244,18 @@ statsList = hooks.after_get_all_vm_stats(statsList) return {'status': doneCode, 'statsList': statsList} + def getDevicesByCaps(self, caps): + """ + """ + devices = self._cif.nodeDeviceMapper.getDevicesByCaps(caps) + return {'status': doneCode, 'devices': devices} + + def getDevicesByDomain(self, vmId): + """ + """ + devices = self._cif.nodeDeviceMapper.getDevicesByDomain(vmId) + return {'status': doneCode, 'devices': devices} + def getStats(self): """ Report host statistics. diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index d5372f3..118652d 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -91,6 +91,7 @@ self.gluster = None try: self.vmContainer = {} + self.nodeDeviceMapper = NodeDeviceMapper(vmContainer, log) self._hostStats = sampling.HostStatsThread(log=log) self._hostStats.start() self.lastRemoteAccess = 0 diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index d6663c3..a31d76c 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -482,6 +482,14 @@ api = API.Global() return api.getAllVmStats() + def getDevicesByCaps(self, caps): + api = API.Global() + return api.getDevicesByCaps(caps) + + def getDevicesByDomain(self, vmId): + api = API.Global() + return api.getDevicesByDomain(vmId) + def vmGetIoTunePolicy(self, vmId): vm = API.VM(vmId) return vm.getIoTunePolicy() @@ -989,6 +997,8 @@ (self.getStats, 'getVdsStats'), (self.vmGetStats, 'getVmStats'), (self.getAllVmStats, 'getAllVmStats'), + (self.getDevicesByCaps, 'getDevicesByCaps'), + (self.getDevicesByDomain, 'getDevicesByDomain'), (self.vmMigrationCreate, 'migrationCreate'), (self.vmDesktopLogin, 'desktopLogin'), (self.vmDesktopLogoff, 'desktopLogoff'), diff --git a/vdsm/virt/Makefile.am b/vdsm/virt/Makefile.am index 423839b..4b7b1cc 100644 --- a/vdsm/virt/Makefile.am +++ b/vdsm/virt/Makefile.am @@ -25,6 +25,7 @@ __init__.py \ guestagent.py \ migration.py \ + nodedev.py \ sampling.py \ vm.py \ vmchannels.py \ -- To view, visit http://gerrit.ovirt.org/30209 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6f21d465a90cfe2eb16ba70943e5fcf1683f1656 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

6 22

Change in vdsm[master]: [WIP] vdsm: add support for PCI passthrough
by mpoledni＠redhat.com 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: [WIP] vdsm: add support for PCI passthrough ...................................................................... [WIP] vdsm: add support for PCI passthrough required functionality: * report PCI devices available on host [x] * handle createVm xml generation [x] * hotplugHostdev [ ] (required for after-migration) * hotpunlugHostdev [ ] (required for migration) Change-Id: I363d2622d72ca2db75f60032fe0892c348bab121 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M lib/vdsm/define.py M vdsm/caps.py M vdsm/vm.py 3 files changed, 83 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/62/22462/1 diff --git a/lib/vdsm/define.py b/lib/vdsm/define.py index eb78633..9605f93 100644 --- a/lib/vdsm/define.py +++ b/lib/vdsm/define.py @@ -132,6 +132,12 @@ 'transientErr': {'status': { 'code': 59, 'message': 'Action not permitted on a VM with transient disks'}}, + 'hotplugHostdev': {'status': { + 'code': 60, + 'message': 'Failed to hotplug hostdev'}}, + 'hotunplugHostdev': {'status': { + 'code': 61, + 'message': 'Failed to hotunplug hostdev'}}, 'recovery': {'status': { 'code': 99, 'message': 'Recovering from crash or Initializing'}}, diff --git a/vdsm/caps.py b/vdsm/caps.py index 3839134..d6af375 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -308,6 +308,38 @@ return dict(release=release, version=version, name=osname) +def hostdevList(): + devices = [] + for device in libvirtconnection.get().listAllDevices(): + devXML = minidom.parseString(device.XMLDesc()) + dev = {} + + # we have to grab attributes that will most likely not only + # uniquely identify device, but also serve as human readable + # representation of the device + try: + dev['name'] = devXML.getElementsByTagName('name')[0].\ + childNodes[0].data + capability = devXML.getElementsByTagName('capability')[0] + try: + dev['product'] = capability.getElementsByTagName('product')[0]\ + .childNodes[0].data + dev['vendor'] = capability.getElementsByTagName('vendor')[0].\ + childNodes[0].data + except IndexError: + # althought the retrieval of product/vendor was not successful, + # we can still report back the name + pass + except IndexError: + # should device not have a name, there is nothing engine could send + # back that we could use to uniquely identify and initiate a device + continue + + devices.append(dev) + + return devices + + def get(): targetArch = platform.machine() @@ -360,6 +392,7 @@ config.getint('vars', 'extra_mem_reserve')) caps['guestOverhead'] = config.get('vars', 'guest_ram_overhead') caps['rngSources'] = _getRngSources() + caps['hostDevices'] = hostdevList() return caps diff --git a/vdsm/vm.py b/vdsm/vm.py index a5d923b..a477bc9 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -78,6 +78,7 @@ WATCHDOG_DEVICES = 'watchdog' CONSOLE_DEVICES = 'console' SMARTCARD_DEVICES = 'smartcard' +HOSTDEV_DEVICES = 'hostdev' def isVdsmImage(drive): @@ -1656,6 +1657,27 @@ return m +class HostDevice(VmDevice): + def getXML(self): + """ + Create domxml for a hostdev device. + + <devices> + <hostdev mode='subsystem' type='usb'> + <source startupPolicy='optional'> + <vendor id='0x1234'/> + <product id='0xbeef'/> + </source> + <boot order='2'/> + </hostdev> + </devices> + """ + # libvirt gives us direct api call to construct the XML + return xml.dom.minidom.parseString(libvirtconnection.get(). + nodeDeviceLookupByName(self.name). + XMLDesc()) + + class WatchdogDevice(VmDevice): def __init__(self, *args, **kwargs): super(WatchdogDevice, self).__init__(*args, **kwargs) @@ -1769,7 +1791,8 @@ (CONSOLE_DEVICES, ConsoleDevice), (REDIR_DEVICES, RedirDevice), (RNG_DEVICES, RngDevice), - (SMARTCARD_DEVICES, SmartCardDevice)) + (SMARTCARD_DEVICES, SmartCardDevice), + (HOSTDEV_DEVICES, HostDevice)) def _makeDeviceDict(self): return dict((dev, []) for dev, _ in self.DeviceMapping) @@ -3127,6 +3150,26 @@ break + def hotplugHostdev(self, params): + hostdev = HostDevice(self.conf, self.log, **params) + self._devices[HOSTDEV_DEVICES].append(hostdev) + hostdevXML = hostdev.getXML().toprettyxml(encoding='utf-8') + hostdev._deviceXML = hostdevXML + self.log.debug("Hotplug hostdev xml: %s", hostdevXML) + + try: + self._dom.attachDevice(hostdevXML) + except libvirt.libvirtError as e: + self.log.error("Hotplug failed", exc_info=True) + if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: + return errCode['noVM'] + return {'status': {'code': + errCode['hotplugHostdev']['status']['code'], + 'message': e.message}} + + def hotunplugHostdev(self, name): + pass + def hotplugNic(self, params): if self.isMigrating(): return errCode['migInProgress'] -- To view, visit http://gerrit.ovirt.org/22462 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I363d2622d72ca2db75f60032fe0892c348bab121 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpoledni(a)redhat.com>

8 50

Change in vdsm[master]: storage_exception: Show data in MetadataOverflowError
by Nir Soffer 01 Sep '15

01 Sep '15

Nir Soffer has uploaded a new change for review. Change subject: storage_exception: Show data in MetadataOverflowError ...................................................................... storage_exception: Show data in MetadataOverflowError This should help to debug such issues. Previously we raised an error without any details, making it hard to debug such issue. Change-Id: I305584438c391cbe7d5da7e358482f255e292187 Backport-To: 3.6 Relates-To: https://bugzilla.redhat.com/1258097 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/blockSD.py M vdsm/storage/storage_exception.py 2 files changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/45471/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 51814ab..978b85b 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -353,7 +353,7 @@ metaStr.write("\n") if metaStr.pos > self._size: - raise se.MetadataOverflowError() + raise se.MetadataOverflowError(metaStr.getvalue()) # Clear out previous data - it is a volume, not a file metaStr.write('\0' * (self._size - metaStr.pos)) diff --git a/vdsm/storage/storage_exception.py b/vdsm/storage/storage_exception.py index 3a4d33f..1e9c3f8 100644 --- a/vdsm/storage/storage_exception.py +++ b/vdsm/storage/storage_exception.py @@ -1665,6 +1665,9 @@ code = 756 message = "Metadata is too big. Cannot change Metadata" + def __init__(self, data): + self.value = "data=%r" % data + ################################################# # Import/Export Exceptions -- To view, visit https://gerrit.ovirt.org/45471 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I305584438c391cbe7d5da7e358482f255e292187 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 10

Change in vdsm[master]: Refactor v2v jobs for reusability
by alitke＠redhat.com 01 Sep '15

01 Sep '15

Adam Litke has uploaded a new change for review. Change subject: Refactor v2v jobs for reusability ...................................................................... Refactor v2v jobs for reusability Change-Id: Ida6b1c460c5030c820c540e836e423d4632410df Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M lib/vdsm/define.py M tests/Makefile.am A tests/jobsTests.py M vdsm.spec.in M vdsm/Makefile.am A vdsm/jobs.py M vdsm/v2v.py 7 files changed, 379 insertions(+), 212 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/44857/1 diff --git a/lib/vdsm/define.py b/lib/vdsm/define.py index 53f530d..b4e3981 100644 --- a/lib/vdsm/define.py +++ b/lib/vdsm/define.py @@ -151,16 +151,16 @@ 'V2VConnection': {'status': { 'code': 65, 'message': 'error connecting to hypervisor'}}, - 'V2VNoSuchJob': {'status': { + 'NoSuchJob': {'status': { 'code': 66, 'message': 'Job Id does not exists'}}, 'V2VNoSuchOvf': {'status': { 'code': 67, 'message': 'OVF file does not exists'}}, - 'V2VJobNotDone': {'status': { + 'JobNotDone': {'status': { 'code': 68, 'message': 'Job status is not done'}}, - 'V2VJobExists': {'status': { + 'JobExists': {'status': { 'code': 69, 'message': 'Job id already exists'}}, 'hotplugMem': {'status': { diff --git a/tests/Makefile.am b/tests/Makefile.am index 174982c..888a866 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -57,6 +57,7 @@ iproute2Tests.py \ ipwrapperTests.py \ iscsiTests.py \ + jobsTests.py \ libvirtconnectionTests.py \ lvmTests.py \ main.py \ diff --git a/tests/jobsTests.py b/tests/jobsTests.py new file mode 100644 index 0000000..01a28f8 --- /dev/null +++ b/tests/jobsTests.py @@ -0,0 +1,81 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import uuid + +import jobs + +from testlib import VdsmTestCase as TestCaseBase +from testValidation import slowtest + + +class TestingJob(jobs.Job): + PROC_WAIT_TIMEOUT = 1 + + def __init__(self, job_id): + jobs.Job.__init__(self, job_id) + self._progress = 0 + + @property + def progress(self): + return self._progress + + @classmethod + def from_shell(cls, job_id, shell_script): + def _cmd(): + return ['bash', '-c', shell_script] + + obj = cls(job_id) + obj._create_command = _cmd + return obj + + +class JobsTests(TestCaseBase): + TIMEOUT = 1 + + def setUp(self): + self.job_id = str(uuid.uuid4()) + + def test_simple(self): + job = TestingJob.from_shell(self.job_id, 'true') + self.assertEqual(jobs.STATUS.STARTING, job.status) + job.start() + self.assertTrue(job.proc_finished.wait(self.TIMEOUT)) + self.assertEqual(jobs.STATUS.DONE, job.status) + + def test_cmd_fail(self): + job = TestingJob.from_shell(self.job_id, 'false') + job.start() + self.assertTrue(job.proc_finished.wait(self.TIMEOUT)) + self.assertEqual(jobs.STATUS.FAILED, job.status) + + def test_abort(self): + job = TestingJob.from_shell(self.job_id, 'sleep 5') + job.start() + self.assertTrue(job.proc_started.wait(self.TIMEOUT)) + job.abort() + self.assertEqual(jobs.STATUS.ABORTED, job.status) + + @slowtest + def test_timeout(self): + job = TestingJob.from_shell(self.job_id, 'sleep 5') + job.PROC_WAIT_TIMEOUT = 1 + job.start() + self.assertTrue(job.proc_finished.wait(2)) + self.assertEqual(jobs.STATUS.FAILED, job.status) diff --git a/vdsm.spec.in b/vdsm.spec.in index d17338d..2acf32a 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -791,6 +791,7 @@ %{_datadir}/%{vdsm_name}/hooking.py* %{_datadir}/%{vdsm_name}/hooks.py* %{_datadir}/%{vdsm_name}/hostdev.py* +%{_datadir}/%{vdsm_name}/jobs.py* %{_datadir}/%{vdsm_name}/mk_sysprep_floppy %{_datadir}/%{vdsm_name}/parted_utils.py* %{_datadir}/%{vdsm_name}/mkimage.py* diff --git a/vdsm/Makefile.am b/vdsm/Makefile.am index 4c0578e..be72875 100644 --- a/vdsm/Makefile.am +++ b/vdsm/Makefile.am @@ -33,6 +33,7 @@ hooking.py \ hooks.py \ hostdev.py \ + jobs.py \ kaxmlrpclib.py \ logUtils.py \ mkimage.py \ diff --git a/vdsm/jobs.py b/vdsm/jobs.py new file mode 100644 index 0000000..8b7d9fb --- /dev/null +++ b/vdsm/jobs.py @@ -0,0 +1,276 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import errno +import logging +import signal +import threading + +from vdsm.define import errCode, doneCode +from vdsm.infra import zombiereaper +from vdsm.utils import traceback, execCmd + + +_lock = threading.Lock() +_jobs = {} + + +class STATUS: + ''' + STARTING: request granted and starting the process + ABORTED: user initiated aborted + FAILED: error during process + DONE: process successfully finished + ''' + STARTING = 'starting' + ABORTED = 'aborted' + FAILED = 'error' + DONE = 'done' + + +class ClientError(Exception): + ''' Base class for client error ''' + + +class JobExistsError(ClientError): + ''' Job already exists in _jobs collection ''' + err_name = 'JobExistsError' + + +class NoSuchJob(ClientError): + ''' Job not exists in _jobs collection ''' + err_name = 'NoSuchJob' + + +class JobNotDone(ClientError): + ''' Import process still in progress ''' + err_name = 'JobNotDone' + + +class JobError(Exception): + ''' Base class for all internal job errors''' + + +class ProcessError(JobError): + ''' process had error in execution ''' + + +def delete_job(job_id): + try: + job = _get_job(job_id) + _validate_job_finished(job) + _remove_job(job_id) + except ClientError as e: + logging.info('Cannot delete job, error: %s', e) + return errCode[e.err_name] + return {'status': doneCode} + + +def abort_job(job_id): + try: + job = _get_job(job_id) + job.abort() + except ClientError as e: + logging.info('Cannot abort job, error: %s', e) + return errCode[e.err_name] + return {'status': doneCode} + + +def get_jobs_status(): + ret = {} + with _lock: + items = tuple(_jobs.items()) + for job_id, job in items: + ret[job_id] = { + 'status': job.status, + 'description': job.description, + 'progress': job.progress + } + return ret + + +def add_job(job_id, job): + with _lock: + if job_id in _jobs: + raise JobExistsError("Job %r exists" % job_id) + _jobs[job_id] = job + + +def _get_job(job_id): + with _lock: + if job_id not in _jobs: + raise NoSuchJob("No such job %r" % job_id) + return _jobs[job_id] + + +def _remove_job(job_id): + with _lock: + if job_id not in _jobs: + raise NoSuchJob("No such job %r" % job_id) + del _jobs[job_id] + + +def _validate_job_done(job): + if job.status != STATUS.DONE: + raise JobNotDone("Job %r is %s" % (job.id, job.status)) + + +def _validate_job_finished(job): + if job.status not in (STATUS.DONE, STATUS.FAILED, STATUS.ABORTED): + raise JobNotDone("Job %r is %s" % (job.id, job.status)) + + +class Job(object): + TERM_DELAY = 30 + PROC_WAIT_TIMEOUT = 30 + + def __init__(self, job_id): + self._id = job_id + self._status = STATUS.STARTING + self._aborted = False + self._description = '' + + self._create_command = None + + # Override this if you need to perform some setup when starting thread + self._run_command = self._run + + self.proc_started = threading.Event() + self.proc_finished = threading.Event() + + def start(self): + t = threading.Thread(target=self._run_command) + t.daemon = True + t.start() + + @property + def id(self): + return self._id + + @property + def status(self): + return self._status + + @property + def description(self): + return self._description + + @property + def progress(self): + """ + Must be overridden by child class + """ + raise NotImplementedError() + + def abort(self): + self._status = STATUS.ABORTED + logging.info('Job %r aborting...', self._id) + self._abort() + + def _abort(self): + self._aborted = True + if self._proc.returncode is None: + logging.debug('Job %r killing process', self._id) + try: + self._proc.kill() + except OSError as e: + if e.errno != errno.ESRCH: + raise + logging.debug('Job %r process not running', + self._id) + else: + logging.debug('Job %r process was killed', + self._id) + finally: + zombiereaper.autoReapPID(self._proc.pid) + + def _execution_environments(self): + """ + May be overridden by child class to set process environment variables + """ + return {} + + def _prepare(self): + """ + May be overridden by child class to perform any pre-processing + """ + pass + + def _watch_process_output(self): + """ + May be overridden by child class to monitor process output and update + progress information. + """ + pass + + def _cleanup(self): + """ + May be overridden by child class to perform any postprocessing + """ + pass + + @traceback(msg="Job failed") + def _run(self): + try: + self._start() + except Exception as ex: + if self._aborted: + logging.debug("Job %r was aborted", self._id) + else: + logging.exception("Job %r failed", self._id) + self._status = STATUS.FAILED + self._description = ex.message + try: + self._abort() + except Exception as e: + logging.exception('Job %r, error trying to abort: %r', + self._id, e) + finally: + self.proc_finished.set() + self._cleanup() + + def _start(self): + # TODO: use the process handling http://gerrit.ovirt.org/#/c/33909/ + cmd = self._create_command() + logging.info('Job %r starting process', self._id) + + self._proc = execCmd(cmd, sync=False, deathSignal=signal.SIGTERM, + env=self._execution_environments()) + self.proc_started.set() + self._proc.blocking = True + self._watch_process_output() + self._wait_for_process() + + if self._proc.returncode != 0: + raise ProcessError('Job %r process failed exit-code: %r' + ', stderr: %s' % + (self._id, self._proc.returncode, + self._proc.stderr.read(1024))) + + if self._status != STATUS.ABORTED: + self._status = STATUS.DONE + logging.info('Job %r finished import successfully', self._id) + + def _wait_for_process(self): + if self._proc.returncode is not None: + return + logging.debug("Job %r waiting for process", self._id) + if not self._proc.wait(timeout=self.PROC_WAIT_TIMEOUT): + raise ProcessError("Job %r timeout waiting for process pid=%s", + self._id, self._proc.pid) diff --git a/vdsm/v2v.py b/vdsm/v2v.py index d0bcc3a..431d264 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -30,8 +30,6 @@ import logging import os import re -import signal -import threading import xml.etree.ElementTree as ET import libvirt @@ -39,14 +37,10 @@ from vdsm.constants import P_VDSM_RUN from vdsm.define import errCode, doneCode from vdsm import libvirtconnection, response -from vdsm.infra import zombiereaper -from vdsm.utils import traceback, CommandPath, execCmd +from vdsm.utils import CommandPath, execCmd import caps - - -_lock = threading.Lock() -_jobs = {} +import jobs _V2V_DIR = os.path.join(P_VDSM_RUN, 'v2v') _VIRT_V2V = CommandPath('virt-v2v', '/usr/bin/virt-v2v') @@ -65,27 +59,15 @@ DiskProgress = namedtuple('DiskProgress', ['progress']) -class STATUS: +class V2VSTATUS(jobs.STATUS): ''' - STARTING: request granted and starting the import process COPYING_DISK: copying disk in progress - ABORTED: user initiated aborted - FAILED: error during import process - DONE: convert process successfully finished ''' - STARTING = 'starting' COPYING_DISK = 'copying_disk' - ABORTED = 'aborted' - FAILED = 'error' - DONE = 'done' class V2VError(Exception): ''' Base class for v2v errors ''' - - -class ClientError(Exception): - ''' Base class for client error ''' class InvalidVMConfiguration(ValueError): @@ -96,23 +78,8 @@ ''' Error while parsing virt-v2v output ''' -class JobExistsError(ClientError): - ''' Job already exists in _jobs collection ''' - err_name = 'V2VJobExistsError' - - -class VolumeError(ClientError): +class VolumeError(jobs.ClientError): ''' Error preparing volume ''' - - -class NoSuchJob(ClientError): - ''' Job not exists in _jobs collection ''' - err_name = 'V2VNoSuchJob' - - -class JobNotDone(ClientError): - ''' Import process still in progress ''' - err_name = 'V2VJobNotDone' class NoSuchOvf(V2VError): @@ -120,11 +87,7 @@ err_name = 'V2VNoSuchOvf' -class V2VProcessError(V2VError): - ''' virt-v2v process had error in execution ''' - - -class InvalidInputError(ClientError): +class InvalidInputError(jobs.ClientError): ''' Invalid input received ''' @@ -169,14 +132,14 @@ def convert_external_vm(uri, username, password, vminfo, job_id, irs): job = ImportVm.from_libvirt(uri, username, password, vminfo, job_id, irs) job.start() - _add_job(job_id, job) + jobs.add_job(job_id, job) return {'status': doneCode} def convert_ova(ova_path, vminfo, job_id, irs): job = ImportVm.from_ova(ova_path, vminfo, job_id, irs) job.start() - _add_job(job_id, job) + jobs.add_job(job_id, job) return response.success() @@ -198,81 +161,16 @@ def get_converted_vm(job_id): try: - job = _get_job(job_id) - _validate_job_done(job) + job = jobs._get_job(job_id) + jobs._validate_job_done(job) ovf = _read_ovf(job_id) - except ClientError as e: + except jobs.ClientError as e: logging.info('Converted VM error %s', e) return errCode[e.err_name] except V2VError as e: logging.error('Converted VM error %s', e) return errCode[e.err_name] return {'status': doneCode, 'ovf': ovf} - - -def delete_job(job_id): - try: - job = _get_job(job_id) - _validate_job_finished(job) - _remove_job(job_id) - except ClientError as e: - logging.info('Cannot delete job, error: %s', e) - return errCode[e.err_name] - return {'status': doneCode} - - -def abort_job(job_id): - try: - job = _get_job(job_id) - job.abort() - except ClientError as e: - logging.info('Cannot abort job, error: %s', e) - return errCode[e.err_name] - return {'status': doneCode} - - -def get_jobs_status(): - ret = {} - with _lock: - items = tuple(_jobs.items()) - for job_id, job in items: - ret[job_id] = { - 'status': job.status, - 'description': job.description, - 'progress': job.progress - } - return ret - - -def _add_job(job_id, job): - with _lock: - if job_id in _jobs: - raise JobExistsError("Job %r exists" % job_id) - _jobs[job_id] = job - - -def _get_job(job_id): - with _lock: - if job_id not in _jobs: - raise NoSuchJob("No such job %r" % job_id) - return _jobs[job_id] - - -def _remove_job(job_id): - with _lock: - if job_id not in _jobs: - raise NoSuchJob("No such job %r" % job_id) - del _jobs[job_id] - - -def _validate_job_done(job): - if job.status != STATUS.DONE: - raise JobNotDone("Job %r is %s" % (job.id, job.status)) - - -def _validate_job_finished(job): - if job.status not in (STATUS.DONE, STATUS.FAILED, STATUS.ABORTED): - raise JobNotDone("Job %r is %s" % (job.id, job.status)) def _read_ovf(job_id): @@ -311,32 +209,24 @@ job_id, file_name) -class ImportVm(object): - TERM_DELAY = 30 - PROC_WAIT_TIMEOUT = 30 - +class ImportVm(jobs.Job): def __init__(self, vminfo, job_id, irs): ''' do not use directly, use a factory method instead! ''' + jobs.Job.__init__(self, job_id) self._vminfo = vminfo - self._id = job_id self._irs = irs - self._status = STATUS.STARTING - self._description = '' self._disk_progress = 0 self._disk_count = 1 self._current_disk = 1 - self._aborted = False self._prepared_volumes = [] self._uri = None self._username = None self._password = None self._passwd_file = None - self._create_command = None - self._run_command = None self._ova_path = None @@ -361,23 +251,6 @@ obj._run_command = obj._run return obj - def start(self): - t = threading.Thread(target=self._run_command) - t.daemon = True - t.start() - - @property - def id(self): - return self._id - - @property - def status(self): - return self._status - - @property - def description(self): - return self._description - @property def progress(self): ''' @@ -393,47 +266,11 @@ with password_file(self._id, self._passwd_file, self._password): self._run() - @traceback(msg="Error importing vm") - def _run(self): - try: - self._import() - except Exception as ex: - if self._aborted: - logging.debug("Job %r was aborted", self._id) - else: - logging.exception("Job %r failed", self._id) - self._status = STATUS.FAILED - self._description = ex.message - try: - self._abort() - except Exception as e: - logging.exception('Job %r, error trying to abort: %r', - self._id, e) - finally: - self._teardown_volumes() - - def _import(self): - # TODO: use the process handling http://gerrit.ovirt.org/#/c/33909/ + def _prepare(self): self._prepare_volumes() - cmd = self._create_command() - logging.info('Job %r starting import', self._id) - self._proc = execCmd(cmd, sync=False, deathSignal=signal.SIGTERM, - env=self._execution_environments()) - - self._proc.blocking = True - self._watch_process_output() - self._wait_for_process() - - if self._proc.returncode != 0: - raise V2VProcessError('Job %r process failed exit-code: %r' - ', stderr: %s' % - (self._id, self._proc.returncode, - self._proc.stderr.read(1024))) - - if self._status != STATUS.ABORTED: - self._status = STATUS.DONE - logging.info('Job %r finished import successfully', self._id) + def _cleanup(self): + self._teardown_volumes() def _execution_environments(self): env = {'LIBGUESTFS_BACKEND': 'direct'} @@ -441,19 +278,11 @@ env['VIRTIO_WIN'] = self._vminfo['virtio_iso_path'] return env - def _wait_for_process(self): - if self._proc.returncode is not None: - return - logging.debug("Job %r waiting for virt-v2v process", self._id) - if not self._proc.wait(timeout=self.PROC_WAIT_TIMEOUT): - raise V2VProcessError("Job %r timeout waiting for process pid=%s", - self._id, self._proc.pid) - def _watch_process_output(self): parser = OutputParser() for event in parser.parse(self._proc.stdout): if isinstance(event, ImportProgress): - self._status = STATUS.COPYING_DISK + self._status = V2VSTATUS.COPYING_DISK logging.info("Job %r copying disk %d/%d", self._id, event.current_disk, event.disk_count) self._disk_progress = 0 @@ -503,28 +332,6 @@ get_storage_domain_path(self._prepared_volumes[0]['path'])] cmd.extend(self._generate_disk_parameters()) return cmd - - def abort(self): - self._status = STATUS.ABORTED - logging.info('Job %r aborting...', self._id) - self._abort() - - def _abort(self): - self._aborted = True - if self._proc.returncode is None: - logging.debug('Job %r killing virt-v2v process', self._id) - try: - self._proc.kill() - except OSError as e: - if e.errno != errno.ESRCH: - raise - logging.debug('Job %r virt-v2v process not running', - self._id) - else: - logging.debug('Job %r virt-v2v process was killed', - self._id) - finally: - zombiereaper.autoReapPID(self._proc.pid) def _get_disk_format(self): fmt = self._vminfo.get('format', 'raw').lower() -- To view, visit https://gerrit.ovirt.org/44857 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ida6b1c460c5030c820c540e836e423d4632410df Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

8 52

Change in vdsm[master]: schema: add v2vJobs to HostStats
by shavivi＠redhat.com 01 Sep '15

01 Sep '15

Shahar Havivi has uploaded a new change for review. Change subject: schema: add v2vJobs to HostStats ...................................................................... schema: add v2vJobs to HostStats v2vJobs reports via HostStats and needs to be declared in schema Change-Id: I20b5b44b2b74c0e1326251d6a3b84ef66dac46c7 Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M vdsm/rpc/vdsmapi-schema.json 1 file changed, 23 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/44919/1 diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 4cac5ae..f21d174 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -1970,6 +1970,24 @@ 'score': 'uint'}} ## +# @V2VJobs: +# +# Structure for current v2v jobs status +# +# @status: Indicates jobs status (done, error, aborted, copying_disk, +# starting) +# +# @description: Description for the current status such as error or copying +# disk description. +# +# @progress: Job progress between 0-100 +# +# Since: 4.17.0 +## +{'type': 'V2VJobs', + 'data': {'status': 'str', 'description': 'str', 'progress': 'uint'}} + +## # @HostStats: # # Statistics about this host. @@ -2065,6 +2083,9 @@ # @outgoingVmMigrations: The number of VMs migrating away from this host # (new in version 4.17.0) # +# @v2vJobs: #optional Current v2v jobs that running/ended on +# current host (new in version 4.17.0) +# # Since: 4.10.0 ## {'type': 'HostStats', @@ -2086,7 +2107,8 @@ 'haStatus': 'HostedEngineStatus', '*bootTime': 'uint', 'numaNodeMemFree': 'NumaNodeMemoryStatsMap', 'cpuStatistics': 'CpuCoreStatsMap', - 'incomingVmMigrations': 'uint', 'outgoingVmMigrations': 'uint'}} + 'incomingVmMigrations': 'uint', 'outgoingVmMigrations': 'uint', + '*v2vJobs': 'V2VJobs'}} ## # @Host.getStats: -- To view, visit https://gerrit.ovirt.org/44919 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I20b5b44b2b74c0e1326251d6a3b84ef66dac46c7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

4 10

Change in vdsm[master]: v2v: add test for getOvaInfo verb
by shavivi＠redhat.com 01 Sep '15

01 Sep '15

Shahar Havivi has uploaded a new change for review. Change subject: v2v: add test for getOvaInfo verb ...................................................................... v2v: add test for getOvaInfo verb Change-Id: I398943c356a87cf93a3e0557f4d78da0490814c5 Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M tests/v2vTests.py 1 file changed, 52 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/44471/1 diff --git a/tests/v2vTests.py b/tests/v2vTests.py index 520de61..20db6ee 100644 --- a/tests/v2vTests.py +++ b/tests/v2vTests.py @@ -90,6 +90,38 @@ def hypervisorConnect(uri, username, passwd): return LibvirtMock() +def read_ovf(ovf_path): + return """<?xml version="1.0" encoding="UTF-8"?> +<Envelope xmlns="http://schemas.dmtf.org/ovf/envelope/1" + xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" + xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSet…"> + <References> + <File ovf:href="First-disk1.vmdk" ovf:id="file1" ovf:size="349405696"/> + </References> + <DiskSection> + <Disk ovf:capacity="32" ovf:fileRef="file1"/> + </DiskSection> + <VirtualSystem ovf:id="First"> + <Name>First</Name> + <VirtualHardwareSection> + <Item> + <rasd:ResourceType>4</rasd:ResourceType> + <rasd:VirtualQuantity>2048</rasd:VirtualQuantity> + </Item> + <Item> + <rasd:ResourceType>3</rasd:ResourceType> + <rasd:VirtualQuantity>1</rasd:VirtualQuantity> + </Item> + <Item> + <rasd:Connection>VM Network</rasd:Connection> + <rasd:ElementName>Ethernet 1</rasd:ElementName> + <rasd:ResourceSubType>E1000</rasd:ResourceSubType> + <rasd:ResourceType>10</rasd:ResourceType> + </Item> + </VirtualHardwareSection> + </VirtualSystem> +</Envelope>""" + class v2vTests(TestCaseBase): @MonkeyPatch(libvirtconnection, 'open_connection', hypervisorConnect) @@ -139,3 +171,23 @@ (v2v.DiskProgress(0)), (v2v.DiskProgress(50)), (v2v.DiskProgress(100))]) + + @MonkeyPatch(v2v, '_read_ovf_from_ova', read_ovf) + def testGetOvaInfo(self): + ret = v2v.get_ova_info("dummy") + vm = ret['vmList'][0] + self.assertEquals(vm['vmName'], 'First') + self.assertEquals(vm['memSize'], 2048) + self.assertEquals(vm['smp'], 1) + + disk = vm['disks'][0] + self.assertEquals(disk['allocation'], '349405696') + self.assertEquals(disk['capacity'], '34359738368') + self.assertEquals(disk['type'], 'disk') + self.assertEquals(disk['alias'], 'First-disk1.vmdk') + + network = vm['networks'][0] + self.assertEquals(network['bridge'], 'VM Network') + self.assertEquals(network['model'], 'E1000') + self.assertEquals(network['type'], 'bridge') + self.assertEquals(network['dev'], 'Ethernet 1') -- To view, visit https://gerrit.ovirt.org/44471 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I398943c356a87cf93a3e0557f4d78da0490814c5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

4 16

Change in vdsm[master]: periodic: explicitely track domain availability
by automation＠ovirt.org 01 Sep '15

01 Sep '15

automation(a)ovirt.org has posted comments on this change. Change subject: periodic: explicitely track domain availability ...................................................................... Patch Set 15: * Update tracker::#1250839::OK * Check Bug-Url::OK * Check Public Bug::#1250839::OK, public bug * Check Product::#1250839::OK, Correct product oVirt * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44544 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I1b1d5173bac8e288474581092b8132dc0df03ad4 Gerrit-PatchSet: 15 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches September 2015