vdsm-patches March 2015

vdsm-patches@lists.fedorahosted.org

57 participants
742 discussions

Change in vdsm[master]: clientIF: Remove unnecessary device is disk check
by sgotliv＠redhat.com 19 Oct '15

19 Oct '15

Sergey Gotliv has uploaded a new change for review. Change subject: clientIF: Remove unnecessary device is disk check ...................................................................... clientIF: Remove unnecessary device is disk check Change-Id: I98317e805e6770df5dacd3237a383aaca78fde1e Signed-off-by: Sergey Gotliv <sgotliv(a)redhat.com> --- M vdsm/clientIF.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/63/22363/1 diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index c083991..124f8e5 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -244,7 +244,7 @@ def prepareVolumePath(self, drive, vmId=None): if type(drive) is dict: # PDIV drive format - if drive['device'] == 'disk' and vm.isVdsmImage(drive): + if vm.isVdsmImage(drive): res = self.irs.prepareImage( drive['domainID'], drive['poolID'], drive['imageID'], drive['volumeID']) -- To view, visit http://gerrit.ovirt.org/22363 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I98317e805e6770df5dacd3237a383aaca78fde1e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Sergey Gotliv <sgotliv(a)redhat.com>

7 23

Change in vdsm[master]: testSetupNetworksAddDelDhcp: Check that an DHCPv4 address is...
by osvoboda＠redhat.com 19 Oct '15

19 Oct '15

Ondřej Svoboda has uploaded a new change for review. Change subject: testSetupNetworksAddDelDhcp: Check that an DHCPv4 address is in range ...................................................................... testSetupNetworksAddDelDhcp: Check that an DHCPv4 address is in range (At least) one address is now required to be in a range we know and control. Also introduce range checking for IPv6. Change-Id: Ied7e11e84b8f81c39c799fadfca06b7bde0a409c Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 26 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/86/36186/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index f23f033..c09e122 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -22,6 +22,8 @@ import json import signal import netaddr +import socket +import struct from hookValidation import ValidatesHook from network.sourceroute import StaticSourceRoute @@ -217,6 +219,20 @@ del attrs['d'] +# adapted from http://stackoverflow.com/questions/10558441/ +# inet-aton-similar-function-for-ipv6 +def _ipv4_to_int(addr): + addr = addr.split('/')[0] # remove the network prefix + return struct.unpack('>L', socket.inet_aton(addr))[0] + + +def _ipv6_to_int(addr): + addr = addr.split('/')[0] # remove the network prefix + addr = socket.inet_pton(socket.AF_INET6, addr) + hi, lo = struct.unpack(">QQ", addr) + return (hi << 64) | lo + + @expandPermutations class NetworkTest(TestCaseBase): @@ -367,6 +383,15 @@ def assertMtu(self, mtu, *elems): for elem in elems: self.assertEquals(int(mtu), int(self.vdsm_net.getMtu(elem))) + + def assertAddrInRange(self, addrs, family=4, + range_from=DHCP_RANGE_FROM, range_to=DHCP_RANGE_TO): + addr_to_int = _ipv4_to_int if family == 4 else _ipv6_to_int + range_from = addr_to_int(range_from) + range_to = addr_to_int(range_to) + passes = (range_from <= addr_to_int(a) <= range_to for a in addrs) + self.assertTrue(any(passes), 'no address {0} in expected range'.format( + addrs)) def testLegacyBonds(self): if not (caps.getos() in (caps.OSName.RHEVH, caps.OSName.RHEL) @@ -1773,6 +1798,7 @@ device_name = right if dhcpv4: + self.assertAddrInRange(test_net['ipv4addrs']) # TODO: source routing not ready for IPv6 ip_addr = test_net['addr'] self.assertSourceRoutingConfiguration(device_name, ip_addr) -- To view, visit http://gerrit.ovirt.org/36186 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ied7e11e84b8f81c39c799fadfca06b7bde0a409c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

4 15

Change in vdsm[master]: Network: Flush all configurators before restoring networks
by ibarkan＠redhat.com 18 Oct '15

18 Oct '15

Ido Barkan has uploaded a new change for review. Change subject: Network: Flush all configurators before restoring networks ...................................................................... Network: Flush all configurators before restoring networks Right now, when there are many networks network service takes time to start up (think DHCP). After it is up, VDSM removes all networks and configures it's own persistent networks, which can also take quite some time. This patch teaches 'vdsm-restore-net-config' to conditionally flush the network configuration and splits this behaviour to a unit that systemd will start before the network service. systemd will now manage the following chain of tasks: 1. vdsm-network-cleanup.service will cleanup system files that configurators may have written on boot before network configuration daemons run, so no time is lost by them configuring vdsm networks (and conflict avoidance). 2. after libvirt is up, reconfigure the persistent network configuration of vdsm using the configured net_configurator. 3. start vdsm and super-vdsm Note that it would be possible with this change to remove from vdsmd.service the after vdsm-network.service so that vdsm would start in parallel to the networks being set up. This could be quite useful so that the engine could communicate with vdsm as soon as the management network is restored. However, for now, the engine would see the host unsynched and we have to think carefully if that is okay and how long that would take. Change-Id: I9e3d0fb35d608a8b019ad80f7e40f4399e7db479 Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M .gitignore M init/systemd/Makefile.am A init/systemd/vdsm-network-cleanup.service.in.py M init/systemd/vdsm-network.service.in M init/sysvinit/vdsmd.init.in M lib/vdsm/tool/restore_nets.py M vdsm.spec.in M vdsm/network/configurators/__init__.py M vdsm/vdsm-restore-net-config 9 files changed, 56 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/67/38767/1 diff --git a/.gitignore b/.gitignore index 54cbc46..4245b44 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,7 @@ init/systemd/systemd-vdsmd init/systemd/vdsm-tmpfiles.d.conf init/systemd/vdsmd.service +init/systemd/vdsm-network-cleanup.service init/systemd/vdsm-network.service init/sysvinit/supervdsmd.init init/sysvinit/vdsmd.init diff --git a/init/systemd/Makefile.am b/init/systemd/Makefile.am index 07c4ae8..000e267 100644 --- a/init/systemd/Makefile.am +++ b/init/systemd/Makefile.am @@ -37,6 +37,7 @@ supervdsmd.service.in \ systemd-vdsmd.in \ vdsmd.service.in \ + vdsm-network-cleanup.service.in \ vdsm-network.service.in \ vdsm-tmpfiles.d.conf.in \ $(NULL) diff --git a/init/systemd/vdsm-network-cleanup.service.in.py b/init/systemd/vdsm-network-cleanup.service.in.py new file mode 100644 index 0000000..b2c8dab --- /dev/null +++ b/init/systemd/vdsm-network-cleanup.service.in.py @@ -0,0 +1,13 @@ +[Unit] +Description=Virtual Desktop Server Manager network system files cleanup +Wants=network.target +Before=network.target network.service NetworkManager.service systemd-networkd.service + +[Service] +Type=oneshot +EnvironmentFile=-/etc/sysconfig/vdsm +ExecStart=@VDSMDIR@/vdsm-restore-net-config flush +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/init/systemd/vdsm-network.service.in b/init/systemd/vdsm-network.service.in index 9eb3e5e..9570ef5 100644 --- a/init/systemd/vdsm-network.service.in +++ b/init/systemd/vdsm-network.service.in @@ -1,7 +1,7 @@ [Unit] Description=Virtual Desktop Server Manager network restoration Wants=network.target -Requires=libvirtd.service +Requires=vdsm-network-cleanup.service libvirtd.service After=libvirtd.service [Service] @@ -9,7 +9,7 @@ EnvironmentFile=-/etc/sysconfig/vdsm ExecStartPre=@BINDIR@/vdsm-tool --vvverbose --append --logfile=@VDSMLOGDIR@/upgrade.log upgrade-unified-persistence ExecStartPre=@BINDIR@/vdsm-tool --vvverbose --append --logfile=@VDSMLOGDIR@/upgrade.log upgrade-3.0.0-networks -ExecStart=@VDSMDIR@/vdsm-restore-net-config +ExecStart=@VDSMDIR@/vdsm-restore-net-config restore --no-flush RemainAfterExit=yes [Install] diff --git a/init/sysvinit/vdsmd.init.in b/init/sysvinit/vdsmd.init.in index 1f8dc64..409ee48 100755 --- a/init/sysvinit/vdsmd.init.in +++ b/init/sysvinit/vdsmd.init.in @@ -142,7 +142,7 @@ } restore_nets(){ - "@PYTHON@" "@VDSMDIR@/vdsm-restore-net-config" || return 1 + "@PYTHON@" "@VDSMDIR@/vdsm-restore-net-config" restore || return 1 return 0 } diff --git a/lib/vdsm/tool/restore_nets.py b/lib/vdsm/tool/restore_nets.py index 99ea9f2..861bf01 100644 --- a/lib/vdsm/tool/restore_nets.py +++ b/lib/vdsm/tool/restore_nets.py @@ -38,7 +38,7 @@ def restore(): rc, out, err = utils.execCmd( - [os.path.join(P_VDSM, 'vdsm-restore-net-config')], raw=True) + [os.path.join(P_VDSM, 'vdsm-restore-net-config'), 'restore'], raw=True) sys.stdout.write(out) sys.stderr.write(err) if rc != 0: diff --git a/vdsm.spec.in b/vdsm.spec.in index 819178c..e369b33 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -762,6 +762,7 @@ %if 0%{?with_systemd} install -Dm 0755 init/systemd/systemd-vdsmd %{buildroot}/usr/lib/systemd/systemd-vdsmd install -Dm 0644 init/systemd/vdsmd.service %{buildroot}%{_unitdir}/vdsmd.service +install -Dm 0644 init/systemd/vdsm-network-cleanup.service %{buildroot}%{_unitdir}/vdsm-network-cleanup.service install -Dm 0644 init/systemd/vdsm-network.service %{buildroot}%{_unitdir}/vdsm-network.service install -Dm 0644 init/systemd/supervdsmd.service %{buildroot}%{_unitdir}/supervdsmd.service @@ -868,6 +869,7 @@ /bin/systemctl restart systemd-modules-load.service >/dev/null 2>&1 || : if [ "$1" -eq 1 ] ; then /bin/systemctl enable vdsmd.service >/dev/null 2>&1 || : + /bin/systemctl enable vdsm-network-cleanup.service >/dev/null 2>&1 || : /bin/systemctl enable vdsm-network.service >/dev/null 2>&1 || : /bin/systemctl enable supervdsmd.service >/dev/null 2>&1 || : fi @@ -905,6 +907,7 @@ %if 0%{?with_systemd} %systemd_preun vdsmd.service %systemd_preun vdsm-network.service +%systemd_preun vdsm-network-cleanup.service %systemd_preun supervdsmd.service %else if [ "$1" -eq 0 ]; then @@ -912,6 +915,7 @@ /bin/systemctl --no-reload disable supervdsmd.service > /dev/null 2>&1 || : /bin/systemctl stop vdsmd.service > /dev/null 2>&1 || : /bin/systemctl stop vdsm-network.service > /dev/null 2>&1 || : + /bin/systemctl stop vdsm-network-cleanup.service > /dev/null 2>&1 || : /bin/systemctl stop supervdsmd.service > /dev/null 2>&1 || : fi exit 0 @@ -1015,6 +1019,7 @@ %if 0%{?with_systemd} /usr/lib/systemd/systemd-vdsmd %{_unitdir}/vdsmd.service +%{_unitdir}/vdsm-network-cleanup.service %{_unitdir}/vdsm-network.service %{_unitdir}/supervdsmd.service %else diff --git a/vdsm/network/configurators/__init__.py b/vdsm/network/configurators/__init__.py index 39ad9a6..ce6ce05 100644 --- a/vdsm/network/configurators/__init__.py +++ b/vdsm/network/configurators/__init__.py @@ -70,7 +70,7 @@ return RunningConfig().diffFrom(self.runningConfig) def flush(self): - libvirt.flush() + pass def configureBridge(self, bridge, **opts): raise NotImplementedError diff --git a/vdsm/vdsm-restore-net-config b/vdsm/vdsm-restore-net-config index 8ffb880..bcd3924 100755 --- a/vdsm/vdsm-restore-net-config +++ b/vdsm/vdsm-restore-net-config @@ -28,6 +28,7 @@ from vdsm.config import config from vdsm import netinfo from vdsm.constants import P_VDSM_RUN +from vdsm.tool import unified_persistence, upgrade # Ifcfg persistence restoration from network.configurators import ifcfg @@ -35,6 +36,7 @@ # Unified persistence restoration from network.api import setupNetworks from network import configurators +from network.configurators import libvirt from vdsm.netconfpersistence import RunningConfig, PersistentConfig import pkgutil @@ -46,7 +48,7 @@ configWriter.restorePersistentBackup() -def unified_restoration(): +def unified_restoration(flush_configurators): """ Builds a setupNetworks command from the persistent configuration to set it as running configuration. @@ -65,8 +67,9 @@ # Flush vdsm configurations left-overs from any configurator on the system # so that changes of configurator and persistence system are smooth. - for configurator_cls in _get_all_configurators(): - configurator_cls().flush() + if flush_configurators: + _configurator_flush() + libvirt.flush() # Restore non-VDSM network devices (BZ#1188251) configWriter = ifcfg.ConfigWriter() @@ -143,17 +146,27 @@ os.utime(file_path, None) -def restore(force_restore=False): +def restore(force_restore=False, flush_configurators=True): if not force_restore and _nets_already_restored(_NETS_RESTORED_MARK): logging.info('networks already restored. doing nothing.') return if config.get('vars', 'net_persistence') == 'unified': - unified_restoration() + unified_restoration(flush_configurators) else: ifcfg_restoration() touch_file(_NETS_RESTORED_MARK) + + +def _configurator_flush(): + """Flush vdsm configurations left-overs from any configurator on the system + so that changes of configurator and persistence system are smooth. Only + done if upgrade to unified persistence is not pending""" + if (not upgrade._upgrade_needed( + unified_persistence.UpgradeUnifiedPersistence)): + for configurator_cls in _get_all_configurators(): + configurator_cls().flush() if __name__ == '__main__': @@ -166,14 +179,25 @@ level=logging.DEBUG) logging.error('Could not init proper logging', exc_info=True) - parser = optparse.OptionParser("usage: %prog [options] nets_restored_mark") + parser = optparse.OptionParser("usage: %prog restore|flush [options]") force_option_help = "the restore action first tests for an existence of " \ "a mark that is made after the last successful " \ "restore action. Unless this option is used, " \ "restore will be a no-op if this mark exists." parser.add_option('--force', action='store_true', default=False, help=force_option_help) + parser.add_option('--no-flush', dest='flush_configurators', default=True, + action='store_false') options, args = parser.parse_args() - restore(options.force) + if len(args) != 1 or args[0] not in ['restore', 'flush']: + parser.error("Wrong number of arguments. only restore or flush are " + "allowed") + arg, = args + if arg == 'restore': + restore(options.force, options.flush_configurators) + elif arg == 'flush': + if not options.flush_configurators: + parser.error('flush cannot be used with --no-flush') + _configurator_flush() -- To view, visit https://gerrit.ovirt.org/38767 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9e3d0fb35d608a8b019ad80f7e40f4399e7db479 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 22

Change in vdsm[master]: Added gluster tag support in getAllTasks()
by barumuga＠redhat.com 14 Oct '15

14 Oct '15

Hello Ayal Baron, Timothy Asir, Saggi Mizrahi, Federico Simoncelli, Dan Kenigsberg, I'd like you to do a code review. Please visit http://gerrit.ovirt.org/7579 to review the following change. Change subject: Added gluster tag support in getAllTasks() ...................................................................... Added gluster tag support in getAllTasks() If param tag is empty, all tasks including gluster tasks are returned, else tasks those tags are in param tag list are returned. As below verbs are not consumed by engine/RHS-C yet, its OK to differ in compatibility issue now. glusterVolumeRebalanceStart glusterVolumeRebalanceStatus glusterVolumeReplaceBrickStart glusterVolumeReplaceBrickStatus glusterVolumeRemoveBrickStart glusterVolumeRemoveBrickStatus Change-Id: I9c765cbfebb5ba22f0d21efa04c824ea4daf6432 Signed-off-by: Bala.FA <barumuga(a)redhat.com> --- M tests/gluster_cli_tests.py M vdsm/gluster/cli.py M vdsm/gluster/exception.py M vdsm/storage/taskManager.py 4 files changed, 367 insertions(+), 95 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/7579/1 diff --git a/tests/gluster_cli_tests.py b/tests/gluster_cli_tests.py index f442893..9c6357c 100644 --- a/tests/gluster_cli_tests.py +++ b/tests/gluster_cli_tests.py @@ -28,6 +28,7 @@ from gluster import cli as gcli except ImportError: pass +import xml.etree.cElementTree as etree class GlusterCliTests(TestCaseBase): @@ -115,3 +116,74 @@ def test_parsePeerStatus(self): self._parsePeerStatus_empty_test() self._parsePeerStatus_test() + + def _parseVolumeStatusAll_test(self): + out = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?> +<cliOutput> + <opRet>0</opRet> + <opErrno>0</opErrno> + <opErrstr></opErrstr> + <volumes> + <volume> + <name>V1</name> + <id>03eace73-9197-49d0-a877-831bc6e9dac2</id> + <tasks> + <task> + <name>rebalance</name> + <id>12345473-9197-49d0-a877-831bc6e9dac2</id> + </task> + </tasks> + </volume> + <volume> + <name>V2</name> + <id>03eace73-1237-49d0-a877-831bc6e9dac2</id> + <tasks> + <task> + <name>replace-brick</name> + <id>12345473-1237-49d0-a877-831bc6e9dac2</id> + <sourceBrick>192.168.122.167:/tmp/V2-b1</sourceBrick> + <destBrick>192.168.122.168:/tmp/V2-b1</destBrick> + </task> + </tasks> + </volume> + <volume> + <name>V3</name> + <id>03eace73-1237-1230-a877-831bc6e9dac2</id> + <tasks> + <task> + <name>remove-brick</name> + <id>12345473-1237-1230-a877-831bc6e9dac2</id> + <BrickCount>2</BrickCount> + <brick>192.168.122.167:/tmp/V3-b1</brick> + <brick>192.168.122.168:/tmp/V3-b1</brick> + </task> + </tasks> + </volume> + </volumes> +</cliOutput>""" + tree = etree.fromstring(out) + status = gcli._parseVolumeStatusAll(tree) + self.assertEquals(status, + {'12345473-1237-1230-a877-831bc6e9dac2': + {'bricks': ['192.168.122.167:/tmp/V3-b1', + '192.168.122.168:/tmp/V3-b1'], + 'taskType': 'remove-brick', + 'volumeId': + '03eace73-1237-1230-a877-831bc6e9dac2', + 'volumeName': 'V3'}, + '12345473-1237-49d0-a877-831bc6e9dac2': + {'bricks': ['192.168.122.167:/tmp/V2-b1', + '192.168.122.168:/tmp/V2-b1'], + 'taskType': 'replace-brick', + 'volumeId': + '03eace73-1237-49d0-a877-831bc6e9dac2', + 'volumeName': 'V2'}, + '12345473-9197-49d0-a877-831bc6e9dac2': + {'bricks': [], + 'taskType': 'rebalance', + 'volumeId': + '03eace73-9197-49d0-a877-831bc6e9dac2', + 'volumeName': 'V1'}}) + + def test_parseVolumeStatusAll(self): + self._parseVolumeStatusAll_test() diff --git a/vdsm/gluster/cli.py b/vdsm/gluster/cli.py index 95de106..1f464f6 100644 --- a/vdsm/gluster/cli.py +++ b/vdsm/gluster/cli.py @@ -84,6 +84,55 @@ raise ge.GlusterCmdFailedException(rc=rv, err=[msg]) +class TaskType: + REBALANCE = 'rebalance' + REPLACE_BRICK = 'replace-brick' + REMOVE_BRICK = 'remove-brick' + + +def _parseVolumeStatusAll(tree): + """ + returns {TaskId: {'volumeName': VolumeName, + 'volumeId': VolumeId, + 'taskType': TaskType, + 'bricks': BrickList}, ...} + """ + tasks = {} + for el in tree.findall('volumes/volume'): + volumeName = el.find('name').text + volumeId = el.find('id').text + for c in el.findall('tasks/task'): + taskType = c.find('name').text + taskId = c.find('id').text + bricks = [] + if taskType == TaskType.REPLACE_BRICK: + bricks.append(c.find('sourceBrick').text) + bricks.append(c.find('destBrick').text) + elif taskType == TaskType.REMOVE_BRICK: + for b in c.findall('brick'): + bricks.append(b.text) + elif taskType == TaskType.REBALANCE: + pass + tasks[taskId] = {'volumeName': volumeName, + 'volumeId': volumeId, + 'taskType': taskType, + 'bricks': bricks} + return tasks + + +@exportToSuperVdsm +def volumeStatusAll(): + command = _getGlusterVolCmd() + ["status", "all"] + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeStatusAllFailedException(rc=e.rc, err=e.err) + try: + return _parseVolumeStatusAll(xmltree) + except: + raise ge.GlusterXmlErrorException(err=out) + + def _parseVolumeInfo(out): if not out[0].strip(): del out[0] @@ -300,11 +349,15 @@ command.append("start") if force: command.append("force") - rc, out, err = _execGluster(command) - if rc: - raise ge.GlusterVolumeRebalanceStartFailedException(rc, out, err) - else: - return True + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRebalanceStartFailedException(rc=e.rc, + err=e.err) + try: + return {'taskId': xmltree.find('id').text} + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm @@ -312,84 +365,147 @@ command = _getGlusterVolCmd() + ["rebalance", volumeName, "stop"] if force: command.append('force') - rc, out, err = _execGluster(command) - if rc: - raise ge.GlusterVolumeRebalanceStopFailedException(rc, out, err) - else: + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRebalanceStopFailedException(rc=e.rc, + err=e.err) + + +class TaskStatus(): + RUNNING = 'RUNNING' + FAILED = 'FAILED' + COMPLETED = 'COMPLETED' + + +def _parseVolumeRebalanceRemoveBrickStatus(xmltree, mode): + """ + returns {'taskId': UUID, + 'host': [{'name': NAME, + 'id': HOSTID, + 'filesScanned': INT, + 'filesMoved': INT, + 'filesFailed': INT, + 'totalSizeMoved': INT, + 'status': TaskStatus},...] + 'summary': {'filesScanned': INT, + 'filesMoved': INT, + 'filesFailed': INT, + 'totalSizeMoved': INT, + 'status': TaskStatus}} + """ + if mode == 'rebalance': + tree = xmltree.find('volRebalance') + elif mode == 'remove-brick': + tree = xmltree.find('volRemoveBrick') + else: + return + status = \ + {'taskId': tree.find('id').text, + 'summary': \ + {'filesScanned': int(tree.find('summary/filesScanned').text), + 'filesMoved': int(tree.find('summary/filesMoved').text), + 'filesFailed': int(tree.find('summary/filesFailed').text), + 'totalSizeMoved': int(tree.find('summary/totalSizeMoved').text), + 'status': tree.find('summary/status').text}, + 'host': []} + for el in tree.findall('node'): + status['host'].append({'name': el.find('name').text, + 'id': el.find('id').text, + 'filesScanned': + int(el.find('filesScanned').text), + 'filesMoved': int(el.find('filesMoved').text), + 'filesFailed': int(el.find('filesFailed').text), + 'totalSizeMoved': + int(el.find('totalSizeMoved').text), + 'status': el.find('status').text}) + return status + + +def _parseVolumeRebalanceStatus(tree): + return _parseVolumeRebalanceRemoveBrickStatus(tree, 'rebalance') @exportToSuperVdsm def volumeRebalanceStatus(volumeName): - rc, out, err = _execGluster(_getGlusterVolCmd() + ["rebalance", volumeName, - "status"]) - if rc: - raise ge.GlusterVolumeRebalanceStatusFailedException(rc, out, err) - if 'in progress' in out[0]: - return BrickStatus.RUNNING, "\n".join(out) - elif 'complete' in out[0]: - return BrickStatus.COMPLETED, "\n".join(out) - else: - return BrickStatus.UNKNOWN, "\n".join(out) + command = _getGlusterVolCmd() + ["rebalance", volumeName, "status"] + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRebalanceStatusFailedException(rc=e.rc, + err=e.err) + try: + return _parseVolumeRebalanceStatus(xmltree) + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm def volumeReplaceBrickStart(volumeName, existingBrick, newBrick): - rc, out, err = _execGluster(_getGlusterVolCmd() + ["replace-brick", - volumeName, - existingBrick, newBrick, - "start"]) - if rc: - raise ge.GlusterVolumeReplaceBrickStartFailedException(rc, out, err) - else: - return True + command = _getGlusterVolCmd() + ["replace-brick", volumeName, + existingBrick, newBrick, "start"] + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeReplaceBrickStartFailedException(rc=e.rc, + err=e.err) + try: + return {'taskId': xmltree.find('id').text} + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm def volumeReplaceBrickAbort(volumeName, existingBrick, newBrick): - rc, out, err = _execGluster(_getGlusterVolCmd() + ["replace-brick", - volumeName, - existingBrick, newBrick, - "abort"]) - if rc: - raise ge.GlusterVolumeReplaceBrickAbortFailedException(rc, out, err) - else: + command = _getGlusterVolCmd() + ["replace-brick", volumeName, + existingBrick, newBrick, "abort"] + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeReplaceBrickAbortFailedException(rc=e.rc, + err=e.err) @exportToSuperVdsm def volumeReplaceBrickPause(volumeName, existingBrick, newBrick): - rc, out, err = _execGluster(_getGlusterVolCmd() + ["replace-brick", - volumeName, - existingBrick, newBrick, - "pause"]) - if rc: - raise ge.GlusterVolumeReplaceBrickPauseFailedException(rc, out, err) - else: + command = _getGlusterVolCmd() + ["replace-brick", volumeName, + existingBrick, newBrick, "pause"] + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeReplaceBrickPauseFailedException(rc=e.rc, + err=e.err) + + +def _parseVolumeReplaceBrickStatus(tree): + """ + returns {'taskId': UUID, + 'filesMoved': INT, + 'movingFile': STRING, + 'status': TaskStatus}} + """ + return {'taskId': tree.find('volReplaceBrick/id').text, + 'filesMoved': int(tree.find('volReplaceBrick/filesMoved').text), + 'movingFile': tree.find('volReplaceBrick/movingFile').text, + 'status': tree.find('volReplaceBrick/status').text} @exportToSuperVdsm def volumeReplaceBrickStatus(volumeName, existingBrick, newBrick): - rc, out, err = _execGluster(_getGlusterVolCmd() + ["replace-brick", - volumeName, - existingBrick, newBrick, - "status"]) - if rc: - raise ge.GlusterVolumeReplaceBrickStatusFailedException(rc, out, - err) - message = "\n".join(out) - statLine = out[0].strip().upper() - if BrickStatus.PAUSED in statLine: - return BrickStatus.PAUSED, message - elif statLine.endswith('MIGRATION COMPLETE'): - return BrickStatus.COMPLETED, message - elif statLine.startswith('NUMBER OF FILES MIGRATED'): - return BrickStatus.RUNNING, message - elif statLine.endswith("UNKNOWN"): - return BrickStatus.UNKNOWN, message - else: - return BrickStatus.NA, message + command = _getGlusterVolCmd() + ["replace-brick", volumeName, + existingBrick, newBrick, "status"] + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeReplaceBrickStatusFailedException(rc=e.rc, + err=e.err) + try: + return _parseVolumeReplaceBrickStatus(xmltree) + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm @@ -399,12 +515,12 @@ existingBrick, newBrick, "commit"] if force: command.append('force') - rc, out, err = _execGluster(command) - if rc: - raise ge.GlusterVolumeReplaceBrickCommitFailedException(rc, out, - err) - else: + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeReplaceBrickCommitFailedException(rc=e.rc, + err=e.err) @exportToSuperVdsm @@ -413,12 +529,15 @@ if replicaCount: command += ["replica", "%s" % replicaCount] command += brickList + ["start"] - - rc, out, err = _execGluster(command) - if rc: - raise ge.GlusterVolumeRemoveBrickStartFailedException(rc, out, err) - else: - return True + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRemoveBrickStartFailedException(rc=e.rc, + err=e.err) + try: + return {'taskId': xmltree.find('id').text} + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm @@ -427,12 +546,16 @@ if replicaCount: command += ["replica", "%s" % replicaCount] command += brickList + ["stop"] - rc, out, err = _execGluster(command) - - if rc: - raise ge.GlusterVolumeRemoveBrickStopFailedException(rc, out, err) - else: + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRemoveBrickStopFailedException(rc=e.rc, + err=e.err) + + +def _parseVolumeRemoveBrickStatus(tree): + return _parseVolumeRebalanceRemoveBrickStatus(tree, 'remove-brick') @exportToSuperVdsm @@ -441,12 +564,15 @@ if replicaCount: command += ["replica", "%s" % replicaCount] command += brickList + ["status"] - rc, out, err = _execGluster(command) - - if rc: - raise ge.GlusterVolumeRemoveBrickStatusFailedException(rc, out, err) - else: - return "\n".join(out) + try: + xmltree, out = _execGlusterXml(command) + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRemoveBrickStatusFailedException(rc=e.rc, + err=e.err) + try: + return _parseVolumeRemoveBrickStatus(xmltree) + except: + raise ge.GlusterXmlErrorException(err=out) @exportToSuperVdsm @@ -455,12 +581,12 @@ if replicaCount: command += ["replica", "%s" % replicaCount] command += brickList + ["commit"] - rc, out, err = _execGluster(command) - - if rc: - raise ge.GlusterVolumeRemoveBrickCommitFailedException(rc, out, err) - else: + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRemoveBrickCommitFailedException(rc=e.rc, + err=e.err) @exportToSuperVdsm @@ -469,12 +595,12 @@ if replicaCount: command += ["replica", "%s" % replicaCount] command += brickList + ["force"] - rc, out, err = _execGluster(command) - - if rc: - raise ge.GlusterVolumeRemoveBrickForceFailedException(rc, out, err) - else: + try: + _execGlusterXml(command) return True + except ge.GlusterCmdFailedException, e: + raise ge.GlusterVolumeRemoveBrickForceFailedException(rc=e.rc, + err=e.err) @exportToSuperVdsm diff --git a/vdsm/gluster/exception.py b/vdsm/gluster/exception.py index f4f497b..f209885 100644 --- a/vdsm/gluster/exception.py +++ b/vdsm/gluster/exception.py @@ -323,6 +323,11 @@ message = "Volume remove brick force failed" +class GlusterVolumeStatusAllFailedException(GlusterVolumeException): + code = 4158 + message = "Volume status all failed" + + # Host class GlusterHostException(GlusterException): code = 4400 diff --git a/vdsm/storage/taskManager.py b/vdsm/storage/taskManager.py index 3bc12f3..0a269cd 100644 --- a/vdsm/storage/taskManager.py +++ b/vdsm/storage/taskManager.py @@ -25,6 +25,12 @@ import storage_exception as se from task import Task, Job, TaskCleanType from threadPool import ThreadPool +try: + from gluster import cli as gcli + from gluster import exception as ge + _glusterEnabled = True +except ImportError: + _glusterEnabled = False class TaskManager: @@ -113,19 +119,82 @@ self.log.debug("Return: %s", subRes) return subRes - def getAllTasks(self): + def _getAllGlusterTasks(self): """ - Return Tasks for all public tasks. + Return all gluster tasks + """ + subRes = {} + if not _glusterEnabled: + return subRes + + for taskId, value in gcli.volumeStatusAll(): + msg = '' + state = '' + try: + if value['taskType'] == gcli.TaskType.REBALANCE: + status = gcli.volumeRebalanceStatus(value['volumeName']) + msg = ('Files [scanned: %d, moved: %d, failed: %d], ' + 'Total size moved: %d') % \ + (status['summary']['filesScanned'], + status['summary']['filesMoved'], + status['summary']['filesFailed'], + status['summary']['totalSizeMoved']) + state = status['summary']['status'] + elif value['taskType'] == gcli.TaskType.REMOVE_BRICK: + status = gcli.volumeRemoveBrickStatus(value['volumeName'], + value['bricks']) + msg = ('Files [scanned: %d, moved: %d, failed: %d], ' + 'Total size moved: %d') % \ + (status['summary']['filesScanned'], + status['summary']['filesMoved'], + status['summary']['filesFailed'], + status['summary']['totalSizeMoved']) + state = status['summary']['status'] + elif value['taskType'] == gcli.TaskType.REPLACE_BRICK: + status = gcli.volumeReplaceBrickStatus(value['volumeName'], + value['bricks'][0], + value['bricks'][1]) + msg = 'Files moved: %d, Moving file: %s' % \ + (status['filesMoved'], status['movingFile']) + state = status['status'] + except ge.GlusterException: + self.log.error("gluster exception occured", exc_info=True) + + subRes[taskId] = {"id": taskId, + "verb": value['volumeName'], + "state": state, + "code": value['taskType'], + "message": msg, + "result": '', + "tag": 'gluster'} + return subRes + + def getAllTasks(self, tag=[]): + """ + Return Tasks for all public tasks if param tag is empty, + else return tasks those tags are in param tag. """ self.log.debug("Entry.") subRes = {} for taskID, task in self._tasks.items(): try: - subRes[taskID] = task.getDetails() + if not tag: + subRes[taskID] = task.getDetails() + elif task.getTags() in tag: + subRes[taskID] = task.getDetails() except se.UnknownTask: # Return info for existing tasks only. self.log.warn("Unknown task %s. Maybe task was already " "cleared.", taskID) + + try: + if not tag: + subRes.update(self._getAllGlusterTasks()) + elif 'gluster' in tag: + subRes.update(self._getAllGlusterTasks()) + except ge.GlusterException: + self.log.error("gluster exception occured", exc_info=True) + self.log.debug("Return: %s", subRes) return subRes -- To view, visit http://gerrit.ovirt.org/7579 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9c765cbfebb5ba22f0d21efa04c824ea4daf6432 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Bala.FA <barumuga(a)redhat.com> Gerrit-Reviewer: Ayal Baron <abaron(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Federico Simoncelli <fsimonce(a)redhat.com> Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com> Gerrit-Reviewer: Timothy Asir <tjeyasin(a)redhat.com>

5 16

Change in vdsm[master]: tests: Fix AttributeError: 'module' object has no attribute ...
by osvoboda＠redhat.com 11 Oct '15

11 Oct '15

Ondřej Svoboda has uploaded a new change for review. Change subject: tests: Fix AttributeError: 'module' object has no attribute 'ifcfg' ...................................................................... tests: Fix AttributeError: 'module' object has no attribute 'ifcfg' This allows running NOSE_EXCLUDE='.*' make rpm on Fedora 20 again. Change-Id: Ibe715c9f4c0f6b2b832f897467ba551416c52432 Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M tests/configNetworkTests.py 1 file changed, 4 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/08/31108/1 diff --git a/tests/configNetworkTests.py b/tests/configNetworkTests.py index 83a4b73..afeaa65 100644 --- a/tests/configNetworkTests.py +++ b/tests/configNetworkTests.py @@ -25,8 +25,9 @@ from testlib import VdsmTestCase as TestCaseBase from monkeypatch import MonkeyPatch -from network import api, configurators +from network import api from network import errors +from network.configurators import ifcfg from network.models import Bond, Bridge, Nic, Vlan @@ -51,10 +52,8 @@ @MonkeyPatch(netinfo, 'networks', _fakeNetworks) @MonkeyPatch(netinfo, 'getMaxMtu', lambda *x: 1500) @MonkeyPatch(netinfo, 'getMtu', lambda *x: 1500) - @MonkeyPatch(configurators.ifcfg, 'ifdown', lambda *x: - _raiseInvalidOpException()) - @MonkeyPatch(configurators.ifcfg, 'ifup', - lambda *x: _raiseInvalidOpException()) + @MonkeyPatch(ifcfg, 'ifdown', lambda *x: _raiseInvalidOpException()) + @MonkeyPatch(ifcfg, 'ifup', lambda *x: _raiseInvalidOpException()) @MonkeyPatch(Bond, 'configure', lambda *x: _raiseInvalidOpException()) @MonkeyPatch(Bridge, 'configure', lambda *x: _raiseInvalidOpException()) @MonkeyPatch(Nic, 'configure', lambda *x: _raiseInvalidOpException()) -- To view, visit http://gerrit.ovirt.org/31108 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe715c9f4c0f6b2b832f897467ba551416c52432 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

5 25

Change in vdsm[master]: qemuimg: add support for convert progress
by Federico Simoncelli 11 Oct '15

11 Oct '15

Federico Simoncelli has uploaded a new change for review. Change subject: qemuimg: add support for convert progress ...................................................................... qemuimg: add support for convert progress Change-Id: Id0b53e418c62bb2e91444ba5f351c916ca417299 Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> --- M lib/vdsm/qemuimg.py M tests/qemuimgTests.py M vdsm/storage/image.py 3 files changed, 172 insertions(+), 58 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/33910/1 diff --git a/lib/vdsm/qemuimg.py b/lib/vdsm/qemuimg.py index 39cd394..9ca9c58 100644 --- a/lib/vdsm/qemuimg.py +++ b/lib/vdsm/qemuimg.py @@ -160,9 +160,9 @@ return check -def convert(srcImage, dstImage, stop, srcFormat=None, dstFormat=None, +def convert(srcImage, dstImage, srcFormat=None, dstFormat=None, backing=None, backingFormat=None): - cmd = [_qemuimg.cmd, "convert", "-t", "none"] + cmd = [_qemuimg.cmd, "convert", "-p", "-t", "none"] options = [] cwdPath = None @@ -190,14 +190,66 @@ cmd.append(dstImage) - (rc, out, err) = utils.watchCmd( - cmd, cwd=cwdPath, stop=stop, nice=utils.NICENESS.HIGH, - ioclass=utils.IOCLASS.IDLE) + return QemuImgProcess(cmd, cwd=cwdPath) - if rc != 0: - raise QImgError(rc, out, err) - return (rc, out, err) +class QemuImgProcess(object): + REGEXPR = re.compile(r'$([\d.]+)/100%$') + + def __init__(self, cmd, cwd=None): + self.progress = 0.0 + + self._stdout = bytearray() + self._stderr = bytearray() + + cmd = utils.ionice_cmd(cmd, utils.IOCLASS.IDLE) + cmd = utils.nice_cmd(cmd, utils.NICENESS.HIGH) + + self.execution = utils.CommandStream( + cmd, self._recvstdout, self._recvstderr, cwd=cwd, + deathSignal=signal.SIGKILL) + + def _recvstderr(self, buffer): + self._stderr += buffer + + def _recvstdout(self, buffer): + last_progress = None + self._stdout += buffer + + while True: + try: + idx = self._stdout.index('\r') + except ValueError: + break + + last_progress = self._stdout[:idx] + del self._stdout[:idx + 1] + + if last_progress: + m = self.REGEXPR.match(last_progress.strip()) + if m is None: + raise ValueError( + 'Unable to parse: "%s"' % last_progress) + self.progress = float(m.group(1)) + + @property + def stderr(self): + return str(self._stderr) + + def wait(self, timeout=None): + returncode = self.execution.wait(timeout=timeout) + + if (self.execution.returncode is not None + and self.execution.returncode != 0): + raise QImgError(returncode, "", self.stderr) + + return returncode + + def terminate(self): + self.execution.terminate() + + def kill(self): + self.execution.kill() def resize(image, newSize, format=None): diff --git a/tests/qemuimgTests.py b/tests/qemuimgTests.py index abc0750..9e61de0 100644 --- a/tests/qemuimgTests.py +++ b/tests/qemuimgTests.py @@ -163,12 +163,11 @@ def test_no_format(self): def convert(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'src', 'dst'] + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'src', 'dst'] self.assertEqual(cmd, expected) - return 0, '', '' - with FakeCmd(utils, 'watchCmd', convert): - qemuimg.convert('src', 'dst', True) + with FakeCmd(qemuimg, 'QemuImgProcess', convert): + qemuimg.convert('src', 'dst') def test_qcow2_compat_unsupported(self): def qcow2_compat_unsupported(cmd, **kw): @@ -176,14 +175,13 @@ return 0, 'Supported options:\nsize ...\n', '' def convert(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'src', '-O', + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'src', '-O', 'qcow2', 'dst'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', qcow2_compat_unsupported): - with FakeCmd(utils, 'watchCmd', convert): - qemuimg.convert('src', 'dst', True, dstFormat='qcow2') + with FakeCmd(qemuimg, 'QemuImgProcess', convert): + qemuimg.convert('src', 'dst', dstFormat='qcow2') def qcow2_compat_supported(self, cmd, **kw): self.check_supports_qcow2_compat(cmd, **kw) @@ -191,14 +189,13 @@ def test_qcow2_compat_supported(self): def convert(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'src', '-O', + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'src', '-O', 'qcow2', '-o', 'compat=0.10', 'dst'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', self.qcow2_compat_supported): - with FakeCmd(utils, 'watchCmd', convert): - qemuimg.convert('src', 'dst', True, dstFormat='qcow2') + with FakeCmd(qemuimg, 'QemuImgProcess', convert): + qemuimg.convert('src', 'dst', dstFormat='qcow2') def check_supports_qcow2_compat(self, cmd, **kw): expected = [QEMU_IMG, 'convert', '-O', 'qcow2', '-o', '?', '/dev/null', @@ -207,49 +204,95 @@ def test_qcow2_no_backing_file(self): def qcow2_no_backing_file(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'source', '-O', - 'qcow2', '-o', 'compat=0.10', 'target'] + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'source', + '-O', 'qcow2', '-o', 'compat=0.10', 'target'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', self.qcow2_compat_supported): - with FakeCmd(utils, 'watchCmd', qcow2_no_backing_file): - qemuimg.convert('source', 'target', None, dstFormat='qcow2') + with FakeCmd(qemuimg, 'QemuImgProcess', qcow2_no_backing_file): + qemuimg.convert('source', 'target', dstFormat='qcow2') def test_qcow2_backing_file(self): def qcow2_backing_file(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'source', '-O', - 'qcow2', '-o', 'compat=0.10,backing_file=backing', - 'target'] + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'source', + '-O', 'qcow2', + '-o', 'compat=0.10,backing_file=backing', 'target'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', self.qcow2_compat_supported): - with FakeCmd(utils, 'watchCmd', qcow2_backing_file): - qemuimg.convert('source', 'target', None, dstFormat='qcow2', + with FakeCmd(qemuimg, 'QemuImgProcess', qcow2_backing_file): + qemuimg.convert('source', 'target', dstFormat='qcow2', backing='backing') def test_qcow2_backing_format(self): def qcow2_backing_format(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'source', '-O', - 'qcow2', '-o', 'compat=0.10', 'target'] + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'source', + '-O', 'qcow2', '-o', 'compat=0.10', 'target'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', self.qcow2_compat_supported): - with FakeCmd(utils, 'watchCmd', qcow2_backing_format): - qemuimg.convert('source', 'target', None, dstFormat='qcow2', + with FakeCmd(qemuimg, 'QemuImgProcess', qcow2_backing_format): + qemuimg.convert('source', 'target', dstFormat='qcow2', backingFormat='qcow2') def test_qcow2_backing_file_and_format(self): def qcow2_backing_format(cmd, **kw): - expected = [QEMU_IMG, 'convert', '-t', 'none', 'source', '-O', - 'qcow2', '-o', 'compat=0.10,backing_file=backing,' + expected = [QEMU_IMG, 'convert', '-p', '-t', 'none', 'source', + '-O', 'qcow2', + '-o', 'compat=0.10,backing_file=backing,' 'backing_fmt=qcow2', 'target'] self.assertEqual(cmd, expected) - return 0, '', '' with FakeCmd(utils, 'execCmd', self.qcow2_compat_supported): - with FakeCmd(utils, 'watchCmd', qcow2_backing_format): - qemuimg.convert('source', 'target', None, dstFormat='qcow2', + with FakeCmd(qemuimg, 'QemuImgProcess', qcow2_backing_format): + qemuimg.convert('source', 'target', dstFormat='qcow2', backing='backing', backingFormat='qcow2') + + +class QemuImgProcessTests(TestCaseBase): + PROGRESS_FORMAT = " (%.2f/100%%)\r" + + @staticmethod + def _progress_iterator(): + return map(lambda x: x / 100.0, xrange(0, 10000, 1)) + + def test_progress_simple(self): + p = qemuimg.QemuImgProcess([]) + + for progress in self._progress_iterator(): + p._recvstdout(self.PROGRESS_FORMAT % progress) + self.assertEquals(p.progress, progress) + + self.assertEquals(p.wait(), 0) + + def test_progress_incomplete(self): + p = qemuimg.QemuImgProcess([]) + + for progress in self._progress_iterator(): + stdout = self.PROGRESS_FORMAT % progress + p._recvstdout(stdout[:12]) + p._recvstdout(stdout[12:]) + self.assertEquals(p.progress, progress) + + self.assertEquals(p.wait(), 0) + + def test_progress_batch(self): + p = qemuimg.QemuImgProcess([]) + + p._recvstdout( + (self.PROGRESS_FORMAT % 10.00) + + (self.PROGRESS_FORMAT % 25.00) + + (self.PROGRESS_FORMAT % 33.33)) + + self.assertEquals(p.progress, 33.33) + self.assertEquals(p.wait(), 0) + + def test_unexpected_output(self): + p = qemuimg.QemuImgProcess([]) + + self.assertRaises(ValueError, p._recvstdout, "Hello World\r") + + p._recvstdout("Hello ") + self.assertRaises(ValueError, p._recvstdout, "World\r") + + self.assertEquals(p.wait(), 0) diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 41e3f30..cdcd82a 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -93,6 +93,7 @@ """ log = logging.getLogger('Storage.Image') _fakeTemplateLock = threading.Lock() + _QEMU_LOGGING_INTERVAL = 60.0 @classmethod def createImageRollback(cls, taskObj, imageDir): @@ -109,6 +110,24 @@ def __init__(self, repoPath): self.repoPath = repoPath + + def qemuImgConvert(self, *args, **kwargs): + self.log.debug('starting qemu-img operation') + command = qemuimg.convert(*args, **kwargs) + + def abortImgConversion(): + self.log.info('aborting ongoing qemu-img operation') + command.terminate() + + retcode = None + + with vars.task.abort_callback(abortImgConversion): + while retcode is None: + retcode = command.wait(self._QEMU_LOGGING_INTERVAL) + self.log.debug('qemu-img operation progress: %s%%', + command.progress) + + self.log.debug('qemu-img operation has completed: %s', retcode) def create(self, sdUUID, imgUUID): """Create placeholder for image's volumes @@ -444,13 +463,12 @@ backingFormat = None self.log.debug("start qemu convert") - qemuimg.convert(srcVol.getVolumePath(), - dstVol.getVolumePath(), - vars.task.aborting, - srcFormat=srcFormat, - dstFormat=dstFormat, - backing=backing, - backingFormat=backingFormat) + self.qemuImgConvert(srcVol.getVolumePath(), + dstVol.getVolumePath(), + srcFormat=srcFormat, + dstFormat=dstFormat, + backing=backing, + backingFormat=backingFormat) except ActionStopped: raise except se.StorageException: @@ -830,10 +848,11 @@ dstVol.prepare(rw=True, setrw=True) try: - qemuimg.convert(volParams['path'], dstPath, - vars.task.aborting, - volume.fmt2str(volParams['volFormat']), - volume.fmt2str(dstVolFormat)) + self.qemuImgConvert( + volParams['path'], + dstPath, + srcFormat=volume.fmt2str(volParams['volFormat']), + dstFormat=volume.fmt2str(dstVolFormat)) except ActionStopped: raise except qemuimg.QImgError: @@ -1045,7 +1064,7 @@ # volume and rebase successor's children (if exists) on top of it. # Step 1: Create an empty volume named sucessor_MERGE similar to # ancestor volume. - # Step 2: qemuimg.convert successor -> sucessor_MERGE + # Step 2: qemuImgConvert successor -> sucessor_MERGE # Step 3: Rename successor to _remove_me__successor # Step 4: Rename successor_MERGE to successor # Step 5: Unsafely rebase successor's children on top of temporary @@ -1071,11 +1090,11 @@ # Step 2: Convert successor to new volume # qemu-img convert -f qcow2 successor -O raw newUUID try: - qemuimg.convert(srcVolParams['path'], - newVol.getVolumePath(), - vars.task.aborting, - volume.fmt2str(srcVolParams['volFormat']), - volume.fmt2str(volParams['volFormat'])) + self.qemuImgConvert( + srcVolParams['path'], + newVol.getVolumePath(), + srcFormat=volume.fmt2str(srcVolParams['volFormat']), + dstFormat=volume.fmt2str(volParams['volFormat'])) except qemuimg.QImgError: self.log.exception('conversion failure for volume %s', srcVol.volUUID) -- To view, visit http://gerrit.ovirt.org/33910 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id0b53e418c62bb2e91444ba5f351c916ca417299 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>

7 52

Change in vdsm[master]: vm: Modify and save state after hotunplugging the disk
by xfrancis＠redhat.com 11 Oct '15

11 Oct '15

Xavi Francisco has uploaded a new change for review. Change subject: vm: Modify and save state after hotunplugging the disk ...................................................................... vm: Modify and save state after hotunplugging the disk The rationale behind this patch is to modify the way the hotunplug process is executed. Now when the hotunplug command is called we first remove the disk from the internal state and after that the libvirt hotunplug command is executed.The problem with that approach is that if VDSM restarts between the modification of the internal state and the actual unplugging of the disk, the disk never gets unplugged leaving the system in an inconsistent state. This patch solves the issue by executing the modification of the internal state after the actual unplugging has successfully happened. This state is later saved so if either the disk has been unplugged or not VDSM is able to recover its state in case of a restart. Change-Id: I5169fa16591283de33aa82c5730626bfd5d3eaf5 Signed-off-by: Xavi Francisco <xfrancis(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 12 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/87/28187/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index bd670b9..1eb2adc 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -3449,20 +3449,6 @@ driveXml = drive.getXML().toprettyxml(encoding='utf-8') self.log.debug("Hotunplug disk xml: %s", driveXml) # Remove found disk from vm's drives list - if isVdsmImage(drive): - self.sdIds.remove(drive.domainID) - self._devices[DISK_DEVICES].remove(drive) - # Find and remove disk device from vm's conf - diskDev = None - for dev in self.conf['devices'][:]: - if (dev['type'] == DISK_DEVICES and - dev['path'] == drive.path): - with self._confLock: - self.conf['devices'].remove(dev) - diskDev = dev - break - - self.saveState() hooks.before_disk_hotunplug(driveXml, self.conf, params=drive.custom) @@ -3472,19 +3458,25 @@ self.log.error("Hotunplug failed", exc_info=True) if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: return errCode['noVM'] - self._devices[DISK_DEVICES].append(drive) - # Restore disk device in vm's conf and _devices - if diskDev: - with self._confLock: - self.conf['devices'].append(diskDev) - self.saveState() return { 'status': {'code': errCode['hotunplugDisk']['status']['code'], 'message': e.message}} else: + if isVdsmImage(drive): + self.sdIds.remove(drive.domainID) + self._devices[DISK_DEVICES].remove(drive) + # Find and remove disk device from vm's conf + for dev in self.conf['devices'][:]: + if (dev['type'] == DISK_DEVICES and + dev['path'] == drive.path): + with self._confLock: + self.conf['devices'].remove(dev) + break hooks.after_disk_hotunplug(driveXml, self.conf, params=drive.custom) self._cleanupDrives(drive) + finally: + self.saveState() return {'status': doneCode, 'vmList': self.status()} -- To view, visit http://gerrit.ovirt.org/28187 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5169fa16591283de33aa82c5730626bfd5d3eaf5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Xavi Francisco <xfrancis(a)redhat.com>

6 6

Change in vdsm[master]: incorporate stop/start logic to functional tests
by ykleinbe＠redhat.com 11 Oct '15

11 Oct '15

Yoav Kleinberger has uploaded a new change for review. Change subject: incorporate stop/start logic to functional tests ...................................................................... incorporate stop/start logic to functional tests Change-Id: I55b86578a681de2b09ac5ee01ca2e3b9443ed13a Signed-off-by: Yoav Kleinberger <ykleinbe(a)redhat.com> --- M tests/functional/new/basicLocalFSStorageDomainTest.py A tests/functional/new/controlvdsm.py 2 files changed, 65 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/23/29423/1 diff --git a/tests/functional/new/basicLocalFSStorageDomainTest.py b/tests/functional/new/basicLocalFSStorageDomainTest.py index a590886..3280e16 100644 --- a/tests/functional/new/basicLocalFSStorageDomainTest.py +++ b/tests/functional/new/basicLocalFSStorageDomainTest.py @@ -6,10 +6,22 @@ import uuid import storage.volume import storage.image +import logging +logging.basicConfig(level=logging.DEBUG, format='%(asctime)s TEST %(levelname)s: %(message)s') from . import testlib +from . import controlvdsm class TestBasicLocalFSStorageDomain: - def notest_flow_connect_create_storage_domain_format_disconnect(self): + def setup(self): + control_vdsm = controlvdsm.ControlVDSM() + control_vdsm.cleanup() + + @classmethod + def teardown_class(cls): + control_vdsm = controlvdsm.ControlVDSM() + control_vdsm.cleanup() + + def test_flow_connect_create_storage_domain_format_disconnect(self): with testlib.TemporaryDirectory() as directory: with testlib.VDSMTestTools() as (vdsm, verify): diff --git a/tests/functional/new/controlvdsm.py b/tests/functional/new/controlvdsm.py new file mode 100644 index 0000000..a7e8e79 --- /dev/null +++ b/tests/functional/new/controlvdsm.py @@ -0,0 +1,52 @@ +import subprocess +import logging +import vdsm.vdscli +import socket +import vdsm.config +import time + +class ControlVDSM: + def cleanup(self): + self._stop_service() + assert not self._service_running() + self._brutally_clean_files() +# self._restart_service() + return self._check_connection() + + def _check_connection(self): + useSSL = vdsm.config.config.getboolean('vars', 'ssl') + vdsmClient = vdsm.vdscli.connect(useSSL=useSSL) + RETRIES = 5 + for _ in range(RETRIES): + try: + vdsmClient.getStorageDomainsList() + logging.info('VDSM ready for testing') + except socket.error as e: + logging.warning('could not talk to VDSM: %s' % e) + time.sleep(1) + + raise Exception('could not connect to VDSM') + + def _stop_service(self): + self._run("sudo service vdsmd stop") + + def _service_running(self): + return_code = subprocess.call('sudo service vdsmd status', shell=True, stdout=open('/dev/null','w'), stderr=open('/dev/null','w')) + logging.info('vdsm running: %s' % (return_code == 0)) + return return_code == 0 + + def _restart_service(self): + self._run("sudo vdsm-tool configure --force") + self._run("sudo service vdsmd start") + + def _run(self, command): + logging.info('running: %s' % command) + return_code = subprocess.call(command, shell=True, close_fds=True, stdout=open('/dev/null','w'), stderr=open('/dev/null','w')) + if return_code != 0: + logging.warning('failure! command was: %s' % command) + else: + logging.info('finished.') + + def _brutally_clean_files(self): + logging.warning('removing /rhev/data-center without asking too many questions') + self._run('sudo rm -fr /rhev/data-center/*') -- To view, visit http://gerrit.ovirt.org/29423 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I55b86578a681de2b09ac5ee01ca2e3b9443ed13a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yoav Kleinberger <ykleinbe(a)redhat.com>

4 6

Change in vdsm[master]: vdsm: extract arch-dependent details
by ykleinbe＠redhat.com 08 Oct '15

08 Oct '15

Yoav Kleinberger has uploaded a new change for review. Change subject: vdsm: extract arch-dependent details ...................................................................... vdsm: extract arch-dependent details we now use other packages... Change-Id: I94bfc8f9d93d8f7a73e69b9390329c342f2904e7 Signed-off-by: Yoav Kleinberger <ykleinbe(a)redhat.com> --- M vdsm.spec.in M vdsm/storage/protect/Makefile.am D vdsm/storage/protect/safelease.c M vdsm/storage/protect/spmprotect.sh 4 files changed, 43 insertions(+), 766 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/97/31297/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index ceb238f..2aa8bdb 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -136,10 +136,7 @@ Requires: mom >= 0.4.1 Requires: numactl -%ifarch x86_64 -Requires: python-dmidecode -Requires: dmidecode -%endif +Requires: vdsm-arch-dependencies %if 0%{?rhel} >= 7 || 0%{?fedora} >= 18 Requires: libvirt-daemon >= 1.0.2-1 @@ -249,6 +246,8 @@ Requires(post): policycoreutils-python Requires(preun): policycoreutils-python +BuildArch: noarch + %description The VDSM service is required by a Virtualization Manager to manage the Linux hosts. VDSM manages and monitors the host's storage, memory and @@ -257,7 +256,7 @@ %package cli Summary: VDSM command line interface -BuildArch: noarch + Requires: %{name}-python = %{version}-%{release} Requires: %{name}-xmlrpc = %{version}-%{release} @@ -267,7 +266,7 @@ %package xmlrpc Summary: VDSM xmlrpc API -BuildArch: noarch + Requires: %{name}-python = %{version}-%{release} @@ -278,7 +277,7 @@ %package jsonrpc Summary: VDSM API Server -BuildArch: noarch + Requires: %{name}-python = %{version}-%{release} Requires: %{name}-yajsonrpc = %{version}-%{release} @@ -291,7 +290,7 @@ %package yajsonrpc Summary: JSON RPC server and client implementation -BuildArch: noarch + Requires: python >= 2.6 @@ -300,7 +299,7 @@ %package python-zombiereaper Summary: Collects zombie processes automatically -BuildArch: noarch + Requires: python >= 2.6 @@ -309,7 +308,7 @@ %package bootstrap Summary: VDSM bootstrapping package -BuildArch: noarch + %description bootstrap VDSM bootstrapping package. Used for delivering the bootstrap code onto the @@ -317,7 +316,7 @@ %package reg Summary: VDSM registration package -BuildArch: noarch + Requires: %{name} = %{version}-%{release} Requires: m2crypto @@ -334,7 +333,7 @@ Requires: python-cpopen >= 1.2.3-5 Requires: m2crypto Requires: python-ioprocess >= 0.5-1 -BuildArch: noarch + %description python Shared libraries between the various VDSM packages. @@ -343,7 +342,7 @@ Summary: VDSM Debug Plugin Requires: %{name} Requires: %{name}-xmlrpc = %{version}-%{release} -BuildArch: noarch + %description debug-plugin Used by the trained monkeys at Red Hat to insert chaos and mayhem in to VDSM. @@ -353,14 +352,14 @@ Requires: %{name} = %{version}-%{release} Requires: dracut Requires: python-nose -BuildArch: noarch + %description tests A test suite for verifying the functionality of a running vdsm instance %package hook-checkimages Summary: Qcow2 disk image format check hook for VDSM -BuildArch: noarch + Requires: %{name} %description hook-checkimages @@ -369,7 +368,7 @@ %package hook-ethtool-options Summary: Allow setting custom ethtool options for vdsm controlled nics -BuildArch: noarch + Requires: %{name} = %{version}-%{release} %description hook-ethtool-options @@ -380,7 +379,7 @@ %package hook-vhostmd Summary: VDSM hook set for interaction with vhostmd Requires: vhostmd -BuildArch: noarch + %description hook-vhostmd VDSM hook to use vhostmd per VM according to Virtualization Manager requests. @@ -388,7 +387,7 @@ %package hook-faqemu Summary: Fake qemu process for VDSM quality assurance -BuildArch: noarch + Requires: %{name} %description hook-faqemu @@ -399,7 +398,7 @@ %package hook-directlun Summary: Direct LUN support for VDSM -BuildArch: noarch + %description hook-directlun VDSM hook enable user to add storage LUN for VDSM @@ -407,7 +406,7 @@ %package hook-macbind Summary: Bind a vNIC to a Bridge -BuildArch: noarch + Requires: %{name} >= 4.14 %description hook-macbind @@ -415,7 +414,7 @@ %package hook-macspoof Summary: Disables MAC spoofing filtering -BuildArch: noarch + %description hook-macspoof VDSM hooks which allow to disable mac spoof filtering @@ -424,7 +423,7 @@ %package hook-extnet Summary: Force a vNIC to connect to a specific libvirt network -BuildArch: noarch + Requires: %{name} = %{version}-%{release} %description hook-extnet @@ -433,7 +432,7 @@ %package hook-fakevmstats Summary: Generate random VM statistics -BuildArch: noarch + Requires: %{name} %description hook-fakevmstats @@ -441,7 +440,7 @@ %package hook-fileinject Summary: Allow uploading file to VMs disk -BuildArch: noarch + Requires: python-libguestfs %description hook-fileinject @@ -450,14 +449,14 @@ %package hook-floppy Summary: Allow adding floppy to VM -BuildArch: noarch + %description hook-floppy Allow adding floppy to VM %package hook-hostusb Summary: Allow attaching USB device from host -BuildArch: noarch + Requires: usbutils %description hook-hostusb @@ -466,7 +465,7 @@ %package hook-hugepages Summary: Huge pages enable user to handle VM with 2048KB page files. -BuildArch: noarch + %description hook-hugepages Hook is getting number of huge pages reserve them for the VM, @@ -474,7 +473,7 @@ %package hook-isolatedprivatevlan Summary: Isolated network environment for VMs -BuildArch: noarch + %description hook-isolatedprivatevlan limit VM traffic to a specific gateway by its mac address, @@ -484,7 +483,7 @@ %package hook-nestedvt Summary: Nested Virtualization support for VDSM -BuildArch: noarch + %description hook-nestedvt If the nested virtualization is enabled in your kvm module @@ -492,7 +491,7 @@ %package hook-numa Summary: NUMA support for VDSM -BuildArch: noarch + %description hook-numa Hooks is getting number/rage of NUMA nodes and NUMA mode, @@ -500,14 +499,14 @@ %package hook-openstacknet Summary: OpenStack Network vNICs support for VDSM -BuildArch: noarch + %description hook-openstacknet Hook for OpenStack Network vNICs. %package hook-pincpu Summary: Hook pin VM so specific CPUs -BuildArch: noarch + %description hook-pincpu pincpu is hook for VDSM. @@ -515,7 +514,7 @@ %package hook-promisc Summary: Network interface promiscuous mode support for VDSM -BuildArch: noarch + %description hook-promisc VDSM promiscuous mode let user define a VM interface that will capture @@ -523,7 +522,7 @@ %package hook-qemucmdline Summary: QEMU cmdline hook for VDSM -BuildArch: noarch + Requires: %{name} %description hook-qemucmdline @@ -533,14 +532,14 @@ %package hook-qos Summary: QoS network in/out traffic support for VDSM -BuildArch: noarch + %description hook-qos Hook adds QoS in/out traffic to VMs interfaces %package hook-scratchpad Summary: One time disk creation for VDSM -BuildArch: noarch + %description hook-scratchpad scratchpad hook for VDSM @@ -550,7 +549,7 @@ %package hook-smbios Summary: Adding custom smbios entries to libvirt domain via VDSM -BuildArch: noarch + %description hook-smbios Adding custom smbios entries to libvirt domain via VDSM @@ -561,7 +560,7 @@ %if 0%{?rhel} >= 7 || 0%{?fedora} >= 18 Requires: libvirt-daemon-driver-nodedev %endif -BuildArch: noarch + %description hook-sriov sr-iov hook enable to add virtual functions exposed by the device @@ -570,7 +569,7 @@ %package hook-spiceoptions Summary: To configure spice options for vm -BuildArch: noarch + %description hook-spiceoptions This vdsm hook can be used to configure some of @@ -578,7 +577,7 @@ %package hook-vmfex Summary: vmfex support for VDSM -BuildArch: noarch + Conflicts: hook-vmfex-dev %description hook-vmfex @@ -586,7 +585,7 @@ %package hook-vmfex-dev Summary: VM-FEX vNIC support for VDSM -BuildArch: noarch + Requires: %{name} = %{version}-%{release} Conflicts: hook-vmfex @@ -596,7 +595,7 @@ %package hook-vmdisk Summary: External disk support for VDSM -BuildArch: noarch + %description hook-vmdisk Hook adds additional disk image for a VM (raw or qcow2) @@ -604,7 +603,7 @@ %if 0%{?with_gluster} %package gluster Summary: Gluster Plugin for VDSM -BuildArch: noarch + Requires: %{name} = %{version}-%{release} Requires: glusterfs-server @@ -1051,7 +1050,6 @@ %{_datadir}/%{vdsm_name}/storage/volume.py* %{_datadir}/%{vdsm_name}/storage/imageRepository/__init__.py* %{_datadir}/%{vdsm_name}/storage/imageRepository/formatConverter.py* -%{_libexecdir}/%{vdsm_name}/safelease %{_libexecdir}/%{vdsm_name}/spmprotect.sh %{_libexecdir}/%{vdsm_name}/spmstop.sh %dir %{_libexecdir}/%{vdsm_name}/hooks diff --git a/vdsm/storage/protect/Makefile.am b/vdsm/storage/protect/Makefile.am index feb40a9..1eb4661 100644 --- a/vdsm/storage/protect/Makefile.am +++ b/vdsm/storage/protect/Makefile.am @@ -6,11 +6,6 @@ # LICENSE_GPL_v2 which accompany this distribution. # -vdsmexec_PROGRAMS = safelease - dist_vdsmexec_SCRIPTS = \ spmprotect.sh \ spmstop.sh - -safelease_SOURCES = \ - safelease.c diff --git a/vdsm/storage/protect/safelease.c b/vdsm/storage/protect/safelease.c deleted file mode 100644 index 3814231..0000000 --- a/vdsm/storage/protect/safelease.c +++ /dev/null @@ -1,716 +0,0 @@ -/* Locker */ -#define _GNU_SOURCE 1 -#include <stdarg.h> -#include <stdio.h> -#include <stdlib.h> -#include <unistd.h> -#include <string.h> -#include <sys/types.h> -#include <sys/time.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <errno.h> -#include <time.h> -#include <signal.h> - -#define WARN(fmt, args...) warn(__FUNCTION__, fmt, ## args) -#define PANIC(fmt, args...) panic(__FUNCTION__, fmt, ## args) -#define DEBUG(fmt, args...) do { if (debug) warn(__FUNCTION__, fmt, ## args); } while (0) - -char *freetag = "------FREE------0000000000000000"; -enum { - idlen = 16, - stamplen = 16, - taglen = idlen + stamplen, -}; - -char *progname; -int debug; - -char *id; -char *path; -char *request; - -long lease_ms; -long op_max_ms; -char *iobuf; - -inline unsigned long long tv2msec(struct timeval *tv); -int renew(int fd, off_t offset, char *id, long long *ts); - -void -panic(const char const *fn, char *msg, ...) -{ - char buf[512]; - va_list va; - int n; - - va_start(va, msg); - n = vsprintf(buf, msg, va); - va_end(va); - buf[n] = 0; - - fprintf(stderr, "panic: [%d] %s: %s: (%m)\n", getpid(), fn, buf); - - exit(-1); -} - -void -warn(const char const *fn, char *msg, ...) -{ - struct timeval tv; - long long unsigned tscurr; - char buf[512]; - va_list va; - int n; - - va_start(va, msg); - n = vsprintf(buf, msg, va); - va_end(va); - buf[n] = 0; - - gettimeofday(&tv, 0); - tscurr = tv2msec(&tv); - - fprintf(stderr, "[%s:%d:%llu]: %s: %s\n", progname, getpid(), tscurr, fn, buf); -} - -void -usage(void) -{ - fprintf(stderr, "Usage: %s [ -h ] <op> [...]\n", progname); - fprintf(stderr, "Ops:\n" - "acquire [ -r <path> ] [ -b ] [ -o offset ] <path> <id> <lease_ms> <op_max_ms>\n" - "renew [ -r <path> ] [ -o offset ] [ -t laststamp ] <path> <id> <lease_ms> <op_max_ms>\n" - "release [ -f ] [ -o offset ] <path> <id>\n" - "query [ -o offset ] <path>\n" - "protect [ -r <path> -i <id>] [ -o offset ] <path> <lease_ms> <op_max_ms> <progname> [<param1> ...]\n" - ); - fprintf(stderr, "\nNotes:\n" - "-b - busy loop on lease until lease acquired\n" - "-f - force release even if lease id is not equal to id\n" - "-o - offset to lease in path (default is 0)\n" - "-t - timestamp of last successful renewal\n" - "Path is a path to a device or a file to use as a sync object.\n" - "Id is an arbitrary unique string\n" - "lease_ms is the maximum time in msec that the owner of the lease\n" - " may hold it without renewing it\n" - "op_max_ms is the maximum time in msec that a single IO operation may take (must be <= lease_ms).\n" - "if -r option is used, the path is a readable file/device.\n" - " The program then validates that its 'id' is written at the given offset.\n" - " If this is not the case, acquire and renew will fail immediately.\n" - ); - exit(1); -} - -inline unsigned long long -tv2msec(struct timeval *tv) -{ - return tv->tv_sec * 1000ull + tv->tv_usec/1000; -} - -int -withintimelimits(struct timeval *start, struct timeval *stop) -{ - unsigned long long delta; - if (op_max_ms <= 0) - return 1; - delta = tv2msec(stop) - tv2msec(start); - if (delta > op_max_ms) { - DEBUG("Error - time limit breached: op_max_ms - %ld, time passed - %lld", op_max_ms, delta); - errno = -ETIMEDOUT; - return 0; - } - return 1; -} - -int -sametag(const char *tag1, const char *tag2) -{ - return !memcmp(tag1, tag2, taglen); -} - -int -isfree(const char *tag) -{ - return sametag(tag, freetag); -} - -void -settag(char *tag, const char *src) -{ - memcpy(tag, src, taglen); -} - -void -buildtag(char *tag, const char *id, long long ts) -{ - snprintf(tag, taglen+1, "%-*s%0*llx", idlen, id, stamplen, ts); - DEBUG("'%s' ts %lld", tag, ts); -} - -int -sameid(const char *tag, const char *id) -{ - char _id[idlen+1]; - - snprintf(_id, idlen+1, "%-*s", idlen, id); - return !memcmp(tag, _id, idlen); -} - -void -querytag(const char *tag, char *id, long long *ts) -{ - char _stamp[stamplen+1] = ""; - - memcpy(id, tag, idlen); - id[idlen] = 0; - memcpy(_stamp, tag+idlen, stamplen); - *ts = strtoull(_stamp, 0, 16); -} - -int -readtag(int fd, off_t offset, char *tag, int limit) -{ - struct timeval start, stop; - int r; - - DEBUG("fd %d offset %ld", fd, offset); - gettimeofday(&start, 0); - r = pread(fd, iobuf, 512, offset); - gettimeofday(&stop, 0); - DEBUG("r %d %m", r); - if (r <= 0 || (limit && !withintimelimits(&start, &stop))) - return -1; - memcpy(tag, iobuf, taglen); - return r; -} - -int -writetag(int fd, off_t offset, const char *tag, int limit) -{ - struct timeval start, stop; - int r; - - DEBUG("Enter"); - memcpy(iobuf, tag, taglen); - gettimeofday(&start, 0); - r = pwrite(fd, iobuf, 512, offset) < taglen ? -1 : 0; - gettimeofday(&stop, 0); - DEBUG("Exit r=%ld", r); - if (r < 0 || (limit && !withintimelimits(&start, &stop))) - return -1; - return r; -} - -int -writetimestamp(int fd, off_t offset, const char *id, char *tag, long long *ts) -{ - struct timeval tv; - long long t; - int r; - - gettimeofday(&tv, 0); - t = tv.tv_sec * 1000000ll + tv.tv_usec; - buildtag(tag, id, t); - r = writetag(fd, offset, tag, 1); - if (r > 0) - *ts = t; - return r; -} - -/* - * Attempt to acquire the lease. - * Return 1 if succedded, 0 if not , and < 0 on errors. - */ -int -acquire(int fd, off_t offset, char *id, int busyloop, long long *ts) -{ - char curr[taglen+1] = "", last[taglen+1] = "", tag[taglen+1] = ""; - long backoff_usec = (lease_ms + 6 * op_max_ms) * 1000; - long contend_usec = (2 * op_max_ms) * 1000; - char dummyid[idlen+1]; - - if (readtag(fd, offset, curr, 1) < 0) - return -errno; - - settag(last, freetag); - - do { - DEBUG("restart: curr tag is '%s'", curr); - if (!sametag(curr, last) && !isfree(curr)) do { - DEBUG("backoff: curr tag is '%s'", curr); - settag(last, curr); - usleep(backoff_usec); - if (readtag(fd, offset, curr, 1) < 0) - return -errno; - } while (busyloop && !sametag(curr, last) && !isfree(curr)); - if (!sametag(curr, last) && !isfree(curr)) { - DEBUG("fail: curr tag is '%s'", curr); - return 0; - } - DEBUG("contend: curr tag is '%s'", curr); - if (writetimestamp(fd, offset, id, tag, ts) < 0) { - DEBUG("lost (writetimestamp failed) : curr tag is %s", curr); - return -errno; - } - usleep(contend_usec); - if (readtag(fd, offset, curr, 1) < 0) { - DEBUG("lost (readtag failed) : curr tag is %s", curr); - return -errno; - } - } while (busyloop && !sametag(curr, tag)); - - if (busyloop || sametag(curr, tag)) { - DEBUG("won : curr tag is %s", curr); - querytag(curr, dummyid, ts); - return renew(fd, offset, id, ts); - } - DEBUG("lost : curr tag is %s\n our tag is %s", curr, tag); - return 0; -} - -static void -handler(int sig) -{ - PANIC("IO op too long"); -} - -long long -timeleft_ms(long long tsprev) -{ - struct timeval tv; - long long tscurr; - - tsprev /= 1000; - gettimeofday(&tv, 0); - tscurr = tv2msec(&tv); - DEBUG("time elapsed: %lld/%lld", tscurr - tsprev, lease_ms); - return lease_ms - (tscurr - tsprev); -} - -/* - * Attempt to renew the lease. - * Return 1 if succeded, 0 if not , and < 0 on errors. - */ -int -renew(int fd, off_t offset, char *id, long long *ts) -{ - char curr[taglen+1] = "", tag[taglen+1] = ""; - char dummyid[idlen+1]; - struct sigaction sa; - long long msleft; - int rc = 0; - - sa.sa_flags = !SA_RESTART; - sigemptyset(&sa.sa_mask); - sa.sa_handler = handler; - if (sigaction(SIGALRM, &sa, NULL) == -1) - PANIC("sigaction: can't set alarm"); - - if (readtag(fd, offset, curr, 0) < 0) { - rc = -errno; - goto out; - } - - DEBUG("curr tag is '%s'", curr); - if (!sameid(curr, id)) { - *ts = 0; - goto out; - } - - querytag(curr, dummyid, ts); - msleft = timeleft_ms(*ts); - if (msleft <= 0) { - rc = -ETIMEDOUT; - goto out; - } - - alarm(msleft / 1000); - DEBUG("updating tag: msleft %lld", msleft); - if (writetimestamp(fd, offset, id, tag, ts) < 0) { - rc = -errno; - goto out; - } - - DEBUG("All good"); - /* disable the alarm because usleep might use the same signal */ - alarm(0); - return 1; - -out: - alarm(0); - return rc; -} - -/* - * Attempt to release the lease. - * Return 1 if succedded, 0 if not , and < 0 on errors. - */ -int -release(int fd, off_t offset, char *id, int force) -{ - char curr[taglen+1] = ""; - - if (!force) { - if (readtag(fd, offset, curr, 0) < 0) - return -errno; - - if (!sameid(curr, id)) - return 0; - } - - return writetag(fd, offset, freetag, 0) < 0 ? -1 : 1; -} - -/* - * Qeury the lease. - * Return 1 if succedded, 0 if not , and < 0 on errors. - */ -int -query(int fd, off_t offset) -{ - char curr[taglen+1] = ""; - char id[idlen+1] = ""; - long long ts; - time_t tsec; - int tusec; - char *t; - - if (readtag(fd, offset, curr, 0) < 0) - return -errno; - - querytag(curr, id, &ts); - tsec = ts / 1000000; - tusec = ts % 1000000; - - t = ctime(&tsec); - t[strlen(t)-1] = 0; - - printf("%s: ID %-*s TS %0*llx (%s, %d usec)\n", - sameid(curr, freetag) ? "FREE" : "LOCKED", - idlen, id, stamplen, ts, t, tusec); - - return 1; -} - -void -validate_path(const char *path) -{ - if (access(path, R_OK | W_OK) < 0) - PANIC("can't access '%s'", path); -} - -void -validate_id(const char *id) -{ - if (strlen(id) > idlen) - PANIC("id must be <= 8 characters"); - if (!strncmp(id, freetag, idlen)) - PANIC("can't lease free stamp"); -} - -void -validate_lease_params(int lease_ms, int op_max_ms) -{ - if (lease_ms <= 0 || op_max_ms <= 0 || lease_ms < op_max_ms || - op_max_ms < 1000 || op_max_ms % 1000 != 0) - PANIC("bad lease/op max timeouts"); -} - -/* - * Initialize the timeout to one op_max_ms. - */ -long long -renew_timeout(void) -{ - struct timeval tv; - - gettimeofday(&tv, 0); - return tv.tv_sec * 1000000ull + tv.tv_usec - (lease_ms - op_max_ms) * 1000; -} - -int -cmd_acquire(int argc, char **argv) -{ - int opt, fd, r, b = 0; - off_t offset = 0; - long long ts; - - optind = 0; - while ((opt = getopt(argc, argv, "+hdr:bo:")) != -1) { - switch (opt) { - case 'h': - usage(); - break; - case 'd': - debug++; - break; - case 'r': - request = optarg; - break; - case 'b': - b = 1; - break; - case 'o': - offset = strtoul(optarg, 0, 0); - break; - } - } - if (argc - optind < 4) - usage(); - - path = argv[optind++]; - validate_path(path); - id = argv[optind++]; - validate_id(id); - lease_ms = strtoul(argv[optind++], 0, 0); - op_max_ms = strtoul(argv[optind++], 0, 0); - validate_lease_params(lease_ms, op_max_ms); - - DEBUG("path '%s' offset %ld id '%s' lease_ms %ld op_max_ms %ld", - path, offset, id, lease_ms, op_max_ms); - - if ((fd = open(path, O_RDWR | O_DIRECT)) < 0) - panic("can't open '%s'", path); - - r = acquire(fd, offset, id, b, &ts); - - close(fd); - - if (r == 1) { - /* print last successful timestamp == aquire time */ - printf("%lld", ts); - DEBUG("Succeeded"); - return 0; - } else - DEBUG("%s (%s)", "Failed", strerror(r)); - - return 1; -} - -int -cmd_renew(int argc, char **argv) -{ - long long ts = renew_timeout(); - off_t offset = 0; - int opt, fd, r; - - optind = 0; - while ((opt = getopt(argc, argv, "+hdr:o:t:")) != -1) { - switch (opt) { - case 'h': - usage(); - break; - case 'd': - debug++; - break; - case 'r': - request = optarg; - break; - case 'o': - offset = strtoul(optarg, 0, 0); - break; - case 't': - ts = strtoll(optarg, 0, 0); - break; - } - } - if (argc - optind < 4) - usage(); - - path = argv[optind++]; - validate_path(path); - id = argv[optind++]; - validate_id(id); - lease_ms = strtoul(argv[optind++], 0, 0); - op_max_ms = strtoul(argv[optind++], 0, 0); - validate_lease_params(lease_ms, op_max_ms); - - DEBUG("path '%s' offset %ld id '%s' lease_ms %ld op_max_ms %ld", - path, offset, id, lease_ms, op_max_ms); - - if ((fd = open(path, O_RDWR | O_DIRECT)) < 0) - panic("can't open '%s'", path); - - r = renew(fd, offset, id, &ts); - - close(fd); - - /* print out the last successful renewal timestamp, or zero for don't renew */ - printf("%lld\n", ts); - - if (r == 1) { - DEBUG("Succeeded"); - return 0; - } - - DEBUG("%s (%s)", "Failed", strerror(r)); - return 1; -} - -int -cmd_release(int argc, char **argv) -{ - int opt, fd, r; - int force = 0; - off_t offset = 0; - - optind = 0; - while ((opt = getopt(argc, argv, "+hdfo:")) != -1) { - switch (opt) { - case 'h': - usage(); - break; - case 'd': - debug++; - break; - case 'f': - force++; - break; - case 'o': - offset = strtoul(optarg, 0, 0); - break; - } - } - if (argc - optind < 2) - usage(); - - path = argv[optind++]; - validate_path(path); - id = argv[optind++]; - validate_id(id); - - DEBUG("path '%s' offset %ld id '%s' force %d", path, offset, id, force); - - if ((fd = open(path, O_RDWR | O_DIRECT)) < 0) - panic("can't open '%s'", path); - - r = release(fd, offset, id, force); - - close(fd); - - if (r == 1) { - DEBUG("Succeeded"); - return 0; - } else - DEBUG("%s (%s)", "Failed", strerror(r)); - - return 1; -} - -int -cmd_query(int argc, char **argv) -{ - int opt, fd, r; - off_t offset = 0; - - optind = 0; - while ((opt = getopt(argc, argv, "+hdr:o:")) != -1) { - switch (opt) { - case 'h': - usage(); - break; - case 'd': - debug++; - break; - case 'r': - request = optarg; - break; - case 'o': - offset = strtoul(optarg, 0, 0); - break; - } - } - if (argc - optind < 4) - usage(); - - path = argv[optind++]; - validate_path(path); - - DEBUG("path '%s' offset %ld id '%s'", path, offset, id); - - if ((fd = open(path, O_RDWR | O_DIRECT)) < 0) - panic("can't open '%s'", path); - - r = query(fd, offset); - - close(fd); - - if (r == 1) { - DEBUG("Succeeded"); - return 0; - } else - DEBUG("%s (%s)", "Failed", strerror(r)); - - return 1; -} - -int -cmd_protect(int argc, char **argv) -{ - return 0; -} - -void -sig_handler(int sig) -{ - fprintf(stderr, "%s: Exiting due to signal %d\n", progname, sig); - exit(0); -} - -int -main(int argc, char **argv) -{ - int opt; - void *v = 0; - - signal(SIGTERM, sig_handler); - signal(SIGINT, sig_handler); - signal(SIGTRAP, sig_handler); - - if (posix_memalign(&v, 4096, 512) != 0) - { - fprintf(stderr, "fatal memory allocation error\n"); - return 1; - } - - iobuf = v; - memset(iobuf, 0, 512); - - progname = strrchr(argv[0], '/'); - if (!progname) - progname = argv[0]; - else - progname++; - - while ((opt = getopt(argc, argv, "+hd")) != -1) { - switch (opt) { - case 'h': - usage(); - break; - case 'd': - debug++; - break; - } - } - if (optind >= argc) - usage(); - - if (!strcmp(argv[optind], "acquire")) - return cmd_acquire(argc - optind, argv + optind); - if (!strcmp(argv[optind], "renew")) - return cmd_renew(argc - optind, argv + optind); - if (!strcmp(argv[optind], "release")) - return cmd_release(argc - optind, argv + optind); - if (!strcmp(argv[optind], "query")) - return cmd_query(argc - optind, argv + optind); - if (!strcmp(argv[optind], "protect")) - return cmd_protect(argc - optind, argv + optind); - - fprintf(stderr, "unknonwn op <%s>\n", argv[optind]); - usage(); - - return 1; -} - diff --git a/vdsm/storage/protect/spmprotect.sh b/vdsm/storage/protect/spmprotect.sh index f569a91..e864b34 100755 --- a/vdsm/storage/protect/spmprotect.sh +++ b/vdsm/storage/protect/spmprotect.sh @@ -24,7 +24,7 @@ SETSID="/usr/bin/setsid" LOGFILE="/var/log/vdsm/spm-lock.log" VDS_CLIENT="/usr/bin/vdsClient" -LEASE_UTIL="./safelease" +LEASE_UTIL="../safelease/safelease" KILL="/bin/kill" PKILL="/usr/bin/pkill" sdUUID=$2 -- To view, visit http://gerrit.ovirt.org/31297 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I94bfc8f9d93d8f7a73e69b9390329c342f2904e7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yoav Kleinberger <ykleinbe(a)redhat.com>

5 6

Change in vdsm[master]: vdsm: require kernel-2.6.32-465 on for NFSv4 fixes
by ykleinbe＠redhat.com 08 Oct '15

08 Oct '15

Yoav Kleinberger has uploaded a new change for review. Change subject: vdsm: require kernel-2.6.32-465 on for NFSv4 fixes ...................................................................... vdsm: require kernel-2.6.32-465 on for NFSv4 fixes This patch updates vdsm's dependencies to incorporate, on Fedora, fixes to NFSv4 introduces in https://bugzilla.redhat.com/show_bug.cgi?id=1075123. This patch fixes https://bugzilla.redhat.com/show_bug.cgi?id=1100285. Change-Id: Ib7fae41d22065061588ef72516070bf8b55bf526 Signed-off-by: Yoav Kleinberger <ykleinbe(a)redhat.com> --- M vdsm.spec.in 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/33810/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 0dcd82b..35cc892 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -200,7 +200,7 @@ Requires: device-mapper-multipath >= 0.4.9-52 Requires: e2fsprogs >= 1.41.12-11 Requires: fence-agents -Requires: kernel >= 2.6.32-279.9.1 +Requires: kernel >= 2.6.32-465.9.1 Requires: initscripts >= 9.03.31-2.el6_3.1 Requires: policycoreutils >= 2.0.83-19.30 Requires: policycoreutils-python >= 2.0.83-19.30 -- To view, visit http://gerrit.ovirt.org/33810 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib7fae41d22065061588ef72516070bf8b55bf526 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yoav Kleinberger <ykleinbe(a)redhat.com>

4 6

← Newer
1
...
5
6
7
8
9
10
11
...
75
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches March 2015