July 2015 - vdsm-patches - Fedora mailing-lists

Change in vdsm[master]: vdsm: expose release call to engine and via vdsClient
by Martin Polednik 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: vdsm: expose release call to engine and via vdsClient ...................................................................... vdsm: expose release call to engine and via vdsClient Release would, in ideal world, always be called when VM dies. Exposing this verbs gives users tool to force release if series of events lead to situation that should never occur(tm). Change-Id: I920ae8b5c82134f09a12e56a529fa3d30fd7ab53 Signed-off-by: Martin Polednik <mpolednik(a)redhat.com> --- M client/vdsClient.py M vdsm/API.py M vdsm/rpc/BindingXMLRPC.py 3 files changed, 17 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/94/32394/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 783afa6..85d00cd 100644 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -521,6 +521,10 @@ def do_getAllVmStats(self, args): return self.ExecAndExit(self.s.getAllVmStats()) + def hostdevRelease(self, args): + device_name = args[0] + return self.ExecAndExit(self.s.hostdevRelease(device_name)) + def desktopLogin(self, args): vmId, domain, user, password = tuple(args[:4]) if len(args) > 4: @@ -2149,6 +2153,10 @@ ('', 'Get Statistics info for all existing VMs' )), + 'hostdevRelease': (serv.do_hostdevRelease, + ('<deviceName>', + 'Release specified device from any VMs on the host' + )), 'getVGList': (serv.getVGList, ('storageType', 'List of all VGs.' diff --git a/vdsm/API.py b/vdsm/API.py index 8da6030..6269d72 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1296,6 +1296,10 @@ devices = hostdev.list_by_caps(self._cif.vmContainer, caps) return {'status': doneCode, 'deviceList': devices} + def hostdevRelease(self, device_name): + hostdev.release(device_name) + return {'status': doneCode} + def getStats(self): """ Report host statistics. diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index 4c1ef20..bdd0886 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -487,6 +487,10 @@ api = API.Global() return api.hostdevListByCaps(caps) + def hostdevRelease(self, device_name): + api = API.Global() + return api.hostdevRelease(device_name) + def vmGetIoTunePolicy(self, vmId): vm = API.VM(vmId) return vm.getIoTunePolicy() @@ -1004,6 +1008,7 @@ (self.vmGetStats, 'getVmStats'), (self.getAllVmStats, 'getAllVmStats'), (self.hostdevListByCaps, 'hostdevListByCaps'), + (self.hostdevRelease, 'hostdevRelease'), (self.vmMigrationCreate, 'migrationCreate'), (self.vmDesktopLogin, 'desktopLogin'), (self.vmDesktopLogoff, 'desktopLogoff'), -- To view, visit http://gerrit.ovirt.org/32394 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I920ae8b5c82134f09a12e56a529fa3d30fd7ab53 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

3 3

Change in vdsm[master]: vdsm hostdev: add support for USB devices
by mpoledni＠redhat.com 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: vdsm hostdev: add support for USB devices ...................................................................... vdsm hostdev: add support for USB devices Libvirt allows passthrough of USB devices (not busses) - this patch exposes the functionality in vdsm Change-Id: Iac74e7537d56bcb940ef07a4654d45cbcdbb1fb0 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M vdsm/caps.py M vdsm/rpc/vdsmapi-schema.json M vdsm/virt/vm.py 3 files changed, 71 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/54/29054/1 diff --git a/vdsm/caps.py b/vdsm/caps.py index d937d4e..8c16af6 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -530,7 +530,12 @@ # back that we could use to uniquely identify and initiate a device continue - if capability in ('pci',): + if capability in ('pci', 'usb_device'): + # Libvirt only allows to attach USB device with capability 'usb', + # but the bus identifies itself as 'usb' while device as + # 'usb_device' + if dev['capability'] == 'usb_device': + dev['capability'] = 'usb' devices.append(dev) return devices diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 921f3d4..ac10144 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -3156,33 +3156,55 @@ 'specParams': 'VmRngDeviceSpecParams'}} ## +# @StartupPolicy: +# +# Possible policies for startup with device +# +# @mandatory: fail if missing for any reason (the default) +# +# @requisite: fail if missing on boot up, drop if missing +# on migrate/restore/revert +# +# @optional: drop if missing at any start attempt +# +# Since: 4.16.0 +## +{'enum': 'StartupPolicy', 'data': ['mandatory', 'requisite', 'optional']} + +## # @HostDeviceCapability: # # Properties of a host device. # # @pci: PCI device # +# @usb: USB device +# # Since: 4.16.0 ## -{'enum': 'HostDeviceCapability', 'data': ['pci']} +{'enum': 'HostDeviceCapability', 'data': ['pci', 'usb']} ## # @HostDeviceSpecParams: # # Properties of a host device. # -# @bootorder: #optional If specified, this device is part of the boot -# sequence at the specified position +# @bootorder: #optional If specified, this device is part of the boot +# sequence at the specified position (for @pci and @usb) # -# @bar: #optional ROM visibility in the guest's memory map (for @pci) +# @bar: #optional ROM visibility in the guest's +# memory map (for @pci) # -# @file: #optional Binary file to be used as device's ROM (for @pci) +# @file: #optional Binary file to be used as device's ROM (for @pci) +# +# @startupPolicy: #optional Possible boot handling with attached device +# (for @usb) # # Since: 4.16.0 ## {'type': 'HostDeviceSpecParams', - 'data': {'*bootorder': 'int', '*bar': 'bool', - '*file': 'str'}} + 'data': {'*bootorder': 'int', '*bar': 'bool', '*file': 'str', + '*startupPolicy': 'StartupPolicy'}} ## # @HostDevice: diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 6143cfa..6a6978e 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1631,13 +1631,16 @@ self.log.debug('Detaching hostdev %s', self.name) self._node.dettach() - def getPciAddr(self): + def _parsecaps(self): capsxml = _domParseStr(self._node.XMLDesc(0)).childNodes[0] self.log.debug('Got nodeDevice XML:\n%s', capsxml.toprettyxml(encoding='utf-8')) - capsxml = capsxml.getElementsByTagName('capability')[0] + return capsxml.getElementsByTagName('capability')[0] + + def getPciAddr(self): + capsxml = self._parsecaps() domain = capsxml.getElementsByTagName('domain')[0]. \ firstChild.nodeValue bus = capsxml.getElementsByTagName('bus')[0].firstChild.nodeValue @@ -1646,6 +1649,26 @@ self.log.debug('PCI device %s at address {domain: %s bus: %s ' 'slot: %s}', self.name, domain, bus, slot) return {'domain': domain, 'bus': bus, 'slot': slot} + + def getUsbAddr(self): + capsxml = self._parsecaps() + addr = {} + + addr['bus'] = capsxml.getElementsByTagName('bus')[0].firstChild. \ + nodeValue + addr['device'] = capsxml.getElementsByTagName('device')[0]. \ + firstChild.nodeValue + # TODO: handle nonexistant product_id and vendor_id by not adding them + # to addr + addr['product_id'] = capsxml.getElementsByTagName('product')[0].\ + getAttribute('id') + addr['vendor_id'] = capsxml.getElementsByTagName('vendor')[0].\ + getAttribute('id') + + self.log.debug('USB device %s {product: %s, vendor: %s} at address ' + '{bus: %s device: %s}', self.name, addr['product_id'], + addr['vendor_id'], addr['bus'], addr['device']) + return addr def getXML(self): """ @@ -1683,6 +1706,17 @@ rom.setAttrs(**romAttrs) + elif self.capability == 'usb_device': + addr = self.getPciAddr() + try: + source.appendChildWithArgs('vendor', None, addr['vendor']) + source.appendChildWithArgs('vendor', None, addr['product']) + except: + source.appendChildWithArgs('address', None, **addr) + + if 'startupPolicy' in self.specParams: + source.setAttrs(startupPolicy=self.specParams['startupPolicy']) + return hostdev -- To view, visit http://gerrit.ovirt.org/29054 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iac74e7537d56bcb940ef07a4654d45cbcdbb1fb0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpoledni(a)redhat.com>

5 5

Change in vdsm[master]: vdsm hostdev: add support for SCSI devices
by mpoledni＠redhat.com 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: vdsm hostdev: add support for SCSI devices ...................................................................... vdsm hostdev: add support for SCSI devices Libvirt allows passthrough of SCSI devices - this patch exposes the functionality in vdsm Change-Id: Ia953bcd5eda1b97235a8dd2f5f9593d8f302e5d6 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M vdsm/caps.py M vdsm/rpc/vdsmapi-schema.json M vdsm/virt/vm.py 3 files changed, 51 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/58/29058/1 diff --git a/vdsm/caps.py b/vdsm/caps.py index 8c16af6..6112763 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -530,7 +530,7 @@ # back that we could use to uniquely identify and initiate a device continue - if capability in ('pci', 'usb_device'): + if capability in ('pci', 'usb_device', 'scsi'): # Libvirt only allows to attach USB device with capability 'usb', # but the bus identifies itself as 'usb' while device as # 'usb_device' diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index ac10144..52f20c1 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -3180,9 +3180,11 @@ # # @usb: USB device # +# @scsi: SCSI device +# # Since: 4.16.0 ## -{'enum': 'HostDeviceCapability', 'data': ['pci', 'usb']} +{'enum': 'HostDeviceCapability', 'data': ['pci', 'usb', 'scsi']} ## # @HostDeviceSpecParams: @@ -3200,11 +3202,18 @@ # @startupPolicy: #optional Possible boot handling with attached device # (for @usb) # +# @readonly #optional If present, indicates that the device is read +# only (for @scsi) +# +# @shareable #optional If present, this indicates the device is +# expected to be shared between domains (for @scsi) +# # Since: 4.16.0 ## {'type': 'HostDeviceSpecParams', 'data': {'*bootorder': 'int', '*bar': 'bool', '*file': 'str', - '*startupPolicy': 'StartupPolicy'}} + '*startupPolicy': 'StartupPolicy', '*shareable': 'bool', + '*readonly*': 'bool'}} ## # @HostDevice: diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 6a6978e..3cb2eb9 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1670,6 +1670,33 @@ addr['vendor_id'], addr['bus'], addr['device']) return addr + def getScsiAddr(self): + capsxml = self._parsecaps() + addr = {} + + addr['type'] = 'scsi' + addr['bus'] = capsxml.getElementsByTagName('bus')[0].firstChild. \ + nodeValue + addr['target'] = capsxml.getElementsByTagName('target')[0]. \ + firstChild.nodeValue + addr['unit'] = capsxml.getElementsByTagName('lun')[0]. \ + firstChild.nodeValue + + self.log.debug('SCSI device %s at address ' + '{bus: %s, target: %s, unit: %s}', + self.name, addr['bus'], addr['target'], addr['unit'], + addr['device']) + return addr + + def getScsiAdapter(self): + capsxml = self._parsecaps() + + adapter = 'scsi_host{}'.format( + capsxml.getElementsByTagName('host')[0].firstChild.nodeValue) + + self.log.debug('SCSI device %s adapter %s', self.name, adapter) + return adapter + def getXML(self): """ Create domxml for a hostdev device. @@ -1717,6 +1744,18 @@ if 'startupPolicy' in self.specParams: source.setAttrs(startupPolicy=self.specParams['startupPolicy']) + elif self.capability == 'scsi': + source.appendChildWithArgs('address', None, + **self.getScsiHost()) + source.appendChildWithArgs('adapter', None, + **self.getScsiAdapter()) + + if 'readonly' in self.specParams: + hostdev.appendChild('readonly') + + if 'shareable' in self.specParams: + hostdev.appendChild('shareable') + return hostdev -- To view, visit http://gerrit.ovirt.org/29058 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia953bcd5eda1b97235a8dd2f5f9593d8f302e5d6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpoledni(a)redhat.com>

5 9

Change in vdsm[master]: VDSM: implement nodeDeviceMapper
by Martin Polednik 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: VDSM: implement nodeDeviceMapper ...................................................................... VDSM: implement nodeDeviceMapper NodeDeviceMapper is structure to keep track of available node devices and provide easy access to querying and managing them. The mapper is needed in order to allow VDSM to meaningfully report the usage of these devices, along with managing their availability. Change-Id: I6f21d465a90cfe2eb16ba70943e5fcf1683f1656 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M client/vdsClient.py M debian/vdsm.install M vdsm.spec.in M vdsm/API.py M vdsm/clientIF.py M vdsm/rpc/BindingXMLRPC.py M vdsm/virt/Makefile.am 7 files changed, 44 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/30209/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 2c09b28..89dceab 100644 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -461,6 +461,14 @@ def do_getAllVmStats(self, args): return self.ExecAndExit(self.s.getAllVmStats()) + def do_getDevicesByCaps(self, args): + caps = list(args) + return self.ExecAndExit(self.s.getDevicesByCaps(caps)) + + def do_getDevicesByDomain(self, args): + vmId = args[0] + return self.ExecAndExit(self.s.getDevicesByDomain(vmId)) + def desktopLogin(self, args): vmId, domain, user, password = tuple(args) response = self.s.desktopLogin(vmId, domain, user, password) @@ -2077,6 +2085,16 @@ ('', 'Get Statistics info for all existing VMs' )), + 'getDevicesByCaps': (serv.do_getDevicesByCaps, + ('[<caps>]', + 'Get available node devices on the host with ' + 'given capability' + )), + 'getDevicesByDomain': (serv.do_getDevicesByDomain, + ('<vmId>', + 'Get available node devices attached to specified ' + 'domain' + )), 'getVGList': (serv.getVGList, ('storageType', 'List of all VGs.' diff --git a/debian/vdsm.install b/debian/vdsm.install index 3af1100..e1d8652 100644 --- a/debian/vdsm.install +++ b/debian/vdsm.install @@ -142,6 +142,7 @@ ./usr/share/vdsm/virt/__init__.py ./usr/share/vdsm/virt/guestagent.py ./usr/share/vdsm/virt/migration.py +./usr/share/vdsm/virt/nodedev.py ./usr/share/vdsm/virt/sampling.py ./usr/share/vdsm/virt/vm.py ./usr/share/vdsm/virt/vmchannels.py diff --git a/vdsm.spec.in b/vdsm.spec.in index 16b7834..0ddacbe 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -939,6 +939,7 @@ %{_datadir}/%{vdsm_name}/virt/__init__.py* %{_datadir}/%{vdsm_name}/virt/guestagent.py* %{_datadir}/%{vdsm_name}/virt/migration.py* +%{_datadir}/%{vdsm_name}/virt/nodedev.py* %{_datadir}/%{vdsm_name}/virt/vmchannels.py* %{_datadir}/%{vdsm_name}/virt/vmstatus.py* %{_datadir}/%{vdsm_name}/virt/vmtune.py* diff --git a/vdsm/API.py b/vdsm/API.py index 6feccf6..e891b80 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1244,6 +1244,18 @@ statsList = hooks.after_get_all_vm_stats(statsList) return {'status': doneCode, 'statsList': statsList} + def getDevicesByCaps(self, caps): + """ + """ + devices = self._cif.nodeDeviceMapper.getDevicesByCaps(caps) + return {'status': doneCode, 'devices': devices} + + def getDevicesByDomain(self, vmId): + """ + """ + devices = self._cif.nodeDeviceMapper.getDevicesByDomain(vmId) + return {'status': doneCode, 'devices': devices} + def getStats(self): """ Report host statistics. diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index d5372f3..118652d 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -91,6 +91,7 @@ self.gluster = None try: self.vmContainer = {} + self.nodeDeviceMapper = NodeDeviceMapper(vmContainer, log) self._hostStats = sampling.HostStatsThread(log=log) self._hostStats.start() self.lastRemoteAccess = 0 diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py index d6663c3..a31d76c 100644 --- a/vdsm/rpc/BindingXMLRPC.py +++ b/vdsm/rpc/BindingXMLRPC.py @@ -482,6 +482,14 @@ api = API.Global() return api.getAllVmStats() + def getDevicesByCaps(self, caps): + api = API.Global() + return api.getDevicesByCaps(caps) + + def getDevicesByDomain(self, vmId): + api = API.Global() + return api.getDevicesByDomain(vmId) + def vmGetIoTunePolicy(self, vmId): vm = API.VM(vmId) return vm.getIoTunePolicy() @@ -989,6 +997,8 @@ (self.getStats, 'getVdsStats'), (self.vmGetStats, 'getVmStats'), (self.getAllVmStats, 'getAllVmStats'), + (self.getDevicesByCaps, 'getDevicesByCaps'), + (self.getDevicesByDomain, 'getDevicesByDomain'), (self.vmMigrationCreate, 'migrationCreate'), (self.vmDesktopLogin, 'desktopLogin'), (self.vmDesktopLogoff, 'desktopLogoff'), diff --git a/vdsm/virt/Makefile.am b/vdsm/virt/Makefile.am index 423839b..4b7b1cc 100644 --- a/vdsm/virt/Makefile.am +++ b/vdsm/virt/Makefile.am @@ -25,6 +25,7 @@ __init__.py \ guestagent.py \ migration.py \ + nodedev.py \ sampling.py \ vm.py \ vmchannels.py \ -- To view, visit http://gerrit.ovirt.org/30209 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6f21d465a90cfe2eb16ba70943e5fcf1683f1656 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

6 22

Change in vdsm[master]: [WIP] vdsm: add support for PCI passthrough
by mpoledni＠redhat.com 01 Sep '15

01 Sep '15

Martin Polednik has uploaded a new change for review. Change subject: [WIP] vdsm: add support for PCI passthrough ...................................................................... [WIP] vdsm: add support for PCI passthrough required functionality: * report PCI devices available on host [x] * handle createVm xml generation [x] * hotplugHostdev [ ] (required for after-migration) * hotpunlugHostdev [ ] (required for migration) Change-Id: I363d2622d72ca2db75f60032fe0892c348bab121 Signed-off-by: Martin Polednik <mpoledni(a)redhat.com> --- M lib/vdsm/define.py M vdsm/caps.py M vdsm/vm.py 3 files changed, 83 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/62/22462/1 diff --git a/lib/vdsm/define.py b/lib/vdsm/define.py index eb78633..9605f93 100644 --- a/lib/vdsm/define.py +++ b/lib/vdsm/define.py @@ -132,6 +132,12 @@ 'transientErr': {'status': { 'code': 59, 'message': 'Action not permitted on a VM with transient disks'}}, + 'hotplugHostdev': {'status': { + 'code': 60, + 'message': 'Failed to hotplug hostdev'}}, + 'hotunplugHostdev': {'status': { + 'code': 61, + 'message': 'Failed to hotunplug hostdev'}}, 'recovery': {'status': { 'code': 99, 'message': 'Recovering from crash or Initializing'}}, diff --git a/vdsm/caps.py b/vdsm/caps.py index 3839134..d6af375 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -308,6 +308,38 @@ return dict(release=release, version=version, name=osname) +def hostdevList(): + devices = [] + for device in libvirtconnection.get().listAllDevices(): + devXML = minidom.parseString(device.XMLDesc()) + dev = {} + + # we have to grab attributes that will most likely not only + # uniquely identify device, but also serve as human readable + # representation of the device + try: + dev['name'] = devXML.getElementsByTagName('name')[0].\ + childNodes[0].data + capability = devXML.getElementsByTagName('capability')[0] + try: + dev['product'] = capability.getElementsByTagName('product')[0]\ + .childNodes[0].data + dev['vendor'] = capability.getElementsByTagName('vendor')[0].\ + childNodes[0].data + except IndexError: + # althought the retrieval of product/vendor was not successful, + # we can still report back the name + pass + except IndexError: + # should device not have a name, there is nothing engine could send + # back that we could use to uniquely identify and initiate a device + continue + + devices.append(dev) + + return devices + + def get(): targetArch = platform.machine() @@ -360,6 +392,7 @@ config.getint('vars', 'extra_mem_reserve')) caps['guestOverhead'] = config.get('vars', 'guest_ram_overhead') caps['rngSources'] = _getRngSources() + caps['hostDevices'] = hostdevList() return caps diff --git a/vdsm/vm.py b/vdsm/vm.py index a5d923b..a477bc9 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -78,6 +78,7 @@ WATCHDOG_DEVICES = 'watchdog' CONSOLE_DEVICES = 'console' SMARTCARD_DEVICES = 'smartcard' +HOSTDEV_DEVICES = 'hostdev' def isVdsmImage(drive): @@ -1656,6 +1657,27 @@ return m +class HostDevice(VmDevice): + def getXML(self): + """ + Create domxml for a hostdev device. + + <devices> + <hostdev mode='subsystem' type='usb'> + <source startupPolicy='optional'> + <vendor id='0x1234'/> + <product id='0xbeef'/> + </source> + <boot order='2'/> + </hostdev> + </devices> + """ + # libvirt gives us direct api call to construct the XML + return xml.dom.minidom.parseString(libvirtconnection.get(). + nodeDeviceLookupByName(self.name). + XMLDesc()) + + class WatchdogDevice(VmDevice): def __init__(self, *args, **kwargs): super(WatchdogDevice, self).__init__(*args, **kwargs) @@ -1769,7 +1791,8 @@ (CONSOLE_DEVICES, ConsoleDevice), (REDIR_DEVICES, RedirDevice), (RNG_DEVICES, RngDevice), - (SMARTCARD_DEVICES, SmartCardDevice)) + (SMARTCARD_DEVICES, SmartCardDevice), + (HOSTDEV_DEVICES, HostDevice)) def _makeDeviceDict(self): return dict((dev, []) for dev, _ in self.DeviceMapping) @@ -3127,6 +3150,26 @@ break + def hotplugHostdev(self, params): + hostdev = HostDevice(self.conf, self.log, **params) + self._devices[HOSTDEV_DEVICES].append(hostdev) + hostdevXML = hostdev.getXML().toprettyxml(encoding='utf-8') + hostdev._deviceXML = hostdevXML + self.log.debug("Hotplug hostdev xml: %s", hostdevXML) + + try: + self._dom.attachDevice(hostdevXML) + except libvirt.libvirtError as e: + self.log.error("Hotplug failed", exc_info=True) + if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: + return errCode['noVM'] + return {'status': {'code': + errCode['hotplugHostdev']['status']['code'], + 'message': e.message}} + + def hotunplugHostdev(self, name): + pass + def hotplugNic(self, params): if self.isMigrating(): return errCode['migInProgress'] -- To view, visit http://gerrit.ovirt.org/22462 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I363d2622d72ca2db75f60032fe0892c348bab121 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpoledni(a)redhat.com>

8 50

Change in vdsm[master]: virt: removing naming confilct in vmstats.py
by mmirecki＠redhat.com 31 Aug '15

31 Aug '15

Marcin Mirecki has uploaded a new change for review. Change subject: virt: removing naming confilct in vmstats.py ...................................................................... virt: removing naming confilct in vmstats.py variable name 'nic' confilcts with function 'nic' renaming variable to 'vm_nic' Change-Id: I77307af135e0f6c3d8c5554e0f3fb4093e36ed0d Signed-off-by: Marcin Mirecki <mmirecki(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/43657/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index b9caeac..53a909f 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -176,17 +176,17 @@ if first_sample is None or last_sample is None: return - for nic in vm.getNicDevices(): - if nic.name.startswith('hostdev'): + for vm_nic in vm.getNicDevices(): + if vm_nic.name.startswith('hostdev'): continue - first_nic = first_sample.get('net', {}).get(nic.name, {}) - last_nic = last_sample.get('net', {}).get(nic.name, {}) + first_nic = first_sample.get('net', {}).get(vm_nic.name, {}) + last_nic = last_sample.get('net', {}).get(vm_nic.name, {}) # may happen if nic is a new hot-plugged one if not first_nic or not last_nic: continue - stats['network'][nic.name] = nic( - nic.name, nic.nicModel, nic.macAddr, + stats['network'][vm_nic.name] = nic( + vm_nic.name, vm_nic.nicModel, vm_nic.macAddr, first_nic, last_nic, interval) -- To view, visit https://gerrit.ovirt.org/43657 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I77307af135e0f6c3d8c5554e0f3fb4093e36ed0d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Marcin Mirecki <mmirecki(a)redhat.com>

4 7

Change in vdsm[master]: shell_helper: remove redundant ./configure calls
by ibarkan＠redhat.com 31 Aug '15

31 Aug '15

Ido Barkan has uploaded a new change for review. Change subject: shell_helper: remove redundant ./configure calls ...................................................................... shell_helper: remove redundant ./configure calls autogen calls configure anyway Change-Id: Ic3d93ef7da9d16ffc57b94e15d47285e7fe18ad6 Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M contrib/shell_helper 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/98/44198/1 diff --git a/contrib/shell_helper b/contrib/shell_helper index 7dc2984..5c10e58 100755 --- a/contrib/shell_helper +++ b/contrib/shell_helper @@ -92,7 +92,7 @@ vdsm_build_with_hooks() { pushd $VDSM_DIR - ./autogen.sh --system && ./configure --enable-hooks && make \ + ./autogen.sh --system --enable-hooks && make \ && NOSE_EXCLUDE=.* make rpm popd } @@ -189,7 +189,7 @@ vdsm_pep8() { pushd $VDSM_DIR - ./autogen.sh --system && ./configure && make pep8 + ./autogen.sh --system && make pep8 popd } -- To view, visit https://gerrit.ovirt.org/44198 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic3d93ef7da9d16ffc57b94e15d47285e7fe18ad6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 3

Change in vdsm[master]: netinfo: improve which ipv4 addr is reported if there are mu...
by asegurap＠redhat.com 30 Aug '15

30 Aug '15

Antoni Segura Puimedon has uploaded a new change for review. Change subject: netinfo: improve which ipv4 addr is reported if there are multiple primary ...................................................................... netinfo: improve which ipv4 addr is reported if there are multiple primary The current code assumed that additional configured addresses for a device would have the 'secondary' flag. However, this is no longer true in recent kernels, as multiple primary addresses can be set for a device. The improvement is that now we will check if any of the addresses is in the subnet of the gateway and report one of them. If there is no default gw in the main table for the device we return the last set primary ip and if there is a default gw but going through another hop, we return the first set ip. Change-Id: I8666cfef5bd8ea63edf8979e501d4785db5f4893 Signed-off-by: Antoni S. Puimedon <asegurap(a)redhat.com> --- M lib/vdsm/netinfo.py 1 file changed, 35 insertions(+), 7 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/75/33375/1 diff --git a/lib/vdsm/netinfo.py b/lib/vdsm/netinfo.py index 6491505..816810d 100644 --- a/lib/vdsm/netinfo.py +++ b/lib/vdsm/netinfo.py @@ -292,25 +292,49 @@ struct.pack("!I", int('1' * prefix + '0' * (32 - prefix), 2))) +def prefix2int(prefix): + if not 0 <= prefix <= 32: + raise ValueError('%s is not a valid prefix value. It must be between ' + '0 and 32') + return (2 ** prefix - 1) << (32 - prefix) + + +def addr2int(address): + return struct.unpack('!I', socket.inet_aton(address))[0] + + def getDefaultGateway(): output = routeShowGateways('main') return Route.fromText(output[0]) if output else None -def getIpInfo(dev, ipaddrs=None): +def getIpInfo(dev, ipaddrs=None, ipv4_gateway=None): if ipaddrs is None: ipaddrs = _getIpAddrs() ipv4addr = ipv4netmask = '' ipv4addrs = [] ipv6addrs = [] + gateway_int = addr2int(ipv4_gateway) if ipv4_gateway else None + for addr in ipaddrs[dev]: if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') - ipv4netmask = prefix2netmask(addr['prefixlen']) + address, _ = addr['address'].split('/') + mask = prefix2int(addr['prefixlen']) + if (gateway_int is None or + addr2int(address) & mask == gateway_int & mask): + ipv4addr = address + ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address']) + if ipv4addrs and ipv4addr == '': + # If we didn't find an address in the gateway subnet (which is + # legal if there is another route that takes us to the gateway) we + # choose to report the first address + ipv4addr, prefix = ipv4addrs[0].split('/') + ipv4netmask = prefix2netmask(int(prefix)) + return ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs @@ -501,14 +525,16 @@ # comment when the version is no longer supported. data['interface'] = iface - ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo(iface, ipaddrs) + gateway = _get_gateway(routes, iface) + ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo( + iface, ipaddrs, gateway) data.update({'iface': iface, 'bridged': bridged, 'addr': ipv4addr, 'netmask': ipv4netmask, 'bootproto4': ('dhcp' if ipv4addr and iface in dhcp4 else 'none'), 'ipv4addrs': ipv4addrs, 'ipv6addrs': ipv6addrs, - 'gateway': _get_gateway(routes, iface), + 'gateway': gateway, 'ipv6gateway': _get_gateway(routes, iface, family=6), 'mtu': str(getMtu(iface))}) except (IOError, OSError) as e: @@ -549,12 +575,14 @@ def _devinfo(link, routes, ipaddrs, dhcp4): - ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo(link.name, ipaddrs) + gateway = _get_gateway(routes, link.name) + ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo( + link.name, ipaddrs, gateway) info = {'addr': ipv4addr, 'cfg': getIfaceCfg(link.name), 'ipv4addrs': ipv4addrs, 'ipv6addrs': ipv6addrs, - 'gateway': _get_gateway(routes, link.name), + 'gateway': gateway, 'ipv6gateway': _get_gateway(routes, link.name, family=6), 'bootproto4': ('dhcp' if ipv4addr and link.name in dhcp4 else 'none'), -- To view, visit http://gerrit.ovirt.org/33375 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8666cfef5bd8ea63edf8979e501d4785db5f4893 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Antoni Segura Puimedon <asegurap(a)redhat.com>

7 38

Change in vdsm[master]: tests: add tests for disk hotplug/hotunplug
by fromani＠redhat.com 26 Aug '15

26 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: tests: add tests for disk hotplug/hotunplug ...................................................................... tests: add tests for disk hotplug/hotunplug Add tests for negative flows. Negative flows are tested less often because of the need to cause or inject faults. Moreover, we want this test to help verify the upcoming switch to response.error(). Change-Id: Idd8b467d49c183c81f6a11494d38984246f5b527 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/vmTests.py 1 file changed, 74 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/39053/1 diff --git a/tests/vmTests.py b/tests/vmTests.py index ab26135..9ca6655 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -53,6 +53,7 @@ from vmTestsData import CONF_TO_DOMXML_PPC64 from vmTestsData import CONF_TO_DOMXML_NO_VDSM import vmfakelib as fake +from vmStorageTests import drive_config from testValidation import slowtest @@ -1417,6 +1418,7 @@ self.assertEqual(NAME, testvm.getStats()['vmName']) +@expandPermutations class ChangeBlockDevTests(TestCaseBase): def test_change_cd_failure(self): @@ -1431,12 +1433,82 @@ virtError=libvirt.VIR_ERR_GET_FAILED) res = fakevm.changeCD({}) + self.assertEqual(res, response.error('changeDisk')) - expected_status = define.errCode['changeDisk']['status'] - self.assertEqual(res['status'], expected_status) + # TODO: need a way to pass message through fake.Domain failIfRequested + @permutations([[libvirt.VIR_ERR_OPERATION_DENIED, 'hotplugDisk', ''], + [libvirt.VIR_ERR_NO_DOMAIN, 'noVM', '']]) + def test_hotplug_disk_failure(self, virtError, vdsmError, customMessage): + # caveat emptor: in this test we DON't + # overwrite the message, thus will be the default one; but for + # errors != 'noVM', it can change. + PATH = '/this/path/does/not/exists' + + def _fail(*args): + # no specific meaning, actually any error != None is good + raise_libvirt_error(libvirt.VIR_ERR_ACCESS_DENIED, + 'Failed to hotplug disk') + + dom = fake.Domain() + dom.updateDeviceFlags = _fail + dom.attachDevice = _fail + + cif = fake.ClientIF() + + with MonkeyPatchScope([(cif, 'prepareVolumePath', + lambda _: PATH), + (cif, 'teardownVolumePath', + lambda _: None), + (hooks, 'before_disk_hotplug', + _fake_disk_hook), + (utils, 'isBlockDevice', + lambda path: True)]): + with fake.VM(devices=[drive_config()], + cif=cif) as fakevm: + fakevm._dom = dom + + res = fakevm.hotplugDisk({ + 'drive': drive_config( + path=PATH, + serial='65-a561-23e5b495b9bc')}) + + self.assertEqual(res, + response.error('hotplugDisk', customMessage)) + + def test_hotunplug_inexistent_disk(self): + PATH = '/this/path/does/not/exists' + + def _fail(*args): + raise_libvirt_error(libvirt.VIR_ERR_ACCESS_DENIED, '') + + dom = fake.Domain() + dom.updateDeviceFlags = _fail + + cif = fake.ClientIF() + with MonkeyPatchScope([(cif, 'prepareVolumePath', + lambda _: PATH), + (cif, 'teardownVolumePath', + lambda _: None), + (hooks, 'before_disk_hotunplug', + _fake_disk_hook), + (utils, 'isBlockDevice', + lambda path: True)]): + with fake.VM(cif=cif) as fakevm: + # no specific meaning, actually any error != None is good + fakevm._dom = dom + + res = fakevm.hotunplugDisk({}) + + self.assertEqual(res, + response.error('hotunplugDisk', + 'Disk not found')) def raise_libvirt_error(code, message): err = libvirt.libvirtError(defmsg=message) err.err = [code] raise err + + +def _fake_disk_hook(driveXml, conf, params): + return driveXml -- To view, visit https://gerrit.ovirt.org/39053 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idd8b467d49c183c81f6a11494d38984246f5b527 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 34

Change in vdsm[master]: virt: fully convert to response module
by fromani＠redhat.com 26 Aug '15

26 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: virt: fully convert to response module ...................................................................... virt: fully convert to response module convert the missing bits to response.{error,success} for the sake of code clarity. Change-Id: If1bb95b46197ba4d6c8eccaed33fd77c426a084c Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py M vdsm/virt/vmpowerdown.py 2 files changed, 35 insertions(+), 37 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/00/40900/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index ce058a0..01c2c47 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -42,7 +42,7 @@ from vdsm import utils from vdsm.compat import pickle from vdsm.config import config -from vdsm.define import ERROR, NORMAL, doneCode +from vdsm.define import ERROR, NORMAL from vdsm.netinfo import DUMMY_BRIDGE from storage import outOfProcess as oop from storage import sd @@ -1093,7 +1093,7 @@ del self.conf['pauseCode'] except KeyError: pass - return {'status': doneCode, 'output': ['']} + return response.success(output=['']) finally: if not guestCpuLocked: self._guestCpuLock.release() @@ -1110,7 +1110,7 @@ guestCpuLocked=True) self._logGuestCpuStatus('pause') self._lastStatus = afterState - return {'status': doneCode, 'output': ['']} + return response.success(output=['']) finally: if not guestCpuLocked: self._guestCpuLock.release() @@ -1919,7 +1919,7 @@ self.hotunplugNic({'nic': nicParams}) return response.error('hotplugNic', e.message) - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def _lookupDeviceByAlias(self, devType, alias): for dev in self._devices[devType][:]: @@ -1977,7 +1977,7 @@ with self.setLinkAndNetwork(netDev, netConf, linkValue, network, custom, specParams): with self.updatePortMirroring(netConf, netsToMirror): - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) except (LookupError, SetLinkAndNetworkError, UpdatePortMirroringError) as e: @@ -2148,7 +2148,7 @@ hooks.after_nic_hotunplug(nicXml, self.conf, params=nic.custom) - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def setNumberOfCpus(self, numberOfCpus): @@ -2169,7 +2169,7 @@ self.conf['smp'] = str(numberOfCpus) self.saveState() hooks.after_set_num_of_cpus() - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def _updateVcpuLimit(self): qos = self._getVmPolicy() @@ -2275,7 +2275,7 @@ else: return response.error('updateVmPolicyErr', e.message) - return {'status': doneCode} + return response.success() def _getVmPolicy(self): """ @@ -2380,7 +2380,7 @@ found_device.name, xml) found_device._deviceXML = xml - return {'status': doneCode} + return response.success() def _createTransientDisk(self, diskParams): if (diskParams.get('shared', None) != @@ -2464,7 +2464,7 @@ hooks.after_disk_hotplug(driveXml, self.conf, params=drive.custom) - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def hotunplugDisk(self, params): if self.isMigrating(): @@ -2521,7 +2521,7 @@ params=drive.custom) self._cleanupDrives(drive) - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def _readPauseCode(self): state, reason = self._dom.state(0) @@ -2864,7 +2864,7 @@ # If all the drives are the current ones, return success if len(newDrives) == 0: self.log.debug('all the drives are already in use, success') - return {'status': doneCode} + return response.success() preparedDrives = {} @@ -2970,9 +2970,10 @@ # Returning quiesce to notify the manager whether the guest agent # froze and flushed the filesystems or not. - return {'status': doneCode, 'quiesce': - (snapFlags & libvirt.VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE - == libvirt.VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE)} + guestDidQuiesce = ( + snapFlags & libvirt.VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE + == libvirt.VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE) + return response.success(quiesce=guestDidQuiesce) def _setDiskReplica(self, srcDrive, dstDisk): """ @@ -3064,7 +3065,7 @@ self.log.exception("Initial extension request failed for %s", srcDrive.name) - return {'status': doneCode} + return response.success() def diskReplicateFinish(self, srcDisk, dstDisk): try: @@ -3148,7 +3149,7 @@ self._delDiskReplica(srcDrive) self.startDisksStatsCollection() - return {'status': doneCode} + return response.success() def _startDriveReplication(self, drive, replica): flags = (libvirt.VIR_DOMAIN_BLOCK_REBASE_COPY | @@ -3198,7 +3199,7 @@ self._setVolumeSize(drive.domainID, drive.poolID, drive.imageID, drive.volumeID, newVirtualSize) - return {'status': doneCode, 'size': str(newVirtualSize)} + return response.success(size=str(newVirtualSize)) def _diskSizeExtendRaw(self, drive, newSizeBytes): # Picking up the volume size extension @@ -3231,7 +3232,7 @@ "reboot", volSize.apparentsize, exc_info=True) return response.error('updateDevice') - return {'status': doneCode, 'size': str(volSize.apparentsize)} + return response.success(size=str(volSize.apparentsize)) def diskSizeExtend(self, driveSpecs, newSizeBytes): try: @@ -3309,7 +3310,7 @@ self.cif.teardownVolumePath(self.conf[vmDev]) self.conf[vmDev] = path - return {'status': doneCode, 'vmList': self.status()} + return response.success(vmList=self.status()) def setTicket(self, otp, seconds, connAct, params): """ @@ -3338,7 +3339,7 @@ res = response.error('ticketErr', unicode(tmo)) else: hooks.after_vm_set_ticket(self._domain.xml, self.conf, params) - res = {'status': doneCode} + res = response.success() return res def _reviveTicket(self, newlife): @@ -3443,7 +3444,7 @@ with self._releaseLock: if self._released: - return {'status': doneCode} + return response.success() self.log.info('Release VM resources') self.lastStatus = vmstatus.POWERING_DOWN @@ -3473,7 +3474,7 @@ self._released = True - return {'status': doneCode} + return response.success() def _destroyVmGraceful(self): try: @@ -3490,7 +3491,7 @@ if e.get_error_code() == libvirt.VIR_ERR_OPERATION_FAILED: return self._destroyVmForceful() return response.error('destroyErr') - return {'status': doneCode} + return response.success() def _destroyVmForceful(self): try: @@ -3499,7 +3500,7 @@ self.log.warning("Failed to destroy VM '%s'", self.conf['vmId'], exc_info=True) return response.error('destroyErr') - return {'status': doneCode} + return response.success() def deleteVm(self): """ @@ -3521,7 +3522,7 @@ # Clean VM from the system self.deleteVm() - return {'status': doneCode} + return response.success() def doDestroy(self): for dev in self._customDevices(): @@ -3561,7 +3562,7 @@ dev['target'] = target # persist the target value to make it consistent after recovery self.saveState() - return {'status': doneCode} + return response.success() def setCpuTuneQuota(self, quota): try: @@ -3593,7 +3594,7 @@ except libvirt.libvirtError as e: return self._reportException(key='cpuTuneErr', msg=e.message) else: - return {'status': doneCode} + return response.success() def _reportException(self, key, msg=None): """ @@ -4454,7 +4455,7 @@ # of getVmStats after this returns will see the new job self.updateVmJobs() - return {'status': doneCode} + return response.success() def _diskXMLGetVolumeChainInfo(self, diskXML, drive): def find_element_by_name(doc, name): diff --git a/vdsm/virt/vmpowerdown.py b/vdsm/virt/vmpowerdown.py index 2382a88..aae927d 100644 --- a/vdsm/virt/vmpowerdown.py +++ b/vdsm/virt/vmpowerdown.py @@ -19,8 +19,8 @@ # import libvirt +from vdsm import response from vdsm import utils -from vdsm.define import doneCode, errCode class VmPowerDown(object): @@ -69,15 +69,12 @@ self.event.clear() self.chain.start() - return {'status': {'code': doneCode['code'], - 'message': self.returnMsg}} + return response.success(message=self.returnMsg) else: # No tools, no ACPI - return { - 'status': { - 'code': errCode['exist']['status']['code'], - 'message': 'VM without ACPI or active oVirt guest agent. ' - 'Try Forced Shutdown.'}} + return response.error('exist', + message='VM without ACPI or active oVirt ' + 'guest agent. Try Forced Shutdown.') class VmShutdown(VmPowerDown): -- To view, visit https://gerrit.ovirt.org/40900 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If1bb95b46197ba4d6c8eccaed33fd77c426a084c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 20

Change in vdsm[master]: v2v: Try to be nicer to other processes
by Nir Soffer 26 Aug '15

26 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: v2v: Try to be nicer to other processes ...................................................................... v2v: Try to be nicer to other processes Under the hood, virt-v2v is running qemu-img convert. We use high nice and ionice values when running qemu-img convert, but we did not specify such options when running virt-v2v. According to bug 1243935, sanlock is loosing access to the storage while v2v is importing a vm. Hopefully running virt-v2v in a nicer manner would solve this issue. Change-Id: Id163844a5976384ca9e9a75d58a581645f08a89f Bug-Url: https://bugzilla.redhat.com/1243935 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/v2v.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/17/43817/1 diff --git a/vdsm/v2v.py b/vdsm/v2v.py index 524cbdf..cac2a0a 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -39,7 +39,7 @@ from vdsm.define import errCode, doneCode from vdsm import libvirtconnection from vdsm.infra import zombiereaper -from vdsm.utils import traceback, CommandPath, execCmd +from vdsm.utils import traceback, CommandPath, execCmd, NICENESS, IOCLASS import caps @@ -368,7 +368,10 @@ cmd = self._create_command() logging.info('Job %r starting import', self._id) + # This is the way we run qemu-img convert jobs. virt-v2v is invoking + # qemu-img convert to perform the migration. self._proc = execCmd(cmd, sync=False, deathSignal=15, + nice=NICENESS.HIGH, ionice=IOCLASS.IDLE, env={'LIBGUESTFS_BACKEND': 'direct'}) self._proc.blocking = True -- To view, visit https://gerrit.ovirt.org/43817 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id163844a5976384ca9e9a75d58a581645f08a89f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

8 27

Change in vdsm[master]: storage: simplify hasVolumeLeases
by alitke＠redhat.com 26 Aug '15

26 Aug '15

Adam Litke has uploaded a new change for review. Change subject: storage: simplify hasVolumeLeases ...................................................................... storage: simplify hasVolumeLeases Whether a storage domain has volume leases depends on the lock implementation used, not the storage domain version. SafeLease never supports volume leases but LocalLock and SANLock both do. Simplify hasVolumeLeases and the clusterLock tables by moving the support flag into the class implementing the lock. Change-Id: Iaf0c1cf341e7d028b5d4e30a27e9a045b43470e4 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/clusterlock.py M vdsm/storage/localFsSD.py M vdsm/storage/sd.py 3 files changed, 13 insertions(+), 14 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/48/43548/1 diff --git a/vdsm/storage/clusterlock.py b/vdsm/storage/clusterlock.py index 2630acf..3a4bfa1 100644 --- a/vdsm/storage/clusterlock.py +++ b/vdsm/storage/clusterlock.py @@ -79,6 +79,7 @@ lockUtilPath = config.get('irs', 'lock_util_path') lockCmd = config.get('irs', 'lock_cmd') freeLockCmd = config.get('irs', 'free_lock_cmd') + volumeLeasesSupported = False def __init__(self, sdUUID, idsPath, leasesPath, lockRenewalIntervalSec, leaseTimeSec, leaseFailRetry, ioOpTimeoutSec): @@ -196,6 +197,7 @@ _sanlock_fd = None _sanlock_lock = threading.Lock() + volumeLeasesSupported = True def __init__(self, sdUUID, idsPath, leasesPath, *args): self._lock = threading.Lock() @@ -356,6 +358,7 @@ _globalLockMap = {} _globalLockMapSync = threading.Lock() + volumeLeasesSupported = True def __init__(self, sdUUID, idsPath, leasesPath, *args): self._sdUUID = sdUUID diff --git a/vdsm/storage/localFsSD.py b/vdsm/storage/localFsSD.py index 1abf018..1dd1ade 100644 --- a/vdsm/storage/localFsSD.py +++ b/vdsm/storage/localFsSD.py @@ -30,11 +30,11 @@ class LocalFsStorageDomain(fileSD.FileStorageDomain): - # version: (clusterLockClass, hasVolumeLeases) + # version: clusterLockClass _clusterLockTable = { - 0: (clusterlock.SafeLease, False), - 2: (clusterlock.SafeLease, False), - 3: (clusterlock.LocalLock, True), + 0: clusterlock.SafeLease, + 2: clusterlock.SafeLease, + 3: clusterlock.LocalLock, } @property diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 336cee0..fedbdee 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -382,11 +382,11 @@ mdBackupVersions = config.get('irs', 'md_backup_versions') mdBackupDir = config.get('irs', 'md_backup_dir') - # version: (clusterLockClass, hasVolumeLeases) + # version: clusterLockClass _clusterLockTable = { - 0: (clusterlock.SafeLease, False), - 2: (clusterlock.SafeLease, False), - 3: (clusterlock.SANLock, True), + 0: clusterlock.SafeLease, + 2: clusterlock.SafeLease, + 3: clusterlock.SANLock, } def __init__(self, manifest): @@ -472,7 +472,7 @@ ) try: - clusterLockClass = self._clusterLockTable[domVersion][0] + clusterLockClass = self._clusterLockTable[domVersion] except KeyError: raise se.UnsupportedDomainVersion(domVersion) @@ -593,11 +593,7 @@ return self._clusterLock.getHostStatus(hostId) def hasVolumeLeases(self): - domVersion = self.getVersion() - try: - return self._clusterLockTable[domVersion][1] - except KeyError: - raise se.UnsupportedDomainVersion(domVersion) + return self._clusterLock.volumeLeasesSupported def getVolumeLease(self, volUUID): """ -- To view, visit https://gerrit.ovirt.org/43548 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf0c1cf341e7d028b5d4e30a27e9a045b43470e4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 20

Change in vdsm[master]: vm: make acpiShutdown more robust
by fromani＠redhat.com 26 Aug '15

26 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: vm: make acpiShutdown more robust ...................................................................... vm: make acpiShutdown more robust In the Vm shutdown flow, the actual 'destruction' of the domain -in libvirt jargon- may already happen asynchonously, e.g. if QEMU dies suddenly. Moreover, we want to make it asynchronously even on our own code for performance reasons. This means that the Vm._dom attribute may become None asyncrhonously, even inside the Vm powerdown flow itself. This patch address the case for the acpiShutdown, fixing this issue: Thread-259072::ERROR::2014-10-15 12:06:52,842::utils::1193::utils.Callback::(__call__) acpiCallback failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 1191, in __call__ result = self.func(*self.args, **self.kwargs) File "/usr/share/vdsm/virt/vmpowerdown.py", line 91, in acpiCallback self.vm.acpiShutdown() File "/usr/share/vdsm/virt/vm.py", line 4942, in acpiShutdown self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_ACPI_POWER_BTN) AttributeError: 'NoneType' object has no attribute 'shutdownFlags' Change-Id: I244f00d62ee24fb42ba3d654961a8fc22f4a6c25 Related-To: https://bugzilla.redhat.com/show_bug.cgi?id=1154389 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/34879/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 9740ab3..2384aca 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -4402,7 +4402,13 @@ def acpiShutdown(self): self._shutdownReason = vmexitreason.ADMIN_SHUTDOWN - self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_ACPI_POWER_BTN) + try: + self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_ACPI_POWER_BTN) + except AttributeError: + if not self._released: + raise + # else the VM was already shut off asynchronously, + # so ignore error and quickly exit def setBalloonTarget(self, target): -- To view, visit http://gerrit.ovirt.org/34879 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I244f00d62ee24fb42ba3d654961a8fc22f4a6c25 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

5 23

Change in vdsm[master]: Add feature to create dispersed volume
by tjeyasin＠redhat.com 26 Aug '15

26 Aug '15

Timothy Asir has uploaded a new change for review. Change subject: Add feature to create dispersed volume ...................................................................... Add feature to create dispersed volume Added disperse count and redundancy count params into create volume function which provides options to create a dispersed volume. Change-Id: I50f6dbae8d179c09480634c64d7d439ae82d72a2 Signed-off-by: Timothy Asir <tjeyasin(a)redhat.com> --- M client/vdsClientGluster.py M vdsm/gluster/api.py M vdsm/gluster/apiwrapper.py M vdsm/gluster/cli.py M vdsm/rpc/vdsmapi-gluster-schema.json 5 files changed, 28 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/74/37474/1 diff --git a/client/vdsClientGluster.py b/client/vdsClientGluster.py index a5065f3..1139222 100644 --- a/client/vdsClientGluster.py +++ b/client/vdsClientGluster.py @@ -32,6 +32,8 @@ params = self._eqSplit(args) brickList = params.get('bricks', '').split(',') volumeName = params.get('volumeName', '') + disperseCount = params.get('disperse', '') + redundancyCount = params.get('redundancy', '') replicaCount = params.get('replica', '') stripeCount = params.get('stripe', '') transport = params.get('transport', '') @@ -39,6 +41,7 @@ force = (params.get('force', 'no').upper() == 'YES') status = self.s.glusterVolumeCreate(volumeName, brickList, + disperseCount, redundancyCount, replicaCount, stripeCount, transportList, force) pp.pprint(status) @@ -522,6 +525,7 @@ {'glusterVolumeCreate': ( serv.do_glusterVolumeCreate, ('volumeName=<volume_name> bricks=<brick[,brick, ...]> ' + '[disperse=<count>] [redundancy=<count>] ' '[replica=<count>] [stripe=<count>] [transport={tcp|rdma}] ' '[force={yes|no}]\n\t' '<volume_name> is name of new volume', diff --git a/vdsm/gluster/api.py b/vdsm/gluster/api.py index 04e0f33..33f1724 100644 --- a/vdsm/gluster/api.py +++ b/vdsm/gluster/api.py @@ -79,10 +79,12 @@ remoteServer)} @exportAsVerb - def volumeCreate(self, volumeName, brickList, replicaCount=0, + def volumeCreate(self, volumeName, brickList, disperseCount=0, + redundancyCount=0, replicaCount=0, stripeCount=0, transportList=[], force=False, options=None): return self.svdsmProxy.glusterVolumeCreate(volumeName, brickList, + disperseCount, redundancyCount, replicaCount, stripeCount, transportList, force) diff --git a/vdsm/gluster/apiwrapper.py b/vdsm/gluster/apiwrapper.py index d42f8ba..4f0689b 100644 --- a/vdsm/gluster/apiwrapper.py +++ b/vdsm/gluster/apiwrapper.py @@ -111,9 +111,10 @@ def list(self, volumeName=None, remoteServer=None): return self._gluster.volumesList(volumeName, remoteServer) - def create(self, volumeName, brickList, replicaCount=0, stripeCount=0, - transportList=[], force=False): - return self._gluster.volumeCreate(volumeName, brickList, replicaCount, + def create(self, volumeName, brickList, disperseCount=0, redundancyCount=0, + replicaCount=0, stripeCount=0, transportList=[], force=False): + return self._gluster.volumeCreate(volumeName, brickList, disperseCount, + redundancyCount, replicaCount, stripeCount, transportList, force) def start(self, volumeName, force=False): diff --git a/vdsm/gluster/cli.py b/vdsm/gluster/cli.py index d9ecd66..d161801 100644 --- a/vdsm/gluster/cli.py +++ b/vdsm/gluster/cli.py @@ -475,9 +475,13 @@ @makePublic -def volumeCreate(volumeName, brickList, replicaCount=0, stripeCount=0, - transportList=[], force=False): +def volumeCreate(volumeName, brickList, disperseCount=0, redundancyCount=0, + replicaCount=0, stripeCount=0, transportList=[], force=False): command = _getGlusterVolCmd() + ["create", volumeName] + if disperseCount: + command += ["disperse", "%s" % disperseCount] + if redundancyCount: + command += ["redundancy", "%s" % redundancyCount] if stripeCount: command += ["stripe", "%s" % stripeCount] if replicaCount: diff --git a/vdsm/rpc/vdsmapi-gluster-schema.json b/vdsm/rpc/vdsmapi-gluster-schema.json index 5924fd9..bba7126 100644 --- a/vdsm/rpc/vdsmapi-gluster-schema.json +++ b/vdsm/rpc/vdsmapi-gluster-schema.json @@ -660,17 +660,21 @@ # # Create a Gluster volume # -# @volumeName: Gluster volume name +# @volumeName: Gluster volume name # -# @bricklist: List of bricks +# @bricklist: List of bricks # -# @replicaCount: #optional Replica count for volume +# @disperseCount: #optional Disperse count for volume # -# @stripeCount: #optional Stripe count for volume +# @redundancyCount: #optional Redundancy count for volume # -# @transportList: #optional Transport type +# @replicaCount: #optional Replica count for volume # -# @force: #optional Force create Volume (new in version 4.14.0) +# @stripeCount: #optional Stripe count for volume +# +# @transportList: #optional Transport type +# +# @force: #optional Force create Volume (new in version 4.14.0) # # Returns: # uuid of the volume @@ -679,6 +683,7 @@ ## {'command': {'class': 'GlusterVolume', 'name': 'create'}, 'data': {'volumeName': 'str', 'bricklist': ['str'], + '*disperseCount': 'int', '*redundancyCount': 'int', '*replicaCount': 'int', '*stripeCount': 'int', '*transportList': ['transType'], '*force': 'bool'}, 'returns': 'UUID'} -- To view, visit http://gerrit.ovirt.org/37474 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I50f6dbae8d179c09480634c64d7d439ae82d72a2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Timothy Asir <tjeyasin(a)redhat.com>

6 22

Change in vdsm[master]: gluster: fix xfs filesystem is created with wrong inode size
by tjeyasin＠redhat.com 25 Aug '15

25 Aug '15

Timothy Asir has uploaded a new change for review. Change subject: gluster: fix xfs filesystem is created with wrong inode size ...................................................................... gluster: fix xfs filesystem is created with wrong inode size Currently brick create function creates an xfs filesystem with default value (ex 256) for the disks other then RAID6 or RAID10. This patch fixes this issue by setting the inode size to 512 and the directory block size to 8192. Change-Id: If3f350d2f79172705a93a9d69fe1b7b78867861e Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1244865 Signed-off-by: Timothy Asir Jeyasingh <tjeyasin(a)redhat.com> --- M vdsm/gluster/storagedev.py 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/47/43847/1 diff --git a/vdsm/gluster/storagedev.py b/vdsm/gluster/storagedev.py index af250417..e79a4fc 100644 --- a/vdsm/gluster/storagedev.py +++ b/vdsm/gluster/storagedev.py @@ -313,6 +313,9 @@ "-f", "-K", "-i", "size=512", "-d", "sw=%s,su=%sk" % (count, raidParams.get('stripeSize')), "-n", "size=8192"] + else: + format._defaultFormatOptions = ["-f", "-i", "size=512", + "-n", "size=8192"] blivetEnv.formatDevice(thinlv, format) blivetEnv.doIt() -- To view, visit https://gerrit.ovirt.org/43847 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If3f350d2f79172705a93a9d69fe1b7b78867861e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Timothy Asir <tjeyasin(a)redhat.com>

6 22

Change in vdsm[master]: WIP: make _getInterfaceStats a function
by fromani＠redhat.com 25 Aug '15

25 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: WIP: make _getInterfaceStats a function ...................................................................... WIP: make _getInterfaceStats a function Change-Id: I0b5796f3463585794d2696ce73f8357cb0fd1f78 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/sampling.py 1 file changed, 56 insertions(+), 72 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/30/40430/1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index f61514b..45afdfd 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -622,10 +622,10 @@ if len(self._samples) < 2: return stats - stats.update(self._getInterfacesStats()) - - hs0, hs1 = self._samples[0], self._samples[-1] + hs0, hs1, _ = self._samples.stats() interval = hs1.timestamp - hs0.timestamp + + stats.update(self._get_interfaces_stats(hs0, hs1, interval)) jiffies = (hs1.pidcpu.user - hs0.pidcpu.user) % (2 ** 32) stats['cpuUserVdsmd'] = jiffies / interval @@ -652,75 +652,6 @@ stats['cpuStatistics'] = _get_cpu_core_stats(hs0, hs1) stats['v2vJobs'] = v2v.get_jobs_status() - return stats - - def _getInterfacesStats(self): - """ - Compile and return a dict containing the stats. - - :returns: a dict that with the following keys: - - * cpuUser - * cpuSys - * cpuIdle - * rxRate - * txRate - """ - stats = {} - if len(self._samples) < 2: - return stats - hs0, hs1 = self._samples[0], self._samples[-1] - interval = hs1.timestamp - hs0.timestamp - - rx = tx = rxDropped = txDropped = 0 - stats['network'] = {} - total_rate = 0 - for ifid in hs1.interfaces: - # it skips hot-plugged devices if we haven't enough information - # to count stats from it - if ifid not in hs0.interfaces: - continue - ifrate = hs1.interfaces[ifid].speed or 1000 - Mbps2Bps = (10 ** 6) / 8 - thisRx = (hs1.interfaces[ifid].rx - hs0.interfaces[ifid].rx) % \ - (2 ** 32) - thisTx = (hs1.interfaces[ifid].tx - hs0.interfaces[ifid].tx) % \ - (2 ** 32) - rxRate = 100.0 * thisRx / interval / ifrate / Mbps2Bps - txRate = 100.0 * thisTx / interval / ifrate / Mbps2Bps - if txRate > 100 or rxRate > 100: - txRate = min(txRate, 100.0) - rxRate = min(rxRate, 100.0) - self._log.debug('Rate above 100%%.') - iface = hs1.interfaces[ifid] - stats['network'][ifid] = {'name': ifid, 'speed': str(ifrate), - 'rxDropped': str(iface.rxDropped), - 'txDropped': str(iface.txDropped), - 'rxErrors': str(iface.rxErrors), - 'txErrors': str(iface.txErrors), - 'state': iface.operstate, - 'rxRate': '%.1f' % rxRate, - 'txRate': '%.1f' % txRate, - 'rx': str(iface.rx), - 'tx': str(iface.tx), - 'sampleTime': hs1.timestamp, - } - rx += thisRx - tx += thisTx - rxDropped += hs1.interfaces[ifid].rxDropped - txDropped += hs1.interfaces[ifid].txDropped - total_rate += ifrate - - total_bytes_per_sec = (total_rate or 1000) * (10 ** 6) / 8 - stats['rxRate'] = 100.0 * rx / interval / total_bytes_per_sec - stats['txRate'] = 100.0 * tx / interval / total_bytes_per_sec - if stats['txRate'] > 100 or stats['rxRate'] > 100: - stats['txRate'] = min(stats['txRate'], 100.0) - stats['rxRate'] = min(stats['rxRate'], 100.0) - logging.debug(stats) - stats['rxDropped'] = rxDropped - stats['txDropped'] = txDropped - return stats @@ -754,6 +685,59 @@ return cpuCoreStats +def _get_interfaces_stats(self, hs0, hs1, interval): + rx = tx = rxDropped = txDropped = 0 + stats['network'] = {} + total_rate = 0 + for ifid in hs1.interfaces: + # it skips hot-plugged devices if we haven't enough information + # to count stats from it + if ifid not in hs0.interfaces: + continue + ifrate = hs1.interfaces[ifid].speed or 1000 + Mbps2Bps = (10 ** 6) / 8 + thisRx = (hs1.interfaces[ifid].rx - hs0.interfaces[ifid].rx) % \ + (2 ** 32) + thisTx = (hs1.interfaces[ifid].tx - hs0.interfaces[ifid].tx) % \ + (2 ** 32) + rxRate = 100.0 * thisRx / interval / ifrate / Mbps2Bps + txRate = 100.0 * thisTx / interval / ifrate / Mbps2Bps + if txRate > 100 or rxRate > 100: + txRate = min(txRate, 100.0) + rxRate = min(rxRate, 100.0) + logging.debug('Rate above 100%%.') + iface = hs1.interfaces[ifid] + stats['network'][ifid] = {'name': ifid, 'speed': str(ifrate), + 'rxDropped': str(iface.rxDropped), + 'txDropped': str(iface.txDropped), + 'rxErrors': str(iface.rxErrors), + 'txErrors': str(iface.txErrors), + 'state': iface.operstate, + 'rxRate': '%.1f' % rxRate, + 'txRate': '%.1f' % txRate, + 'rx': str(iface.rx), + 'tx': str(iface.tx), + 'sampleTime': hs1.timestamp, + } + rx += thisRx + tx += thisTx + rxDropped += hs1.interfaces[ifid].rxDropped + txDropped += hs1.interfaces[ifid].txDropped + total_rate += ifrate + + total_bytes_per_sec = (total_rate or 1000) * (10 ** 6) / 8 + stats['rxRate'] = 100.0 * rx / interval / total_bytes_per_sec + stats['txRate'] = 100.0 * tx / interval / total_bytes_per_sec + if stats['txRate'] > 100 or stats['rxRate'] > 100: + stats['txRate'] = min(stats['txRate'], 100.0) + stats['rxRate'] = min(stats['rxRate'], 100.0) + logging.debug(stats) + stats['rxDropped'] = rxDropped + stats['txDropped'] = txDropped + + return stats + + def _getLinkSpeed(dev): if dev.isNIC(): speed = netinfo.nicSpeed(dev.name) -- To view, visit https://gerrit.ovirt.org/40430 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0b5796f3463585794d2696ce73f8357cb0fd1f78 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

5 34

Change in vdsm[master]: WIP: sampling: make _getCpuCoresStats a function
by fromani＠redhat.com 25 Aug '15

25 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: WIP: sampling: make _getCpuCoresStats a function ...................................................................... WIP: sampling: make _getCpuCoresStats a function Change-Id: If602a84096db7aa7d1a6672b84d62287e08b0ac2 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/sampling.py 1 file changed, 31 insertions(+), 31 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/29/40429/1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index cbe53ef..f61514b 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -649,40 +649,10 @@ stats['bootTime'] = self._boot_time() stats['numaNodeMemFree'] = hs1.numaNodeMem.nodesMemSample - stats['cpuStatistics'] = self._getCpuCoresStats() + stats['cpuStatistics'] = _get_cpu_core_stats(hs0, hs1) stats['v2vJobs'] = v2v.get_jobs_status() return stats - - def _getCpuCoresStats(self): - """ - :returns: a dict that with the following formats: - - {'<cpuId>': {'numaNodeIndex': int, 'cpuSys': 'str', - 'cpuIdle': 'str', 'cpuUser': 'str'}, ...} - """ - cpuCoreStats = {} - for nodeIndex, numaNode in caps.getNumaTopology().iteritems(): - cpuCores = numaNode['cpus'] - for cpuCore in cpuCores: - coreStat = {} - coreStat['nodeIndex'] = int(nodeIndex) - hs0, hs1 = self._samples[0], self._samples[-1] - interval = hs1.timestamp - hs0.timestamp - jiffies = (hs1.cpuCores.getCoreSample(cpuCore)['user'] - - hs0.cpuCores.getCoreSample(cpuCore)['user']) % \ - (2 ** 32) - coreStat['cpuUser'] = ("%.2f" % (jiffies / interval)) - jiffies = (hs1.cpuCores.getCoreSample(cpuCore)['sys'] - - hs0.cpuCores.getCoreSample(cpuCore)['sys']) % \ - (2 ** 32) - coreStat['cpuSys'] = ("%.2f" % (jiffies / interval)) - coreStat['cpuIdle'] = ("%.2f" % - max(0.0, 100.0 - - float(coreStat['cpuUser']) - - float(coreStat['cpuSys']))) - cpuCoreStats[str(cpuCore)] = coreStat - return cpuCoreStats def _getInterfacesStats(self): """ @@ -754,6 +724,36 @@ return stats +def _get_cpu_core_stats(hs0, hs1): + """ + :returns: a dict that with the following formats: + + {'<cpuId>': {'numaNodeIndex': int, 'cpuSys': 'str', + 'cpuIdle': 'str', 'cpuUser': 'str'}, ...} + """ + cpuCoreStats = {} + for nodeIndex, numaNode in caps.getNumaTopology().iteritems(): + cpuCores = numaNode['cpus'] + for cpuCore in cpuCores: + coreStat = {} + coreStat['nodeIndex'] = int(nodeIndex) + interval = hs1.timestamp - hs0.timestamp + jiffies = (hs1.cpuCores.getCoreSample(cpuCore)['user'] - + hs0.cpuCores.getCoreSample(cpuCore)['user']) % \ + (2 ** 32) + coreStat['cpuUser'] = ("%.2f" % (jiffies / interval)) + jiffies = (hs1.cpuCores.getCoreSample(cpuCore)['sys'] - + hs0.cpuCores.getCoreSample(cpuCore)['sys']) % \ + (2 ** 32) + coreStat['cpuSys'] = ("%.2f" % (jiffies / interval)) + coreStat['cpuIdle'] = ("%.2f" % + max(0.0, 100.0 - + float(coreStat['cpuUser']) - + float(coreStat['cpuSys']))) + cpuCoreStats[str(cpuCore)] = coreStat + return cpuCoreStats + + def _getLinkSpeed(dev): if dev.isNIC(): speed = netinfo.nicSpeed(dev.name) -- To view, visit https://gerrit.ovirt.org/40429 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If602a84096db7aa7d1a6672b84d62287e08b0ac2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

5 39

Change in vdsm[master]: periodic: make VmDispatcher ignore TooManyTasks
by fromani＠redhat.com 25 Aug '15

25 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: periodic: make VmDispatcher ignore TooManyTasks ...................................................................... periodic: make VmDispatcher ignore TooManyTasks VmDispatcher is meant to make N independent operations from on (VM bulk sampling), being N the number of active VMs. To fullfill this claim, we ignore TooManyTasks in the vm dispatching loop, even though is highly likely that once one TooManyTasks error is encountered, many subsequent dispatch() attempt will fail. Change-Id: Iad2d186327f8607aaeb41ca132a9a8a0806869cf Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/periodic.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/41657/1 diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py index acfe2f6..f226424 100644 --- a/vdsm/virt/periodic.py +++ b/vdsm/virt/periodic.py @@ -223,6 +223,7 @@ def __call__(self): vms = self._get_vms() skipped = [] + op = None for vm_id, vm_obj in vms.iteritems(): try: @@ -245,8 +246,11 @@ # we want to make sure to have VM UUID logged self._log.exception("while dispatching %s to VM '%s'", self._create, vm_id) - else: + + try: self._executor.dispatch(op, self._timeout) + except executor.TooManyTasks: + skipped.append(vm_id) if skipped: self._log.warning('could not run %s on %s', -- To view, visit https://gerrit.ovirt.org/41657 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iad2d186327f8607aaeb41ca132a9a8a0806869cf Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 46

Change in vdsm[master]: vmstats: fine-grained KeyError handling
by fromani＠redhat.com 25 Aug '15

25 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: fine-grained KeyError handling ...................................................................... vmstats: fine-grained KeyError handling one of the cornerstones behind bulk stats implementation is to fail only in critical conditions (e.g. memory exaustion) and report as much as partial data it is possible. This means that is legal to omit random keys in the output. We must make sure that our code deals with this case gracefully. This patch addresses the missing pieces. Change-Id: I04ecbf98d36d576ede3c3cde36586245ea5e4218 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 68 insertions(+), 40 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/77/43777/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index e36ef22..7254cd6 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -148,11 +148,17 @@ # MOM will ignore VMs with missing balloon information instead # using incomplete data and computing wrong balloon targets if balloon_target is not None and sample is not None: + + try: + balloon_cur = sample['balloon.current'] + except KeyError: + balloon_cur = 0 + stats['balloonInfo'].update({ 'balloon_max': str(max_mem), 'balloon_min': str( int(vm.conf.get('memGuaranteedSize', '0')) * 1024), - 'balloon_cur': str(sample['balloon.current']), + 'balloon_cur': str(balloon_cur), 'balloon_target': str(balloon_target) }) @@ -180,31 +186,39 @@ 'speed': str(ifSpeed), 'state': 'unknown'} - ifStats['rxErrors'] = str(end_sample['net.%d.rx.errs' % end_index]) - ifStats['rxDropped'] = str(end_sample['net.%d.rx.drop' % end_index]) - ifStats['txErrors'] = str(end_sample['net.%d.tx.errs' % end_index]) - ifStats['txDropped'] = str(end_sample['net.%d.tx.drop' % end_index]) + try: + ifStats['rxErrors'] = str(end_sample['net.%d.rx.errs' % end_index]) + ifStats['rxDropped'] = str(end_sample['net.%d.rx.drop' % end_index]) + ifStats['txErrors'] = str(end_sample['net.%d.tx.errs' % end_index]) + ifStats['txDropped'] = str(end_sample['net.%d.tx.drop' % end_index]) + except KeyError: + pass - rxDelta = ( - end_sample['net.%d.rx.bytes' % end_index] - - start_sample['net.%d.rx.bytes' % start_index] - ) - ifRxBytes = (100.0 * - (rxDelta % 2 ** 32) / - interval / ifSpeed / _MBPS_TO_BPS) - txDelta = ( - end_sample['net.%d.tx.bytes' % end_index] - - start_sample['net.%d.tx.bytes' % start_index] - ) - ifTxBytes = (100.0 * - (txDelta % 2 ** 32) / - interval / ifSpeed / _MBPS_TO_BPS) + try: + rxDelta = ( + end_sample['net.%d.rx.bytes' % end_index] - + start_sample['net.%d.rx.bytes' % start_index] + ) + txDelta = ( + end_sample['net.%d.tx.bytes' % end_index] - + start_sample['net.%d.tx.bytes' % start_index] + ) + except KeyError: + pass + else: + ifRxBytes = (100.0 * + (rxDelta % 2 ** 32) / + interval / ifSpeed / _MBPS_TO_BPS) + ifTxBytes = (100.0 * + (txDelta % 2 ** 32) / + interval / ifSpeed / _MBPS_TO_BPS) - ifStats['rxRate'] = '%.1f' % ifRxBytes - ifStats['txRate'] = '%.1f' % ifTxBytes + ifStats['rxRate'] = '%.1f' % ifRxBytes + ifStats['txRate'] = '%.1f' % ifTxBytes - ifStats['rx'] = str(end_sample['net.%d.rx.bytes' % end_index]) - ifStats['tx'] = str(end_sample['net.%d.tx.bytes' % end_index]) + ifStats['rx'] = str(end_sample['net.%d.rx.bytes' % end_index]) + ifStats['tx'] = str(end_sample['net.%d.tx.bytes' % end_index]) + ifStats['sampleTime'] = monotonic_time() return ifStats @@ -288,15 +302,21 @@ def _disk_rate(first_sample, first_index, last_sample, last_index, interval): - return { - 'readRate': ( - (last_sample['block.%d.rd.bytes' % last_index] - - first_sample['block.%d.rd.bytes' % first_index]) - / interval), - 'writeRate': ( - (last_sample['block.%d.wr.bytes' % last_index] - - first_sample['block.%d.wr.bytes' % first_index]) - / interval)} + try: + return { + 'readRate': ( + (last_sample['block.%d.rd.bytes' % last_index] - + first_sample['block.%d.rd.bytes' % first_index]) + / interval + ), + 'writeRate': ( + (last_sample['block.%d.wr.bytes' % last_index] - + first_sample['block.%d.wr.bytes' % first_index]) + / interval + ), + } + except KeyError: + return {} def _disk_latency(first_sample, first_index, last_sample, last_index): @@ -313,18 +333,26 @@ ) return elapsed_time / operations - return {'readLatency': str(compute_latency('rd')), + try: + return { + 'readLatency': str(compute_latency('rd')), 'writeLatency': str(compute_latency('wr')), - 'flushLatency': str(compute_latency('fl'))} + 'flushLatency': str(compute_latency('fl')) + } + except KeyError: + return {} def _disk_iops_bytes(first_sample, first_index, last_sample, last_index): - return { - 'readOps': str(last_sample['block.%d.rd.reqs' % last_index]), - 'writeOps': str(last_sample['block.%d.wr.reqs' % last_index]), - 'readBytes': str(last_sample['block.%d.rd.bytes' % last_index]), - 'writtenBytes': str(last_sample['block.%d.wr.bytes' % last_index]), - } + try: + return { + 'readOps': str(last_sample['block.%d.rd.reqs' % last_index]), + 'writeOps': str(last_sample['block.%d.wr.reqs' % last_index]), + 'readBytes': str(last_sample['block.%d.rd.bytes' % last_index]), + 'writtenBytes': str(last_sample['block.%d.wr.bytes' % last_index]), + } + except KeyError: + return {} def _diff(prev, curr, val): -- To view, visit https://gerrit.ovirt.org/43777 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I04ecbf98d36d576ede3c3cde36586245ea5e4218 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 29

Change in vdsm[master]: sampling: host stats: switch to monotonic_time
by fromani＠redhat.com 25 Aug '15

25 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: sampling: host stats: switch to monotonic_time ...................................................................... sampling: host stats: switch to monotonic_time Makes the time stamping function injectable, and default to monotonic_time. The resolution of monotonic_time is sufficient for the task of host stats, and it is a bit more robust. Also, test becomes easier if the time stamping code is injectable. Change-Id: I4ca60654836018f040b7aa1b506b3479a0f4c996 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/sampling.py 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/01/40901/1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 2eedee1..cb5527c 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -546,9 +546,10 @@ class HostMonitor(object): _connlog = logging.getlogger('connectivity') - def __init__(self, log): + def __init__(self, log, clock=utils.monotonic_time): self._log = log - self._start_time = time.time() + self._clock = clock + self._start_time = self._clock() self._samples = SampleWindow(size=5) self._pid = os.getpid() @@ -584,7 +585,7 @@ 'txRate': 0.0, # REQUIRED_FOR: engine < 3.6 'cpuSysVdsmd': 0.0, 'cpuUserVdsmd': 0.0, - 'elapsedTime': int(time.time() - self._start_time) + 'elapsedTime': int(self._clock() - self._start_time) } if len(self._samples) < 2: -- To view, visit https://gerrit.ovirt.org/40901 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4ca60654836018f040b7aa1b506b3479a0f4c996 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 12

Change in vdsm[master]: StorageDomainManifest: move image and volume enumeration
by alitke＠redhat.com 24 Aug '15

24 Aug '15

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: move image and volume enumeration ...................................................................... StorageDomainManifest: move image and volume enumeration Change-Id: I56214a7024b1f894b8e4e0b951faf0567a1606c9 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py M vdsm/storage/fileSD.py M vdsm/storage/sd.py 3 files changed, 137 insertions(+), 132 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/69/42269/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 273cf51..d685313 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -628,6 +628,43 @@ deleteVolumes(sdUUID, toDel) self.rmDCImgDir(imgUUID, volsImgs) + def getAllVolumesImages(self): + """ + Return all the images that depend on a volume. + + Return dicts: + vols = {volUUID: ([imgUUID1, imgUUID2], parentUUID)]} + for complete images. + remnants (same) for broken imgs, orphan volumes, etc. + """ + vols = {} # The "legal" volumes: not half deleted/removed volumes. + remnants = {} # Volumes which are part of failed image deletes. + allVols = getAllVolumes(self.sdUUID) + for volName, ip in allVols.iteritems(): + if (volName.startswith(sd.REMOVED_IMAGE_PREFIX) or + ip.imgs[0].startswith(sd.REMOVED_IMAGE_PREFIX)): + remnants[volName] = ip + else: + # Deleted images are not dependencies of valid volumes. + images = [img for img in ip.imgs + if not img.startswith(sd.REMOVED_IMAGE_PREFIX)] + vols[volName] = sd.ImgsPar(images, ip.parent) + return vols, remnants + + def getAllVolumes(self): + vols, rems = self.getAllVolumesImages() + return vols + + def getAllImages(self): + """ + Get the set of all images uuids in the SD. + """ + vols = self.getAllVolumes() # {volName: ([imgs], parent)} + images = set() + for imgs, parent in vols.itervalues(): + images.update(imgs) + return images + class BlockStorageDomain(sd.StorageDomain): manifestClass = BlockStorageDomainManifest @@ -1020,16 +1057,6 @@ mdavalid=vgMetadataStatus['mdavalid'], mdathreshold=vgMetadataStatus['mdathreshold']) - def getAllImages(self): - """ - Get the set of all images uuids in the SD. - """ - vols = self.getAllVolumes() # {volName: ([imgs], parent)} - images = set() - for imgs, parent in vols.itervalues(): - images.update(imgs) - return images - def rmDCImgDir(self, imgUUID, volsImgs): return self._manifest.rmDCImgDir(imgUUID, volsImgs) @@ -1057,37 +1084,6 @@ allVols = self.getAllVolumes() volUUIDs = self._getImgExclusiveVols(imgUUID, allVols) lvm.deactivateLVs(self.sdUUID, volUUIDs) - - def getAllVolumesImages(self): - """ - Return all the images that depend on a volume. - - Return dicts: - vols = {volUUID: ([imgUUID1, imgUUID2], parentUUID)]} - for complete images. - remnants (same) for broken imgs, orphan volumes, etc. - """ - vols = {} # The "legal" volumes: not half deleted/removed volumes. - remnants = {} # Volumes which are part of failed image deletes. - allVols = getAllVolumes(self.sdUUID) - for volName, ip in allVols.iteritems(): - if (volName.startswith(sd.REMOVED_IMAGE_PREFIX) or - ip.imgs[0].startswith(sd.REMOVED_IMAGE_PREFIX)): - remnants[volName] = ip - else: - # Deleted images are not dependencies of valid volumes. - images = [img for img in ip.imgs - if not img.startswith(sd.REMOVED_IMAGE_PREFIX)] - vols[volName] = sd.ImgsPar(images, ip.parent) - return vols, remnants - - def getAllVolumes(self): - vols, rems = self.getAllVolumesImages() - return vols - - def getAllRemnants(self): - vols, rems = self.getAllVolumesImages() - return rems def linkBCImage(self, imgPath, imgUUID): dst = self.getLinkBCImagePath(imgUUID) diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index 9f44342..07fc1c6 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -213,6 +213,87 @@ self.log.error("removed image dir: %s can't be removed", toDelDir) raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) + def getAllVolumes(self): + """ + Return dict {volUUID: ((imgUUIDs,), parentUUID)} of the domain. + + (imgUUIDs,) is a tuple of all the images that contain a certain + volUUID. For non-templates volumes, this tuple consists of a single + image. For template volume it consists of all the images that are + based on the template volume. In that case, the first imgUUID in the + tuple is the self-image of the template. + + The parent of a non-template volume cannot be determined in file domain + without reading the metadata. However, in order to have an output + compatible to block domain, we report parent as None. + + Template volumes have no parent, and thus we report BLANK_UUID as their + parentUUID. + """ + volMetaPattern = os.path.join(self.mountpoint, self.sdUUID, + sd.DOMAIN_IMAGES, "*", "*.meta") + volMetaPaths = self.oop.glob.glob(volMetaPattern) + + # First create mapping from images to volumes + images = collections.defaultdict(list) + for metaPath in volMetaPaths: + head, tail = os.path.split(metaPath) + volUUID, volExt = os.path.splitext(tail) + imgUUID = os.path.basename(head) + images[imgUUID].append(volUUID) + + # Using images to volumes mapping, we can create volumes to images + # mapping, detecting template volumes and template images, based on + # these rules: + # + # Template volumes are hard linked in every image directory + # which is derived from that template, therefore: + # + # 1. A template volume which is in use will appear at least twice + # (in the template image dir and in the derived image dir) + # + # 2. Any volume which appears more than once in the dir tree is + # by definition a template volume. + # + # 3. Any image which has more than 1 volume is not a template + # image. + + volumes = {} + for imgUUID, volUUIDs in images.iteritems(): + for volUUID in volUUIDs: + if volUUID in volumes: + # This must be a template volume (rule 2) + volumes[volUUID]['parent'] = sd.BLANK_UUID + if len(volUUIDs) > 1: + # This image is not a template (rule 3) + volumes[volUUID]['imgs'].append(imgUUID) + else: + # This image is a template (rule 3) + volumes[volUUID]['imgs'].insert(0, imgUUID) + else: + volumes[volUUID] = {'imgs': [imgUUID], 'parent': None} + + return dict((k, sd.ImgsPar(tuple(v['imgs']), v['parent'])) + for k, v in volumes.iteritems()) + + def getAllImages(self): + """ + Fetch the set of the Image UUIDs in the SD. + """ + # Get Volumes of an image + pattern = os.path.join(sd.storage_repository, + # ISO domains don't have images, + # we can assume single domain + self.getPools()[0], + self.sdUUID, sd.DOMAIN_IMAGES) + pattern = os.path.join(pattern, constants.UUID_GLOB_PATTERN) + files = self.oop.glob.glob(pattern) + images = set() + for i in files: + if self.oop.os.path.isdir(i): + images.add(os.path.basename(i)) + return images + class FileStorageDomain(sd.StorageDomain): manifestClass = FileStorageDomainManifest @@ -390,24 +471,6 @@ def validateMasterMount(self): return self.oop.fileUtils.pathExists(self.getMasterDir()) - def getAllImages(self): - """ - Fetch the set of the Image UUIDs in the SD. - """ - # Get Volumes of an image - pattern = os.path.join(self.storage_repository, - # ISO domains don't have images, - # we can assume single domain - self.getPools()[0], - self.sdUUID, sd.DOMAIN_IMAGES) - pattern = os.path.join(pattern, constants.UUID_GLOB_PATTERN) - files = self.oop.glob.glob(pattern) - images = set() - for i in files: - if self.oop.os.path.isdir(i): - images.add(os.path.basename(i)) - return images - def zeroImage(self, sdUUID, imgUUID, volsImgs): self.log.warning("image %s on a fileSD %s won't be zeroed." % (imgUUID, sdUUID)) @@ -420,69 +483,6 @@ imgUUID: the image to be deactivated. """ pass - - def getAllVolumes(self): - """ - Return dict {volUUID: ((imgUUIDs,), parentUUID)} of the domain. - - (imgUUIDs,) is a tuple of all the images that contain a certain - volUUID. For non-templates volumes, this tuple consists of a single - image. For template volume it consists of all the images that are - based on the template volume. In that case, the first imgUUID in the - tuple is the self-image of the template. - - The parent of a non-template volume cannot be determined in file domain - without reading the metadata. However, in order to have an output - compatible to block domain, we report parent as None. - - Template volumes have no parent, and thus we report BLANK_UUID as their - parentUUID. - """ - volMetaPattern = os.path.join(self.mountpoint, self.sdUUID, - sd.DOMAIN_IMAGES, "*", "*.meta") - volMetaPaths = self.oop.glob.glob(volMetaPattern) - - # First create mapping from images to volumes - images = collections.defaultdict(list) - for metaPath in volMetaPaths: - head, tail = os.path.split(metaPath) - volUUID, volExt = os.path.splitext(tail) - imgUUID = os.path.basename(head) - images[imgUUID].append(volUUID) - - # Using images to volumes mapping, we can create volumes to images - # mapping, detecting template volumes and template images, based on - # these rules: - # - # Template volumes are hard linked in every image directory - # which is derived from that template, therefore: - # - # 1. A template volume which is in use will appear at least twice - # (in the template image dir and in the derived image dir) - # - # 2. Any volume which appears more than once in the dir tree is - # by definition a template volume. - # - # 3. Any image which has more than 1 volume is not a template - # image. - - volumes = {} - for imgUUID, volUUIDs in images.iteritems(): - for volUUID in volUUIDs: - if volUUID in volumes: - # This must be a template volume (rule 2) - volumes[volUUID]['parent'] = sd.BLANK_UUID - if len(volUUIDs) > 1: - # This image is not a template (rule 3) - volumes[volUUID]['imgs'].append(imgUUID) - else: - # This image is a template (rule 3) - volumes[volUUID]['imgs'].insert(0, imgUUID) - else: - volumes[volUUID] = {'imgs': [imgUUID], 'parent': None} - - return dict((k, sd.ImgsPar(tuple(v['imgs']), v['parent'])) - for k, v in volumes.iteritems()) def linkBCImage(self, imgPath, imgUUID): # Nothing to do here other than returning the path diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 873c8fa..9e81917 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -351,6 +351,18 @@ def getFormat(self): return str(self.getVersion()) + def getPools(self): + try: + pools = self.getMetaParam(key=DMDK_POOLS) + except KeyError: + pools = [] + else: + # Old pool MD marked SDs not belonging to any pool with + # BLANK_UUID as the pool uuid. + if BLANK_UUID in pools: + pools.remove(BLANK_UUID) + return pools + class StorageDomain(object): log = logging.getLogger("Storage.StorageDomain") @@ -406,6 +418,12 @@ def deleteImage(self, sdUUID, imgUUID, volsImgs): self._manifest.deleteImage(sdUUID, imgUUID, volsImgs) + + def getAllImages(self): + return self._manifest.getAllImages() + + def getAllVolumes(self): + return self._manifest.getAllVolumes() def prepareMailbox(self): """ @@ -542,16 +560,7 @@ return self._manifest.getFormat() def getPools(self): - try: - pools = self.getMetaParam(key=DMDK_POOLS) - except KeyError: - pools = [] - else: - # Old pool MD marked SDs not belonging to any pool with - # BLANK_UUID as the pool uuid. - if BLANK_UUID in pools: - pools.remove(BLANK_UUID) - return pools + return self._manifest.getPools() def getIdsFilePath(self): return self._manifest.getIdsFilePath() -- To view, visit https://gerrit.ovirt.org/42269 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I56214a7024b1f894b8e4e0b951faf0567a1606c9 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 32

Change in vdsm[master]: StorageDomainManifest: deleteImage and deps
by alitke＠redhat.com 24 Aug '15

24 Aug '15

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: deleteImage and deps ...................................................................... StorageDomainManifest: deleteImage and deps Change-Id: I7e418262d70ed50b45fdddf2fc77eb80a03a3811 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py M vdsm/storage/fileSD.py M vdsm/storage/sd.py 3 files changed, 98 insertions(+), 85 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/42268/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index caa7baf..273cf51 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -570,6 +570,64 @@ newsize = self.metaSize(self.sdUUID) lvm.extendLV(self.sdUUID, sd.METADATA, newsize) + def _getImgExclusiveVols(self, imgUUID, volsImgs): + """Filter vols belonging to imgUUID only.""" + exclusives = dict((vName, v) for vName, v in volsImgs.iteritems() + if v.imgs[0] == imgUUID) + return exclusives + + def _markForDelVols(self, sdUUID, imgUUID, volUUIDs, opTag): + """ + Mark volumes that will be zeroed or removed. + + Mark for delete just in case that lvremove [lvs] success partialy. + Mark for zero just in case that zero process is interrupted. + + Tagging is preferable to rename since it can be done in a single lvm + operation and is resilient to open LVs, etc. + """ + try: + lvm.changelv(sdUUID, volUUIDs, + (("-a", "y"), + ("--deltag", blockVolume.TAG_PREFIX_IMAGE + imgUUID), + ("--addtag", blockVolume.TAG_PREFIX_IMAGE + + opTag + imgUUID))) + except se.StorageException as e: + log.error("Can't activate or change LV tags in SD %s. " + "failing Image %s %s operation for vols: %s. %s", + sdUUID, imgUUID, opTag, volUUIDs, e) + raise + + def _rmDCVolLinks(self, imgPath, volsImgs): + for vol in volsImgs: + lPath = os.path.join(imgPath, vol) + removedPaths = [] + try: + os.unlink(lPath) + except OSError as e: + self.log.warning("Can't unlink %s. %s", lPath, e) + else: + removedPaths.append(lPath) + self.log.debug("removed: %s", removedPaths) + return tuple(removedPaths) + + def rmDCImgDir(self, imgUUID, volsImgs): + imgPath = os.path.join(self.domaindir, sd.DOMAIN_IMAGES, imgUUID) + self._rmDCVolLinks(imgPath, volsImgs) + try: + os.rmdir(imgPath) + except OSError: + self.log.warning("Can't rmdir %s", imgPath, exc_info=True) + else: + self.log.debug("removed image dir: %s", imgPath) + return imgPath + + def deleteImage(self, sdUUID, imgUUID, volsImgs): + toDel = self._getImgExclusiveVols(imgUUID, volsImgs) + self._markForDelVols(sdUUID, imgUUID, toDel, sd.REMOVED_IMAGE_PREFIX) + deleteVolumes(sdUUID, toDel) + self.rmDCImgDir(imgUUID, volsImgs) + class BlockStorageDomain(sd.StorageDomain): manifestClass = BlockStorageDomainManifest @@ -972,62 +1030,14 @@ images.update(imgs) return images - def rmDCVolLinks(self, imgPath, volsImgs): - for vol in volsImgs: - lPath = os.path.join(imgPath, vol) - removedPaths = [] - try: - os.unlink(lPath) - except OSError as e: - self.log.warning("Can't unlink %s. %s", lPath, e) - else: - removedPaths.append(lPath) - self.log.debug("removed: %s", removedPaths) - return tuple(removedPaths) - def rmDCImgDir(self, imgUUID, volsImgs): - imgPath = os.path.join(self.domaindir, sd.DOMAIN_IMAGES, imgUUID) - self.rmDCVolLinks(imgPath, volsImgs) - try: - os.rmdir(imgPath) - except OSError: - self.log.warning("Can't rmdir %s", imgPath, exc_info=True) - else: - self.log.debug("removed image dir: %s", imgPath) - return imgPath + return self._manifest.rmDCImgDir(imgUUID, volsImgs) def _getImgExclusiveVols(self, imgUUID, volsImgs): - """Filter vols belonging to imgUUID only.""" - exclusives = dict((vName, v) for vName, v in volsImgs.iteritems() - if v.imgs[0] == imgUUID) - return exclusives + return self._manifest._getImgExclusiveVols(imgUUID, volsImgs) def __markForDelVols(self, sdUUID, imgUUID, volUUIDs, opTag): - """ - Mark volumes that will be zeroed or removed. - - Mark for delete just in case that lvremove [lvs] success partialy. - Mark for zero just in case that zero process is interrupted. - - Tagging is preferable to rename since it can be done in a single lvm - operation and is resilient to open LVs, etc. - """ - try: - lvm.changelv(sdUUID, volUUIDs, (("-a", "y"), - ("--deltag", blockVolume.TAG_PREFIX_IMAGE + imgUUID), - ("--addtag", blockVolume.TAG_PREFIX_IMAGE + - opTag + imgUUID))) - except se.StorageException as e: - log.error("Can't activate or change LV tags in SD %s. " - "failing Image %s %s operation for vols: %s. %s", - sdUUID, imgUUID, opTag, volUUIDs, e) - raise - - def deleteImage(self, sdUUID, imgUUID, volsImgs): - toDel = self._getImgExclusiveVols(imgUUID, volsImgs) - self.__markForDelVols(sdUUID, imgUUID, toDel, sd.REMOVED_IMAGE_PREFIX) - deleteVolumes(sdUUID, toDel) - self.rmDCImgDir(imgUUID, volsImgs) + return self._manifest._markForDelVols(sdUUID, imgUUID, volUUIDs, opTag) def zeroImage(self, sdUUID, imgUUID, volsImgs): toZero = self._getImgExclusiveVols(imgUUID, volsImgs) diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index ff083cd..9f44342 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -179,6 +179,40 @@ def getIdsFilePath(self): return os.path.join(self.getMDPath(), sd.IDS) + def getImagePath(self, imgUUID): + return os.path.join(self.domaindir, sd.DOMAIN_IMAGES, imgUUID) + + def deleteImage(self, sdUUID, imgUUID, volsImgs): + currImgDir = self.getImagePath(imgUUID) + dirName, baseName = os.path.split(currImgDir) + toDelDir = os.path.join(dirName, sd.REMOVED_IMAGE_PREFIX + baseName) + self.log.debug("Renaming dir %s to %s", currImgDir, toDelDir) + try: + self.oop.os.rename(currImgDir, toDelDir) + except OSError as e: + self.log.error("image: %s can't be moved", currImgDir) + raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) + for volUUID in volsImgs: + volPath = os.path.join(toDelDir, volUUID) + try: + self.log.debug("Removing file: %s", volPath) + self.oop.os.remove(volPath) + metaFile = volPath + '.meta' + self.log.debug("Removing file: %s", metaFile) + self.oop.os.remove(metaFile) + leaseFile = volPath + '.lease' + self.log.debug("Removing file: %s", leaseFile) + self.oop.os.remove(leaseFile) + except OSError: + self.log.error("vol: %s can't be removed.", + volPath, exc_info=True) + self.log.debug("Removing directory: %s", toDelDir) + try: + self.oop.os.rmdir(toDelDir) + except OSError as e: + self.log.error("removed image dir: %s can't be removed", toDelDir) + raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) + class FileStorageDomain(sd.StorageDomain): manifestClass = FileStorageDomainManifest @@ -373,40 +407,6 @@ if self.oop.os.path.isdir(i): images.add(os.path.basename(i)) return images - - def getImagePath(self, imgUUID): - return os.path.join(self.domaindir, sd.DOMAIN_IMAGES, imgUUID) - - def deleteImage(self, sdUUID, imgUUID, volsImgs): - currImgDir = self.getImagePath(imgUUID) - dirName, baseName = os.path.split(currImgDir) - toDelDir = os.path.join(dirName, sd.REMOVED_IMAGE_PREFIX + baseName) - self.log.debug("Renaming dir %s to %s", currImgDir, toDelDir) - try: - self.oop.os.rename(currImgDir, toDelDir) - except OSError as e: - self.log.error("image: %s can't be moved", currImgDir) - raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) - for volUUID in volsImgs: - volPath = os.path.join(toDelDir, volUUID) - try: - self.log.debug("Removing file: %s", volPath) - self.oop.os.remove(volPath) - metaFile = volPath + '.meta' - self.log.debug("Removing file: %s", metaFile) - self.oop.os.remove(metaFile) - leaseFile = volPath + '.lease' - self.log.debug("Removing file: %s", leaseFile) - self.oop.os.remove(leaseFile) - except OSError: - self.log.error("vol: %s can't be removed.", - volPath, exc_info=True) - self.log.debug("Removing directory: %s", toDelDir) - try: - self.oop.os.rmdir(toDelDir) - except OSError as e: - self.log.error("removed image dir: %s can't be removed", toDelDir) - raise se.ImageDeleteError("%s %s" % (imgUUID, str(e))) def zeroImage(self, sdUUID, imgUUID, volsImgs): self.log.warning("image %s on a fileSD %s won't be zeroed." % diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index c5dd224..873c8fa 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -404,6 +404,9 @@ def getVSize(self, imgUUID, volUUID): return self._manifest.getVSize(imgUUID, volUUID) + def deleteImage(self, sdUUID, imgUUID, volsImgs): + self._manifest.deleteImage(sdUUID, imgUUID, volsImgs) + def prepareMailbox(self): """ This method has been introduced in order to prepare the mailbox -- To view, visit https://gerrit.ovirt.org/42268 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7e418262d70ed50b45fdddf2fc77eb80a03a3811 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 36

Change in vdsm[master]: StorageDomainManifest: Move getRepoPath and other basic meta...
by alitke＠redhat.com 24 Aug '15

24 Aug '15

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: Move getRepoPath and other basic metadata getters ...................................................................... StorageDomainManifest: Move getRepoPath and other basic metadata getters Change-Id: I571437a10a53d5aa20c83438350476d03111a704 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/sd.py 1 file changed, 27 insertions(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/67/42267/1 diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 1f3ec36..c5dd224 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -306,6 +306,25 @@ def replaceMetadata(self, md): self._metadata = md + def getDomainRole(self): + return self.getMetaParam(DMDK_ROLE) + + def getDomainClass(self): + return self.getMetaParam(DMDK_CLASS) + + def getStorageType(self): + return self.getMetaParam(DMDK_TYPE) + + def getRepoPath(self): + # This is here to make sure no one tries to get a repo + # path from an ISO domain. + if self.getDomainClass() == ISO_DOMAIN: + raise se.ImagesNotSupportedError() + + # Get the datacenter ID. When using storage pools this will be the + # spUUID. Else, it's just a UUID to establish a storage namespace. + return os.path.join(storage_repository, self.getPools()[0]) + def getIsoDomainImagesDir(self): """ Get 'images' directory from Iso domain @@ -328,6 +347,9 @@ def getVersion(self): return self.getMetaParam(DMDK_VERSION) + + def getFormat(self): + return str(self.getVersion()) class StorageDomain(object): @@ -514,7 +536,7 @@ return self._manifest.getVersion() def getFormat(self): - return str(self.getVersion()) + return self._manifest.getFormat() def getPools(self): try: @@ -723,13 +745,7 @@ self.createMasterTree() def _getRepoPath(self): - # This is here to make sure no one tries to get a repo - # path from an ISO domain. - if self.getDomainClass() == ISO_DOMAIN: - raise se.ImagesNotSupportedError() - - # If it has a repo we don't have multiple domains. Assume single pool - return os.path.join(self.storage_repository, self.getPools()[0]) + return self._manifest.getRepoPath() def getLinkBCImagePath(self, imgUUID): return image.Image(self._getRepoPath()) \ @@ -808,13 +824,13 @@ return self._manifest.getMetaParam(key) def getStorageType(self): - return self.getMetaParam(DMDK_TYPE) + return self._manifest.getStorageType() def getDomainRole(self): - return self.getMetaParam(DMDK_ROLE) + return self._manifest.getDomainRole() def getDomainClass(self): - return self.getMetaParam(DMDK_CLASS) + return self._manifest.getDomainClass() def getRemotePath(self): pass -- To view, visit https://gerrit.ovirt.org/42267 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I571437a10a53d5aa20c83438350476d03111a704 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 32

Change in vdsm[master]: StorageDomainManifest: move extend and extendVolume
by alitke＠redhat.com 24 Aug '15

24 Aug '15

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: move extend and extendVolume ...................................................................... StorageDomainManifest: move extend and extendVolume Change-Id: I0ad1769d7e4db6fc0883b37a67d071c278b96c8b Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockSD.py M vdsm/storage/sd.py 2 files changed, 142 insertions(+), 118 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/66/42266/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index e368e0b..caa7baf 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -404,6 +404,10 @@ metadata = selectMetadata(sdUUID) sd.StorageDomainManifest.__init__(self, sdUUID, domaindir, metadata) + # _extendlock is used to prevent race between + # VG extend and LV extend. + self._extendlock = threading.Lock() + try: self.logBlkSize = self.getMetaParam(DMDK_LOGBLKSIZE) self.phyBlkSize = self.getMetaParam(DMDK_PHYBLKSIZE) @@ -414,6 +418,15 @@ # Such domains supported only 512 sizes self.logBlkSize = 512 self.phyBlkSize = 512 + + def readMetadataMapping(self): + meta = self.getMetadata() + for key in meta.keys(): + if not DMDK_PV_REGEX.match(key): + del meta[key] + + self.log.info("META MAPPING: %s" % meta) + return meta def getReadDelay(self): stats = misc.readspeed(lvm.lvPath(self.sdUUID, sd.METADATA), 4096) @@ -444,6 +457,119 @@ lvm.activateLVs(self.sdUUID, [sd.IDS]) return lvm.lvPath(self.sdUUID, sd.IDS) + def extendVolume(self, volumeUUID, size, isShuttingDown=None): + self._extendlock.acquire() + try: + # FIXME: following line. + lvm.extendLV(self.sdUUID, volumeUUID, size) # , isShuttingDown) + finally: + self._extendlock.release() + + @classmethod + def getMetaDataMapping(cls, vgName, oldMapping={}): + firstDev, firstExtent = lvm.getFirstExt(vgName, sd.METADATA) + firstExtent = int(firstExtent) + if firstExtent != 0: + cls.log.error("INTERNAL: metadata ext is not 0") + raise se.MetaDataMappingError("vg %s: metadata extent is not the " + "first extent" % vgName) + + pvlist = list(lvm.listPVNames(vgName)) + + pvlist.remove(firstDev) + pvlist.insert(0, firstDev) + cls.log.info("Create: SORT MAPPING: %s" % pvlist) + + mapping = {} + devNum = len(oldMapping) + for dev in pvlist: + knownDev = False + for pvID, oldInfo in oldMapping.iteritems(): + if os.path.basename(dev) == oldInfo["guid"]: + mapping[pvID] = oldInfo + knownDev = True + break + + if knownDev: + continue + + pv = lvm.getPV(dev) + pvInfo = {} + pvInfo["guid"] = os.path.basename(pv.name) + pvInfo["uuid"] = pv.uuid + # this is another trick, it's not the + # the pestart value you expect, it's just + # 0, always + pvInfo["pestart"] = 0 + pvInfo["pecount"] = pv.pe_count + if devNum == 0: + mapOffset = 0 + else: + prevDevNum = devNum - 1 + try: + prevInfo = mapping["PV%d" % (prevDevNum,)] + except KeyError: + prevInfo = oldMapping["PV%d" % (prevDevNum,)] + + mapOffset = int(prevInfo["mapoffset"]) + \ + int(prevInfo["pecount"]) + + pvInfo["mapoffset"] = mapOffset + mapping["PV%d" % devNum] = pvInfo + devNum += 1 + + return mapping + + def updateMapping(self): + # First read existing mapping from metadata + with self._metadata.transaction(): + mapping = self.getMetaDataMapping(self.sdUUID, + self.readMetadataMapping()) + for key in set(self._metadata.keys() + mapping.keys()): + if DMDK_PV_REGEX.match(key): + if key in mapping: + self._metadata[key] = mapping[key] + else: + del self._metadata[key] + + @classmethod + def metaSize(cls, vgroup): + ''' Calc the minimal meta volume size in MB''' + # In any case the metadata volume cannot be less than 512MB for the + # case of 512 bytes per volume metadata, 2K for domain metadata and + # extent size of 128MB. In any case we compute the right size on line. + vg = lvm.getVG(vgroup) + minmetasize = (SD_METADATA_SIZE / sd.METASIZE * int(vg.extent_size) + + (1024 * 1024 - 1)) / (1024 * 1024) + metaratio = int(vg.extent_size) / sd.METASIZE + metasize = (int(vg.extent_count) * sd.METASIZE + + (1024 * 1024 - 1)) / (1024 * 1024) + metasize = max(minmetasize, metasize) + if metasize > int(vg.free) / (1024 * 1024): + raise se.VolumeGroupSizeError( + "volume group has not enough extents %s (Minimum %s), VG may " + "be too small" % (vg.extent_count, + (1024 * 1024) / sd.METASIZE)) + cls.log.info("size %s MB (metaratio %s)" % (metasize, metaratio)) + return metasize + + def extend(self, devlist, force): + with self._extendlock: + if self.getVersion() in VERS_METADATA_LV: + mapping = self.readMetadataMapping().values() + if len(mapping) + len(devlist) > MAX_PVS: + raise se.StorageDomainIsMadeFromTooManyPVs() + + knowndevs = set(multipath.getMPDevNamesIter()) + unknowndevs = set(devlist) - knowndevs + if unknowndevs: + raise se.InaccessiblePhysDev(unknowndevs) + + lvm.extendVG(self.sdUUID, devlist, force) + self.updateMapping() + newsize = self.metaSize(self.sdUUID) + lvm.extendLV(self.sdUUID, sd.METADATA, newsize) + class BlockStorageDomain(sd.StorageDomain): manifestClass = BlockStorageDomainManifest @@ -458,9 +584,6 @@ # block sizes. lvm.checkVGBlockSizes(sdUUID, (self.logBlkSize, self.phyBlkSize)) - # _extendlock is used to prevent race between - # VG extend and LV extend. - self._extendlock = threading.Lock() self.imageGarbageCollector() self._registerResourceNamespaces() self._lastUncachedSelftest = 0 @@ -495,24 +618,7 @@ @classmethod def metaSize(cls, vgroup): - ''' Calc the minimal meta volume size in MB''' - # In any case the metadata volume cannot be less than 512MB for the - # case of 512 bytes per volume metadata, 2K for domain metadata and - # extent size of 128MB. In any case we compute the right size on line. - vg = lvm.getVG(vgroup) - minmetasize = (SD_METADATA_SIZE / sd.METASIZE * int(vg.extent_size) + - (1024 * 1024 - 1)) / (1024 * 1024) - metaratio = int(vg.extent_size) / sd.METASIZE - metasize = (int(vg.extent_count) * sd.METASIZE + - (1024 * 1024 - 1)) / (1024 * 1024) - metasize = max(minmetasize, metasize) - if metasize > int(vg.free) / (1024 * 1024): - raise se.VolumeGroupSizeError( - "volume group has not enough extents %s (Minimum %s), VG may " - "be too small" % (vg.extent_count, - (1024 * 1024) / sd.METASIZE)) - cls.log.info("size %s MB (metaratio %s)" % (metasize, metaratio)) - return metasize + return cls.manifestClass.metaSize(vgroup) @classmethod def create(cls, sdUUID, domainName, domClass, vgUUID, storageType, @@ -657,87 +763,10 @@ @classmethod def getMetaDataMapping(cls, vgName, oldMapping={}): - firstDev, firstExtent = lvm.getFirstExt(vgName, sd.METADATA) - firstExtent = int(firstExtent) - if firstExtent != 0: - cls.log.error("INTERNAL: metadata ext is not 0") - raise se.MetaDataMappingError("vg %s: metadata extent is not the " - "first extent" % vgName) - - pvlist = list(lvm.listPVNames(vgName)) - - pvlist.remove(firstDev) - pvlist.insert(0, firstDev) - cls.log.info("Create: SORT MAPPING: %s" % pvlist) - - mapping = {} - devNum = len(oldMapping) - for dev in pvlist: - knownDev = False - for pvID, oldInfo in oldMapping.iteritems(): - if os.path.basename(dev) == oldInfo["guid"]: - mapping[pvID] = oldInfo - knownDev = True - break - - if knownDev: - continue - - pv = lvm.getPV(dev) - pvInfo = {} - pvInfo["guid"] = os.path.basename(pv.name) - pvInfo["uuid"] = pv.uuid - # this is another trick, it's not the - # the pestart value you expect, it's just - # 0, always - pvInfo["pestart"] = 0 - pvInfo["pecount"] = pv.pe_count - if devNum == 0: - mapOffset = 0 - else: - prevDevNum = devNum - 1 - try: - prevInfo = mapping["PV%d" % (prevDevNum,)] - except KeyError: - prevInfo = oldMapping["PV%d" % (prevDevNum,)] - - mapOffset = int(prevInfo["mapoffset"]) + \ - int(prevInfo["pecount"]) - - pvInfo["mapoffset"] = mapOffset - mapping["PV%d" % devNum] = pvInfo - devNum += 1 - - return mapping - - def updateMapping(self): - # First read existing mapping from metadata - with self._metadata.transaction(): - mapping = self.getMetaDataMapping(self.sdUUID, - self.readMetadataMapping()) - for key in set(self._metadata.keys() + mapping.keys()): - if DMDK_PV_REGEX.match(key): - if key in mapping: - self._metadata[key] = mapping[key] - else: - del self._metadata[key] + return cls.manifestClass.getMetaDataMapping(vgName, oldMapping) def extend(self, devlist, force): - with self._extendlock: - if self.getVersion() in VERS_METADATA_LV: - mapping = self.readMetadataMapping().values() - if len(mapping) + len(devlist) > MAX_PVS: - raise se.StorageDomainIsMadeFromTooManyPVs() - - knowndevs = set(multipath.getMPDevNamesIter()) - unknowndevs = set(devlist) - knowndevs - if unknowndevs: - raise se.InaccessiblePhysDev(unknowndevs) - - lvm.extendVG(self.sdUUID, devlist, force) - self.updateMapping() - newsize = self.metaSize(self.sdUUID) - lvm.extendLV(self.sdUUID, sd.METADATA, newsize) + self._manifest.extend(devlist, force) _lvTagMetaSlotLock = threading.Lock() @@ -822,13 +851,7 @@ (self.sdUUID, dev, ext)) def readMetadataMapping(self): - meta = self.getMetadata() - for key in meta.keys(): - if not DMDK_PV_REGEX.match(key): - del meta[key] - - self.log.info("META MAPPING: %s" % meta) - return meta + return self._manifest.readMetadataMapping() def getLeasesFileSize(self): lv = lvm.getLV(self.sdUUID, sd.LEASES) @@ -1325,12 +1348,7 @@ os.symlink(src, dst) def extendVolume(self, volumeUUID, size, isShuttingDown=None): - self._extendlock.acquire() - try: - # FIXME: following line. - lvm.extendLV(self.sdUUID, volumeUUID, size) # , isShuttingDown) - finally: - self._extendlock.release() + return self._manifest.extendVolume(volumeUUID, size, isShuttingDown) def refresh(self): self.refreshDirTree() diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 4a89805..1f3ec36 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -317,8 +317,17 @@ return os.path.join(self.domaindir, DOMAIN_META_DATA) return None + def getMetadata(self): + """ + Unified Metadata accessor/mutator + """ + return self._metadata.copy() + def getMetaParam(self, key): return self._metadata[key] + + def getVersion(self): + return self.getMetaParam(DMDK_VERSION) class StorageDomain(object): @@ -502,7 +511,7 @@ exc_info=True) def getVersion(self): - return self.getMetaParam(DMDK_VERSION) + return self._manifest.getVersion() def getFormat(self): return str(self.getVersion()) @@ -778,10 +787,7 @@ pass def getMetadata(self): - """ - Unified Metadata accessor/mutator - """ - return self._metadata.copy() + return self._manifest.getMetadata() def setMetadata(self, newMetadata): # Backup old md (rotate old backup files) -- To view, visit https://gerrit.ovirt.org/42266 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0ad1769d7e4db6fc0883b37a67d071c278b96c8b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 57

Change in vdsm[master]: lib: migration: moving create of migration status to migrate.py
by mmirecki＠redhat.com 24 Aug '15

24 Aug '15

Marcin Mirecki has uploaded a new change for review. Change subject: lib: migration: moving create of migration status to migrate.py ...................................................................... lib: migration: moving create of migration status to migrate.py Until now the status of migration was partly composed in migrate.py and partly in API.py. The part done in API.py had a hardcoded error code which did not allow the code in migrate.py to return any error code. Now the entire status is composed in migrate.py, allowing to return any error code which is required. Change-Id: Icbc1525f52dc5ee49647aa6d690f221f2dc96e5d Signed-off-by: Marcin Mirecki <mmirecki(a)redhat.com> --- M vdsm/API.py M vdsm/virt/migration.py M vdsm/virt/vm.py 3 files changed, 11 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/43357/1 diff --git a/vdsm/API.py b/vdsm/API.py index 9399445..73e4647 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -354,7 +354,7 @@ v = self._cif.vmContainer[self._UUID] except KeyError: return errCode['noVM'] - return {'status': doneCode, 'migrationStats': v.migrateStatus()} + return v.migrateStatus() def getStats(self): """ diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py index d6994ed..e71b6e2 100644 --- a/vdsm/virt/migration.py +++ b/vdsm/virt/migration.py @@ -95,11 +95,13 @@ config.get('vars', 'migration_downtime') self._autoConverge = autoConverge self._compressed = compressed + self._progress = 0 self.status = { 'status': { 'code': 0, - 'message': 'Migration in progress'}} - self._progress = 0 + 'message': 'Migration in progress'}, + 'migrationStats': { + 'progress': self._progress}} threading.Thread.__init__(self) self._preparingMigrationEvt = True self._migrationCanceledEvt = False @@ -113,14 +115,16 @@ """ Get the status of the migration. """ + if 'migrationStats' not in self.status: + self.status['migrationStats'] = {} if self._monitorThread is not None: # fetch migration status from the monitor thread self._progress = self._monitorThread.progress - self.status['progress'] = self._progress + self.status['migrationStats']['progress'] = self._progress stat = self._vm._dom.jobStats(libvirt.VIR_DOMAIN_JOB_STATS_COMPLETED) if 'downtime_net' in stat: - self.status['downtime'] = stat['downtime_net'] + self.status['migrationStats']['downtime'] = stat['downtime_net'] return self.status diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 1ec03d4..9c05973 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1375,7 +1375,8 @@ if 'pauseCode' in self.conf: stats['pauseCode'] = self.conf['pauseCode'] if self.isMigrating(): - stats['migrationProgress'] = self.migrateStatus()['progress'] + migrationStats = self.migrateStatus()['migrationStats'] + stats['migrationProgress'] = migrationStats['progress'] decStats = {} try: -- To view, visit https://gerrit.ovirt.org/43357 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icbc1525f52dc5ee49647aa6d690f221f2dc96e5d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Marcin Mirecki <mmirecki(a)redhat.com>

4 22

Change in vdsm[master]: gluster: fix mount point issue for snapshot thin lv
by tjeyasin＠redhat.com 23 Aug '15

23 Aug '15

Timothy Asir has uploaded a new change for review. Change subject: gluster: fix mount point issue for snapshot thin lv ...................................................................... gluster: fix mount point issue for snapshot thin lv Currently storage devices list provides snapshot lv's mount point for both snapshot lv and base lv. This patch uses blivet.utils module to fix this issue by fetching the actual mount point. This patch will be reverted once the issue is fixed in blivet Change-Id: Ifadcfaedf87d325b09211c52c77a08769f57cf53 Signed-off-by: Timothy Asir Jeyasingh <tjeyasin(a)redhat.com> Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1230669 Signed-off-by: Timothy Asir Jeyasingh <tjeyasin(a)redhat.com> --- M tests/glusterTestData.py M vdsm/gluster/storagedev.py 2 files changed, 7 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/42424/1 diff --git a/tests/glusterTestData.py b/tests/glusterTestData.py index e412349..9682ae7 100644 --- a/tests/glusterTestData.py +++ b/tests/glusterTestData.py @@ -846,7 +846,7 @@ 'bus': '', 'fsType': 'xfs', 'model': 'Virtio Block Device (partition)', - 'mountPoint': '/', + 'mountPoint': '', 'name': 'vda5', 'size': '1024.0', 'uuid': '12345-12345-12345'}, @@ -974,7 +974,7 @@ device.format = TestStorageDev() device.format.type = 'xfs' device.format.uuid = "12345-12345-12345" - device.format.mountpoint = '/' + device.format.mountpoint = '' devices.append(device) # check fstype, no mountpoint attribute diff --git a/vdsm/gluster/storagedev.py b/vdsm/gluster/storagedev.py index eb8b103..df2612b 100644 --- a/vdsm/gluster/storagedev.py +++ b/vdsm/gluster/storagedev.py @@ -93,8 +93,12 @@ else: dev = {} info['fsType'] = device.format.type or dev.get('ID_FS_TYPE', '') + # device.format.mountpoint does not provide valide mount point for all + # the devices. Blivet.Util module used here to fix the issue as a + # workaround. This part will be removed once its fixed in blivet.devices if hasattr(device.format, 'mountpoint'): - info['mountPoint'] = device.format.mountpoint or '' + mountPaths = blivet.util.get_mount_paths(device.path) + info['mountPoint'] = mountPaths and mountPaths[0] or '' return info -- To view, visit https://gerrit.ovirt.org/42424 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifadcfaedf87d325b09211c52c77a08769f57cf53 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Timothy Asir <tjeyasin(a)redhat.com>

4 5

Change in vdsm[master]: gluster: fix AttributeError and TypeError in exception.py
by tjeyasin＠redhat.com 23 Aug '15

23 Aug '15

Timothy Asir has uploaded a new change for review. Change subject: gluster: fix AttributeError and TypeError in exception.py ...................................................................... gluster: fix AttributeError and TypeError in exception.py This patch fix exceptions.AttributeError object has no attribute 'out' TypeError: ('__init__() takes at least 2 arguments (1 given) Change-Id: Ic4c72daf30aee8f0024b1ef9ee064a205a500620 Signed-off-by: Timothy Asir Jeyasingh <tjeyasin(a)redhat.com> --- M vdsm/gluster/exception.py 1 file changed, 20 insertions(+), 7 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/30/41530/1 diff --git a/vdsm/gluster/exception.py b/vdsm/gluster/exception.py index 8a9f538..d93a351 100644 --- a/vdsm/gluster/exception.py +++ b/vdsm/gluster/exception.py @@ -84,6 +84,7 @@ message = "Missing argument" def __init__(self, *args, **kwargs): + GlusterGeneralException.__init__(self) self.message = 'Missing argument: args=%s, kwargs=%s' % (args, kwargs) @@ -382,7 +383,9 @@ class GlusterHostStorageDeviceMountFailedException(GlusterHostException): code = 4411 - def __init__(self, device, mountPoint, fsType, mountOpts): + def __init__(self, device=None, mountPoint=None, + fsType=None, mountOpts=None): + GlusterHostException.__init__(self) self.message = "Failed to mount device %s on mount point %s using " \ "fs-type %s with mount options %s" % ( device, mountPoint, fsType, mountOpts) @@ -391,14 +394,17 @@ class GlusterHostStorageDeviceFsTabFoundException(GlusterHostException): code = 4412 - def __init__(self, device): + def __init__(self, device=None): + GlusterHostException.__init__(self) self.message = "fstab entry for device %s already exists" % device class GlusterHostStorageDevicePVCreateFailedException(GlusterHostException): code = 4413 - def __init__(self, device, alignment, rc=0, out=(), err=()): + def __init__(self, device=None, alignment=None, + rc=0, out=(), err=()): + GlusterHostException.__init__(self) self.rc = rc self.out = out self.err = err @@ -409,7 +415,8 @@ class GlusterHostStorageDeviceLVConvertFailedException(GlusterHostException): code = 4414 - def __init__(self, device, alignment, rc=0, out=(), err=()): + def __init__(self, device=None, alignment=None, rc=0, out=(), err=()): + GlusterHostException.__init__(self) self.rc = rc self.out = out self.err = err @@ -420,7 +427,8 @@ class GlusterHostStorageDeviceLVChangeFailedException(GlusterHostException): code = 4415 - def __init__(self, poolName, rc=0, out=(), err=()): + def __init__(self, poolName=None, rc=0, out=(), err=()): + GlusterHostException.__init__(self) self.rc = rc self.out = out self.err = err @@ -436,7 +444,8 @@ class GlusterHostStorageMountPointInUseException(GlusterHostException): code = 4517 - def __init__(self, mountPoint): + def __init__(self, mountPoint=None): + GlusterHostException.__init__(self) self.message = "Mount point %s is in use" % mountPoint @@ -465,6 +474,7 @@ code = 4504 def __init__(self, glusterCmd=None, level=None, hookName=None): + GlusterHookException.__init__(self) self.glusterCmd = glusterCmd self.level = level self.hookName = hookName @@ -487,6 +497,7 @@ code = 4507 def __init__(self, glusterCmd=None, level=None, hookName=None): + GlusterHookException.__init__(self) self.glusterCmd = glusterCmd self.level = level self.hookName = hookName @@ -498,7 +509,8 @@ class GlusterHookCheckSumMismatchException(GlusterException): code = 4508 - def __init__(self, computedMd5Sum, expectedMd5Sum): + def __init__(self, computedMd5Sum=None, expectedMd5Sum=None): + GlusterHookException.__init__(self) self.computedMd5Sum = computedMd5Sum self.expectedMd5Sum = expectedMd5Sum self.message = 'Hook file check sum:%s mismatch %s' % (computedMd5Sum, @@ -524,6 +536,7 @@ code = 4551 def __init__(self, action=''): + GlusterServiceException.__init__(self) prefix = "%s: " % (action) self.message = prefix + "Service action is not supported" self.err = [self.message] -- To view, visit https://gerrit.ovirt.org/41530 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic4c72daf30aee8f0024b1ef9ee064a205a500620 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Timothy Asir <tjeyasin(a)redhat.com>

6 27

Change in vdsm[master]: vm: remove unused 'output' value in response
by fromani＠redhat.com 23 Aug '15

23 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: vm: remove unused 'output' value in response ...................................................................... vm: remove unused 'output' value in response The 'output' value is not documented in schema, and apparently Engine doesn't care about it. This patch removes it. Change-Id: I0721885876fbe15651737aad4f3b903deb5ef4bb Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/16/44116/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 83d8ad8..164ca29 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1168,7 +1168,7 @@ del self.conf['pauseCode'] except KeyError: pass - return {'status': doneCode, 'output': ['']} + return response.success() finally: if not guestCpuLocked: self._guestCpuLock.release() @@ -1185,7 +1185,7 @@ guestCpuLocked=True) self._logGuestCpuStatus('pause') self._lastStatus = afterState - return {'status': doneCode, 'output': ['']} + return response.success() finally: if not guestCpuLocked: self._guestCpuLock.release() -- To view, visit https://gerrit.ovirt.org/44116 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0721885876fbe15651737aad4f3b903deb5ef4bb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 23

Change in vdsm[master]: tests: functional - convert to run over jsonrpc
by ykaplan＠redhat.com 22 Aug '15

22 Aug '15

Yeela Kaplan has uploaded a new change for review. Change subject: tests: functional - convert to run over jsonrpc ...................................................................... tests: functional - convert to run over jsonrpc will run functional tests by default over jsonrpc if its enabled, otherwise will run over xmlrpc. Change-Id: I468e7eae74dd37b915b15e18ac4b0a830675cb47 Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M lib/vdsm/jsonrpcvdscli.py M tests/functional/utils.py 2 files changed, 17 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/37/43237/1 diff --git a/lib/vdsm/jsonrpcvdscli.py b/lib/vdsm/jsonrpcvdscli.py index 64c9879..affe281 100644 --- a/lib/vdsm/jsonrpcvdscli.py +++ b/lib/vdsm/jsonrpcvdscli.py @@ -37,6 +37,16 @@ 'destroy': 'VM.destroy', 'getVmStats': 'VM.getStats', 'migrationCreate': 'VM.migrationCreate', + 'updateVmPolicy': 'VM.updateVmPolicy', + 'getVdsCapabilities': 'Host.getCapabilities', + 'list': 'Host.getVMList', + 'getAllVmStats': 'Host.getAllVmStats', + 'getVdsStats': 'Host.getStats', + 'setupNetworks': 'Host.setupNetworks', + 'editNetwork': 'Host.editNetwork', + 'delNetwork': 'Host.delNetwork', + 'addNetwork': 'Host.addNetwork', + 'setSafeNetworkConfig': 'Host.setSafeNetworkConfig', } diff --git a/tests/functional/utils.py b/tests/functional/utils.py index 263fd31..05d5f5c 100644 --- a/tests/functional/utils.py +++ b/tests/functional/utils.py @@ -26,6 +26,7 @@ from vdsm.config import config from vdsm.utils import retry from vdsm import ipwrapper +from vdsm import jsonrpcvdscli from vdsm import netinfo from vdsm import vdscli from vdsm.netconfpersistence import RunningConfig @@ -71,7 +72,12 @@ retry(self.start, (socket.error, KeyError), tries=30) def start(self): - self.vdscli = vdscli.connect() + if config.getboolean('vars', 'jsonrpc_enable'): + requestQueues = config.get('addresses', 'request_queues') + requestQueue = requestQueues.split(",")[0] + self.vdscli = jsonrpcvdscli.create(requestQueue) + else: + self.vdscli = vdscli.connect() self.netinfo = self._get_netinfo() if config.get('vars', 'net_persistence') == 'unified': self.config = RunningConfig() -- To view, visit https://gerrit.ovirt.org/43237 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I468e7eae74dd37b915b15e18ac4b0a830675cb47 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

5 26

Change in vdsm[master]: json-rpc: export setLogLevel API
by fromani＠redhat.com 21 Aug '15

21 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: json-rpc: export setLogLevel API ...................................................................... json-rpc: export setLogLevel API This API was already available over XMl-RPC. There is no reason to leave JSON-RPC behind. Change-Id: I7481a59fb5f0598a671118394e368eab79a381b5 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/rpc/Bridge.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/24/38424/1 diff --git a/vdsm/rpc/Bridge.py b/vdsm/rpc/Bridge.py index 1d44b90..62773df 100644 --- a/vdsm/rpc/Bridge.py +++ b/vdsm/rpc/Bridge.py @@ -415,6 +415,7 @@ 'Host_getVMList': {'call': Host_getVMList_Call, 'ret': 'vmList'}, 'Host_getVMFullList': {'call': Host_getVMFullList_Call, 'ret': 'vmList'}, 'Host_getAllVmStats': {'ret': 'statsList'}, + 'Host_setLogLevel': {}, 'Host_setupNetworks': {'ret': 'status'}, 'Image_cloneStructure': {'ret': 'uuid'}, 'Image_delete': {'ret': 'uuid'}, -- To view, visit https://gerrit.ovirt.org/38424 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7481a59fb5f0598a671118394e368eab79a381b5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

6 18

Change in vdsm[master]: API: restore setLogLevel backward compatibility
by fromani＠redhat.com 21 Aug '15

21 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: API: restore setLogLevel backward compatibility ...................................................................... API: restore setLogLevel backward compatibility Host.setLogLevel implementations was different from the one documented in the schema, but it was consistent to what vdsClient sends. This suggests that the API was somehow used through vdsClient, although not in the advertised way. This patch adds fallback backward compatibility to the old format. Change-Id: I7afabd0676f420621e04377f7781a9133671df05 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/API.py 1 file changed, 23 insertions(+), 13 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/42419/1 diff --git a/vdsm/API.py b/vdsm/API.py index de65e4f..17d306f 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1379,22 +1379,13 @@ Doesn't survive a restart """ - LEVELS = { - 'DEBUG': logging.DEBUG, - 'INFO': logging.INFO, - 'WARNING': logging.WARNING, - 'ERROR': logging.ERROR, - 'CRITICAL': logging.CRITICAL - } - - try: - log_level = LEVELS[level.upper()] - except KeyError: - return errCode['unavail'] - else: + log_level = _parse_log_level(level) + if log_level is not None: logging.warning('Setting loglevel to %s (%d)', level, log_level) _set_log_level(logging.getLogger(), log_level) return {'status': doneCode} + else: + return errCode['unavail'] # VM-related functions def getVMList(self, fullStatus=False, vmList=(), onlyUUID=False): @@ -1820,6 +1811,25 @@ options[_translationMap[k]] = options.pop(k) +def _parse_log_level(level): + LEVELS = { + 'DEBUG': logging.DEBUG, + 'INFO': logging.INFO, + 'WARNING': logging.WARNING, + 'ERROR': logging.ERROR, + 'CRITICAL': logging.CRITICAL + } + + try: + return LEVELS[level.upper()] + except KeyError: + # for backward compatibility + try: + return int(level) + except ValueError: + return None + + def _set_log_level(logger, log_level): for handler in logger.handlers: if isinstance(handler, logging.FileHandler): -- To view, visit https://gerrit.ovirt.org/42419 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7afabd0676f420621e04377f7781a9133671df05 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 8

Change in vdsm[master]: client: update to keep up with setLogLevel changes
by fromani＠redhat.com 21 Aug '15

21 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: client: update to keep up with setLogLevel changes ...................................................................... client: update to keep up with setLogLevel changes Bring vdsClient up to speed with setLogLevel changes. Change-Id: I094201e0ef4f739839061f323e7db2d92dbd7c9b Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M client/vdsClient.py 1 file changed, 12 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/83/42283/1 diff --git a/client/vdsClient.py b/client/vdsClient.py index 98f46ea..37c1090 100755 --- a/client/vdsClient.py +++ b/client/vdsClient.py @@ -1562,9 +1562,13 @@ return 0, '' def do_setLogLevel(self, args): - validateArgTypes(args, [int], requiredArgsNumber=1) + validateArgTypes(args, [str, str], requiredArgsNumber=1) level = args[0] - stats = self.s.setLogLevel(level) + if len(args) >= 2: + name = args[1] + else: + args = None + stats = self.s.setLogLevel(level, name) if stats['status']['code']: return stats['status']['code'], stats['status']['message'] return 0, '' @@ -2556,8 +2560,12 @@ 'prepareForShutdown': (serv.prepareForShutdown, ('', '')), 'setLogLevel': (serv.do_setLogLevel, - ('<level> [logName][,logName]...', 'set log verbosity' - ' level (10=DEBUG, 50=CRITICAL' + ('<level> [loggerName]', + 'set log verbosity. Arguments:\n' + ' <level> is one of DEBUG INFO WARNING ERROR CRITICAL' + ' case sensitive\n' + ' <loggerName> is the logger to set.' + ' Skip it to set the system logger' )), 'setMOMPolicy': (serv.do_setMOMPolicy, ('<policyfile>', 'set MOM policy')), -- To view, visit https://gerrit.ovirt.org/42283 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I094201e0ef4f739839061f323e7db2d92dbd7c9b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 10

Change in vdsm[master]: netinfo: Replace misused asserts with InvalidConfiguration
by Nir Soffer 19 Aug '15

19 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: netinfo: Replace misused asserts with InvalidConfiguration ...................................................................... netinfo: Replace misused asserts with InvalidConfiguration The code was assuming that asserts are always available, and if running with in optimized mode, the code would skip the assert and use invalid configuration, possibly corrupting application state or failing with unrelated errors, hiding the root cause of the error. The asserts are replaced now with new InvalidConfiguration exception and the error messages are improved. Change-Id: I99ee49aa7e2364f57112e2452e3eab3940b6b00a Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/netinfo.py 1 file changed, 17 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/61/34361/1 diff --git a/lib/vdsm/netinfo.py b/lib/vdsm/netinfo.py index d818249..e848abc 100644 --- a/lib/vdsm/netinfo.py +++ b/lib/vdsm/netinfo.py @@ -84,6 +84,10 @@ DUMMY_BRIDGE # Appease flake8 since dummy bridge should be exported from here +class InvalidConfiguration(Exception): + """ Raise when getting an invalid conifguration """ + + def _visible_devs(predicate): """Returns a list of visible (vdsm manageable) links for which the predicate is True""" @@ -834,8 +838,9 @@ bondings = [b for (b, attrs) in self.bondings.iteritems() if nic in attrs['slaves']] if bondings: - assert len(bondings) == 1, \ - "Unexpected configuration: More than one bonding per nic" + if len(bondings) != 1: + raise InvalidConfiguration( + "Unexpected configuration: More than one bonding per nic") return bondings[0] return None @@ -853,13 +858,20 @@ for port in ports: if port in self.vlans: - assert vlan is None + if vlan is not None: + raise InvalidConfiguration( + "Unexpected vlan: %s exepected: None" % (vlan,)) nic = getVlanDevice(port) vlan = getVlanID(port) - assert self.vlans[port]['iface'] == nic + if self.vlans[port]['iface'] != nic: + raise InvalidConfiguration( + "Unexpected iface: %s expected: %s" % + (self.vlans[port]['iface'], nic)) port = nic if port in self.bondings: - assert bonding is None + if bonding is not None: + raise InvalidConfiguration( + "Unexpected bonding: %s expected: None" % bonding) bonding = port lnics += self.bondings[bonding]['slaves'] elif port in self.nics: -- To view, visit http://gerrit.ovirt.org/34361 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I99ee49aa7e2364f57112e2452e3eab3940b6b00a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 23

Change in vdsm[master]: tests: Add RWLock tests
by Nir Soffer 18 Aug '15

18 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Add RWLock tests ...................................................................... tests: Add RWLock tests Current RWLock look too complex and has no tests. This patch fixes the later issue. Change-Id: If404f0e8c68fcdb2f7643bdd6d5c1f97f230a227 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A tests/rwlock_test.py 1 file changed, 219 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/07/42907/1 diff --git a/tests/rwlock_test.py b/tests/rwlock_test.py new file mode 100644 index 0000000..80d04f1 --- /dev/null +++ b/tests/rwlock_test.py @@ -0,0 +1,219 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from __future__ import absolute_import, print_function +import threading +import time + +from testlib import VdsmTestCase +from testValidation import slowtest, stresstest + +from storage.misc import RWLock + + +class RWLockTests(VdsmTestCase): + + @slowtest + def test_concurrent_readers(self): + lock = RWLock() + readers = [] + log = [] + with self.assertElapsed(0.1, 0.1): + for i in range(5): + t = start_thread(reader, lock, log, hold=0.1) + readers.append(t) + for t in readers: + t.join() + self.assertEqual(log, ["reader acquired"] * 5) + + @slowtest + def test_wakeup_blocked_writer(self): + lock = RWLock() + readers = [] + log = [] + try: + for i in range(5): + t = start_thread(reader, lock, log, hold=0.2) + readers.append(t) + time.sleep(0.1) + # All readers should hold the lock now + with self.assertElapsed(0.1, 0.02): + # Should block until last reader release the lock + lock.acquireWrite() + lock.release() + finally: + for t in readers: + t.join() + + @slowtest + def test_wakeup_blocked_readers(self): + lock = RWLock() + readers = [] + log = [] + lock.acquireWrite() + try: + for i in range(10): + t = start_thread(reader, lock, log) + readers.append(t) + time.sleep(0.1) + # All readers should be blocked now + finally: + lock.release() + # Should wake up all readers, taking a read lock + for t in readers: + t.join() + self.assertEqual(log, ["reader acquired"] * 10) + + @slowtest + def test_release_other_thread_write_lock(self): + lock = RWLock() + t = start_thread(writer, lock, [], hold=0.2) + try: + time.sleep(0.1) + self.assertRaises(RuntimeError, lock.release) + finally: + t.join() + + @slowtest + def test_release_other_thread_read_lock(self): + lock = RWLock() + t = start_thread(reader, lock, [], hold=0.2) + try: + time.sleep(0.1) + self.assertRaises(RuntimeError, lock.release) + finally: + t.join() + + @slowtest + def test_fifo(self): + lock = RWLock() + log = [] + lock.acquireRead() + try: + # t1 is waiting on the first reader... + t1 = start_thread(writer, lock, log) + time.sleep(0.1) + # t2 joined after t1, so it should get the lock after it + t2 = start_thread(reader, lock, log) + time.sleep(0.1) + finally: + lock.release() + t1.join() + t2.join() + self.assertEqual(log, ["writer acquired", "reader acquired"]) + + @stresstest + def test_fairness(self): + lock = RWLock() + ready = threading.Event() + done = threading.Event() + reads = [0] + writes = [0] + + def read(): + ready.wait() + while not done.is_set(): + lock.acquireRead() + try: + reads[0] += 1 + finally: + lock.release() + + def write(): + ready.wait() + while not done.is_set(): + lock.acquireWrite() + try: + writes[0] += 1 + finally: + lock.release() + + t1 = start_thread(read) + t2 = start_thread(write) + try: + time.sleep(0.1) + ready.set() + time.sleep(1) + finally: + done.set() + t1.join() + t2.join() + + print('reads: ', reads[0], 'writes:', writes[0]) + self.assertTrue(abs(reads[0] - writes[0]) < reads[0] / 10) + + +class HelpersTests(VdsmTestCase): + """ + Test helpers used in the RWLock tests. + """ + + def test_writer(self): + lock = RWLock() + log = [] + start_thread(writer, lock, log).join() + self.assertEqual(log, ["writer acquired"]) + + @slowtest + def test_writer_hold(self): + lock = RWLock() + log = [] + with self.assertElapsed(0.1, 0.05): + start_thread(writer, lock, log, hold=0.1).join() + self.assertEqual(log, ["writer acquired"]) + + def test_reader(self): + lock = RWLock() + log = [] + start_thread(reader, lock, log, hold=0).join() + self.assertEqual(log, ["reader acquired"]) + + @slowtest + def test_reader_hold(self): + lock = RWLock() + log = [] + with self.assertElapsed(0.1, 0.05): + start_thread(reader, lock, log, hold=0.1).join() + self.assertEqual(log, ["reader acquired"]) + + +def writer(lock, log, hold=0): + lock.acquireWrite() + try: + log.append("writer acquired") + time.sleep(hold) + finally: + lock.release() + + +def reader(lock, log, hold=0): + lock.acquireRead() + try: + log.append("reader acquired") + time.sleep(hold) + finally: + lock.release() + + +def start_thread(func, *args, **kwargs): + t = threading.Thread(target=func, args=args, kwargs=kwargs) + t.daemon = True + t.start() + return t -- To view, visit https://gerrit.ovirt.org/42907 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If404f0e8c68fcdb2f7643bdd6d5c1f97f230a227 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 43

Change in vdsm[master]: clientIF: add logs during the recovery
by fromani＠redhat.com 18 Aug '15

18 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: clientIF: add logs during the recovery ...................................................................... clientIF: add logs during the recovery The recovery flow is supposed to be sporadic, but it is nevertheless very important. The current flow is pretty opaque, and on big setups it may take some time, leaving the admin to guess what's going on. This patch adds logs during the process, to make it obvious what is happened, what is going on and what is left to do. Change-Id: I31dddf0a2bc760c5ad383ff6bfee9a72adc87c4f Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/clientIF.py 1 file changed, 44 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/70/43770/1 diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index ef27874..0223cb6 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -463,36 +463,55 @@ caps.CpuTopology().cores()) migration.SourceThread.setMaxOutgoingMigrations(mog) - # Recover - for v in getVDSMDomains(): + # Recover stage 1: domains from libvirt + doms = getVDSMDomains() + for idx, v in enumerate(doms): vmId = v.UUIDString() - if not self._recoverVm(vmId): + if self._recoverVm(vmId): + self.log.info( + 'recovery: from libvirt got domain %d/%d: %s', + idx, len(doms), vmId) + else: # RH qemu proc without recovery - self.log.info('loose qemu process with id: ' - '%s found, killing it.', vmId) + self.log.info( + 'recovery: loose qemu process with id: ' + '%s found, killing it.', vmId) try: v.destroy() except libvirt.libvirtError: - self.log.error('failed to kill loose qemu ' - 'process with id: %s', - vmId, exc_info=True) + self.log.exception( + 'recovery: failed to kill loose qemu ' + 'process with id: %s', vmId) + # Recover stage 2: domains from recovery files # we do this to safely handle VMs which disappeared # from the host while VDSM was down/restarting recVms = self._getVDSMVmsFromRecovery() if recVms: - self.log.warning('Found %i VMs from recovery files not' + self.log.warning('recovery: found %i VMs from data files not' ' reported by libvirt.' ' This should not happen!' ' Will try to recover them.', len(recVms)) - for vmId in recVms: - if not self._recoverVm(vmId): - self.log.warning('VM %s failed to recover from recovery' - ' file, reported as Down', vmId) + for idx, vmId in enumerate(recVms): + if self._recoverVm(vmId): + self.log.info( + 'recovery: from data file domain %d/%d: %s', + idx, len(recVms), vmId) + else: + self.log.warning( + 'recovery: VM %s failed to recover from data' + ' file, reported as Down', vmId) - while (self._enabled and - vmstatus.WAIT_FOR_LAUNCH in [v.lastStatus for v in - self.vmContainer.values()]): + # recover stage 3: waiting for domains to go up + while self._enabled: + launching = sum(v.lastStatus == vmstatus.WAIT_FOR_LAUNCH + for v in self.vmContainer.values()]) + if not launching: + break + else: + self.log.info( + 'recovery: waiting for %d domains to go up', + launching) time.sleep(1) self._cleanOldFiles() self._recovery = False @@ -503,20 +522,25 @@ # volumes manipulations while self._enabled and self.vmContainer and \ not self.irs.getConnectedStoragePoolsList()['poollist']: + self.log.info('recovery: waiting for storage pool to go up') time.sleep(5) - for vmId, vmObj in self.vmContainer.items(): + vmObjects = self.vmContainer.values() + for idx, vmObj in enumerate(vmObjects): # Let's recover as much VMs as possible try: # Do not prepare volumes when system goes down if self._enabled: + self.log.info( + 'recovery: preparing paths for vm %d/%d: %s', + idx, len(vmObjects), vmObj.id) vmObj.preparePaths( vmObj.devSpecMapFromConf()[hwclass.DISK]) except: - self.log.error("Vm %s recovery failed", - vmId, exc_info=True) + self.log.exception( + "recovery: failed for vm %s", vmObj.id) except: - self.log.error("Vm's recovery failed", exc_info=True) + self.log.exception("recovery: failed") raise def _getVDSMVmsFromRecovery(self): -- To view, visit https://gerrit.ovirt.org/43770 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I31dddf0a2bc760c5ad383ff6bfee9a72adc87c4f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 24

Change in vdsm[master]: v2v: Import VM from OVA file
by shavivi＠redhat.com 17 Aug '15

17 Aug '15

Shahar Havivi has uploaded a new change for review. Change subject: v2v: Import VM from OVA file ...................................................................... v2v: Import VM from OVA file OVA is a tar file which contain a VM with its Ovf file and its disks as well. Change-Id: I0e440748ecc503f4d61e8f4f61bb0c7387589354 Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M vdsm/API.py M vdsm/rpc/bindingxmlrpc.py M vdsm/rpc/vdsmapi-schema.json M vdsm/v2v.py 4 files changed, 58 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/67/43367/1 diff --git a/vdsm/API.py b/vdsm/API.py index 4c3724e..704f37b 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1440,6 +1440,9 @@ return v2v.convert_external_vm(uri, username, password, vminfo, jobid, self._cif.irs) + def convertOva(self, ova_path, vminfo, jobid): + return v2v.convert_ova(ova_path, vminfo, jobid, self._cif.irs) + def getConvertedVm(self, jobid): return v2v.get_converted_vm(jobid) diff --git a/vdsm/rpc/bindingxmlrpc.py b/vdsm/rpc/bindingxmlrpc.py index b9dd54e..bc74e22 100644 --- a/vdsm/rpc/bindingxmlrpc.py +++ b/vdsm/rpc/bindingxmlrpc.py @@ -381,6 +381,10 @@ api = API.Global() return api.convertExternalVm(uri, username, password, vminfo, jobid) + def convertOva(self, ova_path, vminfo, jobid): + api = API.Global() + return api.convert_ova(ova_path, vminfo, jobid) + def getConvertedVm(self, jobid): api = API.Global() return api.getConvertedVm(jobid) @@ -1109,6 +1113,7 @@ (self.getExternalVMs, 'getExternalVMs'), (self.getOvaInfo, 'getOvaInfo'), (self.convertExternalVm, 'convertExternalVm'), + (self.convertOva, 'convertOva'), (self.getConvertedVm, 'getConvertedVm'), (self.abortV2VJob, 'abortV2VJob'), (self.deleteV2VJob, 'deleteV2VJob'), diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 291aee5..b4c577b 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -3947,6 +3947,23 @@ 'data': {'ova_path': 'str'}, 'returns': 'ExternalVmInfo'} +## +# @Host.convertOva: +# +# Convert VM from external file (OVA) to data domain +# +# @ova_path: actual path to the ova file +# +# @vminfo: information of the VM such as name, id etc +# +# @jobid: Assign a UUID to this operation which can be used +# to identify it in @HostStats +# +# Since: 4.17.0 +## +{'command': {'class': 'Host', 'name': 'convertOva'}, + 'data': {'ova_path': 'str', 'vminfo': 'ExternalVmInfo', + 'jobid': 'UUID'}} ## # @Host.convertExternalVm: diff --git a/vdsm/v2v.py b/vdsm/v2v.py index e891894..28fb55e 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -167,6 +167,13 @@ return {'status': doneCode} +def convert_ova(ova_path, vminfo, job_id, irs): + job = ImportVm.from_ova(ova_path, vminfo, job_id, irs) + job.start() + _add_job(job_id, job) + return {'status': doneCode} + + def get_ova_info(ova_path): ns = {'ovf': _OVF_NS, 'rasd': _RASD_NS} @@ -325,6 +332,8 @@ self._passwd_file = None self._create_command = None + self._ova_path = None + @classmethod def from_libvirt(cls, uri, username, password, vminfo, job_id, irs): obj = cls(vminfo, job_id, irs) @@ -334,6 +343,14 @@ obj._password = password obj._passwd_file = os.path.join(_V2V_DIR, "%s.tmp" % job_id) obj._create_command = cls._from_libvirt_command + return obj + + @classmethod + def from_ova(cls, ova_path, vminfo, job_id, irs): + obj = cls(vminfo, job_id, irs) + + obj._ova_path = ova_path + obj._create_command = cls._from_ova_command return obj def start(self): @@ -453,6 +470,22 @@ self._vminfo['vmName']]) return cmd + def _from_ova_command(self): + cmd = [_VIRT_V2V.cmd, + '-i', 'ova', self._ova_path, + '-o', 'vdsm', + '-of', self._get_disk_format(), + '-oa', self._vminfo.get('allocation', 'sparse').lower(), + self._generate_disk_parameters(), + '--vdsm-vm-uuid', + self._id, + '--vdsm-ovf-output', + _V2V_DIR, + '--machine-readable', + '-os', + get_storage_domain_path(self._prepared_volumes[0]['path'])] + return cmd + def abort(self): self._status = STATUS.ABORTED logging.info('Job %r aborting...', self._id) -- To view, visit https://gerrit.ovirt.org/43367 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0e440748ecc503f4d61e8f4f61bb0c7387589354 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

5 24

Change in vdsm[master]: v2v: get VM information from OVA file
by shavivi＠redhat.com 17 Aug '15

17 Aug '15

Shahar Havivi has uploaded a new change for review. Change subject: v2v: get VM information from OVA file ...................................................................... v2v: get VM information from OVA file In order to import a VM that exists in an OVA file the oVirt engine needs to get information regarding the VM, such as disks sizes, interfaces memory etc. Adding a new verb: getOvfInfo Change-Id: I3c55c846f837bb5bf363717e05daabf5ee6631ca Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M vdsm/API.py M vdsm/rpc/Bridge.py M vdsm/rpc/bindingxmlrpc.py M vdsm/rpc/vdsmapi-schema.json M vdsm/v2v.py 5 files changed, 97 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/43271/1 diff --git a/vdsm/API.py b/vdsm/API.py index eafceb7..645b962 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1418,6 +1418,13 @@ """ return v2v.get_external_vms(uri, username, password) + def getOvaInfo(self, ova_path): + """ + return a dictionary, information regading a VM that is a part + of the ova + """ + return v2v.get_ova_info(ova_path) + def convertExternalVm(self, uri, username, password, vminfo, jobid): return v2v.convert_external_vm(uri, username, password, vminfo, jobid, self._cif.irs) diff --git a/vdsm/rpc/Bridge.py b/vdsm/rpc/Bridge.py index 4302baa..99396c1 100644 --- a/vdsm/rpc/Bridge.py +++ b/vdsm/rpc/Bridge.py @@ -410,6 +410,7 @@ 'Host_getDeviceList': {'ret': 'devList'}, 'Host_getDevicesVisibility': {'ret': 'visible'}, 'Host_getExternalVMs': {'ret': 'vmList'}, + 'Host_getOvaInfo': {'ret': 'vm'}, 'Host_getConvertedVm': {'ret': 'ovf'}, 'Host_getHardwareInfo': {'ret': 'info'}, 'Host_getLVMVolumeGroups': {'ret': 'vglist'}, diff --git a/vdsm/rpc/bindingxmlrpc.py b/vdsm/rpc/bindingxmlrpc.py index 05110c4..c3d4022 100644 --- a/vdsm/rpc/bindingxmlrpc.py +++ b/vdsm/rpc/bindingxmlrpc.py @@ -372,6 +372,10 @@ api = API.Global() return api.getExternalVMs(uri, username, password) + def getOvaInfo(self, ova_path): + api = API.Global() + return api.getOvaInfo(ova_path) + def convertExternalVm(self, uri, username, password, vminfo, jobid): password = ProtectedPassword(password) api = API.Global() @@ -1093,6 +1097,7 @@ (self.vmSetCpuTuneQuota, 'vmSetCpuTuneQuota'), (self.vmSetCpuTunePeriod, 'vmSetCpuTunePeriod'), (self.getExternalVMs, 'getExternalVMs'), + (self.getOvaInfo, 'getOvaInfo'), (self.convertExternalVm, 'convertExternalVm'), (self.getConvertedVm, 'getConvertedVm'), (self.abortV2VJob, 'abortV2VJob'), diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json index 7c4fe1c..08b85d7 100644 --- a/vdsm/rpc/vdsmapi-schema.json +++ b/vdsm/rpc/vdsmapi-schema.json @@ -3930,6 +3930,24 @@ 'data': {'uri': 'str', 'username': 'str', 'password': 'str'}, 'returns': ['ExternalVmInfo']} + +## +# @Host.getOvaInfo: +# +# Get VM information (disks, interfaces, memory etc) from OVA file +# +# @ova_path: path to the ova file +# +# Returns: +# A dictionary with information regarding the VM +# +# Since: 4.17.0 +## +{'command': {'class': 'Host', 'name': 'getOvaInfo'}, + 'data': {'ova_path': 'str'}, + 'returns': 'ExternalVmInfo'} + + ## # @Host.convertExternalVm: # diff --git a/vdsm/v2v.py b/vdsm/v2v.py index 18cc9bc..e891894 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -50,6 +50,10 @@ _V2V_DIR = os.path.join(P_VDSM_RUN, 'v2v') _VIRT_V2V = CommandPath('virt-v2v', '/usr/bin/virt-v2v') +_OVF_NS = 'http://schemas.dmtf.org/ovf/envelope/1' +_RASD_NS = 'http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/' \ + 'CIM_ResourceAllocationSettingData' + ImportProgress = namedtuple('ImportProgress', ['current_disk', 'disk_count', 'description']) DiskProgress = namedtuple('DiskProgress', ['progress']) @@ -161,6 +165,22 @@ job.start() _add_job(job_id, job) return {'status': doneCode} + + +def get_ova_info(ova_path): + ns = {'ovf': _OVF_NS, 'rasd': _RASD_NS} + + try: + root = ET.fromstring(_read_ovf_from_ova(ova_path)) + except ET.ParseError as e: + raise V2VError('Error reading ovf from ova, position: %r' % e.position) + + vm = {} + _add_general_ovf_info(vm, root, ns) + _add_disks_ovf_info(vm, root, ns) + _add_networks_ovf_info(vm, root, ns) + + return {'status': doneCode, 'vm': vm} def get_converted_vm(job_id): @@ -651,3 +671,49 @@ if model is not None: i['model'] = model.get('type') params['networks'].append(i) + + +def _read_ovf_from_ova(ova_path): + cmd = ['/usr/bin/tar', 'xf', ova_path, '*.ovf', '--to-stdout'] + rc, output, error = execCmd(cmd) + if rc: + raise V2VError(error) + + return output + + +def _add_general_ovf_info(vm, node, ns): + try: + vm['status'] = 'Down' + vm['vmName'] = node.find('./ovf:VirtualSystem/ovf:Name', ns).text + vm['memSize'] = node.find('.//ovf:Item[rasd:ResourceType="4"]/' + 'rasd:VirtualQuantity', ns).text + vm['smp'] = node.find('.//ovf:Item[rasd:ResourceType="3"]/' + 'rasd:VirtualQuantity', ns).text + except Exception as e: + raise V2VError('Error parsing ovf information: %s' % e.message) + + +def _add_disks_ovf_info(vm, node, ns): + try: + vm['disks'] = [] + for d in node.findall(".//ovf:DiskSection/ovf:Disk", ns): + disk = {} + disk['capacity'] = str(d.attrib.get('{%s}capacity' % _OVF_NS)) + disk['allocation'] = str(d.attrib.get('{%s}populatedSize' + % _OVF_NS)) + vm['disks'].append(disk) + except Exception as e: + raise V2VError('Error parsing ovf disk information: %s' % e.message) + + +def _add_networks_ovf_info(vm, node, ns): + try: + vm['networks'] = [] + for n in node.findall('.//ovf:Item[rasd:ResourceType="10"]', ns): + net = {} + net['dev'] = net.find('./rasd:ElementName', ns).text + net['model'] = net.find('./rasd:ResourceSubType', ns).text + vm['networks'].append(net) + except Exception as e: + raise V2VError('Error parsing ovf network information: %s' % e.message) -- To view, visit https://gerrit.ovirt.org/43271 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3c55c846f837bb5bf363717e05daabf5ee6631ca Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

6 33

Change in vdsm[master]: ipwrapper: Use zombiereaper to wait for process
by Nir Soffer 16 Aug '15

16 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: ipwrapper: Use zombiereaper to wait for process ...................................................................... ipwrapper: Use zombiereaper to wait for process Commit aea8aab95f eliminated ip process zombies by waiting for ip process. This is may introduce unlimited wait if the process is not responsive. We can avoid this issue by using zombiereaper. Note that zombiereaper requires explicit initialization (registering signal handler for SIGCHLD), so this make ipwrapper less useful as generic utility. Change-Id: I26263c5b4811d7f46f7c65d1d1b00bd96af02eb8 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/ipwrapper.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/16/32216/1 diff --git a/lib/vdsm/ipwrapper.py b/lib/vdsm/ipwrapper.py index af31159..781e95b 100644 --- a/lib/vdsm/ipwrapper.py +++ b/lib/vdsm/ipwrapper.py @@ -27,6 +27,7 @@ import signal import socket import struct +import zombiereaper from netaddr.core import AddrFormatError from netaddr import IPAddress @@ -638,7 +639,7 @@ def stop(self): self.proc.kill() - self.proc.wait() + zombiereaper.autoReapPID(self.proc.pid) @classmethod def _parseLine(cls, line): -- To view, visit http://gerrit.ovirt.org/32216 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I26263c5b4811d7f46f7c65d1d1b00bd96af02eb8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 6

Change in vdsm[master]: vdscli: jsonrpc: fix getVdsCapabilities and Stats result
by ykaplan＠redhat.com 16 Aug '15

16 Aug '15

Yeela Kaplan has uploaded a new change for review. Change subject: vdscli: jsonrpc: fix getVdsCapabilities and Stats result ...................................................................... vdscli: jsonrpc: fix getVdsCapabilities and Stats result so it will be compatible with result we receive using xmlrpc vdscli Change-Id: Ibd0c7f787b1ad3475328080d9043046ce81a2cbd Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M lib/vdsm/jsonrpcvdscli.py 1 file changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/36/43436/1 diff --git a/lib/vdsm/jsonrpcvdscli.py b/lib/vdsm/jsonrpcvdscli.py index 16a5d5e..c56ec1d 100644 --- a/lib/vdsm/jsonrpcvdscli.py +++ b/lib/vdsm/jsonrpcvdscli.py @@ -45,6 +45,15 @@ def __init__(self, client): self._client = client + def _addInfo(self, res): + newRes = { + 'status': res['status'] + } + del res['status'] + newRes['info'] = res + + return newRes + def _callMethod(self, methodName, *args): try: method = _COMMAND_CONVERTER[methodName] @@ -68,7 +77,12 @@ # None is translated to True inside our JSONRPC implementation return response.success(**resp.result) - return response.success() + result = response.success() + + if method in ('getVdsCapabilities', 'getVdsStats'): + return self._addInfo(result) + + return result def migrationCreate(self, params): return self._callMethod('migrationCreate', -- To view, visit https://gerrit.ovirt.org/43436 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibd0c7f787b1ad3475328080d9043046ce81a2cbd Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

3 5

Change in vdsm[master]: storage: Refactor new volume metadata generation
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: storage: Refactor new volume metadata generation ...................................................................... storage: Refactor new volume metadata generation To prepare for garbage-collectible volume metadata needed by the SDM flows, we need to break out the functionality of creating new volume metadata. Introduce makeMetadata() which builds a metadata dict from a given set of args. Also, modify _putMetadata to allow an optional extension to be supplied. Change-Id: I028996b3a5967f96a103f0b03d6b8f82b25da888 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 2 files changed, 12 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/45/44045/1 diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index f408e95..15ca05b 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -234,9 +234,11 @@ self.file_setrw(self.getVolumePath(), rw=rw) @classmethod - def _putMetadata(cls, metaId, meta): + def _putMetadata(cls, metaId, meta, gcSuffix=None): volPath, = metaId metaPath = cls._metaVolumePath(volPath) + if gcSuffix: + metaPath += gcSuffix with open(metaPath + ".new", "w") as f: for key, value in meta.iteritems(): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 723b3d9..0d61643 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -456,7 +456,15 @@ @classmethod def newMetadata(cls, metaId, sdUUID, imgUUID, puuid, size, format, type, voltype, disktype, desc="", legality=ILLEGAL_VOL): - meta = { + meta = cls.makeMetadata(sdUUID, imgUUID, puuid, size, format, type, + voltype, disktype, desc, legality) + cls.createMetadata(metaId, meta) + return meta + + @classmethod + def makeMetadata(cls, sdUUID, imgUUID, puuid, size, format, type, voltype, + disktype, desc="", legality=ILLEGAL_VOL): + return { FORMAT: str(format), TYPE: str(type), VOLTYPE: str(voltype), @@ -471,9 +479,6 @@ MTIME: 0, LEGALITY: str(legality), } - - cls.createMetadata(metaId, meta) - return meta def refreshVolume(self): pass -- To view, visit https://gerrit.ovirt.org/44045 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I028996b3a5967f96a103f0b03d6b8f82b25da888 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 5

Change in vdsm[master]: StorageDomainManifest: Manifest objects produce VolumeMetada...
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: StorageDomainManifest: Manifest objects produce VolumeMetadata objects ...................................................................... StorageDomainManifest: Manifest objects produce VolumeMetadata objects When we are working directly with StorageDomainManifest objects we do not want to produce Volume objects. Instead, we'll produce VolumeMetadata objects. The interface between Volume and VolumeMetadata is consistent (except that VolumeMetadata lacks functions that can only be called when working with an SPM). Change-Id: Iacb2723c8ce2108b3e5ac2f35be2f59f083095e0 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/manifest_tests.py M vdsm/storage/blockSD.py M vdsm/storage/fileSD.py M vdsm/storage/sd.py 4 files changed, 22 insertions(+), 9 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/40/44040/1 diff --git a/tests/manifest_tests.py b/tests/manifest_tests.py index 6891033..b5f0a7f 100644 --- a/tests/manifest_tests.py +++ b/tests/manifest_tests.py @@ -92,7 +92,7 @@ self.assertEquals('3', manifest.getFormat()) self.assertEquals(os.path.join(STORAGE_REPO, pooluuid), manifest.getRepoPath()) - self.assertEquals(fileVolume.FileVolume, + self.assertEquals(fileVolume.FileVolumeMetadata, manifest.getVolumeClass()) def test_getrepopath_with_iso_domain(self): @@ -504,7 +504,7 @@ self.assertEquals('3', manifest.getFormat()) self.assertEquals(os.path.join(STORAGE_REPO, poolID), manifest.getRepoPath()) - self.assertEquals(blockVolume.BlockVolume, + self.assertEquals(blockVolume.BlockVolumeMetadata, manifest.getVolumeClass()) def test_deleteimage(self): diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index 2aad14e..b48b236 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -583,7 +583,7 @@ """ Return a type specific volume generator object """ - return blockVolume.BlockVolume + return blockVolume.BlockVolumeMetadata def _getImgExclusiveVols(self, imgUUID, volsImgs): """Filter vols belonging to imgUUID only.""" @@ -1397,6 +1397,12 @@ raise se.StorageDomainDoesNotExist(sdUUID) + def getVolumeClass(self): + """ + Return a type specific volume generator object + """ + return blockVolume.BlockVolume + def _createVMSfs(dev): """ diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py index 1669c86..82e6fb5 100644 --- a/vdsm/storage/fileSD.py +++ b/vdsm/storage/fileSD.py @@ -187,7 +187,7 @@ """ Return a type specific volume generator object """ - return fileVolume.FileVolume + return fileVolume.FileVolumeMetadata def deleteImage(self, sdUUID, imgUUID, volsImgs): currImgDir = self.getImagePath(imgUUID) @@ -677,6 +677,12 @@ self.log.debug("Force linking %s to %s", tVol, tLink) self.oop.utils.forceLink(tVol, tLink) + def getVolumeClass(self): + """ + Return a type specific volume generator object + """ + return fileVolume.FileVolume + def _getMountsList(pattern="*"): fileDomPattern = os.path.join( diff --git a/vdsm/storage/sd.py b/vdsm/storage/sd.py index 70425c6..ddad760 100644 --- a/vdsm/storage/sd.py +++ b/vdsm/storage/sd.py @@ -374,8 +374,12 @@ pools.remove(BLANK_UUID) return pools - def getVolumeClass(self): - pass + def produceVolume(self, imgUUID, volUUID): + """ + Produce a type specific volume object + """ + return self.getVolumeClass()(self.mountpoint, self.sdUUID, imgUUID, + volUUID) def isISO(self): return self.getMetaParam(DMDK_CLASS) == ISO_DOMAIN @@ -574,9 +578,6 @@ """ return self.getVolumeClass()(self.mountpoint, self.sdUUID, imgUUID, volUUID) - - def getVolumeClass(self): - return self._manifest.getVolumeClass() def validateCreateVolumeParams(self, volFormat, srcVolUUID, preallocate=None): -- To view, visit https://gerrit.ovirt.org/44040 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iacb2723c8ce2108b3e5ac2f35be2f59f083095e0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 5

Change in vdsm[master]: VolumeMetadata: Move getImageVolumes
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: Move getImageVolumes ...................................................................... VolumeMetadata: Move getImageVolumes Change-Id: I4e99008b0df26a0bd2f714ac060ceee437e0060e Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py 2 files changed, 30 insertions(+), 22 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/43/44043/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 19ee9c0..47336b9 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -323,6 +323,15 @@ self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) os.symlink(self.getDevPath(), dstPath) + @classmethod + def getImageVolumes(cls, repoPath, sdUUID, imgUUID): + """ + Fetch the list of the Volumes UUIDs, not including the shared base + (template) + """ + lvs = lvm.lvsByTag(sdUUID, "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)) + return [lv.name for lv in lvs] + class BlockVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -664,12 +673,7 @@ @classmethod def getImageVolumes(cls, repoPath, sdUUID, imgUUID): - """ - Fetch the list of the Volumes UUIDs, not including the shared base - (template) - """ - lvs = lvm.lvsByTag(sdUUID, "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)) - return [lv.name for lv in lvs] + return BlockVolumeMetadata.getImageVolumes(repoPath, sdUUID, imgUUID) @classmethod def __putMetadata(cls, metaId, meta): diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 87feb18..f408e95 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -316,6 +316,25 @@ if sdCache.produce(self.sdUUID).hasVolumeLeases(): self._shareLease(dstImgPath) + @classmethod + def getImageVolumes(cls, repoPath, sdUUID, imgUUID): + """ + Fetch the list of the Volumes UUIDs, + not including the shared base (template) + """ + # Get Volumes of an image + pattern = os.path.join(repoPath, sdUUID, sd.DOMAIN_IMAGES, + imgUUID, "*.meta") + files = oop.getProcessPool(sdUUID).glob.glob(pattern) + volList = [] + for i in files: + volid = os.path.splitext(os.path.basename(i))[0] + if (sdCache.produce(sdUUID). + produceVolume(imgUUID, volid). + getImage() == imgUUID): + volList.append(volid) + return volList + class FileVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -494,22 +513,7 @@ @classmethod def getImageVolumes(cls, repoPath, sdUUID, imgUUID): - """ - Fetch the list of the Volumes UUIDs, - not including the shared base (template) - """ - # Get Volumes of an image - pattern = os.path.join(repoPath, sdUUID, sd.DOMAIN_IMAGES, - imgUUID, "*.meta") - files = oop.getProcessPool(sdUUID).glob.glob(pattern) - volList = [] - for i in files: - volid = os.path.splitext(os.path.basename(i))[0] - if (sdCache.produce(sdUUID). - produceVolume(imgUUID, volid). - getImage() == imgUUID): - volList.append(volid) - return volList + return cls.MetadataClass.getImageVolumes(repoPath, sdUUID, imgUUID) def setParentMeta(self, puuid): """ -- To view, visit https://gerrit.ovirt.org/44043 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4e99008b0df26a0bd2f714ac060ceee437e0060e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 5

Change in vdsm[master]: VolumeMetadata: volume share utilities
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: volume share utilities ...................................................................... VolumeMetadata: volume share utilities Change-Id: I43c7aaf4a8a8b8bce18331c0a55ba533192777f7 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 58 insertions(+), 49 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/04/42004/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index f14ce2a..4fac4e1 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -278,6 +278,15 @@ def refreshVolume(self): lvm.refreshLVs(self.sdUUID, (self.volUUID,)) + def _share(self, dstImgPath): + """ + Share this volume to dstImgPath + """ + dstPath = os.path.join(dstImgPath, self.volUUID) + + self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) + os.symlink(self.getDevPath(), dstPath) + class BlockVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -554,15 +563,6 @@ lvm.renameLV(self.sdUUID, self.volUUID, newUUID) self.md.volUUID = newUUID self.md.volumePath = os.path.join(self.imagePath, newUUID) - - def _share(self, dstImgPath): - """ - Share this volume to dstImgPath - """ - dstPath = os.path.join(dstImgPath, self.volUUID) - - self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) - os.symlink(self.md.getDevPath(), dstPath) @classmethod def shareVolumeRollback(cls, taskObj, volPath): diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 54cfe9d..4b3fbaf 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -237,6 +237,43 @@ sanlock.init_resource(sdUUID, volUUID, [(leasePath, LEASE_FILEOFFSET)]) + def _getLeaseVolumePath(self, vol_path=None): + """ + Get the volume lease file/link path + """ + if not vol_path: + vol_path = self.getVolumePath() + return self._leaseVolumePath(vol_path) + + def _shareLease(self, dstImgPath): + """ + Internal utility method used to share the template volume lease file + with the images based on such template. + """ + self.log.debug("Share volume lease of %s to %s", self.volUUID, + dstImgPath) + dstLeasePath = self._getLeaseVolumePath( + os.path.join(dstImgPath, self.volUUID)) + self.oop.utils.forceLink(self._getLeaseVolumePath(), dstLeasePath) + + def _share(self, dstImgPath): + """ + Share this volume to dstImgPath, including the metadata and the lease + """ + dstVolPath = os.path.join(dstImgPath, self.volUUID) + dstMetaPath = self._getMetaVolumePath(dstVolPath) + + self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) + self.oop.utils.forceLink(self.getVolumePath(), dstVolPath) + + self.log.debug("Share volume metadata of %s to %s", self.volUUID, + dstImgPath) + self.oop.utils.forceLink(self._getMetaVolumePath(), dstMetaPath) + + # Link the lease file if the domain uses sanlock + if sdCache.produce(self.sdUUID).hasVolumeLeases(): + self._shareLease(dstImgPath) + class FileVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -380,35 +417,6 @@ self.volUUID, exc_info=True) raise eFound - - def _shareLease(self, dstImgPath): - """ - Internal utility method used to share the template volume lease file - with the images based on such template. - """ - self.log.debug("Share volume lease of %s to %s", self.volUUID, - dstImgPath) - dstLeasePath = self._getLeaseVolumePath( - os.path.join(dstImgPath, self.volUUID)) - self.oop.utils.forceLink(self._getLeaseVolumePath(), dstLeasePath) - - def _share(self, dstImgPath): - """ - Share this volume to dstImgPath, including the metadata and the lease - """ - dstVolPath = os.path.join(dstImgPath, self.volUUID) - dstMetaPath = self._getMetaVolumePath(dstVolPath) - - self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) - self.oop.utils.forceLink(self.getVolumePath(), dstVolPath) - - self.log.debug("Share volume metadata of %s to %s", self.volUUID, - dstImgPath) - self.oop.utils.forceLink(self._getMetaVolumePath(), dstMetaPath) - - # Link the lease file if the domain uses sanlock - if sdCache.produce(self.sdUUID).hasVolumeLeases(): - self._shareLease(dstImgPath) @classmethod def shareVolumeRollback(cls, taskObj, volPath): @@ -555,12 +563,7 @@ return self.md._getMetaVolumePath(vol_path) def _getLeaseVolumePath(self, vol_path=None): - """ - Get the volume lease file/link path - """ - if not vol_path: - vol_path = self.getVolumePath() - return self.md._leaseVolumePath(vol_path) + return self.md._getLeaseVolumePath(vol_path) def validateVolumePath(self): """ diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 71f47a1..16c2eee 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -459,6 +459,13 @@ def refreshVolume(self): pass + def _shareLease(self, dstImgPath): + """ + Internal utility method used during the share process and by the + domain V3 upgrade. + """ + pass # Do not remove this method or the V3 upgrade will fail. + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -530,6 +537,9 @@ def removeMetadata(self): self.md.removeMetadata() + + def _share(self, dstImgPath): + return self.md._share(dstImgPath) @classmethod def _getModuleAndClass(cls): @@ -657,11 +667,7 @@ raise se.CannotCloneVolume(self.volumePath, dstPath, str(e)) def _shareLease(self, dstImgPath): - """ - Internal utility method used during the share process and by the - domain V3 upgrade. - """ - pass # Do not remove this method or the V3 upgrade will fail. + self.md._shareLease(dstImgPath) def share(self, dstImgPath): """ -- To view, visit https://gerrit.ovirt.org/42004 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I43c7aaf4a8a8b8bce18331c0a55ba533192777f7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: move validateDelete
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move validateDelete ...................................................................... VolumeMetadata: move validateDelete Change-Id: I06c8a87cb5b07d342b47fc54e75bc54ba00f5be2 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 20 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/00/42000/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 82f74bc..73b6a5e 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -393,7 +393,7 @@ self.recheckIfLeaf() if not force: - self.validateDelete() + self.md.validateDelete() # Mark volume as illegal before deleting self.setLegality(volume.ILLEGAL_VOL) diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 7956ad4..cfd05f8 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -314,7 +314,7 @@ lease_path = self.__leaseVolumePath(vol_path) if not force: - self.validateDelete() + self.md.validateDelete() # Mark volume as illegal before deleting self.setLegality(volume.ILLEGAL_VOL) diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index d2dbd3c..abbd906 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -412,6 +412,24 @@ def setFormat(self, volFormat): self.setMetaParam(FORMAT, type2name(volFormat)) + def validateDelete(self): + """ + Validate volume before deleting + """ + try: + if self.isShared(): + raise se.CannotDeleteSharedVolume("img %s vol %s" % + (self.imgUUID, self.volUUID)) + except se.MetaDataKeyNotFoundError as e: + # In case of metadata key error, we have corrupted + # volume (One of metadata corruptions may be + # previous volume deletion failure). + # So, there is no reasons to avoid its deletion + self.log.warn("Volume %s metadata error (%s)", + self.volUUID, str(e)) + if self.getChildren(): + raise se.VolumeImageHasChildren(self) + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -822,24 +840,6 @@ ) return volUUID - - def validateDelete(self): - """ - Validate volume before deleting - """ - try: - if self.isShared(): - raise se.CannotDeleteSharedVolume("img %s vol %s" % - (self.imgUUID, self.volUUID)) - except se.MetaDataKeyNotFoundError as e: - # In case of metadata key error, we have corrupted - # volume (One of metadata corruptions may be - # previous volume deletion failure). - # So, there is no reasons to avoid its deletion - self.log.warn("Volume %s metadata error (%s)", - self.volUUID, str(e)) - if self.getChildren(): - raise se.VolumeImageHasChildren(self) def extend(self, newsize): """ -- To view, visit https://gerrit.ovirt.org/42000 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I06c8a87cb5b07d342b47fc54e75bc54ba00f5be2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: move newMetadata
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move newMetadata ...................................................................... VolumeMetadata: move newMetadata Change-Id: I115fb4da48ef39bbbda5475cefbed1ab26ee1764 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 56 insertions(+), 45 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/01/42001/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 73b6a5e..d80c2a8 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -253,6 +253,16 @@ # tags self.setMetaParam(volume.IMAGE, imgUUID) + def removeMetadata(self, metaId): + """ + Just wipe meta. + """ + try: + self._putMetadata(metaId, {"NONE": "#" * (sd.METASIZE - 10)}) + except Exception as e: + self.log.error(e, exc_info=True) + raise se.VolumeMetadataWriteError("%s: %s" % (metaId, e)) + class BlockVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -646,23 +656,9 @@ lvs = lvm.lvsByTag(sdUUID, "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)) return [lv.name for lv in lvs] - def removeMetadata(self, metaId): - """ - Just wipe meta. - """ - try: - self.__putMetadata(metaId, {"NONE": "#" * (sd.METASIZE - 10)}) - except Exception as e: - self.log.error(e, exc_info=True) - raise se.VolumeMetadataWriteError("%s: %s" % (metaId, e)) - @classmethod def __putMetadata(cls, metaId, meta): cls.MetadataClass._putMetadata(metaId, meta) - - @classmethod - def createMetadata(cls, metaId, meta): - cls.__putMetadata(metaId, meta) def getMetaOffset(self): return self.md.getMetaOffset() diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index cfd05f8..06ac8f8 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -210,6 +210,15 @@ """ self.setMetaParam(volume.IMAGE, imgUUID) + def removeMetadata(self): + """ + Remove the meta file + """ + metaPath = self._getMetaVolumePath() + if self.oop.os.path.lexists(metaPath): + self.log.debug("Removing: %s", metaPath) + self.oop.os.unlink(metaPath) + class FileVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -411,22 +420,9 @@ if not self.oop.os.access(volPath, os.R_OK): raise se.VolumeAccessError(volPath) - def removeMetadata(self): - """ - Remove the meta file - """ - metaPath = self._getMetaVolumePath() - if self.oop.os.path.lexists(metaPath): - self.log.debug("Removing: %s", metaPath) - self.oop.os.unlink(metaPath) - @classmethod def __putMetadata(cls, metaId, meta): FileVolumeMetadata._putMetadata(metaId, meta) - - @classmethod - def createMetadata(cls, metaId, meta): - cls.__putMetadata(metaId, meta) @classmethod def getImageVolumes(cls, repoPath, sdUUID, imgUUID): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index abbd906..c8a7dc0 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -430,9 +430,36 @@ if self.getChildren(): raise se.VolumeImageHasChildren(self) + @classmethod + def createMetadata(cls, metaId, meta): + cls._putMetadata(metaId, meta) + + @classmethod + def newMetadata(cls, metaId, sdUUID, imgUUID, puuid, size, format, type, + voltype, disktype, desc="", legality=ILLEGAL_VOL): + meta = { + FORMAT: str(format), + TYPE: str(type), + VOLTYPE: str(voltype), + DISKTYPE: str(disktype), + SIZE: int(size), + CTIME: int(time.time()), + sd.DMDK_POOLS: "", # obsolete + DOMAIN: str(sdUUID), + IMAGE: str(imgUUID), + DESCRIPTION: str(desc), + PUUID: str(puuid), + MTIME: 0, + LEGALITY: str(legality), + } + + cls.createMetadata(metaId, meta) + return meta + class Volume(object): log = logging.getLogger('Storage.Volume') + MetadataClass = VolumeMetadata def __init__(self, md): self.md = md @@ -497,6 +524,9 @@ Set the read/write permission on the volume (deprecated) """ self.md._setrw(rw) + + def removeMetadata(self): + self.md.removeMetadata() @classmethod def _getModuleAndClass(cls): @@ -1069,24 +1099,9 @@ @classmethod def newMetadata(cls, metaId, sdUUID, imgUUID, puuid, size, format, type, voltype, disktype, desc="", legality=ILLEGAL_VOL): - meta = { - FORMAT: str(format), - TYPE: str(type), - VOLTYPE: str(voltype), - DISKTYPE: str(disktype), - SIZE: int(size), - CTIME: int(time.time()), - sd.DMDK_POOLS: "", # obsolete - DOMAIN: str(sdUUID), - IMAGE: str(imgUUID), - DESCRIPTION: str(desc), - PUUID: str(puuid), - MTIME: 0, - LEGALITY: str(legality), - } - - cls.createMetadata(metaId, meta) - return meta + return cls.MetadataClass.newMetadata( + metaId, sdUUID, imgUUID, puuid, size, format, type, voltype, + disktype, desc, legality) def getInfo(self): return self.md.getInfo() @@ -1136,3 +1151,7 @@ by reducing the lv to minimal size required """ pass + + @classmethod + def createMetadata(cls, metaId, meta): + return cls.MetadataClass.createMetadata(metaId, meta) -- To view, visit https://gerrit.ovirt.org/42001 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I115fb4da48ef39bbbda5475cefbed1ab26ee1764 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: move newVolumeLease
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move newVolumeLease ...................................................................... VolumeMetadata: move newVolumeLease Change-Id: If18aa7d526d9eb334c638fdf4cc93d31a4943cb6 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 38 insertions(+), 35 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/02/42002/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index d80c2a8..bf1a002 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -38,7 +38,6 @@ import lvm import resourceManager as rm from threadLocal import vars -from sdc import sdCache from resourceFactories import LVM_ACTIVATION_NAMESPACE import fileUtils @@ -262,6 +261,19 @@ except Exception as e: self.log.error(e, exc_info=True) raise se.VolumeMetadataWriteError("%s: %s" % (metaId, e)) + + @classmethod + def newVolumeLease(cls, metaId, sdUUID, volUUID): + cls.log.debug("Initializing volume lease volUUID=%s sdUUID=%s, " + "metaId=%s", volUUID, sdUUID, metaId) + manifest = blockSD.BlockStorageDomainManifest(sdUUID) + metaSdUUID, mdSlot = metaId + + leasePath = manifest.getLeasesFilePath() + leaseOffset = ((mdSlot + RESERVED_LEASES) + * manifest.logBlkSize * sd.LEASE_BLOCKS) + + sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)]) class BlockVolume(volume.Volume): @@ -665,19 +677,6 @@ def setMetadata(self, meta, metaId=None): return self.md.setMetadata(meta, metaId) - - @classmethod - def newVolumeLease(cls, metaId, sdUUID, volUUID): - cls.log.debug("Initializing volume lease volUUID=%s sdUUID=%s, " - "metaId=%s", volUUID, sdUUID, metaId) - dom = sdCache.produce(sdUUID) - metaSdUUID, mdSlot = metaId - - leasePath = dom.getLeasesFilePath() - leaseOffset = ((mdSlot + RESERVED_LEASES) - * dom.logBlkSize * sd.LEASE_BLOCKS) - - sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)]) def _extendSizeRaw(self, newSize): # Since this method relies on lvm.extendLV (lvextend) when the diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 06ac8f8..54cfe9d 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -219,6 +219,24 @@ self.log.debug("Removing: %s", metaPath) self.oop.os.unlink(metaPath) + @classmethod + def _leaseVolumePath(cls, vol_path): + if vol_path: + return vol_path + LEASE_FILEEXT + else: + return None + + @classmethod + def newVolumeLease(cls, metaId, sdUUID, volUUID): + cls.log.debug("Initializing volume lease volUUID=%s sdUUID=%s, " + "metaId=%s", volUUID, sdUUID, metaId) + volPath, = metaId + leasePath = cls._leaseVolumePath(volPath) + oop.getProcessPool(sdUUID).truncateFile(leasePath, LEASE_FILEOFFSET) + cls.file_setrw(leasePath, rw=True) + sanlock.init_resource(sdUUID, volUUID, [(leasePath, + LEASE_FILEOFFSET)]) + class FileVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -320,7 +338,7 @@ self.log.info("Request to delete volume %s", self.volUUID) vol_path = self.getVolumePath() - lease_path = self.__leaseVolumePath(vol_path) + lease_path = self.md._leaseVolumePath(vol_path) if not force: self.md.validateDelete() @@ -398,7 +416,7 @@ procPool = oop.getProcessPool(getDomUuidFromVolumePath(volPath)) procPool.utils.rmFile(volPath) procPool.utils.rmFile(cls.__metaVolumePath(volPath)) - procPool.utils.rmFile(cls.__leaseVolumePath(volPath)) + procPool.utils.rmFile(cls.MetadataClass._leaseVolumePath(volPath)) def llPrepare(self, rw=False, setrw=False): """ @@ -442,17 +460,6 @@ getImage() == imgUUID): volList.append(volid) return volList - - @classmethod - def newVolumeLease(cls, metaId, sdUUID, volUUID): - cls.log.debug("Initializing volume lease volUUID=%s sdUUID=%s, " - "metaId=%s", volUUID, sdUUID, metaId) - volPath, = metaId - leasePath = cls.__leaseVolumePath(volPath) - oop.getProcessPool(sdUUID).truncateFile(leasePath, LEASE_FILEOFFSET) - cls.file_setrw(leasePath, rw=True) - sanlock.init_resource(sdUUID, volUUID, [(leasePath, - LEASE_FILEOFFSET)]) def setParentMeta(self, puuid): """ @@ -541,13 +548,6 @@ def __metaVolumePath(cls, volPath): return cls.MetadataClass._metaVolumePath(volPath) - @classmethod - def __leaseVolumePath(cls, vol_path): - if vol_path: - return vol_path + LEASE_FILEEXT - else: - return None - def _getMetaVolumePath(self, vol_path=None): """ Get the volume metadata file/link path @@ -560,7 +560,7 @@ """ if not vol_path: vol_path = self.getVolumePath() - return self.__leaseVolumePath(vol_path) + return self.md._leaseVolumePath(vol_path) def validateVolumePath(self): """ diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index c8a7dc0..461aa3c 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -1155,3 +1155,7 @@ @classmethod def createMetadata(cls, metaId, meta): return cls.MetadataClass.createMetadata(metaId, meta) + + @classmethod + def newVolumeLease(cls, metaId, sdUUID, volUUID): + return cls.MetadataClass.newVolumeLease(metaId, sdUUID, volUUID) -- To view, visit https://gerrit.ovirt.org/42002 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If18aa7d526d9eb334c638fdf4cc93d31a4943cb6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: move refreshVolume
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move refreshVolume ...................................................................... VolumeMetadata: move refreshVolume Change-Id: Id5041428d8d2e73edc1aa1362fcea8801e5b9116 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/volume.py 2 files changed, 7 insertions(+), 7 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/03/42003/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index bf1a002..f14ce2a 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -275,6 +275,9 @@ sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)]) + def refreshVolume(self): + lvm.refreshLVs(self.sdUUID, (self.volUUID,)) + class BlockVolume(volume.Volume): """ Actually represents a single volume (i.e. part of virtual disk). @@ -297,9 +300,6 @@ except se.LogicalVolumeDoesNotExistError: raise se.VolumeDoesNotExist(self.volUUID) volume.Volume.validate(self) - - def refreshVolume(self): - lvm.refreshLVs(self.sdUUID, (self.volUUID,)) @classmethod def halfbakedVolumeRollback(cls, taskObj, sdUUID, volUUID, volPath): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 461aa3c..71f47a1 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -456,6 +456,9 @@ cls.createMetadata(metaId, meta) return meta + def refreshVolume(self): + pass + class Volume(object): log = logging.getLogger('Storage.Volume') @@ -687,10 +690,7 @@ raise se.CannotShareVolume(self.getVolumePath(), dstPath, str(e)) def refreshVolume(self): - """ - Refresh volume - """ - pass + return self.md.refreshVolume() @classmethod def parentVolumeRollback(cls, taskObj, sdUUID, pimgUUID, pvolUUID): -- To view, visit https://gerrit.ovirt.org/42003 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id5041428d8d2e73edc1aa1362fcea8801e5b9116 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: Move simple metadata getters and setters
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: Move simple metadata getters and setters ...................................................................... VolumeMetadata: Move simple metadata getters and setters Change-Id: I94a36dcb0d6e586612f869e8ef83925954709362 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 162 insertions(+), 105 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/97/41997/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 8668e3f..c22af74 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -146,6 +146,18 @@ "%s%s" % (TAG_PREFIX_PARENT, self.volUUID)) return tuple(lv.name for lv in lvs) + def getImage(self): + """ + Return image UUID + """ + return self.getVolumeTag(TAG_PREFIX_IMAGE) + + def getDevPath(self): + """ + Return the underlying device (for sharing) + """ + return lvm.lvPath(self.sdUUID, self.volUUID) + def setMetadata(self, meta, metaId=None): """ Set the meta data hash as the new meta data of the Volume @@ -218,6 +230,15 @@ LV metadata it may only be performed by an SPM. """ self.changeVolumeTag(TAG_PREFIX_PARENT, puuid) + + def setImage(self, imgUUID): + """ + Set image UUID + """ + self.changeVolumeTag(TAG_PREFIX_IMAGE, imgUUID) + # FIXME In next version we should remove imgUUID, as it is saved on lvm + # tags + self.setMetaParam(volume.IMAGE, imgUUID) class BlockVolume(volume.Volume): @@ -499,12 +520,6 @@ self.md.volUUID = newUUID self.md.volumePath = os.path.join(self.imagePath, newUUID) - def getDevPath(self): - """ - Return the underlying device (for sharing) - """ - return lvm.lvPath(self.sdUUID, self.volUUID) - def _share(self, dstImgPath): """ Share this volume to dstImgPath @@ -512,7 +527,7 @@ dstPath = os.path.join(dstImgPath, self.volUUID) self.log.debug("Share volume %s to %s", self.volUUID, dstImgPath) - os.symlink(self.getDevPath(), dstPath) + os.symlink(self.md.getDevPath(), dstPath) @classmethod def shareVolumeRollback(cls, taskObj, volPath): @@ -603,26 +618,11 @@ def getParentTag(self): return self.md.getParentTag() - def getImage(self): - """ - Return image UUID - """ - return self.getVolumeTag(TAG_PREFIX_IMAGE) - def setParentMeta(self, puuid): return self.md.setParentMeta(puuid) def setParentTag(self, puuid): return self.md.setParentTag(puuid) - - def setImage(self, imgUUID): - """ - Set image UUID - """ - self.changeVolumeTag(TAG_PREFIX_IMAGE, imgUUID) - # FIXME In next version we should remove imgUUID, as it is saved on lvm - # tags - self.setMetaParam(volume.IMAGE, imgUUID) @classmethod def getImageVolumes(cls, repoPath, sdUUID, imgUUID): diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index b332ebb..5be21ee 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -119,6 +119,18 @@ return tuple(children) + def getImage(self): + """ + Return image UUID + """ + return self.getMetaParam(volume.IMAGE) + + def getDevPath(self): + """ + Return the underlying device (for sharing) + """ + return self.getVolumePath() + def setMetadata(self, meta, metaId=None): """ Set the meta data hash as the new meta data of the Volume @@ -176,6 +188,12 @@ sdUUID = getDomUuidFromVolumePath(volPath) oop.getProcessPool(sdUUID).os.rename(metaPath + ".new", metaPath) + + def setImage(self, imgUUID): + """ + Set image UUID + """ + self.setMetaParam(volume.IMAGE, imgUUID) class FileVolume(volume.Volume): @@ -321,12 +339,6 @@ raise eFound - def getDevPath(self): - """ - Return the underlying device (for sharing) - """ - return self.getVolumePath() - def _shareLease(self, dstImgPath): """ Internal utility method used to share the template volume lease file @@ -431,12 +443,6 @@ sanlock.init_resource(sdUUID, volUUID, [(leasePath, LEASE_FILEOFFSET)]) - def getImage(self): - """ - Return image UUID - """ - return self.getMetaParam(volume.IMAGE) - def setParentMeta(self, puuid): """ Set parent volume UUID in Volume metadata. This operation can be done @@ -450,12 +456,6 @@ For file volumes we do not use any LV tags """ pass - - def setImage(self, imgUUID): - """ - Set image UUID - """ - self.setMetaParam(volume.IMAGE, imgUUID) @classmethod def renameVolumeRollback(cls, taskObj, oldPath, newPath): diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index da616b3..21e779d 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -172,6 +172,57 @@ def isShared(self): return self.getVolType() == type2name(SHARED_VOL) + def getDescription(self): + """ + Return volume description + """ + return self.getMetaParam(DESCRIPTION) + + def getLegality(self): + """ + Return volume legality + """ + try: + legality = self.getMetaParam(LEGALITY) + return legality + except se.MetaDataKeyNotFoundError: + return LEGAL_VOL + + def isLegal(self): + try: + legality = self.getMetaParam(LEGALITY) + return legality != ILLEGAL_VOL + except se.MetaDataKeyNotFoundError: + return True + + def isFake(self): + try: + legality = self.getMetaParam(LEGALITY) + return legality == FAKE_VOL + except se.MetaDataKeyNotFoundError: + return False + + def getSize(self): + size = int(self.getMetaParam(SIZE)) + if size < 1: # Size stored in the metadata is not valid + raise se.MetaDataValidationError() + return size + + def getFormat(self): + return name2type(self.getMetaParam(FORMAT)) + + def getType(self): + return name2type(self.getMetaParam(TYPE)) + + def getDiskType(self): + return self.getMetaParam(DISKTYPE) + + def isInternal(self): + return self.getVolType() == type2name(INTERNAL_VOL) + + def isSparse(self): + return self.getType() == SPARSE_VOL + def setMetaParam(self, key, value): """ Set a value of a specific key @@ -227,6 +278,56 @@ self.setInternal() return self.isLeaf() + + def setDescription(self, descr): + """ + Set Volume Description + 'descr' - volume description + """ + self.log.info("volUUID = %s descr = %s ", self.volUUID, descr) + self.setMetaParam(DESCRIPTION, descr) + + def setLegality(self, legality): + """ + Set Volume Legality + 'legality' - volume legality + """ + self.log.info("sdUUID=%s imgUUID=%s volUUID = %s legality = %s ", + self.sdUUID, self.imgUUID, self.volUUID, legality) + self.setMetaParam(LEGALITY, legality) + + def setDomain(self, sdUUID): + self.setMetaParam(DOMAIN, sdUUID) + self.sdUUID = sdUUID + return self.sdUUID + + def setShared(self): + self.setMetaParam(VOLTYPE, type2name(SHARED_VOL)) + self.voltype = type2name(SHARED_VOL) + self.setrw(rw=False) + return self.voltype + + def setSize(self, size): + self.setMetaParam(SIZE, size) + + def updateInvalidatedSize(self): + # During some complex flows the volume size might have been marked as + # invalidated (e.g. during a transaction). Here we are checking + # NOTE: the prerequisite to run this is that the volume is accessible + # (e.g. lv active) and not in use by another process (e.g. dd, qemu). + # Going directly to the metadata parameter as we should skip the size + # validation in getSize. + if int(self.getMetaParam(SIZE)) < 1: + volInfo = qemuimg.info( + self.getVolumePath(), fmt2str(self.getFormat())) + # qemu/qemu-img rounds down + self.setSize(volInfo['virtualsize'] / BLOCK_SIZE) + + def setType(self, prealloc): + self.setMetaParam(TYPE, type2name(prealloc)) + + def setFormat(self, volFormat): + self.setMetaParam(FORMAT, type2name(volFormat)) class Volume(object): @@ -285,6 +386,9 @@ Children can be found in any image of the volume SD. """ return self.md.getChildren() + + def getImage(self): + return self.md.getImage() @deprecated # valid only for domain version < 3, see volume.setrw def _setrw(self, rw): @@ -735,48 +839,22 @@ self.syncMetadata() # update the metadata def setDescription(self, descr): - """ - Set Volume Description - 'descr' - volume description - """ - self.log.info("volUUID = %s descr = %s ", self.volUUID, descr) - self.setMetaParam(DESCRIPTION, descr) + self.md.setDescription(descr) def getDescription(self): - """ - Return volume description - """ - return self.getMetaParam(DESCRIPTION) + return self.md.getDescription() def getLegality(self): - """ - Return volume legality - """ - try: - legality = self.getMetaParam(LEGALITY) - return legality - except se.MetaDataKeyNotFoundError: - return LEGAL_VOL + return self.md.getLegality() def setLegality(self, legality): - """ - Set Volume Legality - 'legality' - volume legality - """ - self.log.info("sdUUID=%s imgUUID=%s volUUID = %s legality = %s ", - self.sdUUID, self.imgUUID, self.volUUID, legality) - self.setMetaParam(LEGALITY, legality) + self.md.setLegality(legality) def setDomain(self, sdUUID): - self.setMetaParam(DOMAIN, sdUUID) - self.md.sdUUID = sdUUID - return self.sdUUID + return self.md.setDomain(sdUUID) def setShared(self): - self.setMetaParam(VOLTYPE, type2name(SHARED_VOL)) - self.md.voltype = type2name(SHARED_VOL) - self.setrw(rw=False) - return self.voltype + return self.md.setShared() @deprecated # valid for domain version < 3 def setrw(self, rw): @@ -792,55 +870,34 @@ return self.md.getVolType() def getSize(self): - size = int(self.getMetaParam(SIZE)) - if size < 1: # Size stored in the metadata is not valid - raise se.MetaDataValidationError() - return size + return self.md.getSize() def setSize(self, size): - self.setMetaParam(SIZE, size) + self.md.setSize(size) def updateInvalidatedSize(self): - # During some complex flows the volume size might have been marked as - # invalidated (e.g. during a transaction). Here we are checking - # NOTE: the prerequisite to run this is that the volume is accessible - # (e.g. lv active) and not in use by another process (e.g. dd, qemu). - # Going directly to the metadata parameter as we should skip the size - # validation in getSize. - if int(self.getMetaParam(SIZE)) < 1: - volInfo = qemuimg.info( - self.getVolumePath(), fmt2str(self.getFormat())) - # qemu/qemu-img rounds down - self.setSize(volInfo['virtualsize'] / BLOCK_SIZE) + self.md.updateInvalidatedSize() def getType(self): - return name2type(self.getMetaParam(TYPE)) + return self.md.getType() def setType(self, prealloc): - self.setMetaParam(TYPE, type2name(prealloc)) + self.md.setType(prealloc) def getDiskType(self): - return self.getMetaParam(DISKTYPE) + return self.md.getDiskType() def getFormat(self): - return name2type(self.getMetaParam(FORMAT)) + return self.md.getFormat() def setFormat(self, volFormat): - self.setMetaParam(FORMAT, type2name(volFormat)) + self.md.setFormat(volFormat) def isLegal(self): - try: - legality = self.getMetaParam(LEGALITY) - return legality != ILLEGAL_VOL - except se.MetaDataKeyNotFoundError: - return True + return self.md.isLegal() def isFake(self): - try: - legality = self.getMetaParam(LEGALITY) - return legality == FAKE_VOL - except se.MetaDataKeyNotFoundError: - return False + return self.md.isFake() def isShared(self): return self.md.isShared() @@ -849,10 +906,10 @@ return self.md.isLeaf() def isInternal(self): - return self.getVolType() == type2name(INTERNAL_VOL) + return self.md.isInternal() def isSparse(self): - return self.getType() == SPARSE_VOL + return self.md.isSparse() def recheckIfLeaf(self): """ -- To view, visit https://gerrit.ovirt.org/41997 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I94a36dcb0d6e586612f869e8ef83925954709362 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: Move getInfo and getVolumeParams
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: Move getInfo and getVolumeParams ...................................................................... VolumeMetadata: Move getInfo and getVolumeParams Change-Id: I80c1fa58d64efbb3cbc7eea84189fc8375b6fa3d Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/volume.py 1 file changed, 85 insertions(+), 79 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/99/41999/1 diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 97afb56..d2dbd3c 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -223,6 +223,89 @@ def isSparse(self): return self.getType() == SPARSE_VOL + def metadata2info(self, meta): + return { + "uuid": self.volUUID, + "type": meta.get(TYPE, ""), + "format": meta.get(FORMAT, ""), + "disktype": meta.get(DISKTYPE, ""), + "voltype": meta.get(VOLTYPE, ""), + "size": int(meta.get(SIZE, "0")), + "parent": self.getParent(), + "description": meta.get(DESCRIPTION, ""), + "pool": meta.get(sd.DMDK_POOLS, ""), + "domain": meta.get(DOMAIN, ""), + "image": self.getImage(), + "ctime": meta.get(CTIME, ""), + "mtime": "0", + "legality": meta.get(LEGALITY, ""), + } + + def getInfo(self): + """ + Get volume info + """ + self.log.info("Info request: sdUUID=%s imgUUID=%s volUUID = %s ", + self.sdUUID, self.imgUUID, self.volUUID) + info = {} + try: + meta = self.getMetadata() + info = self.metadata2info(meta) + info["capacity"] = str(int(info["size"]) * BLOCK_SIZE) + del info["size"] + # Get the image actual size on disk + vsize = self.getVolumeSize(bs=1) + avsize = self.getVolumeTrueSize(bs=1) + info['apparentsize'] = str(vsize) + info['truesize'] = str(avsize) + info['status'] = "OK" + except se.StorageException as e: + self.log.debug("exception: %s:%s" % (str(e.message), str(e.value))) + info['apparentsize'] = "0" + info['truesize'] = "0" + info['status'] = "INVALID" + + # Both engine and dumpStorageTable don't use this option so + # only keeping it to not break existing scripts that look for the key + info['children'] = [] + + # If image was set to illegal, mark the status same + # (because of VDC constraints) + if info.get('legality', None) == ILLEGAL_VOL: + info['status'] = ILLEGAL_VOL + self.log.info("%s/%s/%s info is %s", + self.sdUUID, self.imgUUID, self.volUUID, str(info)) + return info + + def getVolumeParams(self, bs=BLOCK_SIZE): + volParams = {} + volParams['volUUID'] = self.volUUID + volParams['imgUUID'] = self.getImage() + volParams['path'] = self.getVolumePath() + volParams['disktype'] = self.getDiskType() + volParams['prealloc'] = self.getType() + volParams['volFormat'] = self.getFormat() + # TODO: getSize returns size in 512b multiples, should move all sizes + # to byte multiples everywhere to avoid conversion errors and change + # only at the end + volParams['size'] = self.getSize() + volParams['apparentsize'] = self.getVolumeSize(bs=bs) + volParams['truesize'] = self.getVolumeTrueSize(bs=bs) + volParams['parent'] = self.getParent() + volParams['descr'] = self.getDescription() + volParams['legality'] = self.getLegality() + return volParams + + def getVmVolumeInfo(self): + """ + Get volume path/info as dict. + Derived classes can use this if they want to represent the + volume to the VM in a different way than the standard 'path' way. + """ + # By default, send path + return {'volType': VmVolumeInfo.TYPE_PATH, + 'path': self.getVolumePath()} + def setMetaParam(self, key, value): """ Set a value of a specific key @@ -983,24 +1066,6 @@ """ pass - def metadata2info(self, meta): - return { - "uuid": self.volUUID, - "type": meta.get(TYPE, ""), - "format": meta.get(FORMAT, ""), - "disktype": meta.get(DISKTYPE, ""), - "voltype": meta.get(VOLTYPE, ""), - "size": int(meta.get(SIZE, "0")), - "parent": self.getParent(), - "description": meta.get(DESCRIPTION, ""), - "pool": meta.get(sd.DMDK_POOLS, ""), - "domain": meta.get(DOMAIN, ""), - "image": self.getImage(), - "ctime": meta.get(CTIME, ""), - "mtime": "0", - "legality": meta.get(LEGALITY, ""), - } - @classmethod def newMetadata(cls, metaId, sdUUID, imgUUID, puuid, size, format, type, voltype, disktype, desc="", legality=ILLEGAL_VOL): @@ -1024,40 +1089,7 @@ return meta def getInfo(self): - """ - Get volume info - """ - self.log.info("Info request: sdUUID=%s imgUUID=%s volUUID = %s ", - self.sdUUID, self.imgUUID, self.volUUID) - info = {} - try: - meta = self.getMetadata() - info = self.metadata2info(meta) - info["capacity"] = str(int(info["size"]) * BLOCK_SIZE) - del info["size"] - # Get the image actual size on disk - vsize = self.getVolumeSize(bs=1) - avsize = self.getVolumeTrueSize(bs=1) - info['apparentsize'] = str(vsize) - info['truesize'] = str(avsize) - info['status'] = "OK" - except se.StorageException as e: - self.log.debug("exception: %s:%s" % (str(e.message), str(e.value))) - info['apparentsize'] = "0" - info['truesize'] = "0" - info['status'] = "INVALID" - - # Both engine and dumpStorageTable don't use this option so - # only keeping it to not break existing scripts that look for the key - info['children'] = [] - - # If image was set to illegal, mark the status same - # (because of VDC constraints) - if info.get('legality', None) == ILLEGAL_VOL: - info['status'] = ILLEGAL_VOL - self.log.info("%s/%s/%s info is %s", - self.sdUUID, self.imgUUID, self.volUUID, str(info)) - return info + return self.md.getInfo() def getParentVolume(self): """ @@ -1083,16 +1115,6 @@ """ return self.md.getVolumePath() - def getVmVolumeInfo(self): - """ - Get volume path/info as dict. - Derived classes can use this if they want to represent the - volume to the VM in a different way than the standard 'path' way. - """ - # By default, send path - return {'volType': VmVolumeInfo.TYPE_PATH, - 'path': self.getVolumePath()} - def getMetaParam(self, key): """ Get a value of a specific key @@ -1106,23 +1128,7 @@ self.md.setMetaParam(key, value) def getVolumeParams(self, bs=BLOCK_SIZE): - volParams = {} - volParams['volUUID'] = self.volUUID - volParams['imgUUID'] = self.getImage() - volParams['path'] = self.getVolumePath() - volParams['disktype'] = self.getDiskType() - volParams['prealloc'] = self.getType() - volParams['volFormat'] = self.getFormat() - # TODO: getSize returns size in 512b multiples, should move all sizes - # to byte multiples everywhere to avoid conversion errors and change - # only at the end - volParams['size'] = self.getSize() - volParams['apparentsize'] = self.getVolumeSize(bs=bs) - volParams['truesize'] = self.getVolumeTrueSize(bs=bs) - volParams['parent'] = self.getParent() - volParams['descr'] = self.getDescription() - volParams['legality'] = self.getLegality() - return volParams + return self.md.getVolumeParams(bs) def shrinkToOptimalSize(self): """ -- To view, visit https://gerrit.ovirt.org/41999 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I80c1fa58d64efbb3cbc7eea84189fc8375b6fa3d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: VolumeMetadata: move getVolumeSize
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: VolumeMetadata: move getVolumeSize ...................................................................... VolumeMetadata: move getVolumeSize Change-Id: Ie122d4c96ad81d31c0e783e8651a99328722f8ed Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/storage/blockVolume.py M vdsm/storage/fileVolume.py M vdsm/storage/volume.py 3 files changed, 34 insertions(+), 27 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/98/41998/1 diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index c22af74..82f74bc 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -30,6 +30,7 @@ import volume import image import sd +import blockSD import misc from misc import logskip from misc import deprecated @@ -157,6 +158,18 @@ Return the underlying device (for sharing) """ return lvm.lvPath(self.sdUUID, self.volUUID) + + def getVolumeSize(self, bs=BLOCK_SIZE): + """ + Return the volume size in blocks + """ + # Just call the SD Manifest method getVSize() - apparently it does what + # we need. We consider incurred overhead of producing the object + # to be a small price for code de-duplication. + manifest = blockSD.BlockStorageDomainManifest(self.sdUUID) + return int(manifest.getVSize(self.imgUUID, self.volUUID) / bs) + + getVolumeTrueSize = getVolumeSize def setMetadata(self, meta, metaId=None): """ @@ -669,18 +682,6 @@ * dom.logBlkSize * sd.LEASE_BLOCKS) sanlock.init_resource(sdUUID, volUUID, [(leasePath, leaseOffset)]) - - def getVolumeSize(self, bs=BLOCK_SIZE): - """ - Return the volume size in blocks - """ - # Just call the SD method getVSize() - apparently it does what - # we need. We consider incurred overhead of producing the SD object - # to be a small price for code de-duplication. - sdobj = sdCache.produce(sdUUID=self.sdUUID) - return int(sdobj.getVSize(self.imgUUID, self.volUUID) / bs) - - getVolumeTrueSize = getVolumeSize def _extendSizeRaw(self, newSize): # Since this method relies on lvm.extendLV (lvextend) when the diff --git a/vdsm/storage/fileVolume.py b/vdsm/storage/fileVolume.py index 5be21ee..7956ad4 100644 --- a/vdsm/storage/fileVolume.py +++ b/vdsm/storage/fileVolume.py @@ -131,6 +131,21 @@ """ return self.getVolumePath() + def getVolumeSize(self, bs=BLOCK_SIZE): + """ + Return the volume size in blocks + """ + volPath = self.getVolumePath() + return int(int(self.oop.os.stat(volPath).st_size) / bs) + + def getVolumeTrueSize(self, bs=BLOCK_SIZE): + """ + Return the size of the storage allocated for this volume + on underlying storage + """ + volPath = self.getVolumePath() + return int(int(self.oop.os.stat(volPath).st_blocks) * BLOCK_SIZE / bs) + def setMetadata(self, meta, metaId=None): """ Set the meta data hash as the new meta data of the Volume @@ -576,21 +591,6 @@ metaVolumePath = self._getMetaVolumePath() if not self.oop.fileUtils.pathExists(metaVolumePath): raise se.VolumeDoesNotExist(self.volUUID) - - def getVolumeSize(self, bs=BLOCK_SIZE): - """ - Return the volume size in blocks - """ - volPath = self.getVolumePath() - return int(int(self.oop.os.stat(volPath).st_size) / bs) - - def getVolumeTrueSize(self, bs=BLOCK_SIZE): - """ - Return the size of the storage allocated for this volume - on underlying storage - """ - volPath = self.getVolumePath() - return int(int(self.oop.os.stat(volPath).st_blocks) * BLOCK_SIZE / bs) def _extendSizeRaw(self, newSize): volPath = self.getVolumePath() diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 21e779d..97afb56 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -872,6 +872,12 @@ def getSize(self): return self.md.getSize() + def getVolumeSize(self, bs=BLOCK_SIZE): + return self.md.getVolumeSize(bs) + + def getVolumeTrueSize(self, bs=BLOCK_SIZE): + return self.md.getVolumeTrueSize(bs) + def setSize(self, size): self.md.setSize(size) -- To view, visit https://gerrit.ovirt.org/41998 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie122d4c96ad81d31c0e783e8651a99328722f8ed Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 16

Change in vdsm[master]: lvm: Allow multiple initial tags when creating an LV
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: lvm: Allow multiple initial tags when creating an LV ...................................................................... lvm: Allow multiple initial tags when creating an LV Change-Id: I2524f0127a7cb0eff5d7639b8ba1d916fd66af6a Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/storagefakelib.py M vdsm/storage/blockVolume.py M vdsm/storage/lvm.py 3 files changed, 4 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/51/44051/1 diff --git a/tests/storagefakelib.py b/tests/storagefakelib.py index af7a3b4..7cbf4c3 100644 --- a/tests/storagefakelib.py +++ b/tests/storagefakelib.py @@ -85,7 +85,7 @@ md['free_count'] += nr_pe_added def createLV(self, vgName, lvName, size, activate=True, contiguous=False, - initialTag=None): + initialTags=[]): lv_attr = dict(voltype='-', permission='w', allocations='i', diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 597e7ee..42ed158 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -435,7 +435,7 @@ volSize = "%s" % ((size + SECTORS_TO_MB - 1) / SECTORS_TO_MB) lvm.createLV(dom.sdUUID, volUUID, volSize, activate=True, - initialTag=TAG_VOL_UNINIT) + initialTags=(TAG_VOL_UNINIT,)) utils.rmFile(volPath) os.symlink(lvm.lvPath(dom.sdUUID, volUUID), volPath) diff --git a/vdsm/storage/lvm.py b/vdsm/storage/lvm.py index 0394ce9..a492a0a 100644 --- a/vdsm/storage/lvm.py +++ b/vdsm/storage/lvm.py @@ -1064,7 +1064,7 @@ def createLV(vgName, lvName, size, activate=True, contiguous=False, - initialTag=None): + initialTags=[]): """ Size units: MB (1024 ** 2 = 2 ** 20)B. """ @@ -1078,8 +1078,7 @@ cmd = ["lvcreate"] cmd.extend(LVM_NOBACKUP) cmd.extend(("--contiguous", cont, "--size", "%sm" % size)) - if initialTag is not None: - cmd.extend(("--addtag", initialTag)) + map(cmd.extend, [("--addtag", tag) for tag in initialTags]) cmd.extend(("--name", lvName, vgName)) rc, out, err = _lvminfo.cmd(cmd, _lvminfo._getVGDevs((vgName, ))) -- To view, visit https://gerrit.ovirt.org/44051 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2524f0127a7cb0eff5d7639b8ba1d916fd66af6a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

2 5

Change in vdsm[master]: tests: Introduce indirection tests for storage refactoring
by alitke＠redhat.com 14 Aug '15

14 Aug '15

Adam Litke has uploaded a new change for review. Change subject: tests: Introduce indirection tests for storage refactoring ...................................................................... tests: Introduce indirection tests for storage refactoring Change-Id: I336b543aa7ba019c91104e860a28bb40acc268b9 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/Makefile.am A tests/sdm_indirection_tests.py 2 files changed, 178 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/98/44098/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 3152735..0b382db 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -91,6 +91,7 @@ scheduleTests.py \ schemaTests.py \ schemaValidationTest.py \ + sdm_indirection_tests.py \ securableTests.py \ sourceroutingTests.py \ sslhelper.py \ diff --git a/tests/sdm_indirection_tests.py b/tests/sdm_indirection_tests.py new file mode 100644 index 0000000..e73964e --- /dev/null +++ b/tests/sdm_indirection_tests.py @@ -0,0 +1,177 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from testlib import VdsmTestCase +from testlib import permutations, expandPermutations +from testlib import recorded + +from storage import sd, blockSD, fileSD + + +class FakeDomainManifest(sd.StorageDomainManifest): + def __init__(self): + self.sdUUID = 'a6ecac0a-5c6b-46d7-9ba5-df8b34df2d01' + self.domaindir = '/a/b/c' + self.mountpoint = '/a/b' + self._metadata = {} + + @recorded + def replaceMetadata(self, md): + pass + + @recorded + def getIsoDomainImagesDir(self): + pass + + @recorded + def getMDPath(self): + pass + + @recorded + def getMetaParam(self, key): + pass + +class FakeBlockDomainManifest(FakeDomainManifest): + def __init__(self): + FakeDomainManifest.__init__(self) + self.logBlkSize = 512 + self.phyBlkSize = 512 + + @recorded + def getReadDelay(self): + pass + + @recorded + def getVSize(self, imgUUID, volUUID): + pass + + @recorded + def getLeasesFilePath(self): + pass + + @recorded + def getIdsFilePath(self): + pass + + +class FakeFileDomainManifest(FakeDomainManifest): + def __init__(self): + FakeDomainManifest.__init__(self) + self.remotePath = 'b' + + @recorded + def getReadDelay(self): + pass + + @recorded + def getVSize(self, imgUUID, volUUID): + pass + + @recorded + def getVAllocSize(self, imgUUID, volUUID): + pass + + @recorded + def getLeasesFilePath(self): + pass + + @recorded + def getIdsFilePath(self): + pass + + +class FakeBlockStorageDomain(blockSD.BlockStorageDomain): + def __init__(self): + self.stat = None + self._manifest = FakeBlockDomainManifest() + + +class FakeFileStorageDomain(fileSD.FileStorageDomain): + def __init__(self): + self.stat = None + self._manifest = FakeFileDomainManifest() + + +@expandPermutations +class StorageDomainManifestTests(VdsmTestCase): + def _get_domain(self, domType): + if domType == 'block': + return FakeBlockStorageDomain() + elif domType == 'file': + return FakeFileStorageDomain() + else: + raise ValueError("%s is not a valid domain type" % domType) + + def _check(self, domType, fn, args, result): + dom = self._get_domain(domType) + getattr(dom, fn)(*args) + self.assertEquals(dom._manifest.__recording__, result) + + def check_call(self, domType, fn, nr_args=0): + args = tuple(range(nr_args)) + self._check(domType, fn, args, [(fn, args, {})]) + + @permutations([['block'], ['file']]) + def test_properties(self, domType): + dom = self._get_domain(domType) + self.assertEquals('a6ecac0a-5c6b-46d7-9ba5-df8b34df2d01', dom.sdUUID) + self.assertEquals('/a/b/c', dom.domaindir) + self.assertEquals({}, dom._metadata) + self.assertEquals('/a/b', dom.mountpoint) + + @permutations([['block']]) + def test_block_properties(self, domType): + dom = self._get_domain(domType) + self.assertEquals(512, dom.logBlkSize) + self.assertEquals(512, dom.phyBlkSize) + + @permutations([['file']]) + def test_getremotepath(self, domType): + dom = self._get_domain(domType) + self.assertEquals('b', dom.getRemotePath()) + + @permutations([['block'], ['file']]) + def test_nonexisting_function(self, domType): + self.assertRaises(AttributeError, self.check_call, domType, 'foo') + + @permutations([ + ['getReadDelay'], + ['replaceMetadata', 1], + ['getVSize', 2], + ['getLeasesFilePath'], + ['getIdsFilePath'], + ['getIsoDomainImagesDir'], + ['getMDPath'], + ['getMetaParam', 1], + ]) + def test_domain_funcs(self, fn, nargs=0): + for domType in 'file', 'block': + self.check_call(domType, fn, nargs) + + @permutations([ + ['getVAllocSize', 2] + ]) + def test_file_domain_funcs(self, fn, nargs=0): + self.check_call('file', fn, nargs) + + @permutations([['block']]) + def test_block_getvallocsize(self, domType): + # On block domains, this call redirects to the getVSize function + recording = [('getVSize', (0, 1), {})] + self._check(domType, 'getVAllocSize', (0, 1), recording) -- To view, visit https://gerrit.ovirt.org/44098 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I336b543aa7ba019c91104e860a28bb40acc268b9 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

3 18

Change in vdsm[master]: multipath: Fix "HITACHI"/"DF.* configuration
by Nir Soffer 14 Aug '15

14 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: multipath: Fix "HITACHI"/"DF.* configuration ...................................................................... multipath: Fix "HITACHI"/"DF.* configuration The multipath configuration we ship is missing important settings embedded in multiapth daemon. Overriding these defaults is needed to enable vdsm required configuration, but also remove any option we don't mention in our mulitpath device configuration, causing possible reliability or performance issues. The new configuration was copied from upstream multiapth-tools settings, updated on 2013-07-16, and override the "features" and "no_path_retry" settings required by vdsm. Requires testing with the relevant storage server. Change-Id: I78195b148aa3953f56b5a0b97366ccbdb3030d00 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/tool/configurators/multipath.py 1 file changed, 10 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/43/41243/1 diff --git a/lib/vdsm/tool/configurators/multipath.py b/lib/vdsm/tool/configurators/multipath.py index aba3d8d..14ae46e 100644 --- a/lib/vdsm/tool/configurators/multipath.py +++ b/lib/vdsm/tool/configurators/multipath.py @@ -50,8 +50,18 @@ devices { device { + # multipath.conf.default vendor "HITACHI" product "DF.*" + path_grouping_policy "group_by_prio" + path_checker "tur" + hardware_handler "0" + prio "hds" + failback "immediate" + rr_weight "uniform" + # vdsm required configuration + features "0" + no_path_retry fail } device { vendor "COMPELNT" -- To view, visit https://gerrit.ovirt.org/41243 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I78195b148aa3953f56b5a0b97366ccbdb3030d00 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 21

Change in vdsm[master]: multipath: Fix "COMPELNT"/"Compellent Vol" configuration
by Nir Soffer 14 Aug '15

14 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: multipath: Fix "COMPELNT"/"Compellent Vol" configuration ...................................................................... multipath: Fix "COMPELNT"/"Compellent Vol" configuration The multipath configuration we ship is missing important settings embedded in multiapth daemon. Overriding these defaults is needed to enable vdsm required configuration, but also remove any option we don't mention in our mulitpath device configuration, causing possible reliability or performance issues. The new configuration was copied from upstream multiapth-tools settings, updated on 2013-07-16, and override the "features" and "no_path_retry" settings required by vdsm. Requires testing with the relevant storage server. Change-Id: Ic3bf5bab082c5063272bf8947a6e2039ce4cf90c Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/tool/configurators/multipath.py 1 file changed, 9 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/44/41244/1 diff --git a/lib/vdsm/tool/configurators/multipath.py b/lib/vdsm/tool/configurators/multipath.py index 14ae46e..3d743af 100644 --- a/lib/vdsm/tool/configurators/multipath.py +++ b/lib/vdsm/tool/configurators/multipath.py @@ -64,8 +64,17 @@ no_path_retry fail } device { + # multipath.conf.default vendor "COMPELNT" product "Compellent Vol" + path_grouping_policy "multibus" + path_checker "tur" + hardware_handler "0" + prio "const" + failback "immediate" + rr_weight "uniform" + # vdsm required configuration + features "0" no_path_retry fail } device { -- To view, visit https://gerrit.ovirt.org/41244 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic3bf5bab082c5063272bf8947a6e2039ce4cf90c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 22

Change in vdsm[master]: schema: Fix asserts that had side effect
by Nir Soffer 14 Aug '15

14 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: schema: Fix asserts that had side effect ...................................................................... schema: Fix asserts that had side effect Asserts should never have side effects, so optimizing them out will not change the semantics of the code. Running with optimization would break parsing, eliminating line.pop(0) operations. Now the only difference is the warning if the line is not empty. Change-Id: I59dbb1c8d3760af7d3f3049227f6ad8ceeaa402a Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/rpc/process-schema.py 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/62/34362/1 diff --git a/vdsm/rpc/process-schema.py b/vdsm/rpc/process-schema.py index 4b1c3cf..8e56a1a 100755 --- a/vdsm/rpc/process-schema.py +++ b/vdsm/rpc/process-schema.py @@ -103,7 +103,8 @@ 'xxx': []}) # Pop a blank line - assert('' == lines.pop(0)) + line = lines.pop(0) + assert line == '', "Expected empty line: %r" % line # Grab the entity description. It might span multiple lines. symbol['desc'] = lines.pop(0) @@ -111,7 +112,8 @@ symbol['desc'] += lines.pop(0) # Pop a blank line - assert ('' == lines.pop(0)) + line = lines.pop(0) + assert line == '', "Expected empty line: %r" % line # Populate the rest of the human-readable data. # First try to read the parameters/members information. We are finished -- To view, visit http://gerrit.ovirt.org/34362 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I59dbb1c8d3760af7d3f3049227f6ad8ceeaa402a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 3

Change in vdsm[master]: spec: Require newer version of sanlock
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: spec: Require newer version of sanlock ...................................................................... spec: Require newer version of sanlock For EL7 PPC, we must use sanlock 3.2.2-2 for compatibility between x86 and ppc hosts. On x86, we can use sanlock 3.2.1, which is good as 3.2.2. Change-Id: I531bf060ef27a6d33a5cdae133fde557afea8ee5 Bug-Url: https://bugzilla.redhat.com/1150008 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm.spec.in 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/36321/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 85c9f83..f2d0f99 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -191,7 +191,7 @@ Requires: iscsi-initiator-utils >= 6.2.0.873-21 %endif -Requires: sanlock >= 2.8, sanlock-python +Requires: sanlock-python %if 0%{?rhel} %if 0%{?rhel} >= 7 @@ -201,6 +201,8 @@ Requires: lvm2 Requires: python Requires: policycoreutils-python +# 3.2.2-2 required for ppc64 +Requires: sanlock >= 3.2.2-2 Requires: selinux-policy-targeted >= 3.12.1-153.el7_0.13 %else Requires: python @@ -214,6 +216,7 @@ Requires: selinux-policy-targeted >= 3.7.19-260.el6_6.1 Requires: lvm2 >= 2.02.100-5 Requires: logrotate < 3.8.0 +Requires: sanlock >= 2.8 %endif %else Requires: fence-agents-all @@ -227,6 +230,8 @@ Requires: sed >= 4.2.1-10 Requires: ed Requires: lvm2 >= 2.02.98-15 +# Same as 3.2.2 on x86 +Requires: sanlock >= 3.2.1-2 Requires: selinux-policy-targeted >= 3.12.1-71 # In order to avoid a policycoreutils bug (rhbz 889698) when selinux is # disabled we now require the version 2.1.13-55 (or newer) of Fedora. -- To view, visit http://gerrit.ovirt.org/36321 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I531bf060ef27a6d33a5cdae133fde557afea8ee5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

2 3

Change in vdsm[master]: misc: Replace assert with AssetionError
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: misc: Replace assert with AssetionError ...................................................................... misc: Replace assert with AssetionError The code was wrongly assumed that asserts are always available. When running in optimized mode, the check would be skipped, leading to disastrous results. The assert was correct in that this is something that should never happen, unless someone is using the barrier incorrectly, or there is a bug in the code. Change-Id: I68f57e393423dc4faf58763a036220849a968e70 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/misc.py 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/63/34363/1 diff --git a/vdsm/storage/misc.py b/vdsm/storage/misc.py index ab2b4fd..5abcdcb 100644 --- a/vdsm/storage/misc.py +++ b/vdsm/storage/misc.py @@ -696,7 +696,9 @@ def exit(self): with self._cond: - assert self._busy, "Attempt to exit a barrier without entering" + if not self._busy: + raise AssertionError("Attempt to exit a barrier without " + "entering") self._busy = False self._cond.notifyAll() -- To view, visit http://gerrit.ovirt.org/34363 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I68f57e393423dc4faf58763a036220849a968e70 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

2 2

Change in vdsm[master]: Configure iscsi_session recovery_tmo for multipath devices
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: Configure iscsi_session recovery_tmo for multipath devices ...................................................................... Configure iscsi_session recovery_tmo for multipath devices This configuration is done by device-mapper-multipath in version 0.4.9-76 and later. However, device looses configuration after recovery from network errors. This patch adds the recovery_tmo configuration to vdsm-multipath udev rules. Since setting the timeout requires a script, the dmsetup command was moved into a new multipath-configure-device script. When we require a multipath version fixing this issue, we can remove the timeout handling code. Change-Id: Iaaa40fa5e6dc86beef501ef4aaefa17c4c1574c1 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M .gitignore M configure.ac M debian/vdsm.install M vdsm.spec.in M vdsm/storage/Makefile.am A vdsm/storage/multipath-configure-device.in A vdsm/storage/vdsm-multipath.rules D vdsm/storage/vdsm-multipath.rules.in 8 files changed, 53 insertions(+), 22 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/82/32582/1 diff --git a/.gitignore b/.gitignore index fca4f2e..5890806 100644 --- a/.gitignore +++ b/.gitignore @@ -61,7 +61,6 @@ vdsm/storage/protect/safelease vdsm/storage/lvm.env vdsm/storage/vdsm-lvm.rules -vdsm/storage/vdsm-multipath.rules vdsm/sudoers.vdsm vdsm/svdsm.logger.conf vdsm/vdscli.py diff --git a/configure.ac b/configure.ac index f40d57a..c467c37 100644 --- a/configure.ac +++ b/configure.ac @@ -335,10 +335,10 @@ vdsm/rpc/Makefile vdsm/sos/Makefile vdsm/storage/Makefile + vdsm/storage/multipath-configure-device vdsm/storage/imageRepository/Makefile vdsm/storage/protect/Makefile vdsm/storage/vdsm-lvm.rules - vdsm/storage/vdsm-multipath.rules vdsm/virt/Makefile vdsm_hooks/Makefile vdsm_hooks/checkimages/Makefile diff --git a/debian/vdsm.install b/debian/vdsm.install index bd63c72..759383d 100644 --- a/debian/vdsm.install +++ b/debian/vdsm.install @@ -25,6 +25,7 @@ ./usr/lib/python2.7/dist-packages/sos/plugins/vdsm.py ./usr/lib/python2.7/dist-packages/vdsmapi.py ./usr/libexec/vdsm/curl-img-wrap +./usr/libexec/vdsm/multiapth-configure-device ./usr/libexec/vdsm/ovirt_functions.sh ./usr/libexec/vdsm/persist-vdsm-hooks ./usr/libexec/vdsm/safelease diff --git a/vdsm.spec.in b/vdsm.spec.in index a5b354c..c912150 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -993,6 +993,7 @@ %{_libexecdir}/%{vdsm_name}/persist-vdsm-hooks %{_libexecdir}/%{vdsm_name}/unpersist-vdsm-hook %{_libexecdir}/%{vdsm_name}/ovirt_functions.sh +%{_libexecdir}/%{vdsm_name}/multipath-configure-device %{_libexecdir}/%{vdsm_name}/vdsm-gencerts.sh %{_libexecdir}/%{vdsm_name}/vdsmd_init_common.sh %{_datadir}/%{vdsm_name}/network/__init__.py* diff --git a/vdsm/storage/Makefile.am b/vdsm/storage/Makefile.am index 99b1460..91c2f9c 100644 --- a/vdsm/storage/Makefile.am +++ b/vdsm/storage/Makefile.am @@ -72,7 +72,8 @@ volume.py dist_vdsmexec_SCRIPTS = \ - curl-img-wrap + curl-img-wrap \ + multipath-configure-device nodist_vdsmstorage_DATA = \ lvm.env \ diff --git a/vdsm/storage/multipath-configure-device.in b/vdsm/storage/multipath-configure-device.in new file mode 100644 index 0000000..dcdfea5 --- /dev/null +++ b/vdsm/storage/multipath-configure-device.in @@ -0,0 +1,33 @@ +#!/bin/sh +# +# Copyright 2014 Red Hat, Inc. and/or its affiliates. +# +# Licensed to you under the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. See the files README and +# LICENSE_GPL_v2 which accompany this distribution. +# + +set -e + +# Ensure that multipath devices use no_path_retry fail, instead of +# no_path_retry queue, which is hardcoded in multipath configuration for many +# devices. +# +# See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/ht… + +@DMSETUP_PATH@ message $DM_NAME 0 fail_if_no_path + +# Set iscsi_session recovery_tmo. This parameter is configured by multipath on +# startup, but after a device fail and become active again, the configuration +# is lost and revert back to iscsid.conf default (120). + +timeout=5 + +for slave in /sys${DEVPATH}/slaves/*; do + path=$(@UDEVADM_PATH@ info --query=path --path="$slave") + session=$(echo "$path" | @SED_PATH@ -rn s'|^/devices/platform/host[0-9]+/(session[0-9]+)/.+$|\1|p') + if [ -n "$session" ]; then + echo $timeout > "/sys/class/iscsi_session/${session}/recovery_tmo" + fi +done diff --git a/vdsm/storage/vdsm-multipath.rules b/vdsm/storage/vdsm-multipath.rules new file mode 100644 index 0000000..d4d0dd7 --- /dev/null +++ b/vdsm/storage/vdsm-multipath.rules @@ -0,0 +1,15 @@ +# +# Copyright 2014 Red Hat, Inc. and/or its affiliates. +# +# Licensed to you under the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. See the files README and +# LICENSE_GPL_v2 which accompany this distribution. +# + +# Vdsm rules for multipath devices. +# +# This rule runs vdsm-mutipath-configure tool to configure multiapth devices to +# work with vdsm. + +ACTION=="add|change", ENV{DM_UUID}=="mpath-?*", RUN+="/usr/libexec/vdsm/multipath-configure-device" diff --git a/vdsm/storage/vdsm-multipath.rules.in b/vdsm/storage/vdsm-multipath.rules.in deleted file mode 100644 index cc2a878..0000000 --- a/vdsm/storage/vdsm-multipath.rules.in +++ /dev/null @@ -1,19 +0,0 @@ -# -# Copyright 2014 Red Hat, Inc. and/or its affiliates. -# -# Licensed to you under the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. See the files README and -# LICENSE_GPL_v2 which accompany this distribution. -# - -# Vdsm rules for multipath devices. -# -# Ensure that multipath devices use no_path_retry fail, instead of -# no_path_retry queue, which is hardcoded in multipath configuration for many -# devices. -# -# See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/ht… -# - -ACTION=="add|change", ENV{DM_UUID}=="mpath-?*", RUN+="@DMSETUP_PATH@ message $env{DM_NAME} 0 fail_if_no_path" -- To view, visit http://gerrit.ovirt.org/32582 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaaa40fa5e6dc86beef501ef4aaefa17c4c1574c1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 10

Change in vdsm[master]: vm: Require format attribute for drives
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: vm: Require format attribute for drives ...................................................................... vm: Require format attribute for drives We have seen sampling errors where a drive has no "format" attribute. These errors spam the system logs, and does not help to debug the real issue - why the required format attribute is not set? This patch ensure that a drive cannot be created without a format attribute, avoding log spam by sampling errors, and hopefully revealing the real reason for this bug. One broken test creating a drive without a format was fixed and new test ensure that creating drive without a format raises. Change-Id: I01ab1e071ecb76f383cc6dc7d99782e10cc90136 Relates-To: http://gerrit.ovirt.org/22551 Relates-To: https://bugzilla.redhat.com/994534 Relates-To: http://lists.ovirt.org/pipermail/users/2014-February/021007.html Bug-Url: https://bugzilla.redhat.com/1055437 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/vmTests.py M vdsm/vm.py 2 files changed, 28 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/34/24234/1 diff --git a/tests/vmTests.py b/tests/vmTests.py index 1e0a3f6..724d458 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -449,6 +449,18 @@ redir = vm.RedirDevice(self.conf, self.log, **dev) self.assertXML(redir.getXML(), redirXML) + def testDriveRequiredParameters(self): + # TODO: It is not clear what are the other required parameters, and it + # the parameters depend on the type of drive. We probbaly need bigger + # test here that test many combinations. + # Currently this test only missing "format" attribute. + conf = {'index': '2', 'propagateErrors': 'off', 'iface': 'ide', + 'name': 'hdc', 'device': 'cdrom', 'path': '/tmp/fedora.iso', + 'type': 'disk', 'readonly': 'True', 'shared': 'none', + 'serial': '54-a672-23e5b495a9ea'} + self.assertRaises(vm.InvalidDeviceParameters, vm.Drive, {}, self.log, + **conf) + def testDriveSharedStatus(self): sharedConfigs = [ # Backward compatibility @@ -470,7 +482,8 @@ 'exclusive', 'shared', 'none', 'transient', ] - driveConfig = {'index': '0', 'iface': 'virtio', 'device': 'disk'} + driveConfig = {'index': '0', 'iface': 'virtio', 'device': 'disk', + 'format': 'raw'} for driveInput, driveOutput in zip(sharedConfigs, expectedStates): driveInput.update(driveConfig) diff --git a/vdsm/vm.py b/vdsm/vm.py index aae8bd6..b7151b1 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -80,6 +80,10 @@ SMARTCARD_DEVICES = 'smartcard' +class InvalidDeviceParameters(Exception): + """ Raised when creating device with invalid parameters """ + + def isVdsmImage(drive): """ Tell if drive looks like a vdsm image @@ -1438,6 +1442,9 @@ VOLWM_CHUNK_REPLICATE_MULT = 2 # Chunk multiplier during replication def __init__(self, conf, log, **kwargs): + if 'format' not in kwargs: + raise InvalidDeviceParameters('"format" attribute is required:' + ' %r' % kwargs) if not kwargs.get('serial'): self.serial = kwargs.get('imageID'[-20:]) or '' VmDevice.__init__(self, conf, log, **kwargs) @@ -3108,8 +3115,13 @@ for devType, devClass in self.DeviceMapping: for dev in devices[devType]: - self._devices[devType].append(devClass(self.conf, self.log, - **dev)) + try: + drive = devClass(self.conf, self.log, **dev) + except InvalidDeviceParameters as e: + self.log.error('Ignoring device with invalid parameters:' + ' %s', e, exc_info=True) + else: + self._devices[devType].append(drive) # We should set this event as a last part of drives initialization self._pathsPreparedEvent.set() -- To view, visit http://gerrit.ovirt.org/24234 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I01ab1e071ecb76f383cc6dc7d99782e10cc90136 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

6 12

Change in vdsm[master]: vm: Prevent multiple threads blocking on same libvirt domain
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: vm: Prevent multiple threads blocking on same libvirt domain ...................................................................... vm: Prevent multiple threads blocking on same libvirt domain If a libvirt call get stuck because a vm is not responding, we could have multiple threads blocked on same vm without any limit, using precious libvirt resources that could be used to run other vms. This patch adds a new TimedLock lock, that raise a LockTimeout if the lock cannot be acquired after configured timeout. Using this lock, a vm allow now only one concurrent libvirt call. If a libvirt call get stuck, and other threads are tyring to invoke libvirt calls on the same vm, they will wait until the current call finish, or fail with a timeout. This should slow down calls for single vm, since each call is invoked only when the previous call returns. However, when using many vms, this creates natural round-robin scheduling, giving each vm equal chance to make progress, and limiting the load on libvirt. Change-Id: Ib459697b8688ebcba987cd6b9e11815826e92990 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M debian/vdsm-python.install M lib/vdsm/Makefile.am A lib/vdsm/locking.py M tests/Makefile.am A tests/lockingTests.py M vdsm.spec.in M vdsm/virt/vm.py 7 files changed, 172 insertions(+), 8 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/30772/1 diff --git a/debian/vdsm-python.install b/debian/vdsm-python.install index 9d6a99c..b80fd4f 100644 --- a/debian/vdsm-python.install +++ b/debian/vdsm-python.install @@ -8,6 +8,7 @@ ./usr/lib/python2.7/dist-packages/vdsm/exception.py ./usr/lib/python2.7/dist-packages/vdsm/ipwrapper.py ./usr/lib/python2.7/dist-packages/vdsm/libvirtconnection.py +./usr/lib/python2.7/dist-packages/vdsm/locking.py ./usr/lib/python2.7/dist-packages/vdsm/netconfpersistence.py ./usr/lib/python2.7/dist-packages/vdsm/netinfo.py ./usr/lib/python2.7/dist-packages/vdsm/netlink/__init__.py diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am index 8e90e6e..ffb642c 100644 --- a/lib/vdsm/Makefile.am +++ b/lib/vdsm/Makefile.am @@ -28,6 +28,7 @@ exception.py \ ipwrapper.py \ libvirtconnection.py \ + locking.py \ netconfpersistence.py \ netinfo.py \ profile.py \ diff --git a/lib/vdsm/locking.py b/lib/vdsm/locking.py new file mode 100644 index 0000000..9e92ca3 --- /dev/null +++ b/lib/vdsm/locking.py @@ -0,0 +1,54 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import threading +import time + + +class LockTimeout(Exception): + """ Timeout acquiring a TimedLock """ + + +class TimedLock(object): + """ + A lock raising a LockTimeout if it cannot be acquired within timeout. + """ + + def __init__(self, name, timeout): + self._name = name + self._timeout = timeout + self._cond = threading.Condition(threading.Lock()) + self._busy = False + + def __enter__(self): + deadline = time.time() + self._timeout + with self._cond: + while self._busy: + now = time.time() + if now >= deadline: + raise LockTimeout("Timeout acquiring lock %r" % self._name) + self._cond.wait(deadline - now) + self._busy = True + return self + + def __exit__(self, *args): + with self._cond: + self._busy = False + self._cond.notify() diff --git a/tests/Makefile.am b/tests/Makefile.am index aa4a45e..10a2727 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -44,6 +44,7 @@ jsonRpcTests.py \ ksmTests.py \ libvirtconnectionTests.py \ + lockingTests.py \ lvmTests.py \ main.py \ miscTests.py \ diff --git a/tests/lockingTests.py b/tests/lockingTests.py new file mode 100644 index 0000000..9ac8a05 --- /dev/null +++ b/tests/lockingTests.py @@ -0,0 +1,89 @@ +# +# Copyright 2014 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import time +import threading +from vdsm import locking +from testlib import VdsmTestCase + + +class TimedLockTests(VdsmTestCase): + + def test_free(self): + lock = locking.TimedLock("xxx-yyy-zzz", 0) + with self.assertNotRaises(): + with lock: + pass + + def test_busy(self): + lock = locking.TimedLock("xxx-yyy-zzz", 0) + with self.assertRaises(locking.LockTimeout): + with lock: + with lock: + pass + + def test_serialize(self): + lock = locking.TimedLock("xxx-yyy-zzz", 0.4) + single_thread = threading.BoundedSemaphore(1) + passed = [0] + timedout = [0] + + def run(): + try: + with lock: + with single_thread: + time.sleep(0.1) + passed[0] += 1 + except locking.LockTimeout: + timedout[0] += 1 + + self.run_threads(3, run) + self.assertEquals(passed[0], 3) + self.assertEquals(timedout[0], 0) + + def test_timeout(self): + lock = locking.TimedLock("xxx-yyy-zzz", 0.1) + single_thread = threading.BoundedSemaphore(1) + passed = [0] + timedout = [0] + + def run(): + try: + with lock: + with single_thread: + time.sleep(0.2) + passed[0] += 1 + except locking.LockTimeout: + timedout[0] += 1 + + self.run_threads(3, run) + self.assertEquals(passed[0], 1) + self.assertEquals(timedout[0], 2) + + def run_threads(self, count, target): + threads = [] + for i in range(count): + t = threading.Thread(target=target) + t.daemon = True + threads.append(t) + for t in threads: + t.start() + for t in threads: + t.join() diff --git a/vdsm.spec.in b/vdsm.spec.in index 2d371b1..c7a2b0b 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -1159,6 +1159,7 @@ %{python_sitearch}/%{vdsm_name}/exception.py* %{python_sitearch}/%{vdsm_name}/ipwrapper.py* %{python_sitearch}/%{vdsm_name}/libvirtconnection.py* +%{python_sitearch}/%{vdsm_name}/locking.py* %{python_sitearch}/%{vdsm_name}/netinfo.py* %{python_sitearch}/%{vdsm_name}/netlink/__init__.py* %{python_sitearch}/%{vdsm_name}/netlink/addr.py* diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 25eec71..9e9a844 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -37,6 +37,7 @@ # vdsm imports from vdsm import constants from vdsm import libvirtconnection +from vdsm import locking from vdsm import netinfo from vdsm import qemuimg from vdsm import utils @@ -619,12 +620,24 @@ class NotifyingVirDomain: - # virDomain wrapper that notifies vm when a method raises an exception with - # get_error_code() = VIR_ERR_OPERATION_TIMEOUT + """ + Wrap virDomain object for limiting concurrent calls and reporting timeouts. - def __init__(self, dom, tocb): + The wrapper allows only one concurrent call per vm, to prevent blocking of + multiple threads when underlying libvirt call get stuck. Invoking a domain + method will block if the domain is busy with another call. If the domain is + not available after timeout seconds, a timeout is reported and a + TimeoutError is raised. + + If a domain method was invoked, and the libvirt call failed with with + VIR_ERR_OPERATION_TIMEOUT error code, the timeout is reported, and + TimeoutError is raised. + """ + + def __init__(self, dom, tocb, vmid, timeout=30): self._dom = dom self._cb = tocb + self._timedlock = locking.TimedLock(vmid, timeout) def __getattr__(self, name): attr = getattr(self._dom, name) @@ -633,7 +646,8 @@ def f(*args, **kwargs): try: - ret = attr(*args, **kwargs) + with self._timedlock: + ret = attr(*args, **kwargs) self._cb(False) return ret except libvirt.libvirtError as e: @@ -643,6 +657,9 @@ toe.err = e.err raise toe raise + except locking.LockTimeout as e: + self._cb(True) + raise TimeoutError(e) return f @@ -2892,7 +2909,7 @@ if self.recovering: self._dom = NotifyingVirDomain( self._connection.lookupByUUIDString(self.id), - self._timeoutExperienced) + self._timeoutExperienced, self.id) elif 'restoreState' in self.conf: fromSnapshot = self.conf.get('restoreFromSnapshot', False) srcDomXML = self.conf.pop('_srcDomXML') @@ -2912,7 +2929,7 @@ self._dom = NotifyingVirDomain( self._connection.lookupByUUIDString(self.id), - self._timeoutExperienced) + self._timeoutExperienced, self.id) else: flags = libvirt.VIR_DOMAIN_NONE if 'launchPaused' in self.conf: @@ -2921,7 +2938,7 @@ del self.conf['launchPaused'] self._dom = NotifyingVirDomain( self._connection.createXML(domxml, flags), - self._timeoutExperienced) + self._timeoutExperienced, self.id) hooks.after_vm_start(self._dom.XMLDesc(0), self.conf) for dev in self._customDevices(): hooks.after_device_create(dev._deviceXML, self.conf, @@ -3690,7 +3707,7 @@ # or restart vdsm if connection to libvirt was lost self._dom = NotifyingVirDomain( self._connection.lookupByUUIDString(self.id), - self._timeoutExperienced) + self._timeoutExperienced, self.id) if not self._incomingMigrationFinished.isSet(): state = self._dom.state(0) -- To view, visit http://gerrit.ovirt.org/30772 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib459697b8688ebcba987cd6b9e11815826e92990 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 3

Change in vdsm[master]: sampling: Add sampling benchmark for profiling
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: sampling: Add sampling benchmark for profiling ...................................................................... sampling: Add sampling benchmark for profiling We have trouble profiling vdsm when using 100's of vms. This patch adds simple benchmark, simulating vdsm sampling, for evaluating the sampling implementation. Change-Id: I7bbb73f38fbbbdfc1fe7050ac3e30327411f8f33 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- A tests/perf/sampling_perf.py 1 file changed, 131 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/87/30987/1 diff --git a/tests/perf/sampling_perf.py b/tests/perf/sampling_perf.py new file mode 100644 index 0000000..f656474 --- /dev/null +++ b/tests/perf/sampling_perf.py @@ -0,0 +1,131 @@ +# Compare sampling in multiple threads vs sampling via the scheduler and +# executor. + +import pthreading +pthreading.monkey_patch() + +import sys +sys.path = ['../../lib', '../../vdsm'] + sys.path + +import libvirt +import logging +import signal +import threading +import time +from xml.dom import minidom + +from vdsm import executor as execute +from vdsm import profile +from vdsm import schedule +from vdsm import utils + +from virt import sampling + +RUN_SEC = 120 +VMS = 100 + +connection = libvirt.openReadOnly('qemu:///system') + + +def sampling_task(): + # Do some libvirt calls and do some xml processing, simulating vdsm + # sampling functions. + sp = connection.listAllStoragePools()[0] + xml = sp.XMLDesc() + dom = minidom.parseString(xml) + uuid = dom.getElementsByTagName('uuid')[0] + return uuid.childNodes[0].data + + +samplings = ( + sampling.AdvancedStatsFunction(sampling_task, 2), # highWrite + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleVmJobs + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleBalloon + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleCpu + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleNet + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleVcpuPinning + sampling.AdvancedStatsFunction(sampling_task, 15), # sampleCpuTune + sampling.AdvancedStatsFunction(sampling_task, 60), # updateVolumes + sampling.AdvancedStatsFunction(sampling_task, 60), # sampleDisk + sampling.AdvancedStatsFunction(sampling_task, 60), # sampleDiskLatency +) + + +class SamplerThread(object): + + def __init__(self, name, samplings): + self.iterator = sampling.delays(sampling.cycle(samplings)) + self.thread = threading.Thread(target=self._run, name=name) + self.thread.daemon = True + self.event = threading.Event() + self.last_sample_time = 0 + + def start(self): + logging.debug("starting %s", self.thread.name) + self.thread.start() + + def stop(self): + logging.debug("stopping %s", self.thread.name) + self.event.set() + + @utils.traceback() + def _run(self): + logging.debug("%s started", self.thread.name) + while not self.event.is_set(): + delay, task = next(self.iterator) + if delay > 0: + self.event.wait(delay) + if self.event.is_set(): + break + self.last_sample_time = time.time() + task() + logging.debug("%s stopped", self.thread.name) + + +(a)profile.profile("threads.prof", builtins=True) +def do_threads(vms): + threads = [] + try: + for i in range(vms): + t = SamplerThread('SamplerThread-%i' % i, samplings) + t.start() + threads.append(t) + time.sleep(RUN_SEC) + finally: + for t in threads: + t.stop() + + +(a)profile.profile("pool.prof", builtins=True) +def do_pool(vms): + scheduler = schedule.Scheduler() + scheduler.start() + executor = execute.Executor(5, 1000, scheduler) + executor.start() + samplers = [] + try: + for i in range(vms): + s = sampling.Sampler("Sampler-%i" % i, samplings, None, + scheduler=scheduler, executor=executor) + s.start() + samplers.append(s) + time.sleep(RUN_SEC) + finally: + for s in samplers: + s.stop() + executor.stop() + scheduler.stop() + + +if __name__ == '__main__': + logging.basicConfig(level=logging.DEBUG) + if len(sys.argv) == 1: + print 'Usage: pyhton sampling_perf.py threads|pool [vms]' + sys.exit(2) + name = 'do_' + sys.argv[1] + func = globals()[name] + if len(sys.argv) > 2: + vms = int(sys.argv[2]) + else: + vms = VMS + func(vms) -- To view, visit http://gerrit.ovirt.org/30987 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7bbb73f38fbbbdfc1fe7050ac3e30327411f8f33 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 29

Change in vdsm[master]: sampling: Scalable vm sampling
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: sampling: Scalable vm sampling ...................................................................... sampling: Scalable vm sampling Previously each vm was running a sampling thread, which is not very efficient in Python when running 100's of vms. This introduces the Sampler class, managing sampling for one vm, replacing the AdvancedStatsThread class. Each VM keeps one Sampler object, running sampling tasks serially using a thread pool, ensuring that each vm has only single sampling task running at any time. If a VM sampling task get stuck, sampling for this vm stops until the stuck task is finished. This keep the same behavior of the current code using one sampling thread per vm. For simplicity, this uses the storage thread pool for running sampling tasks. In the final version, this should use a more robust thread pool handling stuck worker threads, see: http://gerrit.ovirt.org/29191 Threads usage: - One scheduler thread serving all vm samplers - 20 worker threads. The final count should be determined by testing, ensuring that we don't overload libvirt. Configurable using new vars:vm_sampling_threads option. Change-Id: I5539a728f3a1b3075712331440eff1749da6c530 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/config.py.in M tests/numaUtilsTests.py M tests/samplingTests.py M tests/vmApiTests.py M tests/vmTests.py M vdsm/vdsm M vdsm/virt/sampling.py M vdsm/virt/vm.py 8 files changed, 420 insertions(+), 121 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/80/29980/1 diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in index 52e0cd9..c411e4e 100644 --- a/lib/vdsm/config.py.in +++ b/lib/vdsm/config.py.in @@ -190,6 +190,14 @@ ('vm_sample_cpu_tune_interval', '15', None), + ('vm_sampling_threads', '20', + 'Number of vm sampling threads. Should match libvirt thread ' + 'pool size (default 20)'), + + ('vm_sampling_tasks', '500', + 'Number of vm sampling tasks. Should be bigger then number of ' + 'running vms (default 500)'), + ('trust_store_path', '@TRUSTSTORE@', 'Where the certificates and keys are situated.'), diff --git a/tests/numaUtilsTests.py b/tests/numaUtilsTests.py index e5dd293..93e30a9 100644 --- a/tests/numaUtilsTests.py +++ b/tests/numaUtilsTests.py @@ -65,7 +65,7 @@ (3, 1, 19590000000L, 2)], 15 -class FakeVmStatsThread: +class FakeVmStats: def __init__(self, vm): self._vm = vm self.sampleVcpuPinning = FakeAdvancedStatsFunction() @@ -98,7 +98,7 @@ 'memory': '1024', 'nodeIndex': 1}]} with vmTests.FakeVM(VM_PARAMS) as fake: - fake._vmStats = FakeVmStatsThread(fake) + fake._vmStats = FakeVmStats(fake) expectedResult = {'0': [0, 1], '1': [0, 1]} vmNumaNodeRuntimeMap = numaUtils.getVmNumaNodeRuntimeInfo(fake) self.assertEqual(expectedResult, vmNumaNodeRuntimeMap) diff --git a/tests/samplingTests.py b/tests/samplingTests.py index 2ff703e..1089960 100644 --- a/tests/samplingTests.py +++ b/tests/samplingTests.py @@ -111,3 +111,228 @@ s1 = sampling.InterfaceSample(lo) s1.operstate = 'x' self.assertEquals('operstate:x', s1.connlog_diff(s0)) + + +class Task(object): + + def __init__(self, interval, name, error=None): + self.interval = interval + self.name = name + self.error = error + self.executed = 0 + + def __call__(self): + self.executed += 1 + if self.error: + raise self.error + + +class SamplingIterationTests(TestCaseBase): + + task1 = Task(2, "task1") + task2 = Task(2, "task2") + task3 = Task(3, "task3") + samplings = [task1, task2, task3] + + virtual_times = [ + # cycle 1 + (2, task1), + (2, task2), + (3, task3), + (4, task1), + (4, task2), + (6, task1), + (6, task2), + (6, task3), + # cycle 2 + (8, task1), + (8, task2), + (9, task3), + (10, task1), + (10, task2), + (12, task1), + (12, task2), + (12, task3), + ] + + delays = [ + (2, task1), + (0, task2), + (1, task3), + (1, task1), + (0, task2), + (2, task1), + (0, task2), + (0, task3), + ] + + def test_virtual_time(self): + iterator = sampling.cycle(self.samplings) + for pair in self.virtual_times: + self.assertEquals(next(iterator), pair) + + def test_delays(self): + iterator = sampling.delays(self.virtual_times) + for i in range(2): + for pair in self.delays: + self.assertEquals(next(iterator), pair) + + def test_chain(self): + iterator = sampling.delays(sampling.cycle(self.samplings)) + for i in range(3): + for pair in self.delays: + self.assertEquals(next(iterator), pair) + + +class SamplerTests(TestCaseBase): + + def setUp(self): + self.scheduler = FakeScheduler() + self.threadpool = FakeThreadPool() + self.handler = FakeHandler(True) + self.task1 = Task(2, "task1") + self.task2 = Task(3, "task2") + self.task3 = Task(4, "task3") + samplings = [self.task1, self.task2, self.task3] + self.sampler = sampling.Sampler("vm-id", samplings, self.handler, + scheduler=self.scheduler, + threadpool=self.threadpool) + + def test_start(self): + self.sampler.start() + self.assertEquals(self.scheduler.call.delay, 2) + self.assertEquals(self.scheduler.call.func, self.sampler._dispatch) + self.assertEquals(self.threadpool.tasks, []) + + def test_stop(self): + self.sampler.start() + self.sampler.stop() + self.assertEquals(self.scheduler.call.func, INVALID) + self.assertEquals(self.threadpool.tasks, []) + + def test_dispatch(self): + self.sampler.start() + self.scheduler.fire() + self.assertEquals(self.threadpool.tasks, + [("vm-id_task1", self.sampler)]) + + def test_run_samplings(self): + self.sampler.start() + + # time = 2 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.scheduler.call.delay, 1) + self.assertEquals(self.task1.executed, 1) + + # time = 3 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.task2.executed, 1) + self.assertEquals(self.scheduler.call.delay, 1) + + # time = 4 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.task1.executed, 2) + self.assertEquals(self.scheduler.call.delay, 0) + + # time = 4 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.task3.executed, 1) + self.assertEquals(self.scheduler.call.delay, 2) + + # time = 6 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.task1.executed, 3) + self.assertEquals(self.scheduler.call.delay, 0) + + # time = 6 + self.scheduler.fire() + self.threadpool.run() + self.assertEquals(self.task2.executed, 2) + self.assertEquals(self.scheduler.call.delay, 2) + + +class SamplerHandlerTests(TestCaseBase): + + def test_does_handle_error(self): + self.check(FakeHandler(True)) + + def test_does_not_handle_error(self): + self.check(FakeHandler(False)) + + def check(self, handler): + scheduler = FakeScheduler() + threadpool = FakeThreadPool() + error = Exception("Sampling task failed") + task = Task(2, "task", error=error) + sampler = sampling.Sampler("vm-id", [task], handler, + scheduler=scheduler, threadpool=threadpool) + sampler.start() + scheduler.fire() + threadpool.run() + self.assertEquals(handler.error, error) + + +# Helpers + + +class FakeHandler(object): + + def __init__(self, result): + self.error = None + self.result = result + + def handleStatsException(self, e): + self.error = e + return self.result + + +class FakeScheduler(object): + + def __init__(self): + # Schedule only single call to simplify the tests + self.call = None + + def schedule(self, delay, func): + assert self.call is None + self.call = Call(delay, func) + return self.call + + def fire(self): + call = self.call + self.call = None + call.func() + + +class Call(object): + + def __init__(self, delay, func): + self.delay = delay + self.func = func + + def cancel(self): + self.func = INVALID + + +def INVALID(): + pass + + +class FakeThreadPool(object): + + def __init__(self): + self.tasks = [] + + def queueTask(self, id, task): + # Executes task on a worker thread in the real object. Does nothing + # here to simplify the tests. + self.tasks.append((id, task)) + + def run(self): + # Run next task manually to simplify the tests + task_id, task = self.tasks.pop() + task() diff --git a/tests/vmApiTests.py b/tests/vmApiTests.py index bc68c6d..ae2f98f 100644 --- a/tests/vmApiTests.py +++ b/tests/vmApiTests.py @@ -22,6 +22,7 @@ import os import os.path +from virt import sampling from virt import vmexitreason from vdsm import define from testrunner import VdsmTestCase as TestCaseBase @@ -56,11 +57,15 @@ @contextmanager def ensureVmStats(vm): - vm._initVmStats() + sampling.start() try: - yield vm + vm._initVmStats() + try: + yield vm + finally: + vm._vmStats.stop() finally: - vm._vmStats.stop() + sampling.stop() class TestVmStats(TestSchemaCompliancyBase): diff --git a/tests/vmTests.py b/tests/vmTests.py index fda71f5..3ef04f9 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -1338,7 +1338,7 @@ self.assertEqual(stats['exitMessage'], msg) -class TestVmStatsThread(TestCaseBase): +class TestVmStats(TestCaseBase): VM_PARAMS = {'displayPort': -1, 'displaySecurePort': -1, 'display': 'qxl', 'displayIp': '127.0.0.1', 'vmType': 'kvm', 'memSize': 1024} @@ -1349,8 +1349,8 @@ GBPS = 10 ** 9 / 8 MAC = '52:54:00:59:F5:3F' with FakeVM() as fake: - mock_stats_thread = vm.VmStatsThread(fake) - res = mock_stats_thread._getNicStats( + mock_stats = vm.VmStats(fake) + res = mock_stats._getNicStats( name='vnettest', model='virtio', mac=MAC, start_sample=(2 ** 64 - 15 * GBPS, 1, 2, 3, 0, 4, 5, 6), end_sample=(0, 7, 8, 9, 5 * GBPS, 10, 11, 12), @@ -1366,9 +1366,9 @@ # bz1073478 - main case with FakeVM(self.VM_PARAMS, self.DEV_BALLOON) as fake: self.assertEqual(fake._dom, None) - mock_stats_thread = vm.VmStatsThread(fake) + mock_stats = vm.VmStats(fake) res = {} - mock_stats_thread._getBalloonStats(res) + mock_stats._getBalloonStats(res) self.assertIn('balloonInfo', res) self.assertIn('balloon_cur', res['balloonInfo']) @@ -1376,9 +1376,9 @@ # bz1073478 - extra case with FakeVM(self.VM_PARAMS, self.DEV_BALLOON) as fake: fake._dom = FakeDomain() - mock_stats_thread = vm.VmStatsThread(fake) + mock_stats = vm.VmStats(fake) res = {} - mock_stats_thread._getBalloonStats(res) + mock_stats._getBalloonStats(res) self.assertIn('balloonInfo', res) self.assertIn('balloon_cur', res['balloonInfo']) diff --git a/vdsm/vdsm b/vdsm/vdsm index cbc16ff..2343606 100755 --- a/vdsm/vdsm +++ b/vdsm/vdsm @@ -38,6 +38,8 @@ from storage.dispatcher import Dispatcher from storage.hsm import HSM +from virt import sampling + import zombiereaper import dsaversion @@ -71,6 +73,7 @@ profile.start() libvirtconnection.start_event_loop() + sampling.start() if config.getboolean('irs', 'irs_enable'): try: @@ -88,6 +91,7 @@ profile.stop() finally: cif.prepareForShutdown() + sampling.stop() def run(pidfile=None): diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 3479eee..0cd823a 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -26,24 +26,50 @@ Contains a reverse dictionary pointing from error string to its error code. """ -import threading -import os -import time -import logging import errno import ethtool +import itertools +import logging +import operator +import os import re +import threading +import time +from vdsm.config import config from vdsm import utils from vdsm import netinfo +from vdsm import schedule from vdsm.ipwrapper import getLinks from vdsm.constants import P_VDSM_RUN import caps +from storage import threadPool _THP_STATE_PATH = '/sys/kernel/mm/transparent_hugepage/enabled' if not os.path.exists(_THP_STATE_PATH): _THP_STATE_PATH = '/sys/kernel/mm/redhat_transparent_hugepage/enabled' + + +_scheduler = None +_threadpool = None + + +def start(): + """ Called during application startup """ + global _scheduler, _threadpool + threads = config.getint('vars', 'vm_sampling_threads') + tasks = config.getint('vars', 'vm_sampling_tasks') + _threadpool = threadPool.ThreadPool(threads, maxTasks=tasks) + _scheduler = schedule.Scheduler("sampling.scheduler") + + +def stop(): + """ Called during application shutdown """ + if _threadpool: + _threadpool.joinAll(waitForTasks=False, waitForThreads=False) + if _scheduler: + _scheduler.cancel() class InterfaceSample: @@ -318,9 +344,13 @@ def interval(self): return self._interval + @property + def name(self): + return self._function.__name__ + def __repr__(self): return "<AdvancedStatsFunction %s at 0x%x>" % ( - self._function.__name__, id(self._function.__name__)) + self.name, id(self)) def __call__(self, *args, **kwargs): retValue = self._function(*args, **kwargs) @@ -347,115 +377,136 @@ return bgn_sample, end_sample, (end_time - bgn_time) -class AdvancedStatsThread(threading.Thread): +class Sampler(object): """ - A thread that runs the registered AdvancedStatsFunction objects - for statistic and monitoring purpose. + Execute sampling tasks on the threadpool according to sampling schedule. + + The sepcified samplings tasks are executed on threadpool one at a time, as + if there was a thread running them one after another in a loop. + + If one of the sampling task get stuck, the rest of the sampling tasks are + delayed. This is important to prevent flooding of the thread pool with + possibly blocking sampling task, when the underlying libvirt thread is + stuck. When a stuck sampling tasks finish, the sampling tasks continue + normally. + + When a sampling task is stuck, no other task can run on the blocked worker + thread. The threadpool is responsible for detecting and handling this. + + Each vm should create one sampler, to ensure that there is only one + sampling task per vm excuting on the threadpool. """ - DEFAULT_LOG = logging.getLogger("AdvancedStatsThread") - def __init__(self, log=DEFAULT_LOG, daemon=False): - """ - Initialize an AdvancedStatsThread object - """ - threading.Thread.__init__(self) - self.daemon = daemon + _log = logging.getLogger("Sampler") - self._log = log - self._stopEvent = threading.Event() - self._contEvent = threading.Event() - - self._statsTime = None - self._statsFunctions = [] - - def addStatsFunction(self, *args): - """ - Register the functions listed as arguments - """ - if self.isAlive(): - raise RuntimeError("AdvancedStatsThread is started") - - for statsFunction in args: - self._statsFunctions.append(statsFunction) + def __init__(self, vm_id, samplings, handler, scheduler=None, + threadpool=None): + self._vm_id = vm_id + self._samplings = samplings + self._handler = handler + self._scheduler = scheduler or _scheduler + self._threadpool = threadpool or _threadpool + self._lock = threading.Lock() + self._running = False + self._iterator = None + self._task = _INVALID + self._call = None + self._last_sample_time = 0 # Not sample taken yet def start(self): - """ - Start the execution of the thread and exit - """ - self._log.debug("Start statistics collection") - threading.Thread.start(self) + with self._lock: + if self._running: + raise AssertionError("Sampler is running") + self._log.debug("Starting sampler for vm %s", self._vm_id) + self._running = True + self._iterator = delays(cycle(self._samplings)) + self._schedule_next_task() def stop(self): + with self._lock: + if self._running: + self._log.debug("Stopping sampler for vm %s", self._vm_id) + self._running = False + self._call.cancel() + self._call = None + self._task = _INVALID + self._iterator = None + + @property + def last_sample_time(self): + return self._last_sample_time + + # Task interface + + def __call__(self): + try: + self._last_sample_time = time.time() + self._task() + except Exception as e: + if not self._handler.handleStatsException(e): + self._log.exception("Stats function failed: %s", self._task) + finally: + with self._lock: + if self._running: + self._schedule_next_task() + + # Private + + def _dispatch(self): """ - Stop the execution of the thread and exit + Called from the scheduler thread when its time to ran the current + sampling task. """ - self._log.debug("Stop statistics collection") - self._stopEvent.set() - self._contEvent.set() + with self._lock: + if self._running: + task_id = self._vm_id + '_' + self._task.name + self._threadpool.queueTask(task_id, self) - def pause(self): - """ - Pause the execution of the registered functions - """ - self._log.debug("Pause statistics collection") - self._contEvent.clear() + def _schedule_next_task(self): + delay, self._task = next(self._iterator) + self._call = self._scheduler.schedule(delay, self._dispatch) - def cont(self): - """ - Resume the execution of the registered functions - """ - self._log.debug("Resume statistics collection") - self._contEvent.set() - def getLastSampleTime(self): - return self._statsTime +# Sentinel used to mark a task as invalid, allowing running sampling task +# without holding a lock. +def _INVALID(): + pass - def run(self): - self._log.debug("Stats thread started") - self._contEvent.set() - while not self._stopEvent.isSet(): - try: - self.collect() - except: - self._log.debug("Stats thread failed", exc_info=True) +def delays(virtual_times): + """ + Accept stream of tuples (virtual_time, task) and return stream of tuples + (delay, task). + """ + last_virtual_time = 0 + for virtual_time, task in virtual_times: + delay = virtual_time - last_virtual_time + last_virtual_time = virtual_time + yield delay, task - self._log.debug("Stats thread finished") - def handleStatsException(self, ex): - """ - Handle the registered function exceptions and eventually stop the - sampling if a fatal error occurred. - """ - return False +def cycle(samplings): + """ + Returns endless stream of tuples (virtual_time, task) + """ + samplings = group_by_interval(samplings) + virtual_time = 1 + while True: + for interval, tasks in samplings: + if virtual_time % interval == 0: + for task in tasks: + yield virtual_time, task + virtual_time += 1 - def collect(self): - # TODO: improve this with lcm - _mInt = map(lambda x: x.interval, self._statsFunctions) - maxInterval = reduce(lambda x, y: x * y, set(_mInt), 1) - intervalAccum = 0 - while not self._stopEvent.isSet(): - self._contEvent.wait() - - self._statsTime = time.time() - waitInterval = maxInterval - - for statsFunction in self._statsFunctions: - thisInterval = statsFunction.interval - ( - intervalAccum % statsFunction.interval) - waitInterval = min(waitInterval, thisInterval) - - if intervalAccum % statsFunction.interval == 0: - try: - statsFunction() - except Exception as e: - if not self.handleStatsException(e): - self._log.error("Stats function failed: %s", - statsFunction, exc_info=True) - - self._stopEvent.wait(waitInterval) - intervalAccum = (intervalAccum + waitInterval) % maxInterval +def group_by_interval(samplings): + """ + Groups samplings by interval. + """ + keyfn = operator.attrgetter('interval') + samplings = sorted(samplings, key=keyfn) + return [(interval, tuple(tasks)) + for interval, tasks in itertools.groupby(samplings, keyfn)] class HostStatsThread(threading.Thread): diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 7835e99..f1d8983 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -65,7 +65,7 @@ from . import vmstatus from .vmtune import updateIoTuneDom -from .sampling import AdvancedStatsFunction, AdvancedStatsThread +from .sampling import AdvancedStatsFunction, Sampler from .utils import isVdsmImage, XMLElement from vmpowerdown import VmShutdown, VmReboot @@ -167,7 +167,7 @@ pass -class VmStatsThread(AdvancedStatsThread): +class VmStats(object): MBPS_TO_BPS = 10 ** 6 / 8 # CPU tune sampling window @@ -184,7 +184,6 @@ _libvirt_metadata_supported = True def __init__(self, vm): - AdvancedStatsThread.__init__(self, log=vm.log, daemon=True) self._vm = vm self.highWrite = ( @@ -237,11 +236,18 @@ config.getint('vars', 'vm_sample_cpu_tune_interval'), self.CPU_TUNE_SAMPLING_WINDOW)) - self.addStatsFunction( - self.highWrite, self.updateVolumes, self.sampleCpu, - self.sampleDisk, self.sampleDiskLatency, self.sampleNet, - self.sampleBalloon, self.sampleVmJobs, self.sampleVcpuPinning, - self.sampleCpuTune) + samplings = (self.highWrite, self.updateVolumes, self.sampleCpu, + self.sampleDisk, self.sampleDiskLatency, self.sampleNet, + self.sampleBalloon, self.sampleVmJobs, + self.sampleVcpuPinning, self.sampleCpuTune) + + self._sampler = Sampler(self._vm.id, samplings, self) + + def start(self): + self._sampler.start() + + def stop(self): + self._sampler.stop() def _highWrite(self): if not self._vm.isDisksStatsCollectionEnabled(): @@ -311,13 +317,13 @@ metadataCpuLimit = None try: - if VmStatsThread._libvirt_metadata_supported: + if VmStats._libvirt_metadata_supported: metadataCpuLimit = self._vm._dom.metadata( libvirt.VIR_DOMAIN_METADATA_ELEMENT, METADATA_VM_TUNE_URI, 0) except libvirt.libvirtError as e: if e.get_error_code() == libvirt.VIR_ERR_ARGUMENT_UNSUPPORTED: - VmStatsThread._libvirt_metadata_supported = False + VmStats._libvirt_metadata_supported = False self._log.error("libvirt does not support metadata") elif (e.get_error_code() @@ -581,7 +587,7 @@ stats = {} try: - stats['statsAge'] = time.time() - self.getLastSampleTime() + stats['statsAge'] = time.time() - self._sampler.last_sample_time except TypeError: self._log.debug("Stats age not available") stats['statsAge'] = -1.0 @@ -2778,7 +2784,7 @@ WARNING: this method should only gather statistics by copying data. Especially avoid costly and dangerous ditrect calls to the _dom - attribute. Use the VmStatsThread instead! + attribute. Use the VmStats instead! """ if self.lastStatus == vmstatus.DOWN: @@ -3059,7 +3065,7 @@ return domxml.toxml() def _initVmStats(self): - self._vmStats = VmStatsThread(self) + self._vmStats = VmStats(self) self._vmStats.start() self._guestEventTime = self._startTime @@ -3174,7 +3180,7 @@ if drive['device'] == 'disk' and isVdsmImage(drive): self._syncVolumeChain(drive) - # VmStatsThread may use block devices info from libvirt. + # VmStats may use block devices info from libvirt. # So, run it after you have this info self._initVmStats() self.guestAgent = guestagent.GuestAgent( -- To view, visit http://gerrit.ovirt.org/29980 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5539a728f3a1b3075712331440eff1749da6c530 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 46

Change in vdsm[master]: resourceManager: Keep resource state if registerResource fails
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: resourceManager: Keep resource state if registerResource fails ...................................................................... resourceManager: Keep resource state if registerResource fails Previous code was increasing resource activeUsers counter, but exceptions raised after that caused the method to fail, leaving a locked resources that nobody can release. Such locked resource may lead to failure of any pool operation, making the host non-operational, and requiring a restart of vdsm. The failure in the field was caused by Python logging bug, raising OSError when message was logged when log file was rotated. However, such failure can happen everywhere, and locking code must be written in such way that failure would never leave a resource locked. This patch ensure that resource is added and activeUsers counter is increased only if everything else was fine. Since simulating logging error is hard, the tests monkeypatch the RequestRef class to simulate a failure. This is little ugly, depending on internal implementation detail, but I could not find a cleaner way. Change-Id: I16abf41ebc8a8a99b292d38c945074752254a34b Relates-To: https://bugzilla.redhat.com/1065650 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/resourceManagerTests.py M vdsm/storage/resourceManager.py 2 files changed, 50 insertions(+), 9 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/84/25284/1 diff --git a/tests/resourceManagerTests.py b/tests/resourceManagerTests.py index 01b0669..e2b6461 100644 --- a/tests/resourceManagerTests.py +++ b/tests/resourceManagerTests.py @@ -29,6 +29,7 @@ import storage.resourceManager as resourceManager from testrunner import VdsmTestCase as TestCaseBase from testValidation import slowtest, stresstest +import monkeypatch class NullResourceFactory(resourceManager.SimpleResourceFactory): @@ -209,6 +210,32 @@ ex.__class__.__name__) self.fail("Managed to access an attribute not exposed by wrapper") + + def testRegisterResourceFailureExclusive(self): + # This regeisterion must fail + with monkeypatch.MonkeyPatchScope( + [(resourceManager, 'RequestRef', FakeRequestRef)]): + self.assertRaises(Failure, self.manager.registerResource, "string", + "test", resourceManager.LockType.exclusive, None) + + # And it should not leave a locked resource + with self.manager.acquireResource("string", "test", + resourceManager.LockType.exclusive, + 0): + pass + + def testRegisterResourceFailureShared(self): + # This regeisterion must fail + with monkeypatch.MonkeyPatchScope( + [(resourceManager, 'RequestRef', FakeRequestRef)]): + self.assertRaises(Failure, self.manager.registerResource, "string", + "test", resourceManager.LockType.shared, None) + + # And it should not leave a locked resource + with self.manager.acquireResource("string", "test", + resourceManager.LockType.exclusive, + 0): + pass def testAccessAttributeNotExposedByRequestRef(self): resources = [] @@ -705,3 +732,14 @@ except: resourceManager.ResourceManager._instance = None raise + +# Helpers + + +class Failure(Exception): + """ Unique exception for testing """ + + +def FakeRequestRef(*a, **kw): + """ Used to simulate failures when registering a resource """ + raise Failure() diff --git a/vdsm/storage/resourceManager.py b/vdsm/storage/resourceManager.py index 1be1450..ce144cf 100644 --- a/vdsm/storage/resourceManager.py +++ b/vdsm/storage/resourceManager.py @@ -559,23 +559,25 @@ if len(resource.queue) == 0 and \ resource.currentLock == LockType.shared and \ request.lockType == LockType.shared: - resource.activeUsers += 1 self._log.debug("Resource '%s' found in shared state " "and queue is empty, Joining current " "shared lock (%d active users)", - fullName, resource.activeUsers) + fullName, resource.activeUsers + 1) request.grant() + ref = RequestRef(request) contextCleanup.defer(request.emit, ResourceRef(namespace, name, resource.realObj, request.reqID)) - return RequestRef(request) + resource.activeUsers += 1 + return ref - resource.queue.insert(0, request) self._log.debug("Resource '%s' is currently locked, " "Entering queue (%d in queue)", - fullName, len(resource.queue)) - return RequestRef(request) + fullName, len(resource.queue) + 1) + ref = RequestRef(request) + resource.queue.insert(0, request) + return ref # TODO : Creating the object inside the namespace lock causes # the entire namespace to lock and might cause @@ -592,19 +594,20 @@ contextCleanup.defer(request.cancel) return RequestRef(request) - resource = resources[name] = ResourceManager.ResourceInfo( - obj, namespace, name) + resource = ResourceManager.ResourceInfo(obj, namespace, name) resource.currentLock = request.lockType resource.activeUsers += 1 self._log.debug("Resource '%s' is free. Now locking as '%s' " "(1 active user)", fullName, request.lockType) request.grant() + ref = RequestRef(request) contextCleanup.defer(request.emit, ResourceRef(namespace, name, resource.realObj, request.reqID)) - return RequestRef(request) + resources[name] = resource + return ref def releaseResource(self, namespace, name): # WARN : unlike in resource acquire the user now has the request -- To view, visit http://gerrit.ovirt.org/25284 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I16abf41ebc8a8a99b292d38c945074752254a34b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

8 31

Change in vdsm[master]: virt: Fix wrong usage of apparentsize and truesize
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: virt: Fix wrong usage of apparentsize and truesize ...................................................................... virt: Fix wrong usage of apparentsize and truesize Drive extension and live-merge code was assuming that apparentsize is the volume allocated size, and truesize is the virtual volume size. Looking in the schema, file-based storage implementation, and engine, it is clear that this is wrong. The reason that this code works, is that in block storage we do not implement apparentsize, so apparentsize is always equal to truesize. This patch switch the usage of apparentsize and truesize in code handling only block stoage. Code that handle both file and block storage is correct. Since apparentsize and truesize are the same in block storage, this patch does not change the behavior of the system. However, it is required for the next patch, implementing apparentsize in block storage. Change-Id: Ie447f40df0dae496c2400d0af9d81247ab5740a5 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 9 insertions(+), 9 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/34/37834/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 04f1a67..88f4783 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1461,9 +1461,9 @@ for drive, volumeID, capacity, alloc, physical in extend: self.log.info( - "Requesting extension for volume %s on domain %s (apparent: " + "Requesting extension for volume %s on domain %s (truesize: " "%s, capacity: %s, allocated: %s, physical: %s)", - volumeID, drive.domainID, drive.apparentsize, capacity, + volumeID, drive.domainID, drive.truesize, capacity, alloc, physical) self.extendDriveVolume(drive, volumeID, physical, capacity) @@ -1518,9 +1518,9 @@ self.log.debug("Verifying extension for volume %s, requested size %s, " "current size %s", volInfo['volumeID'], - volInfo['newSize'], apparentSize) + volInfo['newSize'], trueSize) - if apparentSize < volInfo['newSize']: + if trueSize < volInfo['newSize']: raise RuntimeError( "Volume extension failed for %s (domainID: %s, volumeID: %s)" % (volInfo['name'], volInfo['domainID'], volInfo['volumeID'])) @@ -3356,8 +3356,8 @@ # We need to stop the collection of the stats for two reasons, one # is to prevent spurious libvirt errors about missing drive paths # (since we're changing them), and also to prevent to trigger a drive - # extension for the new volume with the apparent size of the old one - # (the apparentsize is updated as last step in updateDriveParameters) + # extension for the new volume with the true size of the old one (the + # true is updated as last step in updateDriveParameters). self.stopDisksStatsCollection() try: @@ -4740,8 +4740,8 @@ flags |= libvirt.VIR_DOMAIN_BLOCK_COMMIT_ACTIVE # If top is the active layer, it's allocated size is stored in - # drive.apparantsize. - topSize = drive.apparentsize + # the drive.truesize. + topSize = drive.truesize else: # If top is an internal volume, we must call getVolumeInfo res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, @@ -4750,7 +4750,7 @@ self.log.error("Unable to get volume info for '%s'", topVolUUID) return errCode['mergeErr'] - topSize = int(res['info']['apparentsize']) + topSize = int(res['info']['truesize']) # Take the jobs lock here to protect the new job we are tracking from # being cleaned up by queryBlockJobs() since it won't exist right away -- To view, visit http://gerrit.ovirt.org/37834 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie447f40df0dae496c2400d0af9d81247ab5740a5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 12

Change in vdsm[master]: hsm: Return true "truesize" for block volumes
by Nir Soffer 13 Aug '15

13 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: hsm: Return true "truesize" for block volumes ...................................................................... hsm: Return true "truesize" for block volumes HSM.getVolumeSize() and HSM.getVolumeInfo() used to return bogus value for "truesize" for block volumes. This evil undocumented lie was introduced in commit commit 1cef1409b1 (BZ#551944 - VDSM: getVolumeInfo: Actual size field is not correct for RAW sparse files) on 2010. Now we obtain block volumes "truesize" using qemu-img info. Change-Id: Ib1e726727ef96f26a5d04ef86c87cb18d1ef658f Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/blockSD.py M vdsm/storage/blockVolume.py 2 files changed, 11 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/85/37785/1 diff --git a/vdsm/storage/blockSD.py b/vdsm/storage/blockSD.py index caaef6b..ebde78c 100644 --- a/vdsm/storage/blockSD.py +++ b/vdsm/storage/blockSD.py @@ -617,7 +617,9 @@ return int(size) - getVAllocSize = getVSize + def getVAllocSize(self, imgUUID, volUUID): + """ Returns block volume true size in bytes. """ + return self.produceVolume(imgUUID, volUUID).getVolumeTrueSize(bs=1) def validateCreateVolumeParams(self, volFormat, srcVolUUID, preallocate=None): diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 2bb9a78..99c8840 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -643,7 +643,14 @@ sdobj = sdCache.produce(sdUUID=self.sdUUID) return int(sdobj.getVSize(self.imgUUID, self.volUUID) / bs) - getVolumeTrueSize = getVolumeSize + def getVolumeTrueSize(self, bs=BLOCK_SIZE): + """ + Return the volume true size in blocks + + Raises qemuimg.QImgError if volume size cannot be obtained. + """ + info = qemuimg.info(self.getVolumePath()) + return int(info["virtualsize"] / bs) def _extendSizeRaw(self, newSize): # Since this method relies on lvm.extendLV (lvextend) when the -- To view, visit http://gerrit.ovirt.org/37785 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib1e726727ef96f26a5d04ef86c87cb18d1ef658f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 22

Change in vdsm[master]: hostdev: don't send address type for source address
by Martin Polednik 12 Aug '15

12 Aug '15

Martin Polednik has uploaded a new change for review. Change subject: hostdev: don't send address type for source address ...................................................................... hostdev: don't send address type for source address Libvirt (domain_conf.c) does not expect the type for the host address, and drops it if we send it. This causes USB address matching to fail when parsing dom XML. Same applies for SR-IOV, the example in libvirt documentation states the type but it is again dropped. Change-Id: I12592dd53f8af25fe8e1312b810e6b6f48411083 Signed-off-by: Martin Polednik <mpolednik(a)redhat.com> --- M tests/hostdevTests.py M vdsm/virt/vmdevices/hostdevice.py 2 files changed, 18 insertions(+), 23 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/77/44177/1 diff --git a/tests/hostdevTests.py b/tests/hostdevTests.py index febce55..ae0809b 100644 --- a/tests/hostdevTests.py +++ b/tests/hostdevTests.py @@ -44,8 +44,7 @@ ''' <hostdev managed="no" mode="subsystem" type="pci"> <source> - <address bus="0" domain="0" function="0" slot="2" - type="pci"/> + <address bus="0" domain="0" function="0" slot="2"/> </source> %s </hostdev> @@ -54,8 +53,7 @@ ''' <hostdev managed="no" mode="subsystem" type="pci"> <source> - <address bus="0" domain="0" function="0" slot="25" - type="pci"/> + <address bus="0" domain="0" function="0" slot="25"/> </source> %s </hostdev> @@ -64,8 +62,7 @@ ''' <hostdev managed="no" mode="subsystem" type="pci"> <source> - <address bus="0" domain="0" function="0" slot="26" - type="pci"/> + <address bus="0" domain="0" function="0" slot="26"/> </source> %s </hostdev> @@ -74,8 +71,7 @@ ''' <hostdev managed="no" mode="subsystem" type="pci"> <source> - <address bus="0" domain="0" function="0" slot="27" - type="pci"/> + <address bus="0" domain="0" function="0" slot="27"/> </source> %s </hostdev> @@ -84,8 +80,7 @@ ''' <hostdev managed="no" mode="subsystem" type="pci"> <source> - <address bus="0" domain="0" function="2" slot="31" - type="pci"/> + <address bus="0" domain="0" function="2" slot="31"/> </source> %s </hostdev> @@ -94,7 +89,7 @@ ''' <hostdev managed="no" mode="subsystem" type="usb"> <source> - <address bus="1" device="2" type="usb"/> + <address bus="1" device="2"/> </source> %s </hostdev> @@ -103,7 +98,7 @@ ''' <hostdev managed="no" mode="subsystem" type="usb"> <source> - <address bus="1" device="10" type="usb"/> + <address bus="1" device="10"/> </source> %s </hostdev> @@ -112,7 +107,7 @@ ''' <hostdev managed="no" mode="subsystem" type="usb"> <source> - <address bus="1" device="1" type="usb"/> + <address bus="1" device="1"/> </source> %s </hostdev> @@ -122,8 +117,7 @@ <interface managed="no" type="hostdev"> <driver name="vfio"/> <source> - <address bus="5" domain="0" function="7" slot="16" - type="pci"/> + <address bus="5" domain="0" function="7" slot="16"/> </source> <mac address="ff:ff:ff:ff:ff:ff"/> <vlan> diff --git a/vdsm/virt/vmdevices/hostdevice.py b/vdsm/virt/vmdevices/hostdevice.py index 976a66b..453dc1d 100644 --- a/vdsm/virt/vmdevices/hostdevice.py +++ b/vdsm/virt/vmdevices/hostdevice.py @@ -36,9 +36,7 @@ self.macAddr = self.specParams.get('macAddr') self.vlanId = self.specParams.get('vlanId') - self.hostAddress = self._update_address( - self._deviceParams.get('address') - ) + self.hostAddress = self._deviceParams.get('address') @property def address(self): @@ -54,9 +52,13 @@ by mutating the address. The address is then returned for convenience. ''' if 'type' not in address: - address['type'] = CAPABILITY_TO_XML_ATTR[ - self._deviceParams['capability'] - ] + try: + address['type'] = CAPABILITY_TO_XML_ATTR[ + self._deviceParams['capability']] + except AttributeError: + # _deviceParams may not be yet exist when accessing the + # address, in that case we leave it untouched + pass return address @@ -66,7 +68,6 @@ called before getXML in order to populate _deviceParams. """ self._deviceParams = detach_detachable(self.device) - self.hostAddress = self._update_address(self.hostAddress) def getXML(self): if any((self.macAddr, self.vlanId)): @@ -120,7 +121,7 @@ <interface type='hostdev' managed='no'> <driver name='vfio'/> <source> - <address type='pci' domain='0x0000' bus='0x00' slot='0x07' + <address domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </source> <mac address='52:54:00:6d:90:02'/> -- To view, visit https://gerrit.ovirt.org/44177 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I12592dd53f8af25fe8e1312b810e6b6f48411083 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

4 23

Change in vdsm[master]: vm: Improve error handling when Vm._dom is None
by Nir Soffer 11 Aug '15

11 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: vm: Improve error handling when Vm._dom is None ...................................................................... vm: Improve error handling when Vm._dom is None Vm._dom is initialized to None on when creating a vm, and set to None if the underlying libvirt domain has died. Since Vm._dom is modified by multiple threads, it is impossible to check for None before using it. Even if it was possible, we don't want to litter the code with None checks everywhere. Some code was using this pattern: try: self._dom.doSomething() except AttributeError: # Oh, it was None This code is not communicating well our intent. Worse, it hides AttributeError in doSomthing()! Most code never check self._dom before using it. In the rare cases it is None, we fail with this Traceback: Thread-460::ERROR::2015-07-04 19:53:21,977::__init__::520::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 515, in _serveRequest res = method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 277, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 727, in freeze return v.freeze() File "/usr/share/vdsm/virt/vm.py", line 2882, in freeze frozen = self._dom.fsFreeze() AttributeError: 'NoneType' object has no attribute 'fsFreeze' This traceback is a poor way to say "the vm is not running". This patch introduces the DisconnectedVirDomain class. This object will raise NotConnectedError for any attribute access. Vm._dom is initialzied to DisconnectedVirDomain(vmid) on startup and after underlying libvirt domain has died. Code trying to talk with a dead vm will fail now with: NotConnectedError: Vm '681f6b09-a9c3-4422-a7e2-2f607368718b' is not running yet or was shut down. Code handling disconnected state is using now: try: self._dom.doSomething() except NotConnctedError: ... This communicates our intent, and does not hide any error from the underlying code. Code checking for None is using now: if self._dom.connected: ... Change-Id: I5349ec51c7accf3b417b3bc9489c7eed5bfd8733 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/vmTests.py M tests/vmfakelib.py M vdsm/virt/sampling.py M vdsm/virt/vm.py 4 files changed, 47 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/25/43225/1 diff --git a/tests/vmTests.py b/tests/vmTests.py index 038ffcc..5b63e89 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -1056,7 +1056,6 @@ def testDomainNotRunningWithoutDomain(self): with fake.VM() as testvm: - self.assertEqual(testvm._dom, None) self.assertFalse(testvm._isDomainRunning()) def testDomainNotRunningByState(self): @@ -1087,7 +1086,6 @@ def testDomainNoneNotReadyForCommands(self): with fake.VM() as testvm: - testvm._dom = None self.assertFalse(testvm.isDomainReadyForCommands()) def testReadyForCommandsRaisesLibvirtError(self): @@ -1492,7 +1490,6 @@ def testVmWithoutDom(self): with fake.VM() as testvm: - self.assertTrue(testvm._dom is None) self.assertAPIFailed(testvm.setBalloonTarget(128)) def testTargetValueNotInteger(self): diff --git a/tests/vmfakelib.py b/tests/vmfakelib.py index 211d647..d75198e 100644 --- a/tests/vmfakelib.py +++ b/tests/vmfakelib.py @@ -179,6 +179,10 @@ self._diskErrors = {} self._downtimes = [] + @property + def connected(self): + return True + def _failIfRequested(self): if self._virtError != libvirt.VIR_ERR_OK: raise Error(self._virtError, self._errorMessage) diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index af0910a..a14a1ef 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -546,6 +546,9 @@ elif not vm_obj.isDomainReadyForCommands(): self._skip_doms[vm_id] = True else: + # TODO: This racy check may fail if the underlying libvirt + # domain has died just after checking isDomainReadyForCommands + # succeeded. doms.append(vm_obj._dom._dom) return doms diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 826e4e9..fa7c517 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -138,6 +138,13 @@ pass +class NotConnectedError(Exception): + """ + Raised when trying to talk with a vm that was not started yet or was shut + down. + """ + + VALID_STATES = (vmstatus.DOWN, vmstatus.MIGRATION_DESTINATION, vmstatus.MIGRATION_SOURCE, vmstatus.PAUSED, vmstatus.POWERING_DOWN, vmstatus.REBOOT_IN_PROGRESS, @@ -188,6 +195,20 @@ pass +class DisconnectedVirDomain(object): + + def __init__(self, vmid): + self.vmid = vmid + + @property + def connected(self): + return False + + def __getattr__(self, name): + raise NotConnectedError("VM %r was not started yet or was shut down" + % self.vmid) + + class NotifyingVirDomain: # virDomain wrapper that notifies vm when a method raises an exception with # get_error_code() = VIR_ERR_OPERATION_TIMEOUT @@ -195,6 +216,10 @@ def __init__(self, dom, tocb): self._dom = dom self._cb = tocb + + @property + def connected(self): + return True def __getattr__(self, name): attr = getattr(self._dom, name) @@ -277,7 +302,7 @@ :param recover: Signal if the Vm is recovering; :type recover: bool """ - self._dom = None + self._dom = DisconnectedVirDomain(params["vmId"]) self.recovering = recover self.conf = {'pid': '0', '_blockJobs': {}, 'clientIp': ''} self.conf.update(params) @@ -810,7 +835,7 @@ def _onQemuDeath(self): self.log.info('underlying process disconnected') - self._dom = None + self._dom = DisconnectedVirDomain(self.id) # Try release VM resources first, if failed stuck in 'Powering Down' # state result = self.releaseVm() @@ -1555,10 +1580,8 @@ if e.get_error_code() == libvirt.VIR_ERR_OPERATION_INVALID: return errCode['migCancelErr'] raise - except AttributeError: - if self._dom is None: - return errCode['migCancelErr'] - raise + except NotConnectedError: + return errCode['migCancelErr'] finally: self._guestCpuLock.release() @@ -1673,10 +1696,10 @@ def _isDomainRunning(self): try: status = self._dom.info() - except AttributeError: + except NotConnectedError: # Known reasons for this: # * on migration destination, and migration not yet completed. - # * self._dom may be set to None asynchronously (_onQemuDeath). + # * self._dom may be disconnected asynchronously (_onQemuDeath). # If so, the VM is shutting down or already shut down. return False else: @@ -1744,7 +1767,7 @@ pass raise Exception('destroy() called before Vm started') - if self._dom is None: + if not self._dom.connected: raise MissingLibvirtDomainError(vmexitreason.LIBVIRT_START_FAILED) self._updateDomainDescriptor() @@ -1852,7 +1875,7 @@ self._connection.lookupByUUIDString(self.id), self._timeoutExperienced) elif 'migrationDest' in self.conf: - pass # self._dom will be None until migration ends. + pass # self._dom will be disconnected until migration ends. elif 'restoreState' in self.conf: fromSnapshot = self.conf.get('restoreFromSnapshot', False) srcDomXML = self.conf.pop('_srcDomXML') @@ -2676,14 +2699,14 @@ """ try: state, details, stateTime = self._dom.controlInfo() - except AttributeError: + except NotConnectedError: # this method may be called asynchronously by periodic # operations. Thus, we must use a try/except block # to avoid racy checks. return False except libvirt.libvirtError as e: if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - # same as AttributeError above: possible race on shutdown + # same as NotConnectedError above: possible race on shutdown return False else: raise @@ -3615,7 +3638,7 @@ # Terminate the VM's creation thread. self._incomingMigrationFinished.set() self.guestAgent.stop() - if self._dom: + if self._dom.connected: result = self._destroyVmGraceful() if result['status']['code']: return result @@ -3707,7 +3730,7 @@ def setBalloonTarget(self, target): - if self._dom is None: + if not self._dom.connected: return self._reportError(key='balloonErr') try: target = int(target) @@ -4363,11 +4386,11 @@ # Libvirt sometimes send the SUSPENDED/SUSPENDED_PAUSED event # after RESUMED/RESUMED_MIGRATED (when VM status is PAUSED # when migration completes, see qemuMigrationFinish function). - # In this case self._dom is None because the function + # In this case self._dom is disconnected because the function # _completeIncomingMigration didn't update it yet. try: domxml = self._dom.XMLDesc(0) - except AttributeError: + except NotConnectedError: pass else: hooks.after_vm_pause(domxml, self.conf) @@ -4383,7 +4406,7 @@ # callback however we do not use it. try: domxml = self._dom.XMLDesc(0) - except AttributeError: + except NotConnectedError: pass else: hooks.after_vm_cont(domxml, self.conf) -- To view, visit https://gerrit.ovirt.org/43225 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5349ec51c7accf3b417b3bc9489c7eed5bfd8733 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 16

Change in vdsm[master]: lvm: Update hsm.__processVGInfos() with lvm.getPV() new API
by ishaby＠redhat.com 11 Aug '15

11 Aug '15

Idan Shaby has uploaded a new change for review. Change subject: lvm: Update hsm.__processVGInfos() with lvm.getPV() new API ...................................................................... lvm: Update hsm.__processVGInfos() with lvm.getPV() new API In change I9932b044c (lvm: Raise an exception if missing physical volume), we changed lvm.getPV() to raise an InaccessiblePhysDev exception when the device is no longer in lvm's cache. This patch updates hsm.__processVGInfos() to log a warning in such a case since if we called lvm.getPV() with a device that is no longer in lvm's cache, we still want to get the information for the other PVs. That is, we don't want to stop hsm.__processVGInfos() operation. Note that blovkSD.getMetaDataMapping() also calls lvm.getPV(), but in contrast to hsm.__processVGInfos(), we don't want blovkSD.getMetaDataMapping() to handle InaccessiblePhysDev because it's a part of a scenario that creates a storage domain, and we want to fail the whole operation in case that the PV is no longer in lvm's cache. Change-Id: I583c0493093d2c9c8bca8713df8ee123c415de7f Bug-Url: https://bugzilla.redhat.com/1048696 Signed-off-by: Idan Shaby <ishaby(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/38421/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index b92349c..bbc0f84 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3000,7 +3000,11 @@ vgType != dev["devtype"]): vgType = multipath.DEV_MIXED - pvInfo = lvm.getPV(pv) + try: + pvInfo = lvm.getPV(pv) + except se.InaccessiblePhysDev: + self.log.error("PV %s no longer exists", pv) + continue vgInfo['pvlist'].append(self.__fillPVDict(dev, pvInfo, vgType)) if vgType == multipath.DEV_FCP: -- To view, visit https://gerrit.ovirt.org/38421 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I583c0493093d2c9c8bca8713df8ee123c415de7f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Idan Shaby <ishaby(a)redhat.com>

7 18

Change in vdsm[master]: clientif: hoststats: move elapsedTime in clientIF
by fromani＠redhat.com 11 Aug '15

11 Aug '15

Francesco Romani has uploaded a new change for review. Change subject: clientif: hoststats: move elapsedTime in clientIF ...................................................................... clientif: hoststats: move elapsedTime in clientIF Make elapsedTime a clientIF property, moving it from HostStatsThread. This will help in making HostStatsThread a periodic operation. Change-Id: I92de260ef3988c9961ccff5a633037849a129a06 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/samplingTests.py M vdsm/API.py M vdsm/clientIF.py M vdsm/virt/sampling.py 4 files changed, 9 insertions(+), 9 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/33/42033/1 diff --git a/tests/samplingTests.py b/tests/samplingTests.py index 5370e73..229d3d4 100644 --- a/tests/samplingTests.py +++ b/tests/samplingTests.py @@ -25,7 +25,6 @@ import random import shutil import threading -import time from vdsm import ipwrapper from vdsm.password import ProtectedPassword @@ -260,11 +259,9 @@ 'cpuUserVdsmd': 0.0, 'rxRate': 0.0, 'txRate': 0.0, - 'elapsedTime': 0, } - with MonkeyPatchScope([(time, 'time', lambda: 0)]): - self._hs = sampling.HostStatsThread(self.log) - self.assertEquals(self._hs.get(), expected) + self._hs = sampling.HostStatsThread(self.log) + self.assertEquals(self._hs.get(), expected) def testSamplesWraparound(self): NUM = sampling.HostStatsThread.AVERAGING_WINDOW + 1 diff --git a/vdsm/API.py b/vdsm/API.py index 80586c7..397f87e 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -1317,7 +1317,8 @@ hooks.before_get_stats() stats = {} - decStats = self._cif._hostStats.get() + decStats = {'elapsedTime': self._cif.elapsed_time} + decStats.update(self._cif._hostStats.get()) if self._cif.irs: decStats['storageDomains'] = self._cif.irs.repoStats() diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index aee9aa0..c8ef6f6 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -96,6 +96,7 @@ self.vmContainer = {} self._hostStats = sampling.HostStatsThread(log=log) self._hostStats.start() + self._startTime = time.time() self.lastRemoteAccess = 0 self._enabled = True self._netConfigDirty = False @@ -131,6 +132,10 @@ return self.vmContainer.copy() @property + def elapsed_time(self): + return int(time.time() - self._startTime) + + @property def ready(self): return (self.irs is None or self.irs.ready) and not self._recovery diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index ffee99b..38c5892 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -552,8 +552,6 @@ _CONNLOG = logging.getLogger('connectivity') def __init__(self, log): - self.startTime = time.time() - threading.Thread.__init__(self) self.daemon = True self._log = log @@ -654,7 +652,6 @@ 'txRate': 0.0, # REQUIRED_FOR: engine < 3.6 'cpuSysVdsmd': 0.0, 'cpuUserVdsmd': 0.0, - 'elapsedTime': int(time.time() - self.startTime) } -- To view, visit https://gerrit.ovirt.org/42033 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I92de260ef3988c9961ccff5a633037849a129a06 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 9

Change in vdsm[master]: network: check for bridge changes only if unified persistenc...
by phoracek＠redhat.com 10 Aug '15

10 Aug '15

Petr Horáček has uploaded a new change for review. Change subject: network: check for bridge changes only if unified persistence is used ...................................................................... network: check for bridge changes only if unified persistence is used In _should_keep_bridge we use running configurator to check if we should keep bridge or remove it. Running configurator is available only when we use unified persistence. If we use ifcfg persistence, set keep_bridge always to False. Change-Id: I788e73a1691db5cb49ad24812b9b93c8d47b8927 Signed-off-by: Petr Horáček <phoracek(a)redhat.com> --- M vdsm/network/api.py 1 file changed, 12 insertions(+), 7 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/49/41349/1 diff --git a/vdsm/network/api.py b/vdsm/network/api.py index 6071f8d..dd55664 100755 --- a/vdsm/network/api.py +++ b/vdsm/network/api.py @@ -422,7 +422,7 @@ def _delNonVdsmNetwork(network, vlan, bonding, nics, _netinfo, configurator): if network in netinfo.bridges(): net_ent = _objectivizeNetwork(bridge=network, vlan=vlan, - bonding=bonding, nics=nics, + bonding=bonding, nics=nics, _netinfo=_netinfo, configurator=configurator, implicitBonding=False) @@ -838,12 +838,17 @@ for network, attrs in networks.items(): if network in _netinfo.networks: logger.debug("Removing network %r", network) - running_conf = configurator.runningConfig.networks.get(network) - keep_bridge = _should_keep_bridge( - network_attrs=attrs, - currently_bridged=_netinfo.networks[network]['bridged'], - network_running_config=running_conf - ) + if config.get('vars', 'net_configurator') == 'unified': + running_conf = configurator.runningConfig.networks.get( + network) + keep_bridge = _should_keep_bridge( + network_attrs=attrs, + currently_bridged=_netinfo.networks[network][ + 'bridged'], + network_running_config=running_conf + ) + else: + keep_bridge = False _delNetwork(network, configurator=configurator, force=force, implicitBonding=False, _netinfo=_netinfo, -- To view, visit https://gerrit.ovirt.org/41349 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I788e73a1691db5cb49ad24812b9b93c8d47b8927 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Petr Horáček <phoracek(a)redhat.com>

5 15

Change in vdsm[master]: virt: total rx/tx stats not send to engine
by mmirecki＠redhat.com 10 Aug '15

10 Aug '15

Marcin Mirecki has uploaded a new change for review. Change subject: virt: total rx/tx stats not send to engine ...................................................................... virt: total rx/tx stats not send to engine The network interface statistics for vm interfaces retrieved from libvirt were incorrectly parsed. The parsing was changed to parse the format in which the information is returned from libvirt Change-Id: I702a80f19553372e01ba022afaa07d766258062f Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1227793 Signed-off-by: Marcin Mirecki <mmirecki(a)redhat.com> --- M tests/vmTests.py M vdsm/virt/vmstats.py 2 files changed, 53 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/43672/1 diff --git a/tests/vmTests.py b/tests/vmTests.py index dfaa1bb..b2ede7c 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -1349,6 +1349,35 @@ testvm._setUnresponsiveIfTimeout(stats, 1) # any value < timeout self.assertEqual(stats['monitorResponse'], '-1') + def test_nic_sample(self): + sts = {'net.1.tx.drop': 0, 'net.3.tx.pkts': 7, 'net.1.rx.drop': 0, + 'net.1.tx.pkts': 0, 'net.0.tx.bytes': 0, 'net.0.name': 'vnet3', + 'net.1.rx.bytes': 0, 'net.2.tx.errs': 0, 'net.0.tx.pkts': 0, + 'net.1.tx.errs': 0, 'net.2.tx.pkts': 0, 'net.1.rx.errs': 0, + 'net.2.name': 'vnet5', 'net.0.rx.pkts': 0, 'net.3.tx.errs': 8, + 'net.0.rx.drop': 0, 'net.3.tx.bytes': 9, 'net.2.rx.bytes': 0, + 'net.2.rx.errs': 0, 'net.2.rx.pkts': 0, 'net.2.tx.drop': 0, + 'net.1.rx.pkts': 0, 'net.3.rx.drop': 11, 'net.1.name': 'vnet4', + 'net.1.tx.bytes': 0, 'net.3.rx.errs': 12, 'net.0.rx.errs': 0, + 'net.0.tx.drop': 0, 'net.0.tx.errs': 0, 'net.3.name': 'vnet6', + 'net.2.rx.drop': 0, 'net.0.rx.bytes': 0, 'vcpu.current': 1, + 'net.3.tx.drop': 5, 'net.3.rx.bytes': 3, 'net.3.rx.pkts': 4, + 'net.2.tx.bytes': 0} + result = vmstats.nic_sample(sts, 'vnet6') + self.assertEqual(result, {'rx.bytes': 3, + 'rx.pkts': 4, + 'rx.errs': 12, + 'rx.drop': 11, + 'tx.bytes': 9, + 'tx.pkts': 7, + 'tx.errs': 8, + 'tx.drop': 5}) + + stats = {'net.0.name': 'vnet3', 'net.1.name': 'vnet4', + 'net.3.name': 'vnet6'} + result = vmstats.nic_sample(stats, 'no_such_name') + self.assertEqual(result, {}) + class TestLibVirtCallbacks(TestCaseBase): FAKE_ERROR = 'EFAKERROR' diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index 53a909f..804efb9 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -180,8 +180,8 @@ if vm_nic.name.startswith('hostdev'): continue - first_nic = first_sample.get('net', {}).get(vm_nic.name, {}) - last_nic = last_sample.get('net', {}).get(vm_nic.name, {}) + first_nic = nic_sample(first_sample, vm_nic.name) + last_nic = nic_sample(last_sample, vm_nic.name) # may happen if nic is a new hot-plugged one if not first_nic or not last_nic: continue @@ -190,6 +190,28 @@ first_nic, last_nic, interval) +def nic_sample(stats, vm_nic_name): + sample = {} + ifc_count = -1 + while True: + ifc_count += 1 + net_name_key = 'net.' + str(ifc_count) + '.name' + if net_name_key not in stats: + break + net_name = stats[net_name_key] + if vm_nic_name != net_name: + continue + sample['rx.bytes'] = stats['net.' + str(ifc_count) + '.rx.bytes'] + sample['rx.pkts'] = stats['net.' + str(ifc_count) + '.rx.pkts'] + sample['rx.errs'] = stats['net.' + str(ifc_count) + '.rx.errs'] + sample['rx.drop'] = stats['net.' + str(ifc_count) + '.rx.drop'] + sample['tx.bytes'] = stats['net.' + str(ifc_count) + '.tx.bytes'] + sample['tx.pkts'] = stats['net.' + str(ifc_count) + '.tx.pkts'] + sample['tx.errs'] = stats['net.' + str(ifc_count) + '.tx.errs'] + sample['tx.drop'] = stats['net.' + str(ifc_count) + '.tx.drop'] + return sample + + def disks(vm, stats, first_sample, last_sample, interval): if first_sample is None or last_sample is None: return -- To view, visit https://gerrit.ovirt.org/43672 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I702a80f19553372e01ba022afaa07d766258062f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Marcin Mirecki <mmirecki(a)redhat.com>

3 5

Change in vdsm[master]: register: simplify check fqdn
by Douglas Schilling Landgraf 07 Aug '15

07 Aug '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: register: simplify check fqdn ...................................................................... register: simplify check fqdn Currently we use --check-fqdn (True or False) which is unneeded. We can simplify using --nocheck-fqdn. Change-Id: I106ee53d2b29c32f872e379bc535e10f616733c4 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/tool/register.py 1 file changed, 9 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/20/44220/1 diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py index 8e464a4..99b7a34 100644 --- a/lib/vdsm/tool/register.py +++ b/lib/vdsm/tool/register.py @@ -39,7 +39,7 @@ def __init__(self, engine_fqdn, engine_https_port=None, fingerprint=None, ssh_port=None, - ssh_user=None, check_fqdn=True, + ssh_user=None, check_fqdn=None, vdsm_port=None, node_fqdn=None, node_name=None): """ @@ -78,10 +78,7 @@ self.logger.debug("Engine https port: {hp}".format( hp=self.engine_port)) - if check_fqdn is None: - self.check_fqdn = True - else: - self.check_fqdn = tobool(check_fqdn) + self.check_fqdn = check_fqdn self.logger.debug("Check FQDN: {v}".format(v=self.check_fqdn)) self.fprint = fingerprint @@ -394,7 +391,7 @@ parser.add_argument( '--engine-fqdn', - help="Engine FQDN (See also: --check-fqdn)", + help="Engine FQDN (See also: --nocheck-fqdn)", required=True ) @@ -419,9 +416,11 @@ ) parser.add_argument( - '--check-fqdn', - help="Disable or Enable FQDN check for Engine CA, this option " - "is enabled by default (Use: True or False)", + '--nocheck-fqdn', + default=True, + action='store_false', + help="Disable FQDN check for Engine CA, this option " + "is enabled by default", ) parser.add_argument( @@ -448,7 +447,7 @@ ssh_user=args.ssh_user, ssh_port=args.ssh_port, fingerprint=args.fingerprint, - check_fqdn=args.check_fqdn) + check_fqdn=args.nocheck_fqdn) try: reg.handshake() -- To view, visit https://gerrit.ovirt.org/44220 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I106ee53d2b29c32f872e379bc535e10f616733c4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

3 9

Change in vdsm[master]: sampling: Handle numa nodes with zero memory assigned
by rmohr＠redhat.com 07 Aug '15

07 Aug '15

Roman Mohr has uploaded a new change for review. Change subject: sampling: Handle numa nodes with zero memory assigned ...................................................................... sampling: Handle numa nodes with zero memory assigned Fix zero division when a numa node has zero memory assigned and report a memory usage of 100% instead. One case where this can happen is when the server is not set up correcly. Bug-Url: https://bugzilla.redhat.com/1247058 Change-Id: I6be69e715e9b4a0cc8ded744b0bcc192a466c6c8 Signed-off-by: Roman Mohr <rmohr(a)redhat.com> --- M tests/samplingTests.py M vdsm/virt/sampling.py 2 files changed, 44 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/44121/1 diff --git a/tests/samplingTests.py b/tests/samplingTests.py index 433cb43..1410f4c 100644 --- a/tests/samplingTests.py +++ b/tests/samplingTests.py @@ -349,6 +349,43 @@ expected) +class NumaNodeMemorySampleTests(TestCaseBase): + def _monkeyPatchedMemorySample(self, freeMemory, totalMemory): + node_id, cpu_id = 0, 0 + + def fakeMemoryStats(): + return { + 'free': freeMemory, + 'total': totalMemory + } + + def fakeNumaTopology(): + return { + node_id: { + 'cpus': [cpu_id] + } + } + + return MonkeyPatchScope([(caps, 'getNumaTopology', + fakeNumaTopology), + (caps, 'getUMAHostMemoryStats', + fakeMemoryStats)]) + + def testMemoryStatsWithZeroMemory(self): + expected = {0: {'memPercent': 100, 'memFree': 0}} + + with self._monkeyPatchedMemorySample(freeMemory=0, totalMemory=0): + memorySample = sampling.NumaNodeMemorySample() + self.assertEqual(memorySample.nodesMemSample, expected) + + def testMemoryStats(self): + expected = {0: {'memPercent': 40, 'memFree': 600}} + + with self._monkeyPatchedMemorySample(freeMemory=600, totalMemory=1000): + memorySample = sampling.NumaNodeMemorySample() + self.assertEqual(memorySample.nodesMemSample, expected) + + class StatsCacheTests(TestCaseBase): FAKE_CLOCK_STEP = 1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index f17a5b3..4270256 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -179,8 +179,13 @@ else: memInfo = caps.getMemoryStatsByNumaCell(int(nodeIndex)) nodeMemSample['memFree'] = memInfo['free'] - nodeMemSample['memPercent'] = 100 - \ - int(100.0 * int(memInfo['free']) / int(memInfo['total'])) + # in case the numa node has zero memory assigned, report the whole + # memory as used + if memInfo['total'] == 0: + nodeMemSample['memPercent'] = 100 + else: + nodeMemSample['memPercent'] = 100 - \ + int(100.0 * int(memInfo['free']) / int(memInfo['total'])) self.nodesMemSample[nodeIndex] = nodeMemSample -- To view, visit https://gerrit.ovirt.org/44121 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6be69e715e9b4a0cc8ded744b0bcc192a466c6c8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Roman Mohr <rmohr(a)redhat.com>

7 25

Change in vdsm[master]: tcTests: include stdout and sterr in ExecError traceback
by Dan Kenigsberg 06 Aug '15

06 Aug '15

Dan Kenigsberg has uploaded a new change for review. Change subject: tcTests: include stdout and sterr in ExecError traceback ...................................................................... tcTests: include stdout and sterr in ExecError traceback Change-Id: I4668d86f3f03ab3f8def3d138b6482045ef2ef61 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M tests/tcTests.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/60/43960/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index a7fbc82..d0eac80 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -49,7 +49,7 @@ class ExecError(RuntimeError): def __init__(self, msg, out, err): - super(ExecError, self).__init__(msg) + super(ExecError, self).__init__(msg, out, err) self.out = out self.err = err -- To view, visit https://gerrit.ovirt.org/43960 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4668d86f3f03ab3f8def3d138b6482045ef2ef61 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

2 2

Change in vdsm[master]: logging: remove log messages which give little or no value
by Piotr Kliczewski 06 Aug '15

06 Aug '15

Piotr Kliczewski has uploaded a new change for review. Change subject: logging: remove log messages which give little or no value ...................................................................... logging: remove log messages which give little or no value Some of the messages logged during jsonrpc request processing give no or little value when diagnosing the issues but are logged quite often. This patch removes those messages. Change-Id: If5c6648f39b24adb1a5950c80a3883de0182b3b4 Signed-off-by: pkliczewski <piotr.kliczewski(a)gmail.com> Bug-Url: https://bugzilla.redhat.com/1239062 --- M lib/yajsonrpc/__init__.py M lib/yajsonrpc/stompreactor.py 2 files changed, 0 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/20/43720/1 diff --git a/lib/yajsonrpc/__init__.py b/lib/yajsonrpc/__init__.py index 68af242..778b9c0 100644 --- a/lib/yajsonrpc/__init__.py +++ b/lib/yajsonrpc/__init__.py @@ -530,7 +530,6 @@ @traceback(on=log.name) def serve_requests(self): while True: - self.log.debug("Waiting for request") obj = self._workQueue.get() if obj is None: break diff --git a/lib/yajsonrpc/stompreactor.py b/lib/yajsonrpc/stompreactor.py index e3b71bf..3ed451d 100644 --- a/lib/yajsonrpc/stompreactor.py +++ b/lib/yajsonrpc/stompreactor.py @@ -232,7 +232,6 @@ self._req_dest[request.get("id")] = req_dest def handle_frame(self, dispatcher, frame): - self.log.debug("Handling message %s", frame) try: self._commands[frame.command](dispatcher, frame) except KeyError: -- To view, visit https://gerrit.ovirt.org/43720 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If5c6648f39b24adb1a5950c80a3883de0182b3b4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>

6 8

Change in vdsm[master]: v2v: refactor create command
by shavivi＠redhat.com 06 Aug '15

06 Aug '15

Shahar Havivi has uploaded a new change for review. Change subject: v2v: refactor create command ...................................................................... v2v: refactor create command When other inputs will be introduced (such as files and ova) there is need to create a different input for virt-v2v command. This patch make _create_command as function variable and each new factory method will introduce its own _create_command. Change-Id: I4b9edabb2ec324693432ff71d2a8e682afbe3a35 Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M vdsm/v2v.py 1 file changed, 19 insertions(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/66/42066/1 diff --git a/vdsm/v2v.py b/vdsm/v2v.py index b7008fe..18cc9bc 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -283,12 +283,12 @@ TERM_DELAY = 30 PROC_WAIT_TIMEOUT = 30 - def __init__(self, vminfo, irs): + def __init__(self, vminfo, job_id, irs): ''' - Don't use this constructor for initialization, - use the factory methods instead. + do not use directly, use a factory method instead! ''' self._vminfo = vminfo + self._id = job_id self._irs = irs self._status = STATUS.STARTING @@ -299,14 +299,22 @@ self._aborted = False self._prepared_volumes = [] + self._uri = None + self._username = None + self._password = None + self._passwd_file = None + self._create_command = None + @classmethod - def from_libvirt(self, uri, username, password, vminfo, job_id, irs): - self._uri = uri - self._username = username - self._password = password - self._id = job_id - self._passwd_file = os.path.join(_V2V_DIR, "%s.tmp" % job_id) - return ImportVm(vminfo, irs) + def from_libvirt(cls, uri, username, password, vminfo, job_id, irs): + obj = cls(vminfo, job_id, irs) + + obj._uri = uri + obj._username = username + obj._password = password + obj._passwd_file = os.path.join(_V2V_DIR, "%s.tmp" % job_id) + obj._create_command = cls._from_libvirt_command + return obj def start(self): t = threading.Thread(target=self._run) @@ -406,7 +414,7 @@ raise RuntimeError("Job %r got unexpected parser event: %s" % (self._id, event)) - def _create_command(self): + def _from_libvirt_command(self): cmd = [_VIRT_V2V.cmd, '-ic', self._uri, '-o', 'vdsm', -- To view, visit https://gerrit.ovirt.org/42066 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4b9edabb2ec324693432ff71d2a8e682afbe3a35 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

4 18

Change in vdsm[master]: virt: Handle disconnects only if endpoints match
by Vinzenz Feenstra 06 Aug '15

06 Aug '15

Vinzenz Feenstra has uploaded a new change for review. Change subject: virt: Handle disconnects only if endpoints match ...................................................................... virt: Handle disconnects only if endpoints match Until now we used to handle every disconnect event with no respect of which remote endpoint caused it. There are currently 3 different types of graphic events: 1. Connected event someone connected to the port 2. Initialized event - Someone had the right credentials to establish the connection and the session has started 3. Disconnected event - Someone disconnected from the port Now any connected event can be triggered even with no credentials and would eventually trigger the disconnected event due to being rejected. This means that a host on the public internet which gets scanned for opened ports and tried to intrude them would trigger the disconnect handling for the affected VM. This can cause that somone connected via Spice, VNC or even RDP with the guest agent installed on the VM gets randomly a locked screen (default behaviour). This patch addresses the above mentioned issue by performing a comparison of the remote endpoint that disconnected and remembering only initiated endpoints. In case that multiple connections are established, like in case of spice, the last initiated channel would be remembered and when that one disconnects, it will cause the disconnect handling to proceed. Due to some bug in the handling of graphic events in libvirt, SPICE connections currently do not include the port. A fix for this has been posted upstream and should eventually become available. Therefore, the handling has been implemented to not ignore the disconnect event when the clientPort value is an empty string. Change-Id: Idfb3441407aa6e7bc31d37304bce0076e12bb1a3 Bug-Url: https://bugzilla.redhat.com/1203891 Signed-off-by: Vinzenz Feenstra <vfeenstr(a)redhat.com> --- M vdsm/clientIF.py M vdsm/virt/vm.py 2 files changed, 14 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/41/43141/1 diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index 1769a00..803fb78 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -594,9 +594,10 @@ 'authScheme %s subject %s', phase, localAddr, remoteAddr, authScheme, subject) if phase == libvirt.VIR_DOMAIN_EVENT_GRAPHICS_INITIALIZE: - v.onConnect(remoteAddr['node']) + v.onConnect(remoteAddr['node'], remoteAddr['service']) elif phase == libvirt.VIR_DOMAIN_EVENT_GRAPHICS_DISCONNECT: - v.onDisconnect() + v.onDisconnect(clientIp=remoteAddr['node'], + clientPort=remoteAddr['service']) elif eventid == libvirt.VIR_DOMAIN_EVENT_ID_WATCHDOG: action, = args[:-1] v._onWatchdogEvent(action) diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 826e4e9..06a7f0d 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -352,6 +352,7 @@ self._vcpuTuneInfo = {} self._numaInfo = {} self._vmJobs = None + self._clientPort = '' def _get_lastStatus(self): # note that we don't use _statusLock here. One of the reasons is the @@ -892,9 +893,10 @@ except Exception: self.log.exception("Reboot event failed") - def onConnect(self, clientIp=''): + def onConnect(self, clientIp='', clientPort=''): if clientIp: self.conf['clientIp'] = clientIp + self._clientPort = clientPort def _timedDesktopLock(self): # This is not a definite fix, we're aware that there is still the @@ -917,7 +919,14 @@ message='Scheduled shutdown on disconnect', force=True) - def onDisconnect(self, detail=None): + def onDisconnect(self, detail=None, clientIp='', clientPort=''): + if self.conf['clientIp'] != clientIp: + self.log.debug('Ignoring disconnect event because ip differs') + return + if self._clientPort and self._clientPort != clientPort: + self.log.debug('Ignoring disconnect event because ports differ') + return + self.conf['clientIp'] = '' # This is a hack to mitigate the issue of spice-gtk not respecting the # configured secure channels. Spice-gtk is always connecting first to -- To view, visit https://gerrit.ovirt.org/43141 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idfb3441407aa6e7bc31d37304bce0076e12bb1a3 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Vinzenz Feenstra <vfeenstr(a)redhat.com>

5 15

Change in vdsm[master]: Removing prefix definition for qemu-kvm-ev
by ybronhei＠redhat.com 06 Aug '15

06 Aug '15

Yaniv Bronhaim has uploaded a new change for review. Change subject: Removing prefix definition for qemu-kvm-ev ...................................................................... Removing prefix definition for qemu-kvm-ev qemu-kvm-ev (Centos) "provides" qemu-kvm-rhev, Since it's also the same version this patch simplies the requirements on both CentOS and RHEL platforms and keep plain qemu on Fedora only. Change-Id: I3335a58799eed2e4881a05f200aad96fa5d2af92 Signed-off-by: Yaniv Bronhaim <ybronhei(a)redhat.com> --- M vdsm.spec.in 1 file changed, 2 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/93/43993/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 4772416..bdd920e 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -208,17 +208,9 @@ Requires: systemd >= 197-1.fc18.2 %endif -%if !%{fedora_koji_build} -%if 0%{?rhev_build} -%global qemu_suffix -rhev -%else -%global qemu_suffix -ev -%endif # rhev_build -%endif # !%{fedora_koji_build} - %if 0%{?rhel} -Requires: qemu-kvm%{?qemu_suffix} >= 10:2.1.2-23.el7_1.2 -Requires: qemu-img%{?qemu_suffix} >= 10:2.1.2-23.el7_1.2 +Requires: qemu-kvm-rhev >= 10:2.1.2-23.el7_1.2 +Requires: qemu-img-rhev >= 10:2.1.2-23.el7_1.2 %else Requires: qemu-kvm >= 2:2.0.0-1 Requires: qemu-img >= 2:2.0.0-1 -- To view, visit https://gerrit.ovirt.org/43993 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3335a58799eed2e4881a05f200aad96fa5d2af92 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yaniv Bronhaim <ybronhei(a)redhat.com>

5 11

Change in vdsm[master]: utils: remove legacy code for uuid
by Douglas Schilling Landgraf 06 Aug '15

06 Aug '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: utils: remove legacy code for uuid ...................................................................... utils: remove legacy code for uuid In 3.6 version we won't support old registration schema. This patch removes the legacy code for collecting UUID from host. Change-Id: I1834eba15c7e5517a7aa62e1f32874244375036e Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/utils.py 1 file changed, 1 insertion(+), 42 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/41/44241/1 diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py index fccc286..f13e8a5 100644 --- a/lib/vdsm/utils.py +++ b/lib/vdsm/utils.py @@ -776,36 +776,10 @@ return False -def _getAllMacs(): - - # ( - # find /sys/class/net/*/device | while read f; do \ - # cat "$(dirname "$f")/address"; \ - # done; \ - # [ -d /proc/net/bonding ] && \ - # find /proc/net/bonding -type f -exec cat '{}' \; | \ - # grep 'Permanent HW addr:' | \ - # sed 's/.* //' - # ) | sed -e '/00:00:00:00/d' -e '/^$/d' - - macs = [] - for b in glob.glob('/sys/class/net/*/device'): - with open(os.path.join(os.path.dirname(b), "address")) as a: - mac = a.readline().replace("\n", "") - macs.append(mac) - - for b in glob.glob('/proc/net/bonding/*'): - with open(b) as bond: - for line in bond: - if line.startswith("Permanent HW addr: "): - macs.append(line.split(": ")[1].replace("\n", "")) - - return set(macs) - set(["", "00:00:00:00:00:00"]) - __hostUUID = None -def getHostUUID(legacy=True): +def getHostUUID(legacy=False): global __hostUUID if __hostUUID: return __hostUUID @@ -841,24 +815,9 @@ except IOError: logging.warning('Could not find host UUID.') - if legacy: - try: - mac = sorted(_getAllMacs())[0] - except: - mac = "" - logging.warning('Could not find host MAC.', exc_info=True) - - # __hostUUID might contain the string 'None' returned - # from dmidecode call - if __hostUUID and __hostUUID is not 'None': - __hostUUID += "_" + mac - else: - __hostUUID = "_" + mac except: logging.error("Error retrieving host UUID", exc_info=True) - if legacy and not __hostUUID: - return 'None' return __hostUUID symbolerror = {} -- To view, visit https://gerrit.ovirt.org/44241 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1834eba15c7e5517a7aa62e1f32874244375036e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

4 10

Change in vdsm[master]: fakevmstats: add config entry to enable/disable the hook
by Martin Polednik 06 Aug '15

06 Aug '15

Martin Polednik has uploaded a new change for review. Change subject: fakevmstats: add config entry to enable/disable the hook ...................................................................... fakevmstats: add config entry to enable/disable the hook Installing fake vmstats hook cases it to be automatically enabled, causing fake stats to be reported. This patch adds configuration directive to control the behavior, and disables the plugin by default. Change-Id: I6949445d8b98d5f4cd4f441362338791dfa664c1 Signed-off-by: Martin Polednik <mpolednik(a)redhat.com> --- M lib/vdsm/config.py.in M vdsm_hooks/fakevmstats/after_get_all_vm_stats.py 2 files changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/31/43931/1 diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in index 3221aab..5a450e5 100644 --- a/lib/vdsm/config.py.in +++ b/lib/vdsm/config.py.in @@ -175,6 +175,8 @@ ('fake_kvm_memory', '0', 'Set memory of fake KVM hosts. ' 'Set to \'0\' to use supplied memory value'), + ('fake_vmstats_enable', 'false', 'Enable reporting of fake VM stats.), + ('xmlrpc_enable', 'true', 'Enable the xmlrpc server'), ('xmlrpc_http11', 'true', diff --git a/vdsm_hooks/fakevmstats/after_get_all_vm_stats.py b/vdsm_hooks/fakevmstats/after_get_all_vm_stats.py index 3315c90..5dd4578 100644 --- a/vdsm_hooks/fakevmstats/after_get_all_vm_stats.py +++ b/vdsm_hooks/fakevmstats/after_get_all_vm_stats.py @@ -21,6 +21,8 @@ import hooking import random +from vdsm.config import config + QUARTER_GB = 1024*1024*256 GB = QUARTER_GB * 4 @@ -177,7 +179,8 @@ if __name__ == '__main__': - statsList = hooking.read_json() - for stats in statsList: - randomizeRuntimeStats(stats) - hooking.write_json(statsList) + if config.getboolean('vars', 'fake_vmstats_enable'): + statsList = hooking.read_json() + for stats in statsList: + randomizeRuntimeStats(stats) + hooking.write_json(statsList) -- To view, visit https://gerrit.ovirt.org/43931 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6949445d8b98d5f4cd4f441362338791dfa664c1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

7 19

Change in vdsm[master]: tests: Add requires_brctl and requries_tc decorators
by Nir Soffer 06 Aug '15

06 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Add requires_brctl and requries_tc decorators ...................................................................... tests: Add requires_brctl and requries_tc decorators Lot of tests used nettestlib.checkDependencies() in setUp() to skip all tests if dependencies are not met. However, this function was checking for two features, but most tests only need one of them, so tests could be skipped for no reason. Now we new specific requires decorators: - @requries_brctl raise SkipTest if running brctl fails - @requires_tc raise SkipTest if creating a bridge and adding a qdisc fails. nettestlib.checkDependencies() was creating a bridge and deleting it for testing if we can run brctl. This could leave a stale bridge if deleting the device failed. Now we use safer "brctl show". Each test class is using now the specific requires decorator needed. Change-Id: I9211cb491b1b6f5e7a8880559ec1e1ca3804e92a Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/iproute2Tests.py M tests/ipwrapperTests.py M tests/nettestlib.py M tests/tcTests.py 4 files changed, 37 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/18/44218/1 diff --git a/tests/iproute2Tests.py b/tests/iproute2Tests.py index 518f350..86818d6 100644 --- a/tests/iproute2Tests.py +++ b/tests/iproute2Tests.py @@ -26,15 +26,15 @@ from vdsm.constants import EXT_BRCTL from network.configurators.iproute2 import _BRCTL_DEV_EXISTS -from nettestlib import Bridge, checkDependencies +from nettestlib import Bridge, requires_brctl class TestBridgeOverwrite(TestCaseBase): _bridge = Bridge() @ValidateRunningAsRoot + @requires_brctl def setUp(self): - checkDependencies() self._bridge.addDevice() def tearDown(self): diff --git a/tests/ipwrapperTests.py b/tests/ipwrapperTests.py index bcef590..e99280d 100644 --- a/tests/ipwrapperTests.py +++ b/tests/ipwrapperTests.py @@ -23,7 +23,7 @@ from vdsm.ipwrapper import Route from vdsm.ipwrapper import Rule -from nettestlib import Bridge, checkDependencies +from nettestlib import Bridge, requires_brctl from testlib import VdsmTestCase as TestCaseBase @@ -94,8 +94,8 @@ class TestLinks(TestCaseBase): @ValidateRunningAsRoot + @requires_brctl def setUp(self): - checkDependencies() self._bridge = Bridge() self._bridge.addDevice() @@ -112,8 +112,8 @@ class TestDrvinfo(TestCaseBase): @ValidateRunningAsRoot + @requires_brctl def setUp(self): - checkDependencies() self._bridge = Bridge() self._bridge.addDevice() @@ -139,8 +139,8 @@ class TestUnicodeDrvinfo(TestCaseBase): @ValidateRunningAsRoot + @requires_brctl def setUp(self): - checkDependencies() # First 3 Hebrew letters # See http://unicode.org/charts/PDF/U0590.pdf self._bridge = Bridge("\xd7\x90\xd7\x91\xd7\x92") diff --git a/tests/nettestlib.py b/tests/nettestlib.py index ab7a8d5..51bafc4 100644 --- a/tests/nettestlib.py +++ b/tests/nettestlib.py @@ -19,6 +19,7 @@ import errno import fcntl +import functools import os import platform import signal @@ -142,7 +143,25 @@ os.write(self._cloneDevice.fileno(), icmp) -def checkDependencies(): +def check_brctl(): + try: + execCmd([EXT_BRCTL, "show"]) + except OSError as e: + if e.errno == errno.ENOENT: + raise SkipTest("Cannot run %r: %s\nDo you have bridge-utils " + "installed?" % (EXT_BRCTL, e)) + raise + + +def requires_brctl(f): + @functools.wraps(f) + def wrapper(*a, **kw): + check_brctl() + return f(*a, **kw) + return wrapper + + +def check_tc(): dev = Bridge() try: dev.addDevice() @@ -159,3 +178,11 @@ "modules installed?" % (EXT_TC, e.err)) finally: dev.delDevice() + + +def requires_tc(f): + @functools.wraps(f) + def wrapper(*a, **kw): + check_tc() + return f(*a, **kw) + return wrapper diff --git a/tests/tcTests.py b/tests/tcTests.py index 8c894ba..d78e8e5 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -30,7 +30,7 @@ from testlib import VdsmTestCase as TestCaseBase from testValidation import ValidateRunningAsRoot -from nettestlib import Bridge, Tap, checkDependencies +from nettestlib import Bridge, Tap, requires_brctl, requires_tc from vdsm.constants import EXT_TC from network import tc @@ -39,8 +39,8 @@ class TestQdisc(TestCaseBase): @ValidateRunningAsRoot + @requires_tc def setUp(self): - checkDependencies() self._bridge = Bridge() self._bridge.addDevice() @@ -286,8 +286,8 @@ '00123456789') # Payload @ValidateRunningAsRoot + @requires_brctl def setUp(self): - checkDependencies() self._tap0 = Tap() self._tap1 = Tap() self._tap2 = Tap() -- To view, visit https://gerrit.ovirt.org/44218 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9211cb491b1b6f5e7a8880559ec1e1ca3804e92a Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 15

Change in vdsm[master]: nettestlib: Remove unneeded hack for improving randomness
by Nir Soffer 06 Aug '15

06 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: nettestlib: Remove unneeded hack for improving randomness ...................................................................... nettestlib: Remove unneeded hack for improving randomness Python seed the random number generator from os.urandom() on startup, so different processes do not use the same seed, and produce different sequences of pseudo random numbers by default. Change-Id: I631ec8572bb6a69ee086011d67daf7ed000d718b Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/nettestlib.py 1 file changed, 0 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/16/44216/1 diff --git a/tests/nettestlib.py b/tests/nettestlib.py index 44a831d..ab7a8d5 100644 --- a/tests/nettestlib.py +++ b/tests/nettestlib.py @@ -21,7 +21,6 @@ import fcntl import os import platform -import random import signal import struct from multiprocessing import Process @@ -144,10 +143,6 @@ def checkDependencies(): - - # hack to avoid name collision in concurrently-running mock-based tests - random.jumpahead(os.getpid()) - dev = Bridge() try: dev.addDevice() -- To view, visit https://gerrit.ovirt.org/44216 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I631ec8572bb6a69ee086011d67daf7ed000d718b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 13

Change in vdsm[master]: network: fix parameters to ConfigNetworkError
by osvoboda＠redhat.com 06 Aug '15

06 Aug '15

Ondřej Svoboda has uploaded a new change for review. Change subject: network: fix parameters to ConfigNetworkError ...................................................................... network: fix parameters to ConfigNetworkError Change-Id: I2358521cbe8c39cb2208c8a3f9f96313dd693c7e Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm/network/configurators/__init__.py 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/72/44172/1 diff --git a/vdsm/network/configurators/__init__.py b/vdsm/network/configurators/__init__.py index e51d3ed..b7ba018 100644 --- a/vdsm/network/configurators/__init__.py +++ b/vdsm/network/configurators/__init__.py @@ -176,5 +176,4 @@ dhclient = DhcpClient(iface.name, family) rc = dhclient.start(iface.blockingdhcp) if iface.blockingdhcp and rc: - raise ConfigNetworkError(ERR_FAILED_IFUP, 'dhclient%s failed', - family) + raise ConfigNetworkError(ERR_FAILED_IFUP, 'dhclient%s failed' % family) -- To view, visit https://gerrit.ovirt.org/44172 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2358521cbe8c39cb2208c8a3f9f96313dd693c7e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

4 7

Change in vdsm[master]: network: wait for a bridge to appear before disabling IPv6 o...
by osvoboda＠redhat.com 06 Aug '15

06 Aug '15

Ondřej Svoboda has uploaded a new change for review. Change subject: network: wait for a bridge to appear before disabling IPv6 on it ...................................................................... network: wait for a bridge to appear before disabling IPv6 on it A traceback such as File "/usr/share/vdsm/network/configurators/ifcfg.py", line 101, in configureBridge sysctl.disable_ipv6(bridge.name) File "/usr/lib/python2.7/site-packages/vdsm/sysctl.py", line 42, in disable_ipv6 with open('/proc/sys/net/ipv6/conf/%s/disable_ipv6' % dev, 'w') as f: IOError: [Errno 2] No such file or directory: u'/proc/sys/net/ipv6/conf/test-network/disable_ipv6' shows that a bridge was not yet created when _ifup in ifcfg's configureBridge already finished. This patch makes configureBridge wait for the bridge for up to 1s. If it is not created by then, a standard exception is triggered. Change-Id: Ic5292e5a297ff50a0e82ba0781333e21fd9dd7f3 Signed-off-by: Ondřej Svoboda <osvoboda(a)redhat.com> --- M vdsm/network/configurators/__init__.py M vdsm/network/configurators/ifcfg.py M vdsm/network/configurators/iproute2.py 3 files changed, 22 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/83/43583/1 diff --git a/vdsm/network/configurators/__init__.py b/vdsm/network/configurators/__init__.py index e51d3ed..6a5b484 100644 --- a/vdsm/network/configurators/__init__.py +++ b/vdsm/network/configurators/__init__.py @@ -24,6 +24,7 @@ from vdsm import netinfo from vdsm.config import config from vdsm.netconfpersistence import RunningConfig +from vdsm.netlink import monitor from .dhclient import DhcpClient from ..errors import ConfigNetworkError, ERR_FAILED_IFUP @@ -178,3 +179,19 @@ if iface.blockingdhcp and rc: raise ConfigNetworkError(ERR_FAILED_IFUP, 'dhclient%s failed', family) + + +def wait_for_bridge(name, timeout=1): + """ + Wait for a bridge to appear in a given timeout. If the bridge is not + created by then, raise a ConfigNetworkError. + """ + with monitor.Monitor(timeout=timeout, groups=('link',), + silent_timeout=True) as mon: + if name in netinfo.bridges(): + return + for event in mon: + if event.get('name') == name and event.get('event') == 'new_link': + return + raise ConfigNetworkError(ERR_FAILED_IFUP, 'Bridge %s was not created ' + 'during a %ss timeout.' % (name, timeout)) diff --git a/vdsm/network/configurators/ifcfg.py b/vdsm/network/configurators/ifcfg.py index e9ef7cc..b42e9b3 100644 --- a/vdsm/network/configurators/ifcfg.py +++ b/vdsm/network/configurators/ifcfg.py @@ -44,7 +44,7 @@ if utils.isOvirtNode(): from ovirt.node.utils import fs as node_fs -from . import Configurator, dhclient, getEthtoolOpts, libvirt +from . import Configurator, dhclient, getEthtoolOpts, libvirt, wait_for_bridge from ..errors import ConfigNetworkError, ERR_FAILED_IFUP from ..models import Nic, Bridge, IPv4, IPv6 from ..sourceroute import StaticSourceRoute, DynamicSourceRoute @@ -98,6 +98,7 @@ _ifup(bridge) if not bridge.ipv6.address and not bridge.ipv6.ipv6autoconf and ( not bridge.ipv6.dhcpv6): + wait_for_bridge(bridge.name) sysctl.disable_ipv6(bridge.name) def configureVlan(self, vlan, **opts): diff --git a/vdsm/network/configurators/iproute2.py b/vdsm/network/configurators/iproute2.py index 0b5b73e..ef36766 100644 --- a/vdsm/network/configurators/iproute2.py +++ b/vdsm/network/configurators/iproute2.py @@ -29,7 +29,8 @@ from vdsm.utils import CommandPath from vdsm.utils import execCmd -from . import Configurator, runDhclient, getEthtoolOpts, libvirt +from . import (Configurator, runDhclient, getEthtoolOpts, libvirt, + wait_for_bridge) from .dhclient import DhcpClient from ..errors import ConfigNetworkError, ERR_FAILED_IFUP, ERR_FAILED_IFDOWN from ..models import Nic @@ -75,6 +76,7 @@ self.configApplier.setIfaceConfigAndUp(bridge) if not bridge.ipv6.address and not bridge.ipv6.ipv6autoconf and ( not bridge.ipv6.dhcpv6): + wait_for_bridge(bridge.name) sysctl.disable_ipv6(bridge.name) self._addSourceRoute(bridge) if 'custom' in opts and 'bridge_opts' in opts['custom']: -- To view, visit https://gerrit.ovirt.org/43583 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic5292e5a297ff50a0e82ba0781333e21fd9dd7f3 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ondřej Svoboda <osvoboda(a)redhat.com>

5 16

Change in vdsm[master]: Live Merge: Allow extension of non-leaf raw volumes
by alitke＠redhat.com 05 Aug '15

05 Aug '15

Adam Litke has uploaded a new change for review. Change subject: Live Merge: Allow extension of non-leaf raw volumes ...................................................................... Live Merge: Allow extension of non-leaf raw volumes volume.extendSize() is currently prohibited for any non-leaf volume. For a very specific live merge scenario we must permit extension of an internal raw base volume. Allow this usage and add a comment explaining the reasoning. The scenario: - User begins with a raw block disk. - User creates a snapshot. - User enlarges the disk (diskSizeExtend) - User performs live merge to remove the snapshot In this case the base volume is too small to accommodate the data from the child volume and an error is raised since libvirt cannot enlarge a block device. The solution is to require engine to call extendVolumeSize on the base volume before requesting the live merge operation. Change-Id: I82a73356a863abe39d367cdc70a9b1e914f75aee Signed-off-by: Adam Litke <alitke(a)redhat.com> Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1232481 --- M vdsm/storage/volume.py 1 file changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/07/43407/1 diff --git a/vdsm/storage/volume.py b/vdsm/storage/volume.py index 980b6e3..1fa1571 100644 --- a/vdsm/storage/volume.py +++ b/vdsm/storage/volume.py @@ -548,10 +548,16 @@ """ Extend the size (virtual disk size seen by the guest) of the volume. """ - if not self.isLeaf() or self.isShared(): + if self.isShared(): raise se.VolumeNonWritable(self.volUUID) + # Note: This function previously prohibited extending non-leaf volumes. + # If a disk is enlarged a volume may become larger than its parent. In + # order to support live merge of a larger volume into its raw parent we + # must permit extension of this raw volume prior to starting the merge. volFormat = self.getFormat() + if volFormat != RAW_FORMAT and not self.isLeaf(): + raise se.VolumeNonWritable(self.volUUID) if volFormat == COW_FORMAT: self.log.debug("skipping cow size extension for volume %s to " -- To view, visit https://gerrit.ovirt.org/43407 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I82a73356a863abe39d367cdc70a9b1e914f75aee Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 16

Change in vdsm[master]: Live merge: Update base size after live merge
by alitke＠redhat.com 05 Aug '15

05 Aug '15

Adam Litke has uploaded a new change for review. Change subject: Live merge: Update base size after live merge ...................................................................... Live merge: Update base size after live merge When performing a live merge, data is copied from a top volume into a base volume. If the top volume is larger than the base volume (which can happen if the drive size was extended), libvirt will change the size of the base volume to match that of the top volume. When synchronizing metadata after the merge, we need to update the 'capacity' field of the base volume to reflect the new size. We do this inside the LiveMergeCleanupThread to ensure that it gets retried in the event of storage connection problems or vdsm restarts. Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1232481 Change-Id: Iae354de36db63ae3bf4b4fc7f72df5e306035784 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 37 insertions(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/42921/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 549cd38..05ff99e 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -4510,7 +4510,7 @@ def queryBlockJobs(self): def startCleanup(job, drive, needPivot): - t = LiveMergeCleanupThread(self, job['jobID'], drive, needPivot) + t = LiveMergeCleanupThread(self, job, drive, needPivot) t.start() self._liveMergeCleanupThreads[job['jobID']] = t @@ -4867,6 +4867,15 @@ (domainID, volumeID)) return VolumeSize(int(res['apparentsize']), int(res['truesize'])) + def _getVolumeCapacity(self, domainID, poolID, imageID, volumeID): + """Return the volume capacity by accessing storage""" + res = self.cif.irs.getVolumeInfo(domainID, poolID, imageID, volumeID) + if res['status']['code'] != 0: + raise StorageUnavailableError( + "Unable to get volume capacity for domain %s volume %s" % + (domainID, volumeID)) + return int(res['info']['capacity']) + def _setVolumeSize(self, domainID, poolID, imageID, volumeID, size): res = self.cif.irs.setVolumeSize(domainID, poolID, imageID, volumeID, size) @@ -4877,11 +4886,11 @@ class LiveMergeCleanupThread(threading.Thread): - def __init__(self, vm, jobId, drive, doPivot): + def __init__(self, vm, job, drive, doPivot): threading.Thread.__init__(self) self.setDaemon(True) self.vm = vm - self.jobId = jobId + self.job = job self.drive = drive self.doPivot = doPivot self.success = False @@ -4905,7 +4914,7 @@ self.vm.stopDisksStatsCollection() self.vm.log.info("Requesting pivot to complete active layer commit " - "(job %s)", self.jobId) + "(job %s)", self.job['jobID']) try: flags = libvirt.VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT ret = self.vm._dom.blockJobAbort(self.drive.name, flags) @@ -4915,22 +4924,38 @@ else: if ret != 0: self.vm.log.error("Pivot failed for job %s (rc=%i)", - self.jobId, ret) + self.job['jobID'], ret) raise RuntimeError("pivot failed") self._waitForXMLUpdate() - self.vm.log.info("Pivot completed (job %s)", self.jobId) + self.vm.log.info("Pivot completed (job %s)", self.job['jobID']) + + def update_base_size(self): + # If the drive size was extended just after creating the snapshot which + # we are removing, the size of the top volume might be larger than the + # size of the base volume. In that case libvirt has enlarged the base + # volume automatically as part of the blockCommit operation. Update + # our metadata to reflect this change. + capacity = self.vm._getVolumeCapacity(self.drive.domainID, + self.drive.poolID, + self.drive.imageID, + self.job['topVolume']) + self.vm._setVolumeSize(self.drive.domainID, self.drive.poolID, + self.drive.imageID, self.job['baseVolume'], + capacity) @utils.traceback() def run(self): + self.update_base_size() if self.doPivot: self.tryPivot() self.vm.log.info("Synchronizing volume chain after live merge " - "(job %s)", self.jobId) + "(job %s)", self.job['jobID']) self.vm._syncVolumeChain(self.drive) if self.doPivot: self.vm.startDisksStatsCollection() self.success = True - self.vm.log.info("Synchronization completed (job %s)", self.jobId) + self.vm.log.info("Synchronization completed (job %s)", + self.job['jobID']) def isSuccessful(self): """ @@ -4976,9 +5001,10 @@ self.vm.log.info("The XML update has been completed") break else: - self.log.error("Bad volume chain found for drive %s. Previous " - "chain: %s, Expected chain: %s, Actual chain: " - "%s", alias, origVols, expectedVols, curVols) + self.vm.log.error("Bad volume chain found for drive %s. " + "Previous chain: %s, Expected chain: %s, " + "Actual chain: %s", alias, origVols, + expectedVols, curVols) raise RuntimeError("Bad volume chain found") -- To view, visit https://gerrit.ovirt.org/42921 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iae354de36db63ae3bf4b4fc7f72df5e306035784 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 17

Change in vdsm[master]: fc-connect-server: Support FCP on connect server
by frolland＠redhat.com 05 Aug '15

05 Aug '15

Hello Fred Rolland, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/44011 to review the following change. Change subject: fc-connect-server: Support FCP on connect server ...................................................................... fc-connect-server: Support FCP on connect server Add FCP support in in hsm._connectionDict2ConnectionInfo. Change-Id: I408d8364278a1a502fc94a2e6537cb160c716ff1 Bug-Url: https://bugzilla.redhat.com/1242200 Signed-off-by: Fred Rolland <frolland(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/44011/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index f22d8d7..a79b7ee 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -116,9 +116,7 @@ sd.LOCALFS_DOMAIN: 'localfs', sd.NFS_DOMAIN: 'nfs', sd.ISCSI_DOMAIN: 'iscsi', - # FCP domain shouldn't even be on the list but VDSM use to just - # accept this type as iscsi so we are stuck with it - sd.FCP_DOMAIN: 'iscsi', + sd.FCP_DOMAIN: 'fcp', sd.POSIXFS_DOMAIN: 'posixfs', sd.GLUSTERFS_DOMAIN: 'glusterfs'} @@ -241,6 +239,8 @@ cred = iscsi.ChapCredentials(username, password) params = storageServer.IscsiConnectionParameters(target, iface, cred) + elif typeName == 'fcp': + params = storageServer.FcpConnectionParameters('fcp') else: raise se.StorageServerActionError() -- To view, visit https://gerrit.ovirt.org/44011 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I408d8364278a1a502fc94a2e6537cb160c716ff1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Freddy Rolland <frolland(a)redhat.com> Gerrit-Reviewer: Fred Rolland <frolland(a)redhat.com>

4 13

Change in vdsm[master]: fc-connect-server: Add FcpConnection class
by frolland＠redhat.com 05 Aug '15

05 Aug '15

Hello Fred Rolland, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/44010 to review the following change. Change subject: fc-connect-server: Add FcpConnection class ...................................................................... fc-connect-server: Add FcpConnection class Add FcpConnection class in storageServer.py. All methods except eq and hash are not implemented. This class permit the engine to call connectStorageServer on FC storage domain so that sdCache.refreshStorage and prefetchDomains will be performed on FC SD also. Change-Id: I0a2c937cb997244df7910fc7cfcae11b088d3cdb Bug-Url: https://bugzilla.redhat.com/1242200 Signed-off-by: Fred Rolland <frolland(a)redhat.com> --- M vdsm/storage/storageServer.py 1 file changed, 23 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/44010/1 diff --git a/vdsm/storage/storageServer.py b/vdsm/storage/storageServer.py index e71b88c..6444d2a 100644 --- a/vdsm/storage/storageServer.py +++ b/vdsm/storage/storageServer.py @@ -69,6 +69,8 @@ "export, retrans, timeout, version, " "extraOptions") +FcpConnectionParameters = namedtuple("FcpConnectionParameters", "type") + ConnectionInfo = namedtuple("ConnectionInfo", "type, params") @@ -548,6 +550,26 @@ return hsh +class FcpConnection(object): + def __init__(self, type): + pass + + def connect(self): + pass + + def disconnect(self): + pass + + def isConnected(self): + pass + + def __eq__(self, other): + return self.__class__ == other.__class + + def __hash__(self): + return hash(type(self)) + + class LocalDirectoryConnection(object): @property def path(self): @@ -700,6 +722,7 @@ "glusterfs": GlusterFSConnection, "iscsi": IscsiConnection, "localfs": LocalDirectoryConnection, + "fcp": FcpConnection, } @classmethod -- To view, visit https://gerrit.ovirt.org/44010 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0a2c937cb997244df7910fc7cfcae11b088d3cdb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Freddy Rolland <frolland(a)redhat.com> Gerrit-Reviewer: Fred Rolland <frolland(a)redhat.com>

4 13

Change in vdsm[master]: fc-connect-server: Move call to refreshStorage
by frolland＠redhat.com 05 Aug '15

05 Aug '15

Hello Fred Rolland, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/44009 to review the following change. Change subject: fc-connect-server: Move call to refreshStorage ...................................................................... fc-connect-server: Move call to refreshStorage For clearer readability, move call to sdCache.refreshStorage from __prefetchDomains to connectStorageServer. Change-Id: If8ac84cc2bfd3490f3da41f0bc79fc372495a5a7 Bug-Url: https://bugzilla.redhat.com/1242200 Signed-off-by: Fred Rolland <frolland(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/44009/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index f68d3bb..f22d8d7 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -2388,7 +2388,6 @@ uuidPatern = "????????-????-????-????-????????????" if domType in (sd.FCP_DOMAIN, sd.ISCSI_DOMAIN): - sdCache.refreshStorage() uuids = tuple(blockSD.getStorageDomainsList()) elif domType is sd.NFS_DOMAIN: lPath = conObj._mountCon._getLocalPath() @@ -2457,6 +2456,11 @@ else: status = 0 try: + # In case there were changes in devices size + # while the VDSM was not connected, we need to + # call refreshStorage. + if domType in (sd.FCP_DOMAIN, sd.ISCSI_DOMAIN): + sdCache.refreshStorage() doms = self.__prefetchDomains(domType, conObj) except: self.log.debug("prefetch failed: %s", -- To view, visit https://gerrit.ovirt.org/44009 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If8ac84cc2bfd3490f3da41f0bc79fc372495a5a7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Freddy Rolland <frolland(a)redhat.com> Gerrit-Reviewer: Fred Rolland <frolland(a)redhat.com>

4 10

Change in vdsm[master]: Live Merge: Prevent merge when base volume is too small
by alitke＠redhat.com 05 Aug '15

05 Aug '15

Adam Litke has uploaded a new change for review. Change subject: Live Merge: Prevent merge when base volume is too small ...................................................................... Live Merge: Prevent merge when base volume is too small When a disk with snapshots is resized you can end up with an image chain containing volumes of different sizes. For example: a VM may start with volume A, a snapshot operation causes volume B to be added to the chain, and a resize operation enlarges volume B. This mixed-size chain presents a special case for live merge. Libvirt's virDomainBlockCommit behaves differently depending on whether the base volume is on block storage or file storage and whether the base volume is raw or qcow2: File volume, cow format: - Update qcow header and truncate volume file to the new size File volume, raw format: - Truncate volume file to the new size Block volume, cow format: - Update qcow header only (volume size is managed by vdsm) Block volume, raw format: - Fail because qemu cannot extend a raw block device Since we know the raw block case is not possible we can detect this and return a special error message to engine. To avoid this error, engine should use the SPM host to extend the raw volume to the correct size before starting the merge. We don't _have_ to change vdsm since the current code fails gracefully (albeit with a traceback in the logs). If engine always performs the resize operation before calling merge we'll never see this error. Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1232481 Change-Id: Ibf77a7c5108b500c6ec34653ef7570a841def1b4 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M lib/vdsm/define.py M vdsm/virt/vm.py 2 files changed, 38 insertions(+), 19 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/36/42836/1 diff --git a/lib/vdsm/define.py b/lib/vdsm/define.py index 284b10f..7c5f7b0 100644 --- a/lib/vdsm/define.py +++ b/lib/vdsm/define.py @@ -169,6 +169,9 @@ 'ksmErr': {'status': { 'code': 71, 'message': 'Failed to update KSM values'}}, + 'destVolumeTooSmall': {'status': { + 'code': 72, + 'message': 'Destination volume is too small'}}, 'recovery': {'status': { 'code': 99, 'message': 'Recovering from crash or Initializing'}}, diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 4977d28..549cd38 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -4570,7 +4570,30 @@ jobsRet[jobID] = entry return jobsRet + def _merge_check_base_size(self, drive, base_info, top_info): + # If the drive waa resized the top volume could be larger than the + # base volume. Libvirt can handle this situation for filw-based + # volumes and block qcow volumes (where extension happens dynamically). + # Raw block volumes cannot be extended by libvirt so we require ovirt + # engine to extend them before calling merge. Check here. + if not drive.blockDev or base_info['format'] != 'RAW': + return True + + if int(base_info['capacity']) < int(top_info['capacity']): + self.log.error("merge: The base volume is undersized and cannot " + "be extended. Aborting.") + return False + return True + def merge(self, driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID): + def get_volume_info(drive, volUUID): + res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, + drive.imageID, volUUID) + if res['status']['code'] != 0: + self.log.error("Unable to get volume info for '%s'", volUUID) + raise LookupError(volUUID) + return res['info'] + if not caps.getLiveMergeSupport(): self.log.error("Live merge is not supported on this host") return errCode['mergeErr'] @@ -4604,14 +4627,15 @@ self.log.error("merge: top volume '%s' not found", topVolUUID) return errCode['mergeErr'] + try: + baseInfo = get_volume_info(drive, baseVolUUID) + topInfo = get_volume_info(drive, topVolUUID) + except LookupError: + return errCode['mergeErr'] + # If base is a shared volume then we cannot allow a merge. Otherwise # We'd corrupt the shared volume for other users. - res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, - drive.imageID, baseVolUUID) - if res['status']['code'] != 0: - self.log.error("Unable to get volume info for '%s'", baseVolUUID) - return errCode['mergeErr'] - if res['info']['voltype'] == 'SHARED': + if baseInfo['voltype'] == 'SHARED': self.log.error("merge: Refusing to merge into a shared volume") return errCode['mergeErr'] @@ -4628,18 +4652,9 @@ # pivot to the new active layer (baseVolUUID). flags |= libvirt.VIR_DOMAIN_BLOCK_COMMIT_ACTIVE - # If top is the active layer, it's allocated size is stored in - # drive.apparentsize. - topSize = drive.apparentsize - else: - # If top is an internal volume, we must call getVolumeInfo - res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID, - drive.imageID, topVolUUID) - if res['status']['code'] != 0: - self.log.error("Unable to get volume info for '%s'", - topVolUUID) - return errCode['mergeErr'] - topSize = int(res['info']['apparentsize']) + # Make sure we can merge into the base in case the drive was enlarged. + if not self._merge_check_base_size(drive, baseInfo, topInfo): + return errCode['destVolumeTooSmall'] # Take the jobs lock here to protect the new job we are tracking from # being cleaned up by queryBlockJobs() since it won't exist right away @@ -4672,7 +4687,8 @@ # live merge operation. if drive.chunked: capacity, alloc, physical = self._getExtendInfo(drive) - self.extendDriveVolume(drive, baseVolUUID, topSize, capacity) + topAllocSize = int(topInfo['apparentsize']) + self.extendDriveVolume(drive, baseVolUUID, topAllocSize, capacity) # Trigger the collection of stats before returning so that callers # of getVmStats after this returns will see the new job -- To view, visit https://gerrit.ovirt.org/42836 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibf77a7c5108b500c6ec34653ef7570a841def1b4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 31

Change in vdsm[ovirt-3.5]: Stop sysv libvirtd instance if running
by ybronhei＠redhat.com 05 Aug '15

05 Aug '15

Yaniv Bronhaim has uploaded a new change for review. Change subject: Stop sysv libvirtd instance if running ...................................................................... Stop sysv libvirtd instance if running Using service.service_stop will try first to stop with initctl. This patch explicitly stop the sysv instance. LABEL: OVIRT-3.5 ONLY Change-Id: I4012cdde1ca5bbf5aca397a5a3d27deeefd83951 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1239285 Signed-off-by: Yaniv Bronhaim <ybronhei(a)redhat.com> --- M lib/vdsm/tool/configurators/libvirt.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/27/43227/1 diff --git a/lib/vdsm/tool/configurators/libvirt.py b/lib/vdsm/tool/configurators/libvirt.py index b8ab5c9..f3500b4 100644 --- a/lib/vdsm/tool/configurators/libvirt.py +++ b/lib/vdsm/tool/configurators/libvirt.py @@ -170,7 +170,7 @@ if hasattr(service, '_chkconfigDisable'): service._chkconfigDisable('libvirtd') if not os.path.isfile(TARGET): - service.service_stop('libvirtd') + service._serviceStop('libvirtd') if (not os.path.isfile(TARGET) or not filecmp.cmp(packaged, TARGET)): oldmod = None -- To view, visit https://gerrit.ovirt.org/43227 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4012cdde1ca5bbf5aca397a5a3d27deeefd83951 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Yaniv Bronhaim <ybronhei(a)redhat.com>

3 11

Change in vdsm[master]: iscsicred: Protect sensitive return value in supervdsm log
by Nir Soffer 05 Aug '15

05 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: iscsicred: Protect sensitive return value in supervdsm log ...................................................................... iscsicred: Protect sensitive return value in supervdsm log SuperVdsm logs all calls and return values, possibly exposive sensitive data in its log file. This patch allows Supervdsm methods to wrap the return value with ProtectedPassword object, so the results are logged as ********. Change-Id: Idcfa4ee17466f75270909587c45ee9f703e5e9f3 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/supervdsmServer 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/78/43178/1 diff --git a/vdsm/supervdsmServer b/vdsm/supervdsmServer index 2a1a8f0..36c67b1 100755 --- a/vdsm/supervdsmServer +++ b/vdsm/supervdsmServer @@ -53,6 +53,7 @@ except ImportError: _glusterEnabled = False +from vdsm import password from vdsm import utils from vdsm import sysctl from vdsm.tool import restore_nets @@ -112,6 +113,8 @@ raise callbackLogger.debug('return %s with %s', func.__name__, res) + if isinstance(res, password.ProtectedPassword): + res = res.value return res return wrapper -- To view, visit https://gerrit.ovirt.org/43178 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idcfa4ee17466f75270909587c45ee9f703e5e9f3 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

7 10

Change in vdsm[master]: iscsicred: Use local iscsi credentials database
by Nir Soffer 05 Aug '15

05 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: iscsicred: Use local iscsi credentials database ...................................................................... iscsicred: Use local iscsi credentials database When discovering send targets or logging in to targets, use the local iscsi credentials database. Change-Id: I065619f31fbefeb5b96bc86ed9da81ce17c1c47d Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/hsm.py 1 file changed, 24 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/58/43258/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index 1b9de05..a236b23 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -57,6 +57,7 @@ import image import volume import iscsi +import iscsicred import misc from misc import deprecated import taskManager @@ -234,17 +235,34 @@ conDict.get('initiatorName', None), conDict.get('netIfaceName', None)) - cred = None - username = conDict.get('user', None) - password = conDict.get('password', None) - if username or password: - cred = iscsi.ChapCredentials(username, password) + cred = _iscsi_chap_credentials(conDict) params = storageServer.IscsiConnectionParameters(target, iface, cred) else: raise se.StorageServerActionError() return storageServer.ConnectionInfo(typeName, params) + + +def _iscsi_chap_credentials(con): + """ + Return iscsi.ChapCredentials from local credentials database of from + connection info. + + TODO: prefer more specific credentials over less specific: + iqn > ip:port > host > engine. + """ + try: + credentials = iscsicred.get_credentials('host') + return iscsi.ChapCredentials(credentials['username'], + credentials['password']) + except iscsicred.NotFound: + username = con.get('user') + if username: + password = con.get('password') + return iscsi.ChapCredentials(username, password) + + return None class HSM(object): @@ -3028,16 +3046,10 @@ """ ip = con['connection'] port = int(con['port']) - username = con['user'] - password = con['password'] - if username == "": - username = password = None iface = iscsi.IscsiInterface("default") portal = iscsi.IscsiPortal(ip, port) - cred = None - if username or password: - cred = iscsi.ChapCredentials(username, password) + cred = _iscsi_chap_credentials(con) try: targets = iscsi.discoverSendTargets(iface, portal, cred) -- To view, visit https://gerrit.ovirt.org/43258 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I065619f31fbefeb5b96bc86ed9da81ce17c1c47d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 16

Change in vdsm[master]: iscsicred: Install the iscsi-cred directory
by Nir Soffer 05 Aug '15

05 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: iscsicred: Install the iscsi-cred directory ...................................................................... iscsicred: Install the iscsi-cred directory This directory will be populated with iscsi target credential files by the system administrator. Change-Id: Iea9359291a4bea9fa0d93d52f44f1e0af5235a4c Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M debian/vdsm.dirs M debian/vdsm.postinst M vdsm.spec.in M vdsm/Makefile.am 4 files changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/43257/1 diff --git a/debian/vdsm.dirs b/debian/vdsm.dirs index ee2bccd..3fd61f5 100644 --- a/debian/vdsm.dirs +++ b/debian/vdsm.dirs @@ -5,6 +5,7 @@ etc/pki/vdsm/libvirt-spice etc/vdsm etc/vdsm/mom.d +etc/vdsm/iscsi-cred rhev rhev/data-center rhev/data-center/hsm-tasks diff --git a/debian/vdsm.postinst b/debian/vdsm.postinst index 5458974..88644cb 100644 --- a/debian/vdsm.postinst +++ b/debian/vdsm.postinst @@ -66,6 +66,7 @@ $INSTALLDIR /var/run/vdsm/trackedInterfaces $INSTALLDIR /var/log/vdsm $INSTALLDIR /var/log/vdsm/backup + install -dDm 0700 /etc/vdsm/iscsi-cred chmod g+x /var/lib/libvirt/qemu chmod 644 /etc/iscsi/initiatorname.iscsi diff --git a/vdsm.spec.in b/vdsm.spec.in index de15c12..733a6e9 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -805,6 +805,8 @@ %dir %{_libexecdir}/%{vdsm_name} %dir %{_sysconfdir}/%{vdsm_name} %dir %{_sysconfdir}/%{vdsm_name}/mom.d +%dir %{_sysconfdir}/%{vdsm_name}/iscsi-cred +%attr(0700, root, root) %{_sysconfdir}/%{vdsm_name}/iscsi-cred %dir %{_datadir}/%{vdsm_name} %dir %{_datadir}/%{vdsm_name}/network %dir %{_datadir}/%{vdsm_name}/network/configurators diff --git a/vdsm/Makefile.am b/vdsm/Makefile.am index 691fb8b..866ee6b 100644 --- a/vdsm/Makefile.am +++ b/vdsm/Makefile.am @@ -158,6 +158,7 @@ $(MKDIR_P) $(DESTDIR)$(vdsmlibdir)/upgrade $(MKDIR_P) $(DESTDIR)$(vdsmbackupdir) $(MKDIR_P) $(DESTDIR)$(localstatedir)/lib/libvirt/qemu/channels + $(MKDIR_P) $(DESTDIR)$(vdsmconfdir)/iscsi-cred uninstall-local: \ uninstall-data-bonding-defaults \ -- To view, visit https://gerrit.ovirt.org/43257 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iea9359291a4bea9fa0d93d52f44f1e0af5235a4c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 9

Change in vdsm[master]: iscsicred: Support per-host iscsi credentials database
by Nir Soffer 05 Aug '15

05 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: iscsicred: Support per-host iscsi credentials database ...................................................................... iscsicred: Support per-host iscsi credentials database Some users need unique host iSCSI credentials for each target, but Engine supports only same credentials for all hosts. This series adds support for simple per-host iSCSI credentials database. Credentials are stored in /etc/vdsm/iscsi-cred/<target-name> Credentials file format: username = foo:bar password = 12345678 This patch adds the iscsicred module, providing readonly access to the iSCSI credentials database. Change-Id: I8f6a838f6b8e132d6b0c1a8135f2c28ef1e7f847 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M debian/vdsm.install M tests/Makefile.am A tests/iscsicred_test.py M vdsm.spec.in M vdsm/storage/Makefile.am A vdsm/storage/iscsicred.py M vdsm/supervdsmServer 7 files changed, 207 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/79/43179/1 diff --git a/debian/vdsm.install b/debian/vdsm.install index 35fcba7..c7cc90e 100644 --- a/debian/vdsm.install +++ b/debian/vdsm.install @@ -110,6 +110,7 @@ ./usr/share/vdsm/storage/imageSharing.py ./usr/share/vdsm/storage/iscsi.py ./usr/share/vdsm/storage/iscsiadm.py +./usr/share/vdsm/storage/iscsicred.py ./usr/share/vdsm/storage/localFsSD.py ./usr/share/vdsm/storage/lvm.env ./usr/share/vdsm/storage/lvm.py diff --git a/tests/Makefile.am b/tests/Makefile.am index b9b8e72..30ee221 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -56,6 +56,7 @@ iproute2Tests.py \ ipwrapperTests.py \ iscsiTests.py \ + iscsicred_test.py \ jsonRpcHelper.py \ jsonRpcTests.py \ libvirtconnectionTests.py \ diff --git a/tests/iscsicred_test.py b/tests/iscsicred_test.py new file mode 100644 index 0000000..715dded --- /dev/null +++ b/tests/iscsicred_test.py @@ -0,0 +1,121 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +from contextlib import contextmanager +import os + +from testlib import VdsmTestCase +from testlib import namedTemporaryDir +from monkeypatch import MonkeyPatchScope, MonkeyClass + +from vdsm.password import ProtectedPassword +from storage import iscsicred + + +class FakeSupervdsm(object): + + def getProxy(self): + return self + + def readTargetCredfile(self, targetName): + return iscsicred._read_credfile(targetName) + + +@MonkeyClass(iscsicred, "supervdsm", FakeSupervdsm()) +class IscsicredTests(VdsmTestCase): + + TARGET = "iqn.1994-05.com.redhat:target7" + + def test_not_found(self): + self.assertRaises(iscsicred.NotFound, iscsicred.get_credentials, + self.TARGET) + + def test_simple(self): + data = "".join(["username=foo:bar\n", + "password=12345678\n"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678")}) + + def test_whitespace(self): + data = "".join([" username = foo:bar \n", + " password = 12345678 \n"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678")}) + + def test_any_whitespace(self): + data = "".join(["\t\tusername\t\t=\t\tfoo:bar\t\t\n", + "\t\tpassword\t\t=\t\t12345678\t\t\n"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678")}) + + def test_empty_lines(self): + data = "".join(["\n", + "username=foo:bar\n", + "\n", + "password=12345678\n", + "\n"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678")}) + + def test_no_eol(self): + data = "".join(["username=foo:bar\n", + "password=12345678"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678")}) + + def test_extra_key(self): + data = "".join(["username=foo:bar\n", + "password=12345678\n" + "extra=value\n"]) + with credfile(self.TARGET, data): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {"username": "foo:bar", + "password": ProtectedPassword("12345678"), + "extra": "value"}) + + def test_empty(self): + with credfile(self.TARGET, ""): + d = iscsicred.get_credentials(self.TARGET) + self.assertEqual(d, {}) + + def test_invalid(self): + with credfile(self.TARGET, "invalid line\n"): + with self.assertRaises(iscsicred.InvalidCredfile): + iscsicred.get_credentials(self.TARGET) + + +@contextmanager +def credfile(target, data): + with namedTemporaryDir() as tmpdir: + with MonkeyPatchScope([(iscsicred, "DIR", tmpdir)]): + filename = os.path.join(tmpdir, target) + with open(filename, "wb") as f: + f.write(data) + yield diff --git a/vdsm.spec.in b/vdsm.spec.in index 58bf5ef..844a041 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -976,6 +976,7 @@ %{_datadir}/%{vdsm_name}/storage/image.py* %{_datadir}/%{vdsm_name}/storage/imageSharing.py* %{_datadir}/%{vdsm_name}/storage/iscsiadm.py* +%{_datadir}/%{vdsm_name}/storage/iscsicred.py* %{_datadir}/%{vdsm_name}/storage/iscsi.py* %{_datadir}/%{vdsm_name}/storage/localFsSD.py* %{_datadir}/%{vdsm_name}/storage/lvm.env diff --git a/vdsm/storage/Makefile.am b/vdsm/storage/Makefile.am index 4ebe0a2..5e64c44 100644 --- a/vdsm/storage/Makefile.am +++ b/vdsm/storage/Makefile.am @@ -42,6 +42,7 @@ image.py \ imageSharing.py \ iscsiadm.py \ + iscsicred.py \ iscsi.py \ localFsSD.py \ lvm.py \ diff --git a/vdsm/storage/iscsicred.py b/vdsm/storage/iscsicred.py new file mode 100644 index 0000000..7e6bfa3 --- /dev/null +++ b/vdsm/storage/iscsicred.py @@ -0,0 +1,76 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +""" +iscsicred - access iscsi credetials database + +This module provides readonly access to per-host iscsi credentails. Each host +in a cluster may be configured with unique credentatils for each iscsi target. +""" + +import errno +import os +from vdsm import password +import supervdsm + +DIR = "/etc/vdsm/iscsi-cred" + + +class NotFound(Exception): + """ Raised when getting non-existent cred file """ + + +class InvalidCredfile(Exception): + """ Raised if cred file cannot be parsed """ + + +def get_credentials(target_name): + data = supervdsm.getProxy().readTargetCredfile(target_name) + return _parse_credfile(data) + + +def _read_credfile(target_name): + """ + Read /etc/vdsm/iscsi-cred/target_name. + + Called from supervdsm since credfiles are root-readonly files. + """ + filename = os.path.join(DIR, target_name) + try: + with open(filename, "rb") as f: + return f.read() + except IOError as e: + if e.errno != errno.ENOENT: + raise + raise NotFound(target_name) + + +def _parse_credfile(data): + cred = {} + for line in data.splitlines(): + line = line.rstrip() + if not line: + continue + if "=" not in line: + raise InvalidCredfile("Invalid line: %r" % line) + key, value = [s.strip() for s in line.split("=", 1)] + if key == "password": + value = password.ProtectedPassword(value) + cred[key] = value + return cred diff --git a/vdsm/supervdsmServer b/vdsm/supervdsmServer index 36c67b1..1d2dd41 100755 --- a/vdsm/supervdsmServer +++ b/vdsm/supervdsmServer @@ -69,6 +69,7 @@ from storage.iscsi import readSessionInfo as _readSessionInfo from supervdsm import _SuperVdsmManager from storage import hba +from storage import iscsicred from storage import multipath from storage.fileUtils import chown, resolveGid, resolveUid from storage.fileUtils import validateAccess as _validateAccess @@ -167,6 +168,11 @@ return _readSessionInfo(sessionID) @logDecorator + def readTargetCredfile(self, target): + data = iscsicred._read_credfile(target) + return password.ProtectedPassword(data) + + @logDecorator def getPathsStatus(self): return _getPathsStatus() -- To view, visit https://gerrit.ovirt.org/43179 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8f6a838f6b8e132d6b0c1a8135f2c28ef1e7f847 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 24

Change in vdsm[master]: tests: Enhance permutations tests
by Nir Soffer 05 Aug '15

05 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Enhance permutations tests ...................................................................... tests: Enhance permutations tests - Move nested class to module to avoid clumsy self.Class syntax. We don't really need to have a nested class since it is not a TestCase class. - Test instance methods instead of class functions - Separate tests for unrelated features. We don't want tests for expanded method to fail because we did not remove the original method. - Use hasattr() for attribute testing - Add test for sub-classing permuted class. Change-Id: I3e28fb348e688970611ddd2e66ed6869047b5317 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/testlibTests.py 1 file changed, 49 insertions(+), 15 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/14/44214/1 diff --git a/tests/testlibTests.py b/tests/testlibTests.py index d631aaf..0e6f604 100644 --- a/tests/testlibTests.py +++ b/tests/testlibTests.py @@ -130,22 +130,56 @@ pass +@expandPermutations +class Permutated(object): + + @permutations([[1, 2], [3, 4]]) + def fn(self, a, b): + return a, b + + +@expandPermutations +class SubPermuated(Permutated): + + @permutations([[1], [2]]) + def fn2(self, param): + return param + + class TestPermutationExpansion(VdsmTestCase): - @expandPermutations - class _Permutations(object): - @permutations([[False], [True]]) - def fn(self, param): - return param - def test_expansion(self): - self.assertNotIn('fn', dir(self._Permutations)) - self.assertIn('fn(False)', dir(self._Permutations)) - self.assertIn('fn(True)', dir(self._Permutations)) + def setUp(self): + self.instance = Permutated() - def test_parameter_values(self): - self.assertFalse(getattr(self._Permutations(), 'fn(False)')()) - self.assertTrue(getattr(self._Permutations(), 'fn(True)')()) + def test_expand_new_methods(self): + self.assertTrue(hasattr(self.instance, 'fn(1, 2)')) + self.assertTrue(hasattr(self.instance, 'fn(3, 4)')) - def test_expanded_attributes(self): - fn = getattr(self._Permutations, 'fn(False)') - self.assertNotIn(PERMUTATION_ATTR, dir(fn)) + def test_remove_expanded_method(self): + self.assertFalse(hasattr(self.instance, "fn")) + + def test_invoke_expanded_1(self): + expanded_method = getattr(self.instance, "fn(1, 2)") + self.assertEqual((1, 2), expanded_method()) + + def test_invoke_expanded_2(self): + expanded_method = getattr(self.instance, "fn(3, 4)") + self.assertEqual((3, 4), expanded_method()) + + def test_clear_permuations_attribute(self): + fn = getattr(Permutated, 'fn(1, 2)') + self.assertFalse(hasattr(fn, PERMUTATION_ATTR)) + + +class TestSubPermuated(VdsmTestCase): + + def setUp(self): + self.instance = SubPermuated() + + def test_super_method(self): + expanded_method = getattr(self.instance, "fn(1, 2)") + self.assertEqual((1, 2), expanded_method()) + + def test_sub_method(self): + expanded_method = getattr(self.instance, "fn2(1)") + self.assertEqual(1, expanded_method()) -- To view, visit https://gerrit.ovirt.org/44214 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3e28fb348e688970611ddd2e66ed6869047b5317 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 6

Change in vdsm[master]: register: use node name and fqdn when needed
by Douglas Schilling Landgraf 04 Aug '15

04 Aug '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: register: use node name and fqdn when needed ...................................................................... register: use node name and fqdn when needed Currently, we collect host fqdn before registering but it's not required. This patch make registration be triggered without setting host fqdn, it will use node IP address. Change-Id: I91c834e10cade4878791be43201ab0966bf251b4 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/tool/register.py 1 file changed, 9 insertions(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/44221/1 diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py index ce9cdbe..3656f1d 100644 --- a/lib/vdsm/tool/register.py +++ b/lib/vdsm/tool/register.py @@ -84,16 +84,10 @@ self.fprint = fingerprint self.logger.debug("Fingerprint: {fp}".format(fp=self.fprint)) - if node_fqdn is None: - self.node_fqdn = socket.gethostname() - else: - self.node_fqdn = node_fqdn + self.node_fqdn = node_fqdn self.logger.debug("Node FQDN: {nf}".format(nf=self.node_fqdn)) - if node_name is None: - self.node_name = socket.gethostname().split(".")[0] - else: - self.node_name = node_name + self.node_name = node_name self.logger.debug("Node name: {na}".format(na=self.node_name)) if ssh_user is None: @@ -145,14 +139,18 @@ uc=ucmd, c="get-ssh-trust") - ureg = "{uc}register&name={name}&address={fqdn}&sshUser={sshu}&" \ + ureg = "{uc}register&sshUser={sshu}&" \ "sshPort={sshp}&port={mp}".format(uc=ucmd, - name=self.node_name, - fqdn=self.node_fqdn, sshu=self.ssh_user, sshp=self.ssh_port, mp=self.vdsm_port) + if self.node_name is not None: + ureg += "&name={name}".format(name=self.node_name) + + if self.node_fqdn is not None: + ureg += "&address={fqdn}".format(fqdn=self.node_fqdn) + self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) self.logger.debug("Download CA via: {u}".format(u=self.url_CA)) -- To view, visit https://gerrit.ovirt.org/44221 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I91c834e10cade4878791be43201ab0966bf251b4 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

3 9

Change in vdsm[master]: tests: Extract nettestlib module
by Nir Soffer 04 Aug '15

04 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Extract nettestlib module ...................................................................... tests: Extract nettestlib module ipwrapperTests was using tcTests as a library. Move the shared code to new module and use it from both tcTests and ipwrapperTests. Change-Id: I22b491120eb8be5b7ad0087ce21c489e6c04e24e Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/Makefile.am M tests/ipwrapperTests.py A tests/nettestlib.py M tests/tcTests.py 4 files changed, 182 insertions(+), 159 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/12/44212/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 3152735..174982c 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -189,6 +189,7 @@ apiData.py \ hookValidation.py \ monkeypatch.py \ + nettestlib.py \ storagefakelib.py \ storagetestlib.py \ testlib.py \ diff --git a/tests/ipwrapperTests.py b/tests/ipwrapperTests.py index bd3007d..bcef590 100644 --- a/tests/ipwrapperTests.py +++ b/tests/ipwrapperTests.py @@ -22,8 +22,8 @@ from vdsm import ipwrapper from vdsm.ipwrapper import Route from vdsm.ipwrapper import Rule -import tcTests +from nettestlib import Bridge, checkDependencies from testlib import VdsmTestCase as TestCaseBase @@ -95,8 +95,8 @@ @ValidateRunningAsRoot def setUp(self): - tcTests._checkDependencies() - self._bridge = tcTests._Bridge() + checkDependencies() + self._bridge = Bridge() self._bridge.addDevice() def tearDown(self): @@ -113,8 +113,8 @@ @ValidateRunningAsRoot def setUp(self): - tcTests._checkDependencies() - self._bridge = tcTests._Bridge() + checkDependencies() + self._bridge = Bridge() self._bridge.addDevice() def tearDown(self): @@ -140,10 +140,10 @@ @ValidateRunningAsRoot def setUp(self): - tcTests._checkDependencies() + checkDependencies() # First 3 Hebrew letters # See http://unicode.org/charts/PDF/U0590.pdf - self._bridge = tcTests._Bridge("\xd7\x90\xd7\x91\xd7\x92") + self._bridge = Bridge("\xd7\x90\xd7\x91\xd7\x92") self._bridge.addDevice() def tearDown(self): diff --git a/tests/nettestlib.py b/tests/nettestlib.py new file mode 100644 index 0000000..44a831d --- /dev/null +++ b/tests/nettestlib.py @@ -0,0 +1,166 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import errno +import fcntl +import os +import platform +import random +import signal +import struct +from multiprocessing import Process + +from nose.plugins.skip import SkipTest + +from vdsm.constants import EXT_BRCTL, EXT_TC +from vdsm.netlink import monitor +from vdsm.utils import execCmd, random_iface_name + +EXT_IP = "/sbin/ip" + + +class ExecError(RuntimeError): + def __init__(self, msg, out, err): + super(ExecError, self).__init__(msg) + self.out = out + self.err = err + + +def check_call(cmd): + rc, out, err = execCmd(cmd, raw=True) + if rc != 0: + raise ExecError( + 'Command %s returned non-zero exit status %s.' % (cmd, rc), + out, err) + + +class Interface(): + + def __init__(self, prefix='vdsm-'): + self.devName = random_iface_name(prefix) + + def _ifUp(self): + check_call([EXT_IP, "link", "set", self.devName, "up"]) + + def _ifDown(self): + with monitor.Monitor(groups=('link',), timeout=2) as mon: + check_call([EXT_IP, "link", "set", self.devName, "down"]) + for event in mon: + if (event.get('name') == self.devName and + event.get('state') == 'down'): + return + + def __str__(self): + return "<{1} {2!r}>".format(self.__class__.__name__, self.devName) + + +class Bridge(Interface): + + def addDevice(self): + check_call([EXT_BRCTL, 'addbr', self.devName]) + # learning interval is different on different kernels, so set it + # explicit for 2.x kernels + if os.uname()[2].startswith("2"): + check_call([EXT_BRCTL, 'setfd', self.devName, '0']) + check_call([EXT_BRCTL, 'setageing', self.devName, '0']) + self._ifUp() + + def delDevice(self): + self._ifDown() + check_call([EXT_BRCTL, 'delbr', self.devName]) + + def addIf(self, dev): + check_call([EXT_BRCTL, 'addif', self.devName, dev]) + + +def _listenOnDevice(fd, icmp): + while True: + packet = os.read(fd, 2048) + # check if it is an IP packet + if (packet[12:14] == chr(0x08) + chr(0x00)): + if packet == icmp: + return + + +class Tap(Interface): + + _IFF_TAP = 0x0002 + _IFF_NO_PI = 0x1000 + arch = platform.machine() + if arch == 'x86_64': + _TUNSETIFF = 0x400454ca + elif arch == 'ppc64': + _TUNSETIFF = 0x800454ca + else: + raise SkipTest("Unsupported Architecture %s" % arch) + + _deviceListener = None + + def addDevice(self): + self._cloneDevice = open('/dev/net/tun', 'r+b') + ifr = struct.pack('16sH', self.devName, self._IFF_TAP | + self._IFF_NO_PI) + fcntl.ioctl(self._cloneDevice, self._TUNSETIFF, ifr) + self._ifUp() + + def delDevice(self): + self._ifDown() + self._cloneDevice.close() + + def startListener(self, icmp): + self._deviceListener = Process(target=_listenOnDevice, + args=(self._cloneDevice.fileno(), icmp)) + self._deviceListener.start() + + def isListenerAlive(self): + if self._deviceListener: + return self._deviceListener.is_alive() + else: + return False + + def stopListener(self): + if self._deviceListener: + os.kill(self._deviceListener.pid, signal.SIGKILL) + self._deviceListener.join() + + def writeToDevice(self, icmp): + os.write(self._cloneDevice.fileno(), icmp) + + +def checkDependencies(): + + # hack to avoid name collision in concurrently-running mock-based tests + random.jumpahead(os.getpid()) + + dev = Bridge() + try: + dev.addDevice() + except OSError as e: + if e.errno == errno.ENOENT: + raise SkipTest("Cannot run %r: %s\nDo you have bridge-utils " + "installed?" % (EXT_BRCTL, e)) + raise + + try: + check_call([EXT_TC, 'qdisc', 'add', 'dev', dev.devName, 'ingress']) + except ExecError as e: + raise SkipTest("%r has failed: %s\nDo you have Traffic Control kernel " + "modules installed?" % (EXT_TC, e.err)) + finally: + dev.delDevice() diff --git a/tests/tcTests.py b/tests/tcTests.py index 942f63f..8c894ba 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -19,173 +19,29 @@ # Refer to the README and COPYING files for full details of the license # -import errno -import random import time import os -import signal import six import sys -from multiprocessing import Process from binascii import unhexlify from itertools import izip_longest from subprocess import Popen, PIPE -import fcntl -import struct from testlib import VdsmTestCase as TestCaseBase from testValidation import ValidateRunningAsRoot +from nettestlib import Bridge, Tap, checkDependencies -from vdsm.constants import EXT_BRCTL, EXT_TC -from vdsm.netlink import monitor -from vdsm.utils import execCmd, random_iface_name - -from nose.plugins.skip import SkipTest - +from vdsm.constants import EXT_TC from network import tc -import platform - -EXT_IP = "/sbin/ip" - - -class ExecError(RuntimeError): - def __init__(self, msg, out, err): - super(ExecError, self).__init__(msg) - self.out = out - self.err = err - - -def check_call(cmd): - rc, out, err = execCmd(cmd, raw=True) - if rc != 0: - raise ExecError( - 'Command %s returned non-zero exit status %s.' % (cmd, rc), - out, err) - - -class _Interface(): - - def __init__(self, prefix='vdsm-'): - self.devName = random_iface_name(prefix) - - def _ifUp(self): - check_call([EXT_IP, "link", "set", self.devName, "up"]) - - def _ifDown(self): - with monitor.Monitor(groups=('link',), timeout=2) as mon: - check_call([EXT_IP, "link", "set", self.devName, "down"]) - for event in mon: - if (event.get('name') == self.devName and - event.get('state') == 'down'): - return - - def __str__(self): - return "<{1} {2!r}>".format(self.__class__.__name__, self.devName) - - -class _Bridge(_Interface): - - def addDevice(self): - check_call([EXT_BRCTL, 'addbr', self.devName]) - # learning interval is different on different kernels, so set it - # explicit for 2.x kernels - if os.uname()[2].startswith("2"): - check_call([EXT_BRCTL, 'setfd', self.devName, '0']) - check_call([EXT_BRCTL, 'setageing', self.devName, '0']) - self._ifUp() - - def delDevice(self): - self._ifDown() - check_call([EXT_BRCTL, 'delbr', self.devName]) - - def addIf(self, dev): - check_call([EXT_BRCTL, 'addif', self.devName, dev]) - - -def _listenOnDevice(fd, icmp): - while True: - packet = os.read(fd, 2048) - # check if it is an IP packet - if (packet[12:14] == chr(0x08) + chr(0x00)): - if packet == icmp: - return - - -class Tap(_Interface): - - _IFF_TAP = 0x0002 - _IFF_NO_PI = 0x1000 - arch = platform.machine() - if arch == 'x86_64': - _TUNSETIFF = 0x400454ca - elif arch == 'ppc64': - _TUNSETIFF = 0x800454ca - else: - raise SkipTest("Unsupported Architecture %s" % arch) - - _deviceListener = None - - def addDevice(self): - self._cloneDevice = open('/dev/net/tun', 'r+b') - ifr = struct.pack('16sH', self.devName, self._IFF_TAP | - self._IFF_NO_PI) - fcntl.ioctl(self._cloneDevice, self._TUNSETIFF, ifr) - self._ifUp() - - def delDevice(self): - self._ifDown() - self._cloneDevice.close() - - def startListener(self, icmp): - self._deviceListener = Process(target=_listenOnDevice, - args=(self._cloneDevice.fileno(), icmp)) - self._deviceListener.start() - - def isListenerAlive(self): - if self._deviceListener: - return self._deviceListener.is_alive() - else: - return False - - def stopListener(self): - if self._deviceListener: - os.kill(self._deviceListener.pid, signal.SIGKILL) - self._deviceListener.join() - - def writeToDevice(self, icmp): - os.write(self._cloneDevice.fileno(), icmp) - - -def _checkDependencies(): - - # hack to avoid name collision in concurrently-running mock-based tests - random.jumpahead(os.getpid()) - - dev = _Bridge() - try: - dev.addDevice() - except OSError as e: - if e.errno == errno.ENOENT: - raise SkipTest("Cannot run %r: %s\nDo you have bridge-utils " - "installed?" % (EXT_BRCTL, e)) - raise - - try: - check_call([EXT_TC, 'qdisc', 'add', 'dev', dev.devName, 'ingress']) - except ExecError as e: - raise SkipTest("%r has failed: %s\nDo you have Traffic Control kernel " - "modules installed?" % (EXT_TC, e.err)) - finally: - dev.delDevice() class TestQdisc(TestCaseBase): @ValidateRunningAsRoot def setUp(self): - _checkDependencies() - self._bridge = _Bridge() + checkDependencies() + self._bridge = Bridge() self._bridge.addDevice() def tearDown(self): @@ -431,13 +287,13 @@ @ValidateRunningAsRoot def setUp(self): - _checkDependencies() + checkDependencies() self._tap0 = Tap() self._tap1 = Tap() self._tap2 = Tap() - self._bridge0 = _Bridge('src-') - self._bridge1 = _Bridge('target-') - self._bridge2 = _Bridge('target2-') + self._bridge0 = Bridge('src-') + self._bridge1 = Bridge('target-') + self._bridge2 = Bridge('target2-') self._devices = [self._tap0, self._tap1, self._tap2, self._bridge0, self._bridge1, self._bridge2] # If setUp raise, teardown is not called, so we should either succeed, -- To view, visit https://gerrit.ovirt.org/44212 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I22b491120eb8be5b7ad0087ce21c489e6c04e24e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 16

Change in vdsm[master]: tests: Remove hack for devices with same names
by Nir Soffer 04 Aug '15

04 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Remove hack for devices with same names ...................................................................... tests: Remove hack for devices with same names After the test isolation issues in tcTests.py and ipwrapperTests.py were resolved, we are not expecting setup to fail because a device with same name exists, so we can remove the hack that used to skip tests. It is possible that slaves contains stale devices from previous run of the broken tests, but this should be solved by cleaning the slaves, not by hiding the error and skipping tests. Change-Id: I675e57756859ca7ffe272607b7f528c281c65256 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/tcTests.py 1 file changed, 1 insertion(+), 11 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/66/44166/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index dbbac95..942f63f 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -87,17 +87,7 @@ class _Bridge(_Interface): def addDevice(self): - try: - check_call([EXT_BRCTL, 'addbr', self.devName]) - except ExecError as e: - # FIXME: we do not know why we sometime see the same bridge names - # on jenkins slaves. This is an ugly hack to mitigate the issue, - # since it is certainly not the fault of the test case - if "can't create bridge with the same name" in e.err: - raise SkipTest(e.err) - else: - raise - + check_call([EXT_BRCTL, 'addbr', self.devName]) # learning interval is different on different kernels, so set it # explicit for 2.x kernels if os.uname()[2].startswith("2"): -- To view, visit https://gerrit.ovirt.org/44166 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I675e57756859ca7ffe272607b7f528c281c65256 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 7

Change in vdsm[master]: tests: Fix ipwrapper tests isolation
by Nir Soffer 04 Aug '15

04 Aug '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Fix ipwrapper tests isolation ...................................................................... tests: Fix ipwrapper tests isolation Similar to tcTests, tests were sharing same devices, so failure to delete a device would cause unrelated test to fail. Even worse, the Unicode tests used *same* device name on all tests. So running concurrent tests is not possible, and once we fail to delete a device, all builds will fail on this slave when tyring to create the same device again. This issue was hidden previously because we used to skip silently such failures. Now we use Unicode prefix with random suffix, so stale devices are unlikely to cause failures in other builds. To simplify cleanup, the Unicode tests was moved to its own test case class. To make it easier to work with the Unicode bridge name, we use plain ASCII ("\xd7\x90\xd7\x91\xd7\x92") instead of raw utf-8 ("אבג"). Change-Id: Ic02b3945a029476f17601608555430dc2fb710de Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/ipwrapperTests.py 1 file changed, 24 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/65/44165/1 diff --git a/tests/ipwrapperTests.py b/tests/ipwrapperTests.py index d75a537..bd3007d 100644 --- a/tests/ipwrapperTests.py +++ b/tests/ipwrapperTests.py @@ -92,11 +92,11 @@ class TestLinks(TestCaseBase): - _bridge = tcTests._Bridge() @ValidateRunningAsRoot def setUp(self): tcTests._checkDependencies() + self._bridge = tcTests._Bridge() self._bridge.addDevice() def tearDown(self): @@ -110,34 +110,46 @@ class TestDrvinfo(TestCaseBase): - _bridge = tcTests._Bridge() - _unicode_bridge = tcTests._Bridge() @ValidateRunningAsRoot def setUp(self): tcTests._checkDependencies() + self._bridge = tcTests._Bridge() self._bridge.addDevice() - self._unicode_bridge.devName = 'test-トトロ' - self._unicode_bridge.addDevice() def tearDown(self): self._bridge.delDevice() - self._unicode_bridge.delDevice() def testBridgeEthtoolDrvinfo(self): self.assertEqual(ipwrapper.drv_name(self._bridge.devName), ipwrapper.LinkType.BRIDGE) - def testUtf8BridgeEthtoolDrvinfo(self): - self.assertEqual( - ipwrapper.drv_name(self._unicode_bridge.devName.decode('utf8')), - ipwrapper.LinkType.BRIDGE) - - def testTogglePromisc(self): + def testEnablePromisc(self): ipwrapper.getLink(self._bridge.devName).promisc = True self.assertTrue(ipwrapper.getLink(self._bridge.devName).promisc, "Could not enable promiscuous mode.") + def testDisablePromisc(self): + ipwrapper.getLink(self._bridge.devName).promisc = True ipwrapper.getLink(self._bridge.devName).promisc = False self.assertFalse(ipwrapper.getLink(self._bridge.devName).promisc, "Could not disable promiscuous mode.") + + +class TestUnicodeDrvinfo(TestCaseBase): + + @ValidateRunningAsRoot + def setUp(self): + tcTests._checkDependencies() + # First 3 Hebrew letters + # See http://unicode.org/charts/PDF/U0590.pdf + self._bridge = tcTests._Bridge("\xd7\x90\xd7\x91\xd7\x92") + self._bridge.addDevice() + + def tearDown(self): + self._bridge.delDevice() + + def testUtf8BridgeEthtoolDrvinfo(self): + self.assertEqual( + ipwrapper.drv_name(self._bridge.devName.decode('utf8')), + ipwrapper.LinkType.BRIDGE) -- To view, visit https://gerrit.ovirt.org/44165 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic02b3945a029476f17601608555430dc2fb710de Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 8

Change in vdsm[master]: call stop_event_loop upon exit
by Dan Kenigsberg 04 Aug '15

04 Aug '15

Dan Kenigsberg has uploaded a new change for review. Change subject: call stop_event_loop upon exit ...................................................................... call stop_event_loop upon exit For cleanliness, whomever starts a thread should stop it when it is no longer needed. Change-Id: I9ab0d9b7be976e37a89a96d2f09a353186008731 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M vdsm/vdsm 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/32/26532/1 diff --git a/vdsm/vdsm b/vdsm/vdsm index 652797c..fd9b3f8 100755 --- a/vdsm/vdsm +++ b/vdsm/vdsm @@ -81,6 +81,7 @@ signal.pause() finally: cif.prepareForShutdown() + libvirtconnection.stop_event_loop() def run(pidfile=None): -- To view, visit http://gerrit.ovirt.org/26532 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9ab0d9b7be976e37a89a96d2f09a353186008731 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

7 19

Change in vdsm[master]: vdsm: drop cluster version < 3.6 for ppc64le
by Martin Polednik 04 Aug '15

04 Aug '15

Martin Polednik has uploaded a new change for review. Change subject: vdsm: drop cluster version < 3.6 for ppc64le ...................................................................... vdsm: drop cluster version < 3.6 for ppc64le Due to moving from powerKVM to ppc64le, we should not support older cluster levels in newest VDSM for POWER8 platform. I5c55f66aa526a6b9498f557c964ee18888f72885 Change-Id: Ibda0fc1b3716fa9be361a0f60e3639395bb9a174 Signed-off-by: Martin Polednik <mpolednik(a)redhat.com> --- M vdsm/dsaversion.py.in 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/92/43892/1 diff --git a/vdsm/dsaversion.py.in b/vdsm/dsaversion.py.in index 374580d..41f6266 100644 --- a/vdsm/dsaversion.py.in +++ b/vdsm/dsaversion.py.in @@ -18,6 +18,8 @@ # Refer to the README and COPYING files for full details of the license # +import platform + """ This module provides DSA software versioning information for python based components @@ -36,3 +38,6 @@ 'supportedENGINEs': ['3.4', '3.5', '3.6'], 'clusterLevels': ['3.4', '3.5', '3.6'], } + +if platform.machine() == 'ppc64le': + version_info['clusterLevels'] = ['3.6'] -- To view, visit https://gerrit.ovirt.org/43892 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibda0fc1b3716fa9be361a0f60e3639395bb9a174 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

5 10

Change in vdsm[master]: v2v: handle exception on probing external VMs
by shavivi＠redhat.com 03 Aug '15

03 Aug '15

Shahar Havivi has uploaded a new change for review. Change subject: v2v: handle exception on probing external VMs ...................................................................... v2v: handle exception on probing external VMs XMLDesc of libvirt domain can throw error, We don't want to fail all the process of fetching external VMs if we fail to read one VM xml. Change-Id: Ie3203dde4878a80c65fe3185cadaef2fe00e6a02 Signed-off-by: Shahar Havivi <shaharh(a)redhat.com> --- M vdsm/v2v.py 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/43259/1 diff --git a/vdsm/v2v.py b/vdsm/v2v.py index b6763da..307c005 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -139,7 +139,11 @@ with closing(conn): vms = [] for vm in conn.listAllDomains(): - root = ET.fromstring(vm.XMLDesc(0)) + try: + root = ET.fromstring(vm.XMLDesc(0)) + except libvirt.libvirtError as e: + logging.error('error reading domain xml, msg: %s', e.message) + continue params = {} _add_vm_info(vm, params) try: -- To view, visit https://gerrit.ovirt.org/43259 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3203dde4878a80c65fe3185cadaef2fe00e6a02 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shahar Havivi <shavivi(a)redhat.com>

4 7

Change in vdsm[ovirt-3.5]: pep8 again
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: pep8 again ...................................................................... pep8 again Too bad that my pep8-correcting patch was written and verified before commit 14e2c07 was merged. Change-Id: I82a181388ce36f8db2e71bfcde3d2eea36c05274 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 0 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/46/44246/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index a908877..d369bde 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1773,7 +1773,6 @@ self.assertEquals(status, SUCCESS, msg) self.assertNetworkDoesntExist(NETWORK_NAME) - @cleanupNet @permutations([[True], [False]]) @RequireDummyMod -- To view, visit https://gerrit.ovirt.org/44246 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I82a181388ce36f8db2e71bfcde3d2eea36c05274 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

2 6

Change in vdsm[master]: vm: event: simplify send_status_event
by fromani＠redhat.com 31 Jul '15

31 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vm: event: simplify send_status_event ...................................................................... vm: event: simplify send_status_event Now that VDSM is fully in charge of sending events to Engine, there is no longer need to protect the send_status_event call. This patch removes the safety checks and greatly simplifies the method. Change-Id: I3013d1e2a1a2f6e8c325adb37c0019fab0c7a4f6 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 5 insertions(+), 15 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/14/43414/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 08ee7ee..c6d3b8f 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -303,8 +303,6 @@ self._lastStatus = vmstatus.RESTORING_STATE else: self._lastStatus = vmstatus.WAIT_FOR_LAUNCH - self._evaluatedStatus = None - self._eventLock = threading.Lock() self._migrationSourceThread = migration.SourceThread(self) self._kvmEnable = self.conf.get('kvmEnable', 'true') self._incomingMigrationFinished = threading.Event() @@ -379,19 +377,11 @@ self._lastStatus = value def send_status_event(self, **kwargs): - vm_status = self._getVmStatus() - stats = {} - with self._eventLock: - if vm_status != self._evaluatedStatus: - self._evaluatedStatus = vm_status - current_status = self.lastStatus - stats['status'] = vm_status - - stats.update(kwargs) - if stats: - self.log.debug('Last status %s and evaluated status %s', - current_status, vm_status) - self._notify('VM_status', stats) + stats = { + 'status': self._getVmStatus() + } + stats.update(kwargs) + self._notify('VM_status', stats) def _notify(self, operation, params): sub_id = '|virt|%s|%s' % (operation, self.id) -- To view, visit https://gerrit.ovirt.org/43414 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3013d1e2a1a2f6e8c325adb37c0019fab0c7a4f6 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 24

Change in vdsm[ovirt-3.5]: make pep8-1.4.6 (el7) happy
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Hello Ido Barkan, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/44239 to review the following change. Change subject: make pep8-1.4.6 (el7) happy ...................................................................... make pep8-1.4.6 (el7) happy Change-Id: Iee593cee60690c43f82f30900f5c992bfba49000 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> Reviewed-on: https://gerrit.ovirt.org/44201 Continuous-Integration: Jenkins CI Reviewed-by: Ido Barkan <ibarkan(a)redhat.com> Tested-by: Ido Barkan <ibarkan(a)redhat.com> --- M tests/functional/networkTests.py M vdsm/vdsm-restore-net-config 2 files changed, 29 insertions(+), 29 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/39/44239/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index e01a612..614131a 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1545,21 +1545,21 @@ IP_ADDR_NET_CHANGED = '240.0.0.101' IP_ADDR_MISSING = '204.0.0.102' nets = { - NET_UNCHANGED: - {'bootproto': 'none', 'ipaddr': IP_ADD_UNCHANGED, - 'vlan': str(VLAN_UNCHANGED), 'netmask': IP_MASK, - 'bonding': BOND_UNCHANGED, 'defaultRoute': True}, - NET_CHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_NET_CHANGED, - 'vlan': str(VLAN_UNCHANGED + 1), - 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, - 'defaultRoute': False}, - NET_MISSING: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_MISSING, - 'vlan': str(VLAN_UNCHANGED + 2), - 'netmask': IP_MASK, 'bonding': BOND_MISSING}, + NET_UNCHANGED: { + 'bootproto': 'none', 'ipaddr': IP_ADD_UNCHANGED, + 'vlan': str(VLAN_UNCHANGED), 'netmask': IP_MASK, + 'bonding': BOND_UNCHANGED, 'defaultRoute': True}, + NET_CHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_NET_CHANGED, + 'vlan': str(VLAN_UNCHANGED + 1), + 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, + 'defaultRoute': False}, + NET_MISSING: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_MISSING, + 'vlan': str(VLAN_UNCHANGED + 2), + 'netmask': IP_MASK, 'bonding': BOND_MISSING}, } def _assert_all_nets_exist(): @@ -1641,18 +1641,18 @@ IP_ADDR_CHANGED = '204.0.0.102' IP_ADDR_ADDITIONAL = '204.0.0.102' nets = { - NET_MGMT: - {'bootproto': 'none', 'ipaddr': IP_MGMT, - 'netmask': IP_MASK, 'defaultRoute': True}, - NET_UNCHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_UNCHANGED, - 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, - 'defaultRoute': False}, - NET_CHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_CHANGED, - 'netmask': IP_MASK, 'bonding': BOND_CHANGED}, + NET_MGMT: { + 'bootproto': 'none', 'ipaddr': IP_MGMT, + 'netmask': IP_MASK, 'defaultRoute': True}, + NET_UNCHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_UNCHANGED, + 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, + 'defaultRoute': False}, + NET_CHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_CHANGED, + 'netmask': IP_MASK, 'bonding': BOND_CHANGED}, } net_additional_attrs = { 'bootproto': 'none', 'ipaddr': IP_ADDR_ADDITIONAL, diff --git a/vdsm/vdsm-restore-net-config b/vdsm/vdsm-restore-net-config index c879442..cf3f464 100755 --- a/vdsm/vdsm-restore-net-config +++ b/vdsm/vdsm-restore-net-config @@ -87,8 +87,8 @@ bonds = changed_config.bonds _convert_to_blocking_dhcp(nets) if nets or bonds: - logging.debug('Calling setupNetworks with networks (%s) and bond (%s).', - nets, bonds) + logging.debug('Calling setupNetworks with networks (%s) ' + 'and bond (%s).', nets, bonds) setupNetworks(nets, bonds, connectivityCheck=False, _inRollback=True) -- To view, visit https://gerrit.ovirt.org/44239 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iee593cee60690c43f82f30900f5c992bfba49000 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com>

4 6

Change in vdsm[ovirt-3.5]: net: always persist owned ifcfg files on ovirt node
by ibarkan＠redhat.com 31 Jul '15

31 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: net: always persist owned ifcfg files on ovirt node ...................................................................... net: always persist owned ifcfg files on ovirt node Change-Id: Ibc717b86194a32c050d346e235a5c35fd318e1ff Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> Buh-Url: https://bugzilla.redhat.com/1203422 --- M vdsm/vdsm-store-net-config.in 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/33/44133/1 diff --git a/vdsm/vdsm-store-net-config.in b/vdsm/vdsm-store-net-config.in index c19d5e0..98b5776 100755 --- a/vdsm/vdsm-store-net-config.in +++ b/vdsm/vdsm-store-net-config.in @@ -37,8 +37,19 @@ rm -rf "$NET_CONF_BACK_DIR"/* } +node_persist_owned_ifcfgs() { + for f in $(find "$NET_CONF_DIR" -type f); do + if grep -q "# Generated by VDSM version" "$f"; then + ovirt_store_config "$f" + fi + done +} + unified_node_persist() { unified_nonnode_persist + # we need to persist ifcfg files as well so that they are available earlier + # on boot time + node_persist_owned_ifcfgs # oVirt node ovirt_store_config puts the dir in persistent storage and # bind mounts it in the original place. So that's all we really need to do. -- To view, visit https://gerrit.ovirt.org/44133 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibc717b86194a32c050d346e235a5c35fd318e1ff Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

4 9

Change in vdsm[ovirt-3.5]: net: ignore pass through ifcfg parameters for selective rest...
by ibarkan＠redhat.com 31 Jul '15

31 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: net: ignore pass through ifcfg parameters for selective restoration ...................................................................... net: ignore pass through ifcfg parameters for selective restoration Ignore keys in persisted networks that might originate from vdsm-reg. these might be a result of calling setupNetworks with ifcfg values that come from the original interface that is serving the management network. for 3.5, VDSM still supports passing arbitrary values directly to the ifcfg files, e.g. 'IPV6_AUTOCONF=no'. we filter them out here since kernelConfig will never report them. Change-Id: Ib1525f7b0238aaa5eb82eedb9ad6f41154c2c29f Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> Bug-Url: https://bugzilla.redhat.com/1203422 --- M lib/vdsm/netconfpersistence.py M tests/functional/networkTests.py 2 files changed, 41 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/44153/1 diff --git a/lib/vdsm/netconfpersistence.py b/lib/vdsm/netconfpersistence.py index cc24228..f5916f6 100644 --- a/lib/vdsm/netconfpersistence.py +++ b/lib/vdsm/netconfpersistence.py @@ -24,6 +24,7 @@ import logging import netaddr import os +import string from .config import config from .tool.restore_nets import restore @@ -316,6 +317,7 @@ self._normalize_bonding_opts(config_copy) self._normalize_bonding_nics(config_copy) self._normalize_address(config_copy) + self._normalize_ifcfg_keys(config_copy) return config_copy @@ -387,6 +389,23 @@ if 'defaultRoute' not in net_attr: net_attr['defaultRoute'] = False + def _normalize_ifcfg_keys(self, config_copy): + # ignore keys in persisted networks that might originate from vdsm-reg. + # these might be a result of calling setupNetworks with ifcfg values + # that come from the original interface that is serving the management + # network. for 3.5, VDSM still supports passing arbitrary values + # directly to the ifcfg files, e.g. 'IPV6_AUTOCONF=no'. we filter them + # out here since kernelConfig will never report them. + # TODO: remove when 3.5 is unsupported. + def unsupported(key): + return set(key) <= set( + string.ascii_uppercase + string.digits + '_') + + for net_attr in config_copy.networks.itervalues(): + for k in net_attr.keys(): + if unsupported(k): + net_attr.pop(k) + def _parse_bond_options(self, opts): if not opts: return {} diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index c5da1f2..9462b60 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1753,6 +1753,28 @@ self.assertBondDoesntExist(BOND_UNCHANGED, [nic_a]) @cleanupNet + def testSelectiveRestoreIgnoresVdsmRegParams(self): + if vdsm.config.config.get('vars', 'net_persistence') == 'ifcfg': + raise SkipTest( + "with ifcfg persistence, vdsm-restore-net-config selective" + "restoration is not supported") + + with dummyIf(1) as nics: + nic, = nics + # let _assert_kernel_config_matches_running_config do the job + status, msg = self.setupNetworks( + {NETWORK_NAME: {'nic': nic, 'IPV6_AUTOCONF': 'no', + 'PEERNTP': 'yes', 'IPV6INIT': 'no'}}, + {}, NOCHK) + self.assertEquals(status, SUCCESS, msg) + self.assertNetworkExists(NETWORK_NAME) + status, msg = self.setupNetworks( + {NETWORK_NAME: {'remove': True}}, {}, NOCHK) + self.assertEquals(status, SUCCESS, msg) + self.assertNetworkDoesntExist(NETWORK_NAME) + + + @cleanupNet @permutations([[True], [False]]) @RequireDummyMod @ValidateRunningAsRoot -- To view, visit https://gerrit.ovirt.org/44153 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib1525f7b0238aaa5eb82eedb9ad6f41154c2c29f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

4 5

Change in vdsm[ovirt-3.5]: restore-net-config: set main thread name
by ibarkan＠redhat.com 31 Jul '15

31 Jul '15

Hello Dan Kenigsberg, I'd like you to do a code review. Please visit https://gerrit.ovirt.org/44199 to review the following change. Change subject: restore-net-config: set main thread name ...................................................................... restore-net-config: set main thread name restore-net-config's log flow to supervdsm.log, and are a bit hard to discern. Change-Id: I7a92962e488e78e57697b6313463c41f63408502 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> Reviewed-on: http://gerrit.ovirt.org/37419 Reviewed-by: Ido Barkan <ibarkan(a)redhat.com> Bug-Url: https://bugzilla.redhat.com/1203422 Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M vdsm/vdsm-restore-net-config 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/99/44199/1 diff --git a/vdsm/vdsm-restore-net-config b/vdsm/vdsm-restore-net-config index c879442..078cf6b 100755 --- a/vdsm/vdsm-restore-net-config +++ b/vdsm/vdsm-restore-net-config @@ -24,6 +24,7 @@ import logging.config import os import re +import threading import time import errno @@ -318,6 +319,7 @@ if __name__ == '__main__': + threading.current_thread().setName('restore-net') try: logging.config.fileConfig('/etc/vdsm/svdsm.logger.conf', disable_existing_loggers=False) -- To view, visit https://gerrit.ovirt.org/44199 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7a92962e488e78e57697b6313463c41f63408502 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>

4 5

Change in vdsm[master]: register: remove legacy code
by Douglas Schilling Landgraf 31 Jul '15

31 Jul '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: register: remove legacy code ...................................................................... register: remove legacy code The legacy code should be dropped. The new registration schema should only support Engine >= 3.6. Change-Id: I6c0bf6c8879538faa5ddc1d3eb92555aba272c22 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/tool/register.py 1 file changed, 26 insertions(+), 78 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/44219/1 diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py index 31121c0..8e464a4 100644 --- a/lib/vdsm/tool/register.py +++ b/lib/vdsm/tool/register.py @@ -121,57 +121,42 @@ self.ca_engine = "{d}{f}".format(d=self.ca_dir, f="ca.pem") self.logger.debug("Engine CA: {ca}".format(ca=self.ca_engine)) - def get_protocol(self): + def handshake(self): """ - Determine if Engine is running in registration - protocol version legacy or service - REQUIRED_FOR: Engine 3.3 + Initial communication with Engine to validate + the registration as a service. """ - self.logger.info("Identifying the registration protocol...") + self.logger.info("Starting registration as a service...") ucmd = "/ovirt-engine/services/host-register?version=1&command=" __GET_VERSION = "https://{e}{u}{c}".format(e=self.engine_fqdn, u=ucmd, c="get-version") + self.logger.debug("Get version via: {0}".format(__GET_VERSION)) + res = requests.get(__GET_VERSION, verify=False) if res.status_code != 200: - self.reg_protocol = "legacy" - self.url_CA = self.engine_url + raise RuntimeError("Cannot get registration version from Engine!") - self.url_ssh_key = "{e}{k}".format(e=self.engine_url, - k="/engine.ssh.key.txt") + self.url_CA = "{e}{uc}{c}".format(e=self.engine_url, + uc=ucmd, + c="get-pki-trust") - ureg = "/OvirtEngineWeb/register?vds_ip={fqdn}" \ - "&vds_name={name}&port={mp}".format(fqdn=self.node_fqdn, - name=self.node_name, - mp=self.vdsm_port) + self.url_ssh_key = "{e}{uc}{c}".format(e=self.engine_url, + uc=ucmd, + c="get-ssh-trust") - self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) - else: - self.reg_protocol = "service" + ureg = "{uc}register&name={name}&address={fqdn}&sshUser={sshu}&" \ + "sshPort={sshp}&port={mp}".format(uc=ucmd, + name=self.node_name, + fqdn=self.node_fqdn, + sshu=self.ssh_user, + sshp=self.ssh_port, + mp=self.vdsm_port) - self.url_CA = "{e}{uc}{c}".format(e=self.engine_url, - uc=ucmd, - c="get-pki-trust") - - self.url_ssh_key = "{e}{uc}{c}".format(e=self.engine_url, - uc=ucmd, - c="get-ssh-trust") - - ureg = "{uc}register&name={name}&address={fqdn}&sshUser={sshu}&" \ - "sshPort={sshp}&port={mp}".format(uc=ucmd, - name=self.node_name, - fqdn=self.node_fqdn, - sshu=self.ssh_user, - sshp=self.ssh_port, - mp=self.vdsm_port) - - self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) - - self.logger.info("Registration procotol selected: {p}".format( - p=self.reg_protocol)) + self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) self.logger.debug("Download CA via: {u}".format(u=self.url_CA)) self.logger.debug("Download SSH via: {u}".format(u=self.url_ssh_key)) @@ -264,15 +249,8 @@ Determine host UUID and if there is no existing /etc/vdsm/vdsm.id it will genereate UUID and save/persist in /etc/vdsm/vdsm.id """ - if self.reg_protocol == "legacy": - # REQUIRED_FOR: Engine 3.3 - # The legacy version uses the format: UUID_MACADDRESS - self.uuid = getHostUUID(legacy=True) - self.url_reg += "&vds_unique_id={u}".format(u=self.uuid) - else: - # Non legacy version uses the format: UUID - self.uuid = getHostUUID(legacy=False) - self.url_reg += "&uniqueId={u}".format(u=self.uuid) + self.uuid = getHostUUID(legacy=False) + self.url_reg += "&uniqueId={u}".format(u=self.uuid) self.logger.debug("Registration via: {u}".format(u=self.url_reg)) @@ -304,14 +282,8 @@ from ovirt.node.utils.fs import Config Config().persist(self.ca_dir) - if self.reg_protocol == "legacy": - # REQUIRED_FOR: Engine 3.3 - res = ssl.get_server_certificate( - (self.engine_fqdn, int(self.engine_port)) - ) - else: - res = self._execute_http_request(self.url_CA, - cert_validation=False) + res = self._execute_http_request(self.url_CA, + cert_validation=False) with tempfile.NamedTemporaryFile( dir=os.path.dirname(self.ca_dir), @@ -479,7 +451,7 @@ check_fqdn=args.check_fqdn) try: - reg.get_protocol() + reg.handshake() reg.host_uuid() reg.download_ca() reg.download_ssh() @@ -504,30 +476,6 @@ - In case, there is no UUID, use auxiliary function from VDSM to generate it and store in /etc/vdsm/vdsm.id - -Legacy reg: -============ - - REQUIRED_FOR: Engine 3.3 - - - Process UUID - - - Download CA via - https://ENGINE_FQDN - - - Download ssh pub key - https://ENGINE_FQDN/engine.ssh.key.txt - - - Register via URL: - (Original .NET version and earlier Linux versions) - https://ENGINE_FQDN/RHEVManagerWeb/VdsAutoRegistration.aspx - ?vds_ip=NODE_FQDN_OR_IP&vds_name=NODE_NAME - &vds_unique_id=NODE_UUID&port=54321 - - or - - https://ENGINE_FQDN/OvirtEngineWeb/register?vds_ip=NODE_FQDN_OR_IP - &vds_name=NODE_NAME - &vds_unique_id=NODE_UUID&port=54321 Service reg: ============ -- To view, visit https://gerrit.ovirt.org/44219 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6c0bf6c8879538faa5ddc1d3eb92555aba272c22 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

3 12

Change in vdsm[master]: make pep8-1.4.6 (el7) happy
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: make pep8-1.4.6 (el7) happy ...................................................................... make pep8-1.4.6 (el7) happy Change-Id: Iee593cee60690c43f82f30900f5c992bfba49000 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 27 insertions(+), 27 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/01/44201/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index ce532bb..c6eedcd 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1539,21 +1539,21 @@ IP_ADDR_NET_CHANGED = '240.0.0.101' IP_ADDR_MISSING = '204.0.0.102' nets = { - NET_UNCHANGED: - {'bootproto': 'none', 'ipaddr': IP_ADD_UNCHANGED, - 'vlan': str(VLAN_UNCHANGED), 'netmask': IP_MASK, - 'bonding': BOND_UNCHANGED, 'defaultRoute': True}, - NET_CHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_NET_CHANGED, - 'vlan': str(VLAN_UNCHANGED + 1), - 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, - 'defaultRoute': False}, - NET_MISSING: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_MISSING, - 'vlan': str(VLAN_UNCHANGED + 2), - 'netmask': IP_MASK, 'bonding': BOND_MISSING}, + NET_UNCHANGED: { + 'bootproto': 'none', 'ipaddr': IP_ADD_UNCHANGED, + 'vlan': str(VLAN_UNCHANGED), 'netmask': IP_MASK, + 'bonding': BOND_UNCHANGED, 'defaultRoute': True}, + NET_CHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_NET_CHANGED, + 'vlan': str(VLAN_UNCHANGED + 1), + 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, + 'defaultRoute': False}, + NET_MISSING: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_MISSING, + 'vlan': str(VLAN_UNCHANGED + 2), + 'netmask': IP_MASK, 'bonding': BOND_MISSING}, } def _assert_all_nets_exist(): @@ -1632,18 +1632,18 @@ IP_ADDR_CHANGED = '204.0.0.102' IP_ADDR_ADDITIONAL = '204.0.0.102' nets = { - NET_MGMT: - {'bootproto': 'none', 'ipaddr': IP_MGMT, - 'netmask': IP_MASK, 'defaultRoute': True}, - NET_UNCHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_UNCHANGED, - 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, - 'defaultRoute': False}, - NET_CHANGED: - {'bootproto': 'none', - 'ipaddr': IP_ADDR_CHANGED, - 'netmask': IP_MASK, 'bonding': BOND_CHANGED}, + NET_MGMT: { + 'bootproto': 'none', 'ipaddr': IP_MGMT, + 'netmask': IP_MASK, 'defaultRoute': True}, + NET_UNCHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_UNCHANGED, + 'netmask': IP_MASK, 'bonding': BOND_UNCHANGED, + 'defaultRoute': False}, + NET_CHANGED: { + 'bootproto': 'none', + 'ipaddr': IP_ADDR_CHANGED, + 'netmask': IP_MASK, 'bonding': BOND_CHANGED}, } net_additional_attrs = { 'bootproto': 'none', 'ipaddr': IP_ADDR_ADDITIONAL, -- To view, visit https://gerrit.ovirt.org/44201 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iee593cee60690c43f82f30900f5c992bfba49000 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

3 9

Change in vdsm[master]: janitorial: vm: switch to response.error()
by fromani＠redhat.com 31 Jul '15

31 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: janitorial: vm: switch to response.error() ...................................................................... janitorial: vm: switch to response.error() response.error() offers a nicer and cleaner way to report error responses instead of the bare errCode['something'], even when we are fine with the default generic message. This patch makes use of response.error() all across vm.py. Change-Id: I3825faf7a144ef8973dee3cb9f9f0e52fabfc039 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 62 insertions(+), 62 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/38268/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index e7fbcf2..cd392c7 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1589,7 +1589,7 @@ if self.lastStatus in (vmstatus.MIGRATION_SOURCE, vmstatus.SAVING_STATE, vmstatus.DOWN): self.log.error('cannot cont while %s', self.lastStatus) - return errCode['unexpected'] + return response.error('unexpected') self._underlyingCont() self._setGuestCpuRunning(self._isDomainRunning(), guestCpuLocked=True) @@ -1638,7 +1638,7 @@ def shutdown(self, delay, message, reboot, timeout, force): if self.lastStatus == vmstatus.DOWN: - return errCode['noVM'] + return response.error('noVM') delay = int(delay) @@ -1913,13 +1913,13 @@ try: if self.isMigrating(): self.log.warning('vm already migrating') - return errCode['exist'] + return response.error('exist') if self.hasTransientDisks(): - return errCode['transientErr'] + return response.error('transientErr') # while we were blocking, another migrationSourceThread could have # taken self Down if self._lastStatus == vmstatus.DOWN: - return errCode['noVM'] + return response.error('noVM') self._migrationSourceThread = migration.SourceThread( self, **params) self._migrationSourceThread.start() @@ -1940,11 +1940,11 @@ return self._migrationSourceThread.status except libvirt.libvirtError as e: if e.get_error_code() == libvirt.VIR_ERR_OPERATION_INVALID: - return errCode['migCancelErr'] + return response.error('migCancelErr') raise except AttributeError: if self._dom is None: - return errCode['migCancelErr'] + return response.error('migCancelErr') raise finally: self._guestCpuLock.release() @@ -2317,7 +2317,7 @@ def hotplugNic(self, params): if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') nicParams = params['nic'] nic = vmdevices.network.Interface(self.conf, self.log, **nicParams) @@ -2336,7 +2336,7 @@ nicXml = hooks.after_nic_hotplug_fail( nicXml, self.conf, params=nic.custom) if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') return response.error('hotplugNic', e.message) else: # FIXME! We may have a problem here if vdsm dies right after @@ -2524,7 +2524,7 @@ response['vmList'] = self.status() return response else: - return errCode['updateDevice'] + return response.error('updateDevice') def updateDevice(self, params): if params.get('deviceType') == hwclass.NIC: @@ -2532,11 +2532,11 @@ elif params.get('deviceType') == hwclass.GRAPHICS: return self._updateGraphicsDevice(params) else: - return errCode['noimpl'] + return response.error('noimpl') def hotunplugNic(self, params): if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') nicParams = params['nic'] @@ -2583,7 +2583,7 @@ except libvirt.libvirtError as e: self.log.exception("Hotunplug failed") if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') # Restore NIC device in vm's conf and _devices if nicDev: with self._confLock: @@ -2602,7 +2602,7 @@ def setNumberOfCpus(self, numberOfCpus): if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') self.log.debug("Setting number of cpus to : %s", numberOfCpus) hooks.before_set_num_of_cpus() @@ -2612,7 +2612,7 @@ except libvirt.libvirtError as e: self.log.exception("setNumberOfCpus failed") if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') return response.error('setNumberOfCpusErr', e.message) self.conf['smp'] = str(numberOfCpus) @@ -2646,7 +2646,7 @@ """ if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') if not params: self.log.error("updateVmPolicy got an empty policy.") @@ -2710,7 +2710,7 @@ except libvirt.libvirtError as e: self.log.exception("updateVmPolicy failed") if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') else: return self._reportError(key='updateVmPolicyErr', msg=e.message) @@ -2797,7 +2797,7 @@ except libvirt.libvirtError as e: self.log.exception("setVmIoTune failed") if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') else: return self._reportError(key='updateIoTuneErr', msg=e.message) @@ -2860,7 +2860,7 @@ def hotplugDisk(self, params): if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') diskParams = params.get('drive', {}) diskParams['path'] = self.cif.prepareVolumePath(diskParams) @@ -2873,7 +2873,7 @@ drive = vmdevices.storage.Drive(self.conf, self.log, **diskParams) if drive.hasVolumeLeases: - return errCode['noimpl'] + return response.error('noimpl') driveXml = drive.getXML().toprettyxml(encoding='utf-8') # TODO: this is debug information. For 3.6.x we still need to @@ -2889,7 +2889,7 @@ self.log.exception("Hotplug failed") self.cif.teardownVolumePath(diskParams) if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') return response.error('hotplugDisk', e.message) else: # FIXME! We may have a problem here if vdsm dies right after @@ -2909,7 +2909,7 @@ def hotunplugDisk(self, params): if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') diskParams = params.get('drive', {}) diskParams['path'] = self.cif.prepareVolumePath(diskParams) @@ -2922,7 +2922,7 @@ return response.error('hotunplugDisk', "Disk not found") if drive.hasVolumeLeases: - return errCode['noimpl'] + return response.error('noimpl') driveXml = drive.getXML().toprettyxml(encoding='utf-8') # TODO: this is debug information. For 3.6.x we still need to @@ -2949,7 +2949,7 @@ except libvirt.libvirtError as e: self.log.exception("Hotunplug failed") if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - return errCode['noVM'] + return response.error('noVM') self._devices[hwclass.DISK].append(drive) # Restore disk device in vm's conf and _devices if diskDev: @@ -3239,7 +3239,7 @@ vmDrives = {} if self.isMigrating(): - return errCode['migInProgress'] + return response.error('migInProgress') for drive in snapDrives: baseDrv, tgetDrv = _normSnapDriveParams(drive) @@ -3260,15 +3260,15 @@ except LookupError: # The volume we want to snapshot doesn't exist self.log.error("The base volume doesn't exist: %s", baseDrv) - return errCode['snapshotErr'] + return response.error('snapshotErr') if vmDrive.hasVolumeLeases: self.log.error('disk %s has volume leases', vmDrive.name) - return errCode['noimpl'] + return response.error('noimpl') if vmDrive.transientDisk: self.log.error('disk %s is a transient disk', vmDrive.name) - return errCode['transientErr'] + return response.error('transientErr') vmDevName = vmDrive.name @@ -3300,7 +3300,7 @@ self.log.exception('unable to prepare the volume path for ' 'disk %s', vmDevName) _rollbackDrives(preparedDrives) - return errCode['snapshotErr'] + return response.error('snapshotErr') snapType = 'block' if vmDrives[vmDevName].blockDev else 'file' snapelem = _diskSnapshot(vmDevName, newDrives[vmDevName]["path"], @@ -3364,7 +3364,7 @@ self.log.exception("Unable to take snapshot") if memoryParams: self.cif.teardownVolumePath(memoryVol) - return errCode['snapshotErr'] + return response.error('snapshotErr') # We are padding the memory volume with block size of zeroes # because qemu-img truncates files such that their size is @@ -3439,20 +3439,20 @@ srcDrive = self._findDriveByUUIDs(srcDisk) except LookupError: self.log.error("Unable to find the disk for '%s'", srcDisk) - return errCode['imageErr'] + return response.error('imageErr') if srcDrive.hasVolumeLeases: - return errCode['noimpl'] + return response.error('noimpl') if srcDrive.transientDisk: - return errCode['transientErr'] + return response.error('transientErr') try: self._setDiskReplica(srcDrive, dstDisk) except Exception: self.log.error("Unable to set the replication for disk '%s' with " "destination '%s'", srcDrive.name, dstDisk) - return errCode['replicaErr'] + return response.error('replicaErr') dstDiskCopy = dstDisk.copy() @@ -3484,7 +3484,7 @@ except Exception: self.log.exception("Cannot complete the disk replication process") self._delDiskReplica(srcDrive) - return errCode['replicaErr'] + return response.error('replicaErr') if srcDrive.chunked: try: @@ -3502,16 +3502,16 @@ try: srcDrive = self._findDriveByUUIDs(srcDisk) except LookupError: - return errCode['imageErr'] + return response.error('imageErr') if srcDrive.hasVolumeLeases: - return errCode['noimpl'] + return response.error('noimpl') if srcDrive.transientDisk: - return errCode['transientErr'] + return response.error('transientErr') if not srcDrive.isDiskReplicationInProgress(): - return errCode['replicaErr'] + return response.error('replicaErr') # Looking for the replication blockJob info (checking its presence) blkJobInfo = self._dom.blockJobInfo(srcDrive.name, 0) @@ -3523,12 +3523,12 @@ # Making sure that we don't have any stale information self._delDiskReplica(srcDrive) - return errCode['replicaErr'] + return response.error('replicaErr') # Checking if we reached the replication mode ("mirroring" in libvirt # and qemu terms) if blkJobInfo['cur'] != blkJobInfo['end']: - return errCode['unavail'] + return response.error('unavail') dstDiskCopy = dstDisk.copy() @@ -3567,7 +3567,7 @@ # There is nothing we can do at this point other than logging self.log.exception("Unable to teardown the replication " "destination disk") - return errCode['changeDisk'] # Finally is evaluated + return response.error('changeDisk') # Finally is evaluated else: try: self.cif.teardownVolumePath(diskToTeardown) @@ -3595,7 +3595,7 @@ "Requested extension size %s for disk %s is smaller " "than the current size %s", newSizeBytes, drive.name, currentSize) - return errCode['resizeErr'] + return response.error('resizeErr') # Uncommit the current volume size (mark as in transaction) self.cif.irs.setVolumeSize(drive.domainID, drive.poolID, @@ -3608,7 +3608,7 @@ self.log.exception( "An error occurred while trying to extend the disk %s " "to size %s", drive.name, newSizeBytes) - return errCode['updateDevice'] + return response.error('updateDevice') finally: # In all cases we want to try and fix the size in the metadata. # Same as above, this is what libvirt would do, see BZ#963881 @@ -3650,7 +3650,7 @@ "Libvirt failed to notify the new size %s to the " "running VM, the change will be available at the ", "reboot", sizeRoundedBytes, exc_info=True) - return errCode['updateDevice'] + return response.error('updateDevice') return {'status': doneCode, 'size': str(sizeRoundedBytes)} @@ -3658,12 +3658,12 @@ try: newSizeBytes = int(newSizeBytes) except ValueError: - return errCode['resizeErr'] + return response.error('resizeErr') try: drive = self._findDriveByUUIDs(driveSpecs) except LookupError: - return errCode['imageErr'] + return response.error('imageErr') try: if drive.format == "cow": @@ -3673,7 +3673,7 @@ except Exception: self.log.exception("Unable to extend disk %s to size %s", drive.name, newSizeBytes) - return errCode['updateDevice'] + return response.error('updateDevice') def _onWatchdogEvent(self, action): def actionToString(action): @@ -3714,7 +3714,7 @@ try: path = self.cif.prepareVolumePath(drivespec) except VolumeError: - return errCode['imageErr'] + return response.error('imageErr') diskelem = vmxml.Element('disk', type='file', device=vmDev) diskelem.appendChildWithArgs('source', file=path) diskelem.appendChildWithArgs('target', dev=blockdev) @@ -3725,7 +3725,7 @@ except Exception: self.log.debug("updateDeviceFlags failed", exc_info=True) self.cif.teardownVolumePath(drivespec) - return errCode['changeDisk'] + return response.error('changeDisk') if vmDev in self.conf: self.cif.teardownVolumePath(self.conf[vmDev]) @@ -3902,7 +3902,7 @@ self.conf['vmId'], exc_info=True) if e.get_error_code() == libvirt.VIR_ERR_OPERATION_FAILED: return self._destroyVmForceful() - return errCode['destroyErr'] + return response.error('destroyErr') return {'status': doneCode} def _destroyVmForceful(self): @@ -3911,7 +3911,7 @@ except libvirt.libvirtError: self.log.warning("Failed to destroy VM '%s'", self.conf['vmId'], exc_info=True) - return errCode['destroyErr'] + return response.error('destroyErr') return {'status': doneCode} def deleteVm(self): @@ -4673,7 +4673,7 @@ def merge(self, driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID): if not caps.getLiveMergeSupport(): self.log.error("Live merge is not supported on this host") - return errCode['mergeErr'] + return response.error('mergeErr') bandwidth = int(bandwidth) if jobUUID is None: @@ -4682,14 +4682,14 @@ try: drive = self._findDriveByUUIDs(driveSpec) except LookupError: - return errCode['imageErr'] + return response.error('imageErr') # Check that libvirt exposes full volume chain information chains = self._driveGetActualVolumeChain([drive]) if drive['alias'] not in chains: self.log.error("merge: libvirt does not support volume chain " "monitoring. Unable to perform live merge.") - return errCode['mergeErr'] + return response.error('mergeErr') base = top = None for v in drive.volumeChain: @@ -4699,10 +4699,10 @@ top = v['path'] if base is None: self.log.error("merge: base volume '%s' not found", baseVolUUID) - return errCode['mergeErr'] + return response.error('mergeErr') if top is None: self.log.error("merge: top volume '%s' not found", topVolUUID) - return errCode['mergeErr'] + return response.error('mergeErr') # If base is a shared volume then we cannot allow a merge. Otherwise # We'd corrupt the shared volume for other users. @@ -4710,10 +4710,10 @@ drive.imageID, baseVolUUID) if res['status']['code'] != 0: self.log.error("Unable to get volume info for '%s'", baseVolUUID) - return errCode['mergeErr'] + return response.error('mergeErr') if res['info']['voltype'] == 'SHARED': self.log.error("merge: Refusing to merge into a shared volume") - return errCode['mergeErr'] + return response.error('mergeErr') # Indicate that we expect libvirt to maintain the relative paths of # backing files. This is necessary to ensure that a volume chain is @@ -4738,7 +4738,7 @@ if res['status']['code'] != 0: self.log.error("Unable to get volume info for '%s'", topVolUUID) - return errCode['mergeErr'] + return response.error('mergeErr') topSize = int(res['info']['apparentsize']) # Take the jobs lock here to protect the new job we are tracking from @@ -4749,7 +4749,7 @@ 'commit') except BlockJobExistsError: self.log.error("A block job is already active on this disk") - return errCode['mergeErr'] + return response.error('mergeErr') self.log.info("Starting merge with jobUUID='%s'", jobUUID) try: @@ -4760,7 +4760,7 @@ except (RuntimeError, libvirt.libvirtError): self.log.exception("Live merge failed (job: %s)", jobUUID) self.untrackBlockJob(jobUUID) - return errCode['mergeErr'] + return response.error('mergeErr') # blockCommit will cause data to be written into the base volume. # Perform an initial extension to ensure there is enough space to -- To view, visit https://gerrit.ovirt.org/38268 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3825faf7a144ef8973dee3cb9f9f0e52fabfc039 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

6 61

Change in vdsm[master]: vdsm-reg: remove the sub-project
by automation＠ovirt.org 31 Jul '15

31 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 4: * Update tracker::#1231379::OK * Set MODIFIED::bug 1231379::::#1231379::::IGNORE, not oVirt prod but Red Hat Enterprise Virtualization Manager -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 4 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has submitted this change and it was merged. Change subject: vdsm-reg: remove the sub-project ...................................................................... vdsm-reg: remove the sub-project As vdsm-reg is deprecated. At moment vdsm-tool has register verb which executes the registration and there is ovirt-register project. oVirt Node doesn't pull vdsm-reg anymore. Based on that, this patch removes vdsm-reg subproject to avoid dead code inside vdsm tree. Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1231379 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> Reviewed-on: https://gerrit.ovirt.org/43254 Reviewed-by: Dan Kenigsberg <danken(a)redhat.com> Continuous-Integration: Dan Kenigsberg <danken(a)redhat.com> --- M .gitignore M Makefile.am M configure.ac M debian/vdsm.install M vdsm.spec.in M vdsm/Makefile.am D vdsm/addNetwork D vdsm/delNetwork D vdsm_reg/Makefile.am D vdsm_reg/config.py.in D vdsm_reg/createDaemon.py D vdsm_reg/deployUtil.py.in D vdsm_reg/logger.conf D vdsm_reg/register-to-engine.py D vdsm_reg/save-config D vdsm_reg/vdsm-complete D vdsm_reg/vdsm-config D vdsm_reg/vdsm-gen-cert D vdsm_reg/vdsm-reg-cmdline D vdsm_reg/vdsm-reg-logrotate D vdsm_reg/vdsm-reg-logrotate.conf D vdsm_reg/vdsm-reg-setup.in D vdsm_reg/vdsm-reg.8 D vdsm_reg/vdsm-reg.conf.in D vdsm_reg/vdsm-reg.init D vdsm_reg/vdsm-reg.service D vdsm_reg/vdsm-upgrade 27 files changed, 0 insertions(+), 3,665 deletions(-) Approvals: Douglas Schilling Landgraf: Verified Dan Kenigsberg: Looks good to me, approved; Passed CI tests Objections: Douglas Schilling Landgraf: I would prefer that you didn't submit this -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 4 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 3: Continuous-Integration+1 merging per QE/PM approval. -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 3: Code-Review+2 vdsm-store-net-config is still much needed. Do not drop it. -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tcTests: Fix problem of left over bridges
by rmohr＠redhat.com 31 Jul '15

31 Jul '15

Roman Mohr has uploaded a new change for review. Change subject: tcTests: Fix problem of left over bridges ...................................................................... tcTests: Fix problem of left over bridges For unknown reason the command which should shut down the bridge fails to do so, but reports back that it successfully shut down the bridge. Therefore the following attempt to delete the bridge can just fail. It appears to be enough to just retry the shutdown and deletion command to work around that problem. Change-Id: Icd0d8f20283fdf80977bd1f78f5b2ac62f7e83c8 Signed-off-by: Roman Mohr <rmohr(a)redhat.com> --- M tests/tcTests.py 1 file changed, 9 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/28/44028/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index 6f0c37e..94f0d05 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -104,8 +104,15 @@ self._ifUp() def delDevice(self): - self._ifDown() - check_call([EXT_BRCTL, 'delbr', self.devName]) + try: + self._ifDown() + check_call([EXT_BRCTL, 'delbr', self.devName]) + except ExecError: + # Sometimes the bridge is not shut down, although ip reports so. + # Just retrying the shutdown and the removal once is enough to + # work around that. + self._ifDown() + check_call([EXT_BRCTL, 'delbr', self.devName]) def addIf(self, dev): check_call([EXT_BRCTL, 'addif', self.devName, dev]) -- To view, visit https://gerrit.ovirt.org/44028 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icd0d8f20283fdf80977bd1f78f5b2ac62f7e83c8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Roman Mohr <rmohr(a)redhat.com>

5 11

Change in vdsm[master]: virt: let Engine start a VM on an UNKOWN OS
by Dan Kenigsberg 31 Jul '15

31 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: virt: let Engine start a VM on an UNKOWN OS ...................................................................... virt: let Engine start a VM on an UNKOWN OS This superflouse validation was introduced as a "bandage" to avoid https://bugzilla.redhat.com/716705, where Vdsm failed to recognize VMs with "UNKOWN" os injected to their bios. Now that we recognize VMs based on guest channels existence, we can let Engine start a VM on whatever host it deems valid. Backward compatibility caveat: if a VM is started on an UNKOWN os, and then migrated to an old Vdsm (pre ovirt-3.4.2), it would be killed when that old Vdsm is to be restarted. It's not probable that someone would start a long-running VM on an UNKOWN platform, we may want to wait with this patch until ovirt-3.3 is deprecated. Change-Id: Iee7b3913e76923043daab8fb85854c3290208237 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M vdsm/API.py 1 file changed, 0 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/32459/1 diff --git a/vdsm/API.py b/vdsm/API.py index 8d1bca6..d6f318f 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -237,11 +237,6 @@ vmParams['vmId']) vmParams['volatileFloppy'] = True - if caps.osversion()['name'] == caps.OSName.UNKNOWN: - return {'status': {'code': errCode['createErr'] - ['status']['code'], - 'message': 'Unknown host operating system'}} - if 'sysprepInf' in vmParams: if not self._createSysprepFloppyFromInf(vmParams['sysprepInf'], vmParams['floppy']): -- To view, visit http://gerrit.ovirt.org/32459 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iee7b3913e76923043daab8fb85854c3290208237 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

4 10

Change in vdsm[master]: caps: report capabilities on UNKNOWN OS
by fromani＠redhat.com 31 Jul '15

31 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: caps: report capabilities on UNKNOWN OS ...................................................................... caps: report capabilities on UNKNOWN OS The capabilities reporting (getCapabilities API) does logic on libvirt version to report, or not, the RNG source availability. The libvirt version is found using the package version, but VDSM knows only how to interact with RPM and APT based system, so it is clueless when it runs on UNKNOWN OS. While it is still OK (from a VDSM PoV) to pass incomplete information about available packages to Engine, this makes the aforementioned RNG check logic fail. Effects of this failure: - Engine cannot connect to the VDS - stacktrace in the VDSM logs (excerpt follows): File "/usr/share/vdsm/API.py", line 1314, in getCapabilities c = caps.get() File "/usr/share/vdsm/caps.py", line 665, in get '-'.join((caps['packages2']['libvirt']['version'], KeyError: 'libvirt' Thread-14::DEBUG::2015-07-31 11:16:52,699::stompreactor::305::yajsonrpc.StompServer::(send) Sending response This patch makes the getCapabilities API work again on UNKNOWN OS. To be conservative, if we don't know about libvirt, we report no RNG sources. Please note that this prevents UNKONWN OS host to join cluster with RNG sources configured. Change-Id: Iaacb20dfd9b3850f477d8b45f221dc2ec186d0c2 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/caps.py 1 file changed, 11 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/32/44232/1 diff --git a/vdsm/caps.py b/vdsm/caps.py index 0c97a96..b443fab 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -660,16 +660,21 @@ caps['guestOverhead'] = config.get('vars', 'guest_ram_overhead') # Verify that our libvirt supports virtio RNG (since 10.0.2-31) - libvirtVer = LooseVersion( - '-'.join((caps['packages2']['libvirt']['version'], - caps['packages2']['libvirt']['release']))) requiredVer = LooseVersion('0.10.2-31') - - if libvirtVer >= requiredVer: - caps['rngSources'] = _getRngSources() + if 'libvirt' not in caps['packages2']: + libvirtVer = None else: + libvirtVer = LooseVersion( + '-'.join((caps['packages2']['libvirt']['version'], + caps['packages2']['libvirt']['release']))) + + if libvirtVer is None: + logging.debug('VirtioRNG DISABLED: unknown libvirt version') + elif libvirtVer < requiredVer: logging.debug('VirtioRNG DISABLED: libvirt version %s required >= %s', libvirtVer, requiredVer) + else: + caps['rngSources'] = _getRngSources() caps['numaNodes'] = getNumaTopology() caps['numaNodeDistance'] = getNumaNodeDistance() -- To view, visit https://gerrit.ovirt.org/44232 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaacb20dfd9b3850f477d8b45f221dc2ec186d0c2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 5

Change in vdsm[master]: change downloaded ca pem filename
by Douglas Schilling Landgraf 31 Jul '15

31 Jul '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: change downloaded ca pem filename ...................................................................... change downloaded ca pem filename To avoid conflicts with ovirt-engine ca.pem name. Change-Id: I74669f9eb3a166af36fc4568881902688d28f8b3 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M lib/vdsm/tool/register.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/37/42837/1 diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py index 3ac3fc6..7c6f7ed 100644 --- a/lib/vdsm/tool/register.py +++ b/lib/vdsm/tool/register.py @@ -118,7 +118,7 @@ self.logger.debug("VDSM Port: {sp}".format(sp=self.vdsm_port)) self.ca_dir = "/etc/pki/ovirt-engine/" - self.ca_engine = "{d}{f}".format(d=self.ca_dir, f="ca.pem") + self.ca_engine = "{d}{f}".format(d=self.ca_dir, f="cert_ca_engine.pem") self.logger.debug("Engine CA: {ca}".format(ca=self.ca_engine)) def get_protocol(self): @@ -292,7 +292,7 @@ Download CA from Engine and save self.ca_engine """ self.logger.info("Collecting CA data from Engine...") - # If engine CA dir doesnt exist create it and download the ca.pem + # If engine CA dir doesnt exist create and download cert_ca_engine.pem temp_ca_file = None if os.path.exists(self.ca_engine): calculated_fprint = self._calculate_fingerprint(self.ca_engine) -- To view, visit https://gerrit.ovirt.org/42837 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I74669f9eb3a166af36fc4568881902688d28f8b3 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

2 5

Change in vdsm[master]: vdsm-tool: remove register verb
by Douglas Schilling Landgraf 31 Jul '15

31 Jul '15

Douglas Schilling Landgraf has uploaded a new change for review. Change subject: vdsm-tool: remove register verb ...................................................................... vdsm-tool: remove register verb git://gerrit.ovirt.org/ovirt-register provides the generic registration which should deprecate the vdsm-tool register as the registration must be triggered without VDSM. Change-Id: I1c2e06b42bc8211ce900316498f28c9214044bd7 Signed-off-by: Douglas Schilling Landgraf <dougsland(a)redhat.com> --- M debian/vdsm-python.install M lib/vdsm/tool/Makefile.am D lib/vdsm/tool/register.py M vdsm.spec.in 4 files changed, 0 insertions(+), 553 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/43921/1 diff --git a/debian/vdsm-python.install b/debian/vdsm-python.install index d9402c3..b74d0e9 100644 --- a/debian/vdsm-python.install +++ b/debian/vdsm-python.install @@ -37,7 +37,6 @@ ./usr/lib/python2.7/dist-packages/vdsm/tool/configurator.py ./usr/lib/python2.7/dist-packages/vdsm/tool/nwfilter.py ./usr/lib/python2.7/dist-packages/vdsm/tool/passwd.py -./usr/lib/python2.7/dist-packages/vdsm/tool/register.py ./usr/lib/python2.7/dist-packages/vdsm/tool/restore_nets.py ./usr/lib/python2.7/dist-packages/vdsm/tool/service.py ./usr/lib/python2.7/dist-packages/vdsm/tool/transient.py diff --git a/lib/vdsm/tool/Makefile.am b/lib/vdsm/tool/Makefile.am index c8979ca..e4afc98 100644 --- a/lib/vdsm/tool/Makefile.am +++ b/lib/vdsm/tool/Makefile.am @@ -37,7 +37,6 @@ nwfilter.py \ configfile.py \ configurator.py \ - register.py \ restore_nets.py \ service.py \ transient.py \ diff --git a/lib/vdsm/tool/register.py b/lib/vdsm/tool/register.py deleted file mode 100644 index 31121c0..0000000 --- a/lib/vdsm/tool/register.py +++ /dev/null @@ -1,550 +0,0 @@ -# Copyright (C) 2012-2015 Red Hat, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; version 2 of the License. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, -# MA 02110-1301, USA. A copy of the GNU General Public License is -# also available at http://www.gnu.org/copyleft/gpl.html. -from __future__ import absolute_import -import argparse -import getpass -import hashlib -import logging -import os -import pwd -import socket -import ssl -import sys -import tempfile - -import requests -import selinux - -from . import expose -from .. import utils - -from vdsm.utils import getHostUUID, tobool - - -class Register(object): - - def __init__(self, engine_fqdn, engine_https_port=None, - fingerprint=None, ssh_port=None, - ssh_user=None, check_fqdn=True, - vdsm_port=None, node_fqdn=None, - node_name=None): - """ - Attributes: - - engine_fqdn - Engine FQDN - engine_https_port - Engine http port - fingeprint - Fingerprint to be validated - ssh_user - SSH user that will establish the connection - from Engine - ssh_port - Port of ssh daemon is running - check_fqdn - Validate Engine FQDN against CA (True or False) - Default is TRUE - vdsm_port - VDSM listen port - node_fqdn - Specify node FQDN - node_name - Specify node name - """ - self.logger = self._set_logger() - self.logger.debug("=======================================") - self.logger.debug("Registering the node") - self.logger.debug("=======================================") - self.logger.debug("Received the following attributes:") - - self.engine_fqdn = engine_fqdn - self.logger.debug("Engine FQDN: {fqdn}".format(fqdn=self.engine_fqdn)) - - self.engine_url = "https://{e}".format(e=engine_fqdn) - if engine_https_port is None: - self.engine_port = "443" - else: - self.engine_port = engine_https_port - self.engine_url = "https://{e}:{p}".format(e=self.engine_fqdn, - p=self.engine_port) - - self.logger.debug("Engine URL: {url}".format(url=self.engine_url)) - self.logger.debug("Engine https port: {hp}".format( - hp=self.engine_port)) - - if check_fqdn is None: - self.check_fqdn = True - else: - self.check_fqdn = tobool(check_fqdn) - self.logger.debug("Check FQDN: {v}".format(v=self.check_fqdn)) - - self.fprint = fingerprint - self.logger.debug("Fingerprint: {fp}".format(fp=self.fprint)) - - if node_fqdn is None: - self.node_fqdn = socket.gethostname() - else: - self.node_fqdn = node_fqdn - self.logger.debug("Node FQDN: {nf}".format(nf=self.node_fqdn)) - - if node_name is None: - self.node_name = socket.gethostname().split(".")[0] - else: - self.node_name = node_name - self.logger.debug("Node name: {na}".format(na=self.node_name)) - - if ssh_user is None: - self.ssh_user = getpass.getuser() - else: - self.ssh_user = ssh_user - self.logger.debug("SSH User: {su}".format(su=self.ssh_user)) - - if ssh_port is None: - self.ssh_port = "22" - else: - self.ssh_port = ssh_port - self.logger.debug("SSH Port: {sp}".format(sp=self.ssh_port)) - - if vdsm_port is None: - self.vdsm_port = "54321" - else: - self.vdsm_port = vdsm_port - self.logger.debug("VDSM Port: {sp}".format(sp=self.vdsm_port)) - - self.ca_dir = "/etc/pki/ovirt-engine/" - self.ca_engine = "{d}{f}".format(d=self.ca_dir, f="ca.pem") - self.logger.debug("Engine CA: {ca}".format(ca=self.ca_engine)) - - def get_protocol(self): - """ - Determine if Engine is running in registration - protocol version legacy or service - REQUIRED_FOR: Engine 3.3 - """ - - self.logger.info("Identifying the registration protocol...") - - ucmd = "/ovirt-engine/services/host-register?version=1&command=" - __GET_VERSION = "https://{e}{u}{c}".format(e=self.engine_fqdn, - u=ucmd, - c="get-version") - - res = requests.get(__GET_VERSION, verify=False) - if res.status_code != 200: - self.reg_protocol = "legacy" - self.url_CA = self.engine_url - - self.url_ssh_key = "{e}{k}".format(e=self.engine_url, - k="/engine.ssh.key.txt") - - ureg = "/OvirtEngineWeb/register?vds_ip={fqdn}" \ - "&vds_name={name}&port={mp}".format(fqdn=self.node_fqdn, - name=self.node_name, - mp=self.vdsm_port) - - self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) - else: - self.reg_protocol = "service" - - self.url_CA = "{e}{uc}{c}".format(e=self.engine_url, - uc=ucmd, - c="get-pki-trust") - - self.url_ssh_key = "{e}{uc}{c}".format(e=self.engine_url, - uc=ucmd, - c="get-ssh-trust") - - ureg = "{uc}register&name={name}&address={fqdn}&sshUser={sshu}&" \ - "sshPort={sshp}&port={mp}".format(uc=ucmd, - name=self.node_name, - fqdn=self.node_fqdn, - sshu=self.ssh_user, - sshp=self.ssh_port, - mp=self.vdsm_port) - - self.url_reg = "{e}{u}".format(e=self.engine_url, u=ureg) - - self.logger.info("Registration procotol selected: {p}".format( - p=self.reg_protocol)) - - self.logger.debug("Download CA via: {u}".format(u=self.url_CA)) - self.logger.debug("Download SSH via: {u}".format(u=self.url_ssh_key)) - - def _set_logger(self): - """ - The logging settings - Saving log in: /var/log/vdsm/register.log - """ - logger = logging.getLogger(__name__) - logger.setLevel(logging.DEBUG) - logger.propagate = False - - fh = logging.FileHandler("/var/log/vdsm/register.log") - fh.setLevel(logging.DEBUG) - debug_fmt = logging.Formatter("%(asctime)s %(message)s", - "%m/%d/%Y %I:%M:%S %p") - - ih = logging.StreamHandler(stream=sys.stdout) - ih.setLevel(logging.INFO) - info_fmt = logging.Formatter("%(message)s", - "%m/%d/%Y %I:%M:%S %p") - - fh.setFormatter(debug_fmt) - ih.setFormatter(info_fmt) - - logger.addHandler(fh) - logger.addHandler(ih) - - logging.captureWarnings(True) - - return logger - - def _execute_http_request(self, url, cert_validation=True): - """ - Execute http requests - url -- URL to be requested - cert_validation -- SSL cert will be verified - - Returns: Content of http request - """ - if self.check_fqdn: - cert_validation = self.ca_engine - else: - cert_validation = False - - res = requests.get("{u}".format(u=url), verify=cert_validation) - if res.status_code != 200: - raise requests.RequestException( - "http response was non OK, code {r}".format(r=res.status_code) - ) - - return res.content - - def _silent_restorecon(self, path): - """ - Execute selinux restorecon cmd to determined file - - Args - path -- full path to file - """ - - try: - if selinux.is_selinux_enabled(): - selinux.restorecon(path) - except: - self.logger.error("restorecon %s failed" % path, exc_info=True) - - def _calculate_fingerprint(self, cert): - """ - Calculate fingerprint of certificate - - Args - cert -- certificate file to be calculated the fingerprint - - Returns - The fingerprint - """ - - with open(cert, 'r') as f: - cert = f.read() - - fp = hashlib.sha1(ssl.PEM_cert_to_DER_cert(cert)).hexdigest() - fp = ':'.join(fp[pos:pos + 2] for pos in range(0, len(fp), 2)) - - return fp - - def host_uuid(self): - """ - Determine host UUID and if there is no existing /etc/vdsm/vdsm.id - it will genereate UUID and save/persist in /etc/vdsm/vdsm.id - """ - if self.reg_protocol == "legacy": - # REQUIRED_FOR: Engine 3.3 - # The legacy version uses the format: UUID_MACADDRESS - self.uuid = getHostUUID(legacy=True) - self.url_reg += "&vds_unique_id={u}".format(u=self.uuid) - else: - # Non legacy version uses the format: UUID - self.uuid = getHostUUID(legacy=False) - self.url_reg += "&uniqueId={u}".format(u=self.uuid) - - self.logger.debug("Registration via: {u}".format(u=self.url_reg)) - - __VDSM_ID = "/etc/vdsm/vdsm.id" - if not os.path.exists(__VDSM_ID): - with open(__VDSM_ID, 'w') as f: - f.write(self.uuid) - - if utils.isOvirtNode(): - from ovirt.node.utils.fs import Config - Config().persist(__VDSM_ID) - - self.logger.info("Host UUID: {u}".format(u=self.uuid)) - - def download_ca(self): - """ - Download CA from Engine and save self.ca_engine - """ - self.logger.info("Collecting CA data from Engine...") - # If engine CA dir doesnt exist create it and download the ca.pem - temp_ca_file = None - if os.path.exists(self.ca_engine): - calculated_fprint = self._calculate_fingerprint(self.ca_engine) - else: - if not os.path.exists(self.ca_dir): - os.makedirs(self.ca_dir, 0o755) - self._silent_restorecon(self.ca_dir) - if utils.isOvirtNode(): - from ovirt.node.utils.fs import Config - Config().persist(self.ca_dir) - - if self.reg_protocol == "legacy": - # REQUIRED_FOR: Engine 3.3 - res = ssl.get_server_certificate( - (self.engine_fqdn, int(self.engine_port)) - ) - else: - res = self._execute_http_request(self.url_CA, - cert_validation=False) - - with tempfile.NamedTemporaryFile( - dir=os.path.dirname(self.ca_dir), - delete=False - ) as f: - f.write(res) - - calculated_fprint = self._calculate_fingerprint(f.name) - temp_ca_file = True - - if self.fprint and self.fprint.lower() != calculated_fprint.lower(): - msg = "The fingeprints doesn't match:\n" \ - "Calculated fingerprint: [{c}]\n" \ - "Attribute fingerprint: [{a}]".format(c=calculated_fprint, - a=self.fprint) - - self.logger.debug(msg) - if temp_ca_file: - os.unlink(f.name) - raise RuntimeError(msg) - - if temp_ca_file: - os.rename(f.name, self.ca_engine) - - self.fprint = calculated_fprint - self.logger.info("Calculated fingerprint: {f}".format( - f=self.fprint)) - - if utils.isOvirtNode(): - from ovirt.node.utils.fs import Config - Config().persist(self.ca_engine) - - def download_ssh(self): - """ - Download ssh authorized keys and save it in the node - """ - self.logger.info("Collecting ssh pub key data...") - _uid = pwd.getpwnam(self.ssh_user).pw_uid - _auth_keys_dir = pwd.getpwuid(_uid).pw_dir + "/.ssh" - _auth_keys = _auth_keys_dir + "/authorized_keys" - self.logger.debug("auth_key is located {f}".format(f=_auth_keys)) - - if not os.path.exists(_auth_keys_dir): - os.makedirs(_auth_keys_dir, 0o700) - self._silent_restorecon(_auth_keys_dir) - if utils.isOvirtNode(): - from ovirt.node.utils.fs import Config - Config().persist(_auth_keys_dir) - os.chown(_auth_keys_dir, _uid, _uid) - - res = self._execute_http_request(self.url_ssh_key) - with tempfile.NamedTemporaryFile( - dir=_auth_keys_dir, - delete=False - ) as f: - f.write(res) - - # If ssh key is new append it into autorized_keys - with open(f.name, "r") as f_ro: - content = f_ro.read() - with open(_auth_keys, "a+") as f_w: - if content not in f_w.read(): - f_w.write(content) - os.chmod(_auth_keys, 0o600) - self._silent_restorecon(_auth_keys) - os.chown(_auth_keys, _uid, _uid) - - os.unlink(f.name) - if utils.isOvirtNode(): - from ovirt.node.utils.fs import Config - Config().persist(_auth_keys) - - def execute_registration(self): - """ - Trigger the registration command against Engine - """ - self._execute_http_request(self.url_reg) - self.logger.info("Registration completed, host is pending approval" - " on Engine: {e}".format(e=self.engine_fqdn)) - - -@expose("register") -def main(*args): - ''' - A tool which register the node against Engine - Note: This comment is required by vdsm-tool which - looks for a doc string. - ''' - parser = argparse.ArgumentParser( - formatter_class=argparse.RawTextHelpFormatter, - description='Tool to register node to Engine', - epilog='Example of use:\n%(prog)s ' - '--engine-fqdn engine.mydomain' - ) - - parser.add_argument( - '--node-fqdn', - help="Define node FQDN or IP address." - " If not provided, will be used system host name", - ) - - parser.add_argument( - '--node-name', - help="Define node name." - " If not provided, will be used system short host name" - " (the name before the first dot in the system host name)", - ) - - parser.add_argument( - '--engine-fqdn', - help="Engine FQDN (See also: --check-fqdn)", - required=True - ) - - parser.add_argument( - '--engine-https-port', - help="Define engine https port." - " If not provided, will be used 443", - ) - - parser.add_argument( - '--ssh-user', - help="SSH username to establish the connection with Engine. " - "If not provided, the user which is " - "executing the script will catch and used", - ) - - parser.add_argument( - '--ssh-port', - help="SSH port to establish the connection with Engine " - "If not provided, the script will use the default " - "SSH port 22" - ) - - parser.add_argument( - '--check-fqdn', - help="Disable or Enable FQDN check for Engine CA, this option " - "is enabled by default (Use: True or False)", - ) - - parser.add_argument( - '--fingerprint', - help="Specify an existing fingerprint to be validated against " - "Engine CA fingerprint", - ) - - parser.add_argument( - '--vdsm-port', - help="Specify the listen port of VDSM" - " If not provided, will be used the default 54321", - ) - - # Using [1:] to remove the 'register' option from arguments - # and avoid vdsm-tool recognize it as an unknown option - args = parser.parse_args(args=args[1:]) - - reg = Register(engine_fqdn=args.engine_fqdn, - engine_https_port=args.engine_https_port, - vdsm_port=args.vdsm_port, - node_fqdn=args.node_fqdn, - node_name=args.node_name, - ssh_user=args.ssh_user, - ssh_port=args.ssh_port, - fingerprint=args.fingerprint, - check_fqdn=args.check_fqdn) - - try: - reg.get_protocol() - reg.host_uuid() - reg.download_ca() - reg.download_ssh() - reg.execute_registration() - except: - reg.logger.exception("Cannot connect to engine. {f} matches " - "the FQDN of Engine?".format(f=args.engine_fqdn)) - return 1 - - return 0 - -if __name__ == '__main__': - sys.exit(main()) - -""" -Registration schema: - -UUID -========= - - If there is UUID already generated for the system will be - available in /etc/vdsm/vdsm.id - - - In case, there is no UUID, use auxiliary function from VDSM - to generate it and store in /etc/vdsm/vdsm.id - -Legacy reg: -============ - - REQUIRED_FOR: Engine 3.3 - - - Process UUID - - - Download CA via - https://ENGINE_FQDN - - - Download ssh pub key - https://ENGINE_FQDN/engine.ssh.key.txt - - - Register via URL: - (Original .NET version and earlier Linux versions) - https://ENGINE_FQDN/RHEVManagerWeb/VdsAutoRegistration.aspx - ?vds_ip=NODE_FQDN_OR_IP&vds_name=NODE_NAME - &vds_unique_id=NODE_UUID&port=54321 - - or - - https://ENGINE_FQDN/OvirtEngineWeb/register?vds_ip=NODE_FQDN_OR_IP - &vds_name=NODE_NAME - &vds_unique_id=NODE_UUID&port=54321 - -Service reg: -============ - - REQUIRED_FOR: Engine >= 3.4 - - - Process UUID - - - Download CA via get-pki-trust URL - https://ENGINE_FQDN/ovirt-engine/services/host-register?version=1 - &command=get-pki-trust - - - Download ssh pub key via get-ssh-trust URL - https://ENGINE_FQDN/ovirt-engine/services/host-register?version=1 - &command=get-ssh-trust - - - Register via URL: - https://ENGINE_FQDN/ovirt-engine/services/host-register?version=1 - &command=register&name=NODE_NAME&address=NO_FQDN_OR_IP - &uniqueId=NODE_UUID&sshUser=SSH_USERNAME&sshPort=SSHD_PORT -""" diff --git a/vdsm.spec.in b/vdsm.spec.in index 4772416..8b91c91 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -1106,7 +1106,6 @@ %{python_sitelib}/%{vdsm_name}/tool/configurators/sebool.py* %{python_sitelib}/%{vdsm_name}/tool/configurators/multipath.py* %{python_sitelib}/%{vdsm_name}/tool/dump_volume_chains.py* -%{python_sitelib}/%{vdsm_name}/tool/register.py* %{python_sitelib}/%{vdsm_name}/tool/restore_nets.py* %{python_sitelib}/%{vdsm_name}/tool/service.py* %{python_sitelib}/%{vdsm_name}/tool/transient.py* -- To view, visit https://gerrit.ovirt.org/43921 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1c2e06b42bc8211ce900316498f28c9214044bd7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com>

3 7

Change in vdsm[master]: virt: avoid spam logs if sampling hollow VMs
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: virt: avoid spam logs if sampling hollow VMs ...................................................................... virt: avoid spam logs if sampling hollow VMs If the recover of a VM fails, the VM may end up being detected as existing by VDSM, while it is actually down. This is of course bad, and it is not something that periodic operations or sampling can fix. However, they can help in making not things worse by spamming the logs with File "/usr/lib64/python2.7/site-packages/libvirt.py", line 938, in controlInfo if ret is None: raise libvirtError ('virDomainGetControlInfo() failed', dom=self) libvirtError: Requested operation is not valid: domain is not running To prevent this spam, we add another libvirt exception to the whitelist of "known unharmful" error codes in Vm.isDomainReadyForCommands(). Change-Id: If130a4b31016e971ac9972b67ba5d3fd70de5b7d Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/06/41406/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 8983965..815f1a7 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -2595,8 +2595,10 @@ # to avoid racy checks. return False except libvirt.libvirtError as e: - if e.get_error_code() == libvirt.VIR_ERR_NO_DOMAIN: - # same as AttributeError above: possible race on shutdown + if e.get_error_code() in ( + libvirt.VIR_ERR_NO_DOMAIN, # possible race on shutdown + libvirt.VIR_ERR_OPERATION_INVALID, # dom misdetected up + ): return False else: raise -- To view, visit https://gerrit.ovirt.org/41406 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If130a4b31016e971ac9972b67ba5d3fd70de5b7d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 16

Change in vdsm[master]: executor: allow caller to resize TaskQueue
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: executor: allow caller to resize TaskQueue ...................................................................... executor: allow caller to resize TaskQueue To prevent having an excessive amount of pending work items, executor._TaskQueue has an hard limit of tasks to accept, which is enforced at every put(). This limit is arbitrary, and it is set only at startup. We want to be able to tune this value at runtime, so this patch exports a simple resize() operation to change the threshold inside _TaskQueue. Change-Id: I8df4d8bbc599fa2de1dc17de8fbd2726809f5de3 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M lib/vdsm/executor.py 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/58/41658/1 diff --git a/lib/vdsm/executor.py b/lib/vdsm/executor.py index fc7880e..b3574f2 100644 --- a/lib/vdsm/executor.py +++ b/lib/vdsm/executor.py @@ -97,6 +97,9 @@ raise NotRunning() self._tasks.put((callable, timeout)) + def resize(self, max_tasks): + self._tasks.resize(max_tasks) + # Serving workers def _worker_discarded(self, worker): @@ -253,3 +256,8 @@ def clear(self): self._tasks.clear() + + def resize(self, max_tasks): + if max_tasks < 1: + raise ValueError + self._max_tasks = max_tasks -- To view, visit https://gerrit.ovirt.org/41658 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8df4d8bbc599fa2de1dc17de8fbd2726809f5de3 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 19

Change in vdsm[master]: periodic: virt: do not poll storage in migrations
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: periodic: virt: do not poll storage in migrations ...................................................................... periodic: virt: do not poll storage in migrations If we monitor storage during migrations, we get harmless but scary warning in the logs. This patch avoids to monitor storage during migrations to silence logs. No functionality is lost since monitoring doesn't do its proper job anyway. Change-Id: I43ec1e64700020b7d143e22c4c3d8bdceb57fb18 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/periodic.py 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/07/40507/1 diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py index 8cc5a21..7354926 100644 --- a/vdsm/virt/periodic.py +++ b/vdsm/virt/periodic.py @@ -264,7 +264,8 @@ @property def required(self): # Avoid queries from storage during recovery process - return self._vm.isDisksStatsCollectionEnabled() + return (not self._vm.isMigrating() and + self._vm.isDisksStatsCollectionEnabled()) @property def runnable(self): @@ -322,7 +323,8 @@ @property def required(self): # Avoid queries from storage during recovery process - return self._vm.isDisksStatsCollectionEnabled() + return (not self._vm.isMigrating() and + self._vm.isDisksStatsCollectionEnabled()) @property def runnable(self): -- To view, visit https://gerrit.ovirt.org/40507 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I43ec1e64700020b7d143e22c4c3d8bdceb57fb18 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 21

Change in vdsm[master]: virt: avoid spam logs with periodic false negative
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: virt: avoid spam logs with periodic false negative ...................................................................... virt: avoid spam logs with periodic false negative Consolidate the periodic.Operation.required checks in a VM helper function, and make it a bit more paranoid, adding check to ensure that Vm._dom is not None. If we get that far and Vm._dom is None, proably something strange already happened (and no chance to recover), so adding this check here skips some annoying log spam. Change-Id: I3f0f5a46bcb8ed1197fe389a0b2904fa6939659d Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/periodic.py M vdsm/virt/vm.py 2 files changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/41659/1 diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py index f226424..efc1d4b 100644 --- a/vdsm/virt/periodic.py +++ b/vdsm/virt/periodic.py @@ -267,7 +267,7 @@ @property def required(self): # Avoid queries from storage during recovery process - return (not self._vm.isMigrating() and + return (self._vm.isDomainQueryable() and self._vm.isDisksStatsCollectionEnabled()) @property @@ -326,7 +326,7 @@ @property def required(self): # Avoid queries from storage during recovery process - return (not self._vm.isMigrating() and + return (self._vm.isDomainQueryable() and self._vm.isDisksStatsCollectionEnabled()) @property diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 0b6773b..894194c 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -2580,6 +2580,10 @@ return 'NOERR' + def isDomainQueryable(self): + # TODO: need better name + return (self._dom is not None and not self.isMigrating()) + def isDomainReadyForCommands(self): """ Returns True if the domain is reported to be in the safest condition -- To view, visit https://gerrit.ovirt.org/41659 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I3f0f5a46bcb8ed1197fe389a0b2904fa6939659d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 15

Change in vdsm[master]: sampling: stats: remove try/except for KeyError
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: sampling: stats: remove try/except for KeyError ...................................................................... sampling: stats: remove try/except for KeyError Catching general exceptions should be done sparingly. Let's remove a try/except block guarding for KeyError. As side effect, this streamlines the function. Change-Id: I40120e3f89a812126b9dd74495be6a1fad461b17 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 15 insertions(+), 14 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/44/42744/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index a434a8c..1978f2a 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -81,22 +81,23 @@ interval, vm.id) return - try: - stats['cpuUsage'] = str(last_sample['cpu.system'] + - last_sample['cpu.user']) + if any(k not in sample + for k in ('cpu.user', 'cpu.system', 'cpu.time') + for sample in (first_sample, last_sample)): + return - stats['cpuSys'] = _usage_percentage( - _diff(last_sample, first_sample, 'cpu.user') + - _diff(last_sample, first_sample, 'cpu.system'), - interval) - stats['cpuUser'] = _usage_percentage( - _diff(last_sample, first_sample, 'cpu.time') - - _diff(last_sample, first_sample, 'cpu.user') - - _diff(last_sample, first_sample, 'cpu.system'), - interval) + stats['cpuUsage'] = str(last_sample['cpu.system'] + + last_sample['cpu.user']) - except KeyError: - logging.warning("CPU stats not available for %s", vm.id) + stats['cpuSys'] = _usage_percentage( + _diff(last_sample, first_sample, 'cpu.user') + + _diff(last_sample, first_sample, 'cpu.system'), + interval) + stats['cpuUser'] = _usage_percentage( + _diff(last_sample, first_sample, 'cpu.time') + - _diff(last_sample, first_sample, 'cpu.user') + - _diff(last_sample, first_sample, 'cpu.system'), + interval) def balloon(vm, stats, sample): -- To view, visit https://gerrit.ovirt.org/42744 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I40120e3f89a812126b9dd74495be6a1fad461b17 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 9

Change in vdsm[master]: WIP: virt: better timeout for periodic operations
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: WIP: virt: better timeout for periodic operations ...................................................................... WIP: virt: better timeout for periodic operations Change-Id: I735d314532c6852889e3ac188651acff044888ed WIP: TODO: Documentation pending Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/periodicTests.py M vdsm/virt/periodic.py M vdsm/virt/utils.py M vdsm/virt/vm.py 4 files changed, 52 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/33/39333/1 diff --git a/tests/periodicTests.py b/tests/periodicTests.py index 80ef09e..d22f5ce 100644 --- a/tests/periodicTests.py +++ b/tests/periodicTests.py @@ -204,8 +204,9 @@ VMS = defaultdict(int) - def __init__(self, vm): + def __init__(self, vm, timeout): self._vm = vm + self._timeout = timeout # unused self.required = True self.runnable = True diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py index fe8bbc8..3159c04 100644 --- a/vdsm/virt/periodic.py +++ b/vdsm/virt/periodic.py @@ -33,6 +33,7 @@ from vdsm.utils import monotonic_time from . import sampling +from .errors import TimeoutError # just a made up number. Maybe should be equal to number of cores? @@ -229,7 +230,7 @@ skipped = [] for vm_id, vm_obj in vms.iteritems(): - op = self._create(vm_obj) + op = self._create(vm_obj, self._timeout) if not op.required: continue @@ -253,8 +254,9 @@ class UpdateVolumes(object): - def __init__(self, vm): + def __init__(self, vm, timeout): self._vm = vm + self._timeout = timeout @property def required(self): @@ -267,15 +269,16 @@ def __call__(self): for drive in self._vm.getDiskDevices(): - # TODO: If this block (it is actually possible?) - # we must make sure we don't overwrite good data - # with stale old data. - self._vm.updateDriveVolume(drive) + try: + self._vm.updateDriveVolume(drive, timeout=self._timeout) + except TimeoutError: + pass # TODO log it? class NumaInfoMonitor(object): - def __init__(self, vm): + def __init__(self, vm, timeout): self._vm = vm + self._timeout = timeout @property def required(self): @@ -286,12 +289,16 @@ return self._vm.isDomainReadyForCommands() def __call__(self): - self._vm.updateNumaInfo() + try: + self._vm.updateNumaInfo(timeout=self._timeout) + except TimeoutError: + pass # TODO log it? class BlockjobMonitor(object): - def __init__(self, vm): + def __init__(self, vm, timeout): self._vm = vm + self._timeout = timeout @property def required(self): @@ -307,7 +314,10 @@ return self._vm.isDomainReadyForCommands() def __call__(self): - self._vm.updateVmJobs() + try: + self._vm.updateVmJobs(timeout=self._timeout) + except TimeoutError: + pass # TODO log it? class DriveWatermarkMonitor(object): diff --git a/vdsm/virt/utils.py b/vdsm/virt/utils.py index 60becfa..385c893 100644 --- a/vdsm/virt/utils.py +++ b/vdsm/virt/utils.py @@ -25,6 +25,7 @@ import threading from vdsm.utils import monotonic_time +from .errors import TimeoutError def isVdsmImage(drive): @@ -110,3 +111,14 @@ raise ItemExpired return value + + +def discard_if_blocked(timeout, func, *args): + start = monotonic_time() + res = func(*args) + elapsed = monotonic_time() - start + + if timeout is None or elapsed <= timeout: + return res + + raise TimeoutError diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 460d45a..eb56ed6 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -72,7 +72,7 @@ from . import vmxml from .errors import TimeoutError -from .utils import isVdsmImage +from .utils import isVdsmImage, discard_if_blocked from vmpowerdown import VmShutdown, VmReboot _VMCHANNEL_DEVICE_NAME = 'com.redhat.rhevm.vdsm' @@ -2652,13 +2652,16 @@ raise LookupError("No such drive: '%s'" % drive) - def updateDriveVolume(self, vmDrive): + # CAN_BLOCK + # PERIODIC_OPERATION + def updateDriveVolume(self, vmDrive, timeout=None): if not vmDrive.device == 'disk' or not isVdsmImage(vmDrive): return - volSize = self.cif.irs.getVolumeSize( - vmDrive.domainID, vmDrive.poolID, vmDrive.imageID, - vmDrive.volumeID) + volSize = discard_if_blocked( + timeout, self.cif.irs.getVolumeSize, + vmDrive.domainID, vmDrive.poolID, + vmDrive.imageID, vmDrive.volumeID) if volSize['status']['code'] != 0: self.log.error( @@ -4248,8 +4251,11 @@ with self._jobsLock: return bool(self.conf['_blockJobs']) - def updateVmJobs(self): - self._vmJobs = self.queryBlockJobs() + # CAN_BLOCK + # PERIODIC_OPERATION + def updateVmJobs(self, timeout=None): + self._vmJobs = discard_if_blocked( + timeout, self.queryBlockJobs) def queryBlockJobs(self): def startCleanup(job, drive, needPivot): @@ -4576,8 +4582,11 @@ statsAge) stats['monitorResponse'] = '-1' - def updateNumaInfo(self): - self._numaInfo = numaUtils.getVmNumaNodeRuntimeInfo(self) + # CAN_BLOCK + # PERIODIC_OPERATION + def updateNumaInfo(self, timeout=None): + self._numaInfo = discard_if_blocked( + timeout, numaUtils.getVmNumaNodeRuntimeInfo, self) @property def hasGuestNumaNode(self): -- To view, visit https://gerrit.ovirt.org/39333 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I735d314532c6852889e3ac188651acff044888ed Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 273

Change in vdsm[master]: tests: Isolate tcTests test properly
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 5: Verified+1 -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[ovirt-3.5]: fix pep8
by ibarkan＠redhat.com 30 Jul '15

30 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: fix pep8 ...................................................................... fix pep8 Change-Id: I4e0d9c4fc179825e9d14b4787e65b77dcfec59d7 Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/26/44026/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index c5da1f2..e01a612 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1505,7 +1505,7 @@ self.assertEquals(status, SUCCESS, msg) self.vdsm_net.save_config() - + @cleanupNet @RequireDummyMod @ValidateRunningAsRoot @@ -1526,7 +1526,7 @@ status, msg = self.setupNetworks( {NETWORK_NAME: {'remove': True}}, {}, NOCHK) self.assertEquals(status, SUCCESS, msg) - + @cleanupNet @RequireDummyMod @ValidateRunningAsRoot @@ -1689,7 +1689,7 @@ def _verify_running_config_intact(): self.assertEqual(set([NET_MGMT, NET_CHANGED, NET_UNCHANGED, - NET_ADDITIONAL]), + NET_ADDITIONAL]), set(self.vdsm_net.config.networks.keys())) self.assertEqual(set([BOND_CHANGED, BOND_UNCHANGED]), set(self.vdsm_net.config.bonds.keys())) @@ -1721,7 +1721,7 @@ _assert_all_nets_exist() _verify_running_config_intact() self.assertEqual(set(['ifcfg-%s' % NET_ADDITIONAL, - 'ifcfg-%s' % nic_d]), + 'ifcfg-%s' % nic_d]), set(os.listdir(NET_CONF_BACK_DIR))) # another 'boot' should restore nothing -- To view, visit https://gerrit.ovirt.org/44026 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4e0d9c4fc179825e9d14b4787e65b77dcfec59d7 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 7

Change in vdsm[master]: net: ignore pass through ifcfg parameters for selective rest...
by ibarkan＠redhat.com 30 Jul '15

30 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: net: ignore pass through ifcfg parameters for selective restoration ...................................................................... net: ignore pass through ifcfg parameters for selective restoration Ignore keys in persisted networks that might originate from vdsm-reg. these might be a result of calling setupNetworks with ifcfg values that come from the original interface that is serving the management network. for 3.5, VDSM still supports passing arbitrary values directly to the ifcfg files, e.g. 'IPV6_AUTOCONF=no'. we filter them out here since kernelConfig will never report them. Change-Id: Ib1525f7b0238aaa5eb82eedb9ad6f41154c2c29f Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M lib/vdsm/netconfpersistence.py M tests/functional/networkTests.py 2 files changed, 41 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/52/44152/1 diff --git a/lib/vdsm/netconfpersistence.py b/lib/vdsm/netconfpersistence.py index c3fbda8..b5c3f14 100644 --- a/lib/vdsm/netconfpersistence.py +++ b/lib/vdsm/netconfpersistence.py @@ -25,6 +25,7 @@ import logging import netaddr import os +import string from .config import config from .tool.restore_nets import restore @@ -319,6 +320,7 @@ self._normalize_bonding_opts(config_copy) self._normalize_bonding_nics(config_copy) self._normalize_address(config_copy) + self._normalize_ifcfg_keys(config_copy) return config_copy @@ -391,6 +393,23 @@ if 'defaultRoute' not in net_attr: net_attr['defaultRoute'] = False + def _normalize_ifcfg_keys(self, config_copy): + # ignore keys in persisted networks that might originate from vdsm-reg. + # these might be a result of calling setupNetworks with ifcfg values + # that come from the original interface that is serving the management + # network. for 3.5, VDSM still supports passing arbitrary values + # directly to the ifcfg files, e.g. 'IPV6_AUTOCONF=no'. we filter them + # out here since kernelConfig will never report them. + # TODO: remove when 3.5 is unsupported. + def unsupported(key): + return set(key) <= set( + string.ascii_uppercase + string.digits + '_') + + for net_attr in config_copy.networks.itervalues(): + for k in net_attr.keys(): + if unsupported(k): + net_attr.pop(k) + def _parse_bond_options(self, opts): if not opts: return {} diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index ce532bb..bec3372 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1744,6 +1744,28 @@ self.assertBondDoesntExist(BOND_UNCHANGED, [nic_a]) @cleanupNet + def testSelectiveRestoreIgnoresVdsmRegParams(self): + if vdsm.config.config.get('vars', 'net_persistence') == 'ifcfg': + raise SkipTest( + "with ifcfg persistence, vdsm-restore-net-config selective" + "restoration is not supported") + + with dummyIf(1) as nics: + nic, = nics + # let _assert_kernel_config_matches_running_config do the job + status, msg = self.setupNetworks( + {NETWORK_NAME: {'nic': nic, 'IPV6_AUTOCONF': 'no', + 'PEERNTP': 'yes', 'IPV6INIT': 'no'}}, + {}, NOCHK) + self.assertEquals(status, SUCCESS, msg) + self.assertNetworkExists(NETWORK_NAME) + status, msg = self.setupNetworks( + {NETWORK_NAME: {'remove': True}}, {}, NOCHK) + self.assertEquals(status, SUCCESS, msg) + self.assertNetworkDoesntExist(NETWORK_NAME) + + + @cleanupNet @permutations([[True], [False]]) def testVolatileConfig(self, bridged): """ -- To view, visit https://gerrit.ovirt.org/44152 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib1525f7b0238aaa5eb82eedb9ad6f41154c2c29f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 7

Change in vdsm[master]: storageServer: Add tests for equality and hash
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: storageServer: Add tests for equality and hash ...................................................................... storageServer: Add tests for equality and hash Current __eq__ and __hash__ are wrong and have no tets. Add tests showing what works and what not. To make the tests results more clear, implement __str__. This can also be useful when logging the objects. Change-Id: Ie1dc9510a9f0f6c44534fd568cbcda161f36497c Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/storageServerTests.py M vdsm/storage/storageServer.py 2 files changed, 69 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/88/43988/1 diff --git a/tests/storageServerTests.py b/tests/storageServerTests.py index 9b2939f..f835419 100644 --- a/tests/storageServerTests.py +++ b/tests/storageServerTests.py @@ -21,6 +21,7 @@ from monkeypatch import MonkeyPatch from testlib import permutations, expandPermutations from testlib import VdsmTestCase +from testValidation import brokentest from storage.storageServer import GlusterFSConnection from storage.storageServer import IscsiConnection from storage.storageServer import MountConnection @@ -66,6 +67,67 @@ @expandPermutations +class TestMountConnectionEquality(VdsmTestCase): + + def test_eq(self): + c1 = MountConnection("spec", "vfstype", "options") + c2 = MountConnection("spec", "vfstype", "options") + self.assertTrue(c1 == c2, "%s should equal %s" % (c1, c2)) + + @brokentest("__eq__ check isinstance instead fo class") + def test_eq_subclass(self): + class Subclass(MountConnection): + pass + c1 = MountConnection("spec", "vfstype", "options") + c2 = Subclass("spec", "vfstype", "options") + self.assertFalse(c1 == c2, "%s should not equal %s" % (c1, c2)) + + @brokentest("__eq__ not using all object state") + @permutations([ + ("s1", "s2", "t", "t", "o", "o"), + ("s", "s", "t1", "t2", "o", "o"), + ("s", "s", "t", "t", "o1", "o2"), + ]) + def test_eq_different(self, s1, s2, t1, t2, o1, o2): + c1 = MountConnection(s1, t1, o1) + c2 = MountConnection(s2, t2, o2) + self.assertFalse(c1 == c2, "%s should not equal %s" % (c1, c2)) + + @brokentest("__ne__ not implemented") + def test_ne(self): + c1 = MountConnection("spec", "vfstype", "options") + c2 = MountConnection("spec", "vfstype", "options") + self.assertFalse(c1 != c2, "%s should equal %s" % (c1, c2)) + + +@expandPermutations +class TestMountConnectionHash(VdsmTestCase): + + def test_eq(self): + c1 = MountConnection("spec", "vfstype", "options") + c2 = MountConnection("spec", "vfstype", "options") + self.assertEqual(hash(c1), hash(c2)) + + def test_eq_subclass(self): + class Subclass(MountConnection): + pass + c1 = MountConnection("spec", "vfstype", "options") + c2 = Subclass("spec", "vfstype", "options") + self.assertNotEqual(hash(c1), hash(c2)) + + @brokentest("__hash__ not using all object state") + @permutations([ + ("s1", "s2", "t", "t", "o", "o"), + ("s", "s", "t1", "t2", "o", "o"), + ("s", "s", "t", "t", "o1", "o2"), + ]) + def test_ne(self, s1, s2, t1, t2, o1, o2): + c1 = MountConnection(s1, t1, o1) + c2 = MountConnection(s2, t2, o2) + self.assertNotEqual(hash(c1), hash(c2)) + + +@expandPermutations class GlusterFSConnectionTests(VdsmTestCase): def test_mountpoint(self): diff --git a/vdsm/storage/storageServer.py b/vdsm/storage/storageServer.py index e71b88c..95e1099 100644 --- a/vdsm/storage/storageServer.py +++ b/vdsm/storage/storageServer.py @@ -264,6 +264,13 @@ def __hash__(self): return hash(type(self)) ^ hash(self._mount) + def __str__(self): + return "<{0} spec={1!r} vfstype={2!r} options={3!r}>".format( + self.__class__.__name__, + self._remotePath, + self._vfsType, + self._options) + class GlusterFSConnection(MountConnection): -- To view, visit https://gerrit.ovirt.org/43988 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie1dc9510a9f0f6c44534fd568cbcda161f36497c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 13

Change in vdsm[master]: vmstats: avoid explicit string conversion
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: avoid explicit string conversion ...................................................................... vmstats: avoid explicit string conversion Now that we streamlined the reporting of disk stats, we can get rid of explicit conversion loop and simplify the code. Change-Id: I99c0353828f3690411faf25edda40e6ddac8641e Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 0 insertions(+), 10 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/18/43718/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index 891c137..17f44fb 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -65,16 +65,6 @@ value = vm_stats[var] if value: stats[var] = value - else: - if 'disks' not in stats: - stats['disks'] = {} - try: - stats['disks'][var] = {} - for value in vm_stats[var]: - stats['disks'][var][value] = \ - convertToStr(vm_stats[var][value]) - except Exception: - logging.exception("Error setting vm disk stats") return stats -- To view, visit https://gerrit.ovirt.org/43718 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I99c0353828f3690411faf25edda40e6ddac8641e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 15

Change in vdsm[master]: vmstats: report disks stats only if present
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: report disks stats only if present ...................................................................... vmstats: report disks stats only if present avoid to report empty values for disks stats, as this may confuse Engine and let it to believe there is no disks at all. Change-Id: I7f1a2553114e3d78f98f978f599497d2ef5750f0 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/17/43717/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index caa2fc2..891c137 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -61,7 +61,7 @@ stats[var] = vm_stats[var] elif type(vm_stats[var]) is not dict: stats[var] = convertToStr(vm_stats[var]) - elif var in ('network', 'balloonInfo'): + elif var in ('disks', 'network', 'balloonInfo'): value = vm_stats[var] if value: stats[var] = value @@ -230,8 +230,11 @@ def disks(vm, stats, first_sample, last_sample, interval): + if first_sample is None or last_sample is None: return + + disk_stats = {} for vm_drive in vm.getDiskDevices(): drive_stats = {} @@ -272,7 +275,10 @@ logging.exception("Disk %s stats not available", vm_drive.name) - stats[vm_drive.name] = drive_stats + disk_stats[vm_drive.name] = drive_stats + + if disk_stats: + stats['disks'] = disk_stats def _disk_rate(first_sample, last_sample, interval): -- To view, visit https://gerrit.ovirt.org/43717 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7f1a2553114e3d78f98f978f599497d2ef5750f0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 14

Change in vdsm[master]: vmstats: do not report empty ioTune
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: do not report empty ioTune ...................................................................... vmstats: do not report empty ioTune Do not report empty value for 'ioTune' stats, much like we do for balloonInfo, disks and networks In general, we should avoid to report empty values if the emptiness carries no meaning (noteworthy counterexample: vmJobs for live merge). Change-Id: I1c34e2ffb1ebeb500be22d6db5f5a8ea2c1bb19f Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 9 insertions(+), 6 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/96/43596/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index c7c6bb4..f6636cf 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -50,12 +50,15 @@ for var in vm_stats: if var == "ioTune": - # Convert ioTune numbers to strings to avoid xml-rpc issue - # with numbers bigger than int32_t - for ioTune in vm_stats["ioTune"]: - ioTune["ioTune"] = dict((k, convertToStr(v)) for k, v - in ioTune["ioTune"].iteritems()) - stats[var] = vm_stats[var] + value = vm_stats[var] + if value: + # Convert ioTune numbers to strings to avoid xml-rpc issue + # with numbers bigger than int32_t + for ioTune in value: + ioTune["ioTune"] = dict( + (k, convertToStr(v)) for k, v + in ioTune["ioTune"].iteritems()) + stats[var] = vm_stats[var] elif type(vm_stats[var]) is not dict: stats[var] = convertToStr(vm_stats[var]) elif var in ('network', 'balloonInfo'): -- To view, visit https://gerrit.ovirt.org/43596 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1c34e2ffb1ebeb500be22d6db5f5a8ea2c1bb19f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 15

Change in vdsm[master]: iscsi: Add ChapCredentials equality and hash tests
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: iscsi: Add ChapCredentials equality and hash tests ...................................................................... iscsi: Add ChapCredentials equality and hash tests Add tests before cleaning up the implementation. Mark __ne__ test as @brokentest for now, as it is not implemented. Change-Id: I7f3fde727c179936ba480f5b4cf4c24cacb195b8 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/iscsiTests.py 1 file changed, 59 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/89/43989/1 diff --git a/tests/iscsiTests.py b/tests/iscsiTests.py index 95e3af4..30df6cd 100644 --- a/tests/iscsiTests.py +++ b/tests/iscsiTests.py @@ -4,8 +4,11 @@ from monkeypatch import MonkeyPatch from testlib import VdsmTestCase as TestCaseBase from testlib import make_config +from testlib import expandPermutations, permutations +from testValidation import brokentest from vdsm import utils +from vdsm.password import ProtectedPassword from storage import iscsi from storage import iscsiadm @@ -59,3 +62,59 @@ Iface('eth2', 'tcp', None, None, 'eth2', None)) self.assertEqual(tuple(iscsi.iscsiadm.iface_list(out=out)), res) + + +@expandPermutations +class TestChapCredentialsEquality(TestCaseBase): + + def test_eq_equal(self): + c1 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + c2 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + self.assertTrue(c1 == c2, "%s should equal %s" % (c1, c2)) + + def test_eq_subclass(self): + class Subclass(iscsi.ChapCredentials): + pass + c1 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + c2 = Subclass("username", ProtectedPassword("password")) + self.assertFalse(c1 == c2, "%s should not equal %s" % (c1, c2)) + + @permutations([ + ("a", "a", "a", "b"), + ("a", "b", "a", "a"), + ]) + def test_eq_different(self, user1, user2, pass1, pass2): + c1 = iscsi.ChapCredentials(user1, ProtectedPassword(pass1)) + c2 = iscsi.ChapCredentials(user2, ProtectedPassword(pass2)) + self.assertFalse(c1 == c2, "%s should not equal %s" % (c1, c2)) + + @brokentest("__ne__ not implemented") + def test_ne_equal(self): + c1 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + c2 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + self.assertFalse(c1 != c2, "%s should equal %s" % (c1, c2)) + + +@expandPermutations +class TestChapCredentialsHash(TestCaseBase): + + def test_equal_same_hash(self): + c1 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + c2 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + self.assertEqual(hash(c1), hash(c2)) + + def test_subclass_diffrent_hash(self): + class Subclass(iscsi.ChapCredentials): + pass + c1 = iscsi.ChapCredentials("username", ProtectedPassword("password")) + c2 = Subclass("username", ProtectedPassword("password")) + self.assertNotEqual(hash(c1), hash(c2)) + + @permutations([ + ("a", "a", "a", "b"), + ("a", "b", "a", "a"), + ]) + def test_not_equal_different_hash(self, user1, user2, pass1, pass2): + c1 = iscsi.ChapCredentials(user1, ProtectedPassword(pass1)) + c2 = iscsi.ChapCredentials(user2, ProtectedPassword(pass2)) + self.assertNotEqual(hash(c1), hash(c2)) -- To view, visit https://gerrit.ovirt.org/43989 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7f3fde727c179936ba480f5b4cf4c24cacb195b8 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 11

Change in vdsm[master]: vmstats: do not report empty balloonInfo
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: do not report empty balloonInfo ...................................................................... vmstats: do not report empty balloonInfo Do not report empty value for 'balloonInfo' stats, much like we do for disks and networks In general, we should avoid to report empty values if the emptiness carries no meaning (noteworthy counterexample: vmJobs for live merge). Change-Id: I146e1f476434f023281f475ffaaa808f4cdf862b Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/39/43539/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index eed10e8..0b02a1e 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -59,7 +59,9 @@ elif type(vm_stats[var]) is not dict: stats[var] = convertToStr(vm_stats[var]) elif var in ('network', 'balloonInfo'): - stats[var] = vm_stats[var] + value = vm_stats[var] + if value: + stats[var] = value else: try: stats['disks'][var] = {} -- To view, visit https://gerrit.ovirt.org/43539 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I146e1f476434f023281f475ffaaa808f4cdf862b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 17

Change in vdsm[master]: vm: event: send status event on pause and resume
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vm: event: send status event on pause and resume ...................................................................... vm: event: send status event on pause and resume Make sure to send to Engine the events when VM is paused and resumed. Change-Id: I8da5e2c8e183e05a7a795ee129984b214fff60ac Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/76/43376/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 7ab953f..0e8a64d 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -1182,6 +1182,7 @@ del self.conf['pauseCode'] except KeyError: pass + self.send_status_event() return {'status': doneCode, 'output': ['']} finally: if not guestCpuLocked: @@ -1199,6 +1200,7 @@ guestCpuLocked=True) self._logGuestCpuStatus('pause') self._lastStatus = afterState + self.send_status_event() return {'status': doneCode, 'output': ['']} finally: if not guestCpuLocked: -- To view, visit https://gerrit.ovirt.org/43376 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8da5e2c8e183e05a7a795ee129984b214fff60ac Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 25

Change in vdsm[master]: vm: event: send status event on IO Error
by automation＠ovirt.org 30 Jul '15

30 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 23: * Update tracker::IGNORE, no Bug-Url found * Set MODIFIED::IGNORE, no Bug-Url found. -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 23 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by Dan Kenigsberg 30 Jul '15

30 Jul '15

Dan Kenigsberg has submitted this change and it was merged. Change subject: vm: event: send status event on IO Error ...................................................................... vm: event: send status event on IO Error Explicitely send back to Engine a status Event after an IO Error reported by libvirt. Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Signed-off-by: Francesco Romani <fromani(a)redhat.com> Reviewed-on: https://gerrit.ovirt.org/43312 Continuous-Integration: Jenkins CI Reviewed-by: Dan Kenigsberg <danken(a)redhat.com> --- M tests/vmTests.py M vdsm/virt/vm.py 2 files changed, 18 insertions(+), 3 deletions(-) Approvals: Jenkins CI: Passed CI tests Dan Kenigsberg: Looks good to me, approved Francesco Romani: Verified -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 23 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by Dan Kenigsberg 30 Jul '15

30 Jul '15

Dan Kenigsberg has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 22: Code-Review+2 -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 22 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: make up pauseCode if libvirt is omitting it
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vm: make up pauseCode if libvirt is omitting it ...................................................................... vm: make up pauseCode if libvirt is omitting it When VDSM receives one IOError notification from libvirt, it can override the VM pauseCode. The same pauseCode will be reported to Engine through Events or through the VM.getStats() API. For historical reasons, libvirt reliably reports only the 'enospc' reason. In the other cases, the reason is an empty string, and this resets the pauseCode to empty string. Engine is not (anymore) expecting that. To make Engine happy(er), and to improve the quality of data VDSM reports in general, it is safe to report EOTHER if the pauseCode is not given onIOError. This patch does that. Change-Id: I0bde0561cae20971effcf01c8a57159edcc4c42e Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 8 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/15/44115/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 71a74b3..83d8ad8 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -3591,22 +3591,26 @@ :param err: one of "eperm", "eio", "enospc" or "eother" Note the different API from that of Vm._onAbnormalStop """ + # libvirt <= 1.2.18 sends either reason="enospc" or reason="". + # Engine doesn't like disappearing pauseCode, so we fill the gap + reason = err.upper() if err else "EOTHER" + if action == libvirt.VIR_DOMAIN_EVENT_IO_ERROR_PAUSE: self.log.info('abnormal vm stop device %s error %s', blockDevAlias, err) - self.conf['pauseCode'] = err.upper() + self.conf['pauseCode'] = reason self._setGuestCpuRunning(False) self._logGuestCpuStatus('onIOError') - if err.upper() == 'ENOSPC': + if reason == 'ENOSPC': if not self.extendDrivesIfNeeded(): self.log.info("No VM drives were extended") elif action == libvirt.VIR_DOMAIN_EVENT_IO_ERROR_REPORT: self.log.info('I/O error %s device %s reported to guest OS', - err, blockDevAlias) + reason, blockDevAlias) else: # we do not support and do not expect other values self.log.warning('unexpected action %i on device %s error %s', - action, blockDevAlias, err) + action, blockDevAlias, reason) @property def hasSpice(self): -- To view, visit https://gerrit.ovirt.org/44115 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0bde0561cae20971effcf01c8a57159edcc4c42e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 12

Change in vdsm[master]: vm: event: send status event on IO Error
by fromani＠redhat.com 30 Jul '15

30 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 22: Verified+1 verified with 44115. - booted VM - started write on disk using dd with oflag=direct - after a few tenth of secs, blocked connection to storage wait for VM go paused and IOError reported. HOji> 10:45:31 root [~]$ vdsClient -s 0 getAllVmStats b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb Status = Paused displayInfo = [{'tlsPort': '5901', 'ipAddress': '0', 'port': '5900', 'type': 'spice'}] memUsage = 0 acpiEnable = true guestFQDN = vNodeRuntimeInfo = {'0': [0]} session = Unknown displaySecurePort = 5901 timeOffset = 0 pauseCode = EOTHER network = {'vnet0': {'macAddr': '00:1a:4a:16:01:52', 'rxDropped': '0', 'tx': '13471', 'txDropped': '0', 'txRate': '0.0', 'rxErrors': '0', 'rx': '27752', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'sampleTime': 4297886.89, 'speed': '1000', 'name': 'vnet0'}} vmJobs = {} cpuUser = 0.25 elapsedTime = 564 displayType = qxl cpuSys = 0.00 appsList = [] vmName = aa_def_desktop_1 vcpuCount = 2 clientIp = 192.168.1.177 hash = -626332171739236010 vmType = kvm pid = 5023 displayIp = 0 cpuUsage = 13050000000 vcpuPeriod = 100000 displayPort = 5900 vcpuQuota = -1 kvmEnable = true disks = {'vda': {'readLatency': '0', 'writtenBytes': '58103808', 'truesize': '2147483648', 'apparentsize': '2147483648', 'readOps': '346', 'writeLatency': '0', 'imageID': 'bb026109-3886-438a-be55-37797c2bbfb0', 'readBytes': '1413632', 'flushLatency': '0', 'readRate': '0.0', 'writeOps': '56742', 'writeRate': '0.0'}, 'hdc': {'readLatency': '0', 'writtenBytes': '0', 'truesize': '0', 'apparentsize': '0', 'readOps': '29099', 'writeLatency': '0', 'readBytes': '394772782', 'flushLatency': '0', 'readRate': '0.0', 'writeOps': '0', 'writeRate': '0.0'}} monitorResponse = 0 username = Unknown guestCPUCount = -1 guestIPs = statusTime = 4297886890 HOji> 10:51:22 root [~]$ grep Notif /var/log/vdsm/vdsm.log Thread-268::DEBUG::2015-07-30 10:41:58,599::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4297322880, "b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb": {"status": "Powering up", "displayInfo": [{"tlsPort": "5901", "ipAddress": "0", "type": "spice", "port": "5900"}], "hash": "-925049597309698524", "displayIp": "0", "displayPort": "5900", "displaySecurePort": "5901", "timeOffset": "0", "pauseCode": "NOERR", "vcpuQuota": "-1", "cpuUser": "0.00", "monitorResponse": "0", "elapsedTime": "0", "displayType": "qxl", "cpuSys": "0.00", "clientIp": "", "vcpuPeriod": 100000}}, "jsonrpc": "2.0", "method": "|virt|VM_status|b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb"} libvirtEventLoop::DEBUG::2015-07-30 10:47:32,559::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4297656840, "b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb": {"status": "Paused", "ioerror": {"alias": "virtio-disk0", "name": "vda", "path": "/rhev/data-center/00000001-0001-0001-0001-0000000001e8/bbed5784-b0ee-4a0a-aff2-801da0bcf39e/images/bb026109-3886-438a-be55-37797c2bbfb0/667fe374-53ed-4498-a388-c90acf3d4659"}, "pauseCode": "EOTHER"}}, "jsonrpc": "2.0", "method": "|virt|VM_status|b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb"} -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 22 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: periodic: Use thread names that can be used as system thread...
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: periodic: Use thread names that can be used as system thread names ...................................................................... periodic: Use thread names that can be used as system thread names Using new executor naming scheme, creating executor named "periodic" will create workers named "periodic/0", "periodic/1", ... Change also periodic scheduler thread to match this style. Change-Id: I50d76e0b8c08070c2f34fc4eba439b9decd4990d Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/virt/periodic.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/43819/1 diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py index b023976..632674f 100644 --- a/vdsm/virt/periodic.py +++ b/vdsm/virt/periodic.py @@ -41,10 +41,10 @@ _TASKS = _WORKERS * _TASK_PER_WORKER -_scheduler = schedule.Scheduler(name="periodic.Scheduler", +_scheduler = schedule.Scheduler(name="periodic-sched", clock=monotonic_time) -_executor = executor.Executor(name="periodic.Executor", +_executor = executor.Executor(name="periodic", workers_count=_WORKERS, max_tasks=_TASKS, scheduler=_scheduler) -- To view, visit https://gerrit.ovirt.org/43819 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I50d76e0b8c08070c2f34fc4eba439b9decd4990d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 12

Change in vdsm[master]: executor: Set worker system name
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: executor: Set worker system name ...................................................................... executor: Set worker system name Previously we used to rename the workers using long name such as "periodic-executor-worker-3". These names are too long for setting thread system name (using pthread.setname()), and little too long in general. Since we managed worker ids in the Worker class, when creating multiple executors, workers ids did not much the number of workers in the executor. This patch changes the naming scheme to "executor-name/<worker-id>", similar to the way kernel threads are named. For example, when creating executor named "periodic" with 4 workers, the workers will use these names: periodic/0 periodic/1 periodic/2 periodic/3 If we create another executor such as "jsonrpc", it will create these worker names: jsonrpc/0 jsonrpc/1 jsonrpc/2 jsonrpc/3 jsonrpc/4 jsonrpc/5 jsonrpc/6 jsonrpc/7 If the executor name is too long, the system thread name will be truncated, as system thread names are limited to 15 characters. Change-Id: Ia6a6b289daff2456213615355d6d6b51fd04b500 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/executor.py M tests/executorTests.py 2 files changed, 50 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/88/43788/1 diff --git a/lib/vdsm/executor.py b/lib/vdsm/executor.py index 2639b26..4447115 100644 --- a/lib/vdsm/executor.py +++ b/lib/vdsm/executor.py @@ -27,6 +27,7 @@ import logging import threading +from . import pthread from . import utils @@ -53,6 +54,7 @@ def __init__(self, name, workers_count, max_tasks, scheduler): self._name = name self._workers_count = workers_count + self._worker_id = 0 self._tasks = TaskQueue(max_tasks) self._scheduler = scheduler self._workers = set() @@ -129,7 +131,9 @@ # Private def _add_worker(self): - worker = _Worker(self, self._scheduler) + name = "%s/%d" % (self.name, self._worker_id) + self._worker_id += 1 + worker = _Worker(self, self._scheduler, name) self._workers.add(worker) @@ -143,14 +147,11 @@ class _Worker(object): _log = logging.getLogger('Executor') - _id = 0 - def __init__(self, executor, scheduler): + def __init__(self, executor, scheduler, name): self._executor = executor self._scheduler = scheduler self._discarded = False - _Worker._id += 1 - name = "%s-worker-%d" % (self._executor.name, _Worker._id) self._thread = threading.Thread(target=self._run, name=name) self._thread.daemon = True self._log.debug('Starting worker %s' % name) @@ -166,6 +167,7 @@ @utils.traceback(on=_log.name) def _run(self): + pthread.setname(self.name[:15]) self._log.debug('Worker started') try: while True: diff --git a/tests/executorTests.py b/tests/executorTests.py index 1f58d4d..e2d093e 100644 --- a/tests/executorTests.py +++ b/tests/executorTests.py @@ -21,8 +21,11 @@ import threading import time +from vdsm import concurrent from vdsm import executor +from vdsm import pthread from vdsm import schedule +from vdsm import utils from testValidation import slowtest from testlib import VdsmTestCase as TestCaseBase @@ -130,6 +133,46 @@ self.assertTrue(task.executed.is_set()) +class TestWorkerSystemNames(TestCaseBase): + + def test_worker_thread_system_name(self): + names = [] + workers = 2 + done = concurrent.Barrier(workers + 1) + + def get_worker_name(): + names.append(pthread.getname()) + done.wait() + + foo = executor.Executor('foo', workers, workers, None) + with utils.running(foo): + for i in range(workers): + foo.dispatch(get_worker_name) + done.wait() + + self.assertEqual(sorted(names), ["foo/0", "foo/1"]) + + def test_multiple_executors(self): + names = [] + workers = 2 + done = concurrent.Barrier(2 * workers + 1) + + def get_worker_name(): + names.append(pthread.getname()) + done.wait() + + foo = executor.Executor('foo', workers, workers, None) + bar = executor.Executor('bar', workers, workers, None) + with utils.running(foo), utils.running(bar): + for i in range(workers): + foo.dispatch(get_worker_name) + bar.dispatch(get_worker_name) + done.wait() + + self.assertEqual(sorted(names), + ["bar/0", "bar/1", "foo/0", "foo/1"]) + + class Task(object): def __init__(self, wait=None, error=None): -- To view, visit https://gerrit.ovirt.org/43788 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6a6b289daff2456213615355d6d6b51fd04b500 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 9

Change in vdsm[master]: concurrent: Add CountedEvent class
by Nir Soffer 30 Jul '15

30 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: concurrent: Add CountedEvent class ...................................................................... concurrent: Add CountedEvent class CountedEvent is like a threading.Event, but it notifies waiters only after it was set multiple times. CountedEvent is useful for waiting until multiple threads are ready. Currently we are synchronizing multiple threads in the tests using timeouts, which make the tests slower and break randomly on overloaded CI slaves. Change-Id: I68525e18f6b3774d7e10af1226a7bc3404c68ae9 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M lib/vdsm/concurrent.py M tests/concurrentTests.py 2 files changed, 79 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/27/42927/1 diff --git a/lib/vdsm/concurrent.py b/lib/vdsm/concurrent.py index 3865873..20d0e09 100644 --- a/lib/vdsm/concurrent.py +++ b/lib/vdsm/concurrent.py @@ -21,6 +21,51 @@ from __future__ import absolute_import import threading from collections import namedtuple +from . import utils + + +class Timeout(Exception): + """ Raised when operation timedout """ + + +class CountedEvent(object): + """ + An Event like object that notify waiters only after it was set multiple + times. + + Typical usage: + + ready = CountedEvent(5) + + def thread(): + ready.set() + + for i in range(5): + threading.Thread(target=thread).start() + + # Will block until all 5 threads invoke ready.set() + ready.wait() + + """ + + def __init__(self, count): + self._count = count + self._cond = threading.Condition(threading.Lock()) + + def set(self): + with self._cond: + self._count -= 1 + if self._count <= 0: + self._cond.notify_all() + + def wait(self, timeout): + deadline = utils.monotonic_time() + timeout + with self._cond: + while self._count > 0: + now = utils.monotonic_time() + if now >= deadline: + raise Timeout("Timeout waiting for counted event") + self._cond.wait(deadline - now) Result = namedtuple("Result", ["succeeded", "value"]) diff --git a/tests/concurrentTests.py b/tests/concurrentTests.py index 307e397..6bd6496 100644 --- a/tests/concurrentTests.py +++ b/tests/concurrentTests.py @@ -20,12 +20,46 @@ import time import random +import threading from testlib import VdsmTestCase +from testValidation import slowtest from vdsm import concurrent +class CountedEventTests(VdsmTestCase): + + def test_wait_until_set_multiple_times(self): + event = concurrent.CountedEvent(2) + self.assertRaises(concurrent.Timeout, event.wait, 0.0) + event.set() + self.assertRaises(concurrent.Timeout, event.wait, 0.0) + event.set() + event.wait(0) + + @slowtest + def test_timeout(self): + event = concurrent.CountedEvent(1) + with self.assertElapsed(1.0, tolerance=0.5): + self.assertRaises(concurrent.Timeout, event.wait, 1.0) + + def test_wait_for_multiple_threads(self): + count = 5 + event = concurrent.CountedEvent(count) + threads = [] + try: + for i in range(count): + t = threading.Thread(target=event.set) + t.daemon = True + t.start() + threads.append(t) + event.wait(2) + finally: + for t in threads: + t.join() + + class TMapTests(VdsmTestCase): def test_results(self): -- To view, visit https://gerrit.ovirt.org/42927 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I68525e18f6b3774d7e10af1226a7bc3404c68ae9 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

8 42

Change in vdsm[master]: tests: Isolate tcTests test properly
by ibarkan＠redhat.com 30 Jul '15

30 Jul '15

Ido Barkan has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 5: Code-Review+1 -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tests: Isolate tcTests test properly
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 5: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 5 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tests: Isolate tcTests test properly
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 4: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 4 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: virt: Add _getVolumeInfo helper
by alitke＠redhat.com 29 Jul '15

29 Jul '15

Adam Litke has uploaded a new change for review. Change subject: virt: Add _getVolumeInfo helper ...................................................................... virt: Add _getVolumeInfo helper The next patch needs to call into to storage to get detailed volume size information. Add a helper to encapsulate this operation which makes error checking and exception raising consistent for all call sites. Change-Id: I11eefb292e5d08458cf3a16ef9c444fb9c08702b Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M vdsm/virt/vm.py 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/60/43560/1 diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index c3d5da2..6989603 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -4896,6 +4896,14 @@ (domainID, volumeID)) return VolumeSize(int(res['apparentsize']), int(res['truesize'])) + def _getVolumeInfo(self, domainID, poolID, imageID, volumeID): + res = self.cif.irs.getVolumeInfo(domainID, poolID, imageID, volumeID) + if res['status']['code'] != 0: + raise StorageUnavailableError( + "Unable to get volume info for domain %s volume %s" % + (domainID, volumeID)) + return res['info'] + def _setVolumeSize(self, domainID, poolID, imageID, volumeID, size): res = self.cif.irs.setVolumeSize(domainID, poolID, imageID, volumeID, size) -- To view, visit https://gerrit.ovirt.org/43560 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I11eefb292e5d08458cf3a16ef9c444fb9c08702b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

5 7

Change in vdsm[master]: tests: Fix expandPermutations use with Mixins
by alitke＠redhat.com 29 Jul '15

29 Jul '15

Adam Litke has uploaded a new change for review. Change subject: tests: Fix expandPermutations use with Mixins ...................................................................... tests: Fix expandPermutations use with Mixins When using the expandPermutations decorator on Mixin classes or in classes which will be inherited from it is important to remove the PERMUTATION_ATTR from functions that have been processed. Otherwise we will attempt to expand already expanded permutations when subclassing or mixing this class. Change-Id: Idcc0805ce2e77bfa4987142ccfd76b4003222471 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/testlib.py M tests/testlibTests.py 2 files changed, 14 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/51/44151/1 diff --git a/tests/testlib.py b/tests/testlib.py index e6bf427..e015269 100644 --- a/tests/testlib.py +++ b/tests/testlib.py @@ -79,9 +79,9 @@ permName = "%s(%s)" % (f.__name__, argsString) wrapper = _getPermutation(f, args) wrapper.__name__ = permName + delattr(wrapper, PERMUTATION_ATTR) setattr(cls, permName, wrapper) - delattr(cls, f.__name__) return cls diff --git a/tests/testlibTests.py b/tests/testlibTests.py index c0961b8..4dc8368 100644 --- a/tests/testlibTests.py +++ b/tests/testlibTests.py @@ -21,6 +21,7 @@ from testlib import AssertingLock from testlib import VdsmTestCase from testlib import recorded +from testlib import permutations, expandPermutations class AssertNotRaisesTests(VdsmTestCase): @@ -127,3 +128,15 @@ @recorded def no_args(self): pass + + +@expandPermutations +class PermutationsMixin(object): + @permutations([['True']]) + def test_mixin_permutations(self, param): + self.assertTrue(param) + + +@expandPermutations +class TestPermutationsMixin(PermutationsMixin, VdsmTestCase): + pass -- To view, visit https://gerrit.ovirt.org/44151 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idcc0805ce2e77bfa4987142ccfd76b4003222471 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 14

Change in vdsm[master]: tests: Use more random interface names
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Use more random interface names ...................................................................... tests: Use more random interface names We used random.sample(), choosing unique characters, instead of choosing any sequence from the charset, which gives more choices. Change-Id: Iffcabc7992e787d1e2cf38ce4ad2164cc80c646e Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/tcTests.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/44111/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index 4453730..7f16e9f 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -69,7 +69,7 @@ def _generateRandomName(self, prefix): char_set = string.ascii_letters + string.digits - return prefix + ''.join(random.sample(char_set, 5)) + return prefix + ''.join(random.choice(char_set) for _ in range(5)) def _ifUp(self): check_call([EXT_IP, "link", "set", self.devName, "up"]) -- To view, visit https://gerrit.ovirt.org/44111 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iffcabc7992e787d1e2cf38ce4ad2164cc80c646e Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

5 16

Change in vdsm[master]: tests: Isolate tcTests test properly
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 3: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tests: Abusing permutations to run tests multiple times
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Abusing permutations to run tests multiple times ...................................................................... tests: Abusing permutations to run tests multiple times Run random failing test 1000 times to check if the previous patch is making a difference. Change-Id: Iebdd83cdce82abb618efc9805156e3554166092c Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/tcTests.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/44109/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index c5f35cb..5a8b8d1 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -33,6 +33,7 @@ import struct from testlib import VdsmTestCase as TestCaseBase +from testlib import expandPermutations, permutations from testValidation import ValidateRunningAsRoot from vdsm.constants import EXT_BRCTL, EXT_TC @@ -397,6 +398,7 @@ self.assertEqual(parsed, correct) +@expandPermutations class TestPortMirror(TestCaseBase): """ @@ -491,7 +493,8 @@ self.assertFalse(self._sendPing(), "Bridge received mirrored ping " "requests, but mirroring is unset.") - def testMirroringWithDistraction(self): + @permutations([["%03d" % i] for i in range(1000)]) + def testMirroringWithDistraction(self, _): "setting another mirror action should not obstract the first one" tc.setPortMirroring(self._bridge0.devName, self._bridge2.devName) self.testMirroring() -- To view, visit https://gerrit.ovirt.org/44109 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iebdd83cdce82abb618efc9805156e3554166092c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

3 10

Change in vdsm[master]: tests: Wait until an interface is up
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Wait until an interface is up ...................................................................... tests: Wait until an interface is up We are waiting until a device is down in ifdown, but we still get this error in the tests from time to time: 00:03:53 root: DEBUG: /usr/sbin/brctl addbr vdsmtest-aijc4 (cwd None) 00:03:53 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:03:53 root: DEBUG: /sbin/ip link set vdsmtest-aijc4 up (cwd None) 00:03:53 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:03:53 root: DEBUG: /usr/sbin/tc qdisc add dev vdsmtest-aijc4 ingress (cwd None) 00:03:53 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:03:53 root: DEBUG: /sbin/ip link set vdsmtest-aijc4 down (cwd None) 00:03:53 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0 00:03:53 root: DEBUG: /usr/sbin/brctl delbr vdsmtest-aijc4 (cwd None) 00:03:53 root: DEBUG: FAILED: <err> = "bridge vdsmtest-aijc4 is still up; can't delete it\n"; <rc> = 1 Since all the logs are in the same second, I suspect that ifdown times out since we send it too early, leading to failure when removing the bridge. Trying to wait until ifup is done. Change-Id: I39192d1e07a8a8b5ba8ed5a7a76b6441146dc91b Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/tcTests.py 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/08/44108/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index 6f0c37e..c5f35cb 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -71,7 +71,12 @@ return prefix + ''.join(random.sample(char_set, 5)) def _ifUp(self): - check_call([EXT_IP, "link", "set", self.devName, "up"]) + with monitor.Monitor(groups=('link',), timeout=2) as mon: + check_call([EXT_IP, "link", "set", self.devName, "up"]) + for event in mon: + if (event.get('name') == self.devName and + event.get('state') == 'up'): + return def _ifDown(self): with monitor.Monitor(groups=('link',), timeout=2) as mon: -- To view, visit https://gerrit.ovirt.org/44108 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I39192d1e07a8a8b5ba8ed5a7a76b6441146dc91b Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 19

Change in vdsm[master]: tests: Isolate tcTests test properly
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 2: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 22: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 22 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by fromani＠redhat.com 29 Jul '15

29 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 21: Verified+1 works: libvirtEventLoop::INFO::2015-07-29 17:56:07,654::vm::3608::virt.vm::(onIOError) vmId=`b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb`::abnormal vm stop device virtio-disk0 error Thread-32::INFO::2015-07-29 17:56:07,655::monitor::273::Storage.Monitor::(_notifyStatusChanges) Domain bbed5784-b0ee-4a0a-aff2-801da0bcf39e became INVALID libvirtEventLoop::INFO::2015-07-29 17:56:07,656::vm::4903::virt.vm::(_logGuestCpuStatus) vmId=`b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb`::CPU stopped: onIOError Thread-136::DEBUG::2015-07-29 17:56:07,656::misc::777::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event libvirtEventLoop::DEBUG::2015-07-29 17:56:07,656::vm::393::virt.vm::(send_status_event) vmId=`b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb`::Last status Paused and evaluated status Paused Thread-136::DEBUG::2015-07-29 17:56:07,656::misc::787::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` libvirtEventLoop::DEBUG::2015-07-29 17:56:07,657::__init__::206::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4406657030, "b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb": {"status": "Paused", "ioerror": {"alias": "virtio-disk0", "name": "vda", "path": "/rhev/data-center/00000001-0001-0001-0001-0000000001e8/bbed5784-b0ee-4a0a-aff2-801da0bcf39e/images/bb026109-3886-438a-be55-37797c2bbfb0/667fe374-53ed-4498-a388-c90acf3d4659"}, "pauseCode": ""}}, "jsonrpc": "2.0", "method": "|virt|VM_status|b0f37f43-ff02-4dbf-ba8c-dfcc0f5056bb"} empty pauseCode is addressed by: https://gerrit.ovirt.org/#/c/44115/ -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 21 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: hyperv: enforce hypervclock presence
by fromani＠redhat.com 29 Jul '15

29 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: hyperv: enforce hypervclock presence ...................................................................... hyperv: enforce hypervclock presence It is not enough for libvirt to have the <clock ...> <timer name='hypervclock'> </clock> attribute in the domain XML to actually enable the timer. Libvirt also want to make sure that the timer is present. Hence, this patch adds the 'present=yes' attribute to enforce the timer activation. Change-Id: I24fb9f8b8414d0550c0478571365f8bd6d4ee363 Bug-Url: https://bugzilla.redhat.com/1083529 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/vmTests.py M vdsm/virt/vmxml.py 2 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/44119/1 diff --git a/tests/vmTests.py b/tests/vmTests.py index bf7af3b..0c30dda 100644 --- a/tests/vmTests.py +++ b/tests/vmTests.py @@ -338,7 +338,7 @@ def testHyperVClockXML(self): clockXML = """ <clock adjustment="-3600" offset="variable"> - <timer name="hypervclock"/> + <timer name="hypervclock" present="yes"/> <timer name="rtc" tickpolicy="catchup"/> <timer name="pit" tickpolicy="delay"/> <timer name="hpet" present="no"/> diff --git a/vdsm/virt/vmxml.py b/vdsm/virt/vmxml.py index 945a60b..2c51272 100644 --- a/vdsm/virt/vmxml.py +++ b/vdsm/virt/vmxml.py @@ -188,7 +188,7 @@ for hyperv: <clock offset="variable" adjustment="-3600"> - <timer name="hypervclock"> + <timer name="hypervclock" present="yes"> <timer name="rtc" tickpolicy="catchup"> </clock> """ @@ -196,7 +196,7 @@ m = Element('clock', offset='variable', adjustment=str(self.conf.get('timeOffset', 0))) if utils.tobool(self.conf.get('hypervEnable', 'false')): - m.appendChildWithArgs('timer', name='hypervclock') + m.appendChildWithArgs('timer', name='hypervclock', present='yes') m.appendChildWithArgs('timer', name='rtc', tickpolicy='catchup') m.appendChildWithArgs('timer', name='pit', tickpolicy='delay') -- To view, visit https://gerrit.ovirt.org/44119 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I24fb9f8b8414d0550c0478571365f8bd6d4ee363 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 6

Change in vdsm[master]: net/api: drop support for legacy mgmt networks
by Dan Kenigsberg 29 Jul '15

29 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: net/api: drop support for legacy mgmt networks ...................................................................... net/api: drop support for legacy mgmt networks Change-Id: I45d956b117f7a6cbfa97ce7e060a08307e62a883 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M lib/vdsm/constants.py.in M vdsm/network/api.py 2 files changed, 2 insertions(+), 12 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/15/43615/1 diff --git a/lib/vdsm/constants.py.in b/lib/vdsm/constants.py.in index 5aff44e..0c85b9f 100644 --- a/lib/vdsm/constants.py.in +++ b/lib/vdsm/constants.py.in @@ -23,9 +23,6 @@ from __future__ import absolute_import import os -# VDSM management networks -LEGACY_MANAGEMENT_NETWORKS = ('ovirtmgmt', 'rhevm') - # SMBIOS manufacturer SMBIOS_MANUFACTURER = '@SMBIOS_MANUFACTURER@' SMBIOS_OSNAME = '@SMBIOS_OSNAME@' diff --git a/vdsm/network/api.py b/vdsm/network/api.py index 5ad222d..0d91da5 100755 --- a/vdsm/network/api.py +++ b/vdsm/network/api.py @@ -87,7 +87,7 @@ bondingOptions=None, nics=None, mtu=None, ipaddr=None, netmask=None, gateway=None, bootproto=None, ipv6addr=None, ipv6gateway=None, ipv6autoconf=None, - dhcpv6=None, defaultRoute=None, _netinfo=None, + dhcpv6=None, defaultRoute=False, _netinfo=None, configurator=None, blockingdhcp=None, implicitBonding=None, opts=None): """ @@ -242,7 +242,7 @@ ipv6addr=None, ipv6gateway=None, ipv6autoconf=None, force=False, configurator=None, bondingOptions=None, bridged=True, _netinfo=None, hostQos=None, - defaultRoute=None, blockingdhcp=False, **options): + defaultRoute=False, blockingdhcp=False, **options): nics = nics or () if _netinfo is None: _netinfo = netinfo.NetInfo() @@ -280,13 +280,6 @@ else: for nic in nics: _validateInterNetworkCompatibility(_netinfo, vlan, nic) - - # defaultRoute is set either explicitly by the client, OR if we're adding - # the management network. - # TODO: When oVirt 3.3 is deprecated, change the default to False and - # remove reference to constants.LEGACY_MANAGEMENT_NETWORKS - if defaultRoute is None: - defaultRoute = network in constants.LEGACY_MANAGEMENT_NETWORKS logging.info("Adding network %s with vlan=%s, bonding=%s, nics=%s," " bondingOptions=%s, mtu=%s, bridged=%s, defaultRoute=%s," -- To view, visit https://gerrit.ovirt.org/43615 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I45d956b117f7a6cbfa97ce7e060a08307e62a883 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

3 5

Change in vdsm[master]: mount: Better and simpler __hash__
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: mount: Better and simpler __hash__ ...................................................................... mount: Better and simpler __hash__ Mixing type(self) in a hash is not a good idea, all instance of old style class have the same type. It is better to use the __class__ attribute to get the hash of the class. The hash values were mixed in a poor way. There is no need to implement when we can use the hash function of the tuple object, which does a better job. Change-Id: I6030c441c7137d65f4038d86e9813719429250c0 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/mount.py 1 file changed, 1 insertion(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/78/43978/1 diff --git a/vdsm/storage/mount.py b/vdsm/storage/mount.py index e818fd1..1626a26 100644 --- a/vdsm/storage/mount.py +++ b/vdsm/storage/mount.py @@ -206,10 +206,7 @@ return not self == other def __hash__(self): - hsh = hash(type(self)) - hsh ^= hash(self.fs_spec) - hsh ^= hash(self.fs_file) - return hsh + return hash((self.__class__, self.fs_spec, self.fs_file)) def mount(self, mntOpts=None, vfstype=None, timeout=None, cgroup=None): cmd = [constants.EXT_MOUNT] -- To view, visit https://gerrit.ovirt.org/43978 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I6030c441c7137d65f4038d86e9813719429250c0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 15

Change in vdsm[master]: tests: Isolate tcTests test properly
by ibarkan＠redhat.com 29 Jul '15

29 Jul '15

Ido Barkan has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 1: Code-Review+1 -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: virt: enable libgfapi with snapshot support
by Federico Simoncelli 29 Jul '15

29 Jul '15

Federico Simoncelli has uploaded a new change for review. Change subject: virt: enable libgfapi with snapshot support ...................................................................... virt: enable libgfapi with snapshot support Change-Id: Ie0965bef605ba67297670c0bf7924f88fa3b0460 Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> --- M vdsm/storage/hsm.py M vdsm/virt/vm.py 2 files changed, 38 insertions(+), 17 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/33768/1 diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py index cd6f4b9..bde04be 100644 --- a/vdsm/storage/hsm.py +++ b/vdsm/storage/hsm.py @@ -3225,8 +3225,7 @@ path = os.path.join(dom.domaindir, sd.DOMAIN_IMAGES, imgUUID, volUUID) volInfo = {'domainID': sdUUID, 'imageID': imgUUID, - 'volumeID': volUUID, 'path': path, - 'volType': "path"} + 'volumeID': volUUID, 'path': path} leasePath, leaseOffset = dom.getVolumeLease(imgUUID, volUUID) @@ -3237,8 +3236,8 @@ }) imgVolumesInfo.append(volInfo) - if volUUID == leafUUID: - leafInfo = volInfo + + leafInfo = dom.produceVolume(imgUUID, volUUID).getVmVolumeInfo() return {'path': leafPath, 'info': leafInfo, 'imgVolumesInfo': imgVolumesInfo} diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index 1eebb69..af0830a 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -3821,20 +3821,43 @@ def snapshot(self, snapDrives, memoryParams): """Live snapshot command""" - def _diskSnapshot(vmDev, newPath, sourceType): + def _diskSnapshot(vmDev, oldDrive, newDriveDict): """Libvirt snapshot XML""" + sourceType = 'file' + sourceAttrs = {} + hostAttrs = {} + + # TODO: unify these with the Drive.getXML() method + if oldDrive.networkDev: + sourceType = 'network' + sourceAttrs.update({ + 'name': newDriveDict['volumeInfo']['path'], + 'protocol': newDriveDict['volumeInfo']['protocol']}) + hostAttrs.update({ + 'name': newDriveDict['volumeInfo']['volfileServer'], + 'port': newDriveDict['volumeInfo']['volPort'], + 'transport': newDriveDict['volumeInfo']['volTransport']}) + else: + sourceAttrs.update({'file': newDriveDict['path']}) + + # Libvirt versions before 1.2.2 do not understand 'type' + # and treat all snapshots as if they are type='file'. + # In order to ensure proper handling of block snapshots + # in modern libvirt versions, we specify type='block' + # and dev=path for block volumes but we always speficy + # the file=path for backwards compatibility. + if oldDrive.blockDev: + sourceType = 'block' + sourceAttrs.update({'dev': newDriveDict['path']}) disk = vmxml.Element('disk', name=vmDev, snapshot='external', type=sourceType) - # Libvirt versions before 1.2.2 do not understand 'type' and treat - # all snapshots as if they are type='file'. In order to ensure - # proper handling of block snapshots in modern libvirt versions, - # we specify type='block' and dev=path for block volumes but we - # always speficy the file=path for backwards compatibility. - args = {'type': sourceType, 'file': newPath} - if sourceType == 'block': - args['dev'] = newPath - disk.appendChildWithArgs('source', **args) + + source = disk.appendChildWithArgs('source', **sourceAttrs) + + if hostAttrs: + source.appendChildWithArgs('host', **hostAttrs) + return disk def _normSnapDriveParams(drive): @@ -3964,9 +3987,8 @@ _rollbackDrives(preparedDrives) return errCode['snapshotErr'] - snapType = 'block' if vmDrives[vmDevName].blockDev else 'file' - snapelem = _diskSnapshot(vmDevName, newDrives[vmDevName]["path"], - snapType) + snapelem = _diskSnapshot(vmDevName, vmDrives[vmDevName], + newDrives[vmDevName]) disks.appendChild(snapelem) snap.appendChild(disks) -- To view, visit http://gerrit.ovirt.org/33768 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie0965bef605ba67297670c0bf7924f88fa3b0460 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>

7 12

Change in vdsm[master]: vm: event: send status event on IO Error
by michal.skrivanek＠redhat.com 29 Jul '15

29 Jul '15

Michal Skrivanek has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 21: (1 comment) https://gerrit.ovirt.org/#/c/43312/21/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 3622: else: Line 3623: io_error_info['name'] = drive.name Line 3624: io_error_info['path'] = drive.path Line 3625: Line 3626: self.send_status_event(pauseCode=reason, ioerror=io_error_info) > we should document the data sent back in events - at least the minimal key I asked Piotr&Oved the same question...we need to have some documentation guidelines before it's a complete mess. I don't think we have anything yet:( Line 3627: Line 3628: @property Line 3629: def hasSpice(self): Line 3630: return (self.conf.get('display') == 'qxl' or -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 21 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: Yes

1 0

Change in vdsm[master]: tests: Isolate tcTests test properly
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 1: (1 comment) https://gerrit.ovirt.org/#/c/44144/1/tests/tcTests.py File tests/tcTests.py: Line 461: try: Line 462: iface.delDevice() Line 463: except Exception: Line 464: self.log.exception("Error removing device %s" % iface) Line 465: raise t, v, tb Need to use python 3 friendly version Line 466: Line 467: def tearDown(self): Line 468: cleanup = [self._tap0, self._tap1, self._tap2, self._bridge0, Line 469: self._bridge1, self._bridge2] -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Ido Barkan <ibarkan(a)redhat.com> Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: Roman Mohr <rmohr(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: Yes

1 0

Change in vdsm[master]: tests: Isolate tcTests test properly
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: tests: Isolate tcTests test properly ...................................................................... Patch Set 1: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/44144 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I3861ccfb432541b63d7ba5814f7e3ae339ec8105 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tests: Silence momPolicyTests
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Silence momPolicyTests ...................................................................... tests: Silence momPolicyTests The code triggered by these tests log to stderr, polluting test progress. We now redirect stderr to stdout during these tests, so log output is displayed only if a test failed. Change-Id: Ice082d703ac36e17a8f3ed3cc3b4e4b144265c27 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/momPolicyTests.py 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/43621/1 diff --git a/tests/momPolicyTests.py b/tests/momPolicyTests.py index 81a5969..c4800c5 100644 --- a/tests/momPolicyTests.py +++ b/tests/momPolicyTests.py @@ -1,10 +1,12 @@ import os.path import ConfigParser +import sys from mom.Policy.Policy import Policy from mom.Entity import Entity from mom.Monitor import Monitor from unittest import TestCase +import monkeypatch # This is a very hacky way of implementing the test scenario # we should update MOM to offer test capability and use it here. @@ -46,6 +48,7 @@ return ent + @monkeypatch.MonkeyPatch(sys, 'stderr', sys.stdout) def testCpuTuneBasicTest(self): p = Policy() @@ -69,6 +72,7 @@ self.assertEqual(vm.controls["vcpu_quota"], 50000) self.assertEqual(vm.controls["vcpu_period"], 100000) + @monkeypatch.MonkeyPatch(sys, 'stderr', sys.stdout) def testCpuTuneHundredCpus(self): p = Policy() @@ -92,6 +96,7 @@ self.assertEqual(vm.controls["vcpu_quota"], 60000) self.assertEqual(vm.controls["vcpu_period"], 100000) + @monkeypatch.MonkeyPatch(sys, 'stderr', sys.stdout) def testCpuTuneNoLimit(self): p = Policy() @@ -115,6 +120,7 @@ self.assertEqual(vm.controls["vcpu_quota"], -1) self.assertEqual(vm.controls["vcpu_period"], 100000) + @monkeypatch.MonkeyPatch(sys, 'stderr', sys.stdout) def testCpuTuneTooSmall(self): p = Policy() -- To view, visit https://gerrit.ovirt.org/43621 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ice082d703ac36e17a8f3ed3cc3b4e4b144265c27 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 12

Change in vdsm[master]: migration: fix parameters
by ykaplan＠redhat.com 29 Jul '15

29 Jul '15

Yeela Kaplan has uploaded a new change for review. Change subject: migration: fix parameters ...................................................................... migration: fix parameters Change-Id: I940eeb557378d53670779607d4a1379be3ac0c09 Signed-off-by: Yeela Kaplan <ykaplan(a)redhat.com> --- M vdsm/virt/migration.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/20/44120/1 diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py index 54cbde0..a3e5027 100644 --- a/vdsm/virt/migration.py +++ b/vdsm/virt/migration.py @@ -143,7 +143,7 @@ client = self._createClient(port) requestQueues = config.get('addresses', 'request_queues') requestQueue = requestQueues.split(",")[0] - self._destServer = jsonrpcvdscli.connect(client, requestQueue) + self._destServer = jsonrpcvdscli.connect(requestQueue, client) self.log.debug('Initiating connection with destination') self._destServer.ping() -- To view, visit https://gerrit.ovirt.org/44120 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I940eeb557378d53670779607d4a1379be3ac0c09 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yeela Kaplan <ykaplan(a)redhat.com>

5 8

Change in vdsm[master]: lib: executor: get and show TID alongside name
by fromani＠redhat.com 29 Jul '15

29 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: lib: executor: get and show TID alongside name ...................................................................... lib: executor: get and show TID alongside name For debug purposes, it is useful to map python threads to linux TIDs. Most prominent example is to understand when an executor Worker thread is replaced, and what happens afterwards. A previous attempt to set system names for threads may have solved this issue, but it stalled on the problem of shrkinking long python names to 15 character most, to fit the linux threads implementation. For the short -and maybe even long- term, a cheap, easy but still useful solution is to grab and log the Linux Task Identifier (TID) alongside the python thread name. Since TIDs are basically PIDs for threads, this make easier to find a python thread froma system tool, like gdk (or pstack). To implement this, we add a trivial wrapper in lib/vdsm/pthread.py, inspired by https://code.google.com/p/psutil/issues/detail?id=186 and we extend lib/vdsm/executor.py to make use of it. Change-Id: Ibb4e4974ef29dad5f764450d4ed29c61cae31d21 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M lib/vdsm/executor.py M lib/vdsm/pthread.py 2 files changed, 15 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/43771/1 diff --git a/lib/vdsm/executor.py b/lib/vdsm/executor.py index 2639b26..065dbb5 100644 --- a/lib/vdsm/executor.py +++ b/lib/vdsm/executor.py @@ -27,6 +27,7 @@ import logging import threading +from . import pthread from . import utils @@ -149,12 +150,17 @@ self._executor = executor self._scheduler = scheduler self._discarded = False + self._tid = None _Worker._id += 1 name = "%s-worker-%d" % (self._executor.name, _Worker._id) self._thread = threading.Thread(target=self._run, name=name) self._thread.daemon = True self._log.debug('Starting worker %s' % name) self._thread.start() + + @property + def tid(self): + return self._tid @property def name(self): @@ -166,14 +172,15 @@ @utils.traceback(on=_log.name) def _run(self): - self._log.debug('Worker started') + self._tid = pthread.gettid() + self._log.debug('Worker started (tid=%d)', self._tid) try: while True: self._execute_task() except NotRunning: self._log.debug('Worker stopped') except _WorkerDiscarded: - self._log.debug('Worker was discarded') + self._log.debug('Worker was discarded (tid=%d)', self._tid) finally: self._executor._worker_stopped(self) diff --git a/lib/vdsm/pthread.py b/lib/vdsm/pthread.py index 851afd1..1433fdb 100644 --- a/lib/vdsm/pthread.py +++ b/lib/vdsm/pthread.py @@ -26,7 +26,9 @@ NAME_MAX_LENGTH = 15 +_SYS_GETTID = 186 +_LIBC = ctypes.CDLL('libc.so.6') _LIBPTHREAD = ctypes.CDLL("libpthread.so.0", use_errno=True) _pthread_setname_np_proto = ctypes.CFUNCTYPE( @@ -88,3 +90,7 @@ thread = threading.current_thread() _pthread_getname_np(thread.ident, buf, bufsize) return buf.value + + +def gettid(): + return _LIBC.syscall(_SYS_GETTID) -- To view, visit https://gerrit.ovirt.org/43771 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibb4e4974ef29dad5f764450d4ed29c61cae31d21 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 9

Change in vdsm[master]: net: always persist owned ifcfg files on ovirt node
by ibarkan＠redhat.com 29 Jul '15

29 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: net: always persist owned ifcfg files on ovirt node ...................................................................... net: always persist owned ifcfg files on ovirt node Change-Id: Ibc717b86194a32c050d346e235a5c35fd318e1ff Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M vdsm/network/configurators/ifcfg.py M vdsm/vdsm-store-net-config.in 2 files changed, 9 insertions(+), 4 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/12/44112/1 diff --git a/vdsm/network/configurators/ifcfg.py b/vdsm/network/configurators/ifcfg.py index f2dbd5e..83b95f9 100644 --- a/vdsm/network/configurators/ifcfg.py +++ b/vdsm/network/configurators/ifcfg.py @@ -479,10 +479,6 @@ self._backup(fileName) configuration = self.CONFFILE_HEADER + '\n' + configuration - # make sure that ifcfg files are never persisted by the node - if self.unifiedPersistence and utils.isOvirtNode(): - node_fs.Config().unpersist(fileName) - logging.debug('Writing to file %s configuration:\n%s', fileName, configuration) with open(fileName, 'w') as confFile: diff --git a/vdsm/vdsm-store-net-config.in b/vdsm/vdsm-store-net-config.in index c19d5e0..4e84fb7 100755 --- a/vdsm/vdsm-store-net-config.in +++ b/vdsm/vdsm-store-net-config.in @@ -37,8 +37,17 @@ rm -rf "$NET_CONF_BACK_DIR"/* } +node_persist_owned_ifcfgs() { + for f in $(find "$NET_CONF_DIR" -type f); do + if grep -q "# Generated by VDSM version" $f; then + ovirt_store_config "$f" + fi + done +} + unified_node_persist() { unified_nonnode_persist + node_persist_owned_ifcfgs # oVirt node ovirt_store_config puts the dir in persistent storage and # bind mounts it in the original place. So that's all we really need to do. -- To view, visit https://gerrit.ovirt.org/44112 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibc717b86194a32c050d346e235a5c35fd318e1ff Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 11

Change in vdsm[master]: vm: event: send status event on IO Error
by michal.skrivanek＠redhat.com 29 Jul '15

29 Jul '15

Michal Skrivanek has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 21: Code-Review+1 -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 21 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by fromani＠redhat.com 29 Jul '15

29 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 21: (1 comment) https://gerrit.ovirt.org/#/c/43312/21/vdsm/virt/vm.py File vdsm/virt/vm.py: Line 3622: else: Line 3623: io_error_info['name'] = drive.name Line 3624: io_error_info['path'] = drive.path Line 3625: Line 3626: self.send_status_event(pauseCode=reason, ioerror=io_error_info) we should document the data sent back in events - at least the minimal key set we ensure to always send back ('status'?) where should this documentation be put? vdsmapi-schema.json? different schema for events only? Line 3627: Line 3628: @property Line 3629: def hasSpice(self): Line 3630: return (self.conf.get('display') == 'qxl' or -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 21 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: Yes

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 21: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 21 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by automation＠ovirt.org 29 Jul '15

29 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 20: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 20 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vmstats: report network stats only if present
by fromani＠redhat.com 29 Jul '15

29 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: report network stats only if present ...................................................................... vmstats: report network stats only if present avoid to report empty values for network stats, as this may confuse Engine and let it to believe there is no networks at all. Change-Id: I8b3fd2fe7913c3a7b80b5d2c7ffe55e8fb5f8976 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/16/43716/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index 10d0776..caa2fc2 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -206,10 +206,11 @@ def networks(vm, stats, first_sample, last_sample, interval): - stats['network'] = {} if first_sample is None or last_sample is None: return + + net_stats = {} for nic in vm.getNicDevices(): if nic.name.startswith('hostdev'): @@ -220,10 +221,13 @@ # may happen if nic is a new hot-plugged one if not first_nic or not last_nic: continue - stats['network'][nic.name] = nic( + net_stats[nic.name] = nic( nic.name, nic.nicModel, nic.macAddr, first_nic, last_nic, interval) + if net_stats: + stats['network'] = net_stats + def disks(vm, stats, first_sample, last_sample, interval): if first_sample is None or last_sample is None: -- To view, visit https://gerrit.ovirt.org/43716 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8b3fd2fe7913c3a7b80b5d2c7ffe55e8fb5f8976 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

2 9

Change in vdsm[master]: net: fix tSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured
by ibarkan＠redhat.com 29 Jul '15

29 Jul '15

Ido Barkan has uploaded a new change for review. Change subject: net: fix tSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured ...................................................................... net: fix tSetupNetworksDeletesTheBridgeOnlyWhenItIsReconfigured since the tap object itself went out of scope, the kernel removed the device during the test since the file descriptor held by the Tap object was closed. The test was also fixed to remove the device before calling setupNetworks that would attempt to delete the bridge. Change-Id: I52d0c04ffd3c047c1e66cec3d1a9646717d90a21 Signed-off-by: Ido Barkan <ibarkan(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 4 insertions(+), 8 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/95/43995/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index 7f5ab37..d930214 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -1005,15 +1005,11 @@ return link.index def add_tap_to_bridge(): - tap = Tap() + tap = Tap(prefix='vnet') tap.addDevice() rc, _, _ = execCmd([EXT_BRCTL, 'addif', NETWORK_NAME, tap.devName]) self.assertEquals(rc, 0, 'brctl addif failed: rc=%s' % (rc,)) - return tap.devName - - def remove_tap_from_bridge(tap_name): - rc, _, _ = execCmd([EXT_BRCTL, 'delif', NETWORK_NAME, tap_name]) - self.assertEquals(rc, 0, 'brctl delif failed: rc=%s' % (rc,)) + return tap STANDARD, BIG = 1500, 2000 with dummyIf(2) as nics: @@ -1025,7 +1021,7 @@ self.assertMtu(STANDARD, NETWORK_NAME, first) bridge_index = get_bridge_index() # simulate a vm connected to the bridge - tap_name = add_tap_to_bridge() + tap = add_tap_to_bridge() second_net = {NETWORK_NAME: dict(bridged=True, nic=second, mtu=BIG, vlan=VLAN_ID)} @@ -1033,6 +1029,7 @@ self.assertEquals(status, SUCCESS, msg) second_bridge_index = get_bridge_index() self.assertEquals(bridge_index, second_bridge_index) + tap.delDevice() # the kernel bridge driver automatically updates the bridge to the # new minimum MTU of all of its connected interfaces self.assertMtu(BIG, NETWORK_NAME, second) @@ -1054,7 +1051,6 @@ self.assertEquals(status, SUCCESS, msg) self.assertNotEqual(second_bridge_index, get_bridge_index()) - remove_tap_from_bridge(tap_name) status, msg = self.setupNetworks({NETWORK_NAME: {'remove': True}}, {}, NOCHK) self.assertEquals(status, SUCCESS, msg) -- To view, visit https://gerrit.ovirt.org/43995 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I52d0c04ffd3c047c1e66cec3d1a9646717d90a21 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ido Barkan <ibarkan(a)redhat.com>

3 14

Change in vdsm[master]: tests: Fix error handling when checking dependencies
by Nir Soffer 29 Jul '15

29 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Fix error handling when checking dependencies ...................................................................... tests: Fix error handling when checking dependencies We used to hide *all* exceptions and skip the tests with possible bogus message. When trying brctl, we skip the test only if we cannot run it, since this is the only error that can cause by missing bridge-utils. Any other error should fail the tests. When trying tc, we show the failure in the error message. Also removed pointless code opening and closing /dev/null. Change-Id: I180ba6449009dc986d2df58958039e2324d2191f Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/tcTests.py 1 file changed, 9 insertions(+), 8 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/44110/1 diff --git a/tests/tcTests.py b/tests/tcTests.py index c5f35cb..92bf4d7 100644 --- a/tests/tcTests.py +++ b/tests/tcTests.py @@ -19,6 +19,7 @@ # Refer to the README and COPYING files for full details of the license # +import errno import random import time import string @@ -178,18 +179,18 @@ dev = _Bridge() try: dev.addDevice() - except: - raise SkipTest("'brctl' has failed. Do you have bridge-utils " - "installed?") + except OSError as e: + if e.errno == errno.ENOENT: + raise SkipTest("Cannot run %r: %s\nDo you have bridge-utils " + "installed?" % (EXT_BRCTL, e)) + raise - null = open("/dev/null", "a") try: check_call([EXT_TC, 'qdisc', 'add', 'dev', dev.devName, 'ingress']) - except: - raise SkipTest("'tc' has failed. Do you have Traffic Control kernel " - "modules installed?") + except ExecError as e: + raise SkipTest("%r has failed: %s\nDo you have Traffic Control kernel " + "modules installed?" % (EXT_TC, e.err)) finally: - null.close() dev.delDevice() -- To view, visit https://gerrit.ovirt.org/44110 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I180ba6449009dc986d2df58958039e2324d2191f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 8

Change in vdsm[master]: mount: Implement __ne__
by Nir Soffer 28 Jul '15

28 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: mount: Implement __ne__ ...................................................................... mount: Implement __ne__ There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected. See https://docs.python.org/2/reference/datamodel.html#object.__eq__ Change-Id: I9b08fe7ee472c3582ee9816988db66e92c2542ea Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/mount.py 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/77/43977/1 diff --git a/vdsm/storage/mount.py b/vdsm/storage/mount.py index becd541..e818fd1 100644 --- a/vdsm/storage/mount.py +++ b/vdsm/storage/mount.py @@ -202,6 +202,9 @@ self.fs_spec == other.fs_spec and self.fs_file == other.fs_file) + def __ne__(self, other): + return not self == other + def __hash__(self): hsh = hash(type(self)) hsh ^= hash(self.fs_spec) -- To view, visit https://gerrit.ovirt.org/43977 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9b08fe7ee472c3582ee9816988db66e92c2542ea Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 12

Change in vdsm[master]: mount: More correct and simpler __eq__
by Nir Soffer 28 Jul '15

28 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: mount: More correct and simpler __eq__ ...................................................................... mount: More correct and simpler __eq__ Instance of different classes should not be considered equal, as subclass may add new state not in the superclass. If an instance is missing an attribute it should have, we don't want to hide this error, and we certainly do not want to swallow *any* Exception during the check. Change-Id: Ib7b182fcd2ae95d2e0c310795a794f9e48d8d13f Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M vdsm/storage/mount.py 1 file changed, 3 insertions(+), 8 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/76/43976/1 diff --git a/vdsm/storage/mount.py b/vdsm/storage/mount.py index 4786ccb..becd541 100644 --- a/vdsm/storage/mount.py +++ b/vdsm/storage/mount.py @@ -198,14 +198,9 @@ self.fs_file = normpath(fs_file) def __eq__(self, other): - if not isinstance(other, Mount): - return False - - try: - return (other.fs_spec == self.fs_spec and - other.fs_file == self.fs_file) - except Exception: - return False + return (self.__class__ == other.__class__ and + self.fs_spec == other.fs_spec and + self.fs_file == other.fs_file) def __hash__(self): hsh = hash(type(self)) -- To view, visit https://gerrit.ovirt.org/43976 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib7b182fcd2ae95d2e0c310795a794f9e48d8d13f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 10

Change in vdsm[master]: tests: Add Mount equality and hash tests
by Nir Soffer 28 Jul '15

28 Jul '15

Nir Soffer has uploaded a new change for review. Change subject: tests: Add Mount equality and hash tests ...................................................................... tests: Add Mount equality and hash tests The relevant code has not tests and I want to clean it up, so lets have some tests to make sure the cleanup does not break anything. Change-Id: Ie958119aebfd2a5f96df14e9bba68a0c4f16f094 Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- M tests/mountTests.py 1 file changed, 37 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/75/43975/1 diff --git a/tests/mountTests.py b/tests/mountTests.py index 16a98b9..a6d9e54 100644 --- a/tests/mountTests.py +++ b/tests/mountTests.py @@ -28,7 +28,7 @@ from nose.plugins.skip import SkipTest from testlib import VdsmTestCase as TestCaseBase -from testlib import namedTemporaryDir +from testlib import namedTemporaryDir, expandPermutations, permutations from storage.misc import execCmd import storage.mount as mount from testValidation import checkSudo @@ -62,6 +62,42 @@ os.unlink(path) +@expandPermutations +class TestMountEquality(TestCaseBase): + + def test_eq(self): + m1 = mount.Mount("spec", "file") + m2 = mount.Mount("spec", "file") + self.assertEqual(m1, m2) + + @permutations([ + ("spec", "spec", "file1", "file2"), + ("spec1", "spec2", "file", "file"), + ]) + def test_ne(self, spec1, spec2, file1, file2): + m1 = mount.Mount(spec1, file1) + m2 = mount.Mount(spec2, file2) + self.assertNotEqual(m1, m2) + + +@expandPermutations +class TestMountHash(TestCaseBase): + + def test_eq(self): + m1 = mount.Mount("spec", "file") + m2 = mount.Mount("spec", "file") + self.assertEqual(hash(m1), hash(m2)) + + @permutations([ + ("spec", "spec", "file1", "file2"), + ("spec1", "spec2", "file", "file"), + ]) + def test_ne(self, spec1, spec2, file1, file2): + m1 = mount.Mount(spec1, file1) + m2 = mount.Mount(spec2, file2) + self.assertNotEqual(hash(m1), hash(m2)) + + class MountTests(TestCaseBase): def testLoopMount(self): checkSudo(["mount", "-o", "loop", "somefile", "target"]) -- To view, visit https://gerrit.ovirt.org/43975 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie958119aebfd2a5f96df14e9bba68a0c4f16f094 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>

4 13

Change in vdsm[ovirt-3.5]: test jenkins, please delete
by Dan Kenigsberg 28 Jul '15

28 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: test jenkins, please delete ...................................................................... test jenkins, please delete Change-Id: I88e0fccfe52b23d675a74e44e6da06c2b74bfbeb Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M lib/vdsm/netinfo.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/66/43566/1 diff --git a/lib/vdsm/netinfo.py b/lib/vdsm/netinfo.py index 01f25c7..129c0d7 100644 --- a/lib/vdsm/netinfo.py +++ b/lib/vdsm/netinfo.py @@ -781,7 +781,10 @@ def getIpAddresses(): "Return a list of the host's IP addresses" - return filter(None, [getaddr(i) for i in ethtool.get_active_devices()]) + try: + return filter(None, [getaddr(i) for i in ethtool.get_active_devices()]) + except UnicodeDecodeError: + raise RuntimeError(unicode(ethtool.get_active_devices())) def IPv4toMapped(ip): -- To view, visit https://gerrit.ovirt.org/43566 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I88e0fccfe52b23d675a74e44e6da06c2b74bfbeb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: ovirt-3.5 Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

2 3

Change in vdsm[master]: functional net test: set both bridge_opts and ethtool_opts
by Dan Kenigsberg 28 Jul '15

28 Jul '15

Dan Kenigsberg has uploaded a new change for review. Change subject: functional net test: set both bridge_opts and ethtool_opts ...................................................................... functional net test: set both bridge_opts and ethtool_opts This test has been written in an attempt to reproduce the referred bug, to no avail. Bug-Url: https://bugzilla.redhat.com/1186203 Change-Id: I57dd7fa0a05e83ab37393651f23cdf9b7e08ed89 Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M tests/functional/networkTests.py 1 file changed, 42 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/51/39251/1 diff --git a/tests/functional/networkTests.py b/tests/functional/networkTests.py index 8671244..81218fe 100644 --- a/tests/functional/networkTests.py +++ b/tests/functional/networkTests.py @@ -2394,3 +2394,45 @@ self.assertNetworkDoesntExist(NETWORK_NAME) self.assertBondDoesntExist(BONDING_NAME, [nic]) self.vdsm_net.save_config() + + @cleanupNet + @ValidateRunningAsRoot + def test_ethtool_bridge_opts(self): + status, msg, info = self.vdsm_net.getVdsCapabilities() + self.assertEqual(status, SUCCESS, msg) + if '30_ethtool_options' not in info['hooks']['after_network_setup']: + raise SkipTest('ethtool hook not installed') + + def get_gso(nic): + GSO = 'generic-segmentation-offload:' + rc, out, err = execCmd(['ethtool', '-k', nic]) + self.assertTrue(rc == 0, 'error reading gso: %s' % err) + for line in out: + if line.startswith(GSO): + return line[len(GSO) + 1:] + + def setup_bridge_and_eth_opts(nic, priority, gso): + bridge_opts = {'priority': priority} + custom = { + 'bridge_opts': 'priority=' + priority, + 'ethtool_opts': '-K * gso ' + gso + } + + status, msg = self.vdsm_net.setupNetworks( + {NETWORK_NAME: {'nic': nic, 'bridged': True, + 'vlan': VLAN_ID, + 'custom': custom}}, + {}, NOCHK) + self.assertEqual(status, SUCCESS, msg) + self.assertNetworkExists(NETWORK_NAME, bridgeOpts=bridge_opts) + self.assertEqual(get_gso(nic), gso) + + with dummyIf(1) as (nic, ): + setup_bridge_and_eth_opts(nic, '42', 'on') + setup_bridge_and_eth_opts(nic, '33', 'off') + + status, msg = self.vdsm_net.setupNetworks( + {NETWORK_NAME: {'remove': True}}, + {}, NOCHK) + self.assertEqual(status, SUCCESS, msg) + self.assertNetworkDoesntExist(NETWORK_NAME) -- To view, visit https://gerrit.ovirt.org/39251 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I57dd7fa0a05e83ab37393651f23cdf9b7e08ed89 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

3 7

Change in vdsm[master]: vm: event: document event on reboot
by automation＠ovirt.org 28 Jul '15

28 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: document event on reboot ...................................................................... Patch Set 19: * Update tracker::IGNORE, no Bug-Url found * Set MODIFIED::IGNORE, no Bug-Url found. -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 19 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: document event on reboot
by Dan Kenigsberg 28 Jul '15

28 Jul '15

Dan Kenigsberg has submitted this change and it was merged. Change subject: vm: event: document event on reboot ...................................................................... vm: event: document event on reboot The guest agent onStatusChange callback automatically delivers RebootInProgess event when VM reboots: we don't need anything extra here, but it is worth to explicitely document this behaviour. Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Signed-off-by: Francesco Romani <fromani(a)redhat.com> Reviewed-on: https://gerrit.ovirt.org/43309 Reviewed-by: Dan Kenigsberg <danken(a)redhat.com> Continuous-Integration: Jenkins CI --- M vdsm/virt/vm.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Jenkins CI: Passed CI tests Dan Kenigsberg: Looks good to me, approved Francesco Romani: Verified -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: merged Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 19 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 19: NON-SPM VDSM - works, but sends the bare miniumum information. We can and should do better. SPM VDSM - VDSM SIGTERMS itself to death, on recovery onIOError is delivered; Event "WaitForLaunch" is sent (useless) onIOError; later, at the end of domDependentInit, "Paused" with pauseCode "EOTHER" is sent. Needs more work. -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 19 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: tests: Create storagetestutils module
by alitke＠redhat.com 28 Jul '15

28 Jul '15

Adam Litke has uploaded a new change for review. Change subject: tests: Create storagetestutils module ...................................................................... tests: Create storagetestutils module The tests for VolumeMetadata want to reuse many of the functions that were implemented in manifest_tests.py. Move these into a helper module that can be shared by all storage tests. Change-Id: I713e19473b45bc05da5a1c888a034306144228c5 Signed-off-by: Adam Litke <alitke(a)redhat.com> --- M tests/Makefile.am A tests/storagetestutils.py 2 files changed, 101 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/47/43547/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 12ccfc2..df3b335 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -188,6 +188,7 @@ hookValidation.py \ monkeypatch.py \ storagefakelib.py \ + storagetestutils.py \ testlib.py \ testrunner.py \ testValidation.py \ diff --git a/tests/storagetestutils.py b/tests/storagetestutils.py new file mode 100644 index 0000000..a04ab58 --- /dev/null +++ b/tests/storagetestutils.py @@ -0,0 +1,100 @@ +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# +import os +import uuid +from contextlib import contextmanager + +from storage import sd, blockSD, fileSD + + +NR_PVS = 5 # The number of fake PVs we use to make a fake VG +MDSIZE = 524288 # The size (in bytes) of fake metadata files + + +class FakeMetadata(dict): + @contextmanager + def transaction(self): + yield + + +def make_blocksd_manifest(metadata=None, tmpdir=None): + sduuid = str(uuid.uuid4()) + if metadata is None: + metadata = FakeMetadata() + manifest = blockSD.BlockStorageDomainManifest(sduuid, metadata) + if tmpdir is not None: + manifest.domaindir = tmpdir + os.makedirs(os.path.join(manifest.domaindir, sduuid, sd.DOMAIN_IMAGES)) + return manifest + + +def get_device_list(count): + return ['/dev/mapper/{0}'.format(os.urandom(16).encode('hex')) + for _ in range(count)] + + +def make_vg(fake_lvm, manifest, devices=None): + vg_name = manifest.sdUUID + if devices is None: + devices = get_device_list(NR_PVS) + fake_lvm.createVG(vg_name, devices, blockSD.STORAGE_UNREADY_DOMAIN_TAG, + blockSD.VG_METADATASIZE) + fake_lvm.createLV(vg_name, sd.METADATA, blockSD.SD_METADATA_SIZE) + + # Fake the PV information for our metadata LV + fake_lvm.lvmd[vg_name][sd.METADATA]['devices'] = \ + '{0}(0)'.format(devices[0]) + return vg_name + + +def get_metafile_path(domaindir): + return os.path.join(domaindir, sd.DOMAIN_META_DATA, sd.METADATA) + + +def make_filesd_manifest(tmpdir, metadata=None): + sduuid = str(uuid.uuid4()) + domain_path = os.path.join(tmpdir, sduuid) + make_fake_metafile(get_metafile_path(domain_path)) + if metadata is None: + metadata = dict() + manifest = fileSD.FileStorageDomainManifest(domain_path, metadata) + os.makedirs(os.path.join(manifest.domaindir, sduuid, sd.DOMAIN_IMAGES)) + return manifest + + +def make_fake_metafile(metafile): + os.makedirs(os.path.dirname(metafile)) + with open(metafile, "w") as f: + f.truncate(0) + + +def make_file_volume(domaindir, size, imguuid=None, voluuid=None): + imguuid = imguuid or str(uuid.uuid4()) + voluuid = voluuid or str(uuid.uuid4()) + imgpath = os.path.join(domaindir, "images", imguuid) + volpath = os.path.join(imgpath, voluuid) + mdfiles = [volpath + '.meta', volpath + '.lease'] + if not os.path.exists(imgpath): + os.makedirs(imgpath) + with open(volpath, "w") as f: + f.truncate(size) + for mdfile in mdfiles: + with open(mdfile, "w") as f: + f.truncate(0) + return imguuid, voluuid \ No newline at end of file -- To view, visit https://gerrit.ovirt.org/43547 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I713e19473b45bc05da5a1c888a034306144228c5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Adam Litke <alitke(a)redhat.com>

4 23

Change in vdsm[master]: vmstats: make sure values are all strings
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has uploaded a new change for review. Change subject: vmstats: make sure values are all strings ...................................................................... vmstats: make sure values are all strings Until we drop support of XML-RPC, we need to make sure that possibly large integers are reported as strings. This patch converts directly the values to strings, avoiding the need for a later conversion step. Change-Id: I8326c284c412cb392619b33a3bd869b12add1206 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/vmstats.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/15/43715/1 diff --git a/vdsm/virt/vmstats.py b/vdsm/virt/vmstats.py index f6636cf..10d0776 100644 --- a/vdsm/virt/vmstats.py +++ b/vdsm/virt/vmstats.py @@ -273,10 +273,10 @@ def _disk_rate(first_sample, last_sample, interval): return { - 'readRate': ( + 'readRate': str( (last_sample['rd.bytes'] - first_sample['rd.bytes']) / interval), - 'writeRate': ( + 'writeRate': str( (last_sample['wr.bytes'] - first_sample['wr.bytes']) / interval)} -- To view, visit https://gerrit.ovirt.org/43715 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8326c284c412cb392619b33a3bd869b12add1206 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 11

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Douglas Schilling Landgraf 28 Jul '15

28 Jul '15

Douglas Schilling Landgraf has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 3: I could build vdsm rpms and vdsm_reg is not generated. -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Douglas Schilling Landgraf 28 Jul '15

28 Jul '15

Douglas Schilling Landgraf has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 3: Verified+1 Code-Review-1 -1 as pending the official test for new reg approach. -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by automation＠ovirt.org 28 Jul '15

28 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 3: * Update tracker::#1231379::OK * Check Bug-Url::OK * Check Public Bug::#1231379::OK, public bug * Check Product::#1231379::OK, Correct product Red Hat Enterprise Virtualization Manager * Check TR::SKIP, not in a monitored branch (ovirt-3.5 ovirt-3.4 ovirt-3.3 ovirt-3.2) * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vdsm-reg: remove the sub-project
by Douglas Schilling Landgraf 28 Jul '15

28 Jul '15

Douglas Schilling Landgraf has posted comments on this change. Change subject: vdsm-reg: remove the sub-project ...................................................................... Patch Set 2: Can we drop vdsm-store-net-config too ? -- To view, visit https://gerrit.ovirt.org/43254 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I5d19ddb0e65657515e2dc0eb9eeae320d8d6bf71 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck(a)redhat.com> Gerrit-Reviewer: Douglas Schilling Landgraf <dougsland(a)redhat.com> Gerrit-Reviewer: Fabian Deutsch <fabiand(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: fix wrong syntax in spec introduced by 7ba0684
by ybronhei＠redhat.com 28 Jul '15

28 Jul '15

Yaniv Bronhaim has uploaded a new change for review. Change subject: fix wrong syntax in spec introduced by 7ba0684 ...................................................................... fix wrong syntax in spec introduced by 7ba0684 Change-Id: Ibd01b281db238d195b97870546aad0e9df63e9e9 Signed-off-by: Yaniv Bronhaim <ybronhei(a)redhat.com> --- M vdsm.spec.in 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/80/44080/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 21321a0..5020d8b 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -601,7 +601,7 @@ %endif %configure %{?with_hooks:--enable-hooks} \ %if 0%{?rhev_build} - --enable-rhev + --enable-rhev \ --with-reg-bridge-name='rhevm' \ --with-engine-name='RHEV-M' \ --with-smbios-manufacturer='Red Hat' \ -- To view, visit https://gerrit.ovirt.org/44080 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibd01b281db238d195b97870546aad0e9df63e9e9 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yaniv Bronhaim <ybronhei(a)redhat.com>

3 4

Change in vdsm[master]: vdsm: fix missing \ in specfile
by Martin Polednik 28 Jul '15

28 Jul '15

Martin Polednik has uploaded a new change for review. Change subject: vdsm: fix missing \ in specfile ...................................................................... vdsm: fix missing \ in specfile The next option shouldn't be treated as a command itself. Change-Id: I781249247f4da6b7922af77fc556162304b64128 Signed-off-by: Martin Polednik <mpolednik(a)redhat.com> --- M vdsm.spec.in 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/74/44074/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 21321a0..5020d8b 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -601,7 +601,7 @@ %endif %configure %{?with_hooks:--enable-hooks} \ %if 0%{?rhev_build} - --enable-rhev + --enable-rhev \ --with-reg-bridge-name='rhevm' \ --with-engine-name='RHEV-M' \ --with-smbios-manufacturer='Red Hat' \ -- To view, visit https://gerrit.ovirt.org/44074 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I781249247f4da6b7922af77fc556162304b64128 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Martin Polednik <mpolednik(a)redhat.com>

3 6

Change in vdsm[master]: vm: event: document event on reboot
by Dan Kenigsberg 28 Jul '15

28 Jul '15

Dan Kenigsberg has posted comments on this change. Change subject: vm: event: document event on reboot ...................................................................... Patch Set 18: Code-Review+2 If you find it helpful, let it be. -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 18 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: document event on reboot
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: document event on reboot ...................................................................... Patch Set 18: Verified+1 v18 is the one actually verified -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 18 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on IO Error
by automation＠ovirt.org 28 Jul '15

28 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: send status event on IO Error ...................................................................... Patch Set 19: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43312 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa4be90f1267e82b8af2abb36c06cfd9a70fb558 Gerrit-PatchSet: 19 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: document event on reboot
by automation＠ovirt.org 28 Jul '15

28 Jul '15

automation(a)ovirt.org has posted comments on this change. Change subject: vm: event: document event on reboot ...................................................................... Patch Set 18: * Update tracker::IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.5', 'ovirt-3.4', 'ovirt-3.3']) -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 18 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on reboot
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on reboot ...................................................................... Patch Set 17: -Verified -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 17 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on reboot
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on reboot ...................................................................... Patch Set 17: Verified+1 now event is sent only once -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 17 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on reboot
by fromani＠redhat.com 28 Jul '15

28 Jul '15

Francesco Romani has posted comments on this change. Change subject: vm: event: send status event on reboot ...................................................................... Patch Set 17: Michal's thought is correct, verified on the field. Thinking about the best way to fix this. We may just want to drop this patch entirely. -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 17 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

Change in vdsm[master]: vm: event: send status event on reboot
by michal.skrivanek＠redhat.com 28 Jul '15

28 Jul '15

Michal Skrivanek has posted comments on this change. Change subject: vm: event: send status event on reboot ...................................................................... Patch Set 17: -Code-Review all 3 are from libvirtEventLoop context...so perhaps libvirt is emitting multiple times? I'd guess setting self.guestStatus = vmstatus.REBOOT_IN_PROGRESS only when it's not already set would resolve this (and also the @guestStatus.setter would take care of sendign the event) -- To view, visit https://gerrit.ovirt.org/43309 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I748b187228a39e55982c1b7d05ab47681481801c Gerrit-PatchSet: 17 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com> Gerrit-Reviewer: Jenkins CI Gerrit-Reviewer: Michal Skrivanek <michal.skrivanek(a)redhat.com> Gerrit-Reviewer: automation(a)ovirt.org Gerrit-HasComments: No

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches July 2015