vdsm-patches March 2013

vdsm-patches@lists.fedorahosted.org

51 participants
424 discussions

Change in vdsm[master]: (Schema) Change BlockDevicePathInfo so that it needs one les...
by smizrahi＠redhat.com 11 May '14

11 May '14

Saggi Mizrahi has uploaded a new change for review. Change subject: (Schema) Change BlockDevicePathInfo so that it needs one less redundant enum ...................................................................... (Schema) Change BlockDevicePathInfo so that it needs one less redundant enum Change-Id: I5ab262a1fbadf89aec0c649822a2a5879823521d Signed-off-by: Saggi Mizrahi <smizrahi(a)redhat.com> --- M vdsm_api/vdsmapi-schema.json 1 file changed, 6 insertions(+), 18 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/43/6843/1 -- To view, visit http://gerrit.ovirt.org/6843 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5ab262a1fbadf89aec0c649822a2a5879823521d Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com>

4 7

Change in vdsm[master]: (Schema) Replace host\port combos with NetworkAdress
by smizrahi＠redhat.com 11 May '14

11 May '14

Saggi Mizrahi has uploaded a new change for review. Change subject: (Schema) Replace host\port combos with NetworkAdress ...................................................................... (Schema) Replace host\port combos with NetworkAdress Change-Id: Iade8796a0168561ea0902ca512a817eb2f44aecb Signed-off-by: Saggi Mizrahi <smizrahi(a)redhat.com> --- M vdsm_api/vdsmapi-schema.json 1 file changed, 21 insertions(+), 15 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/64/6964/1 -- To view, visit http://gerrit.ovirt.org/6964 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iade8796a0168561ea0902ca512a817eb2f44aecb Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com>

4 4

Change in vdsm[master]: Related to 846307 - Simplifying clientIF._cleanOldFiles().
by ewarszaw＠redhat.com 11 May '14

11 May '14

Eduardo has uploaded a new change for review. Change subject: Related to 846307 - Simplifying clientIF._cleanOldFiles(). ...................................................................... Related to 846307 - Simplifying clientIF._cleanOldFiles(). Change-Id: Ifd63a1f4b1c64f84d5734da033a2420b97ce0ada Bug-Id: https://bugzilla.redhat.com/show_bug.cgi?id=846307 Signed-off-by: Eduardo <ewarszaw(a)redhat.com> --- M vdsm/clientIF.py 1 file changed, 15 insertions(+), 20 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/26/8626/1 diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index e9ca7c9..669c5be 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -1,3 +1,4 @@ +import os.path # # Copyright 2011 Red Hat, Inc. # @@ -453,24 +454,18 @@ return None def _cleanOldFiles(self): - for f in os.listdir(constants.P_VDSM_RUN): + for fName in os.listdir(constants.P_VDSM_RUN): + if fName in ('vdsmd.pid', 'respawn.pid', 'svdsm.pid', + 'svdsm.sock'): + continue try: - vmId, fileType = f.split(".", 1) - if fileType in ["guest.socket", "monitor.socket", "pid", - "stdio.dump", "recovery"]: - if vmId in self.vmContainer: - continue - if f == 'vdsmd.pid': - continue - if f == 'respawn.pid': - continue - if f == 'svdsm.pid': - continue - if f == 'svdsm.sock': - continue - else: - continue - self.log.debug("removing old file " + f) - utils.rmFile(constants.P_VDSM_RUN + f) - except: - pass + vmId, fileType = fName.split('.', 1) + except ValueError: + pass # fName does not contains '.' + if not fileType in ("guest.socket", "monitor.socket", "pid", + "stdio.dump", "recovery"): + continue + if vmId in self.vmContainer: + continue + self.log.debug("Removing old file %s", fName) + utils.rmFile(constants.P_VDSM_RUN + fName) -- To view, visit http://gerrit.ovirt.org/8626 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifd63a1f4b1c64f84d5734da033a2420b97ce0ada Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo <ewarszaw(a)redhat.com>

8 20

Change in vdsm[master]: Extend Image.move to move and copy subchains
by Federico Simoncelli 11 May '14

11 May '14

Federico Simoncelli has uploaded a new change for review. Change subject: Extend Image.move to move and copy subchains ...................................................................... Extend Image.move to move and copy subchains The method Image.move was only capable of moving and copying the entire image chain. Now it's possible to specify an (optional) volUUID to move or copy a subchain from the base up to volUUID. Change-Id: Icb11818f830c6f02bcb74d69f1c119efaf100cc2 --- M vdsm/storage/image.py M vdsm/storage/sp.py 2 files changed, 33 insertions(+), 22 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/17/3217/1 -- To view, visit http://gerrit.ovirt.org/3217 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icb11818f830c6f02bcb74d69f1c119efaf100cc2 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>

5 8

Change in vdsm[master]: image: unify the getChainTemplate method
by Federico Simoncelli 11 May '14

11 May '14

Federico Simoncelli has uploaded a new change for review. Change subject: image: unify the getChainTemplate method ...................................................................... image: unify the getChainTemplate method Change-Id: I32fedf40e8c05d4ef7d26cb9172c22d7ee6e9086 Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com> --- M vdsm/storage/image.py 1 file changed, 11 insertions(+), 16 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/12868/1 diff --git a/vdsm/storage/image.py b/vdsm/storage/image.py index 92021ff..4c6047e 100644 --- a/vdsm/storage/image.py +++ b/vdsm/storage/image.py @@ -260,7 +260,7 @@ """ chain = self.getChain(sdUUID, imgUUID, volUUID) newsize = 0 - template = chain[0].getParentVolume() + template = self.getChainTemplate(chain) if template: newsize = template.getVolumeSize() for vol in chain: @@ -327,27 +327,22 @@ self.log.info("sdUUID=%s imgUUID=%s chain=%s ", sdUUID, imgUUID, chain) return chain - def getTemplate(self, sdUUID, imgUUID): + def getChainTemplate(self, chain): """ - Return template of the image + Return the template of a volume chain. None if not present. """ - tmpl = None - # Find all volumes of image (excluding template) - chain = self.getChain(sdUUID, imgUUID) - # check if the chain is build above a template, or it is a standalone - pvol = chain[0].getParentVolume() - if pvol: - tmpl = pvol - elif chain[0].isShared(): - tmpl = chain[0] + try: + templateVolume = chain[0].getParentVolume() + except IndexError: + raise se.ImageValidationError("The chain has no volumes") - return tmpl + return templateVolume def prepare(self, sdUUID, imgUUID, volUUID=None): chain = self.getChain(sdUUID, imgUUID, volUUID) # Adding the image template to the chain - tmplVolume = chain[0].getParentVolume() + tmplVolume = self.getChainTemplate(chain) if tmplVolume: chain.insert(0, tmplVolume) @@ -489,7 +484,7 @@ fakeTemplate = False pimg = volume.BLANK_UUID # standalone chain # check if the chain is build above a template, or it is a standalone - pvol = srcChain[0].getParentVolume() + pvol = self.getChainTemplate(srcChain) if pvol: # find out parent volume parameters volParams = pvol.getVolumeParams() @@ -790,7 +785,7 @@ raise se.ImageIsNotLegalChain(imgUUID) chain = self.getChain(sdUUID, imgUUID) # check if the chain is build above a template, or it is a standalone - pvol = chain[0].getParentVolume() + pvol = self.getChainTemplate(chain) if pvol: if not pvol.isLegal() or pvol.isFake(): raise se.ImageIsNotLegalChain(imgUUID) -- To view, visit http://gerrit.ovirt.org/12868 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I32fedf40e8c05d4ef7d26cb9172c22d7ee6e9086 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>

4 6

Change in vdsm[master]: Added a support for a guest hibernate command.
by ghammer＠redhat.com 28 Apr '14

28 Apr '14

Gal Hammer has uploaded a new change for review. Change subject: Added a support for a guest hibernate command. ...................................................................... Added a support for a guest hibernate command. Change-Id: I1ad6836e61e9d91ec6f46a599541f61ff12e9737 --- M vdsm/clientIF.py M vdsm/guestIF.py M vdsm_cli/vdsClient.py 3 files changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/1121/1 -- To view, visit http://gerrit.ovirt.org/1121 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1ad6836e61e9d91ec6f46a599541f61ff12e9737 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Gal Hammer <ghammer(a)redhat.com>

14 64

Change in vdsm[master]: [WIP] Added gluster geo-replication support
by barumuga＠redhat.com 28 Apr '14

28 Apr '14

Hello Ayal Baron, Timothy Asir, Saggi Mizrahi, Federico Simoncelli, Dan Kenigsberg, I'd like you to do a code review. Please visit http://gerrit.ovirt.org/8375 to review the following change. Change subject: [WIP] Added gluster geo-replication support ...................................................................... [WIP] Added gluster geo-replication support Below new verbs are added glusterValidateSshConnection glusterSetupSshConnection Change-Id: Ic783abd5f1b63bc5116ce4ff2a3c7be92001a387 Signed-off-by: Bala.FA <barumuga(a)redhat.com> --- M vdsm.spec.in M vdsm/gluster/api.py M vdsm/gluster/exception.py 3 files changed, 137 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/75/8375/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 22e48fb..020344f 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -352,6 +352,7 @@ Requires: %{name} = %{version}-%{release} Requires: glusterfs glusterfs-server glusterfs-fuse +Requires: python-paramiko %description gluster Gluster plugin enables VDSM to serve Gluster functionalities. diff --git a/vdsm/gluster/api.py b/vdsm/gluster/api.py index a825de2..338cb06 100644 --- a/vdsm/gluster/api.py +++ b/vdsm/gluster/api.py @@ -19,11 +19,24 @@ # from functools import wraps +import socket +import paramiko +import re from vdsm.define import doneCode import supervdsm as svdsm +from vdsm.config import config +from vdsm import utils +import exception as ge _SUCCESS = {'status': doneCode} +_KEYFILE = config.get('vars', 'trust_store_path') + '/keys/vdsmkey.pem' +_sshKeyGenCommandPath = utils.CommandPath("ssh-keygen", + "/usr/bin/ssh-keygen", + ) +_SSH_COPY_ID_CMD = "umask 077; test -d ~/.ssh || mkdir ~/.ssh ; " \ + "cat >> ~/.ssh/authorized_keys && (test -x /sbin/restorecon && " \ + "/sbin/restorecon ~/.ssh ~/.ssh/authorized_keys >/dev/null 2>&1 || true)" def exportAsVerb(func): @@ -43,6 +56,52 @@ class VolumeStatus(): ONLINE = 'ONLINE' OFFLINE = 'OFFLINE' + + +class HostKeyMatchException(paramiko.SSHException): + def __init__(self, hostname, fingerprint, expected_fingerprint): + err = 'Fingerprint of Host key ' \ + '%s for server %s does not match with %s' % \ + (fingerprint, hostname, expected_fingerprint) + paramiko.SSHException.__init__(self, err) + self.hostname = hostname + self.fingerprint = fingerprint + self.expected_fingerprint = expected_fingerprint + + +class HostKeyMatchPolicy(paramiko.AutoAddPolicy): + def __init__(self, expected_fingerprint): + self.expected_fingerprint = expected_fingerprint + + def missing_host_key(self, client, hostname, key): + s = paramiko.util.hexlify(key.get_fingerprint()) + fingerprint = ':'.join(re.findall('..', s)) + if fingerprint == self.expected_fingerprint.upper(): + paramiko.AutoAddPolicy.missing_host_key(self, client, hostname, + key) + else: + raise HostKeyMatchException(hostname, fingerprint, + self.expected_fingerprint) + + +class GlusterSsh(paramiko.SSHClient): + def __init__(self, hostname, fingerprint, port=22, username=None, + password=None, pkey=None, key_filename=None, timeout=None, + allow_agent=True, look_for_keys=True, compress=False): + paramiko.SSHClient.__init__(self) + key_file_list = [_KEYFILE] + if key_filename: + key_file_list.append(list(key_filename)) + self.set_missing_host_key_policy(HostKeyMatchPolicy(fingerprint)) + try: + paramiko.SSHClient.connect(self, hostname, port, username, + password, pkey, key_file_list, timeout, + allow_agent, look_for_keys, compress) + except socket.error, e: + err = ['%s: %s' % (hostname, e)] + raise ge.GlusterSshConnectionFailedException(err=err) + except HostKeyMatchException, e: + raise ge.GlusterSshHostKeyMismatchException(err=[e.err]) class GlusterApi(object): @@ -236,6 +295,47 @@ def volumeProfileStop(self, volumeName, options=None): self.svdsmProxy.glusterVolumeProfileStop(volumeName) + def _validateSshConnection(self, hostname, fingerprint, username): + try: + ssh = GlusterSsh(hostname, + fingerprint, + username=username) + ssh.close() + return True + except paramiko.AuthenticationException, e: + raise ge.GlusterSshHostKeyAuthException(err=[str(e)]) + + @exportAsVerb + def validateSshConnection(self, hostname, fingerprint, username): + return self._validateSshConnection(hostname, fingerprint, username) + + @exportAsVerb + def setupSshConnection(self, hostname, fingerprint, username, password): + rc, out, err = utils.execCmd([_sshKeyGenCommandPath.cmd, '-y', '-f', + _KEYFILE]) + if rc != 0: + raise ge.GlusterSshPubKeyGenerationFailedException(rc=rc, err=err) + + try: + ssh = GlusterSsh(hostname, + fingerprint, + username=username, + password=password) + c = ssh.get_transport().open_session() + stdin, stdout, stderr = c.exec_command(_SSH_COPY_ID_CMD) + stdin.write('\n'.join(out) + '\n') + stdin.flush() + stdin.close() + rc = c.recv_exit_status() + ssh.close() + if rc != 0: + err = stderr.read().splitlines() + raise ge.GlusterSshSetupExecFailedException(rc=rc, err=err) + except paramiko.AuthenticationException, e: + raise ge.GlusterSshHostAuthException(err=[str(e)]) + + return self._validateSshConnection(hostname, fingerprint, username) + def getGlusterMethods(gluster): l = [] diff --git a/vdsm/gluster/exception.py b/vdsm/gluster/exception.py index 6d94ae3..0143a5e 100644 --- a/vdsm/gluster/exception.py +++ b/vdsm/gluster/exception.py @@ -377,3 +377,39 @@ class GlusterHostsListFailedException(GlusterHostException): code = 4407 message = "Hosts list failed" + + +# Ssh +class GlusterSshException(GlusterException): + code = 4500 + message = "Gluster ssh exception" + + +class GlusterSshConnectionFailedException(GlusterSshException): + code = 4501 + message = "SSH connection failed" + + +class GlusterSshHostKeyMismatchException(GlusterSshException): + code = 4502 + message = "Host key match failed" + + +class GlusterSshHostKeyAuthException(GlusterSshException): + code = 4503 + message = "SSH host key authentication failed" + + +class GlusterSshHostAuthException(GlusterSshException): + code = 4504 + message = "SSH host authentication failed" + + +class GlusterSshPubKeyGenerationFailedException(GlusterSshException): + code = 4505 + message = "SSH public key generation failed" + + +class GlusterSshSetupExecFailedException(GlusterSshException): + code = 4506 + message = "SSH key setup execution failed" -- To view, visit http://gerrit.ovirt.org/8375 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic783abd5f1b63bc5116ce4ff2a3c7be92001a387 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Bala.FA <barumuga(a)redhat.com> Gerrit-Reviewer: Ayal Baron <abaron(a)redhat.com> Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com> Gerrit-Reviewer: Federico Simoncelli <fsimonce(a)redhat.com> Gerrit-Reviewer: Saggi Mizrahi <smizrahi(a)redhat.com> Gerrit-Reviewer: Timothy Asir <tjeyasin(a)redhat.com>

3 4

Change in vdsm[master]: Add qemu's memory usage to VM statistics.
by ghammer＠redhat.com 28 Apr '14

28 Apr '14

Gal Hammer has uploaded a new change for review. Change subject: Add qemu's memory usage to VM statistics. ...................................................................... Add qemu's memory usage to VM statistics. Change-Id: Ibeb35759454c4a9b41e1303956267e93ca3545a0 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=799285 Signed-off-by: Gal Hammer <ghammer(a)redhat.com> --- M vdsm/config.py.in M vdsm/libvirtvm.py 2 files changed, 14 insertions(+), 1 deletion(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/06/9006/1 diff --git a/vdsm/config.py.in b/vdsm/config.py.in index df85e7e..ee1627b 100644 --- a/vdsm/config.py.in +++ b/vdsm/config.py.in @@ -111,6 +111,8 @@ ('vm_sample_net_interval', '5', None), ('vm_sample_net_window', '2', None), + + ('vm_sample_memory_interval', '2', None), ('trust_store_path', '@TRUSTSTORE@', 'Where the certificates and keys are situated.'), diff --git a/vdsm/libvirtvm.py b/vdsm/libvirtvm.py index 86e39a3..f76f35c 100644 --- a/vdsm/libvirtvm.py +++ b/vdsm/libvirtvm.py @@ -91,10 +91,13 @@ self._sampleNet, config.getint('vars', 'vm_sample_net_interval'), config.getint('vars', 'vm_sample_net_window'))) + self.sampleMem = (utils.AdvancedStatsFunction(self._sampleMem, + config.getint('vars', 'vm_sample_memory_interval'))) self.addStatsFunction( self.highWrite, self.updateVolumes, self.sampleCpu, - self.sampleDisk, self.sampleDiskLatency, self.sampleNet) + self.sampleDisk, self.sampleDiskLatency, self.sampleNet, + self.sampleMem) def _highWrite(self): if not self._vm.isDisksStatsCollectionEnabled(): @@ -168,6 +171,14 @@ netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name) return netSamples + def _sampleMem(self): + memUsage = {} + for line in open('/proc/%d/status' %(self.conf['pid'])): + var, value = line.strip().split()[0:2] + if var in ('VmSize:', 'VmRSS:', 'VmData:'): + memUsage[var[:-1]] = long(value) + return memUsage + def _diff(self, prev, curr, val): return prev[val] - curr[val] -- To view, visit http://gerrit.ovirt.org/9006 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ibeb35759454c4a9b41e1303956267e93ca3545a0 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Gal Hammer <ghammer(a)redhat.com>

8 29

Change in vdsm[master]: [WIP] Start moving proc parsing to it's own module
by smizrahi＠redhat.com 28 Apr '14

28 Apr '14

Saggi Mizrahi has uploaded a new change for review. Change subject: [WIP] Start moving proc parsing to it's own module ...................................................................... [WIP] Start moving proc parsing to it's own module Change-Id: I7ba84c7ece95bdef7448a7c7af277e7f58695401 Signed-off-by: Saggi Mizrahi <smizrahi(a)redhat.com> --- M vdsm.spec.in M vdsm/API.py M vdsm/Makefile.am M vdsm/caps.py A vdsm/procfs.py M vdsm/utils.py M vdsm/vm.py 7 files changed, 53 insertions(+), 40 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/13/7513/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index bd01c2a..1f01961 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -573,6 +573,7 @@ %{_datadir}/%{vdsm_name}/supervdsmServer.py* %{_datadir}/%{vdsm_name}/vmChannels.py* %{_datadir}/%{vdsm_name}/vmContainer.py* +%{_datadir}/%{vdsm_name}/procfs.py* %{_datadir}/%{vdsm_name}/tc.py* %{_datadir}/%{vdsm_name}/vdsm %{_datadir}/%{vdsm_name}/vdsm-restore-net-config diff --git a/vdsm/API.py b/vdsm/API.py index 720c3b9..aab69cd 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -37,6 +37,7 @@ from vdsm.define import doneCode, errCode, Kbytes, Mbytes import caps from vdsm.config import config +import procfs import supervdsm @@ -864,7 +865,7 @@ """ def _readSwapTotalFree(): - meminfo = utils.readMemInfo() + meminfo = procfs.meminfo() return meminfo['SwapTotal'] / 1024, meminfo['SwapFree'] / 1024 stats = {} @@ -1111,17 +1112,16 @@ memCommitted = self._memCommitted() resident = 0 for v in self._cif.vmContainer.getVMs(): - if v.conf['pid'] == '0': - continue try: - statmfile = file('/proc/' + v.conf['pid'] + '/statm') - resident += int(statmfile.read().split()[1]) + resident += v.statm().resident except: pass + resident *= PAGE_SIZE_BYTES - meminfo = utils.readMemInfo() - freeOrCached = (meminfo['MemFree'] + - meminfo['Cached'] + meminfo['Buffers']) * Kbytes + + meminfo = procfs.meminfo() + freeOrCached = (meminfo['MemFree'] + meminfo['Cached'] + + meminfo['Buffers']) * Kbytes return freeOrCached + resident - memCommitted - \ config.getint('vars', 'host_mem_reserve') * Mbytes diff --git a/vdsm/Makefile.am b/vdsm/Makefile.am index 574d762..1a3ac43 100644 --- a/vdsm/Makefile.am +++ b/vdsm/Makefile.am @@ -47,6 +47,7 @@ momIF.py \ neterrors.py \ parted_utils.py \ + procfs.py \ pthread.py \ supervdsm.py \ supervdsmServer.py \ diff --git a/vdsm/caps.py b/vdsm/caps.py index f1641ff..39fc837 100644 --- a/vdsm/caps.py +++ b/vdsm/caps.py @@ -41,6 +41,7 @@ from vdsm import utils from vdsm import constants import storage.hba +import procfs # For debian systems we can use python-apt if available try: @@ -271,7 +272,7 @@ caps['HBAInventory'] = storage.hba.HBAInventory() caps['vmTypes'] = ['kvm'] - caps['memSize'] = str(utils.readMemInfo()['MemTotal'] / 1024) + caps['memSize'] = str(procfs.meminfo()['MemTotal'] / 1024) caps['reservedMem'] = str( config.getint('vars', 'host_mem_reserve') + config.getint('vars', 'extra_mem_reserve')) diff --git a/vdsm/procfs.py b/vdsm/procfs.py new file mode 100644 index 0000000..29fc973 --- /dev/null +++ b/vdsm/procfs.py @@ -0,0 +1,31 @@ +from collections import namedtuple + +buffsize = 4096 + +MemStat = namedtuple("MemStat", + "size, resident, share, text, UNUSED1, data, UNUSED2") + + +def statm(pid): + """ + Parses statm for a pid. Note all results are in pages. + """ + with open("/proc/%d/statm" % pid, "rb") as f: + return MemStat(*(int(val) for val in f.read().split())) + + +def meminfo(): + """ + Parse ``/proc/meminfo`` and return its content as a dictionary. + + note. + All values are in KB + """ + meminfo = {} + with open("/proc/meminfo", "rb") as f: + f.seek(0) + lines = f.readlines() + for var, val in (l.split()[0:2] for l in lines): + meminfo[var[:-1]] = int(val) + + return meminfo diff --git a/vdsm/utils.py b/vdsm/utils.py index 5e2d4e5..048a528 100644 --- a/vdsm/utils.py +++ b/vdsm/utils.py @@ -19,7 +19,8 @@ # """ -A module containing miscellaneous functions and classes that are user plentifuly around vdsm. +A module containing miscellaneous functions and classes that are user +plentifuly around vdsm. .. attribute:: utils.symbolerror @@ -28,7 +29,8 @@ from SimpleXMLRPCServer import SimpleXMLRPCServer import SocketServer import threading -import os, time +import os +import time import logging import errno import subprocess @@ -42,6 +44,7 @@ import constants from config import config import netinfo +import procfs _THP_STATE_PATH = '/sys/kernel/mm/transparent_hugepage/enabled' if not os.path.exists(_THP_STATE_PATH): @@ -63,34 +66,6 @@ os.unlink(fileToRemove) except: pass - -def readMemInfo(): - """ - Parse ``/proc/meminfo`` and return its content as a dictionary. - - For a reason unknown to me, ``/proc/meminfo`` is is sometime - empty when opened. If that happens, the function retries to open it - 3 times. - - :returns: a dictionary representation of ``/proc/meminfo`` - """ - # FIXME the root cause for these retries should be found and fixed - tries = 3 - meminfo = {} - while True: - tries -= 1 - try: - lines = [] - lines = file('/proc/meminfo').readlines() - for line in lines: - var, val = line.split()[0:2] - meminfo[var[:-1]] = int(val) - return meminfo - except: - logging.warning(lines, exc_info=True) - if tries <= 0: - raise - time.sleep(0.1) #Threaded version of SimpleXMLRPCServer class SimpleThreadedXMLRPCServer(SocketServer.ThreadingMixIn, SimpleXMLRPCServer): @@ -225,7 +200,7 @@ """ BaseSample.__init__(self, pid, ifids) self.totcpu = TotalCpuSample() - meminfo = readMemInfo() + meminfo = procfs.meminfo() freeOrCached = (meminfo['MemFree'] + meminfo['Cached'] + meminfo['Buffers']) self.memUsed = 100 - int(100.0 * (freeOrCached) / meminfo['MemTotal']) diff --git a/vdsm/vm.py b/vdsm/vm.py index c1a22b0..bd436e0 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -36,6 +36,7 @@ import libvirt from vdsm import vdscli import caps +import procfs DEFAULT_BRIDGE = config.get("vars", "default_bridge") @@ -693,6 +694,9 @@ load = len(self.cif.vmContainer.getVMs()) return base * (doubler + load) / doubler + def statm(self): + return procfs.statm(int(self.conf['pid'])) + def saveState(self): if self.destroyed: return -- To view, visit http://gerrit.ovirt.org/7513 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I7ba84c7ece95bdef7448a7c7af277e7f58695401 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com>

4 4

Change in vdsm[master]: Refactor prepareVolumePath
by smizrahi＠redhat.com 28 Apr '14

28 Apr '14

Saggi Mizrahi has uploaded a new change for review. Change subject: Refactor prepareVolumePath ...................................................................... Refactor prepareVolumePath Change-Id: I57bb8684fd11a47843a158d13fcc2815147fa7ef Signed-off-by: Saggi Mizrahi <smizrahi(a)redhat.com> --- M vdsm/API.py M vdsm/clientIF.py M vdsm/libvirtvm.py M vdsm/storage/devicemapper.py M vdsm/vm.py 5 files changed, 93 insertions(+), 63 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/55/7755/1 diff --git a/vdsm/API.py b/vdsm/API.py index 720c3b9..c324d79 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -173,8 +173,7 @@ # NOTE: pickled params override command-line params. this # might cause problems if an upgrade took place since the # parmas were stored. - fname = self._cif.prepareVolumePath(paramFilespec) - try: + with self._cif.preparedDrive(paramFilespec) as fname: with file(fname) as f: pickledMachineParams = pickle.load(f) @@ -183,8 +182,6 @@ + str(pickledMachineParams)) self.log.debug('former conf ' + str(vmParams)) vmParams.update(pickledMachineParams) - finally: - self._cif.teardownVolumePath(paramFilespec) except: self.log.error("Error restoring VM parameters", exc_info=True) @@ -299,9 +296,15 @@ :param hiberVolHandle: opaque string, indicating the location of hibernation images. """ - params = {'vmId': self._UUID, 'mode': 'file', - 'hiberVolHandle': hibernationVolHandle} - response = self.migrate(params) + v = self._getVmObject() + if v is None: + return errCode['noVM'] + + try: + response = self.hibernate(hibernationVolHandle) + except vm.WrongStateError: + response = errCode['noVM'] + if not response['status']['code']: response['status']['message'] = 'Hibernation process starting' return response diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index 55a7fc9..0446eb2 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -25,6 +25,7 @@ from xml.dom import minidom import uuid import errno +from contextlib import contextmanager from storage.dispatcher import Dispatcher from storage.hsm import HSM @@ -44,6 +45,7 @@ import blkid import supervdsm import vmContainer +from storage import devicemapper try: import gluster.api as gapi _glusterEnabled = True @@ -239,50 +241,74 @@ self.log.info('Error finding path for device', exc_info=True) raise vm.VolumeError(uuid) + def _preparePoolImage(self, drive): + res = self.irs.prepareImage( + drive['domainID'], drive['poolID'], + drive['imageID'], drive['volumeID']) + + if res['status']['code']: + raise vm.VolumeError(drive) + + drive['volumeChain'] = res['chain'] + return res['path'] + + def _prepareDmDevice(self, drive, vmId): + volPath = devicemapper.getDevicePathByGuid(drive["GUID"]) + + if not os.path.exists(volPath): + raise vm.VolumeError(drive) + + res = self.irs.appropriateDevice(drive["GUID"], vmId) + if res['status']['code']: + raise vm.VolumeError(drive) + + return volPath + + def _prepareScsiDevice(self, drive): + return self._getUUIDSpecPath(drive["UUID"]) + + def _prepareVmPayload(self, drive, vmId): + ''' + vmPayload is a key in specParams + 'vmPayload': {'file': {'filename': 'content'}} + ''' + for key, files in drive['specParams']['vmPayload'].iteritems(): + if key == 'file': + svdsm = supervdsm.getProxy() + if drive['device'] == 'cdrom': + return svdsm.mkIsoFs(vmId, files) + elif drive['device'] == 'floppy': + return svdsm.mkFloppyFs(vmId, files) + + raise vm.VolumeError(drive) + + def _preparePath(self, drive): + return drive['path'] + + @contextmanager + def perparedDrive(self, drive, vmId=None): + path = self.prepareVolumePath(drive, vmId) + try: + yield path + finally: + self.teardownVolumePath(drive, vmId) + def prepareVolumePath(self, drive, vmId=None): if type(drive) is dict: - # PDIV drive format if drive['device'] == 'disk' and vm.isVdsmImage(drive): - res = self.irs.prepareImage( - drive['domainID'], drive['poolID'], - drive['imageID'], drive['volumeID']) + volPath = self._preparePoolImage(drive) - if res['status']['code']: - raise vm.VolumeError(drive) - - volPath = res['path'] - drive['volumeChain'] = res['chain'] - - # GUID drive format elif "GUID" in drive: - volPath = os.path.join("/dev/mapper", drive["GUID"]) + volPath = self._prepareDmDevice(drive, vmId) - if not os.path.exists(volPath): - raise vm.VolumeError(drive) - - res = self.irs.appropriateDevice(drive["GUID"], vmId) - if res['status']['code']: - raise vm.VolumeError(drive) - - # UUID drive format elif "UUID" in drive: - volPath = self._getUUIDSpecPath(drive["UUID"]) + volPath = self._prepareScsiDevice(drive) elif 'specParams' in drive and 'vmPayload' in drive['specParams']: - ''' - vmPayload is a key in specParams - 'vmPayload': {'file': {'filename': 'content'}} - ''' - for key, files in drive['specParams']['vmPayload'].iteritems(): - if key == 'file': - if drive['device'] == 'cdrom': - volPath = supervdsm.getProxy().mkIsoFs(vmId, files) - elif drive['device'] == 'floppy': - volPath = \ - supervdsm.getProxy().mkFloppyFs(vmId, files) + volPath = self._prepareVmPayload(drive, vmId) elif "path" in drive: - volPath = drive['path'] + volPath = self._preparePath(drive) else: raise vm.VolumeError(drive) @@ -301,17 +327,22 @@ self.log.info("prepared volume path: %s", volPath) return volPath - def teardownVolumePath(self, drive): - res = {'status': doneCode} - if type(drive) == dict: - try: - res = self.irs.teardownImage(drive['domainID'], - drive['poolID'], drive['imageID']) - except KeyError: - #This drive is not a vdsm image (quartet) - self.log.info("Avoiding tear down drive %s", str(drive)) + def _teardownPoolImage(self, drive): + try: + res = self.irs.teardownImage(drive['domainID'], + drive['poolID'], drive['imageID']) + return res['status']['code'] + except KeyError: + #This drive is not a vdsm image (quartet) + self.log.info("Avoiding tear down drive %s", str(drive)) + return doneCode - return res['status']['code'] + def teardownVolumePath(self, drive): + if type(drive) == dict: + return self._teardownPoolImage(drive) + else: + # Other types don't require tear down + return 0 def createVm(self, vmParams): try: @@ -320,6 +351,7 @@ except vmContainer.VmContainerError as e: if e.errno == errno.EEXIST: return errCode['exist'] + return def waitForShutdown(self, timeout=None): diff --git a/vdsm/libvirtvm.py b/vdsm/libvirtvm.py index a530228..ea0d017 100644 --- a/vdsm/libvirtvm.py +++ b/vdsm/libvirtvm.py @@ -404,11 +404,8 @@ hooks.before_vm_hibernate(self._vm._dom.XMLDesc(0), self._vm.conf) try: self._vm._vmStats.pause() - fname = self._vm.cif.prepareVolumePath(self._dst) - try: + with self._vm.cif.preparedDrive(self._dst) as fname: self._vm._dom.save(fname) - finally: - self._vm.cif.teardownVolumePath(self._dst) except: self._vm._vmStats.cont() raise @@ -1397,11 +1394,8 @@ elif 'restoreState' in self.conf: hooks.before_vm_dehibernate(self.conf.pop('_srcDomXML'), self.conf) - fname = self.cif.prepareVolumePath(self.conf['restoreState']) - try: + with self.cif.preparedDrive(self.conf['restoreState']) as fname: self._connection.restore(fname) - finally: - self.cif.teardownVolumePath(self.conf['restoreState']) self._dom = NotifyingVirDomain( self._connection.lookupByUUIDString(self.id), diff --git a/vdsm/storage/devicemapper.py b/vdsm/storage/devicemapper.py index a1651e0..388c1cd 100644 --- a/vdsm/storage/devicemapper.py +++ b/vdsm/storage/devicemapper.py @@ -46,6 +46,10 @@ (major, minor))) +def getDevicePathByGuid(devGuid): + return DMPATH_FORMAT % devGuid + + def getSysfsPath(devName): if "/" in devName: raise ValueError("devName has an illegal format. " diff --git a/vdsm/vm.py b/vdsm/vm.py index 3aa9f52..49193d3 100644 --- a/vdsm/vm.py +++ b/vdsm/vm.py @@ -210,12 +210,9 @@ if ignoreParam in self._machineParams: del self._machineParams[ignoreParam] - fname = self._vm.cif.prepareVolumePath(self._dstparams) - try: - with file(fname, "w") as f: + with self._vm.cif.preparedDrive(self._dstparams) as fname: + with file(fname, "wb") as f: pickle.dump(self._machineParams, f) - finally: - self._vm.cif.teardownVolumePath(self._dstparams) self._vm.setDownStatus(NORMAL, "SaveState succeeded") self.status = { -- To view, visit http://gerrit.ovirt.org/7755 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I57bb8684fd11a47843a158d13fcc2815147fa7ef Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Saggi Mizrahi <smizrahi(a)redhat.com>

2 2

← Newer
1
2
3
4
5
6
7
8
9
...
43
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches March 2013