Adam Litke has uploaded a new change for review.
Change subject: Live merge: Update base size after live merge
......................................................................
Live merge: Update base size after live merge
When performing a live merge, data is copied from a top volume into a
base volume. If the top volume is larger than the base volume (which
can happen if the drive size was extended), libvirt will change the size
of the base volume to match that of the top volume. When synchronizing
metadata after the merge, we need to update the 'capacity' field of the
base volume to reflect the new size. We do this inside the
LiveMergeCleanupThread to ensure that it gets retried in the event of
storage connection problems or vdsm restarts.
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1232481
Change-Id: Iae354de36db63ae3bf4b4fc7f72df5e306035784
Signed-off-by: Adam Litke <alitke(a)redhat.com>
---
M vdsm/virt/vm.py
1 file changed, 37 insertions(+), 11 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/42921/1
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 549cd38..05ff99e 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -4510,7 +4510,7 @@
def queryBlockJobs(self):
def startCleanup(job, drive, needPivot):
- t = LiveMergeCleanupThread(self, job['jobID'], drive, needPivot)
+ t = LiveMergeCleanupThread(self, job, drive, needPivot)
t.start()
self._liveMergeCleanupThreads[job['jobID']] = t
@@ -4867,6 +4867,15 @@
(domainID, volumeID))
return VolumeSize(int(res['apparentsize']), int(res['truesize']))
+ def _getVolumeCapacity(self, domainID, poolID, imageID, volumeID):
+ """Return the volume capacity by accessing storage"""
+ res = self.cif.irs.getVolumeInfo(domainID, poolID, imageID, volumeID)
+ if res['status']['code'] != 0:
+ raise StorageUnavailableError(
+ "Unable to get volume capacity for domain %s volume %s" %
+ (domainID, volumeID))
+ return int(res['info']['capacity'])
+
def _setVolumeSize(self, domainID, poolID, imageID, volumeID, size):
res = self.cif.irs.setVolumeSize(domainID, poolID, imageID, volumeID,
size)
@@ -4877,11 +4886,11 @@
class LiveMergeCleanupThread(threading.Thread):
- def __init__(self, vm, jobId, drive, doPivot):
+ def __init__(self, vm, job, drive, doPivot):
threading.Thread.__init__(self)
self.setDaemon(True)
self.vm = vm
- self.jobId = jobId
+ self.job = job
self.drive = drive
self.doPivot = doPivot
self.success = False
@@ -4905,7 +4914,7 @@
self.vm.stopDisksStatsCollection()
self.vm.log.info("Requesting pivot to complete active layer commit "
- "(job %s)", self.jobId)
+ "(job %s)", self.job['jobID'])
try:
flags = libvirt.VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT
ret = self.vm._dom.blockJobAbort(self.drive.name, flags)
@@ -4915,22 +4924,38 @@
else:
if ret != 0:
self.vm.log.error("Pivot failed for job %s (rc=%i)",
- self.jobId, ret)
+ self.job['jobID'], ret)
raise RuntimeError("pivot failed")
self._waitForXMLUpdate()
- self.vm.log.info("Pivot completed (job %s)", self.jobId)
+ self.vm.log.info("Pivot completed (job %s)", self.job['jobID'])
+
+ def update_base_size(self):
+ # If the drive size was extended just after creating the snapshot which
+ # we are removing, the size of the top volume might be larger than the
+ # size of the base volume. In that case libvirt has enlarged the base
+ # volume automatically as part of the blockCommit operation. Update
+ # our metadata to reflect this change.
+ capacity = self.vm._getVolumeCapacity(self.drive.domainID,
+ self.drive.poolID,
+ self.drive.imageID,
+ self.job['topVolume'])
+ self.vm._setVolumeSize(self.drive.domainID, self.drive.poolID,
+ self.drive.imageID, self.job['baseVolume'],
+ capacity)
@utils.traceback()
def run(self):
+ self.update_base_size()
if self.doPivot:
self.tryPivot()
self.vm.log.info("Synchronizing volume chain after live merge "
- "(job %s)", self.jobId)
+ "(job %s)", self.job['jobID'])
self.vm._syncVolumeChain(self.drive)
if self.doPivot:
self.vm.startDisksStatsCollection()
self.success = True
- self.vm.log.info("Synchronization completed (job %s)", self.jobId)
+ self.vm.log.info("Synchronization completed (job %s)",
+ self.job['jobID'])
def isSuccessful(self):
"""
@@ -4976,9 +5001,10 @@
self.vm.log.info("The XML update has been completed")
break
else:
- self.log.error("Bad volume chain found for drive %s. Previous "
- "chain: %s, Expected chain: %s, Actual chain: "
- "%s", alias, origVols, expectedVols, curVols)
+ self.vm.log.error("Bad volume chain found for drive %s. "
+ "Previous chain: %s, Expected chain: %s, "
+ "Actual chain: %s", alias, origVols,
+ expectedVols, curVols)
raise RuntimeError("Bad volume chain found")
--
To view, visit https://gerrit.ovirt.org/42921
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iae354de36db63ae3bf4b4fc7f72df5e306035784
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Adam Litke <alitke(a)redhat.com>
Hello Fred Rolland,
I'd like you to do a code review. Please visit
https://gerrit.ovirt.org/44011
to review the following change.
Change subject: fc-connect-server: Support FCP on connect server
......................................................................
fc-connect-server: Support FCP on connect server
Add FCP support in in hsm._connectionDict2ConnectionInfo.
Change-Id: I408d8364278a1a502fc94a2e6537cb160c716ff1
Bug-Url: https://bugzilla.redhat.com/1242200
Signed-off-by: Fred Rolland <frolland(a)redhat.com>
---
M vdsm/storage/hsm.py
1 file changed, 3 insertions(+), 3 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/44011/1
diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py
index f22d8d7..a79b7ee 100644
--- a/vdsm/storage/hsm.py
+++ b/vdsm/storage/hsm.py
@@ -116,9 +116,7 @@
sd.LOCALFS_DOMAIN: 'localfs',
sd.NFS_DOMAIN: 'nfs',
sd.ISCSI_DOMAIN: 'iscsi',
- # FCP domain shouldn't even be on the list but VDSM use to just
- # accept this type as iscsi so we are stuck with it
- sd.FCP_DOMAIN: 'iscsi',
+ sd.FCP_DOMAIN: 'fcp',
sd.POSIXFS_DOMAIN: 'posixfs',
sd.GLUSTERFS_DOMAIN: 'glusterfs'}
@@ -241,6 +239,8 @@
cred = iscsi.ChapCredentials(username, password)
params = storageServer.IscsiConnectionParameters(target, iface, cred)
+ elif typeName == 'fcp':
+ params = storageServer.FcpConnectionParameters('fcp')
else:
raise se.StorageServerActionError()
--
To view, visit https://gerrit.ovirt.org/44011
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I408d8364278a1a502fc94a2e6537cb160c716ff1
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Freddy Rolland <frolland(a)redhat.com>
Gerrit-Reviewer: Fred Rolland <frolland(a)redhat.com>
Hello Fred Rolland,
I'd like you to do a code review. Please visit
https://gerrit.ovirt.org/44009
to review the following change.
Change subject: fc-connect-server: Move call to refreshStorage
......................................................................
fc-connect-server: Move call to refreshStorage
For clearer readability, move call to sdCache.refreshStorage from
__prefetchDomains to connectStorageServer.
Change-Id: If8ac84cc2bfd3490f3da41f0bc79fc372495a5a7
Bug-Url: https://bugzilla.redhat.com/1242200
Signed-off-by: Fred Rolland <frolland(a)redhat.com>
---
M vdsm/storage/hsm.py
1 file changed, 5 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/44009/1
diff --git a/vdsm/storage/hsm.py b/vdsm/storage/hsm.py
index f68d3bb..f22d8d7 100644
--- a/vdsm/storage/hsm.py
+++ b/vdsm/storage/hsm.py
@@ -2388,7 +2388,6 @@
uuidPatern = "????????-????-????-????-????????????"
if domType in (sd.FCP_DOMAIN, sd.ISCSI_DOMAIN):
- sdCache.refreshStorage()
uuids = tuple(blockSD.getStorageDomainsList())
elif domType is sd.NFS_DOMAIN:
lPath = conObj._mountCon._getLocalPath()
@@ -2457,6 +2456,11 @@
else:
status = 0
try:
+ # In case there were changes in devices size
+ # while the VDSM was not connected, we need to
+ # call refreshStorage.
+ if domType in (sd.FCP_DOMAIN, sd.ISCSI_DOMAIN):
+ sdCache.refreshStorage()
doms = self.__prefetchDomains(domType, conObj)
except:
self.log.debug("prefetch failed: %s",
--
To view, visit https://gerrit.ovirt.org/44009
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If8ac84cc2bfd3490f3da41f0bc79fc372495a5a7
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Freddy Rolland <frolland(a)redhat.com>
Gerrit-Reviewer: Fred Rolland <frolland(a)redhat.com>
Dan Kenigsberg has uploaded a new change for review.
Change subject: spec: Enable vhostmd on non-koji Fedora builds
......................................................................
spec: Enable vhostmd on non-koji Fedora builds
with_vhostmd is configurable since vhostmd is missing from Centos 6/7 +
EPEL. For everywhere else, we want to set with_vhostmd=1
Recently, we've added
http://jenkins.ovirt.org/job/vhostmd_create-rpms_el6/http://jenkins.ovirt.org/job/vhostmd_create-rpms_el7/
which let us ship vhostmd within oVirt repos.
So basically, we can take vhostmd if it's not el, or if it's built out of koji.
Change-Id: Ie0bbca861f60d28bb23404b70888321f90ab101a
Signed-off-by: Dima Kuznetsov <dkuznets(a)redhat.com>
Reviewed-on: http://gerrit.ovirt.org/36316
Reviewed-by: Yaniv Bronhaim <ybronhei(a)redhat.com>
Reviewed-by: Dan Kenigsberg <danken(a)redhat.com>
---
M vdsm.spec.in
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/50/44350/1
diff --git a/vdsm.spec.in b/vdsm.spec.in
index 8a54ed8..9d68e9b 100644
--- a/vdsm.spec.in
+++ b/vdsm.spec.in
@@ -39,7 +39,7 @@
%global with_gluster 1
%endif
-%if ! 0%{?rhel}
+%if ! 0%{?rhel} || ! 0%{fedora_koji_build}
%global with_vhostmd 1
%endif
--
To view, visit https://gerrit.ovirt.org/44350
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie0bbca861f60d28bb23404b70888321f90ab101a
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.5
Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Dima Kuznetsov <dkuznets(a)redhat.com>
Adam Litke has uploaded a new change for review.
Change subject: Live Merge: Prevent merge when base volume is too small
......................................................................
Live Merge: Prevent merge when base volume is too small
When a disk with snapshots is resized you can end up with an image chain
containing volumes of different sizes. For example: a VM may start with
volume A, a snapshot operation causes volume B to be added to the chain,
and a resize operation enlarges volume B. This mixed-size chain
presents a special case for live merge.
Libvirt's virDomainBlockCommit behaves differently depending on whether
the base volume is on block storage or file storage and whether
the base volume is raw or qcow2:
File volume, cow format:
- Update qcow header and truncate volume file to the new size
File volume, raw format:
- Truncate volume file to the new size
Block volume, cow format:
- Update qcow header only (volume size is managed by vdsm)
Block volume, raw format:
- Fail because qemu cannot extend a raw block device
Since we know the raw block case is not possible we can detect this and
return a special error message to engine. To avoid this error, engine
should use the SPM host to extend the raw volume to the correct size
before starting the merge.
We don't _have_ to change vdsm since the current code fails gracefully
(albeit with a traceback in the logs). If engine always performs the
resize operation before calling merge we'll never see this error.
Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1232481
Change-Id: Ibf77a7c5108b500c6ec34653ef7570a841def1b4
Signed-off-by: Adam Litke <alitke(a)redhat.com>
---
M lib/vdsm/define.py
M vdsm/virt/vm.py
2 files changed, 38 insertions(+), 19 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/36/42836/1
diff --git a/lib/vdsm/define.py b/lib/vdsm/define.py
index 284b10f..7c5f7b0 100644
--- a/lib/vdsm/define.py
+++ b/lib/vdsm/define.py
@@ -169,6 +169,9 @@
'ksmErr': {'status': {
'code': 71,
'message': 'Failed to update KSM values'}},
+ 'destVolumeTooSmall': {'status': {
+ 'code': 72,
+ 'message': 'Destination volume is too small'}},
'recovery': {'status': {
'code': 99,
'message': 'Recovering from crash or Initializing'}},
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 4977d28..549cd38 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -4570,7 +4570,30 @@
jobsRet[jobID] = entry
return jobsRet
+ def _merge_check_base_size(self, drive, base_info, top_info):
+ # If the drive waa resized the top volume could be larger than the
+ # base volume. Libvirt can handle this situation for filw-based
+ # volumes and block qcow volumes (where extension happens dynamically).
+ # Raw block volumes cannot be extended by libvirt so we require ovirt
+ # engine to extend them before calling merge. Check here.
+ if not drive.blockDev or base_info['format'] != 'RAW':
+ return True
+
+ if int(base_info['capacity']) < int(top_info['capacity']):
+ self.log.error("merge: The base volume is undersized and cannot "
+ "be extended. Aborting.")
+ return False
+ return True
+
def merge(self, driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID):
+ def get_volume_info(drive, volUUID):
+ res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID,
+ drive.imageID, volUUID)
+ if res['status']['code'] != 0:
+ self.log.error("Unable to get volume info for '%s'", volUUID)
+ raise LookupError(volUUID)
+ return res['info']
+
if not caps.getLiveMergeSupport():
self.log.error("Live merge is not supported on this host")
return errCode['mergeErr']
@@ -4604,14 +4627,15 @@
self.log.error("merge: top volume '%s' not found", topVolUUID)
return errCode['mergeErr']
+ try:
+ baseInfo = get_volume_info(drive, baseVolUUID)
+ topInfo = get_volume_info(drive, topVolUUID)
+ except LookupError:
+ return errCode['mergeErr']
+
# If base is a shared volume then we cannot allow a merge. Otherwise
# We'd corrupt the shared volume for other users.
- res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID,
- drive.imageID, baseVolUUID)
- if res['status']['code'] != 0:
- self.log.error("Unable to get volume info for '%s'", baseVolUUID)
- return errCode['mergeErr']
- if res['info']['voltype'] == 'SHARED':
+ if baseInfo['voltype'] == 'SHARED':
self.log.error("merge: Refusing to merge into a shared volume")
return errCode['mergeErr']
@@ -4628,18 +4652,9 @@
# pivot to the new active layer (baseVolUUID).
flags |= libvirt.VIR_DOMAIN_BLOCK_COMMIT_ACTIVE
- # If top is the active layer, it's allocated size is stored in
- # drive.apparentsize.
- topSize = drive.apparentsize
- else:
- # If top is an internal volume, we must call getVolumeInfo
- res = self.cif.irs.getVolumeInfo(drive.domainID, drive.poolID,
- drive.imageID, topVolUUID)
- if res['status']['code'] != 0:
- self.log.error("Unable to get volume info for '%s'",
- topVolUUID)
- return errCode['mergeErr']
- topSize = int(res['info']['apparentsize'])
+ # Make sure we can merge into the base in case the drive was enlarged.
+ if not self._merge_check_base_size(drive, baseInfo, topInfo):
+ return errCode['destVolumeTooSmall']
# Take the jobs lock here to protect the new job we are tracking from
# being cleaned up by queryBlockJobs() since it won't exist right away
@@ -4672,7 +4687,8 @@
# live merge operation.
if drive.chunked:
capacity, alloc, physical = self._getExtendInfo(drive)
- self.extendDriveVolume(drive, baseVolUUID, topSize, capacity)
+ topAllocSize = int(topInfo['apparentsize'])
+ self.extendDriveVolume(drive, baseVolUUID, topAllocSize, capacity)
# Trigger the collection of stats before returning so that callers
# of getVmStats after this returns will see the new job
--
To view, visit https://gerrit.ovirt.org/42836
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibf77a7c5108b500c6ec34653ef7570a841def1b4
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Adam Litke <alitke(a)redhat.com>
Nir Soffer has uploaded a new change for review.
Change subject: nettestlib: Delete qdisc if it was added
......................................................................
nettestlib: Delete qdisc if it was added
After we cleanup up the networking tests, we have now one issue left:
17:05:43 root: DEBUG: /usr/sbin/brctl addbr vdsm-fboHCFgRN0 (cwd None)
17:05:43 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
17:05:43 root: DEBUG: /sbin/ip link set vdsm-fboHCFgRN0 up (cwd None)
17:05:43 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
17:05:43 root: DEBUG: /usr/sbin/tc qdisc add dev vdsm-fboHCFgRN0 ingress (cwd None)
17:05:43 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
17:05:43 root: DEBUG: /sbin/ip link set vdsm-fboHCFgRN0 down (cwd None)
17:05:43 root: DEBUG: SUCCESS: <err> = ''; <rc> = 0
17:05:43 root: DEBUG: /usr/sbin/brctl delbr vdsm-fboHCFgRN0 (cwd None)
17:05:43 root: DEBUG: FAILED: <err> = "bridge vdsm-fboHCFgRN0 is still up; can't delete it\n"; <rc> = 1
This failure effect now on the tcTests, using @reqire_tc.
I suspect that adding a qdisc is asynchronous, causing the bridge state
to change after bringing the device down and waiting for netlink "down"
event.
Trying to delete the qdisc before bringing the device down.
Change-Id: Ia2db2b307350c4c8131f393b89fa5e9a78eafc3a
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M tests/nettestlib.py
1 file changed, 2 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/68/44368/1
diff --git a/tests/nettestlib.py b/tests/nettestlib.py
index 51bafc4..c3457d9 100644
--- a/tests/nettestlib.py
+++ b/tests/nettestlib.py
@@ -176,6 +176,8 @@
except ExecError as e:
raise SkipTest("%r has failed: %s\nDo you have Traffic Control kernel "
"modules installed?" % (EXT_TC, e.err))
+ else:
+ check_call([EXT_TC, 'qdisc', 'delete', 'dev', dev.devName, 'ingress'])
finally:
dev.delDevice()
--
To view, visit https://gerrit.ovirt.org/44368
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia2db2b307350c4c8131f393b89fa5e9a78eafc3a
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
Nir Soffer has uploaded a new change for review.
Change subject: iscsicred: Protect sensitive return value in supervdsm log
......................................................................
iscsicred: Protect sensitive return value in supervdsm log
SuperVdsm logs all calls and return values, possibly exposive sensitive
data in its log file.
This patch allows Supervdsm methods to wrap the return value with
ProtectedPassword object, so the results are logged as ********.
Change-Id: Idcfa4ee17466f75270909587c45ee9f703e5e9f3
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M vdsm/supervdsmServer
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/78/43178/1
diff --git a/vdsm/supervdsmServer b/vdsm/supervdsmServer
index 2a1a8f0..36c67b1 100755
--- a/vdsm/supervdsmServer
+++ b/vdsm/supervdsmServer
@@ -53,6 +53,7 @@
except ImportError:
_glusterEnabled = False
+from vdsm import password
from vdsm import utils
from vdsm import sysctl
from vdsm.tool import restore_nets
@@ -112,6 +113,8 @@
raise
callbackLogger.debug('return %s with %s',
func.__name__, res)
+ if isinstance(res, password.ProtectedPassword):
+ res = res.value
return res
return wrapper
--
To view, visit https://gerrit.ovirt.org/43178
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idcfa4ee17466f75270909587c45ee9f703e5e9f3
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>