Hello Ayal Baron, Haim Ateya,
I'd like you to do a code review. Please visit
to review the following change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Fix block volume atomicity creation issue.
BlockVolume creation is a 2 step process where first an LV is created and then tags are added to it. If another host refreshes it's cache between the 2 commands it will have an LV with partial data on it. To solve this issue, we add an initial tag to the lvcreate command so that any other host would be able to identify this volume as incomplete and ignore it. Special LVs still have not MD tags.
Bug-uri: https://bugzilla.redhat.com/show_bug.cgi?id=876558
Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Signed-off-by: Eduardo ewarszaw@redhat.com Reviewed-on: https://gerrit.eng.lab.tlv.redhat.com/3299 Reviewed-by: Ayal Baron abaron@redhat.com Tested-by: Haim Ateya hateya@redhat.com --- M vdsm/storage/blockVolume.py 1 file changed, 29 insertions(+), 14 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/58/9358/1
diff --git a/vdsm/storage/blockVolume.py b/vdsm/storage/blockVolume.py index 98c471b..692f8cd 100644 --- a/vdsm/storage/blockVolume.py +++ b/vdsm/storage/blockVolume.py @@ -20,6 +20,7 @@
import os import threading +import logging import sanlock
from vdsm.config import config @@ -42,6 +43,7 @@ TAG_PREFIX_MDNUMBLKS = "MS_" TAG_PREFIX_IMAGE = "IU_" TAG_PREFIX_PARENT = "PU_" +TAG_VOL_UNINIT = "RHAT_VOL_INITIALIZING" VOLUME_TAGS = [TAG_PREFIX_PARENT, TAG_PREFIX_IMAGE, TAG_PREFIX_MD, @@ -58,6 +60,7 @@ # - 2..100 (Unassigned) RESERVED_LEASES = 100
+log = logging.getLogger('Storage.Volume') rmanager = rm.ResourceManager.getInstance()
@@ -159,7 +162,8 @@ else: volSize = "%s" % (size / 2 / 1024)
- lvm.createLV(dom.sdUUID, volUUID, volSize, activate=True) + lvm.createLV(dom.sdUUID, volUUID, volSize, activate=True, + initialTag=TAG_VOL_UNINIT)
fileUtils.safeUnlink(volPath) os.symlink(lvm.lvPath(dom.sdUUID, volUUID), volPath) @@ -180,11 +184,11 @@
with cls._tagCreateLock: mdSlot = dom.mapMetaOffset(volUUID, VOLUME_MDNUMBLKS) - lvm.addLVTags(dom.sdUUID, volUUID, ( - "%s%s" % (TAG_PREFIX_MD, mdSlot), - "%s%s" % (TAG_PREFIX_PARENT, srcVolUUID,), - "%s%s" % (TAG_PREFIX_IMAGE, imgUUID,) - )) + mdTags = ["%s%s" % (TAG_PREFIX_MD, mdSlot), + "%s%s" % (TAG_PREFIX_PARENT, srcVolUUID), + "%s%s" % (TAG_PREFIX_IMAGE, imgUUID)] + lvm.changeLVTags(dom.sdUUID, volUUID, delTags=[TAG_VOL_UNINIT], + addTags=mdTags)
try: lvm.deactivateLVs(dom.sdUUID, volUUID) @@ -541,13 +545,15 @@ def getMetaOffset(self): if self.metaoff: return self.metaoff - l = lvm.getLV(self.sdUUID, self.volUUID).tags - for t in l: - if t.startswith(TAG_PREFIX_MD): - return int(t[3:]) - self.log.error("missing offset tag on volume %s", self.volUUID) - raise se.VolumeMetadataReadError("missing offset tag on volume %s" - % self.volUUID) + try: + md = _getVolumeTag(self.sdUUID, self.volUUID, TAG_PREFIX_MD) + except se.MissingTagOnLogicalVolume: + self.log.error("missing offset tag on volume %s/%s", + self.sdUUID, self.volUUID, exc_info=True) + raise se.VolumeMetadataReadError("missing offset tag on volume" + "%s/%s" % (self.sdUUID, self.volUUID)) + else: + return int(md)
def getMetadataId(self): """ @@ -633,7 +639,16 @@
def _getVolumeTag(sdUUID, volUUID, tagPrefix): - for tag in lvm.getLV(sdUUID, volUUID).tags: + tags = lvm.getLV(sdUUID, volUUID).tags + if TAG_VOL_UNINIT in tags: + log.warning("Reloading unfinished volume %s/%s", sdUUID, volUUID) + lvm.invalidateVG(sdUUID) + tags = lvm.getLV(sdUUID, volUUID).tags + if TAG_VOL_UNINIT in tags: + log.error("Unfinished volume %s/%s", sdUUID, volUUID) + raise se.VolumeDoesNotExist("%s/%s" % (sdUUID, volUUID)) + + for tag in tags: if tag.startswith(tagPrefix): return tag[len(tagPrefix):]
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/64/ (1/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/87/ (2/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: Fails; I would prefer that you didn't submit this
Build Failed
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/64/ : UNSTABLE
http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/87/ : FAILURE
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: No score; No score
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/88/ (2/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: Fails; I would prefer that you didn't submit this
Build Failed
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/64/ : UNSTABLE
http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/88/ : FAILURE
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: No score; No score
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/89/ (2/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: Fails; I would prefer that you didn't submit this
Build Failed
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/64/ : UNSTABLE
http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/89/ : FAILURE
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: No score; No score
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/103/ (2/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: Fails; I would prefer that you didn't submit this
Build Failed
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/64/ : UNSTABLE
http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/103/ : FAILURE
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Ayal Baron has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(3 inline comments)
.................................................... File vdsm/storage/blockVolume.py Line 547: return self.metaoff Line 548: try: Line 549: md = _getVolumeTag(self.sdUUID, self.volUUID, TAG_PREFIX_MD) Line 550: except se.MissingTagOnLogicalVolume: Line 551: self.log.error("missing offset tag on volume %s/%s", I think this log should be in: _getVolumeTag and not here Line 552: self.sdUUID, self.volUUID, exc_info=True) Line 553: raise se.VolumeMetadataReadError("missing offset tag on volume" Line 554: "%s/%s" % (self.sdUUID, self.volUUID)) Line 555: else:
Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading unfinished volume %s/%s", sdUUID, volUUID) I see no reason to log this, this situation is totally expected due to the way we work. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Unfinished volume %s/%s", sdUUID, volUUID)
Line 644: log.warning("Reloading unfinished volume %s/%s", sdUUID, volUUID) Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Unfinished volume %s/%s", sdUUID, volUUID) s/Unfinished volume/Found uninitialized volume: "/ Line 649: raise se.VolumeDoesNotExist("%s/%s" % (sdUUID, volUUID)) Line 650: Line 651: for tag in tags: Line 652: if tag.startswith(tagPrefix):
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2:
Build Started http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/132/ (2/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2:
Build Started http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/98/ (1/2)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Eduardo has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 1: (2 inline comments)
.................................................... File vdsm/storage/blockVolume.py Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading unfinished volume %s/%s", sdUUID, volUUID) This situation is changing the rules for the hsm system including the design of the lvm cache. Until we work only with volulmes in the hsm, is meaning that engine is stressing too much the cluster. We don't expect too many entries of this log. If they are is better to know it and think again the whole issue. IMHO should be harmless, not clutter the log too much and useful for debugging. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Unfinished volume %s/%s", sdUUID, volUUID)
Line 644: log.warning("Reloading unfinished volume %s/%s", sdUUID, volUUID) Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Unfinished volume %s/%s", sdUUID, volUUID) Done Line 649: raise se.VolumeDoesNotExist("%s/%s" % (sdUUID, volUUID)) Line 650: Line 651: for tag in tags: Line 652: if tag.startswith(tagPrefix):
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
oVirt Jenkins CI Server has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: I would prefer that you didn't submit this
Build Unstable
http://jenkins.ovirt.org/job/vdsm_pep8_gerrit/98/ : UNSTABLE
http://jenkins.ovirt.org/job/vdsm_unit_tests_manual_gerrit/132/ : SUCCESS
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Ayal Baron has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: (2 inline comments)
.................................................... File vdsm/storage/blockVolume.py Line 547: return self.metaoff Line 548: try: Line 549: md = _getVolumeTag(self.sdUUID, self.volUUID, TAG_PREFIX_MD) Line 550: except se.MissingTagOnLogicalVolume: Line 551: self.log.error("missing offset tag on volume %s/%s", you've already logged this in _getVolumeTag, why again? Line 552: self.sdUUID, self.volUUID, exc_info=True) Line 553: raise se.VolumeMetadataReadError("missing offset tag on volume" Line 554: "%s/%s" % (self.sdUUID, self.volUUID)) Line 555: else:
Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading uninitialized volume %s/%s", sdUUID, volUUID) still, it is 'info' at most. nothing wrong here. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Found uninitialized volume: %s/%s", sdUUID, volUUID)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Eduardo has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: (2 inline comments)
.................................................... File vdsm/storage/blockVolume.py Line 547: return self.metaoff Line 548: try: Line 549: md = _getVolumeTag(self.sdUUID, self.volUUID, TAG_PREFIX_MD) Line 550: except se.MissingTagOnLogicalVolume: Line 551: self.log.error("missing offset tag on volume %s/%s", More clear since we have no need to deduce which tag is absent and which is the operation that failed. The stack trace will be more complete here and you requested the lower log. Since it is actually a low possibility race (negative flow) should be rare and not clutter the log. If not we should fix the race and not remove logs. Line 552: self.sdUUID, self.volUUID, exc_info=True) Line 553: raise se.VolumeMetadataReadError("missing offset tag on volume" Line 554: "%s/%s" % (self.sdUUID, self.volUUID)) Line 555: else:
Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading uninitialized volume %s/%s", sdUUID, volUUID) This race is new and is a challenge for some design assumptions of the whole system. It is assumed that the storage ovirt data (MD, pool Md, volume layout, etc) is stable when the HSM is using it and if it is changed, the SPM or the engine should request a refresh of the HSM view of the storage. Then this race should be extraordinary (like today) but in any case is not the only possible one. Then a log warning would alert us that we are changing the usage pattern of the system. Other races should be checked too and avoided. This is not a "normal" situation and this type of behaviour should be avoided. The complete solution is looking in the HSM for "volumes only", only when they are needed, and only at the start of the requested API operation. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Found uninitialized volume: %s/%s", sdUUID, volUUID)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: oVirt Jenkins CI Server
Shu Ming has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm/storage/blockVolume.py Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading uninitialized volume %s/%s", sdUUID, volUUID) The original assumption depends on the SPM or engine to refresh the HSM view of the storage. And the racing comes from the HSM is using it while the volume metadata(lvm tag here) updating is undergoing. We should make the metadata updating to be atomic and block any metadata reading in any other HSM host. Only after the updating operations is done, can the HSM invalidate the lvm metadata cache and read the volume metadata. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Found uninitialized volume: %s/%s", sdUUID, volUUID)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Eduardo has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: (1 inline comment)
.................................................... File vdsm/storage/blockVolume.py Line 640: Line 641: def _getVolumeTag(sdUUID, volUUID, tagPrefix): Line 642: tags = lvm.getLV(sdUUID, volUUID).tags Line 643: if TAG_VOL_UNINIT in tags: Line 644: log.warning("Reloading uninitialized volume %s/%s", sdUUID, volUUID) """ The original assumption depends on the SPM or engine to refresh the HSM view of the storage. """ This is an axiom of the system up to now. Actually this issue can arise for other items but the LV metadata. This patch prevents the HSM using incomplete LV metadata. A complete solution can't be addressed at this point. Line 645: lvm.invalidateVG(sdUUID) Line 646: tags = lvm.getLV(sdUUID, volUUID).tags Line 647: if TAG_VOL_UNINIT in tags: Line 648: log.error("Found uninitialized volume: %s/%s", sdUUID, volUUID)
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
Haim Ateya has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 2: Verified
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 2 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com
Dan Kenigsberg has posted comments on this change.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Patch Set 3: Verified; Looks good to me, approved
copying score, as the only change is the dropping of RHAT trademark reference.
Other questions regarding logging, which can be handled post ovirt-3.2 release.
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com
Dan Kenigsberg has submitted this change and it was merged.
Change subject: Fix block volume atomicity creation issue. ......................................................................
Fix block volume atomicity creation issue.
BlockVolume creation is a 2 step process where first an LV is created and then tags are added to it. If another host refreshes it's cache between the 2 commands it will have an LV with partial data on it. To solve this issue, we add an initial tag to the lvcreate command so that any other host would be able to identify this volume as incomplete and ignore it. Special LVs still have not MD tags.
Bug-uri: https://bugzilla.redhat.com/show_bug.cgi?id=876558
Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Signed-off-by: Eduardo ewarszaw@redhat.com Reviewed-on: https://gerrit.eng.lab.tlv.redhat.com/3299 Reviewed-by: Ayal Baron abaron@redhat.com Tested-by: Haim Ateya hateya@redhat.com --- M vdsm/storage/blockVolume.py 1 file changed, 33 insertions(+), 16 deletions(-)
Approvals: Dan Kenigsberg: Verified; Looks good to me, approved
-- To view, visit http://gerrit.ovirt.org/9358 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: merged Gerrit-Change-Id: I40cd67e563935de663d938cbc1bc9cf152802448 Gerrit-PatchSet: 3 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Ayal Baron abaron@redhat.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Eduardo ewarszaw@redhat.com Gerrit-Reviewer: Haim Ateya hateya@redhat.com Gerrit-Reviewer: Shu Ming shuming@linux.vnet.ibm.com
vdsm-patches@lists.fedorahosted.org