Change in vdsm[master]: virt: Ensure cleanup after failed recovered migration
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: Ensure cleanup after failed recovered migration
......................................................................
virt: Ensure cleanup after failed recovered migration
We check for migrations in Vdsm recovery. If we detect that a recovered VM is
migrating to another host, we put it to Migration Source status, which is a
right thing to do. However the other part of that is to make sure the VM gets
out of Migration Source status once the migration finishes, either successfully
or unsuccessfully. This is normally ensured by SourceThread but it doesn't run
anymore after Vdsm recovery.
In case of successful migration libvirt will notify us and we do what's needed
in VM.onJobCompleted. However in case of a failed migration, we may not
receive any notification from libvirt. In order to prevent the VM from
remaining in Migration Source status “forever” (let's say to the next Vdsm
restart), we must check the status of the VM ourselves.
One way to do that is to start a special thread in SourceThread for that
purpose. It checks the migration job status in regular intervals; those
intervals should be not too short to not query libvirt too often and not too
long to avoid possible interference with possibly following domain jobs. Once
the job is finished, we check whether the migration failed and do what's needed
in such a case.
Change-Id: I93394e7a805bad4cf0a1fbf8e7975ee1f04130d0
Signed-off-by: Milan Zamazal <mzamazal(a)redhat.com>
---
M lib/vdsm/virt/migration.py
M vdsm/virt/vm.py
2 files changed, 23 insertions(+), 0 deletions(-)
Approvals:
Jenkins CI: Passed CI tests
Francesco Romani: Looks good to me, approved
Milan Zamazal: Verified
--
To view, visit https://gerrit.ovirt.org/73738
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I93394e7a805bad4cf0a1fbf8e7975ee1f04130d0
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years
Change in vdsm[master]: virt: host: stats: do not replace HostMonitor
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: host: stats: do not replace HostMonitor
......................................................................
virt: host: stats: do not replace HostMonitor
Some time ago we moved the HostMonitor operation inside the periodic
framework. This was made out of convenience, to kill one extra thread,
under the assumption that HostMonitor never blocks.
But from https://bugzilla.redhat.com/show_bug.cgi?id=1419856#c15
we learn that indeed HostMonitor can indeed block under rare, still
not completely clear, circumstances.
If more than two thread enter the netlink interface, all of them block.
There is perhaps a bug in libnl, but we should never access concurrently
this interface in the first place. So, if one HostMonitor is slow, we
must never replace it, so we can never access the same interface more
than once.
This is a partial fix. It is still unclear why the first HostMonitor is
slow, so why we end up with a timeout in the first place; but still,
given
1. the original intentions for moving HostMonitor in the periodic
framework
2. the fact that we don't need the host stats in the critical flow
3. the fact that we should never access concurrently the netlink
interface
then it is safe to never replace the HostMonitor operation. If it is
blocked, it will stay stuck unless it unblocks, or Vdsm is restarted.
Change-Id: I60ccd4e0e239ce5dfa2c90947bd6cd59a23e51b3
Related-To: https://bugzilla.redhat.com/1419856
Backport-To: 4.1
Backport-To: 4.0
Signed-off-by: Francesco Romani <fromani(a)redhat.com>
---
M lib/vdsm/virt/periodic.py
1 file changed, 3 insertions(+), 1 deletion(-)
Approvals:
Piotr Kliczewski: Looks good to me, but someone else must approve
Yaniv Bronhaim: Looks good to me, but someone else must approve
Jenkins CI: Passed CI tests
Dan Kenigsberg: Looks good to me, approved
Francesco Romani: Verified
Milan Zamazal: Looks good to me, but someone else must approve
--
To view, visit https://gerrit.ovirt.org/73133
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I60ccd4e0e239ce5dfa2c90947bd6cd59a23e51b3
Gerrit-PatchSet: 12
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Edward Haas <edwardh(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Martin Polednik <mpolednik(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years
Change in vdsm[master]: periodic: rename: _step -> _reschedule
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: periodic: rename: _step -> _reschedule
......................................................................
periodic: rename: _step -> _reschedule
Trivial rename for the sake of readability.
No change in logic.
Change-Id: Ie12fe886c60c43fed8f8cf03e34defaf6fb6e82c
Signed-off-by: Francesco Romani <fromani(a)redhat.com>
---
M lib/vdsm/virt/periodic.py
1 file changed, 4 insertions(+), 4 deletions(-)
Approvals:
Jenkins CI: Passed CI tests
Dan Kenigsberg: Looks good to me, approved
Francesco Romani: Verified
Milan Zamazal: Looks good to me, but someone else must approve
--
To view, visit https://gerrit.ovirt.org/73676
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie12fe886c60c43fed8f8cf03e34defaf6fb6e82c
Gerrit-PatchSet: 8
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years
Change in vdsm[master]: tests: Fix flaky picklecopy benchmark
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: tests: Fix flaky picklecopy benchmark
......................................................................
Patch Set 3: Code-Review-1
placing -1 becuase print() in tests is awkward, and not really required here.
--
To view, visit https://gerrit.ovirt.org/69738
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ice02137c9e6285401a9af29fd1b5ecd2a9c6a2cb
Gerrit-PatchSet: 3
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: No
7 years
Change in vdsm[master]: virt: periodic: HostMonitor: set not-discardable
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: virt: periodic: HostMonitor: set not-discardable
......................................................................
Patch Set 2:
(1 comment)
https://gerrit.ovirt.org/#/c/74492/2/lib/vdsm/virt/periodic.py
File lib/vdsm/virt/periodic.py:
Line 109: config.getint('vars', 'host_sample_stats_interval'),
Line 110: scheduler,
Line 111: timeout=None,
Line 112: exclusive=True,
Line 113: discard=False),
I think that we should not whine every timeout (15 seconds), but only after an hour or so of constant discarding
Line 114:
Line 115: Operation(
Line 116: containersconnection.monitor,
Line 117: config.getint('vars', 'vm_sample_interval'),
--
To view, visit https://gerrit.ovirt.org/74492
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If9d6923ad397c85909067747f0d7dd2a94dce8a1
Gerrit-PatchSet: 2
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: Yes
7 years
Change in vdsm[master]: periodic: rename: _step -> _reschedule
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: periodic: rename: _step -> _reschedule
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit https://gerrit.ovirt.org/73676
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie12fe886c60c43fed8f8cf03e34defaf6fb6e82c
Gerrit-PatchSet: 7
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: No
7 years
Change in vdsm[master]: virt: host: stats: do not replace HostMonitor
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: virt: host: stats: do not replace HostMonitor
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit https://gerrit.ovirt.org/73133
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I60ccd4e0e239ce5dfa2c90947bd6cd59a23e51b3
Gerrit-PatchSet: 10
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Edward Haas <edwardh(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Martin Polednik <mpolednik(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: No
7 years
Change in vdsm[master]: virt: periodic: support for exclusive operations
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: periodic: support for exclusive operations
......................................................................
virt: periodic: support for exclusive operations
One "exclusive operation" is one operation that guarantee
that any given time either none or exactly one instance of
the callable are being executed, regardless of its state (perhaps
running fine, perhaps slow, perhaps blocked).
Doing like this we can avoid that more than one instance somehow
enters one critical region, exhausting limited system resources
(libvirt worker threads, netlink socket connections).
If we enable this flag, the executor behaves much like a regular thread
pool. Please also note that the concept of exclusive operation is
related, but different, to the timeout.
The timeout controls when a slow worker should be discarded, while
the exclusive flag controls the new attempt to reschedule one periodic
operation. You use the timeout to control when/if to give up, and the
exclusive flag to control how aggressively retry.
Up until now, there it was no need for this `exclusive` flag, because
all the periodic operations were considered important enough to deserve
aggressive rescheduling. The exception is the HostMonitor operations,
for which we can afford delayed reporting of the state, and for which
we discovered that, because of a still not confirmed underlying netlink
issue, aggressively retrying causes more harm than benefits.
In the future, we may want reevaluate if other periodic operations
may benefit of less aggressive scheduling.
Change-Id: I2cbb708a246c25065a46688de739ea0ec4f52c77
Related-To: https://bugzilla.redhat.com/1419856
Backport-To: 4.1
Backport-To: 4.0
Signed-off-by: Francesco Romani <fromani(a)redhat.com>
---
M lib/vdsm/virt/periodic.py
M tests/periodic_test.py
2 files changed, 62 insertions(+), 4 deletions(-)
Approvals:
Jenkins CI: Passed CI tests
Dan Kenigsberg: Looks good to me, approved
Francesco Romani: Verified
Milan Zamazal: Looks good to me, but someone else must approve
--
To view, visit https://gerrit.ovirt.org/73534
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2cbb708a246c25065a46688de739ea0ec4f52c77
Gerrit-PatchSet: 9
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Martin Polednik <mpolednik(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
7 years
Change in vdsm[master]: virt: periodic: support for exclusive operations
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: virt: periodic: support for exclusive operations
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit https://gerrit.ovirt.org/73534
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cbb708a246c25065a46688de739ea0ec4f52c77
Gerrit-PatchSet: 8
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Francesco Romani <fromani(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Martin Polednik <mpolednik(a)redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal(a)redhat.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: No
7 years
Change in vdsm[master]: xmlrpc: stop detection
by Code Review
From Dan Kenigsberg <danken(a)redhat.com>:
Dan Kenigsberg has posted comments on this change.
Change subject: xmlrpc: stop detection
......................................................................
Patch Set 4: Code-Review+1
--
To view, visit https://gerrit.ovirt.org/72740
To unsubscribe, visit https://gerrit.ovirt.org/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I6ab7d42efab98f23736d323b2b0b614a568b7112
Gerrit-PatchSet: 4
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
Gerrit-Reviewer: Dan Kenigsberg <danken(a)redhat.com>
Gerrit-Reviewer: Irit Goihman <igoihman(a)redhat.com>
Gerrit-Reviewer: Jenkins CI
Gerrit-Reviewer: Nir Soffer <nsoffer(a)redhat.com>
Gerrit-Reviewer: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>
Gerrit-Reviewer: gerrit-hooks <automation(a)ovirt.org>
Gerrit-HasComments: No
7 years