vdsm-patches July 2015

vdsm-patches@lists.fedorahosted.org

44 participants
919 discussions

Change in vdsm[master]: gluster: Added VDSM verb to stop gluster related processes
by shtripat＠redhat.com 11 Nov '15

11 Nov '15

Shubhendu Tripathi has uploaded a new change for review. Change subject: gluster: Added VDSM verb to stop gluster related processes ...................................................................... gluster: Added VDSM verb to stop gluster related processes Added a verb which stops the gluster related process like brick processes, gsyncd process. This needs to be done as part of a host moving to maintenance mode. as data should not be available if a host is moved to maintenence mode for some migration etc. Change-Id: Id686e098b323eededcf1f89de331a1d524274995 Bug-URL: https://bugzilla.redhat.com/1205724 Signed-off-by: Shubhendu Tripathi <shtripat(a)redhat.com> --- M client/vdsClientGluster.py M vdsm/gluster/api.py M vdsm/gluster/apiwrapper.py M vdsm/gluster/exception.py M vdsm/rpc/vdsmapi-gluster-schema.json 5 files changed, 51 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/21/43821/1 diff --git a/client/vdsClientGluster.py b/client/vdsClientGluster.py index a4bce63..a72d4e3 100644 --- a/client/vdsClientGluster.py +++ b/client/vdsClientGluster.py @@ -740,6 +740,11 @@ pp.pprint(status) return status['status']['code'], status['status']['message'] + def do_glusterStopProcesses(self, args): + status = self.s.glusterStopProcesses() + pp.pprint(status) + return status['status']['code'], status['status']['message'] + def getGlusterCmdDict(serv): return \ @@ -1252,5 +1257,10 @@ serv.do_glusterSnapshotScheduleReset, ('', 'Reset gluster snapshot scheduling' + )), + 'glusterStopProcesses' : ( + serv.do_glusterStopProcesses, + ('', + 'Stop gluster processes' )) } diff --git a/vdsm/gluster/api.py b/vdsm/gluster/api.py index 781ab77..3467412 100644 --- a/vdsm/gluster/api.py +++ b/vdsm/gluster/api.py @@ -52,6 +52,12 @@ "/usr/sbin/snap_scheduler.py", ) +_stopAllProcessesPath = utils.CommandPath( + "stop-all-gluster-processes.sh", + "/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh", +) + + GLUSTER_RPM_PACKAGES = ( ('glusterfs', ('glusterfs',)), ('glusterfs-fuse', ('glusterfs-fuse',)), @@ -262,6 +268,15 @@ except (IOError, OSError) as e: raise ge.GlusterSnapshotScheduleFlagUpdateFailedException( err=[str(e)]) + return True + + +@makePublic +def stopProcesses(): + command = ["/bin/sh", _stopAllProcessesPath.cmd] + rc, out, err = utils.execCmd(command) + if rc: + raise ge.GlusterStopProcessesFailedException(rc) return True @@ -752,6 +767,10 @@ def snapshotScheduleReset(self, options=None): self.svdsmProxy.glusterSnapshotScheduleFlagUpdate("none") + @exportAsVerb + def stopProcesses(self): + self.svdsmProxy.glusterStopProcesses() + def getGlusterMethods(gluster): l = [] diff --git a/vdsm/gluster/apiwrapper.py b/vdsm/gluster/apiwrapper.py index 4768a86..13dc859 100644 --- a/vdsm/gluster/apiwrapper.py +++ b/vdsm/gluster/apiwrapper.py @@ -86,6 +86,9 @@ return self._gluster.createBrick(name, mountPoint, devList, fsType, raidParams) + def stopProcesses(self): + return self._gluster.stopProcesses() + class GlusterService(GlusterApiBase): def __init__(self): diff --git a/vdsm/gluster/exception.py b/vdsm/gluster/exception.py index 17ad018..2898d3d 100644 --- a/vdsm/gluster/exception.py +++ b/vdsm/gluster/exception.py @@ -594,6 +594,12 @@ message = "Failed to disable snapshot schedule through cli" +class GlusterStopProcessesFailedException( + GlusterVolumeException): + code = 4579 + message = "Failed to stop gluster processes" + + # geo-replication class GlusterGeoRepException(GlusterException): code = 4200 diff --git a/vdsm/rpc/vdsmapi-gluster-schema.json b/vdsm/rpc/vdsmapi-gluster-schema.json index c86f43c..2853851 100644 --- a/vdsm/rpc/vdsmapi-gluster-schema.json +++ b/vdsm/rpc/vdsmapi-gluster-schema.json @@ -1299,6 +1299,19 @@ 'returns': 'bool'} ## +# @GlusterHost.stopProcesses: +# +# Stops the gluster processes on the host +# +# Returns: +# Success or failure +# +# Since: 4.17.0 +## +{'command': {'class': 'GlusterHost', 'name': 'stopProcesses'}, + 'returns': 'bool'} + +## # @GlusterVolumeStatsInfo: # # Gluster Volumes disk usage statistics -- To view, visit https://gerrit.ovirt.org/43821 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id686e098b323eededcf1f89de331a1d524274995 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Shubhendu Tripathi <shtripat(a)redhat.com>

7 30

Change in vdsm[master]: vmchannel: performance enhancment. changing thread name
by emarcian＠redhat.com 10 Nov '15

10 Nov '15

Eldad Marciano has uploaded a new change for review. Change subject: vmchannel: performance enhancment. changing thread name ...................................................................... vmchannel: performance enhancment. changing thread name Change-Id: I8489ec3a65ed9025eb91dfc2f408450b02fd7e0c Signed-off-by: emarcian <emarcian(a)redhat.com> --- M vdsm/virt/vmchannels.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/42/36342/1 diff --git a/vdsm/virt/vmchannels.py b/vdsm/virt/vmchannels.py index c143162..e0b305e 100644 --- a/vdsm/virt/vmchannels.py +++ b/vdsm/virt/vmchannels.py @@ -167,8 +167,8 @@ events = NoIntrPoll(self._epoll.poll, 1) if events: for (fileno, event) in events: - _wait_for_events = threading.Thread(name='handle_events-%s' % fileno, target=self._handle_event, args=(fileno, event)) - _wait_for_events.start() + _handle_event = threading.Thread(name='handle_events-%s' % fileno, target=self._handle_event, args=(fileno, event)) + _handle_event.start() else: self._update_channels() if (self._timeout is not None) and (self._timeout > 0): -- To view, visit http://gerrit.ovirt.org/36342 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8489ec3a65ed9025eb91dfc2f408450b02fd7e0c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Eldad Marciano <emarcian(a)redhat.com>

5 6

Change in vdsm[master]: jsonrpc: executor based thread factory
by Piotr Kliczewski 09 Nov '15

09 Nov '15

Piotr Kliczewski has uploaded a new change for review. Change subject: jsonrpc: executor based thread factory ...................................................................... jsonrpc: executor based thread factory Creating new thread for every request is not efficient so we introduce usage of the executor for request processing. Change-Id: I56b307633a8bf7e4aad8f87cc97a4129c9ed0970 Signed-off-by: pkliczewski <piotr.kliczewski(a)gmail.com> --- M lib/vdsm/config.py.in M vdsm/rpc/bindingjsonrpc.py 2 files changed, 49 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/59/43759/1 diff --git a/lib/vdsm/config.py.in b/lib/vdsm/config.py.in index 3221aab..bffa949 100644 --- a/lib/vdsm/config.py.in +++ b/lib/vdsm/config.py.in @@ -201,6 +201,12 @@ ('connection_stats_timeout', '3600', 'Time in seconds defining how frequently we log transport stats'), + + ('worker_threads', '8', + 'Number of worker threads to serve jsonrpc server.'), + + ('tasks_per_worker', '100', + 'Max number of tasks which can be queued per workers.'), ]), # Section: [mom] diff --git a/vdsm/rpc/bindingjsonrpc.py b/vdsm/rpc/bindingjsonrpc.py index 80e3388..ca15fde 100644 --- a/vdsm/rpc/bindingjsonrpc.py +++ b/vdsm/rpc/bindingjsonrpc.py @@ -19,18 +19,53 @@ from yajsonrpc import JsonRpcServer from yajsonrpc.stompreactor import StompReactor +from vdsm import executor +from vdsm import schedule +from vdsm.config import config +from vdsm.utils import monotonic_time -def _simpleThreadFactory(func): - t = threading.Thread(target=func) - t.setDaemon(False) - t.start() + +# TODO test what should be the default values +_THREADS = config.getint('vars', 'worker_threads') +_TASK_PER_WORKER = config.getint('vars', 'tasks_per_worker') +_TASKS = _THREADS * _TASK_PER_WORKER + + +_scheduler = schedule.Scheduler(name="jsonrpc.Scheduler", + clock=monotonic_time) + +_executor = executor.Executor(name="jsonrpc.Executor", + workers_count=_THREADS, + max_tasks=_TASKS, + scheduler=_scheduler) + + +class _RequestWorker(object): + + _log = logging.getLogger("jsonrpc._RequestWorker") + + def __init__(self, func): + self._func = func + + def __call__(self): + try: + self._func() + except Exception: + self._log.exception("%s processing request failed", self._func) + + +def _executorFactory(func): + _executor.dispatch(_RequestWorker(func)) class BindingJsonRpc(object): log = logging.getLogger('BindingJsonRpc') def __init__(self, bridge, subs, timeout): - self._server = JsonRpcServer(bridge, timeout, _simpleThreadFactory) + _scheduler.start() + _executor.start() + + self._server = JsonRpcServer(bridge, timeout, _executorFactory) self._reactor = StompReactor(subs) self.startReactor() @@ -60,3 +95,6 @@ def stop(self): self._server.stop() self._reactor.stop() + + _scheduler.stop() + _executor.stop() -- To view, visit https://gerrit.ovirt.org/43759 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I56b307633a8bf7e4aad8f87cc97a4129c9ed0970 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Piotr Kliczewski <piotr.kliczewski(a)gmail.com>

6 45

Change in vdsm[master]: tests: add tests for sampling.SampleVMs
by fromani＠redhat.com 06 Nov '15

06 Nov '15

Francesco Romani has uploaded a new change for review. Change subject: tests: add tests for sampling.SampleVMs ...................................................................... tests: add tests for sampling.SampleVMs The SampleVMs class is responisble for handling bulk stats sampling, and includes all the logic to deal with blocked domains. However, this logic has the time among its variables, and it is built on quite some assumptions, like running on an executor and so on. All those factors make it difficult to test the bare logic without some (minor) cheating and without some significant faking. However, the nasty spots should be the ones more tested, not less tested, so this patch adds the initial batch of unit tests for the sampling logic. Change-Id: Id66dbd420ca29d08ae4063dc83b858be34b8940f Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/Makefile.am A tests/bulkSamplingTests.py 2 files changed, 213 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/53/40053/1 diff --git a/tests/Makefile.am b/tests/Makefile.am index 7fa1c09..4f984dd 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -26,6 +26,7 @@ alignmentScanTests.py \ blocksdTests.py \ bridgeTests.py \ + bulkSamplingTests.py \ cPopenTests.py \ capsTests.py \ clientifTests.py \ diff --git a/tests/bulkSamplingTests.py b/tests/bulkSamplingTests.py new file mode 100644 index 0000000..5c536ff --- /dev/null +++ b/tests/bulkSamplingTests.py @@ -0,0 +1,212 @@ +# +# Copyright 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +# +# Refer to the README and COPYING files for full details of the license +# + +import virt.sampling as sampling + +from testlib import VdsmTestCase as TestCaseBase + + +# these tests are complex and fragile. In order to maximize +# readability and robustness, intentionally includes tailor-made +# minimal Fakes to support those tests. + + +class FakeStatsCache(object): + def __init__(self): + self.data = [] + self._clock = 0 # private only for API clash + + def clock(self): + return self._clock + + def set_clock(self, value): + self._clock = value + + def put(self, bulk_stats, timestamp): + self.data.append((bulk_stats, timestamp)) + + +class FakeDomain(object): + def __init__(self, name): + self._name = name + self._dom = self # yep, this is an ugly hack + + def UUIDString(self): + # yes this is cheating + return self._name + + +class FakeVM(object): + def __init__(self, vmid): + self.id = vmid + self._dom = FakeDomain(vmid) + self.ready_for_commands = True + + def isDomainReadyForCommands(self): + return self.ready_for_commands + + +class CollectMode(object): + NONE = 0 + SLOW = 1 + FAST = 2 + + +class FakeConnection(object): + def __init__(self, vms): + self.vms = vms + # testable fields + self.executions = [] + + def getVMs(self): + return self.vms + + def getAllDomainStats(self, flags=0): + self.executions.append(CollectMode.FAST) + return [(vm._dom, {'vmid': vm._dom.UUIDString()}) + for vm in self.vms.values()] + + def domainListGetStats(self, doms, flags=0): + self.executions.append(CollectMode.SLOW) + return [(dom, {'vmid': dom.UUIDString()}) + for dom in doms + if dom.UUIDString() in self.vms] + + +class SampleVMsTests(TestCaseBase): + + def test_collect_fast_path_as_default(self): + cache = FakeStatsCache() + vms = { + '1': FakeVM('1'), + '2': FakeVM('2') + } + conn = FakeConnection(vms) + + sampler = sampling.SampleVMs(conn, conn.getVMs, cache) + sampler() + + self.assertEqual(conn.executions[0], CollectMode.FAST) + self.assertVmsInBulkStats(vms.keys(), cache.data[0][0]) + + def test_collect_slow_path_after_blocked(self): + cache = FakeStatsCache() + vms = { + '1': FakeVM('1'), + '2': FakeVM('2') + } + conn = FakeConnection(vms) + + sampler = sampling.SampleVMs(conn, conn.getVMs, cache) + # this is cheating, but setting up proper blocking is costly + # and fragile. + sampler._sampling = True + sampler() + + self.assertEqual(conn.executions[0], CollectMode.SLOW) + # SampleVMs doesn't care in the slow path if all registed + # VMs are actually responsive or not. Quite the opposite, it is + # good sign if all VMs are now responsive, it means we can + # restart using getAllDomainStats on the next cycle. + self.assertVmsInBulkStats(vms.keys(), cache.data[0][0]) + + def test_collect_unresponsive_vm(self): + cache = FakeStatsCache() + vms = { + '1': FakeVM('1'), + '2': FakeVM('2'), + '3': FakeVM('3') + } + vms['2'].ready_for_commands = False + conn = FakeConnection(vms) + + sampler = sampling.SampleVMs(conn, conn.getVMs, cache) + # this is cheating, but setting up proper blocking is costly + # and fragile. + sampler._sampling = True + sampler() + + self.assertEqual(conn.executions[0], CollectMode.SLOW) + self.assertVmsInBulkStats(('1', '3'), cache.data[0][0]) + self.assertVmStuck(sampler, '2') + + def test_slow_collect_while_unresponsive_vm(self): + cache = FakeStatsCache() + vms = { + '1': FakeVM('1'), + '2': FakeVM('2'), + '3': FakeVM('3') + } + vms['2'].ready_for_commands = False + conn = FakeConnection(vms) + + sampler = sampling.SampleVMs(conn, conn.getVMs, cache) + sampler._sampling = True # cheat to bootstrap slow collection + sampler() + sampler() + sampler() + + self.assertSamplingExecutions( + conn.executions, + (CollectMode.SLOW, CollectMode.SLOW, CollectMode.SLOW)) + for datum in cache.data: + self.assertVmsInBulkStats(('1', '3'), datum[0]) + self.assertVmStuck(sampler, '2') + + def test_fast_collect_once_vm_responsive(self): + cache = FakeStatsCache() + vms = { + '1': FakeVM('1'), + '2': FakeVM('2'), + '3': FakeVM('3') + } + vms['2'].ready_for_commands = False + conn = FakeConnection(vms) + + sampler = sampling.SampleVMs(conn, conn.getVMs, cache) + sampler._sampling = True # cheat to bootstrap slow collection + sampler() + sampler() + vms['2'].ready_for_commands = True + sampler._skip_doms.clear() # cheating, again + sampler() + + self.assertSamplingExecutions( + conn.executions, + (CollectMode.SLOW, CollectMode.SLOW, CollectMode.FAST)) + for datum in cache.data[:-1]: + self.assertVmsInBulkStats(('1', '3'), datum[0]) + self.assertVmsInBulkStats(vms.keys(), cache.data[-1][0]) + + def assertSamplingExecutions(self, executions, expected): + for done, exp in zip(executions[-len(expected):], expected): + self.assertEqual(done, exp) + + def assertVmsInBulkStats(self, vm_ids, bulk_stats): + """ + checks only for the presence of stats. Stats data is fake, + so ignore it as meaningless. + """ + self.assertEqual(len(vm_ids), len(bulk_stats)) + for vm_id in vm_ids: + self.assertIn(vm_id, bulk_stats) + + def assertVmStuck(self, sampler, vm_id): + self.assertTrue(sampler._skip_doms.get(vm_id)) -- To view, visit https://gerrit.ovirt.org/40053 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Id66dbd420ca29d08ae4063dc83b858be34b8940f Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

5 103

Change in vdsm[master]: janitorial: sslutils, kaxmlrpclib: drop python-2.6-specific ...
by Dan Kenigsberg 06 Nov '15

06 Nov '15

Dan Kenigsberg has uploaded a new change for review. Change subject: janitorial: sslutils, kaxmlrpclib: drop python-2.6-specific code ...................................................................... janitorial: sslutils, kaxmlrpclib: drop python-2.6-specific code Change-Id: Icdb4ba3a52a144d7d28d9d8adf36e8b0621f10dd Signed-off-by: Dan Kenigsberg <danken(a)redhat.com> --- M lib/vdsm/sslutils.py M vdsm/kaxmlrpclib.py 2 files changed, 12 insertions(+), 64 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/64/40964/1 diff --git a/lib/vdsm/sslutils.py b/lib/vdsm/sslutils.py index 4b863a7..7e9319a 100644 --- a/lib/vdsm/sslutils.py +++ b/lib/vdsm/sslutils.py @@ -112,52 +112,13 @@ self._timeout = timeout def make_connection(self, host): - """Return VerifyingHTTPS object that is aware of ca_certs, and will - create VerifyingHTTPSConnection. - In Python 2.7, return VerifyingHTTPSConnection object - """ + """Return a VerifyingHTTPSConnection object""" chost, self._extra_headers, x509 = self.get_host_info(host) - if hasattr(xmlrpclib.SafeTransport, "single_request"): # Python 2.7 - return VerifyingHTTPSConnection( - chost, None, key_file=self.key_file, strict=None, - cert_file=self.cert_file, timeout=self._timeout, - ca_certs=self.ca_certs, - cert_reqs=self.cert_reqs) - else: - return VerifyingHTTPS( - chost, None, key_file=self.key_file, - cert_file=self.cert_file, timeout=self._timeout, - ca_certs=self.ca_certs, - cert_reqs=self.cert_reqs) - - -class VerifyingHTTPS(httplib.HTTPS): - _connection_class = VerifyingHTTPSConnection - - def __init__(self, host='', port=None, key_file=None, cert_file=None, - strict=None, timeout=SOCKET_DEFAULT_TIMEOUT, - ca_certs=None, cert_reqs=ssl.CERT_REQUIRED): - """A ca_cert-aware HTTPS object, - that creates a VerifyingHTTPSConnection - """ - # provide a default host, pass the X509 cert info - - # urf. compensate for bad input. - if port == 0: - port = None - self._setup(self._connection_class(host=host, - port=port, - key_file=key_file, - cert_file=cert_file, - strict=strict, - timeout=timeout, - ca_certs=ca_certs, - cert_reqs=cert_reqs)) - - # we never actually use these for anything, but we keep them - # here for compatibility with post-1.5.2 CVS. - self.key_file = key_file - self.cert_file = cert_file + return VerifyingHTTPSConnection( + chost, None, key_file=self.key_file, strict=None, + cert_file=self.cert_file, timeout=self._timeout, + ca_certs=self.ca_certs, + cert_reqs=self.cert_reqs) class SSLHandshakeDispatcher(object): diff --git a/vdsm/kaxmlrpclib.py b/vdsm/kaxmlrpclib.py index 6374726..e5ff677 100644 --- a/vdsm/kaxmlrpclib.py +++ b/vdsm/kaxmlrpclib.py @@ -57,10 +57,7 @@ class TcpkeepTransport(xmlrpclib.Transport): def make_connection(self, host): - if hasattr(xmlrpclib.Transport, "single_request"): # Python 2.7 - return TcpkeepHTTPConnection(host) - else: - return TcpkeepHTTP(host) + return TcpkeepHTTPConnection(host) class TcpkeepHTTPConnection(httplib.HTTPConnection): @@ -126,17 +123,11 @@ def make_connection(self, host): chost, self._extra_headers, x509 = self.get_host_info(host) - if hasattr(xmlrpclib.SafeTransport, "single_request"): # Python 2.7 - return TcpkeepHTTPSConnection( - chost, None, key_file=self.key_file, strict=None, - timeout=CONNECTTIMEOUT, - cert_file=self.cert_file, ca_certs=self.ca_certs, - cert_reqs=self.cert_reqs) - else: - return TcpkeepHTTPS( - chost, None, key_file=self.key_file, - cert_file=self.cert_file, ca_certs=self.ca_certs, - cert_reqs=self.cert_reqs) + return TcpkeepHTTPSConnection( + chost, None, key_file=self.key_file, strict=None, + timeout=CONNECTTIMEOUT, + cert_file=self.cert_file, ca_certs=self.ca_certs, + cert_reqs=self.cert_reqs) class TcpkeepHTTPSConnection(sslutils.VerifyingHTTPSConnection): @@ -158,7 +149,3 @@ self.sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, KEEPIDLE) self.sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, KEEPINTVL) self.sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT, KEEPCNT) - - -class TcpkeepHTTPS(sslutils.VerifyingHTTPS): - _connection_class = TcpkeepHTTPSConnection -- To view, visit https://gerrit.ovirt.org/40964 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icdb4ba3a52a144d7d28d9d8adf36e8b0621f10dd Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Dan Kenigsberg <danken(a)redhat.com>

4 7

Change in vdsm[master]: v2v: janitorial: use the response module
by fromani＠redhat.com 06 Nov '15

06 Nov '15

Francesco Romani has uploaded a new change for review. Change subject: v2v: janitorial: use the response module ...................................................................... v2v: janitorial: use the response module This patch makes v2v.py use the response module. Change-Id: I678e764a2895e7c34df6852759b1d43baeb687a1 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/v2v.py 1 file changed, 4 insertions(+), 5 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/47/38447/1 diff --git a/vdsm/v2v.py b/vdsm/v2v.py index 4fbecf0..18f59d9 100644 --- a/vdsm/v2v.py +++ b/vdsm/v2v.py @@ -22,8 +22,8 @@ import libvirt -from vdsm.define import errCode, doneCode from vdsm import libvirtconnection +from vdsm import response import caps @@ -39,7 +39,7 @@ def get_external_vms(uri, username, password): if not supported(): - return errCode["noimpl"] + return response.error("noimpl") try: conn = libvirtconnection.open_connection(uri=uri, @@ -47,8 +47,7 @@ passwd=password) except libvirt.libvirtError as e: logging.error('error connection to hypervisor: %r', e.message) - return {'status': {'code': errCode['V2VConnection']['status']['code'], - 'message': e.message}} + return response.error('V2VConnection', e.message) with closing(conn): vms = [] @@ -67,7 +66,7 @@ for disk in params['disks']: _add_disk_info(conn, disk) vms.append(params) - return {'status': doneCode, 'vmList': vms} + return response.success(vmList=vms) def _mem_to_mib(size, unit): -- To view, visit https://gerrit.ovirt.org/38447 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I678e764a2895e7c34df6852759b1d43baeb687a1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

7 71

Change in vdsm[master]: migration: make stop() update internal status
by fromani＠redhat.com 05 Nov '15

05 Nov '15

Francesco Romani has uploaded a new change for review. Change subject: migration: make stop() update internal status ...................................................................... migration: make stop() update internal status The SourceThread.stop() operation should update the internal status accordingly in case of success. Previously, it was the caller that updated the status (!) adding unneeded coupling. Change-Id: I0ab50fc789dde969b2fb9ab969241ed4ad12545c Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/migration.py 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/20/40520/1 diff --git a/vdsm/virt/migration.py b/vdsm/virt/migration.py index 1eaed05..61d5a7b 100644 --- a/vdsm/virt/migration.py +++ b/vdsm/virt/migration.py @@ -371,6 +371,10 @@ except libvirt.libvirtError: if not self._preparingMigrationEvt: raise + else: + self.status['status']['message'] = \ + 'Migration process cancelled' + return self.status def exponential_downtime(downtime, steps): -- To view, visit https://gerrit.ovirt.org/40520 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0ab50fc789dde969b2fb9ab969241ed4ad12545c Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

3 13

Change in vdsm[master]: WIP
by fromani＠redhat.com 05 Nov '15

05 Nov '15

Francesco Romani has uploaded a new change for review. Change subject: WIP ...................................................................... WIP Change-Id: I39c2e6e4bca286a513992b7231f1356e8dd871a1 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M vdsm/virt/sampling.py 1 file changed, 47 insertions(+), 37 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/31/40431/1 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 45afdfd..eec5ae5 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -551,51 +551,30 @@ return doms -class HostStatsThread(threading.Thread): - """ - A thread that periodically samples host statistics. - """ - _CONNLOG = logging.getLogger('connectivity') +class HostSampler(object): + _connlog = logging.getlogger('connectivity') def __init__(self, log): - self.startTime = time.time() - - threading.Thread.__init__(self) - self.daemon = True self._log = log - self._stopEvent = threading.Event() + self._start_time = time.time() self._samples = SampleWindow(size=5) self._pid = os.getpid() self._ncpus = max(os.sysconf('SC_NPROCESSORS_ONLN'), 1) - self._sampleInterval = \ - config.getint('vars', 'host_sample_stats_interval') + self._sample_interval = config.getint( + 'vars', 'host_sample_stats_interval') - def stop(self): - self._stopEvent.set() - - def run(self): - try: - # wait a bit before starting to sample - time.sleep(self._sampleInterval) - while not self._stopEvent.isSet(): - try: - sample = HostSample(self._pid) - self._samples.append(sample) - prev, last = self._samples.last_pair() - if prev is None: - self._CONNLOG.debug('%s', sample.to_connlog()) - else: - diff = sample.connlog_diff(prev) - if diff: - self._CONNLOG.debug('%s', diff) - except TimeoutError: - self._log.exception("Timeout while sampling stats") - self._stopEvent.wait(self._sampleInterval) - except: - if not self._stopEvent.isSet(): - self._log.exception("Error while sampling stats") + def __call__(self): + sample = HostSample(self._pid) + self._samples.append(sample) + prev, last = self._samples.last_pair() + if prev is None: + self._CONNLOG.debug('%s', sample.to_connlog()) + else: + diff = sample.connlog_diff(prev) + if diff: + self._CONNLOG.debug('%s', diff) @utils.memoized def _boot_time(self): @@ -625,7 +604,7 @@ hs0, hs1, _ = self._samples.stats() interval = hs1.timestamp - hs0.timestamp - stats.update(self._get_interfaces_stats(hs0, hs1, interval)) + stats.update(_get_interfaces_stats(hs0, hs1, interval)) jiffies = (hs1.pidcpu.user - hs0.pidcpu.user) % (2 ** 32) stats['cpuUserVdsmd'] = jiffies / interval @@ -655,6 +634,37 @@ return stats + +class HostStatsThread(threading.Thread): + """ + A thread that periodically samples host statistics. + """ + _connlog = logging.getlogger('connectivity') + + def __init__(self, log): + threading.Thread.__init__(self) + self._sampler = HostSampler(log) + self.daemon = True + self._stopEvent = threading.Event() + + def stop(self): + self._stopEvent.set() + + def run(self): + try: + # wait a bit before starting to sample + time.sleep(self._sampleInterval) + while not self._stopEvent.isSet(): + try: + self._sampler() + except TimeoutError: + self._log.exception("Timeout while sampling stats") + self._stopEvent.wait(self._sampleInterval) + except: + if not self._stopEvent.isSet(): + self._log.exception("Error while sampling stats") + + def _get_cpu_core_stats(hs0, hs1): """ :returns: a dict that with the following formats: -- To view, visit https://gerrit.ovirt.org/40431 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I39c2e6e4bca286a513992b7231f1356e8dd871a1 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

6 42

Change in vdsm[master]: sampling: remove HostStatsThread.get()
by fromani＠redhat.com 05 Nov '15

05 Nov '15

Francesco Romani has uploaded a new change for review. Change subject: sampling: remove HostStatsThread.get() ...................................................................... sampling: remove HostStatsThread.get() Remove the get() operation of HostStatsThread, decouple sample storage and stats production, Change-Id: I80d919b0adb48a40d41f387543f14be265610405 Signed-off-by: Francesco Romani <fromani(a)redhat.com> --- M tests/samplingTests.py M vdsm/API.py M vdsm/clientIF.py M vdsm/virt/sampling.py 4 files changed, 22 insertions(+), 16 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/36/42036/1 diff --git a/tests/samplingTests.py b/tests/samplingTests.py index ab85834..0b0bd2b 100644 --- a/tests/samplingTests.py +++ b/tests/samplingTests.py @@ -262,11 +262,13 @@ 'rxRate': 0.0, 'txRate': 0.0, } - self._hs = sampling.HostStatsThread(self.log) - self.assertEquals(self._hs.get(), expected) + self.assertEquals(hoststats.produce(None, None), expected) def testSamplesWraparound(self): - NUM = sampling.HostStatsThread.AVERAGING_WINDOW + 1 + NUM = sampling.HOST_STATS_AVERAGING_WINDOW + 1 + + samples = sampling.SampleWindow( + sampling.HOST_STATS_AVERAGING_WINDOW) class FakeEvent(object): def __init__(self, *args): @@ -299,16 +301,16 @@ pass with MonkeyPatchScope([(sampling, 'HostSample', FakeHostSample)]): - self._hs = sampling.HostStatsThread(self.log) + self._hs = sampling.HostStatsThread(self.log, samples) self._hs._sampleInterval = 0 # we cannot monkey patch, it will interfer on threading internals self._hs._stopEvent = FakeEvent() self._hs.start() self._hs.join() - first, last, _ = self._hs._samples.stats() + first, last, _ = samples.stats() self.assertEqual(first.id, FakeHostSample.counter - - sampling.HostStatsThread.AVERAGING_WINDOW) + sampling.HOST_STATS_AVERAGING_WINDOW) self.assertEqual(last.id, FakeHostSample.counter - 1) diff --git a/vdsm/API.py b/vdsm/API.py index 397f87e..e05a939 100644 --- a/vdsm/API.py +++ b/vdsm/API.py @@ -43,6 +43,8 @@ import storage.volume import storage.sd import storage.image +from virt import hoststats +from virt import sampling from virt import vmstatus from virt.vmdevices import graphics from virt.vmdevices import hwclass @@ -1318,7 +1320,8 @@ hooks.before_get_stats() stats = {} decStats = {'elapsedTime': self._cif.elapsed_time} - decStats.update(self._cif._hostStats.get()) + first_sample, last_sample, _ = sampling.host_samples.stats() + decStats.update(hoststats.produce(first_sample, last_sample)) if self._cif.irs: decStats['storageDomains'] = self._cif.irs.repoStats() diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py index c8ef6f6..6cd5e7f 100644 --- a/vdsm/clientIF.py +++ b/vdsm/clientIF.py @@ -94,7 +94,8 @@ self.gluster = None try: self.vmContainer = {} - self._hostStats = sampling.HostStatsThread(log=log) + self._hostStats = sampling.HostStatsThread( + log, sampling.host_samples) self._hostStats.start() self._startTime = time.time() self.lastRemoteAccess = 0 diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py index 401bae9..40cb3ee 100644 --- a/vdsm/virt/sampling.py +++ b/vdsm/virt/sampling.py @@ -37,7 +37,6 @@ from vdsm.config import config from .utils import ExpiringCache -from . import hoststats import caps @@ -524,19 +523,24 @@ return doms +HOST_STATS_AVERAGING_WINDOW = 5 + + +host_samples = SampleWindow(size=HOST_STATS_AVERAGING_WINDOW) + + class HostStatsThread(threading.Thread): """ A thread that periodically samples host statistics. """ - AVERAGING_WINDOW = 5 _CONNLOG = logging.getLogger('connectivity') - def __init__(self, log): + def __init__(self, log, samples): threading.Thread.__init__(self) self.daemon = True self._log = log self._stopEvent = threading.Event() - self._samples = SampleWindow(size=self.AVERAGING_WINDOW) + self._samples = samples self._pid = os.getpid() self._ncpus = max(os.sysconf('SC_NPROCESSORS_ONLN'), 1) @@ -569,10 +573,6 @@ except: if not self._stopEvent.isSet(): self._log.exception("Error while sampling stats") - - def get(self): - first_sample, last_sample, _ = self._samples.stats() - return hoststats.produce(first_sample, last_sample) def _getLinkSpeed(dev): -- To view, visit https://gerrit.ovirt.org/42036 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I80d919b0adb48a40d41f387543f14be265610405 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Francesco Romani <fromani(a)redhat.com>

4 24

Change in vdsm[master]: Renaming policycoreutils-python dependency for fedora 23 and...
by ybronhei＠redhat.com 04 Nov '15

04 Nov '15

Yaniv Bronhaim has uploaded a new change for review. Change subject: Renaming policycoreutils-python dependency for fedora 23 and above ...................................................................... Renaming policycoreutils-python dependency for fedora 23 and above The utilities of policycoreutils-python is renamed to policycoreutils-python-utils in fedora 23 Change-Id: I4d0b9fbfd77419cbf885c7663d799f19213592f5 Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1246626 Signed-off-by: Yaniv Bronhaim <ybronhei(a)redhat.com> --- M vdsm.spec.in 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/44019/1 diff --git a/vdsm.spec.in b/vdsm.spec.in index 4772416..aafd309 100644 --- a/vdsm.spec.in +++ b/vdsm.spec.in @@ -91,7 +91,11 @@ BuildRequires: m2crypto BuildRequires: mom >= 0.4.5 BuildRequires: openssl +%if 0%{?fedora} >= 23 +BuildRequires: policycoreutils-python-utils +%else BuildRequires: policycoreutils-python +%endif # fedora 23 BuildRequires: psmisc BuildRequires: python-cpopen >= 1.3 BuildRequires: python-inotify @@ -197,7 +201,11 @@ Requires: initscripts >= 9.42.2-1 Requires: device-mapper-multipath >= 0.4.9-56 Requires: e2fsprogs >= 1.41.14 +%if 0%{?fedora} >= 23 +Requires: policycoreutils-python-utils +%else Requires: policycoreutils-python +%endif # fedora 23 Requires: sed >= 4.2.1-10 Requires: ed Requires: lvm2 >= 2.02.98-15 -- To view, visit https://gerrit.ovirt.org/44019 To unsubscribe, visit https://gerrit.ovirt.org/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4d0b9fbfd77419cbf885c7663d799f19213592f5 Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Yaniv Bronhaim <ybronhei(a)redhat.com>

3 15

← Newer
1
...
5
6
7
8
9
10
11
...
92
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

vdsm-patches July 2015