Change in vdsm[master]: contrib: Simple jsonrpc client
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: contrib: Simple jsonrpc client
......................................................................
contrib: Simple jsonrpc client
This is a simple jsonrpc client for communicating with the jsonrpc
server from the command line.
Arguments
method one of the mehtods described in json schema
params optionl json object with message parameters
Examples
Calling method without arguements:
# jsonrpc Host.getVMList
{
"jsonrpc": "2.0",
"id": "0e043d83-294a-4d31-b1b6-6dc2f2747494",
"result": [
"b3f6fa00-b315-4ad4-8108-f73da817b5c5"
]
}
Calling method with arguements:
# jsonrpc VM.getStats '{"vmID": "b3f6fa00-b315-4ad4-8108-f73da817b5c5"}'
{
"jsonrpc": "2.0",
"id": "cefd25a3-6250-4123-8a56-d7047899e19e",
"result": [
{
"status": "Down",
"exitMessage": "Admin shut down from the engine",
"vmId": "b3f6fa00-b315-4ad4-8108-f73da817b5c5",
"exitReason": 6,
"timeOffset": "0",
"exitCode": 0
}
]
}
Requires stomp.py library:
https://pypi.python.org/pypi/stomp.py
Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
A contrib/jsonrpc
1 file changed, 104 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/81/35181/1
diff --git a/contrib/jsonrpc b/contrib/jsonrpc
new file mode 100755
index 0000000..3080193
--- /dev/null
+++ b/contrib/jsonrpc
@@ -0,0 +1,104 @@
+#!/usr/bin/python
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+"""
+jsonrpc-cli - Vdsm jsonrpc client
+"""
+
+import json
+import os
+import signal
+import sys
+import time
+import uuid
+import optparse
+
+import stomp
+
+# Copied from lib/vdsm/vdscli.py
+PKIDIR = '/etc/pki/vdsm'
+KEYFILE = os.path.join(PKIDIR, 'keys/vdsmkey.pem')
+CERTFILE = os.path.join(PKIDIR, 'certs/vdsmcert.pem')
+CACERT = os.path.join(PKIDIR, 'certs/cacert.pem')
+
+PORT = 54321
+DESTINATION = "/queue/_local/vdsm/requests"
+
+
+class Listener(stomp.ConnectionListener):
+
+ def on_error(self, headers, message):
+ print 'Error: %s' % message
+ terminate()
+
+ def on_message(self, headers, message):
+ msg = json.loads(message)
+ print json.dumps(msg, indent=4)
+ terminate()
+
+
+def main(args):
+ parser = option_parser()
+ options, args = parser.parse_args(args)
+ if not args:
+ parser.error("method required")
+
+ msg = {
+ "id": str(uuid.uuid4()),
+ "jsonrpc": "2.0",
+ "method": args[0]
+ }
+
+ if len(args) > 1:
+ msg["params"] = json.loads(args[1])
+
+ conn = stomp.Connection10(
+ host_and_ports=((options.host, PORT),),
+ use_ssl=True,
+ ssl_key_file=KEYFILE,
+ ssl_cert_file=CERTFILE,
+ ssl_ca_certs=CACERT)
+
+ conn.set_listener("", Listener())
+ conn.start()
+ conn.send(body=json.dumps(msg), destination=DESTINATION)
+
+ try:
+ signal.pause()
+ except KeyboardInterrupt:
+ pass
+
+ conn.disconnect()
+
+
+def option_parser():
+ parser = optparse.OptionParser(usage='%prog [options] method [params]')
+ parser.add_option("-a", "--host", dest="host",
+ help="host address (default localhost)")
+ parser.set_defaults(host="localhost")
+ return parser
+
+
+def terminate():
+ os.kill(os.getpid(), signal.SIGINT)
+
+
+if __name__ == "__main__":
+ main(sys.argv[1:])
--
To view, visit http://gerrit.ovirt.org/35181
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia6273eabf6f3601602659d1e4e748d8025ae8084
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years
Change in vdsm[master]: GuestIF Refactoring
by Vinzenz Feenstra
Vinzenz Feenstra has uploaded a new change for review.
Change subject: GuestIF Refactoring
......................................................................
GuestIF Refactoring
Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d
Signed-off-by: Vinzenz Feenstra <vfeenstr(a)redhat.com>
---
M vdsm/guestIF.py
1 file changed, 114 insertions(+), 76 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/18/24618/1
diff --git a/vdsm/guestIF.py b/vdsm/guestIF.py
index 229a55d..96ad68c 100644
--- a/vdsm/guestIF.py
+++ b/vdsm/guestIF.py
@@ -39,6 +39,115 @@
union(set(range(0x86, 0x9F + 1)))
+class UnknownMessageError(Exception):
+ def __init__(self, message, args):
+ Exception.__init__(self, 'Unknown or unsupported guest agent message '
+ '"%s" received with args "%s"' % (message,
+ str(args)))
+
+
+class MessageHandler(object):
+ def __init__(self, agent):
+ self.log = agent.log
+ self._agent = agent
+
+ def __call__(self, message, args):
+ handler = self.getattr(self, message.replace('-', '_'), None)
+ if handler:
+ handler(args)
+ else:
+ raise UnknownMessageError(message, args)
+
+ def applications(self, args):
+ self._agent.guestInfo['appsList'] = args['applications']
+
+ def fqdn(self, args):
+ self._agent.guestInfo['guestFQDN'] = args['fqdn']
+
+ def host_name(self, args):
+ self._agent.guestInfo['guestName'] = args['name']
+
+ def os_version(self, args):
+ self._agent.guestInfo['guestOs'] = args['version']
+
+ def session_lock(self, args):
+ self.agent.guestInfo['session'] = 'Locked'
+
+ def session_logoff(self, args):
+ self.agent.guestInfo['session'] = 'LoggedOff'
+
+ def session_logon(self, args):
+ self.agent.guestInfo['session'] = 'UserLoggedOn'
+
+ def session_unlock(self, args):
+ self.agent.guestInfo['session'] = 'Active'
+
+ def session_shutdown(self, args):
+ self.log.debug('Guest system shuts down')
+
+ def session_startup(self, args):
+ self.log.debug('Guest system started or restarted')
+
+ def uninstalled(self, args):
+ self.log.debug('Guest agent was uninstalled')
+ self._agent.guestInfo['appsList'] = []
+
+ def heartbeat(self, args):
+ self._agent.guestStatus = 'Up'
+ self._agent.guestInfo['memUsage'] = int(args['free-ram'])
+ # ovirt-guest-agent reports the following fields in 'memory-stat':
+ # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out',
+ # 'pageflt' and 'majflt'
+ if 'memory-stat' in args:
+ for (k, v) in args['memory-stat'].iteritems():
+ # Convert the value to string since 64-bit integer is not
+ # supported in XMLRPC
+ self._agent.guestInfo['memoryStats'][k] = str(v)
+
+ if 'apiVersion' in args:
+ # The guest agent supports API Versioning
+ self._agent._handleAPIVersion(args['apiVersion'])
+ elif self._agent.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO:
+ # Older versions of the guest agent (before the introduction
+ # of API versioning) do not report this field
+ # Disable the API if not already disabled (e.g. after
+ # downgrade of the guest agent)
+ self.log.debug("API versioning no longer reported by guest.")
+ self._agent.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO
+
+ def network_interfaces(self, args):
+ interfaces = []
+ old_ips = ''
+ for iface in args['interfaces']:
+ iface['inet'] = iface.get('inet', [])
+ iface['inet6'] = iface.get('inet6', [])
+ interfaces.append(iface)
+ # Provide the old information which includes
+ # only the IP addresses.
+ old_ips += ' '.join(iface['inet']) + ' '
+ self._agent.guestInfo['netIfaces'] = interfaces
+ self._agent.guestInfo['guestIPs'] = old_ips.strip()
+
+ def active_user(self, args):
+ currentUser = args['name']
+ if ((currentUser != self._agent.guestInfo['username']) and
+ not (currentUser == 'Unknown' and
+ self._agent.guestInfo['username'] == 'None')):
+ self._agent.guestInfo['username'] = currentUser
+ self._agent.guestInfo['lastLogin'] = time.time()
+ self.log.debug("username: %s", repr(self.guestInfo['username']))
+
+ def disks_usage(self, args):
+ disks = []
+ for disk in args['disks']:
+ # Converting to string because XML-RPC doesn't support 64-bit
+ # integers.
+ disk['total'] = str(disk['total'])
+ disk['used'] = str(disk['used'])
+ disks.append(disk)
+ self._agent.guestInfo['disksUsage'] = disks
+
+
def _filterXmlChars(u):
"""
The set of characters allowed in XML documents is described in
@@ -109,6 +218,7 @@
def __init__(self, socketName, channelListener, log, user='Unknown',
ips='', connect=True):
+ self.handler = MessageHandler(self)
self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO
self.log = log
self._socketName = socketName
@@ -223,82 +333,10 @@
self.log.log(logging.TRACE, "Guest's message %s: %s", message, args)
if self.guestStatus is None:
self.guestStatus = 'Up'
- if message == 'heartbeat':
- self.guestStatus = 'Up'
- self.guestInfo['memUsage'] = int(args['free-ram'])
- # ovirt-guest-agent reports the following fields in 'memory-stat':
- # 'mem_total', 'mem_free', 'mem_unused', 'swap_in', 'swap_out',
- # 'pageflt' and 'majflt'
- if 'memory-stat' in args:
- for (k, v) in args['memory-stat'].iteritems():
- # Convert the value to string since 64-bit integer is not
- # supported in XMLRPC
- self.guestInfo['memoryStats'][k] = str(v)
-
- if 'apiVersion' in args:
- # The guest agent supports API Versioning
- self._handleAPIVersion(args['apiVersion'])
- elif self.effectiveApiVersion != _IMPLICIT_API_VERSION_ZERO:
- # Older versions of the guest agent (before the introduction
- # of API versioning) do not report this field
- # Disable the API if not already disabled (e.g. after
- # downgrade of the guest agent)
- self.log.debug("API versioning no longer reported by guest.")
- self.effectiveApiVersion = _IMPLICIT_API_VERSION_ZERO
- elif message == 'host-name':
- self.guestInfo['guestName'] = args['name']
- elif message == 'os-version':
- self.guestInfo['guestOs'] = args['version']
- elif message == 'network-interfaces':
- interfaces = []
- old_ips = ''
- for iface in args['interfaces']:
- iface['inet'] = iface.get('inet', [])
- iface['inet6'] = iface.get('inet6', [])
- interfaces.append(iface)
- # Provide the old information which includes
- # only the IP addresses.
- old_ips += ' '.join(iface['inet']) + ' '
- self.guestInfo['netIfaces'] = interfaces
- self.guestInfo['guestIPs'] = old_ips.strip()
- elif message == 'applications':
- self.guestInfo['appsList'] = args['applications']
- elif message == 'active-user':
- currentUser = args['name']
- if ((currentUser != self.guestInfo['username']) and
- not (currentUser == 'Unknown' and
- self.guestInfo['username'] == 'None')):
- self.guestInfo['username'] = currentUser
- self.guestInfo['lastLogin'] = time.time()
- self.log.debug("username: %s", repr(self.guestInfo['username']))
- elif message == 'session-logon':
- self.guestInfo['session'] = "UserLoggedOn"
- elif message == 'session-lock':
- self.guestInfo['session'] = "Locked"
- elif message == 'session-unlock':
- self.guestInfo['session'] = "Active"
- elif message == 'session-logoff':
- self.guestInfo['session'] = "LoggedOff"
- elif message == 'uninstalled':
- self.log.debug("RHEV agent was uninstalled.")
- self.guestInfo['appsList'] = []
- elif message == 'session-startup':
- self.log.debug("Guest system is started or restarted.")
- elif message == 'fqdn':
- self.guestInfo['guestFQDN'] = args['fqdn']
- elif message == 'session-shutdown':
- self.log.debug("Guest system shuts down.")
- elif message == 'disks-usage':
- disks = []
- for disk in args['disks']:
- # Converting to string because XML-RPC doesn't support 64-bit
- # integers.
- disk['total'] = str(disk['total'])
- disk['used'] = str(disk['used'])
- disks.append(disk)
- self.guestInfo['disksUsage'] = disks
- else:
- self.log.error('Unknown message type %s', message)
+ try:
+ self.handler(message, args)
+ except UnknownMessageError as e:
+ self.log.error(e)
def stop(self):
self._stopped = True
--
To view, visit http://gerrit.ovirt.org/24618
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib357d770a26ef1dc80b89a32bf6808551a7d622d
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Vinzenz Feenstra <vfeenstr(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: libvirtconnection: Replace assert with AssertionError
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: libvirtconnection: Replace assert with AssertionError
......................................................................
libvirtconnection: Replace assert with AssertionError
The code wrongly assumed that assert always exists. When running in
optimized mode, the check would be skipped, and instead of getting an
AssertionError, which is the expected error for programmer error
(starting the eventloop twice), we could get a confusing
RuntimeException or RuntimeError from Thread.start (depending on Python
version).
RuntimeError misused in the standard library for all kinds of errors
that do not have builtin errors. It is particularry bad option when used
for usage error.
Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M lib/vdsm/libvirtconnection.py
1 file changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/64/34364/1
diff --git a/lib/vdsm/libvirtconnection.py b/lib/vdsm/libvirtconnection.py
index 5430c82..009f8b7 100644
--- a/lib/vdsm/libvirtconnection.py
+++ b/lib/vdsm/libvirtconnection.py
@@ -37,7 +37,8 @@
self.__thread = None
def start(self):
- assert not self.run
+ if self.run:
+ raise AssertionError("EventLoop is running")
self.__thread = threading.Thread(target=self.__run,
name="libvirtEventLoop")
self.__thread.setDaemon(True)
--
To view, visit http://gerrit.ovirt.org/34364
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icf1564f81f4c1fbf77ccaff6d93c047a02d946da
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: cache: Add caching decorator with invalidation
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: cache: Add caching decorator with invalidation
......................................................................
cache: Add caching decorator with invalidation
The new cache.memoized extends utils.memoized, adding invalidation
support.
Features added:
- An optional "validate" argument. This is a callable invoked each time
the memoized function is called. When the callable returns False, the
cache is invalidated.
- Memoized functions have an "invalidate" method, used to invalidate the
cache during testing.
- file_validator - invalidates the cache when a file changes.
Example usage:
from vdsm.cache import memoized, file_validator
@memoized(file_validator('/bigfile'))
def parse_bigfile():
# Expensive code processing '/bigfile' contents
Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M lib/vdsm/Makefile.am
A lib/vdsm/cache.py
M tests/Makefile.am
A tests/cacheTests.py
4 files changed, 366 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/09/34709/1
diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am
index b862e71..6f0040d 100644
--- a/lib/vdsm/Makefile.am
+++ b/lib/vdsm/Makefile.am
@@ -23,6 +23,7 @@
dist_vdsmpylib_PYTHON = \
__init__.py \
+ cache.py \
compat.py \
define.py \
exception.py \
diff --git a/lib/vdsm/cache.py b/lib/vdsm/cache.py
new file mode 100644
index 0000000..9806e40
--- /dev/null
+++ b/lib/vdsm/cache.py
@@ -0,0 +1,98 @@
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+import errno
+import os
+import functools
+
+
+def memoized(validate=None):
+ """
+ Return a caching decorator supporting invalidation.
+
+ The decorator accepts an optional validate callable, called each time the
+ memoized function is called. If the validate callable return True, the
+ memoized function will use the cache. If the validate callable return
+ False, the memoized cache is cleared.
+
+ The memoized function may accept multiple positional arguments. The
+ cache store the result for each combination of arguments. Functions with
+ kwargs are not supported.
+
+ Memoized functions have an "invalidate" method, used to invalidate the
+ memoized cache during testing.
+
+ To invalidate the cache when a file changes, use the file_validator from
+ this module.
+
+ Example usage:
+
+ from vdsm.cache import memoized, file_validator
+
+ @memoized(file_validator('/bigfile'))
+ def parse_bigfile():
+ # Expensive code processing '/bigfile' contents
+
+ """
+ def decorator(f):
+ cache = {}
+
+ @functools.wraps(f)
+ def wrapper(*args):
+ if validate is not None and not validate():
+ cache.clear()
+ try:
+ value = cache[args]
+ except KeyError:
+ value = cache[args] = f(*args)
+ return value
+
+ wrapper.invalidate = cache.clear
+ return wrapper
+
+ return decorator
+
+
+class file_validator(object):
+ """
+ I'm a validator returning False when a file has changed since the last
+ validation.
+ """
+
+ UNKNOWN = 0
+ MISSING = 1
+
+ def __init__(self, path):
+ self.path = path
+ self.stats = self.UNKNOWN
+
+ def __call__(self):
+ try:
+ stats = os.stat(self.path)
+ except OSError as e:
+ if e.errno != errno.ENOENT:
+ raise
+ stats = self.MISSING
+ else:
+ stats = stats.st_ino, stats.st_size, stats.st_mtime
+ if stats != self.stats:
+ self.stats = stats
+ return False
+ return True
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 36a1cdd..6fa7e64 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -26,6 +26,7 @@
alignmentScanTests.py \
blocksdTests.py \
bridgeTests.py \
+ cacheTests.py \
cPopenTests.py \
capsTests.py \
clientifTests.py \
diff --git a/tests/cacheTests.py b/tests/cacheTests.py
new file mode 100644
index 0000000..8927b39
--- /dev/null
+++ b/tests/cacheTests.py
@@ -0,0 +1,266 @@
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+import os
+from vdsm.cache import memoized
+from vdsm.cache import file_validator
+from testlib import VdsmTestCase
+from testlib import namedTemporaryDir
+
+
+class Validator(object):
+ """ I'm a callable returning a boolean value (self.valid) """
+
+ def __init__(self):
+ self.valid = True
+ self.count = 0
+
+ def __call__(self):
+ self.count += 1
+ return self.valid
+
+
+class Accessor(object):
+ """ I'm recording how many times a dict was accessed. """
+
+ def __init__(self, d):
+ self.d = d
+ self.count = 0
+
+ def get(self, key):
+ self.count += 1
+ return self.d[key]
+
+
+class MemoizedTests(VdsmTestCase):
+
+ def setUp(self):
+ self.values = {'a': 0, 'b': 10, ('a',): 20, ('a', 'b'): 30}
+
+ def test_no_args(self):
+ accessor = Accessor(self.values)
+
+ @memoized()
+ def func(key):
+ return accessor.get(key)
+
+ # Fill the cache
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 2)
+
+ # Values served now from the cache
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 2)
+
+ def test_validation(self):
+ accessor = Accessor(self.values)
+ validator = Validator()
+
+ @memoized(validator)
+ def func(key):
+ return accessor.get(key)
+
+ # Fill the cache
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 2)
+ self.assertEqual(validator.count, 2)
+
+ # Values served now from the cache
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 2)
+ self.assertEqual(validator.count, 4)
+
+ # Values has changed
+ self.values['a'] += 1
+ self.values['b'] += 1
+
+ # Next call should clear the cache
+ validator.valid = False
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(accessor.count, 3)
+ self.assertEqual(validator.count, 5)
+
+ # Next call should add next value to cache
+ validator.valid = True
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 4)
+ self.assertEqual(validator.count, 6)
+
+ # Values served now from the cache
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(func('b'), self.values['b'])
+ self.assertEqual(accessor.count, 4)
+ self.assertEqual(validator.count, 8)
+
+ def test_raise_errors_in_memoized_func(self):
+ accessor = Accessor(self.values)
+ validator = Validator()
+
+ @memoized(validator)
+ def func(key):
+ return accessor.get(key)
+
+ # First run should fail, second shold fill the cache
+ self.assertRaises(KeyError, func, 'no such key')
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(accessor.count, 2)
+ self.assertEqual(validator.count, 2)
+
+ def test_multiple_args(self):
+ accessor = Accessor(self.values)
+
+ @memoized()
+ def func(*args):
+ return accessor.get(args)
+
+ # Fill the cache
+ self.assertEqual(func('a'), self.values[('a',)])
+ self.assertEqual(func('a', 'b'), self.values[('a', 'b')])
+ self.assertEqual(accessor.count, 2)
+
+ # Values served now from the cache
+ self.assertEqual(func('a'), self.values[('a',)])
+ self.assertEqual(func('a', 'b'), self.values[('a', 'b')])
+ self.assertEqual(accessor.count, 2)
+
+ def test_kwargs_not_supported(self):
+ @memoized()
+ def func(a=None, b=None):
+ pass
+ self.assertRaises(TypeError, func, a=1, b=2)
+
+ def test_invalidate(self):
+ accessor = Accessor(self.values)
+
+ @memoized()
+ def func(key):
+ return accessor.get(key)
+
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(accessor.count, 1)
+
+ func.invalidate()
+
+ self.assertEqual(func('a'), self.values['a'])
+ self.assertEqual(accessor.count, 2)
+
+
+class FileValidatorTests(VdsmTestCase):
+
+ def test_no_file(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+
+ # Must be False so memoise call the decorated function
+ self.assertEqual(validator(), False)
+
+ # Since file state did not change, must remain True
+ self.assertEqual(validator(), True)
+
+ def test_file_created(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ with open(path, 'w') as f:
+ f.write('data')
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ def test_file_removed(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+
+ with open(path, 'w') as f:
+ f.write('data')
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ os.unlink(path)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ def test_size_changed(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+ data = 'old data'
+ with open(path, 'w') as f:
+ f.write(data)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ with open(path, 'w') as f:
+ f.write(data + ' new data')
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ def test_mtime_changed(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+ data = 'old data'
+ with open(path, 'w') as f:
+ f.write(data)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ # Fake timestamp change, as timestamp resolution may not be good
+ # enough when comparing changes during the test.
+ atime = mtime = os.path.getmtime(path) + 1
+ os.utime(path, (atime, mtime))
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ def test_ino_changed(self):
+ with namedTemporaryDir() as tempdir:
+ path = os.path.join(tempdir, 'data')
+ validator = file_validator(path)
+ data = 'old data'
+ with open(path, 'w') as f:
+ f.write(data)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
+
+ tmp = path + '.tmp'
+ with open(tmp, 'w') as f:
+ f.write(data)
+ os.rename(tmp, path)
+
+ self.assertEqual(validator(), False)
+ self.assertEqual(validator(), True)
--
To view, visit http://gerrit.ovirt.org/34709
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6dd8fb29d94286e3e3a3e29b8218501cbdc5c018
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: fencing: Make getHostLeaseStatus API public
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: fencing: Make getHostLeaseStatus API public
......................................................................
fencing: Make getHostLeaseStatus API public
Getting host lease status will allow engine to make smarter decisions
when a host is non-responsive by using a proxy host to get the
non-responsive host status.
See http://pastebin.com/KqqeAdSu for example output from this API.
Change-Id: I415c1fee6256bf8d4e03ee542cc58e193162e9b8
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M client/vdsClient.py
M vdsm/API.py
M vdsm/rpc/BindingXMLRPC.py
M vdsm/rpc/Bridge.py
M vdsm/rpc/vdsmapi-schema.json
5 files changed, 62 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/57/29157/1
diff --git a/client/vdsClient.py b/client/vdsClient.py
index 2c09b28..aea1503 100644
--- a/client/vdsClient.py
+++ b/client/vdsClient.py
@@ -1760,6 +1760,18 @@
status = self.s.stopMonitoringDomain(sdUUID)
return status['status']['code'], status['status']['message']
+ def getHostLeaseStatus(self, args):
+ domains = {}
+ for pair in args:
+ sdUUID, hostId = pair.split('=', 1)
+ domains[sdUUID] = int(hostId)
+ response = self.s.getHostLeaseStatus(domains)
+ if response['status']['code']:
+ print "Cannot get host storage liveliness"
+ return response['status']['code'], response['status']['message']
+ pp.pprint(response['domains'])
+ return 0, ''
+
def snapshot(self, args):
vmUUID, sdUUID, imgUUID, baseVolUUID, volUUID = args
@@ -2579,6 +2591,11 @@
('<sdUUID>',
'Stop monitoring SD: sdUUID'
)),
+ 'getHostLeaseStatus': (serv.getHostLeaseStatus,
+ ('<sdUUID>=<hostId> [<sdUUID>=<hostId>] ...',
+ 'Returns host lease status for hostId on '
+ 'each domain.'
+ )),
'snapshot': (serv.snapshot,
('<vmId> <sdUUID> <imgUUID> <baseVolUUID> <volUUID>',
'Take a live snapshot'
diff --git a/vdsm/API.py b/vdsm/API.py
index e739294..0b44459 100644
--- a/vdsm/API.py
+++ b/vdsm/API.py
@@ -1497,6 +1497,9 @@
def stopMonitoringDomain(self, sdUUID):
return self._irs.stopMonitoringDomain(sdUUID)
+ def getHostLeaseStatus(self, domains):
+ return self._irs.getHostLeaseStatus(domains)
+
def getLVMVolumeGroups(self, storageType=None):
return self._irs.getVGList(storageType)
diff --git a/vdsm/rpc/BindingXMLRPC.py b/vdsm/rpc/BindingXMLRPC.py
index c1c7490..a06a3b4 100644
--- a/vdsm/rpc/BindingXMLRPC.py
+++ b/vdsm/rpc/BindingXMLRPC.py
@@ -917,6 +917,10 @@
api = API.Global()
return api.stopMonitoringDomain(sdUUID)
+ def getHostLeaseStatus(self, domains, options=None):
+ api = API.Global()
+ return api.getHostLeaseStatus(domains)
+
def vgsGetList(self, storageType=None, options=None):
api = API.Global()
return api.getLVMVolumeGroups(storageType)
@@ -1070,6 +1074,7 @@
(self.storageRepoGetStats, 'repoStats'),
(self.startMonitoringDomain, 'startMonitoringDomain'),
(self.stopMonitoringDomain, 'stopMonitoringDomain'),
+ (self.getHostLeaseStatus, 'getHostLeaseStatus'),
(self.vgsGetList, 'getVGList'),
(self.devicesGetList, 'getDeviceList'),
(self.devicesGetVisibility, 'getDevicesVisibility'),
diff --git a/vdsm/rpc/Bridge.py b/vdsm/rpc/Bridge.py
index 7e898de..ba700d1 100644
--- a/vdsm/rpc/Bridge.py
+++ b/vdsm/rpc/Bridge.py
@@ -349,6 +349,7 @@
'Host_getStorageRepoStats': {'ret': Host_getStorageRepoStats_Ret},
'Host_startMonitoringDomain': {},
'Host_stopMonitoringDomain': {},
+ 'Host_getHostLeaseStatus': {'ret': 'domains'},
'Host_getVMList': {'call': Host_getVMList_Call, 'ret': Host_getVMList_Ret},
'Host_getVMFullList': {'call': Host_getVMFullList_Call, 'ret': 'vmList'},
'Host_getAllVmStats': {'ret': 'statsList'},
diff --git a/vdsm/rpc/vdsmapi-schema.json b/vdsm/rpc/vdsmapi-schema.json
index 0c8a6f6..7617185 100644
--- a/vdsm/rpc/vdsmapi-schema.json
+++ b/vdsm/rpc/vdsmapi-schema.json
@@ -2052,6 +2052,42 @@
'returns': ''}
##
+# @HostIdMap:
+#
+# A mapping of hostId indexed by domain UUID.
+#
+# Since: 4.15.0
+##
+{'map': 'HostIdMap',
+ 'key': 'UUID', 'value': 'int'}
+
+##
+# @HostLeaseStatusMap:
+#
+# A mapping of status codes indexed by domain UUID.
+#
+# Since: 4.15.0
+##
+{'map': 'HostLeaseStatusMap',
+ 'key': 'UUID', 'value': 'str'}
+
+##
+# @Host.getHostLeaseStatus:
+#
+# Returns host status for for specified domains
+#
+# @domains: A mapping of hostId indexed by domain UUID
+#
+# Returns:
+# Host status code for each domain
+#
+# Since: 4.15.0
+##
+{'command': {'class': 'Host', 'name': 'getHostLeaseStatus'},
+ 'data': {'domains': 'HostIdMap'}
+ 'returns': {'domains': 'HostLeaseStatusMap'}}
+
+##
# @VmStatus:
#
# An enumeration of possible virtual machine statuses.
--
To view, visit http://gerrit.ovirt.org/29157
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I415c1fee6256bf8d4e03ee542cc58e193162e9b8
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: cache: Replace utils.memoized with cache.memoized()
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: cache: Replace utils.memoized with cache.memoized()
......................................................................
cache: Replace utils.memoized with cache.memoized()
Repalce users of utils.memoized with cache.memoized() and remove
utils.memoized. No behavior change is expected.
Change-Id: I12e2f2919cf92ff7d0758d70e2ed40523d66174f
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M lib/vdsm/netinfo.py
M lib/vdsm/utils.py
M tests/vmApiTests.py
M vdsm/caps.py
M vdsm/dmidecodeUtil.py
M vdsm/ppc64HardwareInfo.py
M vdsm/supervdsmServer
M vdsm/virt/sampling.py
8 files changed, 30 insertions(+), 51 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/10/34710/1
diff --git a/lib/vdsm/netinfo.py b/lib/vdsm/netinfo.py
index d2ef90b..8c56863 100644
--- a/lib/vdsm/netinfo.py
+++ b/lib/vdsm/netinfo.py
@@ -43,7 +43,7 @@
from .ipwrapper import routeShowGateways
from . import libvirtconnection
from .netconfpersistence import RunningConfig
-from .utils import memoized
+from .cache import memoized
from .netlink import link as nl_link
from .netlink import addr as nl_addr
from .netlink import route as nl_route
@@ -395,7 +395,7 @@
return paddr
-@memoized
+@memoized()
def _getAllDefaultBondingOptions():
"""
Return default options per mode, in a dictionary of dictionaries. All keys
@@ -405,7 +405,7 @@
return json.loads(defaults.read())
-@memoized
+@memoized()
def _getDefaultBondingOptions(mode=None):
"""
Return default options for the given mode. If it is None, return options
diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py
index feb4079..131145c 100644
--- a/lib/vdsm/utils.py
+++ b/lib/vdsm/utils.py
@@ -51,6 +51,7 @@
import vdsm.infra.zombiereaper as zombiereaper
from cpopen import CPopen
+from . import cache
from . import constants
# Buffsize is 1K because I tested it on some use cases and 1K was fastest. If
@@ -828,32 +829,6 @@
fcntl.fcntl(fd, fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)
-class memoized(object):
- """
- Decorator that caches a function's return value each time it is called.
- If called later with the same arguments, the cached value is returned, and
- not re-evaluated. There is no support for uncachable arguments.
-
- Adaptation from http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize
- """
- def __init__(self, func):
- self.func = func
- self.cache = {}
- functools.update_wrapper(self, func)
-
- def __call__(self, *args):
- try:
- return self.cache[args]
- except KeyError:
- value = self.func(*args)
- self.cache[args] = value
- return value
-
- def __get__(self, obj, objtype):
- """Support instance methods."""
- return functools.partial(self.__call__, obj)
-
-
def validateMinimalKeySet(dictionary, reqParams):
if not all(key in dictionary for key in reqParams):
raise ValueError
@@ -993,7 +968,7 @@
sys.exit(-3)
-@memoized
+(a)cache.memoized()
def isOvirtNode():
return (os.path.exists('/etc/rhev-hypervisor-release') or
bool(glob.glob('/etc/ovirt-node-*-release')))
diff --git a/tests/vmApiTests.py b/tests/vmApiTests.py
index eb89fb1..4f940cf 100644
--- a/tests/vmApiTests.py
+++ b/tests/vmApiTests.py
@@ -25,14 +25,14 @@
from virt import vmexitreason
from vdsm import define
from testlib import VdsmTestCase as TestCaseBase
-from vdsm import utils
+from vdsm import cache
from rpc import vdsmapi
import vmfakelib as fake
class TestSchemaCompliancyBase(TestCaseBase):
- @utils.memoized
+ @cache.memoized()
def _getAPI(self):
testPath = os.path.realpath(__file__)
dirName = os.path.split(testPath)[0]
diff --git a/vdsm/caps.py b/vdsm/caps.py
index dba9b5b..6e261d8 100644
--- a/vdsm/caps.py
+++ b/vdsm/caps.py
@@ -40,6 +40,7 @@
import dsaversion
from vdsm import netinfo
import hooks
+from vdsm import cache
from vdsm import utils
import storage.hba
from network.configurators import qos
@@ -218,12 +219,12 @@
ENABLED = 1
-(a)utils.memoized
+(a)cache.memoized()
def _getCapsXMLStr():
return libvirtconnection.get().getCapabilities()
-(a)utils.memoized
+(a)cache.memoized()
def _getCpuTopology(capabilities):
if capabilities is None:
capabilities = _getCapsXMLStr()
@@ -277,7 +278,7 @@
return None
-(a)utils.memoized
+(a)cache.memoized()
def _getLiveSnapshotSupport(arch, capabilities=None):
if capabilities is None:
capabilities = _getCapsXMLStr()
@@ -294,7 +295,7 @@
return None
-(a)utils.memoized
+(a)cache.memoized()
def getLiveMergeSupport():
"""
Determine if libvirt provides the necessary features to enable live merge.
@@ -318,7 +319,7 @@
return True
-(a)utils.memoized
+(a)cache.memoized()
def getNumaTopology():
capabilities = _getCapsXMLStr()
caps = minidom.parseString(capabilities)
@@ -369,7 +370,7 @@
return memDict
-(a)utils.memoized
+(a)cache.memoized()
def getNumaNodeDistance():
nodeDistance = {}
retcode, out, err = utils.execCmd(['numactl', '--hardware'])
@@ -385,7 +386,7 @@
return nodeDistance
-(a)utils.memoized
+(a)cache.memoized()
def getAutoNumaBalancingInfo():
retcode, out, err = utils.execCmd(['sysctl', '-n', '-e',
'kernel.numa_balancing'])
@@ -399,7 +400,7 @@
return AutoNumaBalancingStatus.UNKNOWN
-(a)utils.memoized
+(a)cache.memoized()
def _getEmulatedMachines(arch, capabilities=None):
if capabilities is None:
capabilities = _getCapsXMLStr()
@@ -461,7 +462,7 @@
return allModels
-(a)utils.memoized
+(a)cache.memoized()
def _getCompatibleCpuModels():
c = libvirtconnection.get()
allModels = _getAllCpuModels()
@@ -532,7 +533,7 @@
return kdumpStatus
-(a)utils.memoized
+(a)cache.memoized()
def getos():
if os.path.exists('/etc/rhev-hypervisor-release'):
return OSName.RHEVH
@@ -550,7 +551,7 @@
return OSName.UNKNOWN
-(a)utils.memoized
+(a)cache.memoized()
def osversion():
version = release = ''
@@ -700,7 +701,7 @@
return info
-(a)utils.memoized
+(a)cache.memoized()
def _getVersionInfo():
if not hasattr(libvirt, 'VIR_MIGRATE_ABORT_ON_ERROR'):
return _dropVersion('3.4',
diff --git a/vdsm/dmidecodeUtil.py b/vdsm/dmidecodeUtil.py
index add2526..f0c72fa 100644
--- a/vdsm/dmidecodeUtil.py
+++ b/vdsm/dmidecodeUtil.py
@@ -18,7 +18,7 @@
# Refer to the README and COPYING files for full details of the license
#
-from vdsm import utils
+from vdsm import cache
# This function gets dict and returns new dict that includes only string
@@ -35,7 +35,7 @@
return ret
-(a)utils.memoized
+(a)cache.memoized()
def getAllDmidecodeInfo():
import dmidecode
@@ -45,7 +45,7 @@
return myLeafDict
-(a)utils.memoized
+(a)cache.memoized()
def getHardwareInfoStructure():
dmiInfo = getAllDmidecodeInfo()
sysStruct = {}
diff --git a/vdsm/ppc64HardwareInfo.py b/vdsm/ppc64HardwareInfo.py
index 23f7388..1a89639 100644
--- a/vdsm/ppc64HardwareInfo.py
+++ b/vdsm/ppc64HardwareInfo.py
@@ -16,6 +16,7 @@
# Refer to the README and COPYING files for full details of the license
#
+from vdsm import cache
from vdsm import utils
import os
@@ -31,7 +32,7 @@
return 'unavailable'
-(a)utils.memoized
+(a)cache.memoized()
def getHardwareInfoStructure():
infoStructure = {'systemSerialNumber': 'unavailable',
'systemFamily': 'unavailable',
@@ -59,7 +60,7 @@
return infoStructure
-(a)utils.memoized
+(a)cache.memoized()
def getCpuTopology(capabilities):
topology = {}
diff --git a/vdsm/supervdsmServer b/vdsm/supervdsmServer
index c6c139b..aba32fb 100755
--- a/vdsm/supervdsmServer
+++ b/vdsm/supervdsmServer
@@ -51,6 +51,7 @@
except ImportError:
_glusterEnabled = False
+from vdsm import cache
from vdsm import utils
from vdsm.tool import restore_nets
from parted_utils import getDevicePartedInfo as _getDevicePartedInfo
@@ -378,7 +379,7 @@
raise OSError(errno.EINVAL, "Could not reload-rules for device "
"%s" % guid)
- @utils.memoized
+ @cache.memoized()
def __udevVersion(self):
cmd = [EXT_UDEVADM, '--version']
rc, out, err = utils.execCmd(cmd)
diff --git a/vdsm/virt/sampling.py b/vdsm/virt/sampling.py
index 6bb4211..b1c3db6 100644
--- a/vdsm/virt/sampling.py
+++ b/vdsm/virt/sampling.py
@@ -30,6 +30,7 @@
import errno
import re
+from vdsm import cache
from vdsm import utils
from vdsm import netinfo
from vdsm.ipwrapper import getLinks
@@ -524,7 +525,7 @@
if not self._stopEvent.isSet():
self._log.error("Error while sampling stats", exc_info=True)
- @utils.memoized
+ @cache.memoized()
def _boot_time(self):
# Try to get boot time only once, if N/A just log the error and never
# include it in the response.
--
To view, visit http://gerrit.ovirt.org/34710
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I12e2f2919cf92ff7d0758d70e2ed40523d66174f
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: qemuimg: Memoize _supports_qcow2_compat
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: qemuimg: Memoize _supports_qcow2_compat
......................................................................
qemuimg: Memoize _supports_qcow2_compat
We used to run qemu-img twice when creating or converting qcow2 images.
The first run check if qemu-img supports the qcow2 "comapt" option, and
the second run uses the result to format the qemu-img command.
Now we run qemu-img once for "create" and "convert" to learn about its
capabilities, and use the cached value on the next runs. If qemu-img
executable is modified, we drop the cache, in case a new version was
installed with different capabilities.
Change-Id: Ic63f5e8c06993df8e4066bf7ac2dabfb4b4bdbfb
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M lib/vdsm/qemuimg.py
M tests/qemuimgTests.py
2 files changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/11/34711/1
diff --git a/lib/vdsm/qemuimg.py b/lib/vdsm/qemuimg.py
index cf428b2..f295acc 100644
--- a/lib/vdsm/qemuimg.py
+++ b/lib/vdsm/qemuimg.py
@@ -23,6 +23,7 @@
import signal
from . import utils
+from . import cache
_qemuimg = utils.CommandPath("qemu-img",
"/usr/bin/qemu-img",) # Fedora, EL6
@@ -221,6 +222,7 @@
raise QImgError(rc, out, err)
+(a)cache.memoized(cache.file_validator(_qemuimg.cmd))
def _supports_qcow2_compat(command):
"""
qemu-img "create" and "convert" commands support a "compat" option in
diff --git a/tests/qemuimgTests.py b/tests/qemuimgTests.py
index 813a497..a642374 100644
--- a/tests/qemuimgTests.py
+++ b/tests/qemuimgTests.py
@@ -118,6 +118,9 @@
class CreateTests(TestCaseBase):
+ def setUp(self):
+ qemuimg._supports_qcow2_compat.invalidate()
+
def test_no_format(self):
def create(cmd, **kw):
expected = [QEMU_IMG, 'create', 'image']
@@ -161,6 +164,9 @@
class ConvertTests(TestCaseBase):
+ def setUp(self):
+ qemuimg._supports_qcow2_compat.invalidate()
+
def test_no_format(self):
def convert(cmd, **kw):
expected = [QEMU_IMG, 'convert', '-t', 'none', 'src', 'dst']
--
To view, visit http://gerrit.ovirt.org/34711
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic63f5e8c06993df8e4066bf7ac2dabfb4b4bdbfb
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 2 months
Change in vdsm[master]: vdsm: add eventfd and EventFile synchronization
by Federico Simoncelli
Federico Simoncelli has uploaded a new change for review.
Change subject: vdsm: add eventfd and EventFile synchronization
......................................................................
vdsm: add eventfd and EventFile synchronization
Change-Id: I0d237f13c42b1f4505c90d30c6d3c3ecbd1e9fa7
Signed-off-by: Federico Simoncelli <fsimonce(a)redhat.com>
---
M lib/vdsm/Makefile.am
A lib/vdsm/eventfd.py
M tests/Makefile.am
A tests/eventfdTests.py
4 files changed, 251 insertions(+), 0 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/87/33687/1
diff --git a/lib/vdsm/Makefile.am b/lib/vdsm/Makefile.am
index 4bebf28..e712cad 100644
--- a/lib/vdsm/Makefile.am
+++ b/lib/vdsm/Makefile.am
@@ -25,6 +25,7 @@
__init__.py \
compat.py \
define.py \
+ eventfd.py \
exception.py \
ipwrapper.py \
libvirtconnection.py \
diff --git a/lib/vdsm/eventfd.py b/lib/vdsm/eventfd.py
new file mode 100644
index 0000000..b2a7084
--- /dev/null
+++ b/lib/vdsm/eventfd.py
@@ -0,0 +1,140 @@
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+"""\
+This module provides the support for eventfd(2).
+
+More information about eventfd and usage examples can be found in the
+eventfd(2) man page.
+
+The EventFile class provides a single synchronization object exposing
+the python Event interface and associated eventfds.
+
+The eventfd() context manager returns a file descriptor that can be
+used to provide the event notice to select, poll and epoll, e.g.
+
+ import os
+ import sys
+ import select
+ import threading
+ import time
+ from vdsm.eventfd import EventFile, DATASIZE
+
+ e = EventFile()
+ p = select.epoll()
+
+ threading.Timer(5, e.set).start()
+
+ with e.eventfd() as efd:
+ p.register(efd, select.EPOLLIN)
+ p.register(sys.stdin.fileno(), select.EPOLLIN)
+
+ print "Echoing lines until event is received"
+ event_received = False
+
+ while not event_received:
+ for fileno, event in p.poll():
+ if not event & select.EPOLLIN:
+ continue
+
+ if fileno == efd:
+ os.read(efd, DATASIZE)
+ event_received = True
+ elif fileno == sys.stdin.fileno():
+ print os.read(sys.stdin.fileno(), 1024),
+
+ print "Event received!"
+
+
+The Event set() semantic is preserved in the eventfd context manager:
+if the event is set then the eventfd already contains the notification.
+This is both to maintain the semantic and to avoid possible races as:
+
+ if not e.is_set():
+ with e.eventfd() as efd:
+ ...
+"""
+
+import os
+import ctypes
+import threading
+
+from contextlib import contextmanager
+
+_libc = ctypes.CDLL('libc.so.6', use_errno=True)
+
+EFD_NONBLOCK = os.O_NONBLOCK
+EFD_CLOEXEC = 02000000 # os.O_CLOEXEC in python 3.3
+EFD_SEMAPHORE = 00000001
+
+DATASIZE = ctypes.sizeof(ctypes.c_ulonglong)
+
+
+def eventfd(initval, flags):
+ return _libc.eventfd(initval, flags)
+
+
+class EventFile(object):
+ def __init__(self, event=None):
+ self.__lock = threading.Lock()
+ self.__fds = set()
+ self.__event = event or threading.Event()
+
+ @staticmethod
+ def __fire_event(fd):
+ os.write(fd, ctypes.c_ulonglong(1))
+
+ def open_eventfd(self):
+ with self.__lock:
+ fd = eventfd(0, 0)
+
+ self.__fds.add(fd)
+
+ if self.__event.is_set():
+ self.__fire_event(fd)
+
+ return fd
+
+ @contextmanager
+ def eventfd(self):
+ fd = self.open_eventfd()
+
+ yield fd
+
+ with self.__lock:
+ self.__fds.remove(fd)
+ os.close(fd)
+
+ def isSet(self):
+ return self.__event.isSet()
+
+ is_set = isSet
+
+ def set(self):
+ with self.__lock:
+ self.__event.set()
+ for fd in self.__fds:
+ self.__fire_event(fd)
+
+ def clear(self):
+ self.__event.clear()
+
+ def wait(self, timeout=None, balancing=True):
+ self.__event.wait(timeout)
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 449d7b1..120712e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -31,6 +31,7 @@
clientifTests.py \
configNetworkTests.py \
domainMonitorTests.py \
+ eventfdTests.py \
fileVolumeTests.py \
fileUtilTests.py \
fuserTests.py \
diff --git a/tests/eventfdTests.py b/tests/eventfdTests.py
new file mode 100644
index 0000000..be15248
--- /dev/null
+++ b/tests/eventfdTests.py
@@ -0,0 +1,109 @@
+#
+# Copyright 2014 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+import os
+import select
+from vdsm.eventfd import EventFile, DATASIZE
+from nose.tools import timed, raises, TimeExpired
+
+TEST_TIMEOUT = 1
+WAIT_TIMEOUT = 2
+
+
+def test_set():
+ e = EventFile()
+ e.set()
+ assert e.is_set()
+ assert e.isSet()
+
+
+def text_clear():
+ e = EventFile()
+ e.set()
+ assert e.is_set()
+ e.clear()
+ assert not e.is_set()
+
+
+@timed(TEST_TIMEOUT)
+def test_wait_set():
+ e = EventFile()
+ e.set()
+ e.wait(WAIT_TIMEOUT)
+
+
+@raises(TimeExpired)
+@timed(TEST_TIMEOUT)
+def test_wait_noset():
+ e = EventFile()
+ e.wait(WAIT_TIMEOUT)
+
+
+@timed(TEST_TIMEOUT)
+def test_eventfd_earlyset():
+ e = EventFile()
+ e.set()
+ with e.eventfd() as fd:
+ assert len(__select_and_read(fd)) == DATASIZE
+
+
+@timed(TEST_TIMEOUT)
+def test_eventfd_lateset():
+ e = EventFile()
+ with e.eventfd() as fd:
+ e.set()
+ assert len(__select_and_read(fd)) == DATASIZE
+
+
+@raises(TimeExpired)
+@timed(TEST_TIMEOUT)
+def test_eventfd_noset():
+ e = EventFile()
+ with e.eventfd() as fd:
+ assert len(__select_and_read(fd)) != DATASIZE
+
+
+@timed(TEST_TIMEOUT)
+def test_eventfd_multiple():
+ e = EventFile()
+ e.set()
+ with e.eventfd() as fd1:
+ assert len(__select_and_read(fd1)) == DATASIZE
+ with e.eventfd() as fd2:
+ assert len(__select_and_read(fd2)) == DATASIZE
+ with e.eventfd() as fd3:
+ assert len(__select_and_read(fd3)) == DATASIZE
+
+
+@raises(TimeExpired)
+@timed(TEST_TIMEOUT)
+def test_eventfd_clear():
+ e = EventFile()
+ e.set()
+ e.clear()
+ with e.eventfd() as fd:
+ assert len(__select_and_read(fd)) != DATASIZE
+
+
+def __select_and_read(fd):
+ rd, wr, ex = select.select((fd,), (), (), WAIT_TIMEOUT)
+ if fd in rd:
+ return os.read(fd, DATASIZE)
+ return ''
--
To view, visit http://gerrit.ovirt.org/33687
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0d237f13c42b1f4505c90d30c6d3c3ecbd1e9fa7
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Federico Simoncelli <fsimonce(a)redhat.com>
7 years, 3 months
Change in vdsm[master]: utils: Add changehash function for change detection
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: utils: Add changehash function for change detection
......................................................................
utils: Add changehash function for change detection
We use Python built-in hash to detect changes in vm state without sending
the state in each response. This function is not suitable for this
usage. Now we use generic utils.changehash(), implemented using md5
hexdigest.
Change-Id: I2242a594383e2d2fe64e3a581f18b8ac662648b0
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M lib/vdsm/utils.py
M vdsm/virt/vm.py
2 files changed, 13 insertions(+), 2 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/45/33045/1
diff --git a/lib/vdsm/utils.py b/lib/vdsm/utils.py
index 23c63e8..1b4a9d5 100644
--- a/lib/vdsm/utils.py
+++ b/lib/vdsm/utils.py
@@ -37,6 +37,7 @@
import glob
import io
import itertools
+import hashlib
import logging
import re
import sys
@@ -1133,3 +1134,13 @@
flags = fcntl.fcntl(fd, fcntl.F_GETFL)
flags |= os.O_NONBLOCK
fcntl.fcntl(fd, fcntl.F_SETFL, flags)
+
+
+def changehash(s):
+ """
+ Returns a hash of string s, suitable for change detection.
+
+ Tipically changehash(s) is sent to client frequently. When a client detect
+ that changehash(s) changed, it ask for s itself, which may be much bigger.
+ """
+ return hashlib.md5(s).hexdigest()
diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 941f283..b1567f9 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -1500,7 +1500,7 @@
self.guestAgent = guestagent.GuestAgent(
self._guestSocketFile, self.cif.channelListener, self.log)
self._lastXMLDesc = '<domain><uuid>%s</uuid></domain>' % self.id
- self._devXmlHash = '0'
+ self._devXmlHash = utils.changehash('')
self._released = False
self._releaseLock = threading.Lock()
self.saveState()
@@ -4495,7 +4495,7 @@
self._lastXMLDesc = self._dom.XMLDesc(0)
devxml = _domParseStr(self._lastXMLDesc).childNodes[0]. \
getElementsByTagName('devices')[0]
- self._devXmlHash = str(hash(devxml.toxml()))
+ self._devXmlHash = utils.changehash(devxml.toxml())
return self._lastXMLDesc
--
To view, visit http://gerrit.ovirt.org/33045
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2242a594383e2d2fe64e3a581f18b8ac662648b0
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 3 months
Change in vdsm[master]: log: Use INFO log level as default
by Nir Soffer
Nir Soffer has uploaded a new change for review.
Change subject: log: Use INFO log level as default
......................................................................
log: Use INFO log level as default
The current logs are much too verbose which cause trouble for users, and
make us look unprofessional. Mature project should not use debug log by
default.
To debug issues that are not clear enough using INFO logs, the relevant
logger level can be modified on a user machine as needed.
Change-Id: I767dcd9bad7b9fbeebb438e9ef13cb0ec3f042ee
Signed-off-by: Nir Soffer <nsoffer(a)redhat.com>
---
M vdsm/logger.conf.in
1 file changed, 4 insertions(+), 4 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/04/32504/1
diff --git a/vdsm/logger.conf.in b/vdsm/logger.conf.in
index 64b154f..8e963dd 100644
--- a/vdsm/logger.conf.in
+++ b/vdsm/logger.conf.in
@@ -8,18 +8,18 @@
keys=long,simple,none,sysform
[logger_root]
-level=DEBUG
+level=INFO
handlers=syslog,logfile
propagate=0
[logger_vds]
-level=DEBUG
+level=INFO
handlers=syslog,logfile
qualname=vds
propagate=0
[logger_Storage]
-level=DEBUG
+level=INFO
handlers=logfile
qualname=Storage
propagate=0
@@ -31,7 +31,7 @@
propagate=1
[logger_connectivity]
-level=DEBUG
+level=INFO
handlers=connlogfile
qualname=connectivity
propagate=0
--
To view, visit http://gerrit.ovirt.org/32504
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I767dcd9bad7b9fbeebb438e9ef13cb0ec3f042ee
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer(a)redhat.com>
7 years, 4 months