Change in vdsm[master]: jsonrpc: Implement the JsonRPC server for the next-gen API - vdsm-patches - Fedora mailing-lists

16 Oct 2012

Adam Litke has uploaded a new change for review.
Change subject: jsonrpc: Implement the JsonRPC server for the next-gen API
......................................................................
jsonrpc: Implement the JsonRPC server for the next-gen API
This patch implements a new Binding plugin to serve the next generation vdsm API
over a JsonRPC wire protocol.  The basic format of a message is:
<size><json-data>
<size> is an unsigned 64 bit integer in big endian format that indicates the
length of <json-data> in bytes.  <json-data> is either a request or a response
in Javascript object notation (JSON).
A request object has the following fields:
    id:  An integer which will be repeated in the matching response
    methodName:  The name of the API method to be called
    args (optional): A JSON object containing arguments to the method
A response object has the following fields:
    id:  An integer which will be the same as the matching request
    result:  The return value of the method (defined in the API schema)
    error:  A JSON object containing error information
            code:  An integer error code
            message:  Error context information
Method calls are dispatched to vdsm using a MethodBridge.  The DynamicBridge
dispatches calls based on an API schema document and a set of schema exceptions.
In this way, the API can be expanded without the need to add more code to this
server infrastructure.  When a request is made, the DynamicBridge attempts to
resolve 'methodName' to a schema defined method.  If found, the necessary
arguments are collected from the request and the function call is dispatched to
the internal vdsm API.
Schema exceptions:
Currently, vdsm does not completely conform to its own API schema.  It is not
possible to correct all of the discrepencies due to the need to maintain
backwards compatibility with current API users.  To facilitate migration to this
API, the DynamicBridge implements a set of overrides to translate between the
schema-defined API and what is implemented in vdsm today.  There are three types
of method overrides and one type of argument override:
Method overrides:
1. Custom call function:  If a new API does not map directly to an existing vdsm
   API (or multiple functions must be called to get the result), a custom 'call'
   function can be defined for the API.  This function will be called instead of
   trying to find a vdsm API to call.
2. Return field name:  Many vdsm functions' return values are nested in a
   dictionary.  This override specifies the key to use when accessing the result
   so that it can be returned un-nested to the caller.
3. Custom result post-processing function:  Some vdsm APIs return results in a
   non-standard format and a special function must be used to reformat the data
   to comply with the schema.
Type override:
Some data types have a different representation internally to vdsm than we have
defined in the schema.  A type override can be used to convert between the two
formats.  Currently, the only kind of translation we do is to rename fields
within a given type.
Change-Id: Idae0faa80ffc6a5af002a8a7151aa40dc9a6673d
Signed-off-by: Adam Litke agl@us.ibm.com
---
M Makefile.am
M configure.ac
M tests/Makefile.am
A tests/apiData.py
A tests/apiTests.py
M tests/run_tests_local.sh.in
M vdsm.spec.in
M vdsm/clientIF.py
M vdsm/config.py.in
A vdsm_api/BindingJsonRpc.py
A vdsm_api/Bridge.py
A vdsm_api/Makefile.am
12 files changed, 725 insertions(+), 4 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/14/8614/1

diff --git a/Makefile.am b/Makefile.am
index 7548f42..b1a0418 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -18,7 +18,7 @@
 # Refer to the README and COPYING files for full details of the license
 #
-SUBDIRS = vdsm vdsm_cli vds_bootstrap vdsm_reg vdsm_hooks tests vdsm-tool
+SUBDIRS = vdsm vdsm_cli vds_bootstrap vdsm_reg vdsm_hooks tests vdsm-tool vdsm_api
include $(top_srcdir)/build-aux/Makefile.subs
@@ -112,8 +112,7 @@
    vds_bootstrap/miniyum.py \
    $(NULL)
-PEP8_BLACKLIST = \
-	restData.py
+PEP8_BLACKLIST = apiData.py,restData.py
check-local:
    find . -path './.git' -prune -type f -o \
diff --git a/configure.ac b/configure.ac
index 450f714..679ad04 100644
--- a/configure.ac
+++ b/configure.ac
@@ -79,6 +79,7 @@
 AC_SUBST([vdsmpylibdir], ['${pyexecdir}/vdsm'])
 AC_SUBST([vdsmtooldir], ['${vdsmpylibdir}/tool'])
 AC_SUBST([vdsmtestsdir], ['${datarootdir}/vdsm/tests'])
+AC_SUBST([vdsmapidir], ['${datarootdir}/vdsm_api'])
# VDSM registration default paths
 AC_SUBST([vdsmregdir], ['${datarootdir}/vdsm-reg'])
@@ -191,6 +192,7 @@
    tests/Makefile
    tests/functional/Makefile
    vds_bootstrap/Makefile
+	vdsm_api/Makefile
    vdsm_cli/Makefile
    vdsm_hooks/directlun/Makefile
    vdsm_hooks/faqemu/Makefile
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2b61dde..d1fca34 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -22,6 +22,7 @@
test_modules = \
    alignmentScanTests.py \
+	apiTests.py \
    betterPopenTests.py \
    betterThreadingTests.py \
    capsTests.py \
@@ -66,6 +67,7 @@
dist_vdsmtests_PYTHON = \
    $(test_modules) \
+	apiData.py \
    monkeypatch.py \
    testrunner.py \
    testValidation.py
diff --git a/tests/apiData.py b/tests/apiData.py
new file mode 100644
index 0000000..fdb2611
--- /dev/null
+++ b/tests/apiData.py
@@ -0,0 +1,10 @@
+class APIData(object):
+    def __init__(self, obj, meth, data):
+        self.obj = obj
+        self.meth = meth
+        self.data = data
+
+testPing_apidata = [
+    APIData('Global', 'ping', {
+        'status': {'code': 0, 'message': 'OK'}})
+]
diff --git a/tests/apiTests.py b/tests/apiTests.py
new file mode 100644
index 0000000..9bfe347
--- /dev/null
+++ b/tests/apiTests.py
@@ -0,0 +1,228 @@
+#
+# Copyright 2012 Adam Litke, IBM Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
+#
+# Refer to the README and COPYING files for full details of the license
+#
+
+import logging
+import os
+import os.path
+import socket
+import json
+import struct
+import time
+
+from testrunner import VdsmTestCase as TestCaseBase
+from vdsm import constants
+import apiData
+
+ip = '0.0.0.0'
+port = 9824
+_fakeret = {}
+
+apiWhitelist = ('StorageDomain.Classes', 'StorageDomain.Types',
+    'Volume.Formats', 'Volume.Types', 'Volume.Roles', 'Image.DiskTypes')
+
+
+def getFakeAPI():
+    """
+    Create a Mock API module for testing.  Mock API will return data from
+    the _fakeret global variable instead of calling into vdsm.  _fakeret is
+    expected to have the following format:
+
+    {
+      '<class1>': {
+        '<func1>': [ <ret1>, <ret2>, ... ],
+        '<func2>': [ ... ],
+      }, '<class2>': {
+        ...
+      }
+    }
+    """
+    class FakeObj(object):
+        def __new__(cls, *args, **kwargs):
+            return object.__new__(cls)
+
+        def default(self, *args, **kwargs):
+            try:
+                return _fakeret[self.type][self.lastFunc].pop(0)
+            except (KeyError, IndexError):
+                raise Exception("No API data avilable for %s.%s" % \
+                                (self.type, self.lastFunc))
+
+        def __getattr__(self, name):
+            # While we are constructing the API module, use the normal getattr
+            if 'API' not in sys.modules:
+                return object.__getattr__(name)
+            self.lastFunc = name
+            return self.default
+
+    import sys
+    import imp
+    from new import classobj
+
+    _API = __import__('API', globals(), locals(), {}, -1)
+    _newAPI = imp.new_module('API')
+
+    for obj in ('Global', 'ConnectionRefs', 'StorageDomain', 'Image', 'Volume',
+                'Task', 'StoragePool', 'VM'):
+        cls = classobj(obj, (FakeObj,), {'type': obj})
+        setattr(_newAPI, obj, cls)
+
+    # Apply the whitelist to our version of API
+    for name in apiWhitelist:
+        parts = name.split('.')
+        dstObj = _newAPI
+        srcObj = _API
+        # Walk the object hierarchy copying each component of the whitelisted
+        # attribute from the real API to our fake one
+        for obj in parts:
+            srcObj = getattr(srcObj, obj)
+            try:
+                dstObj = getattr(dstObj, obj)
+            except AttributeError:
+                setattr(dstObj, obj, srcObj)
+
+    # Install our fake API into the module table for use by the whole program
+    sys.modules['API'] = _newAPI
+
+
+def findSchema():
+    """
+    Find the API schema file whether we are running tests from the source dir
+    or from the tests install location
+    """
+    scriptdir = os.path.dirname(__file__)
+    localpath = os.path.join(scriptdir, '../vdsm_api/vdsmapi-schema.json')
+    installedpath = os.path.join(constants.P_VDSM, 'vdsmapi-schema.json')
+    for f in localpath, installedpath:
+        if os.access(f, os.R_OK):
+            return f
+    raise Exception("Unable to find schema in %s or %s",
+                      localpath, installedpath)
+
+def setUpModule():
+    """
+    Set up the environment for all tests:
+    1. Override the API so we can program our own return values
+    2. Start an embedded server to process our requests
+    """
+    log = logging.getLogger('apiTests')
+    handler = logging.StreamHandler()
+    fmt_str = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+    formatter = logging.Formatter(fmt_str)
+    handler.setFormatter(formatter)
+    handler.setLevel(logging.INFO)
+    log.addHandler(handler)
+
+    schema = findSchema()
+    getFakeAPI()
+    import BindingJsonRpc
+    import Bridge
+    bridge = Bridge.DynamicBridge(schema)
+    server = BindingJsonRpc.BindingJsonRpc(bridge, ip, port)
+    server.start()
+
+
+class APITest(TestCaseBase):
+    def expectAPI(self, obj, meth, retval):
+        global _fakeret
+        if obj not in _fakeret:
+            _fakeret[obj] = {}
+        if meth not in _fakeret[obj]:
+            _fakeret[obj][meth] = []
+        _fakeret[obj][meth].append(retval)
+
+    def programAPI(self, key):
+        key += '_apidata'
+        for item in getattr(apiData, key):
+            self.expectAPI(item.obj, item.meth, item.data)
+
+    def clearAPI(self):
+        global _fakeret
+        _fakeret = {}
+
+class JsonRawTest(APITest):
+    _Size = struct.Struct("!Q")
+
+    def buildMessage(self, data):
+        msg = json.dumps(data)
+        msg = msg.encode('utf-8')
+        msize = JsonRawTest._Size.pack(len(msg))
+        resp = msize + msg
+        return resp
+
+    def sendMessage(self, msg):
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            sock.connect((ip, port))
+            sock.sendall(msg)
+            data = sock.recv(JsonRawTest._Size.size)
+            msgLen = JsonRawTest._Size.unpack(data)[0]
+            data = sock.recv(msgLen)
+            return json.loads(data)
+        finally:
+            sock.close()
+
+    def testPing(self):
+        self.clearAPI()
+        self.programAPI("testPing")
+        msg = self.buildMessage({'id': 1, 'methodName': 'Host.ping',
+                                   'args': {}})
+        reply = self.sendMessage(msg)
+        self.assertEquals(0, reply['error']['code'])
+        self.assertEquals('Success', reply['error']['message'])
+
+    def testNoMethod(self):
+        msg = self.buildMessage({'id': 1, 'methodName': 'Host.fake'})
+        reply = self.sendMessage(msg)
+        self.assertEquals(4, reply['error']['code'])
+
+    def testBadMethod(self):
+        msg = self.buildMessage({'id': 1, 'methodName': 'malformed''})
+        reply = self.sendMessage(msg)
+        self.assertEquals(4, reply['error']['code'])
+
+    def testMissingSize(self):
+        self.assertRaises(struct.error, self.sendMessage,
+                           "malformed message")
+
+    def testNotJson(self):
+        msg = "malformed message"
+        msize = JsonRawTest._Size.pack(len(msg))
+        msg = msize + msg
+        self.assertRaises(struct.error, self.sendMessage, msg)
+
+    def testSynchronization(self):
+        def doPing(msg):
+            self.clearAPI()
+            self.programAPI("testPing")
+            return self.sendMessage(msg)['error']['code']
+
+        msg = self.buildMessage({'id': 1, 'methodName': 'Host.ping'})
+        # Send Truncated message
+        self.assertRaises(struct.error, doPing, msg[:-1])
+
+        # Test that the server recovers
+        self.assertEquals(0, doPing(msg))
+
+        # Send too much data
+        self.assertEquals(0, doPing(msg + "Hello"))
+
+        # Test that the server recovers
+        self.assertEquals(0, doPing(msg))
+
diff --git a/tests/run_tests_local.sh.in b/tests/run_tests_local.sh.in
index fc4ac37..8f668f9 100644
--- a/tests/run_tests_local.sh.in
+++ b/tests/run_tests_local.sh.in
@@ -1,2 +1,2 @@
 #!/bin/sh
-PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@builddir@/vdsm:@top_srcdir@/vdsm:@top_srcdir@/vdsm_cli:$PYTHONPATH" @PYTHON@ @top_srcdir@/tests/testrunner.py --local-modules $@
+PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@builddir@/vdsm:@top_srcdir@/vdsm:@top_srcdir@/vdsm_cli:@top_srcdir@/vdsm_api:$PYTHONPATH" @PYTHON@ @top_srcdir@/tests/testrunner.py --local-modules $@
diff --git a/vdsm.spec.in b/vdsm.spec.in
index dce14a9..8e7fa6c 100644
--- a/vdsm.spec.in
+++ b/vdsm.spec.in
@@ -154,6 +154,15 @@
 %description rest
 A REST interface for interacting with vdsmd.
+%package jsonrpc
+Summary:        VDSM API Server
+BuildArch:      noarch
+
+Requires: %{name}-python = %{version}-%{release}
+
+%description jsonrpc
+A Json-based RPC interface that serves as the protocol for libvdsm.
+
 %package bootstrap
 Summary:        VDSM bootstrapping package
 BuildArch:      noarch
@@ -912,6 +921,12 @@
 %{_datadir}/%{vdsm_name}/rest/templates/api.xsd
 %{_datadir}/%{vdsm_name}/rest/templates/rsdl.xml
+%files jsonrpc
+%{_datadir}/%{vdsm_name}/BindingJsonRpc.py*
+%{_datadir}/%{vdsm_name}/Bridge.py*
+%{_datadir}/%{vdsm_name}/vdsmapi.py*
+%{_datadir}/%{vdsm_name}/vdsmapi-schema.json
+
 %files bootstrap
 %defattr(-, root, root, -)
 %doc COPYING
diff --git a/vdsm/clientIF.py b/vdsm/clientIF.py
index e9ca7c9..cf8468c 100644
--- a/vdsm/clientIF.py
+++ b/vdsm/clientIF.py
@@ -158,6 +158,15 @@
         self.bindings['rest'] = BindingREST(self, self.log, ip, rest_port,
                                             templatePath)
+    def _loadBindingJsonRpc(self):
+        from BindingJsonRpc import BindingJsonRpc
+        from Bridge import DynamicBridge
+        schema = os.path.join(constants.P_VDSM, 'vdsmapi-schema.json')
+        ip = self._getServerIP(config.get('addresses', 'management_ip'))
+        port = config.getint('addresses', 'json_port')
+        self.bindings['json'] = BindingJsonRpc(DynamicBridge(schema),
+                                                ip, port)
+
     def _prepareBindings(self):
         self.bindings = {}
         if config.getboolean('vars', 'xmlrpc_enable'):
@@ -172,6 +181,12 @@
             except ImportError:
                 self.log.warn('Unable to load the rest server module. '
                               'Please make sure it is installed.')
+        if config.getboolean('vars', 'jsonrpc_enable'):
+            try:
+                self._loadBindingJsonRpc()
+            except ImportError:
+                self.log.warn('Unable to load the json rpc server module. '
+                              'Please make sure it is installed.')
def _prepareMOM(self):
         try:
diff --git a/vdsm/config.py.in b/vdsm/config.py.in
index df85e7e..51a7ad6 100644
--- a/vdsm/config.py.in
+++ b/vdsm/config.py.in
@@ -130,6 +130,8 @@
('rest_enable', 'true', 'Enable the REST server'),
+        ('jsonrpc_enable', 'true', 'Enable the JSON RPC server'),
+
         ('report_host_threads_as_cores', 'false',
             'Count each cpu hyperthread as an individual core'),
     ]),
@@ -251,6 +253,10 @@
             'Port on which the vdsmd REST server listens to network '
             'clients.'),
+        ('json_port', '4044',
+            'Port on which the vdsmd Json RPC server listens to network '
+            'clients.'),
+
         ('management_ip', '', None),
('guests_gateway_ip', '', None),
diff --git a/vdsm_api/BindingJsonRpc.py b/vdsm_api/BindingJsonRpc.py
new file mode 100644
index 0000000..9ac8b5d
--- /dev/null
+++ b/vdsm_api/BindingJsonRpc.py
@@ -0,0 +1,99 @@
+# VDSM JsonRPC Server
+# Copyright (C) 2012 Adam Litke, IBM Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+import threading
+import SocketServer
+import json
+import logging
+
+import struct
+
+_Size = struct.Struct("!Q")
+
+__bridge__ = None
+
+class BindingJsonRpc:
+    def __init__(self, bridge, ip, port):
+        self.bridge = bridge
+        self.serverPort = port
+        self.serverIP = ip
+        self.log = logging.getLogger('BindingJsonRpc')
+        self._createServer()
+
+    def _createServer(self):
+        global __bridge__
+        __bridge__ = self.bridge
+        ip = self.serverIP or '0.0.0.0'
+        self.server = JsonRpcServer((ip, self.serverPort), JsonRpcTCPHandler)
+
+    def start(self):
+        def threaded_start():
+            self.server.serve_forever()
+        t = threading.Thread(target=threaded_start,
+                        name='JsonRpc')
+        t.setDaemon(True)
+        t.start()
+
+    def prepareForShutdown(self):
+        self.server.shutdown()
+
+
+class JsonRpcServer(SocketServer.TCPServer):
+    def __init__(self, addrInfo, handler):
+        self.allow_reuse_address = True
+        SocketServer.TCPServer.__init__(self, addrInfo, handler)
+
+
+class JsonRpcTCPHandler(SocketServer.StreamRequestHandler):
+    """
+    The RequestHandler class for our server.
+
+    It is instantiated once per connection to the server, and must
+    override the handle() method to implement communication to the
+    client.
+    """
+
+    def handle(self):
+        bridge = __bridge__
+        log = logging.getLogger('JsonRpcTCPHandler')
+        while True:
+            # self.request is the TCP socket connected to the client
+            try:
+                data = self.request.recv(_Size.size)
+                if len(data) == 0:
+                    log.debug("Connection closed")
+                    return
+                msgLen = _Size.unpack(data)[0]
+                msg = json.loads(self.request.recv(msgLen))
+            except:
+                log.warn("Unexpected exception", exc_info=True)
+                return
+            log.debug('--> %s', msg)
+
+            try:
+                ret = bridge._dispatch(msg['methodName'], msg.get('args', {}))
+            except Exception:
+                log.error("Dispatch error", exc_info=True)
+                continue
+            ret['id'] = msg['id']
+            msg = json.dumps(ret)
+            msg = msg.encode('utf-8')
+            msize = _Size.pack(len(msg))
+            resp = msize + msg
+            log.debug('<-- %s', msg)
+
+            self.wfile.write(resp)
+            self.wfile.flush()
+
diff --git a/vdsm_api/Bridge.py b/vdsm_api/Bridge.py
new file mode 100644
index 0000000..7402e08
--- /dev/null
+++ b/vdsm_api/Bridge.py
@@ -0,0 +1,332 @@
+# VDSM API Connector
+# Copyright (C) 2012 Adam Litke, IBM Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+
+from functools import partial
+import vdsmapi
+import logging
+import types
+import API
+
+class VdsmError(Exception):
+    def __init__(self, code, message):
+        self.code = code
+        self.message = message
+
+class MethodBridge(object):
+
+    def _dispatch(self, name, argobj):
+        method = name.replace('.', '_')
+        result = None
+        error = {'code': 0, 'message': 'Success'}
+        try:
+            fn = getattr(self, method)
+        except AttributeError:
+            error = {'code': 4,
+                     'message': "Operation '%s' not supported" % name}
+            return {'result': result, 'error': error}
+        try:
+            result = fn(argobj)
+        except VdsmError, e:
+            error = {'code': e.code, 'message': e.message}
+        except Exception, e:
+            error = {'code': 5, 'message': 'Internal error: %s' % e}
+        return {'result': result, 'error': error}
+
+    def getArgs(self, argobj, arglist):
+        ret = ()
+        for arg in arglist:
+            if arg in argobj:
+                ret += (argobj[arg],)
+        return ret
+
+    def getResult(self, response, member=None):
+        if member is None:
+            return None
+        try:
+            return response[member]
+        except KeyError:
+            raise VdsmError(5, "Response is missing '%s' member" % member)
+
+class DynamicBridge(MethodBridge):
+    def __init__(self, schema):
+        self.parseSchema(schema)
+
+
+    def parseSchema(self, schema):
+        self.commands = {}
+        self.classes = {}
+        self.types = {}
+        with open(schema) as f:
+            symbols = vdsmapi.parse_schema(f)
+            for s in symbols:
+                if 'command' in s:
+                    key = "%s_%s" % (s['command']['class'],
+                                     s['command']['name'])
+                    self.commands[key] = s
+                elif 'class' in s:
+                    cls = s['class']
+                    self.classes[cls] = s
+                elif 'type' in s:
+                    t = s['type']
+                    self.types[t] = s
+
+
+    def __getattr__(self, attr):
+        if attr in self.commands:
+            cls, name = attr.split('_')
+            return partial(self.dynamicMethod, cls, name)
+        else:
+            raise AttributeError
+
+
+    def getArgList(self, cmd):
+        return self.commands[cmd].get('data', {}).keys()
+
+
+    def getObjargList(self, cls):
+        return self.classes[cls]['data'].keys()
+
+    def getApiClass(self, cls):
+        name_map = { 'Host': 'Global' }
+        try:
+            cls = name_map[cls]
+        except KeyError:
+            pass
+        return getattr(API, cls)
+
+    def typeFixup(self, sym_name, sym_type, obj):
+        logging.info("typeFixup: '%s': '%s'" % (sym_name, sym_type))
+        isList = False
+        if type(sym_type) is list:
+            sym_type = sym_type[0]
+            isList = True
+        if sym_name[0] == '*':
+            sym_name = sym_name[1:]
+
+        try:
+            symbol = self.types[sym_type]
+        except KeyError:
+            return
+
+        if isList:
+            itemList = obj
+        else:
+            itemList = [obj]
+
+        for item in itemList:
+            if sym_type in typefixups:
+                logging.error("Fixing up type %s", sym_type)
+                typefixups[sym_type](item)
+            for (k, v) in symbol.get('data', {}).items():
+                if k[0] == '*':
+                    k = k[1:]
+                if k in item:
+                    self.typeFixup(k, v, item[k])
+
+
+    def fixupArgs(self, cmd, args):
+        argInfo = zip(self.commands[cmd].get('data', {}).items(), args)
+        for typeInfo, val in argInfo:
+            argName, argType = typeInfo
+            if argType not in self.types:
+                continue
+            self.typeFixup(argName, argType, val)
+
+    def fixupRet(self, cmd, result):
+        retType = self.commands[cmd].get('returns', None)
+        if retType is not None:
+            self.typeFixup('return', retType, result)
+        return result
+
+    def dynamicMethod(self, cls, name, argobj):
+        cmd = '%s_%s' % (cls, name)
+        ctorArgObj = argobj.get('__obj__', {})
+        ctorargs = self.getArgs(ctorArgObj, self.getObjargList(cls))
+        args = self.getArgs(argobj, self.getArgList(cmd))
+        apiobj = self.getApiClass(cls)
+        api = apiobj(*ctorargs)
+
+        self.fixupArgs(cmd, args)
+
+        # Call the override function (if given).  Otherwise, just call directly
+        fn = command_info.get(cmd, {}).get('call')
+        if fn:
+            result = fn(api, argobj)
+        else:
+            fn = getattr(api, name)
+            result = fn(*args)
+
+        if result['status']['code']:
+            code = result['status']['code']
+            msg = result['status']['message']
+            return {'code': code, 'message': msg}
+
+        retfield = command_info.get(cmd, {}).get('ret')
+        if type(retfield) == types.FunctionType:
+            ret = retfield(result)
+        else:
+            ret = self.getResult(result, retfield)
+        return self.fixupRet(cmd, ret)
+
+
+def Host_getStorageRepoStats_Ret(ret):
+    """
+    The returned dictionary doesn't separate the stats from the status code
+    so we need to rebuild the result.
+    """
+    del ret['status']
+    return ret
+
+
+def Host_getVMList_Call(api, args):
+    """
+    This call is only interested in returning the VM UUIDs so pass False for
+    the first argument in order to suppress verbose results.
+    """
+    vmList = args.get('vmList', [])
+    return API.Global().getVMList(False, vmList)
+
+def Host_getVMList_Ret(ret):
+    """
+    Just return a list of VM UUIDs
+    """
+    return [v['vmId'] for v in ret['vmList']]
+
+
+def StoragePool_getInfo_Ret(ret):
+    """
+    The result contains two data structures which must be merged
+    """
+    return {'info': ret['info'], 'dominfo': ret['dominfo']}
+
+
+def VM_getInfo_Call(api, args):
+    """
+    The VM object has no getInfo method.  We use the method from 'Global' and
+    pass arguments to get verbose information for only this one VM.
+    """
+    vmId = api._UUID
+    return API.Global().getVMList(True, [vmId])
+
+def VM_getInfo_Ret(ret):
+    """
+    The result will be a list with only one element.
+    """
+    return ret['vmList'][0]
+
+def VM_migrationCreate_Ret(ret):
+    """
+    The result contains two data structures which must be merged
+    """
+    return {'params': ret['params'], 'migrationPort': ret['migrationPort']}
+
+
+def Volume_getsize_Ret(ret):
+    """
+    Merge the two sizes into a single dictionary result.
+    """
+    return {'truesize': ret['truesize'], 'apparentsize': ret['apparentsize']}
+
+
+##
+# Possible ways to override a command:
+# - Supply a custom call function if the function name doesn't map directly to
+#   a vdsm API.
+# - Specify the name of the field in the result that is the return value
+# - Specify a custom function to post-process the result into a return value
+##
+command_info = {
+    'ConnectionRefs_acquire': {'ret': 'results'},
+    'ConnectionRefs_release': {'ret': 'results'},
+    'ConnectionRefs_statuses': {'ret': 'connectionslist'},
+    'Host_fenceNode': {'ret': 'power'},
+    'Host_getAllTasksInfo': {'ret': 'allTasksInfo'},
+    'Host_getAllTasksStatuses': {'ret': 'allTasksStatus'},
+    'Host_getCapabilities': {'ret': 'info'},
+    'Host_getConnectedStoragePools': {'ret': 'poollist'},
+    'Host_getDeviceInfo': {'ret': 'info'},
+    'Host_getDeviceList': {'ret': 'devList'},
+    'Host_getDevicesVisibility': {'ret': 'visibility'},
+    'Host_getLVMVolumeGroups': {'ret': 'vglist'},
+    'Host_getStats': {'ret': 'info'},
+    'Host_getStorageDomains': {'ret': 'domlist'},
+    'Host_getStorageRepoStats': {'ret': Host_getStorageRepoStats_Ret},
+    'Host_getVMList': {'call': Host_getVMList_Call, 'ret': Host_getVMList_Ret},
+    'Image_delete': {'ret': 'uuid'},
+    'Image_deleteVolumes': {'ret': 'uuid'},
+    'Image_getVolumes': {'ret': 'uuidlist'},
+    'Image_mergeSnapshots': {'ret': 'uuid'},
+    'Image_move': {'ret': 'uuid'},
+    'ISCSIConnection_discoverSendTargets': {'ret': 'fullTargets'},
+    'LVMVolumeGroup_create': {'ret': 'uuid'},
+    'LVMVolumeGroup_getInfo': {'ret': 'info'},
+    'StorageDomain_getFileList': {'ret': 'files'},
+    'StorageDomain_getImages': {'ret': 'imageslist'},
+    'StorageDomain_getInfo': {'ret': 'info'},
+    'StorageDomain_getStats': {'ret': 'stats'},
+    'StorageDomain_getVolumes': {'ret': 'uuidlist'},
+    'StoragePool_connectStorageServer': {'ret': 'statuslist'},
+    'StoragePool_disconnectStorageServer': {'ret': 'statuslist'},
+    'StoragePool_fence': {'ret': 'spm_st'},
+    'StoragePool_getBackedUpVmsInfo': {'ret': 'vmlist'},
+    'StoragePool_getBackedUpVmsList': {'ret': 'vmlist'},
+    'StoragePool_getDomainsContainingImage': {'ret': 'domainslist'},
+    'StoragePool_getFloppyList': {'ret': 'isolist'},
+    'StoragePool_getInfo': {'ret': StoragePool_getInfo_Ret},
+    'StoragePool_getIsoList': {'ret': 'isolist'},
+    'StoragePool_getSpmStatus': {'ret': 'spm_st'},
+    'StoragePool_spmStart': {'ret': 'uuid'},
+    'StoragePool_upgrade': {'ret': 'upgradeStatus'},
+    'StoragePool_validateStorageServerConnection': {'ret': 'statuslist'},
+    'Task_getInfo': {'ret': 'TaskInfo'},
+    'Task_getStatus': {'ret': 'taskStatus'},
+    'VM_changeCD': {'ret': 'vmList'},
+    'VM_changeFloppy': {'ret': 'vmList'},
+    'VM_create': {'ret': 'vmList'},
+    'VM_getInfo': {'call': VM_getInfo_Call, 'ret': VM_getInfo_Ret},
+    'VM_getStats': {'ret': 'statsList'},
+    'VM_hotplugDisk': {'ret': 'vmList'},
+    'VM_hotplugNic': {'ret': 'vmList'},
+    'VM_hotUnplugDisk': {'ret': 'vmList'},
+    'VM_hotUnplugNic': {'ret': 'vmList'},
+    'VM_mergeStatus': {'ret': 'mergeStatus'},
+    'VM_migrationCreate': {'ret': VM_migrationCreate_Ret},
+    'Volume_copy': {'ret': 'uuid'},
+    'Volume_create': {'ret': 'uuid'},
+    'Volume_delete': {'ret': 'uuid'},
+    'Volume_getInfo': {'ret': 'info'},
+    'Volume_getPath': {'ret': 'path'},
+    'Volume_getSize': {'ret': Volume_getsize_Ret},
+    'Host_getAllTasks': {'ret': 'TasksDetails'},
+}
+
+
+def fieldClone(oldName, newName, obj):
+    logging.warning("fieldClone: %s -> %s", oldName, newName)
+    if oldName in obj:
+        obj[newName] = obj[oldName]
+    elif newName in obj:
+        obj[oldName] = obj[newName]
+
+typefixups = {
+    'VmDevice': partial(fieldClone, 'type', 'deviceType'),
+    'BlockDevicePathInfo': partial(fieldClone, 'type', 'deviceType'),
+    'VolumeGroupInfo': partial(fieldClone, 'type', 'deviceType'),
+    'VmDeviceAddress': partial(fieldClone, 'type', 'addressType'),
+    'IscsiCredentials': partial(fieldClone, 'type', 'authType'),
+    'ConnectionRefArgs': partial(fieldClone, 'type', 'connType'),
+    'VolumeInfo': partial(fieldClone, 'type', 'allocType'),
+}
diff --git a/vdsm_api/Makefile.am b/vdsm_api/Makefile.am
new file mode 100644
index 0000000..655586f
--- /dev/null
+++ b/vdsm_api/Makefile.am
@@ -0,0 +1,13 @@
+EXTRA_DIST = \
+	vdsmapi-schema.json \
+	$(NULL)
+
+dist_vdsm_PYTHON = \
+	BindingJsonRpc.py \
+	Bridge.py \
+	vdsmapi.py \
+	$(NULL)
+
+dist_vdsm_DATA = \
+	vdsmapi-schema.json \
+	$(NULL)
--
To view, visit http://gerrit.ovirt.org/8614
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idae0faa80ffc6a5af002a8a7151aa40dc9a6673d
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Adam Litke agl@us.ibm.com

    