[RFC v2 00/47] Python Recipes

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

v2 changes:
  add lnst.Common.IpAddress
    * modify exception string.
  add lnst.Common.Parameters
    * add docstring explaining the __getattribute__ magic
    * make val a property
    * rename x to attr in Parameters dir() iterations
    * fix IpParam value setter for string values
    * modify IpParam Exception string
  add lnst.Devices
    * make Device a Meta class and register RemoteDevice as implementing
      the Device interface
    * removed unnecessary imports
    * added MasterDevice class for slave* methods
    * made master_set to be a 'master' property setter
    * renamed {add, del}_route to route_{add, del}
    * renamed set_speed to speed_set
    * renamed set_autoneg to autoneg_set
    * removed modprobes where unnecessary
    * link_stats are now retrieved from netlink message
    * added link_stats64
    * OvsBridge calls parent _type_init for modprobe
    * remove SoftDevice import - not required
  add lnst.Controller.Host
    * removed debug pprint import
    * added docstrings to requested methods
    * Host::run throws an exception when netns set (not supported yet)
    * added TODO comments to some parts that need to be implemented
  add lnst.Controller.MachineMapper
    * docstring now explains that implementing your own MachineMapper is
      not supported at this moment
    * removed debug pprint import
  add lnst.Controller.Recipe
    * renamed variable 'x' to 'attr'
  Machine, NetTestSlave: heavy reimplementation
    * improved class and module synchronization implementation
    * join split lines in deviceref_to_device method
  lnst.Common.Parameters: IpParam accepts Device objects
    * import Device on demand (Device class arrives on Slave later) and
      check isinstance Device instead of RemoteDevice (not available on
      Slave)
    * fix IpParam Exception string

v2 new commits:
  Controller, Devices: removing unused imports
  NetTestSlave: create dynamic lnst.Tests package
  lnst.Tests: add package docstring
  lnst.Common.Parameters: add DeviceParam
  lnst.Tests.IcmpPing: use DeviceParam for iface parameter

Including the original cover letter as well:

Hi all,

the long avaited patchset for Python Recipes is here.

At this point there's still quite a lot of work left, but since the current
state is functional and the tester facing API should be mostly stable I think
it's a good time to get this merged into upstream so that more people can get
involved with filling in the missing pieces and can slowly start experimenting
and porting their XML recipes.

This I know are missing:
* Recipe run summary - how the Summary should look like was described in the
  running proposal document, but I didn't manage to start with the
  implementation yet...

* Porting of all the old test_modules into the lnst.Tests package, so far we
  only have the IcmpPing module that should still be reworked to be more
  universal (ip4 and ip6 in one class). Porting test_modules should be fairly
  easy, but at this point there's no guide how to do it so if you're having
  trouble feel free to contact me either on irc or email.

* test_tools - we haven't even thought of these yet

* network namespaces - also didn't think about them yet, though I'm hoping this
  will be simple

* Ip address and network generators as we've discussed them on the upstream
  meeting

Please review and provide any comments or features I forgot that should be
added to the above list.

Regards,
Ondrej

Ondrej Lichtner (47):
  add lnst.Common.LnstError
  add lnst.Common.DeviceError
  add lnst.Common.DeviceRef
  add lnst.Common.IpAddress
  add lnst.Common.Parameters
  add lnst.Common.TestModule
  add lnst.Common.JobError
  add lnst.Controller.Common
  add lnst.Devices
  add lnst.Controller.Requirements
  add lnst.Controller.Job
  add lnst.Controller.Host
  add lnst.Controller.Config
  add lnst.Slave.Config
  add lnst.Controller.MachineMapper
  lnst.Controller.Machine: change object initialization
  add lnst.Controller.MessageDispatcher
  add lnst.Controller.SlavePoolManager
  add lnst.Controller.Recipe
  add lnst.Controller.Controller
  various files: retype exceptions
  add lnst.Tests package
  lnst.Common.Config: remove {controller, slave}_init
  lnst.Common.Config: remove global lnst_config
  Slave: use a local config object instead of a global one
  lnst.Common.Utils: add sha256sum function
  lnst.Common.ResourceCache: simplification
  lnst.Controller.CtlSecSocket: remove lnst_config import
  lnst.Controller.SlaveMachineParser: make standalone
  add lnst.Common.InterfaceManagerError
  add lnst.Slave.Job
  lnst.Slave.InterfaceManager: heavy reimplementation
  Machine, NetTestSlave: heavy reimplementation
  lnst.Slave.InterfaceManager: remove Device class implementation
  add lnst.Controller package imports
  lnst/__init__.py remove imports
  lnst.Controller: remove old modules
  setup.py: add new packages
  add lnst.Devices.VirtNetCtl
  lnst.Controller: move VirtUtils to VirtDomainCtl, remove VirtNetCtl
    class
  lnst.Common.Parameters: IpParam accepts Device objects
  add example python_recipe.py script
  Controller, Devices: removing unused imports
  NetTestSlave: create dynamic lnst.Tests package
  lnst.Tests: add package docstring
  lnst.Common.Parameters: add DeviceParam
  lnst.Tests.IcmpPing: use DeviceParam for iface parameter

 lnst-slave                            |   20 +-
 lnst/Common/Config.py                 |  152 +---
 lnst/Common/DeviceError.py            |   22 +
 lnst/Common/DeviceRef.py              |   19 +
 lnst/Common/ExecCmd.py                |    3 +-
 lnst/Common/InterfaceManagerError.py  |   16 +
 lnst/Common/IpAddress.py              |   99 +++
 lnst/Common/JobError.py               |   22 +
 lnst/Common/LnstError.py              |   18 +
 lnst/Common/NetTestCommand.py         |    5 +-
 lnst/Common/Parameters.py             |  157 ++++
 lnst/Common/ResourceCache.py          |  128 ++--
 lnst/Common/SecureSocket.py           |    3 +-
 lnst/Common/ShellProcess.py           |    3 +-
 lnst/Common/TestModule.py             |   67 ++
 lnst/Common/TestsCommon.py            |    3 +-
 lnst/Common/Utils.py                  |   11 +
 lnst/Controller/Common.py             |   17 +
 lnst/Controller/Config.py             |   99 +++
 lnst/Controller/Controller.py         |  218 ++++++
 lnst/Controller/CtlSecSocket.py       |    2 -
 lnst/Controller/Host.py               |  160 ++++
 lnst/Controller/Job.py                |  196 +++++
 lnst/Controller/Machine.py            | 1356 +++++++--------------------------
 lnst/Controller/MachineMapper.py      |  328 ++++++++
 lnst/Controller/MessageDispatcher.py  |  203 +++++
 lnst/Controller/NetTestController.py  |  620 ---------------
 lnst/Controller/Recipe.py             |   97 +++
 lnst/Controller/RecipeParser.py       |  572 --------------
 lnst/Controller/Requirements.py       |  113 +++
 lnst/Controller/SlaveMachineParser.py |  144 +++-
 lnst/Controller/SlavePool.py          |  648 ----------------
 lnst/Controller/SlavePoolManager.py   |  273 +++++++
 lnst/Controller/Task.py               |    4 +-
 lnst/Controller/VirtDomainCtl.py      |   98 +++
 lnst/Controller/VirtUtils.py          |  268 -------
 lnst/Controller/XmlParser.py          |  188 -----
 lnst/Controller/XmlProcessing.py      |  235 ------
 lnst/Controller/XmlTemplates.py       |  438 -----------
 lnst/Controller/__init__.py           |    3 +
 lnst/Devices/BondDevice.py            |   38 +
 lnst/Devices/BridgeDevice.py          |   33 +
 lnst/Devices/Device.py                |  355 +++++++++
 lnst/Devices/MacvlanDevice.py         |   38 +
 lnst/Devices/MasterDevice.py          |   33 +
 lnst/Devices/OvsBridgeDevice.py       |  115 +++
 lnst/Devices/RemoteDevice.py          |   98 +++
 lnst/Devices/SoftDevice.py            |   51 ++
 lnst/Devices/TeamDevice.py            |   60 ++
 lnst/Devices/VethDevice.py            |   58 ++
 lnst/Devices/VethPair.py              |   24 +
 lnst/Devices/VirtNetCtl.py            |   85 +++
 lnst/Devices/VirtualDevice.py         |   98 +++
 lnst/Devices/VlanDevice.py            |   35 +
 lnst/Devices/VtiDevice.py             |   71 ++
 lnst/Devices/VxlanDevice.py           |   65 ++
 lnst/Devices/__init__.py              |   42 +
 lnst/RecipeCommon/ModuleWrap.py       |   33 +-
 lnst/Slave/Config.py                  |   73 ++
 lnst/Slave/InterfaceManager.py        |  648 ++--------------
 lnst/Slave/Job.py                     |  251 ++++++
 lnst/Slave/NetTestSlave.py            |  846 ++++++++++----------
 lnst/Tests/IcmpPing.py                |   62 ++
 lnst/Tests/__init__.py                |   17 +
 lnst/__init__.py                      |    1 -
 recipes/examples/python_recipe.py     |   32 +
 setup.py                              |    2 +-
 67 files changed, 4991 insertions(+), 5301 deletions(-)
 create mode 100644 lnst/Common/DeviceError.py
 create mode 100644 lnst/Common/DeviceRef.py
 create mode 100644 lnst/Common/InterfaceManagerError.py
 create mode 100644 lnst/Common/IpAddress.py
 create mode 100644 lnst/Common/JobError.py
 create mode 100644 lnst/Common/LnstError.py
 create mode 100644 lnst/Common/Parameters.py
 create mode 100644 lnst/Common/TestModule.py
 create mode 100644 lnst/Controller/Common.py
 create mode 100644 lnst/Controller/Config.py
 create mode 100644 lnst/Controller/Controller.py
 create mode 100644 lnst/Controller/Host.py
 create mode 100644 lnst/Controller/Job.py
 create mode 100644 lnst/Controller/MachineMapper.py
 create mode 100644 lnst/Controller/MessageDispatcher.py
 delete mode 100644 lnst/Controller/NetTestController.py
 create mode 100644 lnst/Controller/Recipe.py
 delete mode 100644 lnst/Controller/RecipeParser.py
 create mode 100644 lnst/Controller/Requirements.py
 delete mode 100644 lnst/Controller/SlavePool.py
 create mode 100644 lnst/Controller/SlavePoolManager.py
 create mode 100644 lnst/Controller/VirtDomainCtl.py
 delete mode 100644 lnst/Controller/VirtUtils.py
 delete mode 100644 lnst/Controller/XmlParser.py
 delete mode 100644 lnst/Controller/XmlProcessing.py
 delete mode 100644 lnst/Controller/XmlTemplates.py
 create mode 100644 lnst/Devices/BondDevice.py
 create mode 100644 lnst/Devices/BridgeDevice.py
 create mode 100644 lnst/Devices/Device.py
 create mode 100644 lnst/Devices/MacvlanDevice.py
 create mode 100644 lnst/Devices/MasterDevice.py
 create mode 100644 lnst/Devices/OvsBridgeDevice.py
 create mode 100644 lnst/Devices/RemoteDevice.py
 create mode 100644 lnst/Devices/SoftDevice.py
 create mode 100644 lnst/Devices/TeamDevice.py
 create mode 100644 lnst/Devices/VethDevice.py
 create mode 100644 lnst/Devices/VethPair.py
 create mode 100644 lnst/Devices/VirtNetCtl.py
 create mode 100644 lnst/Devices/VirtualDevice.py
 create mode 100644 lnst/Devices/VlanDevice.py
 create mode 100644 lnst/Devices/VtiDevice.py
 create mode 100644 lnst/Devices/VxlanDevice.py
 create mode 100644 lnst/Devices/__init__.py
 create mode 100644 lnst/Slave/Config.py
 create mode 100644 lnst/Slave/Job.py
 create mode 100644 lnst/Tests/IcmpPing.py
 create mode 100644 lnst/Tests/__init__.py
 create mode 100755 recipes/examples/python_recipe.py

-- 
2.13.0

[PATCH v2 01/47] add lnst.Common.LnstError

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the Base LNST Exception. All LNST related exceptions should
inherit from this.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/LnstError.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100644 lnst/Common/LnstError.py

diff --git a/lnst/Common/LnstError.py b/lnst/Common/LnstError.py
new file mode 100644
index 0000000..ae8d9bc
--- /dev/null
+++ b/lnst/Common/LnstError.py
@@ -0,0 +1,18 @@
+"""
+Defines the LnstError exception class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+class LnstError(Exception):
+    """Base LNST Exception type
+
+    All LNST related Exceptions should inherit from this class.
+    """
+    pass
-- 
2.13.0

[PATCH v2 02/47] add lnst.Common.DeviceError

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines some basic Exceptions signaling some error with a Device object
(to be added in a later commit). Since these Exceptions can be
transmitted to the Controller, they need to be in the Common package.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/DeviceError.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 lnst/Common/DeviceError.py

diff --git a/lnst/Common/DeviceError.py b/lnst/Common/DeviceError.py
new file mode 100644
index 0000000..cf1b7b7
--- /dev/null
+++ b/lnst/Common/DeviceError.py
@@ -0,0 +1,22 @@
+"""
+Defines the DeviceError, DeviceDeleted and DeviceNotFound exceptions.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.LnstError import LnstError
+
+class DeviceError(LnstError):
+    pass
+
+class DeviceDeleted(DeviceError):
+    pass
+
+class DeviceNotFound(DeviceError):
+    pass
-- 
2.13.0

[PATCH v2 03/47] add lnst.Common.DeviceRef

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the DeviceRef class that will be used by the Controller-Slave
communication protocol.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/DeviceRef.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 lnst/Common/DeviceRef.py

diff --git a/lnst/Common/DeviceRef.py b/lnst/Common/DeviceRef.py
new file mode 100644
index 0000000..cd7ee15
--- /dev/null
+++ b/lnst/Common/DeviceRef.py
@@ -0,0 +1,19 @@
+"""
+Defines the DeviceRef class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+class DeviceRef(object):
+    """Device reference transferable over network
+
+    Used in Controller-Slave commucation protocol.
+    """
+    def __init__(self, if_index):
+        self.if_index = int(if_index)
-- 
2.13.0

[PATCH v2 04/47] add lnst.Common.IpAddress

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines BaseIpAddress and derived classes and the IpAddress factory
method. We define our own instead of using the ipaddress module because
it doesn't always fit how LNST works and would require workarounds.
Defining our own classes for ip address handling therefore simplifies
our code.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * modify exception string.
---
 lnst/Common/IpAddress.py | 99 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 lnst/Common/IpAddress.py

diff --git a/lnst/Common/IpAddress.py b/lnst/Common/IpAddress.py
new file mode 100644
index 0000000..0bb5bec
--- /dev/null
+++ b/lnst/Common/IpAddress.py
@@ -0,0 +1,99 @@
+"""
+Defines BaseIpAddress and derived classes and the IpAddress factory method.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import re
+from socket import inet_pton, AF_INET, AF_INET6
+from lnst.Common.LnstError import LnstError
+
+class BaseIpAddress(object):
+    def __init__(self, addr):
+        self.addr, self.prefixlen = self._parse_addr(addr)
+
+        self.family = None
+
+    def __str__(self):
+        return str(self.addr)
+
+    def __eq__(self, other):
+        if self.addr != other.addr or\
+           self.prefixlen != other.prefixlen:
+            return False
+        else:
+            return True
+
+    @staticmethod
+    def _parse_addr(addr):
+        raise NotImplementedError()
+
+class Ip4Address(BaseIpAddress):
+    def __init__(self, addr):
+        super(Ip4Address, self).__init__(addr)
+
+        self.family = AF_INET
+
+    @staticmethod
+    def _parse_addr(addr):
+        addr = addr.split('/')
+        if len(addr) == 1:
+            addr = addr[0]
+            prefixlen = 32
+        elif len(addr) == 2:
+            addr, prefixlen = addr
+            prefixlen = int(prefixlen)
+        else:
+            raise LnstError("Invalid IPv4 format.")
+
+        try:
+            inet_pton(AF_INET, addr)
+        except:
+            raise LnstError("Invalid IPv4 format.")
+
+        return addr, prefixlen
+
+class Ip6Address(BaseIpAddress):
+    def __init__(self, addr):
+        super(Ip6Address, self).__init__(addr)
+
+        self.family = AF_INET6
+
+    @staticmethod
+    def _parse_addr(addr):
+        addr = addr.split('/')
+        if len(addr) == 1:
+            addr = addr[0]
+            prefixlen = 128
+        elif len(addr) == 2:
+            addr, prefixlen = addr
+            prefixlen = int(prefixlen)
+        else:
+            raise LnstError("Invalid IPv6 format.")
+
+        try:
+            type(inet_pton(AF_INET6, addr))
+        except:
+            raise LnstError("Invalid IPv6 format.")
+
+        return addr, prefixlen
+
+def IpAddress(addr):
+    """Factory method to create a BaseIpAddress object"""
+    if isinstance(addr, BaseIpAddress):
+        return addr
+    #TODO add switches for host, interface etc...
+    elif isinstance(addr, str):
+        try:
+            return Ip4Address(addr)
+        except:
+            return Ip6Address(addr)
+    else:
+        raise LnstError("Value must be a BaseIpAddress or string object."
+                        "Not {}".format(type(addr)))
-- 
2.13.0

[PATCH v2 05/47] add lnst.Common.Parameters

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This module defines the Param class, it's type specific derivatives
(IntParam, StrParam) and the Parameters class which serves as a container for
Param instances. This will be used to specify parameters for recipes and
for test modules.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * add docstring explaining the __getattribute__ magic
    * make val a property
    * rename x to attr in Parameters dir() iterations
    * fix IpParam value setter for string values
    * modify IpParam Exception string
---
 lnst/Common/Parameters.py | 133 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100644 lnst/Common/Parameters.py

diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py
new file mode 100644
index 0000000..dcf9bab
--- /dev/null
+++ b/lnst/Common/Parameters.py
@@ -0,0 +1,133 @@
+"""
+This module defines the Param class, it's type specific derivatives
+(IntParam, StrParam) and the Parameters class which serves as a container for
+Param instances. This can be used by a BaseRecipe class to specify
+optional/mandatory parameters for the entire test, or by HostReq and DeviceReq
+classes to define specific parameters needed for the matching algorithm.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.IpAddress import BaseIpAddress, IpAddress
+from lnst.Common.LnstError import LnstError
+
+class ParamError(LnstError):
+    pass
+
+class Param(object):
+    def __init__(self, mandatory=False, default=None):
+        self.mandatory = mandatory
+        self.default = default
+        self._val = None
+        self.set = False
+        if self.default:
+            self.val = self.default
+
+    @property
+    def val(self):
+        return self._val
+
+    @val.setter
+    def val(self, value):
+        self._val = value
+        self.set = True
+
+    def __str__(self):
+        return str(self.val)
+
+class IntParam(Param):
+    @Param.val.setter
+    def val(self, value):
+        try:
+            self._val = int(value)
+        except:
+            raise ParamError("Value must be a valid integer")
+        self.set = True
+
+    def __int__(self):
+        return self.val
+
+class FloatParam(Param):
+    @Param.val.setter
+    def val(self, value):
+        try:
+            self._val = float(value)
+        except:
+            raise ParamError("Value must be a valid float")
+        self.set = True
+
+    def __float__(self):
+        return self.val
+
+class StrParam(Param):
+    @Param.val.setter
+    def val(self, value):
+        try:
+            self._val = str(value)
+        except:
+            raise ParamError("Value must be a string")
+        self.set = True
+
+class IpParam(Param):
+    @Param.val.setter
+    def val(self, value):
+        if isinstance(value, BaseIpAddress):
+            self._val = value
+        elif isinstance(value, str):
+            self._val = IpAddress(value)
+        else:
+            raise ParamError("Value must be a BaseIpAddress or string object."
+                             "Not {}".format(type(value)))
+        self.set = True
+
+class Parameters(object):
+    def __getattribute__(self, name):
+        """
+        Overriding the default __getattribute__ method is important for being
+        able to deepcopy a Parameters object while also allowing to return None
+        for undefined Parameter names. This is because the copy module relies
+        on an exception being raised for certain private attributes and
+        returning None would break it.
+        """
+        if name[:2] == "__" or name[:1] == "_":
+            return object.__getattribute__(self, name)
+
+        try:
+            return object.__getattribute__(self, name)
+        except:
+            return None
+
+    def __iter__(self):
+        for attr in dir(self):
+            val = getattr(self, attr)
+            if isinstance(val, Param):
+                yield (attr, val)
+
+    def _to_dict(self):
+        res = {}
+        for name, param in self:
+            res[name] = str(param.val)
+        return res
+
+    def _from_dict(self, d):
+        for name, val in d.items():
+            if isinstance(val, Param):
+                setattr(self, name, val)
+            else:
+                new_param = StrParam()
+                new_param.val = val
+                setattr(self, name, new_param)
+
+    def __str__(self):
+        result = ""
+        for attr in dir(self):
+            val = getattr(self, attr)
+            if isinstance(val, Param):
+                result += "%s = %s\n" % (attr, str(val))
+        return result
-- 
2.13.0

[PATCH v2 06/47] add lnst.Common.TestModule

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the BaseTestModule class and the TestModuleError exception.

BaseTestModule is a Base class for test modules, all user defined
testmodule classes should inherit from this class. The class itself
defines the interface for a test module that is required by LNST and
implements Parameter checking.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/TestModule.py | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
 create mode 100644 lnst/Common/TestModule.py

diff --git a/lnst/Common/TestModule.py b/lnst/Common/TestModule.py
new file mode 100644
index 0000000..10e5b8f
--- /dev/null
+++ b/lnst/Common/TestModule.py
@@ -0,0 +1,67 @@
+"""
+Defines the BaseTestModule class and the TestModuleError exception.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import copy
+from lnst.Common.Parameters import Parameters, Param
+from lnst.Common.LnstError import LnstError
+
+class TestModuleError(LnstError):
+    """Exception used by BaseTestModule and derived
classes"""
+    pass
+
+class BaseTestModule(object):
+    """Base class for test modules
+
+    All user defined testmodule classes should inherit from this class. The
+    class itself defines the interface for a test module that is required by
+    LNST - the virtual run method.
+
+    It also implements the __init__ method that should be called by the derived
+    classes as it implements Parameter checking.
+
+    Derived classes can define the test parameters by assigning 'Param'
+    instances to class attributes, these will be parsed during initialization
+    and copied to the self.params instance attribute and loaded with values
+    provided to the __init__ method. This will also check mandatory attributes.
+    """
+    def __init__(self, **kwargs):
+        """
+        Args:
+            kwargs -- dictionary of arbitrary named arguments that correspond
+                to class attributes (Param type). Values will be parsed and
+                set to Param instances under the self.params object.
+        """
+        #by defaults loads the params into self.params - no checks pseudocode:
+        self.params = Parameters()
+        for x in dir(self):
+            val = getattr(self, x)
+            if isinstance(val, Param):
+                setattr(self.params, x, copy.deepcopy(val))
+
+        for name, val in kwargs.items():
+            try:
+                param = getattr(self.params, name)
+            except:
+                raise TestModuleError("Unknown parameter {}".format(name))
+            param.val = val
+
+        for name, param in self.params:
+            if param.mandatory and not param.set:
+                raise TestModuleError("Parameter {} is
mandatory".format(name))
+
+        self._res_data = None
+
+    def run(self):
+        raise NotImplementedError("Method 'run' MUST be defined")
+
+    def _get_res_data(self):
+        return self._res_data
-- 
2.13.0

[PATCH v2 07/47] add lnst.Common.JobError

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This module defines the common JobError exception that will be used by
both the Controller and the Slave Job classes.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/JobError.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
 create mode 100644 lnst/Common/JobError.py

diff --git a/lnst/Common/JobError.py b/lnst/Common/JobError.py
new file mode 100644
index 0000000..a86a8be
--- /dev/null
+++ b/lnst/Common/JobError.py
@@ -0,0 +1,22 @@
+"""
+This module defines the common JobError exception used by both the Controller
+and the Slave
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.LnstError import LnstError
+
+class JobError(LnstError):
+    """Base class for client errors."""
+    def __init__(self, s):
+        self._s = s
+
+    def __str__(self):
+        return "JobError: " + str(self._s)
-- 
2.13.0

[PATCH v2 08/47] add lnst.Controller.Common

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Common Controller module. At the moment it only defines the
ControllerError exception class, but other common functionality can be
added later.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Common.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
 create mode 100644 lnst/Controller/Common.py

diff --git a/lnst/Controller/Common.py b/lnst/Controller/Common.py
new file mode 100644
index 0000000..c9e5771
--- /dev/null
+++ b/lnst/Controller/Common.py
@@ -0,0 +1,19 @@
+"""
+Common Controller module. At the moment it only defines the ControllerError
+exception class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import os
+import sys
+from lnst.Common.LnstError import LnstError
+
+class ControllerError(LnstError):
+    pass
-- 
2.13.0

[PATCH v2 09/47] add lnst.Devices

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This package contains the implementation of Device
configuration/manipulation classes that are currently supported by LNST.
This implementation has been moved into a separate package that will be
distributed together with the Controller package. These will then be
sent to the Slave during recipe execution so the slave can use them when
required.

The reasoning behind this is that it is often easier to update or make
changes on the Controller than on all the Slaves. Making the Slave
simpler and independent of the implementation of device configuration
means we don't need to update all our slaves as often and also that
changes to the Device configuration implementation don't require any
changes in Slave or Controller code.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * make Device a Meta class and register RemoteDevice as implementing
      the Device interface
    * removed unnecessary imports
    * added MasterDevice class for slave* methods
    * made master_set to be a 'master' property setter
    * renamed {add, del}_route to route_{add, del}
    * renamed set_speed to speed_set
    * renamed set_autoneg to autoneg_set
    * removed modprobes where unnecessary
    * link_stats are now retrieved from netlink message
    * added link_stats64
    * OvsBridge calls parent _type_init for modprobe
    * remove SoftDevice import - not required
---
 lnst/Devices/BondDevice.py      |  38 +++++
 lnst/Devices/BridgeDevice.py    |  33 ++++
 lnst/Devices/Device.py          | 355 ++++++++++++++++++++++++++++++++++++++++
 lnst/Devices/MacvlanDevice.py   |  38 +++++
 lnst/Devices/MasterDevice.py    |  33 ++++
 lnst/Devices/OvsBridgeDevice.py | 115 +++++++++++++
 lnst/Devices/RemoteDevice.py    |  98 +++++++++++
 lnst/Devices/SoftDevice.py      |  51 ++++++
 lnst/Devices/TeamDevice.py      |  60 +++++++
 lnst/Devices/VethDevice.py      |  58 +++++++
 lnst/Devices/VethPair.py        |  24 +++
 lnst/Devices/VirtualDevice.py   |  98 +++++++++++
 lnst/Devices/VlanDevice.py      |  35 ++++
 lnst/Devices/VtiDevice.py       |  71 ++++++++
 lnst/Devices/VxlanDevice.py     |  65 ++++++++
 lnst/Devices/__init__.py        |  43 +++++
 16 files changed, 1215 insertions(+)
 create mode 100644 lnst/Devices/BondDevice.py
 create mode 100644 lnst/Devices/BridgeDevice.py
 create mode 100644 lnst/Devices/Device.py
 create mode 100644 lnst/Devices/MacvlanDevice.py
 create mode 100644 lnst/Devices/MasterDevice.py
 create mode 100644 lnst/Devices/OvsBridgeDevice.py
 create mode 100644 lnst/Devices/RemoteDevice.py
 create mode 100644 lnst/Devices/SoftDevice.py
 create mode 100644 lnst/Devices/TeamDevice.py
 create mode 100644 lnst/Devices/VethDevice.py
 create mode 100644 lnst/Devices/VethPair.py
 create mode 100644 lnst/Devices/VirtualDevice.py
 create mode 100644 lnst/Devices/VlanDevice.py
 create mode 100644 lnst/Devices/VtiDevice.py
 create mode 100644 lnst/Devices/VxlanDevice.py
 create mode 100644 lnst/Devices/__init__.py

diff --git a/lnst/Devices/BondDevice.py b/lnst/Devices/BondDevice.py
new file mode 100644
index 0000000..5d75546
--- /dev/null
+++ b/lnst/Devices/BondDevice.py
@@ -0,0 +1,38 @@
+"""
+Defines the BondDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.MasterDevice import MasterDevice
+
+class BondDevice(MasterDevice):
+    _name_template = "t_bond"
+
+    def create(self):
+        exec_cmd("ip link add %s type bond" % self.name)
+
+    def _get_bond_dir(self):
+        return "/sys/class/net/%s/bonding" % self.name
+
+    def set_option(self, option, value):
+        if option == "primary":
+            '''
+            "primary" option is not direct value but it's
+            a Device reference
+            '''
+            value = value.name
+        exec_cmd('echo "%s" > %s/%s' % (value,
+                                        self._get_bond_dir(),
+                                        option))
+
+    def set_options(self, options):
+        for option, value in options:
+            self.set_option(option, value)
diff --git a/lnst/Devices/BridgeDevice.py b/lnst/Devices/BridgeDevice.py
new file mode 100644
index 0000000..c9678c4
--- /dev/null
+++ b/lnst/Devices/BridgeDevice.py
@@ -0,0 +1,33 @@
+"""
+Defines the BridgeDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.MasterDevice import MasterDevice
+
+class BridgeDevice(MasterDevice):
+    _name_template = "t_br"
+
+    def create(self):
+        exec_cmd("ip link add dev {} type bridge".format(self._name))
+
+    def _get_bridge_dir(self):
+        return "/sys/class/net/%s/bridge" % self.name
+
+    def set_option(self, option, value):
+        #TODO redo to work with iproute
+        exec_cmd('echo "%s" > %s/%s' % (value,
+                                        self._get_bridge_dir(),
+                                        option))
+
+    def set_options(self, options):
+        for option, value in options:
+            self.set_option(option, value)
diff --git a/lnst/Devices/Device.py b/lnst/Devices/Device.py
new file mode 100644
index 0000000..8ba59f2
--- /dev/null
+++ b/lnst/Devices/Device.py
@@ -0,0 +1,355 @@
+"""
+Defines the Device class implementing the common methods for all device types.
+Every other device type needs to inherit from this class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import re
+from abc import ABCMeta
+from lnst.Common.NetUtils import normalize_hwaddr
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Common.DeviceError import DeviceError, DeviceDeleted
+from lnst.Common.IpAddress import Ip4Address, Ip6Address, IpAddress
+
+try:
+    from pyroute2.netlink.iproute import RTM_NEWLINK
+    from pyroute2.netlink.iproute import RTM_NEWADDR
+    from pyroute2.netlink.iproute import RTM_DELADDR
+except ImportError:
+    from pyroute2.iproute import RTM_NEWLINK
+    from pyroute2.iproute import RTM_NEWADDR
+    from pyroute2.iproute import RTM_DELADDR
+
+class Device(object):
+    """The base Device class
+
+    Implemented using the pyroute2 package to access different attributes of
+    a kernel netdevice object.
+    Changing attributes of a netdevice is right now implemented by calling
+    shell commands (e.g. from iproute2 package).
+
+    The Controller-Slave communication is implemented in such a way that all
+    public methods defined in this and derived class are directly available
+    as a tester facing API.
+    """
+    __metaclass__ = ABCMeta
+
+    def __init__(self, if_manager):
+        self.if_index = None
+        self._nl_msg = None
+        self._devlink = None
+        self._if_manager = if_manager
+        self._enabled = True
+        self._deleted = False
+
+        self._ip_addrs = []
+        # TODO self._netns = None ???
+
+    def create(self):
+        """Creates a new netdevice of the corresponding type
+
+        Method to be implemented by derived classes where applicable.
+        """
+        msg = "Can't create a hardware ethernet device."
+        raise DeviceError(msg)
+
+    def destroy(self):
+        """Destroys the netdevice of the corresponding type
+
+        For the basic eth device it calls the destroy method of it's master
+        device (if there is one). Flushes the configured IP addresses and sets
+        the device 'down'.
+        """
+        if self.master:
+            self.master.destroy()
+        self.ip_flush()
+        self.down()
+        return True
+
+    def enable(self):
+        """Enables the Device object"""
+        self._enabled = True
+
+    def disable(self):
+        """Disables the Device object
+
+        When a Device object is disabled, any calls to it's methods will result
+        in a "no operation", however attribute access will still work.
+
+        The justification for this is to disable the Device used by the
+        Controller-Slave connection to avoid accidental disconnects.
+        """
+        self._enabled = False
+
+    def __getattribute__(self, name):
+        what = object.__getattribute__(self, name)
+
+        if object.__getattribute__(self, "_deleted"):
+            raise DeviceDeleted()
+
+        if not callable(what):
+            return what
+        else:
+            if (object.__getattribute__(self, "_enabled") or
+                    name in ["enable", "disable"]):
+                return what
+            else:
+                def noop(*args, **kwargs):
+                    pass
+                return noop
+
+    def _set_devlink(self, devlink_port_data):
+        self._devlink = devlink_port_data
+
+    def _init_netlink(self, nl_msg):
+        self.if_index = nl_msg['index']
+
+        self._nl_msg = nl_msg
+
+    def _update_netlink(self, nl_msg):
+        if self.if_index != nl_msg['index']:
+            msg = "if_index of netlink message (%s) doesn't match "\
+                  "the device's (%s)." % (nl_msg['index'],
self.if_index)
+            raise DeviceError(msg)
+
+        if nl_msg['header']['type'] == RTM_NEWLINK:
+            if self.if_index != nl_msg['index']:
+                raise DeviceError("RTM_NEWLINK message passed to incorrect "\
+                                  "Device object.")
+
+            self._nl_msg = nl_msg
+        elif nl_msg['header']['type'] == RTM_NEWADDR:
+            if self.if_index != nl_msg['index']:
+                raise DeviceError("RTM_NEWADDR message passed to incorrect "\
+                                  "Device object.")
+
+            addr = IpAddress(nl_msg.get_attr('IFA_ADDRESS'))
+            addr.prefixlen = nl_msg["prefixlen"]
+
+            if addr not in self._ip_addrs:
+                self._ip_addrs.append(addr)
+        elif nl_msg['header']['type'] == RTM_DELADDR:
+            if self.if_index != nl_msg['index']:
+                raise DeviceError("RTM_DELADDR message passed to incorrect "\
+                                  "Device object.")
+
+            addr = IpAddress(nl_msg.get_attr('IFA_ADDRESS'))
+            addr.prefixlen = nl_msg["prefixlen"]
+
+            if addr in self._ip_addrs:
+                self._ip_addrs.remove(addr)
+
+    @property
+    def ifi_type(self):
+        """ifi_type attribute
+
+        Returns the integer type of the device as reported by the kernel.
+        """
+        return self._nl_msg['ifi_type']
+
+    @property
+    def name(self):
+        """name attribute
+
+        Returns string name of the device as reported by the kernel.
+        """
+        return self._nl_msg.get_attr("IFLA_IFNAME")
+
+    @property
+    def hwaddr(self):
+        """hwaddr attribute
+
+        Returns string hardware address of the device as reported by the kernel.
+        """
+        return normalize_hwaddr(self._nl_msg.get_attr("IFLA_ADDRESS"))
+
+    @property
+    def state(self):
+        """state attribute
+
+        Returns string state of the device as reported by the kernel.
+        """
+        #TODO check flags for admin up and lower up!!!
+        #TODO or expand this to check all possibilities?
+        #TODO also, add passive wait until lower up, with timeout
+        return self._nl_msg.get_attr("IFLA_OPERSTATE")
+
+    @property
+    def ips(self):
+        """list of configured ip addresses
+
+        Returns list of BaseIpAddress objects.
+        """
+        return self._ip_addrs
+
+    @property
+    def mtu(self):
+        """mtu attribute
+
+        Returns integer mtu as reported by the kernel.
+        """
+        return self._nl_msg.get_attr("IFLA_MTU")
+
+    @property
+    def master(self):
+        """master device
+
+        Returns Device object of the master device or None when the device has
+        no master.
+        """
+        master_if_index = self._nl_msg.get_attr("IFLA_MASTER")
+        if master_if_index is not None:
+            return self._if_manager.get_device(master_if_index)
+        else:
+            return None
+
+    @master.setter
+    def master(self, dev):
+        """set dev as the master of this device
+
+        Args:
+            dev -- accepts a Device object of the master object.
+                When None, removes the current master from the Device."""
+        if isinstance(dev, Device):
+            exec_cmd("ip link set %s master %s" % (self.name, dev.name))
+        elif dev is None:
+            exec_cmd("ip link set %s nomaster" % self.name)
+        else:
+            raise DeviceError("Invalid dev argument.")
+
+    @property
+    def driver(self):
+        """driver attribute
+
+        Returns string name of the device driver based on an ethtool -i call
+        """
+        if self.ifi_type == 772:  #loopback ifi type
+            return 'loopback'
+        out, _ = exec_cmd("ethtool -i %s" % self.name, False, False, False)
+        match = re.search("^driver: (.*)$", out, re.MULTILINE)
+        if match is not None:
+            return match.group(1)
+        else:
+            return None
+
+    @property
+    def link_stats(self):
+        """Link statistics
+
+        Returns dictionary of interface statistics, IFLA_STATS
+        """
+        return self._nl_msg.get_attr("IFLA_STATS")
+
+    @property
+    def link_stats64(self):
+        """Link statistics
+
+        Returns dictionary of interface statistics, IFLA_STATS64
+        """
+        return self._nl_msg.get_attr("IFLA_STATS64")
+
+    def _clear_ips(self):
+        self._ip_addrs = []
+
+    def _clear_tc_qdisc(self):
+        exec_cmd("tc qdisc replace dev %s root pfifo" % self.name)
+        out, _ = exec_cmd("tc filter show dev %s" % self.name)
+        ingress_handles = re.findall("ingress (\\d+):", out)
+        for ingress_handle in ingress_handles:
+            exec_cmd("tc qdisc del dev %s handle %s: ingress" %
+                     (self.name, ingress_handle))
+        out, _ = exec_cmd("tc qdisc show dev %s" % self.name)
+        ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out)
+        if len(ingress_qdiscs) != 0:
+                exec_cmd("tc qdisc del dev %s ingress" % self.name)
+
+    def _clear_tc_filters(self):
+        out, _ = exec_cmd("tc filter show dev %s" % self.name)
+        egress_prefs = re.findall("pref (\\d+) .* handle", out)
+
+        for egress_pref in egress_prefs:
+            exec_cmd("tc filter del dev %s pref %s" % (self.name,
+                     egress_pref))
+
+    def ip_add(self, addr):
+        """add an ip address
+
+        Args:
+            addr -- accepts a BaseIpAddress object
+        """
+        if addr not in self.ips:
+            exec_cmd("ip addr add %s/%d dev %s" % (addr, addr.prefixlen,
+                                                   self.name))
+
+    def ip_del(self, addr):
+        """remove an ip address
+
+        Args:
+            addr -- accepts a BaseIpAddress object
+        """
+        if addr in self.ips:
+            exec_cmd("ip addr del %s/%d dev %s" % (addr, addr.prefixlen,
+                                                   self.name))
+
+    def ip_flush(self):
+        """flush all ip addresses of the device"""
+        for ip in self.ips:
+            self.ip_del(ip)
+
+    def up(self):
+        """set device up"""
+        exec_cmd("ip link set %s up" % self.name)
+
+    def down(self):
+        """set device down"""
+        exec_cmd("ip link set %s down" % self.name)
+
+    def route_add(self, dest):
+        """add specified route for this device
+
+        Args:
+            dest -- string accepted by the "ip route add " command
+        """
+        exec_cmd("ip route add %s dev %s" % (dest, self.name))
+
+    def route_del(self, dest):
+        """remove specified route for this device
+
+        Args:
+            dest -- string accepted by the "ip route del " command
+        """
+        exec_cmd("ip route del %s dev %s" % (dest, self.name))
+
+    def _get_if_data(self):
+        if_data = {"if_index": self.if_index,
+                   "hwaddr": self.hwaddr,
+                   "name": self.name,
+                   "ip_addrs": self.ips,
+                   "ifi_type": self.ifi_type,
+                   "state": self.state,
+                   "master": self.master,
+                   "mtu": self.mtu,
+                   "driver": self.driver,
+                   "devlink": self._devlink}
+        return if_data
+
+    def speed_set(self, speed):
+        """set the device speed
+
+        Also disables automatic speed negotiation
+
+        Args:
+            speed -- string accepted by the 'ethtool -s dev speed ' command
+        """
+        exec_cmd("ethtool -s %s speed %s autoneg off" % (self.name, speed))
+
+    def autoneg_set_on(self):
+        """enable automatic negotiation of speed for this
device"""
+        exec_cmd("ethtool -s %s autoneg on" % self.name)
diff --git a/lnst/Devices/MacvlanDevice.py b/lnst/Devices/MacvlanDevice.py
new file mode 100644
index 0000000..418cf6d
--- /dev/null
+++ b/lnst/Devices/MacvlanDevice.py
@@ -0,0 +1,38 @@
+"""
+Defines the MacvlanDevice class
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.SoftDevice import SoftDevice
+
+class MacvlanDevice(SoftDevice):
+    _name_template = "t_macvlan"
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(MacvlanDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._real_dev = kwargs["realdev"]
+        self._mode = kwargs.get("mode", None)
+        self._hwaddr = kwargs.get("hwaddr", None)
+
+    def create(self):
+        create_cmd = "ip link add link {} {}".format(self._real_dev.name,
+                                                     self.name)
+
+        if self._hwaddr is not None:
+            create_cmd += " address {}".format(self._hwaddr)
+
+        if self._mode is not None:
+            create_cmd += " mode {}".format(self._mode)
+
+        create_cmd += " type macvlan"
+
+        exec_cmd(create_cmd)
diff --git a/lnst/Devices/MasterDevice.py b/lnst/Devices/MasterDevice.py
new file mode 100644
index 0000000..afd2060
--- /dev/null
+++ b/lnst/Devices/MasterDevice.py
@@ -0,0 +1,33 @@
+"""
+Defines the MasterDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Devices.SoftDevice import SoftDevice
+
+class MasterDevice(SoftDevice):
+    """Common class for all master device types
+
+    Implements the slaves attribute getter and the slave_{add, del} methods.
+    """
+    @property
+    def slaves(self):
+        ret = []
+
+        for dev in self._if_manager.get_devices():
+            if dev.master is self:
+                ret.append(dev)
+        return ret
+
+    def slave_add(self, dev):
+        dev.master_set(self)
+
+    def slave_del(self, dev):
+        dev.master_set(None)
diff --git a/lnst/Devices/OvsBridgeDevice.py b/lnst/Devices/OvsBridgeDevice.py
new file mode 100644
index 0000000..26f75f4
--- /dev/null
+++ b/lnst/Devices/OvsBridgeDevice.py
@@ -0,0 +1,115 @@
+"""
+Defines the OvsBridgeDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.Device import Device, DeviceError
+from lnst.Devices.SoftDevice import SoftDevice
+
+class OvsBridgeDevice(SoftDevice):
+    _name_template = "t_ovsbr"
+
+    _modulename = "openvswitch"
+
+    @classmethod
+    def _type_init(cls):
+        if not cls._type_initialized:
+            super(OvsBridgeDevice, cls)._type_init()
+
+            exec_cmd("mkdir -p /var/run/openvswitch/")
+            exec_cmd("ovsdb-server --detach --pidfile "\
+                     "--remote=punix:/var/run/openvswitch/db.sock",
+                     die_on_err=False)
+            exec_cmd("ovs-vswitchd --detach --pidfile", die_on_err=False)
+
+            cls._type_initialized = True
+
+    def create(self):
+        exec_cmd("ovs-vsctl add-br %s" % self.name)
+
+    def destroy(self):
+        exec_cmd("ovs-vsctl del-br %s" % self.name)
+
+    def port_add(self, dev, **kwargs):
+        options = ""
+        for opt_name, opt_value in kwargs.items():
+            options += " %s=%s" % (opt_name, opt_value)
+
+        exec_cmd("ovs-vsctl add-port %s %s%s" % (self.name, dev.name,
options))
+
+    def port_del(self, dev):
+        if isinstance(dev, Device):
+            exec_cmd("ovs-vsctl del-port %s %s" % (self.name, dev.name))
+        elif isinstance(dev, str):
+            exec_cmd("ovs-vsctl del-port %s %s" % (self.name, dev))
+        else:
+            raise DeviceError("Invalid port_del argument %s" % str(dev))
+
+    def bond_add(self, port_name, devices, **kwargs):
+        dev_names = ""
+        for dev in devices:
+            dev_names += " %s" % dev.name
+
+        options = ""
+        for opt_name, opt_value in kwargs.items():
+            options += " %s=%s" % (opt_name, opt_value)
+
+        exec_cmd("ovs-vsctl add-bond %s %s %s %s" % (self.name, port_name,
+                                                     dev_names, options))
+
+    def bond_del(self, dev):
+        self.port_del(dev)
+
+    def internal_port_add(self, **kwargs):
+        name = self._if_manager.assign_name("int")
+
+        options = ""
+        for opt_name, opt_value in kwargs.items():
+            if opt_name == "name":
+                name = opt_value
+                continue
+
+            options += " %s=%s" % (opt_name, opt_value)
+
+        exec_cmd("ovs-vsctl add-port %s %s -- set Interface %s "\
+                 "type=internal %s" % (self.name, name,
+                                       name, options))
+
+        dev = self._if_manager.get_device_by_name(name)
+        return dev
+
+    def tunnel_add(self, tunnel_type, options):
+        name = self._if_manager.assign_name(tunnel_type)
+
+        options = ""
+        for opt_name, opt_value in options.items():
+            if opt_name == "name":
+                name = opt_value
+                continue
+
+            options += " %s=%s" % (opt_name, opt_value)
+
+        exec_cmd("ovs-vsctl add-port %s %s -- set Interface %s "\
+                 "type=%s %s" % (self.name, name, name,
+                                 tunnel_type, options))
+
+    def tunnel_del(self, name):
+        self.port_del(name)
+
+    def flow_add(self, entry):
+        exec_cmd("ovs-ofctl add-flow %s '%s'" % (self.name, entry))
+
+    def flows_add(self, entries):
+        for entry in entries:
+            self.flow_add(entry)
+
+    def flows_del(self, entry):
+        exec_cmd("ovs-ofctl del-flows %s" % (self.name))
diff --git a/lnst/Devices/RemoteDevice.py b/lnst/Devices/RemoteDevice.py
new file mode 100644
index 0000000..d15d0c5
--- /dev/null
+++ b/lnst/Devices/RemoteDevice.py
@@ -0,0 +1,98 @@
+"""
+Defines the RemoteDevice class. This class wraps all other Device classes
+when creating device instances on the Controller.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Devices.Device import Device
+from lnst.Common.DeviceError import DeviceDeleted
+
+def remotedev_decorator(cls):
+    def func(*args, **kwargs):
+        return RemoteDevice(cls, args, kwargs)
+    return func
+
+class RemoteDevice(object):
+    """Wraps all other Device classes on the Controller
+
+    Ensures that all public methods of Device objects also act as the tester
+    facing API even though the Device objects are instantiated on the Slave,
+    not where the recipe script is actually running.
+    """
+    def __init__(self, dev_cls, args=[], kwargs={}):
+        self.__dev_cls = dev_cls
+        self.__dev_args = args
+        self.__dev_kwargs = kwargs
+
+        self.host = None
+        self.if_index = None
+        self.deleted = False
+
+    @property
+    def _dev_cls(self):
+        return self.__dev_cls
+
+    @property
+    def _dev_args(self):
+        return self.__dev_args
+
+    @property
+    def _dev_kwargs(self):
+        return self.__dev_kwargs
+
+    def _get_dev_cls(self):
+        return self._dev_cls
+
+    def __getattr__(self, name):
+        attr = getattr(self._dev_cls, name)
+
+        if self.deleted:
+            raise DeviceDeleted("This device was deleted on the slave and does not
exist anymore.")
+
+        if callable(attr):
+            def dev_method(*args, **kwargs):
+                return self.host.rpc_call("dev_method", self.if_index,
+                                          name, args, kwargs)
+            return dev_method
+        else:
+            return self.host.rpc_call("dev_attr", self.if_index, name)
+
+    def __iter__(self):
+        for x in dir(self._dev_cls):
+            if x[0] == '_' or x[0:1] == "__":
+                continue
+            attr = getattr(self._dev_cls, x)
+
+            if not callable(attr):
+                yield (x, getattr(self, x))
+
+    def _match_update_data(self, data):
+        return False
+
+class PairedRemoteDevice(RemoteDevice):
+    """RemoteDevice class for paired Devices (such as
veth)"""
+    def __init__(self, peer, dev_cls, args=[], kwargs={}):
+        super(PairedRemoteDevice, self).__init__(dev_cls, args, kwargs)
+
+        self._peer = peer
+
+    @property
+    def _dev_kwargs(self):
+        ret = super(PairedRemoteDevice, self)._dev_kwargs
+        ret["peer_if_id"] = self._peer.if_index
+        return ret
+
+#register the RemoteDevice class as implementing the interface of the Device
+#class - this is true because it just proxies method/attribute calls to the
+#remote Slave where the correct method gets called.
+#registering the RemoteDevice class as an implementation of the Device
+#Interface is required for isinstance() checks in Common code -- Device is
+#available on both Controller and Slave, but RemoteDevice only on Controller
+Device.register(RemoteDevice)
diff --git a/lnst/Devices/SoftDevice.py b/lnst/Devices/SoftDevice.py
new file mode 100644
index 0000000..3dc3b7f
--- /dev/null
+++ b/lnst/Devices/SoftDevice.py
@@ -0,0 +1,51 @@
+"""
+Defines the SoftDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Common.DeviceError import DeviceError
+from lnst.Devices.Device import Device
+
+class SoftDevice(Device):
+    _name_template = "soft_dev"
+
+    _modulename = ""
+    _moduleparams = ""
+    _type_initialized = False
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(SoftDevice, self).__init__(ifmanager)
+
+        self._name = kwargs.get("name", None)
+        if self._name is None:
+            self._name = ifmanager.assign_name(self._name_template)
+
+        self._type_init()
+
+    @classmethod
+    def _type_init(cls):
+        if cls._modulename and not cls._type_initialized:
+            exec_cmd("modprobe %s %s" % (cls._modulename, cls._moduleparams))
+            cls._type_initialized = True
+
+    @property
+    def name(self):
+        try:
+            return super(SoftDevice, self).name
+        except:
+            return self._name
+
+    def create(self):
+        msg = "Classes derived from SoftDevice MUST define a create method."
+        raise DeviceError(msg)
+
+    def destroy(self):
+        exec_cmd("ip link del dev %s" % self.name)
diff --git a/lnst/Devices/TeamDevice.py b/lnst/Devices/TeamDevice.py
new file mode 100644
index 0000000..14ad0ad
--- /dev/null
+++ b/lnst/Devices/TeamDevice.py
@@ -0,0 +1,60 @@
+"""
+Defines the TeamDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import re
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Common.Utils import bool_it
+from lnst.Devices.MasterDevice import MasterDevice
+
+def prepare_json_str(json_str):
+    if not json_str:
+        return "{}"
+    json_str = json_str.replace('"', '\\"')
+    json_str = re.sub('\s+', ' ', json_str)
+    return json_str
+
+class TeamDevice(MasterDevice):
+    _name_template = "t_team"
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(TeamDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._config = kwargs.get("config", None)
+        self._dbus = not bool_it(kwargs.get("disable_dbus", False))
+
+    @property
+    def config(self):
+        return self._config
+
+    @property
+    def dbus(self):
+        return self._dbus
+
+    def create(self):
+        teamd_config = prepare_json_str(self.config)
+
+        exec_cmd("teamd -r -d -c \"%s\" -t %s %s" %\
+                    (teamd_config,
+                     self.name,
+                     " -D" if self.dbus else ""))
+
+    def destroy(self):
+        exec_cmd("teamd -k -t %s" % self.name)
+
+    def slave_add(self, dev, port_config=None):
+        exec_cmd("teamdctl %s %s port config update %s \"%s\"" %\
+                    (" -D" if self.dbus else "",
+                     self.name,
+                     dev.name,
+                     prepare_json_str(port_config)))
+
+        dev.master_set(self)
diff --git a/lnst/Devices/VethDevice.py b/lnst/Devices/VethDevice.py
new file mode 100644
index 0000000..07a6c33
--- /dev/null
+++ b/lnst/Devices/VethDevice.py
@@ -0,0 +1,58 @@
+"""
+Defines the VethDevice and PairedVethDevice classes.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from copy import deepcopy
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.Device import Device
+from lnst.Devices.SoftDevice import SoftDevice
+
+class VethDevice(SoftDevice):
+    _name_template = "veth"
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(VethDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._name = kwargs.get("name", None)
+        self._peer_name = kwargs.get("peer_name", None)
+
+        if self._name is None:
+            self._name = ifmanager.assign_name(self._name_template)
+
+        if self._peer_name is None:
+            self._peer_name =
ifmanager.assign_name("peer_"+self._name_template)
+
+    def create(self):
+        exec_cmd("ip link add {name} type veth peer name {peer}".
+                 format(name=self.name,
+                        peer=self._peer_name))
+
+    @property
+    def peer(self):
+        if self._nl_msg is None:
+            return None
+
+        peer_if_id = self._nl_msg.get_attr("IFLA_LINK")
+        return self._if_manager.get_device(peer_if_id)
+
+class PairedVethDevice(VethDevice):
+    def __init__(self, ifmanager, *args, **kwargs):
+        Device.__init__(self, ifmanager)
+
+        self._peer_if_id = kwargs["peer_if_id"]
+
+    def create(self):
+        peer = self._if_manager.get_device(self._peer_if_id)
+        me = peer.peer
+        self._init_netlink(me._nl_msg)
+        self._ip_addrs = deepcopy(me._ip_addrs)
+
+        self._if_manager.replace_dev(self.if_index, self)
diff --git a/lnst/Devices/VethPair.py b/lnst/Devices/VethPair.py
new file mode 100644
index 0000000..2cf3ff1
--- /dev/null
+++ b/lnst/Devices/VethPair.py
@@ -0,0 +1,24 @@
+"""
+Defines the VethPair factory method.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Devices.VethDevice import VethDevice, PairedVethDevice
+from lnst.Devices.RemoteDevice import RemoteDevice, PairedRemoteDevice
+
+def VethPair(*args, **kwargs):
+    """Creates a pair of Veth Devices
+
+    Args:
+        args, kwargs passed to the VethDevice constructor on the Slave.
+    """
+    first = RemoteDevice(VethDevice, args, kwargs)
+    second = PairedRemoteDevice(first, PairedVethDevice)
+    return (first, second)
diff --git a/lnst/Devices/VirtualDevice.py b/lnst/Devices/VirtualDevice.py
new file mode 100644
index 0000000..49fe088
--- /dev/null
+++ b/lnst/Devices/VirtualDevice.py
@@ -0,0 +1,98 @@
+"""
+Defines the VirtualDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+from time import sleep
+from lnst.Common.Utils import check_process_running
+from lnst.Common.NetUtils import normalize_hwaddr
+from lnst.Devices.Device import Device, DeviceError
+from lnst.Devices.RemoteDevice import RemoteDevice
+
+# conditional support for libvirt
+if check_process_running("libvirtd"):
+    from lnst.Controller.VirtUtils import VirtNetCtl
+
+class VirtualDevice(RemoteDevice):
+    """Remote eth device created on the controller through libvirt
+
+    To support creation of new devices on virtual machines, we derive from
+    the RemoteDevice class and override the create method (which would be
+    remotely called on the slave for the Device class type). Instead, the
+    create method is called on the controller where libvirt is running.
+
+    The Tester shouldn't create instances of this class. They're created
+    automatically if matching virtual machines is allowed.
+
+    Theoretically if a match is virtual the tester could also dynamically add
+    devices during test execution, however this is NOT SUPPORTED at the moment.
+    """
+    def __init__(self, network, driver=None, hwaddr=None):
+        super(VirtualDevice, self).__init__(Device, None, None)
+
+        self.virt_driver = driver if driver is not None else "virtio"
+        self.orig_hwaddr = hwaddr
+        self._network = network
+
+    @property
+    def network(self):
+        return self._network
+
+    @network.setter
+    def network(self, network):
+        self._network = network
+
+    def _match_update_data(self, data):
+        if self.orig_hwaddr == data["hwaddr"]:
+            return True
+
+        return super(VirtualDevice, self)._match_update_data(data)
+
+    def create(self):
+        domain_ctl = self.host.get_domain_ctl()
+
+        if self.orig_hwaddr:
+            if self.host.get_dev_by_hwaddr(self.orig_hwaddr):
+                msg = "Device with hwaddr %s already exists" %
self.orig_hwaddr
+                raise DeviceError(msg)
+        else:
+            mac_pool = self.host.get_mac_pool()
+            while True:
+                hwaddr = normalize_hwaddr(mac_pool.get_addr())
+                if not self.host.get_dev_by_hwaddr(hwaddr):
+                    self.orig_hwaddr = hwaddr
+                    break
+
+        bridges = self.host.get_network_bridges()
+        if self.network in bridges:
+            net_ctl = bridges[self.network]
+        else:
+            bridges[self.network] = net_ctl = VirtNetCtl()
+            net_ctl.init()
+
+        net_name = net_ctl.get_name()
+
+        logging.info("Creating virtual device with hwaddr='%s' on machine
%s",
+                     self.orig_hwaddr, self.host.get_id())
+
+        domain_ctl.attach_interface(self.orig_hwaddr,
+                                    net_name,
+                                    self.virt_driver)
+        # The sleep here is necessary, because udev sometimes renames the
+        # newly created device
+        sleep(1)
+
+    def destroy(self):
+        logging.info("Destroying virtual device with hwaddr='%s' on machine
%s",
+                     self.orig_hwaddr, self.host.get_id())
+
+        domain_ctl = self.host.get_domain_ctl()
+        domain_ctl.detach_interface(self.orig_hwaddr)
diff --git a/lnst/Devices/VlanDevice.py b/lnst/Devices/VlanDevice.py
new file mode 100644
index 0000000..00cb63d
--- /dev/null
+++ b/lnst/Devices/VlanDevice.py
@@ -0,0 +1,35 @@
+"""
+Defines the VlanDevice class
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.SoftDevice import SoftDevice
+
+class VlanDevice(SoftDevice):
+    _name_template = "t_vlan"
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(VlanDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._real_dev = kwargs["real_dev"]
+        self._vlan_id = int(kwargs["vlan_id"])
+
+    @property
+    def real_dev(self):
+        return self._real_dev
+
+    @property
+    def vlan_id(self):
+        return self._vlan_id
+
+    def create(self):
+        exec_cmd("ip link add link %s %s type vlan id %d" %\
+                 (self.real_dev.name, self.name, self.vlan_id))
diff --git a/lnst/Devices/VtiDevice.py b/lnst/Devices/VtiDevice.py
new file mode 100644
index 0000000..53f6712
--- /dev/null
+++ b/lnst/Devices/VtiDevice.py
@@ -0,0 +1,71 @@
+"""
+Defines the VtiDevice and Vti6Device classes.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.Device import DeviceError
+from lnst.Devices.SoftDevice import SoftDevice
+
+class _BaseVtiDevice(SoftDevice):
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(_BaseVtiDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._key = kwargs["key"]
+        self._local = kwargs.get("local", None)
+        self._remote = kwargs.get("remote", None)
+        self._device = kwargs.get("dev", None)
+
+        if self.local is None and self.remote is None:
+            raise DeviceError("One of local/remote MUST be defined.")
+
+    @property
+    def key(self):
+        return self._key
+
+    @property
+    def local(self):
+        return self._local
+
+    @property
+    def remote(self):
+        return self._remote
+
+    @property
+    def device(self):
+        return self._device
+
+    @property
+    def vti_type(self):
+        raise NotImplementedError
+
+    def create(self):
+        exec_cmd("ip link add {name} type {type}{local}{remote}{key}{device}".
+                 format(name=self.name,
+                        type=self.vti_type,
+                        local=(" local " + str(self.local)
+                               if self.local
+                               else ""),
+                        remote=(" remote " + str(self.remote)
+                                if self.remote
+                                else ""),
+                        key=" key " + self.key,
+                        device=(" dev " + self.device.name
+                                if self.device
+                                else "")))
+
+
+class VtiDevice(_BaseVtiDevice):
+    vti_type = "vti"
+    _name_template = "t_vti"
+
+class Vti6Device(_BaseVtiDevice):
+    vti_type = "vti6"
+    _name_template = "t_ip6vti"
diff --git a/lnst/Devices/VxlanDevice.py b/lnst/Devices/VxlanDevice.py
new file mode 100644
index 0000000..079ced8
--- /dev/null
+++ b/lnst/Devices/VxlanDevice.py
@@ -0,0 +1,65 @@
+"""
+Defines the VxlanDevice class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.ExecCmd import exec_cmd
+from lnst.Devices.Device import DeviceError
+from lnst.Devices.SoftDevice import SoftDevice
+
+class VxlanDevice(SoftDevice):
+    _name_template = "t_vxlan"
+
+    def __init__(self, ifmanager, *args, **kwargs):
+        super(VxlanDevice, self).__init__(ifmanager, args, kwargs)
+
+        self._vxlan_id = int(kwargs["vxlan_id"])
+        self._real_dev = kwargs.get("real_dev", None)
+        self._group_ip = kwargs.get("group_ip", None)
+        self._remote_ip = kwargs.get("remote_ip", None)
+        self._dstport = int(kwargs.get("dst_port", 0))
+
+        if self.group_ip is None and self.remote_ip is None:
+            raise DeviceError("group or remote must be specified for vxlan")
+
+    @property
+    def real_dev(self):
+        return self._real_dev
+
+    @property
+    def vxlan_id(self):
+        return self._vxlan_id
+
+    @property
+    def group_ip(self):
+        return self._group_ip
+
+    @property
+    def remote_ip(self):
+        return self._remote_ip
+
+    @property
+    def dst_port(self):
+        return self._dst_port
+
+    def create(self):
+        dev_param = "dev %s" % self.real_dev.name if self.real_dev else
""
+
+        if self.group_ip:
+            group_or_remote = "group %s" % self.group_ip
+        elif self.remote_ip:
+            group_or_remote = "remote %s" % self.remote_ip
+
+        exec_cmd("ip link add %s type vxlan id %d %s %s dstport %d"
+                                % (self.name,
+                                   self.vxlan_id,
+                                   dev_param,
+                                   group_or_remote,
+                                   self.dstport))
diff --git a/lnst/Devices/__init__.py b/lnst/Devices/__init__.py
new file mode 100644
index 0000000..2f8d4d1
--- /dev/null
+++ b/lnst/Devices/__init__.py
@@ -0,0 +1,43 @@
+from lnst.Devices.Device import Device, DeviceError
+from lnst.Devices.BridgeDevice import BridgeDevice
+from lnst.Devices.OvsBridgeDevice import OvsBridgeDevice
+from lnst.Devices.BondDevice import BondDevice
+from lnst.Devices.TeamDevice import TeamDevice
+from lnst.Devices.MacvlanDevice import MacvlanDevice
+from lnst.Devices.VlanDevice import VlanDevice
+from lnst.Devices.VxlanDevice import VxlanDevice
+from lnst.Devices.VtiDevice import VtiDevice, Vti6Device
+from lnst.Devices.VethDevice import VethDevice, PairedVethDevice
+from lnst.Devices.VethPair import VethPair
+from lnst.Devices.VirtualDevice import VirtualDevice
+from lnst.Devices.RemoteDevice import RemoteDevice, remotedev_decorator
+
+device_classes = [
+        ("Device", Device),
+        ("BridgeDevice", BridgeDevice),
+        ("OvsBridgeDevice", OvsBridgeDevice),
+        ("MacvlanDevice", MacvlanDevice),
+        ("VlanDevice", VlanDevice),
+        ("VxlanDevice", VxlanDevice),
+        ("VethDevice", VethDevice),
+        ("PairedVethDevice", PairedVethDevice),
+        ("VtiDevice", VtiDevice),
+        ("Vti6Device", Vti6Device),
+        ("BondDevice", BondDevice),
+        ("TeamDevice", TeamDevice)]
+
+class Devices(object):
+    def __init__(self, host):
+        self._host = host
+
+    def __iter__(self):
+        for x in self._host._device_database.values():
+            if isinstance(x, RemoteDevice):
+                yield x
+
+for name, cls in device_classes:
+    globals()[name] = remotedev_decorator(cls)
+
+#Remove the PairedVethDevice from globals... doesn't make sense to use it on
+#it's own, not even for isinstance... VethDevice works fine for that
+del globals()["PairedVethDevice"]
-- 
2.13.0

[PATCH v2 10/47] add lnst.Controller.Requirements

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This module defines the DeviceReq and HostReq classes, which can be used
to create a global description of Requirements for a network test. You
can use these to define class attributes of a BaseRecipe derived class
to specify "general" requirements for that Recipe, or you can add them
to an instance of a Recipe derived class based on it's parameters to
define requirements "specific" for that single test run.

The module also defines a Requirements class which serves as a container
for HostReq objects, while HostReq classes also serve as containers for
DeviceReq objects. The object tree created this way is translated to a
dictionary used by the internal LNST matching algorithm against
available machines.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Requirements.py | 113 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 113 insertions(+)
 create mode 100644 lnst/Controller/Requirements.py

diff --git a/lnst/Controller/Requirements.py b/lnst/Controller/Requirements.py
new file mode 100644
index 0000000..ffefdfb
--- /dev/null
+++ b/lnst/Controller/Requirements.py
@@ -0,0 +1,113 @@
+"""
+This module defines the DeviceReq and HostReq classes, which can be used to
+create a global description of Requirements for a network test. You can use
+these to define class attributes of a BaseRecipe derived class to specify
+"general" requirements for that Recipe, or you can add them to an instance of
a
+Recipe derived class based on it's parameters to define requirements
"specific"
+for that single test run.
+
+The module also specifies a Requirements class which serves as a container for
+HostReq objects, while HostReq classes also serve as containers for DeviceReq
+objects. The object tree created this way is translated to a dictionary used by
+the internal LNST matching algorithm against available machines.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.LnstError import LnstError
+from lnst.Common.Parameters import Parameters, Param
+
+class RequirementError(LnstError):
+    pass
+
+class HostReq(object):
+    """Specifies a Slave machine requirement
+
+    To define a Host requirement you assign a HostReq instance to a class
+    attribute of a BaseRecipe derived class.
+    Example:
+    class MyRecipe(BaseRecipe):
+        m1 = HostReq()
+
+    Args:
+        kwargs -- any other arguments will be treated as arbitrary string
+            parameters that will be matched to parameters of Slave machines
+            which can define their parameter values based on the implementation
+            of the SlaveMachineParser
+    """
+    def __init__(self, **kwargs):
+        self.params = Parameters()
+        for name, val in kwargs.items():
+            if name == "params":
+                raise RequirementError("'params' is a reserved
keyword.")
+            p = Param()
+            p.val = val
+            setattr(self.params, name, p)
+
+    def __iter__(self):
+        for x in dir(self):
+            val = getattr(self, x)
+            if isinstance(val, DeviceReq):
+                yield (x, val)
+
+    def _to_dict(self):
+        res = {'interfaces': {}, 'params': {}}
+        for dev_id, dev in self:
+            res['interfaces'][dev_id] = dev._to_dict()
+        res['params'] = self.params._to_dict()
+        return res
+
+class DeviceReq(object):
+    """Specifies an Ethernet Device requirement
+
+    To define a Device requirement you assign a DeviceReq instance to a HostReq
+    instance in a BaseRecipe derived class.
+    Example:
+    class MyRecipe(BaseRecipe):
+        m1 = HostReq()
+        m1.eth0 = DeviceReq(label="net1")
+
+    Args:
+        label -- string value indicating the network the Device is connected to
+        kwargs -- any other arguments will be treated as arbitrary string
+            parameters that will be matched to parameters of Slave machines
+            which can define their parameter values based on the implementation
+            of the SlaveMachineParser
+    """
+    def __init__(self, label, **kwargs):
+        self.label = label
+        self.params = Parameters()
+        for name, val in kwargs.items():
+            if name == "params":
+                raise RequirementError("'params' is a reserved
keyword.")
+            p = Param()
+            p.val = val
+            setattr(self.params, name, p)
+
+    def _to_dict(self):
+        res = {'network': self.label,
+               'params': self.params._to_dict()}
+        return res
+
+class _Requirements(object):
+    """Hosts a copy of requirements for a Recipe instance
+
+    Internally used class.
+    """
+    def _to_dict(self):
+        res = {}
+        for h_id, host in self:
+            res[h_id] = host._to_dict()
+        return res
+
+    def __iter__(self):
+        for x in dir(self):
+            val = getattr(self, x)
+            if isinstance(val, HostReq):
+                yield (x, val)
-- 
2.13.0

[PATCH v2 11/47] add lnst.Controller.Job

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the Job class, representing the tester facing API for
manipulating remotely running tasks. Documented in the class doc
strings.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Job.py | 197 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 197 insertions(+)
 create mode 100644 lnst/Controller/Job.py

diff --git a/lnst/Controller/Job.py b/lnst/Controller/Job.py
new file mode 100644
index 0000000..985364e
--- /dev/null
+++ b/lnst/Controller/Job.py
@@ -0,0 +1,197 @@
+"""
+Defines the Job class, representing the tester facing API for manipulating
+remotely running tasks.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+import signal
+import copy_reg
+from lnst.Common.JobError import JobError
+from lnst.Common.TestModule import BaseTestModule
+
+class Job(object):
+    """Tester facing Job API
+
+    Objects of this class are created when a tester calls the 'run' method of
+    a Host object. A Job object can represent both a remotely running task (a
+    background job) or a remote task that already finished.
+    Example:
+        job = m1.run("ls ~/")
+        print job.stdout
+    """
+    def __init__(self, host, what,
+                 expect=True, json=False, netns=None, desc=None):
+        self._host = host
+        self._what = what
+        self._expect = expect
+        self._json = json
+        self._netns = netns
+        self._desc = desc
+
+        self._res = None
+
+        if isinstance(what, BaseTestModule):
+            self._type = "module"
+        elif type(what) == str:
+            self._type = "shell"
+        else:
+            raise JobError("Unable to run '%s'" % str(what))
+
+        self._id = None
+
+    @property
+    def stdout(self):
+        """standard output of the Job
+
+        Type: string
+        Only applicable for Jobs running a shell command
+        """
+        try:
+            return self._res["res_data"]["stdout"]
+        except:
+            return ""
+
+    @property
+    def stderr(self):
+        """standard error output of the Job
+
+        Type: string
+        Only applicable for Jobs running a shell command
+        """
+        try:
+            return self._res["res_data"]["stderr"]
+        except:
+            return ""
+
+    @property
+    def result(self):
+        """result of the Job
+
+        Type:
+            depends on the type of the job. For python modules it is whatever
+            the module sets as the _res_data attribute.
+            For shell commands it is a dictionary with stdout and stderr.
+        """
+        try:
+            return self._res["res_data"]
+        except:
+            return None
+
+    @property
+    def passed(self):
+        """Indicates whether or not the Job passed
+
+        Type: Boolean
+        """
+        try:
+            return self._res["passed"]
+        except:
+            return False
+
+    @property
+    def finished(self):
+        """Indicates whether or not the Job finished running
+
+        Type: Boolean
+        """
+        if self._res is not None:
+            return True
+        else:
+            return False
+
+    @property
+    def netns(self):
+        """name of the network namespace the Job is running in
+
+        Not relevant yet as network namespaces aren't supported yet.
+        """
+        if self._cmd is not None:
+            return self._cmd["netns"]
+        else:
+            return None
+
+    @property
+    def id(self):
+        """id of the job
+
+        Used internally by the Machine class to identify results coming
+        from the slave.
+        TODO make private?
+        """
+        return self._id
+
+    @id.setter
+    def id(self, val):
+        if self._id is not None:
+            raise Exception("Id already set")
+        self._id = val
+
+    def wait(self, timeout=0):
+        """waits for the Job to finish for the specified amount of time
+
+        Args:
+            timeout -- integer value indicating how long to wait for.
+                Default is 0, means wait forever. Don't use for infinitelly
+                running Jobs.
+                If non-zero LNST uses a timed SIGALARM signal to return from
+                this method.
+        Returns:
+            True if the Job finished, False if the Job is still running and
+            the wait method just timed out.
+        """
+        if self.finished:
+            return True
+        if timeout < 0:
+            raise JobError("Negative timeout value not allowed.")
+        return self._host.wait_for_job(self, timeout)
+
+    def kill(self, signal=signal.SIGKILL):
+        """send specified signal to the remotely running Job process
+
+        Args:
+            signal -- integer value of the signal to be sent
+                Default is SIGKILL
+        Returns:
+            True if the Job finished before the signal was sent or if the
+                signal was sent successfully.
+            False if an exception was raised while sending the signal.
+        """
+        logging.info("Sending signal {} to job {} on host {}".format(signal,
+                     self._id, self._host.get_id()))
+        return self._host.kill(self, signal)
+
+    def _to_dict(self):
+        d = {"job_id": self._id,
+             "type": self._type,
+             "json": self._json}
+        if self._type == "shell":
+            d["command"] = self._what
+        elif self._type == "module":
+            d["module"] = self._what
+        else:
+            raise JobError("Unknown Job type %s" % self._type)
+        return d
+
+    def __str__(self):
+        attrs = ["type(%s)" % self._type]
+
+        if self._type == "module":
+            attrs.append("module(%s)" % self._what.__class__.__name__)
+        elif self._type == "shell":
+            attrs.append("command(%s)" % self._what)
+
+        if self._netns is not None:
+            attrs.append("netns(%s)" % self._netns)
+
+        if not self._expect:
+            attrs.append("expecting FAIL")
+
+        return ", ".join(attrs)
-- 
2.13.0

[PATCH v2 12/47] add lnst.Controller.Host

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the Host class that acts as the tester facing API to manipulate
and work with remote machines running the LNST Slave. Documented in the
class doc strings.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * removed debug pprint import
    * added docstrings to requested methods
    * Host::run throws an exception when netns set (not supported yet)
    * added TODO comments to some parts that need to be implemented
---
 lnst/Controller/Host.py | 162 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 162 insertions(+)
 create mode 100644 lnst/Controller/Host.py

diff --git a/lnst/Controller/Host.py b/lnst/Controller/Host.py
new file mode 100644
index 0000000..23cdadf
--- /dev/null
+++ b/lnst/Controller/Host.py
@@ -0,0 +1,162 @@
+"""
+Defines the Host class that acts as the tester facing API to manipulate and
+work with remote machines running the LNST Slave.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+from lnst.Common.Colours import decorate_with_preset
+from lnst.Common.Parameters import Parameters
+from lnst.Common.TestModule import BaseTestModule
+from lnst.Common.NetTestCommand import DEFAULT_TIMEOUT
+from lnst.Devices import Devices
+from lnst.Devices.VirtualDevice import VirtualDevice
+from lnst.Devices.RemoteDevice import RemoteDevice
+from lnst.Controller.Common import ControllerError
+from lnst.Controller.Job import Job
+
+class HostError(ControllerError):
+    pass
+
+class Hosts(object):
+    """Container object for Host class instances
+
+    Implements the __iter__ method to allow iterating Host objects. Created
+    automatically by the LNST Controller class and provided to the tester
+    in the test() method of a BaseRecipe class as the 'matched' attribute.
+    """
+    def __iter__(self):
+        for x in dir(self):
+            val = getattr(self, x)
+            if isinstance(val, Host):
+                yield val
+
+class Host(object):
+    """Tester facing slave Host API
+
+    Objects of this class are created by the Controller and provided to the
+    Recipe object to use from it's 'test()' method. This tester facing API
+    allows the tester to create new Devices and run Jobs on the remote Host.
+    Example:
+        m1.bond0 = Bond() # to create a new bond device
+        m1.run("ip a") # to run a shell command
+    """
+    #TODO add packet capture options
+    def __init__(self, host, **kwargs):
+        self._host = host
+        self.params = Parameters()
+        self.params._from_dict(self._host._slave_desc)
+
+        self.devices = Devices(self._host)
+        self._device_mapping = {}
+
+    def __getattr__(self, name):
+        """direct access to Device objects
+
+        All mapped devices of a Host are directly accessible as attributes of
+        the Host objects. This is implemented by this __getattr__
override"""
+        try:
+            return self._device_mapping[name]
+        except:
+            raise AttributeError("%s object has no attribute named %r" %
+                                 (self.__class__.__name__, name))
+
+    def __setattr__(self, name, value):
+        """allows for dynamic creation of devices
+
+        During execution of the recipes 'test' method, a tester can create new
+        soft devices by assigning a Device object to the Host object instance,
+        this is implemented by overriding this __setattr__ method. It also
+        handles VirtualDevice creation before recipe execution (virtual match).
+        """
+        if isinstance(value, VirtualDevice):
+            # TODO creation of VirtualDevices should be disabled during test
+            # execution, it's commented out right now because I haven't found
+            # a good solution yet...
+            # msg = "Creating VirtualDevices in recipe execution is "\
+                  # "not supported right now."
+            # raise HostError(msg)
+            if name in self._device_mapping:
+                raise HostError("Device with name '%s' already
assigned." % name)
+
+            value.host = self._host
+            self._host.add_tmp_device(value)
+            value.create()
+            self._host.wait_for_tmp_devices(DEFAULT_TIMEOUT)
+
+            self._device_mapping[name] = value
+        elif isinstance(value, RemoteDevice):
+            if name in self._device_mapping:
+                raise HostError("Device with name '%s' already
assigned." % name)
+
+            if value.if_index is None:
+                value.host = self._host
+                self._host.create_remote_device(value)
+            self._device_mapping[name] = value
+        else:
+            super(Host, self).__setattr__(name, value)
+
+    def _map_device(self, dev_id, how):
+        hwaddr = how["hwaddr"]
+        dev = self._host.get_dev_by_hwaddr(hwaddr)
+        self._device_mapping[dev_id] = dev
+
+    def run(self, what, bg=False, fail=False, timeout=DEFAULT_TIMEOUT,
+            json=False, netns=None, desc=None):
+        """
+        Args:
+            what (mandatory) -- what should be run on the host. Can be either a
+                string, that will be executed on the Host as a shell command,
+                or a TestModule object.
+            bg --  run in background flag. Default 'False'. When True, the
+                method will return immediately after the Job request is sent
+                to the Slave Host.
+            fail -- default 'False'. If True, a Failure will be reported as PASS
+            timeout: time limit in seconds. Default is 60. Only respected for
+                jobs running in foreground (background Jobs don't have a time
+                limit)
+            json: Process JSON output into dictionary. Default 'False'.
+            netns: Run in the specified network namespace. Currently not
+                functional.
+            desc: Decription printed in logs. Accepts a string value.
+
+        Returns:
+            a Job object that acts as a handle to access the remote Job. If
+            the Job was ran on foreground, the returned Job object will be
+            filled with result data. If the Job was ran in background, the
+            immediately returned Job object can be used to manipulate the
+            running Job remotely and when the result data arrives from the Slave
+            the Job object will be automatically updated.
+        """
+        #TODO support network namespaces
+        if netns is not None:
+            raise HostError("netns parameter not supported yet.")
+
+        job = Job(self._host, what, expect=not fail, json=json, netns=netns,
+                  desc=desc)
+
+        try:
+            self._host.run_job(job)
+
+            if not bg:
+                if not job.wait(timeout):
+                    logging.debug("Killing timed-out job")
+                    job.kill()
+        except:
+            raise
+        finally:
+            pass
+            #TODO check expect result here
+            # if bg=True:
+            #     add "job started" result
+            # else:
+            #     add job result
+
+        return job
-- 
2.13.0

[PATCH v2 13/47] add lnst.Controller.Config

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the CtlConfig class. This is part of an effort to move from a
single global lnst_config object to using a non-global config object,
that uses a different scheme for Controller and Slave.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Config.py | 99 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100644 lnst/Controller/Config.py

diff --git a/lnst/Controller/Config.py b/lnst/Controller/Config.py
new file mode 100644
index 0000000..b82787c
--- /dev/null
+++ b/lnst/Controller/Config.py
@@ -0,0 +1,99 @@
+"""
+Defines the CtlConfig class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import os
+import sys
+from lnst.Common.Config import DefaultRPCPort, Config
+
+class CtlConfig(Config):
+    """Configuration scheme used by the Controller"""
+    def _init_options(self):
+        self._options['environment'] = dict()
+        self._options['environment']['mac_pool_range'] = {\
+                "value" : ['52:54:01:00:00:01',
'52:54:01:FF:FF:FF'],
+                "additive" : False,
+                "action" : self.optionMacRange,
+                "name" : "mac_pool_range"}
+        self._options['environment']['rpcport'] = {\
+                "value" : DefaultRPCPort,
+                "additive" : False,
+                "action" : self.optionPort,
+                "name" : "rpcport"}
+        self._options['environment']['tool_dirs'] = {\
+                "value" : [],
+                "additive" : True,
+                "action" : self.optionDirList,
+                "name" : "test_tool_dirs"}
+        self._options['environment']['module_dirs'] = {\
+                "value" : [],
+                "additive" : True,
+                "action" : self.optionDirList,
+                "name" : "test_module_dirs"}
+        self._options['environment']['log_dir'] = {\
+                "value" : os.path.abspath(os.path.join(
+                    os.path.dirname(sys.argv[0]), './Logs')),
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "log_dir"}
+        self._options['environment']['resource_dir'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "resource_dir"}
+        self._options['environment']['xslt_url'] = {
+                "value" :
"http://www.lnst-project.org/files/result_xslt/xml_to_html.xsl",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "xslt_url"
+                }
+        self._options['environment']['allow_virtual'] = {
+                "value" : False,
+                "additive" : False,
+                "action" : self.optionBool,
+                "name" : "allow_virtual"
+                }
+
+        self._options['perfrepo'] = dict()
+        self._options['perfrepo']['url'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "url"
+                }
+        self._options['perfrepo']['username'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "username"
+                }
+        self._options['perfrepo']['password'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "password"
+                }
+
+        self._options['pools'] = dict()
+
+        self._options['security'] = dict()
+        self._options['security']['identity'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "identity"}
+        self._options['security']['privkey'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "privkey"}
+
+        self.colours_scheme()
-- 
2.13.0

[PATCH v2 14/47] add lnst.Slave.Config

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the SlaveConfig class. This is part of an effort to move from a
single global lnst_config object to using a non-global config object,
that uses a different scheme for Controller and Slave.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Slave/Config.py | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
 create mode 100644 lnst/Slave/Config.py

diff --git a/lnst/Slave/Config.py b/lnst/Slave/Config.py
new file mode 100644
index 0000000..e749528
--- /dev/null
+++ b/lnst/Slave/Config.py
@@ -0,0 +1,73 @@
+"""
+Defines the SlaveConfig class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import os
+import sys
+from lnst.Common.Config import DefaultRPCPort, Config
+
+class SlaveConfig(Config):
+    def _init_options(self):
+        self._options['environment'] = dict()
+        self._options['environment']['log_dir'] = {\
+                "value" : os.path.abspath(os.path.join(
+                    os.path.dirname(sys.argv[0]), './Logs')),
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "log_dir"}
+        self._options['environment']['use_nm'] = {\
+                "value" : True,
+                "additive" : False,
+                "action" : self.optionBool,
+                "name" : "use_nm"}
+        self._options['environment']['rpcport'] = {\
+                "value" : DefaultRPCPort,
+                "additive" : False,
+                "action" : self.optionPort,
+                "name" : "rpcport"}
+
+        self._options['cache'] = dict()
+        self._options['cache']['dir'] = {\
+                "value" : os.path.abspath(os.path.join(
+                    os.path.dirname(sys.argv[0]), './cache')),
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "cache_dir"}
+
+        self._options['cache']['expiration_period'] = {\
+                "value" : 7*24*60*60, # 1 week
+                "additive" : False,
+                "action" : self.optionTimeval,
+                "name" : "expiration_period"}
+
+        self._options['security'] = dict()
+        self._options['security']['auth_types'] = {\
+                "value" : "none",
+                "additive" : False,
+                "action" : self.optionPlain, #TODO list??
+                "name" : "auth_types"}
+        self._options['security']['auth_password'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPlain,
+                "name" : "auth_password"}
+        self._options['security']['privkey'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "privkey"}
+        self._options['security']['ctl_pubkeys'] = {\
+                "value" : "",
+                "additive" : False,
+                "action" : self.optionPath,
+                "name" : "ctl_pubkeys"}
+
+        self.colours_scheme()
-- 
2.13.0

[PATCH v2 15/47] add lnst.Controller.MachineMapper

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the MachineMapper class. Implements a matching algorithm that
maps requirements to available hosts. In this specific class this is
implemented with backtracking, however testers are free to implement
their own algorithm as long as they respect the API of this class as it
needs to integrate with the rest of LNST.

Most of this class is based on the SetupMapper class and includes some
functionality from the old SlavePool class, but has been modified to
work with Python Recipes.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * docstring now explains that implementing your own MachineMapper is
      not supported at this moment
    * removed debug pprint import
---
 lnst/Controller/MachineMapper.py | 328 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 328 insertions(+)
 create mode 100644 lnst/Controller/MachineMapper.py

diff --git a/lnst/Controller/MachineMapper.py b/lnst/Controller/MachineMapper.py
new file mode 100644
index 0000000..8f98ef9
--- /dev/null
+++ b/lnst/Controller/MachineMapper.py
@@ -0,0 +1,328 @@
+"""
+Defines the MachineMapper class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+from lnst.Controller.Common import ControllerError
+
+class MapperError(ControllerError):
+    pass
+
+class MachineMapper(object):
+    """Implements a matching algorithm that maps requirements to available
hosts
+
+    In this specific class this is implemented with backtracking, however
+    testers are free to implement their own algorithm as long as they respect
+    the API of this class as it needs to integrate with the rest of LNST.
+
+    Since the API is not fully defined yet and depends on the interaction with
+    the SlavePoolManager (also needs a fully defined API), implementing your
+    own MachineMapper class is not recommended yet. However since, the
+    Controller class accepts the 'mapper' parameter it is possible if done
+    properly.
+
+    TODO The Interface will be separated into an abstract class to clearly
+    define the required API. ABC?
+    """
+    def __init__(self):
+        self._pools = {}
+        self._pool_stack = []
+        self._pool = {}
+        self._pool_name = None
+        self._mreqs = {}
+        self._unmatched_req_machines = []
+        self._matched_pool_machines = []
+        self._machine_stack = []
+        self._net_label_mapping = {}
+        self._virtual_matching = False
+
+    def set_requirements(self, mreqs):
+        """set the requirements to be used by the matching algorithm
+
+        This should be a specially formatted dictionary to work.
+        TODO Should probably be reworked to work with something more
+        flexible.
+        """
+        self._mreqs = mreqs
+
+    def set_pools(self, pools):
+        """set the pools to be used by the matching algorithm
+
+        This needs to be a specially formatted dictionary returned by get_pools
+        method of a SlavePoolManager class.
+        """
+        self._pools = pools
+
+    def reset_match_state(self):
+        """resets the state of the backtracking
algorithm"""
+        self._net_label_mapping = {}
+        self._machine_stack = []
+        self._unmatched_req_machines = sorted(self._mreqs.keys(), reverse=True)
+
+        self._pool_stack = list(self._pools.keys())
+        if len(self._pool_stack) > 0:
+            self._pool_name = self._pool_stack.pop()
+            self._pool = self._pools[self._pool_name]
+
+        self._unmatched_pool_machines = []
+        for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True):
+            if self._virtual_matching:
+                if "libvirt_domain" in p_machine["params"]:
+                    self._unmatched_pool_machines.append(p_id)
+            else:
+                self._unmatched_pool_machines.append(p_id)
+
+        if len(self._pool) > 0 and len(self._mreqs) > 0:
+            self._push_machine_stack()
+
+    def matches(self, **kwargs):
+        """Generator method which calls the matching algorithm
+
+        Args:
+            multimatch -- if False or not specified, will only return the first
+                match. Otherwise repeated calls of this method will return
+                more possible mappings until no more are possible.
+
+        Returns:
+            The matched mapping or requirements to pool Machines.
+        """
+        logging.info("Matching machines, without virtuals.")
+        self.reset_match_state()
+        matched = False
+
+        while self._match():
+            matched = True
+            yield self.get_mapping()
+            if "multimatch" not in kwargs or not
kwargs["multimatch"]:
+                return
+
+        if "allow_virt" in kwargs and kwargs["allow_virt"]:
+            logging.info("Match failed for normal machines, falling back "\
+                         "to matching virtual machines.")
+            self._virtual_matching = True
+            self.reset_match_state()
+            while self._match():
+                matched = True
+                yield self.get_mapping()
+                if "multimatch" not in kwargs or not
kwargs["multimatch"]:
+                    return
+        if not matched:
+            msg = "This setup cannot be provisioned with the current pool."
+            raise MapperError(msg)
+
+    def _match(self):
+        logging.info("Trying match with pool: %s" % self._pool_name)
+        while len(self._machine_stack)>0:
+            stack_top = self._machine_stack[-1]
+            if self._virtual_matching and stack_top["virt_matched"]:
+                if stack_top["current_match"] != None:
+                    cur_match = stack_top["current_match"]
+                    self._unmatched_pool_machines.append(cur_match)
+                    stack_top["current_match"] = None
+                stack_top["virt_matched"] = False
+
+            if self._if_match():
+                if len(self._unmatched_req_machines) > 0:
+                    self._push_machine_stack()
+                    continue
+                else:
+                    return True
+            else:
+                #unmap the pool machine
+                if stack_top["current_match"] != None:
+                    cur_match = stack_top["current_match"]
+                    self._unmatched_pool_machines.append(cur_match)
+                stack_top["current_match"] = None
+
+                mreq_m_id = stack_top["m_id"]
+                while len(stack_top["remaining_matches"]) > 0:
+                    pool_m_id = stack_top["remaining_matches"].pop()
+                    if self._check_machine_compatibility(mreq_m_id, pool_m_id):
+                        #map compatible pool machine
+                        stack_top["current_match"] = pool_m_id
+                        stack_top["unmatched_pool_ifs"] = \
+                            sorted(self._pool[pool_m_id]["interfaces"].keys(),
+                                   reverse=True)
+                        self._unmatched_pool_machines.remove(pool_m_id)
+                        break
+
+                if stack_top["current_match"] != None:
+                    #clear if mapping
+                    stack_top["if_stack"] = []
+                    #next iteration will match the interfaces
+                    if not self._virtual_matching:
+                        self._push_if_stack()
+                    continue
+                else:
+                    self._pop_machine_stack()
+                    if len(self._machine_stack) == 0 and\
+                       len(self._pool_stack) > 0:
+                        logging.info("Match with pool %s not found." %
+                                     self._pool_name)
+                        self._pool_name = self._pool_stack.pop()
+                        self._pool = self._pools[self._pool_name]
+                        logging.info("Trying match with pool: %s" %
+                                     self._pool_name)
+
+                        self._unmatched_pool_machines = []
+                        for p_id, p_machine in sorted(self._pool.iteritems(),
reverse=True):
+                            if self._virtual_matching:
+                                if "libvirt_domain" in
p_machine["params"]:
+                                    self._unmatched_pool_machines.append(p_id)
+                            else:
+                                self._unmatched_pool_machines.append(p_id)
+
+                        if len(self._pool) > 0 and len(self._mreqs) > 0:
+                            self._push_machine_stack()
+                    continue
+        return False
+
+    def _if_match(self):
+        m_stack_top = self._machine_stack[-1]
+        if_stack = m_stack_top["if_stack"]
+
+        if self._virtual_matching:
+            if m_stack_top["current_match"] != None:
+                m_stack_top["virt_matched"] = True
+                return True
+            else:
+                return False
+
+        while len(if_stack) > 0:
+            stack_top = if_stack[-1]
+
+            req_m = self._mreqs[m_stack_top["m_id"]]
+            pool_m = self._pool[m_stack_top["current_match"]]
+            req_if = req_m["interfaces"][stack_top["if_id"]]
+            req_net_label = req_if["network"]
+
+            if stack_top["current_match"] != None:
+                cur_match = stack_top["current_match"]
+                m_stack_top["unmatched_pool_ifs"].append(cur_match)
+                pool_if = pool_m["interfaces"][cur_match]
+                pool_net_label = pool_if["network"]
+                net_label_mapping = self._net_label_mapping[req_net_label]
+                if net_label_mapping == (pool_net_label, m_stack_top["m_id"],
+                                         stack_top["if_id"]):
+                    del self._net_label_mapping[req_net_label]
+            stack_top["current_match"] = None
+
+            while len(stack_top["remaining_matches"]) > 0:
+                pool_if_id = stack_top["remaining_matches"].pop()
+                pool_if = pool_m["interfaces"][pool_if_id]
+                if self._check_interface_compatibility(req_if, pool_if):
+                    #map compatible interfaces
+                    stack_top["current_match"] = pool_if_id
+                    if req_net_label not in self._net_label_mapping:
+                        self._net_label_mapping[req_net_label] =\
+                                                   (pool_if["network"],
+                                                   m_stack_top["m_id"],
+                                                   stack_top["if_id"])
+                    m_stack_top["unmatched_pool_ifs"].remove(pool_if_id)
+                    break
+
+            if stack_top["current_match"] != None:
+                if len(m_stack_top["unmatched_ifs"]) > 0:
+                    self._push_if_stack()
+                    continue
+                else:
+                    return True
+            else:
+                self._pop_if_stack()
+                continue
+        return False
+
+    def _push_machine_stack(self):
+        machine_match = {}
+        machine_match["m_id"] = self._unmatched_req_machines.pop()
+        machine_match["current_match"] = None
+        machine_match["remaining_matches"] =
list(self._unmatched_pool_machines)
+        machine_match["if_stack"] = []
+
+        machine = self._mreqs[machine_match["m_id"]]
+        machine_match["unmatched_ifs"] =
sorted(machine["interfaces"].keys(),
+                                                reverse=True)
+        machine_match["unmatched_pool_ifs"] = []
+
+        if self._virtual_matching:
+            machine_match["virt_matched"] = False
+
+        self._machine_stack.append(machine_match)
+
+    def _pop_machine_stack(self):
+        stack_top = self._machine_stack.pop()
+        self._unmatched_req_machines.append(stack_top["m_id"])
+
+    def _push_if_stack(self):
+        m_stack_top = self._machine_stack[-1]
+        if_match = {}
+        if_match["if_id"] = m_stack_top["unmatched_ifs"].pop()
+        if_match["current_match"] = None
+        if_match["remaining_matches"] =
list(m_stack_top["unmatched_pool_ifs"])
+
+        m_stack_top["if_stack"].append(if_match)
+
+    def _pop_if_stack(self):
+        m_stack_top = self._machine_stack[-1]
+        if_stack_top = m_stack_top["if_stack"].pop()
+        m_stack_top["unmatched_ifs"].append(if_stack_top["if_id"])
+
+    def _check_machine_compatibility(self, req_id, pool_id):
+        req_machine = self._mreqs[req_id]
+        pool_machine = self._pool[pool_id]
+        for param, value in req_machine["params"].iteritems():
+            if param not in pool_machine["params"] or\
+               value != pool_machine["params"][param]:
+                return False
+        return True
+
+    def _check_interface_compatibility(self, req_if, pool_if):
+        label_mapping = self._net_label_mapping
+        for req_label, mapping in label_mapping.iteritems():
+            if req_label == req_if["network"] and\
+               mapping[0] != pool_if["network"]:
+                return False
+            if mapping[0] == pool_if["network"] and\
+               req_label != req_if["network"]:
+                return False
+        for param, value in req_if["params"].iteritems():
+            if param not in pool_if["params"] or\
+               value != pool_if["params"][param]:
+                return False
+        return True
+
+    def get_mapping(self):
+        mapping = {"machines": {}, "networks": {},
"virtual": False,
+                   "pool_name": self._pool_name}
+
+        for req_label, label_map in self._net_label_mapping.iteritems():
+            mapping["networks"][req_label] = label_map[0]
+
+        for machine in self._machine_stack:
+            m_map = mapping["machines"][machine["m_id"]] = {}
+
+            m_map["target"] = machine["current_match"]
+
+            hostname =
self._pool[m_map["target"]]["params"]["hostname"]
+            m_map["hostname"] = hostname
+
+            interfaces = m_map["interfaces"] = {}
+            if_stack = machine["if_stack"]
+            for interface in if_stack:
+                i = interfaces[interface["if_id"]] = {}
+                i["target"] = interface["current_match"]
+                pool_if =
self._pool[m_map["target"]]["interfaces"][i["target"]]
+                i["hwaddr"] = pool_if["params"]["hwaddr"]
+
+
+        if self._virtual_matching:
+            mapping["virtual"] = True
+        return mapping
-- 
2.13.0

[PATCH v2 16/47] lnst.Controller.Machine: change object initialization

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Changed to work with the new SlavePoolManager instead of how the old
SlavePool class initialized Machine objects.

This means 2 changes:
* Machine object now uses a CtlConfig object provided by the
  SlavePoolManager
* The MessageDispatcher object is also provided by the SlavePoolManager
  during initialization instead of later.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Machine.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index e4d759a..539952e 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -19,7 +19,6 @@ import signal
 from time import sleep
 from xmlrpclib import Binary
 from functools import wraps
-from lnst.Common.Config import lnst_config
 from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.Utils import wait_for, create_tar_archive
 from lnst.Common.Utils import check_process_running
@@ -44,18 +43,19 @@ class Machine(object):
         deconfiguration, and running commands.
     """
 
-    def __init__(self, m_id, hostname=None, libvirt_domain=None, rpcport=None,
-                 security=None):
+    def __init__(self, m_id, hostname, msg_dispatcher, ctl_config,
+                 libvirt_domain=None, rpcport=None, security=None):
         self._id = m_id
         self._hostname = hostname
+        self._ctl_config = ctl_config
         self._slave_desc = None
         self._connection = None
         self._configured = False
         self._system_config = {}
         self._security = security
-        self._security["identity"] =
lnst_config.get_option("security",
+        self._security["identity"] =
ctl_config.get_option("security",
                                                             "identity")
-        self._security["privkey"] =
lnst_config.get_option("security",
+        self._security["privkey"] =
ctl_config.get_option("security",
                                                            "privkey")
 
         self._domain_ctl = None
@@ -67,9 +67,9 @@ class Machine(object):
         if rpcport:
             self._port = rpcport
         else:
-            self._port = lnst_config.get_option('environment',
'rpcport')
+            self._port = ctl_config.get_option('environment', 'rpcport')
 
-        self._msg_dispatcher = None
+        self._msg_dispatcher = msg_dispatcher
         self._mac_pool = None
 
         self._interfaces = []
@@ -249,7 +249,7 @@ class Machine(object):
 
         slave_version = slave_desc["lnst_version"]
         slave_is_git = self.is_git_version(slave_version)
-        ctl_version = lnst_config.version
+        ctl_version = self._ctl_config.version
         ctl_is_git = self.is_git_version(ctl_version)
         if slave_version != ctl_version:
             if ctl_is_git and slave_is_git:
-- 
2.13.0

[PATCH v2 17/47] add lnst.Controller.MessageDispatcher

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the MessageDispatcher class used by the Controller to multiplex
communication from all the connected Slave machines.

In addition to that it defines functions used by the MessageDispatcher to
transparently translate Device objects while communicating with the Slave.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/MessageDispatcher.py | 188 +++++++++++++++++++++++++++++++++++
 1 file changed, 188 insertions(+)
 create mode 100644 lnst/Controller/MessageDispatcher.py

diff --git a/lnst/Controller/MessageDispatcher.py b/lnst/Controller/MessageDispatcher.py
new file mode 100644
index 0000000..a6d7f44
--- /dev/null
+++ b/lnst/Controller/MessageDispatcher.py
@@ -0,0 +1,188 @@
+"""
+Defines the MessageDispatcher class used by the Controller to multiplex
+communication from all the connected Slave machines.
+
+In addition to that it defines functions used by the MessageDispatcher to
+transparently translate Device objects while communicating with the Slave.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+from lnst.Common.ConnectionHandler import send_data, recv_data
+from lnst.Common.ConnectionHandler import ConnectionHandler
+from lnst.Common.DeviceRef import DeviceRef
+from lnst.Controller.Common import ControllerError
+from lnst.Devices.RemoteDevice import RemoteDevice
+
+def deviceref_to_remote_device(machine, obj):
+    if isinstance(obj, DeviceRef):
+        dev = machine.dev_db_get_if_index(obj.if_index)
+        return dev
+    elif isinstance(obj, dict):
+        new_dict = {}
+        for key, value in obj.items():
+            new_dict[key] = deviceref_to_remote_device(machine,
+                                                       value)
+        return new_dict
+    elif isinstance(obj, list):
+        new_list = []
+        for value in obj:
+            new_list.append(deviceref_to_remote_device(machine,
+                                                       value))
+        return new_list
+    elif isinstance(obj, tuple):
+        new_list = []
+        for value in obj:
+            new_list.append(deviceref_to_remote_device(machine,
+                                                       value))
+        return tuple(new_list)
+    else:
+        return obj
+
+def remote_device_to_deviceref(obj):
+    if isinstance(obj, RemoteDevice):
+        return DeviceRef(obj.if_index)
+    elif isinstance(obj, dict):
+        new_dict = {}
+        for key, value in obj.items():
+            new_dict[key] = remote_device_to_deviceref(value)
+        return new_dict
+    elif isinstance(obj, list):
+        new_list = []
+        for value in obj:
+            new_list.append(remote_device_to_deviceref(value))
+        return new_list
+    elif isinstance(obj, tuple):
+        new_list = []
+        for value in obj:
+            new_list.append(remote_device_to_deviceref(value))
+        return tuple(new_list)
+    else:
+        return obj
+
+class ConnectionError(ControllerError):
+    pass
+
+class MessageDispatcher(ConnectionHandler):
+    def __init__(self, log_ctl):
+        super(MessageDispatcher, self).__init__()
+        self._log_ctl = log_ctl
+        self._machines = dict()
+
+    def add_slave(self, machine, connection):
+        self._machines[machine] = machine
+        self.add_connection(machine, connection)
+
+    def send_message(self, machine, data):
+        soc = self.get_connection(machine)
+
+        if data["type"] == "command":
+            data["args"] = remote_device_to_deviceref(data["args"])
+            data["kwargs"] =
remote_device_to_deviceref(data["kwargs"])
+
+        if send_data(soc, data) == False:
+            msg = "Connection error from slave %s" % machine.get_id()
+            raise ConnectionError(msg)
+
+    def wait_for_result(self, machine):
+        wait = True
+        while wait:
+            connected_slaves = self._connection_mapping.keys()
+
+            messages = self.check_connections()
+
+            remaining_slaves = self._connection_mapping.keys()
+
+            for msg in messages:
+                if msg[1]["type"] == "result" and msg[0] == machine:
+                    wait = False
+                    result = msg[1]["result"]
+
+                    machine = self._machines[machine]
+                    result = deviceref_to_remote_device(machine,
+                                                        result)
+                else:
+                    self._process_message(msg)
+
+            if connected_slaves != remaining_slaves:
+                disconnected_slaves = set(connected_slaves) -\
+                                      set(remaining_slaves)
+                msg = "Slaves " + str(list(disconnected_slaves)) + \
+                      " disconnected from the controller."
+                raise ConnectionError(msg)
+
+        return result
+
+    def wait_for_finish(self, machine, job_id):
+        wait = True
+        while wait:
+            connected_slaves = self._connection_mapping.keys()
+
+            messages = self.check_connections()
+
+            remaining_slaves = self._connection_mapping.keys()
+
+            for msg in messages:
+                self._process_message(msg)
+                if msg[1]["type"] == "job_finished" and msg[0] ==
machine:
+                    wait = False
+
+            if connected_slaves != remaining_slaves:
+                disconnected_slaves = set(connected_slaves) -\
+                                      set(remaining_slaves)
+                msg = "Slaves " + str(list(disconnected_slaves)) + \
+                      " disconnected from the controller."
+                raise ConnectionError(msg)
+        return True
+
+    def handle_messages(self):
+        connected_slaves = self._connection_mapping.keys()
+
+        messages = self.check_connections()
+
+        remaining_slaves = self._connection_mapping.keys()
+
+        for msg in messages:
+            self._process_message(msg)
+
+        if connected_slaves != remaining_slaves:
+            disconnected_slaves = set(connected_slaves) -\
+                                  set(remaining_slaves)
+            msg = "Slaves " + str(list(disconnected_slaves)) + \
+                  " disconnected from the controller."
+            raise ConnectionError(msg)
+        return True
+
+    def _process_message(self, message):
+        if message[1]["type"] == "log":
+            record = message[1]["record"]
+            self._log_ctl.add_client_log(message[0].get_id(), record)
+        elif message[1]["type"] == "result":
+            msg = "Recieved result message from different slave %s" %
message[0].get_id()
+            logging.debug(msg)
+        elif message[1]["type"] == "dev_created":
+            machine = self._machines[message[0]]
+            machine.device_created(message[1]["dev_data"])
+        elif message[1]["type"] == "dev_deleted":
+            machine = self._machines[message[0]]
+            machine.device_delete(message[1])
+        elif message[1]["type"] == "exception":
+            raise message[1]["Exception"]
+        elif message[1]["type"] == "job_finished":
+            machine = self._machines[message[0]]
+            machine.job_finished(message[1])
+        else:
+            msg = "Unknown message type: %s" % message[1]["type"]
+            raise ConnectionError(msg)
+
+    def disconnect_slave(self, machine):
+        soc = self.get_connection(machine)
+        self.remove_connection(soc)
+        del self._machines[machine]
-- 
2.13.0

[PATCH v2 18/47] add lnst.Controller.SlavePoolManager

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the SlavePoolManager class that takes care of loading pools and
checking machine availability. Most of the SlavePoolManager class is
copied over from the old SlavePool class, but modified to work with
Python recipes.

The idea is to later generalize the SlavePoolManager class into an
interface to support custom tester provided poolmanager classes (e.g. to
support different pool types or SlaveMachine description files).

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/SlavePoolManager.py | 273 ++++++++++++++++++++++++++++++++++++
 1 file changed, 273 insertions(+)
 create mode 100644 lnst/Controller/SlavePoolManager.py

diff --git a/lnst/Controller/SlavePoolManager.py b/lnst/Controller/SlavePoolManager.py
new file mode 100644
index 0000000..dc61e11
--- /dev/null
+++ b/lnst/Controller/SlavePoolManager.py
@@ -0,0 +1,273 @@
+"""
+This module contains implementaion of SlavePoolManager class that
+takes care of loading pools and checking machine availability
+
+Most of the SlavePoolManager class is copied over from the old SlavePool class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+import os
+import re
+import socket
+import select
+from lnst.Common.NetUtils import normalize_hwaddr
+from lnst.Controller.Common import ControllerError
+from lnst.Controller.Machine import Machine
+from lnst.Controller.SlaveMachineParser import SlaveMachineParser
+from lnst.Common.Colours import decorate_with_preset
+from lnst.Common.Utils import check_process_running
+
+class PoolManagerError(ControllerError):
+    pass
+
+class SlavePoolManager:
+    """
+    This class is responsible for managing test machines that
+    are available at the controler and can be used for testing.
+    """
+    def __init__(self, pools, msg_dispatcher, ctl_config, pool_checks=True):
+        self._map = {}
+        self._pools = {}
+        self._pool = {}
+        self._msg_dispatcher = msg_dispatcher
+        self._ctl_config = ctl_config
+
+        self._allow_virt = ctl_config.get_option("environment",
+                                                 "allow_virtual")
+        self._allow_virt &= check_process_running("libvirtd")
+        self._pool_checks = pool_checks
+
+        logging.info("Checking machine pool availability.")
+        for pool_name, pool_dir in pools.items():
+            self._pools[pool_name] = {}
+            self.add_dir(pool_name, pool_dir)
+            if len(self._pools[pool_name]) == 0:
+                del self._pools[pool_name]
+
+        self._machines = {}
+        for pool_name, machines in self._pools.items():
+            pool = self._machines[pool_name] = {}
+            for m_id, m_spec in machines.items():
+                params = m_spec["params"]
+
+                hostname = params["hostname"]
+
+                if "libvirt_domain" in params:
+                    libvirt_domain = params["libvirt_domain"]
+                else:
+                    libvirt_domain = None
+
+                if "rpc_port" in params:
+                    rpc_port = params["rpc_port"]
+                else:
+                    rpc_port = None
+
+                pool[m_id] = Machine(m_id, hostname, self._msg_dispatcher,
+                                     ctl_config, libvirt_domain, rpc_port,
+                                     m_spec["security"])
+                #TODO check if all described devices are available
+
+        logging.info("Finished loading pools.")
+
+    def get_pools(self):
+        return self._pools
+
+    def get_pool(self, pool_name):
+        return self._pools[pool_name]
+
+    def get_machine_pools(self):
+        return self._machines
+
+    def get_machine_pool(self, pool_name):
+        return self._machines[pool_name]
+
+    def add_dir(self, pool_name, dir_path):
+        logging.info("Processing pool '%s', directory '%s'" %
(pool_name,
+                                                               dir_path))
+        pool = self._pools[pool_name]
+
+        try:
+            dentries = os.listdir(dir_path)
+        except OSError:
+            logging.warn("Directory '%s' does not exist for pool
'%s'" %
+                                                                  (dir_path,
+                                                                   pool_name))
+            return
+
+        for dirent in dentries:
+            m_id, m = self.add_file(pool_name, dir_path, dirent)
+            if m_id != None and m != None:
+                pool[m_id] = m
+
+        if len(pool) == 0:
+            logging.warn("No machines found in pool '%s', directory
'%s'" %
+                                                                   (pool_name,
+                                                                    dir_path))
+
+        max_len = 0
+        for m_id in pool.keys():
+            if len(m_id) > max_len:
+                max_len = len(m_id)
+
+        if self._pool_checks:
+            check_sockets = {}
+            for m_id, m in sorted(pool.iteritems()):
+                hostname = m["params"]["hostname"]
+                if "rpc_port" in m["params"]:
+                    port = int(m["params"]["rpc_port"])
+                else:
+                    port = self._ctl_config.get_option('environment',
'rpcport')
+
+                logging.debug("Querying machine '%s': %s:%s" %\
+                                                (m_id, hostname, port))
+
+                s = socket.socket()
+                s.settimeout(0)
+                try:
+                    s.connect((hostname, port))
+                except:
+                    pass
+                check_sockets[s] = m_id
+
+            while len(check_sockets) > 0:
+                rl, wl, el = select.select([], check_sockets.keys(), [])
+                for s in wl:
+                    err = s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
+                    m_id = check_sockets[s]
+                    if err == 0:
+                        pool[m_id]["available"] = True
+                        s.shutdown(socket.SHUT_RDWR)
+                        s.close()
+                        del check_sockets[s]
+                    else:
+                        pool[m_id]["available"] = False
+                        s.close()
+                        del check_sockets[s]
+        else:
+            for m_id in pool.keys():
+                pool[m_id]["available"] = True
+
+        for m_id in sorted(list(pool.keys())):
+            m = pool[m_id]
+            if m["available"]:
+                if 'libvirt_domain' in m['params']:
+                    libvirt_msg = "   libvirt_domain: %s" %\
+                                        m['params']['libvirt_domain']
+                else:
+                    libvirt_msg = ""
+                msg = "%s%s [%s] %s" % (m_id, (max_len - len(m_id)) * "
",
+                                        decorate_with_preset("UP",
"pass"),
+                                        libvirt_msg)
+            else:
+                msg = "%s%s [%s]" % (m_id, (max_len - len(m_id)) * "
",
+                                     decorate_with_preset("DOWN",
"fail"))
+                del pool[m_id]
+
+            logging.info(msg)
+
+    def add_file(self, pool_name, dir_path, dirent):
+        filepath = dir_path + "/" + dirent
+        pool = self._pools[pool_name]
+        if os.path.isfile(filepath) and re.search("\.xml$", filepath, re.I):
+            dirname, basename = os.path.split(filepath)
+            m_id = re.sub("\.[xX][mM][lL]$", "", basename)
+
+            parser = SlaveMachineParser(filepath, self._ctl_config)
+            xml_data = parser.parse()
+            machine_spec = self._process_machine_xml_data(m_id, xml_data)
+
+            if 'libvirt_domain' in machine_spec['params'] and \
+               not self._allow_virt:
+                   logging.debug("libvirtd not running disabled. "\
+                                 "Removing libvirt_domain from "\
+                                 "machine '%s'" % m_id)
+                   del machine_spec['params']['libvirt_domain']
+
+            # Check if there isn't any machine with the same
+            # hostname or libvirt_domain already in the pool
+            for pm_id, m in pool.iteritems():
+                pm = m["params"]
+                rm = machine_spec["params"]
+                if pm["hostname"] == rm["hostname"]:
+                    msg = "You have the same machine listed twice in " \
+                          "your pool ('%s' and '%s')." % (m_id,
pm_id)
+                    raise PoolManagerError(msg)
+
+                if "libvirt_domain" in rm and "libvirt_domain" in pm
and \
+                   pm["libvirt_domain"] == rm["libvirt_domain"]:
+                    msg = "You have the same libvirt_domain listed twice in "
\
+                          "your pool ('%s' and '%s')." % (m_id,
pm_id)
+                    raise PoolManagerError(msg)
+
+            return (m_id, machine_spec)
+        return (None, None)
+
+    def _process_machine_xml_data(self, m_id, machine_xml_data):
+        machine_spec = {"interfaces": {}, "params":{},
"security": {}}
+
+        # process parameters
+        if "params" in machine_xml_data:
+            for param in machine_xml_data["params"]:
+                name = str(param["name"])
+                value = str(param["value"])
+                machine_spec["params"][name] = value
+
+        mandatory_params = ["hostname"]
+        for p in mandatory_params:
+            if p not in machine_spec["params"]:
+                msg = "Mandatory parameter '%s' missing for machine
%s." \
+                        % (p, m_id)
+                raise PoolManagerError(msg, machine_xml_data["params"])
+
+        # process interfaces
+        if "interfaces" in machine_xml_data:
+            for iface in machine_xml_data["interfaces"]:
+                if_id = iface["id"]
+                iface_spec = self._process_iface_xml_data(m_id, iface)
+
+                if if_id not in machine_spec["interfaces"]:
+                    machine_spec["interfaces"][if_id] = iface_spec
+                else:
+                    msg = "Duplicate interface id '%s'." % if_id
+                    raise PoolManagerError(msg, iface)
+        else:
+            if "libvirt_domain" not in machine_spec["params"]:
+                msg = "Machine '%s' has no testing interfaces. " \
+                      "This setup is supported only for virtual slaves." \
+                      % m_id
+                raise PoolManagerError(msg, machine_xml_data)
+
+        machine_spec["security"] = machine_xml_data["security"]
+
+        return machine_spec
+
+    def _process_iface_xml_data(self, m_id, iface):
+        if_id = iface["id"]
+        iface_spec = {"params": {}}
+        iface_spec["network"] = iface["network"]
+
+        for param in iface["params"]:
+            name = str(param["name"])
+            value = str(param["value"])
+
+            if name == "hwaddr":
+                iface_spec["params"][name] = normalize_hwaddr(value)
+            else:
+                iface_spec["params"][name] = value
+
+        mandatory_params = ["hwaddr"]
+        for p in mandatory_params:
+            if p not in iface_spec["params"]:
+                msg = "Mandatory parameter '%s' missing for machine %s,
" \
+                      "interface '%s'." % (p, m_id, if_id)
+                raise PoolManagerError(msg, iface["params"])
+
+        return iface_spec
-- 
2.13.0

[PATCH v2 19/47] add lnst.Controller.Recipe

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Module implementing the BaseRecipe class. Every LNST Recipe written by
testers should be inherited from this class. An LNST Recipe is composed
of several parts:
* Requirements definition
* Parameter definition (optional)
* Test definition

Further documentation can be read in the classes doc string.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * renamed variable 'x' to 'attr'
---
 lnst/Controller/Recipe.py | 98 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 lnst/Controller/Recipe.py

diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
new file mode 100644
index 0000000..d4f33f9
--- /dev/null
+++ b/lnst/Controller/Recipe.py
@@ -0,0 +1,98 @@
+"""
+Module implementing the BaseRecipe class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import copy
+from lnst.Common.Parameters import Parameters, Param
+from lnst.Controller.Requirements import _Requirements, HostReq
+from lnst.Controller.Host import Hosts, Host
+from lnst.Controller.Common import ControllerError
+
+class RecipeError(ControllerError):
+    """Exception thrown by the BaseRecipe class"""
+    pass
+
+class BaseRecipe(object):
+    """BaseRecipe class
+
+    Every LNST Recipe written by testers should be inherited from this class.
+    An LNST Recipe is composed of several parts:
+    * Requirements definition - you define recipe requirements in a derived
+        class by defining class attributes of the HostReq type. You can further
+        specify Ethernet Device requirements by defining DeviceReq attributes
+        of the HostReq object.
+        Example:
+        m1 = HostReq(arch="x86_64")
+        m1.eth0 = DeviceReq(hwaddr="52:54:00:12:34:56")
+    * Parameter definition (optional) - you can define paramaters of you Recipe
+        by defining class attributes of the Param type (or inherited). These
+        parameters can then be accessed from the test() method to change it's
+        behaviour. Parameter validity (type) is checked during the
+        instantiation of the Recipe object by the base __init__ method.
+        You can define your own __init__ method to implement more complex
+        Parameter checking if needed, but you MUST call the base __init__
+        method first.
+    * Test definition - this is done by defining the test() method, in this
+        method the tester has direct access to mapped LNST slave Hosts, can
+        manipulate them and implement his tests.
+
+    Attributes:
+        matched -- when running the Recipe the Controller will fill this
+            attribute with a Hosts object after the Mapper finds suitable slave
+            hosts.
+        req -- instantiated Requirements object, you can optionally change the
+            Recipe requirements through this object during runtime (e.g.
+            variable number of hosts or devices of a host based on a Parameter)
+        params -- instantiated Parameters object, can be used to access the
+            calculated parameters during Recipe initialization/execution
+    """
+    def __init__(self, **kwargs):
+        """
+        The __init__ method does 2 things:
+        * copies Requirements -- since Requirements are defined as class
+            attributes, we need to copy the objects to avoid conflicts with
+            multiple instances of the same class etc...
+            The copied objects are stored under a Requirements object available
+            through the 'req' attribute. This way you can optionally change the
+            Requirements of an instantiated Recipe.
+        * copies and instantiates Parameters -- Parameters are also class
+            attributes so they need to be copied into a Parameters() object
+            (accessible in the 'params' attribute).
+            Next, the copied objects are loaded with values from kwargs
+            and checked if mandatory Parameters have values.
+        """
+        self.matched = None
+        self.req = _Requirements()
+        self.params = Parameters()
+        for attr in dir(self):
+            val = getattr(self, attr)
+            if isinstance(val, HostReq):
+                setattr(self.req, attr, copy.deepcopy(val))
+            elif isinstance(val, Param):
+                setattr(self.params, attr, copy.deepcopy(val))
+
+        for name, val in kwargs.items():
+            try:
+                param = getattr(self.params, name)
+                param.val = val
+            except:
+                raise RecipeError("Unknown parameter {}".format(name))
+
+        for name, param in self.params:
+            if param.mandatory and not param.set:
+                raise RecipeError("Parameter {} is mandatory".format(name))
+
+    def _set_hosts(self, hosts):
+        self.matched = hosts
+
+    def test(self):
+        """Method to be implemented by the Tester"""
+        raise NotImplementedError("Method test must be defined by a child
class.")
-- 
2.13.0

[PATCH v2 20/47] add lnst.Controller.Controller

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This module defines the Controller class that brings together individual
implementation parts of an LNST Controller. When instantiated, it allows
the tester to configure and run his own recipes with the LNST
'infrastructure'.

This way, the lnst.Controller package is a Python library allowing the
creation of your own LNST Controller.

In theory multiple instances of the Controller class are possible and
you can therefore run multiple recipes in parallel, though this is not
supported and wasn't tested yet.

More info is available in the doc strings of the Controller class.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Controller.py | 220 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 220 insertions(+)
 create mode 100644 lnst/Controller/Controller.py

diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py
new file mode 100644
index 0000000..7848750
--- /dev/null
+++ b/lnst/Controller/Controller.py
@@ -0,0 +1,220 @@
+"""
+This module defines the Controller class that brings together individual
+implementation parts of an LNST Controller. When instantiated, it allows the
+tester to configure and run his own recipes with the LNST 'infrastructure'.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import os
+import sys
+import datetime
+import logging
+import socket
+from lnst.Common.Logs import LoggingCtl
+from lnst.Common.NetUtils import MacPool
+from lnst.Common.LnstError import LnstError
+from lnst.Common.Utils import mkdir_p
+from lnst.Devices import VirtualDevice
+from lnst.Controller.Common import ControllerError
+from lnst.Controller.Config import CtlConfig
+from lnst.Controller.MessageDispatcher import MessageDispatcher
+from lnst.Controller.SlavePoolManager import SlavePoolManager
+from lnst.Controller.MachineMapper import MachineMapper
+from lnst.Controller.Host import Hosts, Host
+from lnst.Controller.Requirements import DeviceReq
+from lnst.Controller.Recipe import BaseRecipe
+
+class Controller(object):
+    """The LNST Controller class
+
+    Most importantly allows the tester to run instantiated Recipe tests using
+    the LNST infrastructure.
+
+    Can be configured with custom implementation of several objects used for
+    setting up the infrastructure.
+    """
+
+    def __init__(self, poolMgr=SlavePoolManager, mapper=MachineMapper,
+                 config=None, pools=[], pool_checks=True, debug=0):
+        """
+        Args:
+            poolMgr -- class that implements the SlavePoolManager interface
+                will be instantiated by the Controller to provide the mapper
+                with pools available for matching, also handles the creation
+                of Machine objects (internal LNST class used to access the
+                slave hosts)
+            mapper -- class that implements the MachineMapper interface
+                will be instantiated by the Controller to match Recipe
+                requirements to the available pools
+            config -- optional LNST configuration object, if None the
+                Controller will load it's own configuration from default paths
+            pools -- a list of pool names to restrict the used pool directories
+            pool_checks -- boolean (default True), if False will disable
+                checking online status of Slaves
+            debug -- integer (default 0), sets debug level of LNST
+        """
+        self._config = self._load_ctl_config(config)
+        config = self._config
+
+        mac_pool_range = config.get_option('environment',
'mac_pool_range')
+        self._mac_pool = MacPool(mac_pool_range[0], mac_pool_range[1])
+        self._log_ctl = LoggingCtl(debug,
+                log_dir=config.get_option('environment','log_dir'),
+                log_subdir=datetime.datetime.now().
+                           strftime("%Y-%m-%d_%H:%M:%S"),
+                colours=not config.get_option("colours",
"disable_colours"))
+
+        self._msg_dispatcher = MessageDispatcher(self._log_ctl)
+
+        self._network_bridges = {}
+        self._mapper = mapper()
+
+        select_pools = {}
+        conf_pools = config.get_pools()
+        if len(pools) > 0:
+            for pool_name in pools:
+                if pool_name in conf_pools:
+                    select_pools[pool_name] = conf_pools[pool_name]
+                elif len(pools) == 1 and os.path.isdir(pool_name):
+                    select_pools = {"cmd_line_pool": pool_name}
+                else:
+                    raise ControllerError("Pool %s does not exist!" %
pool_name)
+        else:
+            select_pools = conf_pools
+
+        self._pools = poolMgr(select_pools, self._msg_dispatcher, config,
+                              pool_checks)
+
+    def run(self, recipe, **kwargs):
+        """Execute the provided Recipe
+
+        This method takes care of both finding a Slave hosts matching the Recipe
+        requirements, provisioning them and calling the 'test' method of the
+        Recipe object with proper references to the mapped Hosts
+
+        Args:
+            recipe -- an instantiated Recipe object (isinstance BaseRecipe)
+            kwargs -- optional keyword arguments passed to the configured Mapper
+        """
+        if not isinstance(recipe, BaseRecipe):
+            raise ControllerError("recipe argument must be a BaseRecipe
instance.")
+
+        req = recipe.req
+
+        self._mapper.set_pools(self._pools.get_pools())
+        self._mapper.set_requirements(req._to_dict())
+
+        i = 0
+        for match in self._mapper.matches(**kwargs):
+            self._log_ctl.set_recipe(recipe.__class__.__name__,
+                                     expand="match_%d" % i)
+            i += 1
+
+            self._print_match_description(match)
+            self._map_match(match, req)
+            try:
+                recipe._set_hosts(self._hosts)
+                recipe.test()
+            except LnstError as exc:
+                logging.error("Recipe execution terminated by unexpected
exception")
+                raise
+            finally:
+                recipe._set_hosts(None)
+                for machine in self._machines.values():
+                    machine.restore_system_config()
+                self._cleanup_slaves()
+
+    def _map_match(self, match, requested):
+        self._machines = {}
+        self._hosts = Hosts()
+        pool = self._pools.get_machine_pool(match["pool_name"])
+        for m_id, m in match["machines"].items():
+            machine = self._machines[m_id] = pool[m["target"]]
+
+            machine.set_id(m_id)
+            self._prepare_machine(machine)
+
+            setattr(self._hosts, m_id, Host(machine))
+            host = getattr(self._hosts, m_id)
+            for if_id, i in m["interfaces"].items():
+                host._map_device(if_id, i)
+
+            if match["virtual"]:
+                req_host = getattr(requested, m_id)
+                for name, dev in req_host:
+                    new_virt_dev = VirtualDevice(network=dev.label,
+                                                 driver=dev.params.driver,
+                                                 hwaddr=dev.params.hwaddr)
+                    setattr(host, name, new_virt_dev)
+
+    def _prepare_machine(self, machine):
+        self._log_ctl.add_slave(machine.get_id())
+        machine.set_mac_pool(self._mac_pool)
+        machine.set_network_bridges(self._network_bridges)
+
+        recipe_name = os.path.basename(sys.argv[0])
+        machine.set_recipe(recipe_name)
+
+    def _cleanup_slaves(self):
+        if self._machines == None:
+            return
+
+        for m_id, machine in self._machines.iteritems():
+            machine.cleanup()
+            #clean-up slave logger
+            self._log_ctl.remove_slave(m_id)
+
+        self._machines.clear()
+
+        # remove dynamically created bridges
+        for bridge in self._network_bridges.itervalues():
+            bridge.cleanup()
+        self._network_bridges = {}
+
+    def _load_ctl_config(self, config):
+        if isinstance(config, CtlConfig):
+            return config
+        else:
+            config = CtlConfig()
+            try:
+                config.load_config('/etc/lnst-ctl.conf')
+            except:
+                pass
+
+            usr_cfg = os.path.expanduser('~/.lnst/lnst-ctl.conf')
+            if os.path.isfile(usr_cfg):
+                config.load_config(usr_cfg)
+            else:
+                usr_cfg_dir = os.path.dirname(usr_cfg)
+                pool_dir = usr_cfg_dir + "/pool"
+                mkdir_p(pool_dir)
+                global_pools = config.get_section("pools")
+                if (len(global_pools) == 0):
+                    config.add_pool("default", pool_dir, usr_cfg)
+                with open(usr_cfg, 'w') as f:
+                    f.write(config.dump_config())
+
+            dirname = os.path.dirname(sys.argv[0])
+            gitcfg = os.path.join(dirname, "lnst-ctl.conf")
+            if os.path.isfile(gitcfg):
+                config.load_config(gitcfg)
+
+            return config
+
+    def _print_match_description(self, match):
+        logging.info("Pool match description:")
+        if match["virtual"]:
+            logging.info("  Setup is using virtual machines.")
+        for m_id, m in sorted(match["machines"].iteritems()):
+            logging.info("  host \"%s\" uses \"%s\"" %
(m_id, m["target"]))
+            for if_id, match in m["interfaces"].iteritems():
+                pool_id = match["target"]
+                logging.info("    interface \"%s\" matched to
\"%s\"" %\
+                                            (if_id, pool_id))
-- 
2.13.0

[PATCH v2 21/47] various files: retype exceptions

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

All LNST related Exceptions should inherit from the LnstError base
exception class.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/Config.py           |  3 ++-
 lnst/Common/ExecCmd.py          |  3 ++-
 lnst/Common/NetTestCommand.py   |  5 +++--
 lnst/Common/ResourceCache.py    |  3 ++-
 lnst/Common/SecureSocket.py     |  3 ++-
 lnst/Common/ShellProcess.py     |  3 ++-
 lnst/Common/TestsCommon.py      |  3 ++-
 lnst/Controller/Machine.py      |  5 +++--
 lnst/Controller/Task.py         |  4 +++-
 lnst/Controller/VirtUtils.py    |  3 ++-
 lnst/RecipeCommon/ModuleWrap.py | 33 +++++++++++++++++----------------
 11 files changed, 40 insertions(+), 28 deletions(-)

diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py
index d47f3e2..04d6f9b 100644
--- a/lnst/Common/Config.py
+++ b/lnst/Common/Config.py
@@ -18,10 +18,11 @@ from lnst.Common.Utils import bool_it
 from lnst.Common.NetUtils import verify_mac_address
 from lnst.Common.Colours import get_preset_conf
 from lnst.Common.Version import LNSTMajorVersion
+from lnst.Common.LnstError import LnstError
 
 DefaultRPCPort = 9999
 
-class ConfigError(Exception):
+class ConfigError(LnstError):
     pass
 
 class Config():
diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py
index 24715ae..6489766 100644
--- a/lnst/Common/ExecCmd.py
+++ b/lnst/Common/ExecCmd.py
@@ -12,8 +12,9 @@ jpirko(a)redhat.com (Jiri Pirko)
 
 import logging
 import subprocess
+from lnst.Common.LnstError import LnstError
 
-class ExecCmdFail(Exception):
+class ExecCmdFail(LnstError):
     _cmd = None
     _retval = None
     _stderr = None
diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py
index f91cc37..ad4fe8f 100644
--- a/lnst/Common/NetTestCommand.py
+++ b/lnst/Common/NetTestCommand.py
@@ -21,6 +21,7 @@ from time import time
 from lnst.Common.ExecCmd import exec_cmd, ExecCmdFail
 from lnst.Common.ConnectionHandler import send_data
 from lnst.Common.Logs import log_exc_traceback
+from lnst.Common.LnstError import LnstError
 
 DEFAULT_TIMEOUT = 60
 
@@ -64,7 +65,7 @@ def str_command(command):
 
     return ", ".join(attrs)
 
-class CommandException(Exception):
+class CommandException(LnstError):
     """Base class for client errors."""
     def __init__(self, command):
         self.command = command
@@ -72,7 +73,7 @@ class CommandException(Exception):
     def __str__(self):
         return "CommandException: " + str(self.command)
 
-class BgCommandException(Exception):
+class BgCommandException(CommandException):
     """Base class for background command errors."""
     def __init__(self, str):
         self._str = str
diff --git a/lnst/Common/ResourceCache.py b/lnst/Common/ResourceCache.py
index 98558a2..2037a0c 100644
--- a/lnst/Common/ResourceCache.py
+++ b/lnst/Common/ResourceCache.py
@@ -15,10 +15,11 @@ import re
 import time
 import shutil
 from lnst.Common.ExecCmd import exec_cmd
+from lnst.Common.LnstError import LnstError
 
 SETUP_SCRIPT_NAME = "lnst-setup.sh"
 
-class ResourceCacheError(Exception):
+class ResourceCacheError(LnstError):
     pass
 
 class ResourceCache(object):
diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py
index b856238..84ff04a 100644
--- a/lnst/Common/SecureSocket.py
+++ b/lnst/Common/SecureSocket.py
@@ -20,6 +20,7 @@ import cPickle
 import hashlib
 import hmac
 from lnst.Common.Utils import not_imported
+from lnst.Common.LnstError import LnstError
 
 DH_GROUP = {"p":
int("0xFFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD1"\
                      "29024E088A67CC74020BBEA63B139B22514A08798E3404DD"\
@@ -71,7 +72,7 @@ SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8
 if bit_length(SRP_GROUP["p"])%8:
     SRP_GROUP["p_size"] += 1
 
-class SecSocketException(Exception):
+class SecSocketException(LnstError):
     pass
 
 cryptography = not_imported
diff --git a/lnst/Common/ShellProcess.py b/lnst/Common/ShellProcess.py
index a4928ed..58c4bd1 100644
--- a/lnst/Common/ShellProcess.py
+++ b/lnst/Common/ShellProcess.py
@@ -13,9 +13,10 @@ import pty, os, termios, time, signal, re, select
 import logging, atexit
 from lnst.Common.Utils import wait_for
 from lnst.Common.ProcessManager import ProcessManager
+from lnst.Common.LnstError import LnstError
 
 class ShellProcess:
-    class ProcessError(Exception):
+    class ProcessError(LnstError):
         def __init__(self, patterns, output):
             Exception.__init__(self, patterns, output)
             self.patterns = patterns
diff --git a/lnst/Common/TestsCommon.py b/lnst/Common/TestsCommon.py
index accb6b8..471d720 100644
--- a/lnst/Common/TestsCommon.py
+++ b/lnst/Common/TestsCommon.py
@@ -16,6 +16,7 @@ import os
 import signal
 import time
 from lnst.Common.NetTestCommand import NetTestCommandGeneric
+from lnst.Common.LnstError import LnstError
 
 class testLogger(logging.Logger):
     def __init__(self, name, level=logging.NOTSET):
@@ -50,7 +51,7 @@ try:
 finally:
     logging._releaseLock()
 
-class TestOptionMissing(Exception):
+class TestOptionMissing(LnstError):
     pass
 
 class TestGeneric(NetTestCommandGeneric):
diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 539952e..3bfcf82 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -23,16 +23,17 @@ from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.Utils import wait_for, create_tar_archive
 from lnst.Common.Utils import check_process_running
 from lnst.Common.NetTestCommand import DEFAULT_TIMEOUT
+from lnst.Controller.Common import ControllerError
 from lnst.Controller.CtlSecSocket import CtlSecSocket
 
 # conditional support for libvirt
 if check_process_running("libvirtd"):
     from lnst.Controller.VirtUtils import VirtNetCtl, VirtDomainCtl
 
-class MachineError(Exception):
+class MachineError(ControllerError):
     pass
 
-class PrefixMissingError(Exception):
+class PrefixMissingError(ControllerError):
     pass
 
 class Machine(object):
diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py
index a144d00..5c288cc 100644
--- a/lnst/Controller/Task.py
+++ b/lnst/Controller/Task.py
@@ -18,6 +18,7 @@ from lnst.Common.Config import lnst_config
 from lnst.Controller.XmlTemplates import XmlTemplateError
 from lnst.Common.Path import Path
 from lnst.Controller.PerfRepoMapping import PerfRepoMapping
+from lnst.Controller.Common import ControllerError
 from lnst.Common.Utils import Noop
 
 try:
@@ -84,7 +85,8 @@ def match():
             return False
 
 
-class TaskError(Exception): pass
+class TaskError(ControllerError):
+    pass
 
 class ControllerAPI(object):
     """ An API class representing the controller. """
diff --git a/lnst/Controller/VirtUtils.py b/lnst/Controller/VirtUtils.py
index 750196c..c8da28a 100644
--- a/lnst/Controller/VirtUtils.py
+++ b/lnst/Controller/VirtUtils.py
@@ -16,6 +16,7 @@ import libvirt
 from libvirt import libvirtError
 from lnst.Common.ExecCmd import exec_cmd, ExecCmdFail
 from lnst.Common.NetUtils import scan_netdevs
+from lnst.Controller.Common import ControllerError
 
 #this is a global object because opening the connection to libvirt in every
 #object instance that uses it sometimes fails - the libvirt server probably
@@ -27,7 +28,7 @@ def init_libvirt_con():
     if _libvirt_conn is None:
         _libvirt_conn = libvirt.open(None)
 
-class VirtUtilsError(Exception):
+class VirtUtilsError(ControllerError):
     pass
 
 def _ip(cmd):
diff --git a/lnst/RecipeCommon/ModuleWrap.py b/lnst/RecipeCommon/ModuleWrap.py
index 2df953c..1f7acc6 100644
--- a/lnst/RecipeCommon/ModuleWrap.py
+++ b/lnst/RecipeCommon/ModuleWrap.py
@@ -10,6 +10,7 @@ __author__ = """
 olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
+from lnst.Common.LnstError import LnstError
 from lnst.Controller.Task import ctl
 
 def ping(src, dst, options={}, expect="pass", bg=False):
@@ -24,10 +25,10 @@ def ping(src, dst, options={}, expect="pass", bg=False):
 
     options = dict(options)
     if 'addr' in options or 'iface' in options:
-        raise Exception("options can't contain keys 'addr' and
'iface'")
+        raise LnstError("options can't contain keys 'addr' and
'iface'")
 
     if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
     try:
         if len(src) == 2:
             h1, if1 = src
@@ -39,10 +40,10 @@ def ping(src, dst, options={}, expect="pass", bg=False):
             h1, if1, addr_index1, addr_selector1 = src
             options["iface"] = if1.get_ip(addr_index1,
selector=addr_selector1)
     except:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
 
     if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
     try:
         if len(dst) == 3:
             h2, if2, addr_index2 = dst
@@ -51,7 +52,7 @@ def ping(src, dst, options={}, expect="pass", bg=False):
             h2, if2, addr_index2, addr_selector2 = dst
             options["addr"] = if2.get_ip(addr_index2, selector=addr_selector2)
     except:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
 
     ping_mod = ctl.get_module("IcmpPing",
                               options = options)
@@ -70,10 +71,10 @@ def ping6(src, dst, options={}, expect="pass", bg=False):
 
     options = dict(options)
     if 'addr' in options or 'iface' in options:
-        raise Exception("options can't contain keys 'addr' and
'iface'")
+        raise LnstError("options can't contain keys 'addr' and
'iface'")
 
     if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
     try:
         if len(src) == 2:
             h1, if1 = src
@@ -85,10 +86,10 @@ def ping6(src, dst, options={}, expect="pass", bg=False):
             h1, if1, addr_index1, addr_selector1 = src
             options["iface"] = if1.get_ip(addr_index1,
selector=addr_selector1)
     except:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
 
     if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
     try:
         if len(dst) == 3:
             h2, if2, addr_index2 = dst
@@ -97,7 +98,7 @@ def ping6(src, dst, options={}, expect="pass", bg=False):
             h2, if2, addr_index2, addr_selector2 = dst
             options["addr"] = if2.get_ip(addr_index2, selector=addr_selector2)
     except:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
 
     ping_mod = ctl.get_module("Icmp6Ping",
                               options = options)
@@ -123,17 +124,17 @@ def netperf(src, dst, server_opts={}, client_opts={}, baseline={},
timeout=60):
 
     server_opts = dict(server_opts)
     if 'bind' in server_opts or 'role' in server_opts:
-        raise Exception("server_opts can't contain keys 'bind' and
'role'")
+        raise LnstError("server_opts can't contain keys 'bind' and
'role'")
 
     client_opts = dict(client_opts)
     if 'bind' in client_opts or\
        'role' in client_opts or\
        'netperf_server' in client_opts:
-        raise Exception("client_opts can't contain keys 'bind',
'role' "\
+        raise LnstError("client_opts can't contain keys 'bind',
'role' "\
                         "and 'netperf_server'")
 
     if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
     try:
         if len(src) == 3:
             h1, if1, addr_index1 = src
@@ -142,10 +143,10 @@ def netperf(src, dst, server_opts={}, client_opts={}, baseline={},
timeout=60):
             h1, if1, addr_index1, addr_selector1 = src
             client_ip = if1.get_ip(addr_index1, selector=addr_selector1)
     except:
-        raise Exception('Invalid source specification')
+        raise LnstError('Invalid source specification')
 
     if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
     try:
         if len(dst) == 3:
             h2, if2, addr_index2 = dst
@@ -154,7 +155,7 @@ def netperf(src, dst, server_opts={}, client_opts={}, baseline={},
timeout=60):
             h2, if2, addr_index2, addr_selector2 = dst
             server_ip = if2.get_ip(addr_index2, addr_selector2)
     except:
-        raise Exception('Invalid destination specification')
+        raise LnstError('Invalid destination specification')
 
     server_opts["role"] = "server"
     server_opts["bind"] = server_ip
-- 
2.13.0

[PATCH v2 22/47] add lnst.Tests package

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This package will contain all upstream tracked test module classes. At
the moment it only contains the IcmpPing class which is a simple port
from the old IcmpPing test_module. In the future it should be heavily
refactored to support both ipv4 and ipv6 (instead of having 2 different
modules) but for early testing of Python Recipes this is good enough.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Tests/IcmpPing.py | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++
 lnst/Tests/__init__.py |  1 +
 2 files changed, 77 insertions(+)
 create mode 100644 lnst/Tests/IcmpPing.py
 create mode 100644 lnst/Tests/__init__.py

diff --git a/lnst/Tests/IcmpPing.py b/lnst/Tests/IcmpPing.py
new file mode 100644
index 0000000..8b07a2c
--- /dev/null
+++ b/lnst/Tests/IcmpPing.py
@@ -0,0 +1,76 @@
+import re
+import logging
+from lnst.Devices import Device
+from lnst.Common.Parameters import IntParam, Param, FloatParam, IpParam
+from lnst.Common.TestModule import BaseTestModule, TestModuleError
+from lnst.Common.ExecCmd import exec_cmd
+
+class IcmpPing(BaseTestModule):
+    """Port of old IcmpPing test modules"""
+    dst = IpParam(mandatory=True)
+    count = IntParam(default=10)
+    interval = FloatParam(default=1.0)
+    iface = Param()
+    size = IntParam()
+
+    limit_rate = IntParam(default=80)
+
+    def __init__(self, **kwargs):
+        super(IcmpPing, self).__init__(**kwargs)
+
+        if self.iface.set:
+            if not isinstance(self.iface.val, Device) and\
+               not isinstance(self.iface.val, str):
+                   raise TestModuleError("Invalid 'iface' parameter.")
+
+    def _compose_cmd(self):
+        cmd = "ping %s" % self.params.dst.val
+        cmd += " -c %d" % self.params.count
+        cmd += " -i %f" % self.params.interval
+        if self.params.iface.set:
+            if isinstance(self.params.iface.val, str):
+                cmd += " -I %s" % self.params.iface
+            elif isinstance(self.params.iface.val, Device):
+                pass
+                # cmd += " -I %s" % iface.val.devname
+        if self.params.size.set:
+            cmd += " -s %d" % self.params.size
+        return cmd
+
+    def run(self):
+        cmd = self._compose_cmd()
+
+        limit_rate = self.params.limit_rate
+
+        data_stdout = exec_cmd(cmd, die_on_err=False)[0]
+        stat_pttr1 = r'(\d+) packets transmitted, (\d+) received'
+        stat_pttr2 = r'rtt min/avg/max/mdev =
(\d+\.\d+)/(\d+\.\d+)/(\d+\.\d+)/(\d+\.\d+) ms'
+
+        match = re.search(stat_pttr1, data_stdout)
+        if not match:
+            self._res_data = {"msg": "expected pattern not found"}
+            return False
+
+        trans_pkts, recv_pkts = match.groups()
+        rate = int(round((float(recv_pkts) / float(trans_pkts)) * 100))
+        logging.debug("Transmitted \"%s\", received \"%s\",
"
+                      "rate \"%d%%\", limit_rate \"%d%%\""
+                      % (trans_pkts, recv_pkts, rate, limit_rate))
+
+        self._res_data = {"rate": rate,
+                          "limit_rate": limit_rate}
+
+        match = re.search(stat_pttr2, data_stdout)
+        if match:
+            tmin, tavg, tmax, tmdev = [float(x) for x in match.groups()]
+            logging.debug("rtt min \"%.3f\", avg \"%.3f\", max
\"%.3f\", "
+                          "mdev \"%.3f\"" % (tmin, tavg, tmax,
tmdev))
+
+            self._res_data["rtt_min"] = tmin
+            self._res_data["rtt_max"] = tmax
+
+        if rate < limit_rate:
+            self._res_data["msg"] = "rate is lower than limit"
+            return False
+
+        return True
diff --git a/lnst/Tests/__init__.py b/lnst/Tests/__init__.py
new file mode 100644
index 0000000..60da773
--- /dev/null
+++ b/lnst/Tests/__init__.py
@@ -0,0 +1 @@
+from lnst.Tests.IcmpPing import IcmpPing
-- 
2.13.0

[PATCH v2 23/47] lnst.Common.Config: remove {controller, slave}_init

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

The schema initialization methods were moved into Slave and Controller
specific Config classes so they're not needed in the base Config class
anymore.

In addition this adds an _init_options call to the __init__ method which
ensures that the schema will be created on object creation.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/Config.py | 143 ++------------------------------------------------
 1 file changed, 4 insertions(+), 139 deletions(-)

diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py
index 04d6f9b..afeab42 100644
--- a/lnst/Common/Config.py
+++ b/lnst/Common/Config.py
@@ -32,6 +32,10 @@ class Config():
     def __init__(self):
         self._options = dict()
         self.version = self._get_version()
+        self._init_options()
+
+    def _init_options(self):
+        raise NotImplementedError()
 
     def _get_version(self):
         # Check if I'm in git
@@ -56,145 +60,6 @@ class Config():
         os.chdir(cwd)
         return version
 
-    def controller_init(self):
-        self._options['environment'] = dict()
-        self._options['environment']['mac_pool_range'] = {\
-                "value" : ['52:54:01:00:00:01',
'52:54:01:FF:FF:FF'],
-                "additive" : False,
-                "action" : self.optionMacRange,
-                "name" : "mac_pool_range"}
-        self._options['environment']['rpcport'] = {\
-                "value" : DefaultRPCPort,
-                "additive" : False,
-                "action" : self.optionPort,
-                "name" : "rpcport"}
-        self._options['environment']['tool_dirs'] = {\
-                "value" : [],
-                "additive" : True,
-                "action" : self.optionDirList,
-                "name" : "test_tool_dirs"}
-        self._options['environment']['module_dirs'] = {\
-                "value" : [],
-                "additive" : True,
-                "action" : self.optionDirList,
-                "name" : "test_module_dirs"}
-        self._options['environment']['log_dir'] = {\
-                "value" : os.path.abspath(os.path.join(
-                    os.path.dirname(sys.argv[0]), './Logs')),
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "log_dir"}
-        self._options['environment']['resource_dir'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "resource_dir"}
-        self._options['environment']['xslt_url'] = {
-                "value" :
"http://www.lnst-project.org/files/result_xslt/xml_to_html.xsl",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "xslt_url"
-                }
-        self._options['environment']['allow_virtual'] = {
-                "value" : False,
-                "additive" : False,
-                "action" : self.optionBool,
-                "name" : "allow_virtual"
-                }
-
-        self._options['perfrepo'] = dict()
-        self._options['perfrepo']['url'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "url"
-                }
-        self._options['perfrepo']['username'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "username"
-                }
-        self._options['perfrepo']['password'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "password"
-                }
-
-        self._options['pools'] = dict()
-
-        self._options['security'] = dict()
-        self._options['security']['identity'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "identity"}
-        self._options['security']['privkey'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "privkey"}
-
-        self.colours_scheme()
-
-    def slave_init(self):
-        self._options['environment'] = dict()
-        self._options['environment']['log_dir'] = {\
-                "value" : os.path.abspath(os.path.join(
-                    os.path.dirname(sys.argv[0]), './Logs')),
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "log_dir"}
-        self._options['environment']['use_nm'] = {\
-                "value" : True,
-                "additive" : False,
-                "action" : self.optionBool,
-                "name" : "use_nm"}
-        self._options['environment']['rpcport'] = {\
-                "value" : DefaultRPCPort,
-                "additive" : False,
-                "action" : self.optionPort,
-                "name" : "rpcport"}
-
-        self._options['cache'] = dict()
-        self._options['cache']['dir'] = {\
-                "value" : os.path.abspath(os.path.join(
-                    os.path.dirname(sys.argv[0]), './cache')),
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "cache_dir"}
-
-        self._options['cache']['expiration_period'] = {\
-                "value" : 7*24*60*60, # 1 week
-                "additive" : False,
-                "action" : self.optionTimeval,
-                "name" : "expiration_period"}
-
-        self._options['security'] = dict()
-        self._options['security']['auth_types'] = {\
-                "value" : "none",
-                "additive" : False,
-                "action" : self.optionPlain, #TODO list??
-                "name" : "auth_types"}
-        self._options['security']['auth_password'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPlain,
-                "name" : "auth_password"}
-        self._options['security']['privkey'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "privkey"}
-        self._options['security']['ctl_pubkeys'] = {\
-                "value" : "",
-                "additive" : False,
-                "action" : self.optionPath,
-                "name" : "ctl_pubkeys"}
-
-        self.colours_scheme()
-
     def colours_scheme(self):
         self._options['colours'] = dict()
         self._options['colours']["disable_colours"] = {\
-- 
2.13.0

[PATCH v2 24/47] lnst.Common.Config: remove global lnst_config

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

All LNST object should now be using a local config object instead of a
global one so this can be removed.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/Config.py | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py
index afeab42..3745094 100644
--- a/lnst/Common/Config.py
+++ b/lnst/Common/Config.py
@@ -318,9 +318,3 @@ class Config():
             string = str(value)
 
         return string
-
-#Global object containing lnst configuration, available across modules
-#The object is created here but the contents are initialized
-#in lnst-ctl and lnst-slave, after that the modules that need the configuration
-#just import this object
-lnst_config = Config()
-- 
2.13.0

[PATCH v2 25/47] Slave: use a local config object instead of a global one

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst-slave | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/lnst-slave b/lnst-slave
index ebb4cd3..20696ec 100755
--- a/lnst-slave
+++ b/lnst-slave
@@ -17,8 +17,8 @@ import os
 import logging
 from lnst.Common.Daemon import Daemon
 from lnst.Common.Logs import LoggingCtl
-from lnst.Common.Config import lnst_config
 from lnst.Common.Colours import load_presets_from_config
+from lnst.Slave.Config import SlaveConfig
 from lnst.Slave.NetTestSlave import NetTestSlave
 
 def usage():
@@ -50,23 +50,23 @@ def main():
         usage()
         sys.exit()
 
-    lnst_config.slave_init()
+    slave_config = SlaveConfig()
     dirname = os.path.dirname(sys.argv[0])
     gitcfg = os.path.join(dirname, "lnst-slave.conf")
     if os.path.isfile(gitcfg):
-        lnst_config.load_config(gitcfg)
+        slave_config.load_config(gitcfg)
     else:
-        lnst_config.load_config('/etc/lnst-slave.conf')
+        slave_config.load_config('/etc/lnst-slave.conf')
 
     usr_cfg = os.path.expanduser('~/.lnst/lnst-slave.conf')
     if os.path.isfile(usr_cfg):
-        lnst_config.load_config(usr_cfg)
+        slave_config.load_config(usr_cfg)
 
     debug = False
     daemon = False
     pidfile = "/var/run/lnst-slave.pid"
     port = None
-    coloured_output = not lnst_config.get_option("colours",
"disable_colours")
+    coloured_output = not slave_config.get_option("colours",
"disable_colours")
     for opt, arg in opts:
         if opt in ("-d", "--debug"):
             debug = True
@@ -81,17 +81,17 @@ def main():
         elif opt in ("-m", "--no-colours"):
             coloured_output = False
 
-    load_presets_from_config(lnst_config)
+    load_presets_from_config(slave_config)
 
     log_ctl = LoggingCtl(debug,
-                     log_dir=lnst_config.get_option('environment',
'log_dir'),
+                     log_dir=slave_config.get_option('environment',
'log_dir'),
                      colours=coloured_output)
     logging.info("Started")
 
     if port != None:
-        lnst_config.set_option("environment", "rpcport", port)
+        slave_config.set_option("environment", "rpcport", port)
 
-    nettestslave = NetTestSlave(log_ctl)
+    nettestslave = NetTestSlave(log_ctl, slave_config)
 
     if daemon:
         daemon = Daemon(pidfile)
-- 
2.13.0

[PATCH v2 26/47] lnst.Common.Utils: add sha256sum function

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Calculates a SHA256 hexdigest of a file.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/Utils.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py
index d6d6c57..93e9560 100644
--- a/lnst/Common/Utils.py
+++ b/lnst/Common/Utils.py
@@ -108,6 +108,17 @@ def md5sum(file_path, block_size=2**20):
 
     return md5.hexdigest()
 
+def sha256sum(file_path):
+    sha256 = hashlib.sha256()
+    with open(file_path, "rb") as f:
+        while True:
+            data = f.read(1024)
+            if not data:
+                break
+            sha256.update(data)
+
+    return sha256.hexdigest()
+
 def create_tar_archive(input_path, target_path, compression=False):
     if compression:
         args = "cfj"
-- 
2.13.0

[PATCH v2 27/47] lnst.Common.ResourceCache: simplification

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Doesn't work with test_modules and test_tools anymore... for now it
simply handles files (add directories in future) and hashes them with
sha256. The cache will now be used for transferring python modules
(Devices or Tests for now), later could be expanded to include tester
specified files/directories.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/ResourceCache.py | 125 ++++++++++++++++++++-----------------------
 1 file changed, 57 insertions(+), 68 deletions(-)

diff --git a/lnst/Common/ResourceCache.py b/lnst/Common/ResourceCache.py
index 2037a0c..9cfac0d 100644
--- a/lnst/Common/ResourceCache.py
+++ b/lnst/Common/ResourceCache.py
@@ -14,10 +14,15 @@ import os
 import re
 import time
 import shutil
+import json
 from lnst.Common.ExecCmd import exec_cmd
+from lnst.Common.Utils import sha256sum
 from lnst.Common.LnstError import LnstError
 
-SETUP_SCRIPT_NAME = "lnst-setup.sh"
+#current index version
+INDEX_VERSION = 1
+#minimal supported index version -- will be updated to current one when loaded
+MIN_INDEX_VERSION = 1
 
 class ResourceCacheError(LnstError):
     pass
@@ -25,7 +30,6 @@ class ResourceCacheError(LnstError):
 class ResourceCache(object):
     _CACHE_INDEX_FILE_NAME = "index"
     _root = None
-    _entries = {}
     _expiration_period = None
 
     def __init__(self, cache_path, expiration_period):
@@ -38,98 +42,83 @@ class ResourceCache(object):
             os.makedirs(cache_path)
             self._root = cache_path
 
+        self._index = {"index_version": INDEX_VERSION,
+                       "entries": {}}
         self._read_index()
         self._expiration_period = expiration_period
 
     def _read_index(self):
-        logging.debug("Test cache index loaded")
         try:
-            f = open(self._get_index_loc(), "r")
+            with open(self.index_path, "w") as f:
+                index = json.load(f)
+                if index["index_version"] > INDEX_VERSION:
+                    raise ResourceCacheError("Incompatible ResourceCache index
versions")
+                elif index["index_version"] == INDEX_VERSION:
+                    self._index = index
+                    logging.debug("Resource cache index loaded")
+                else:
+                    self._index = self._update_old_index(index)
+                    logging.debug("Resource cache index loaded")
+                    self._save_index()
         except:
-            return
-
-        for line in f.readlines():
-            if not re.match("^\s*#", line) and not re.match("^\s*$",
line):
-                try:
-                    entry_hash, last_used, entry_type, \
-                    entry_name, entry_path = line.split()
-                except:
-                    raise ResourceCacheError("Malformed cache index")
+            pass
 
-                entry = {"type": entry_type, "name": entry_name,
-                         "last_used": int(last_used), "path":
entry_path }
-                self._entries[entry_hash] = entry
+    def _update_old_index(self, old):
+        if old["index_version"] < MIN_INDEX_VERSION:
+            raise ResourceCacheError("ResourceCache index version too old to
update")
+        logging.debug("Updating old index to newer version")
+        return old
 
     def _save_index(self):
-        logging.debug("Test cache index commited")
-        with open(self._get_index_loc(), "w") as f:
-            header = "# hash                           " \
-                     "last_used  type   name         path\n"
-            f.write(header)
-            for entry_hash, entry in self._entries.iteritems():
-                values = (entry_hash, entry["last_used"],
entry["type"],
-                            entry["name"], entry["path"])
-                line = "%s %d %s %s %s\n" % values
-                f.write(line)
-
-    def _get_index_loc(self):
-        return "%s/%s" % (self._root, self._CACHE_INDEX_FILE_NAME)
+        with open(self.index_path, "w") as f:
+            json.dump(self._index, f)
+            logging.debug("Resource cache index commited")
+
+    @property
+    def index_path(self):
+        return "%s/%s" % (self.root, self._CACHE_INDEX_FILE_NAME)
+
+    @property
+    def root(self):
+        return self._root
 
     def query(self, res_hash):
-        return res_hash in self._entries
+        return res_hash in self._index["entries"]
 
     def get_path(self, res_hash):
-        return "%s/%s" % (self._root,
self._entries[res_hash]["path"])
+        return self._index["entries"][res_hash]["path"]
 
     def renew_entry(self, entry_hash):
-        self._entries[entry_hash]["last_used"] = int(time.time())
+        self._index["entries"][entry_hash]["last_used"] =
int(time.time())
         self._save_index()
 
-    def add_cache_entry(self, entry_hash, filepath, entry_name, entry_type):
-        if entry_hash in self._entries:
+    def add_file_entry(self, filepath, entry_name):
+        entry_hash = sha256sum(filepath)
+
+        if entry_hash in self._index["entries"]:
             raise ResourceCacheError("File already in cache")
 
-        entry_dir = "%s/%s" % (self._root, entry_hash)
-        if os.path.exists(entry_dir):
-            try:
-                shutil.rmtree(entry_dir)
-            except OSError as e:
-                if e.errno != 2:
-                    raise
-        os.makedirs(entry_dir)
-
-        shutil.move(filepath, entry_dir)
-        entry_path = "%s/%s" % (entry_dir, os.path.basename(filepath))
-
-        if entry_type == "module":
-            filename = "%s.py" % entry_name
-            shutil.move(entry_path, "%s/%s" % (entry_dir, filename))
-        elif entry_type == "tools":
-            filename = entry_name
-            tools_dir = "%s/%s" % (entry_dir, filename)
-
-            exec_cmd("tar xjmf \"%s\" -C \"%s\"" %
(entry_path, entry_dir))
-
-            if os.path.exists("%s/%s" % (tools_dir, SETUP_SCRIPT_NAME)):
-                exec_cmd("cd \"%s\" && ./%s" % (tools_dir,
SETUP_SCRIPT_NAME))
-            else:
-                msg = "%s not found in %s tools, skipping initialization." % \
-                           (SETUP_SCRIPT_NAME, entry_name)
-                logging.warn(msg)
+        entry_path = "%s/%s" % (self._root, entry_hash)
+        if os.path.exists(entry_path):
+            os.remove(entry_path)
+
+        shutil.move(filepath, entry_path)
 
-        entry = {"type": entry_type, "name": entry_name,
+        entry = {"name": entry_name,
+                 "path": entry_path,
                  "last_used": int(time.time()),
-                 "path": "%s/%s" % (entry_hash, filename)}
-        self._entries[entry_hash] = entry
+                 "digest": entry_hash,
+                 "type": "file"}
+        self._index["entries"][entry_hash] = entry
 
         self._save_index()
 
         return entry_hash
 
     def del_cache_entry(self, entry_hash):
-        if entry_hash in self._entries:
-            shutil.rmtree("%s/%s" % (self._root, entry_hash))
-            del self._entries[entry_hash]
+        if entry_hash in self._index["entries"]:
+            os.remove(self._index["entries"][entry_hash].path)
+            del self._index["entries"][entry_hash]
             self._save_index()
 
     def del_old_entries(self):
@@ -138,7 +127,7 @@ class ResourceCache(object):
 
         rm = []
         now = time.time()
-        for entry_hash, entry in self._entries.iteritems():
+        for entry_hash, entry in self._index["entries"].iteritems():
             if entry["last_used"] <= (now - self._expiration_period):
                 rm.append(entry_hash)
 
-- 
2.13.0

[PATCH v2 28/47] lnst.Controller.CtlSecSocket: remove lnst_config import

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Wasn't used by the module.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/CtlSecSocket.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lnst/Controller/CtlSecSocket.py b/lnst/Controller/CtlSecSocket.py
index fd9ba26..c80917b 100644
--- a/lnst/Controller/CtlSecSocket.py
+++ b/lnst/Controller/CtlSecSocket.py
@@ -18,7 +18,6 @@ import logging
 from lnst.Common.SecureSocket import SecureSocket
 from lnst.Common.SecureSocket import DH_GROUP, SRP_GROUP
 from lnst.Common.SecureSocket import SecSocketException
-from lnst.Common.Config import lnst_config
 from lnst.Common.Utils import not_imported
 
 ser = not_imported
-- 
2.13.0

[PATCH v2 29/47] lnst.Controller.SlaveMachineParser: make standalone

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Since we're moving away from XML recipes, only slave machine description
files need to parse XML files. This commit pulls required methods from
the parent XmlParser into the SlaveMachineParser to make it standalone
so we can later remove all other XML related code.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/SlaveMachineParser.py | 144 ++++++++++++++++++++++++++++++----
 1 file changed, 128 insertions(+), 16 deletions(-)

diff --git a/lnst/Controller/SlaveMachineParser.py
b/lnst/Controller/SlaveMachineParser.py
index 70af0dc..7e08152 100644
--- a/lnst/Controller/SlaveMachineParser.py
+++ b/lnst/Controller/SlaveMachineParser.py
@@ -12,19 +12,79 @@ rpazdera(a)redhat.com (Radek Pazdera)
 """
 
 import os
-from lnst.Controller.XmlParser import XmlParser
-from lnst.Controller.XmlProcessing import XmlProcessingError, XmlData
-from lnst.Controller.XmlProcessing import XmlCollection
-
-class SlaveMachineError(XmlProcessingError):
-    pass
-
-class SlaveMachineParser(XmlParser):
-    def __init__(self, sm_path):
-        super(SlaveMachineParser, self).__init__("schema-sm.rng", sm_path)
+import sys
+from lxml import etree
+from lnst.Controller.Common import ControllerError
+
+class SlaveMachineParser(object):
+    def __init__(self, sm_path, ctl_config):
+        # locate the schema file
+        # try git path
+        dirname = os.path.dirname(sys.argv[0])
+        schema_path = os.path.join(dirname, "schema-sm.rng")
+        if not os.path.exists(schema_path):
+            # try configuration
+            res_dir = ctl_config.get_option("environment",
"resource_dir")
+            schema_path = os.path.join(res_dir, "schema-sm.rng")
+
+        if not os.path.exists(schema_path):
+            raise Exception("The schema file was not found. " + \
+                            "Your LNST installation is corrupt!")
+
+        self._path = sm_path
+        relaxng_doc = etree.parse(schema_path)
+        self._schema = etree.RelaxNG(relaxng_doc)
+
+    def parse(self):
+        try:
+            doc = self._parse(self._path)
+            self._remove_comments(doc)
+            self._schema.assertValid(doc)
+        except:
+            err = self._schema.error_log[0]
+            loc = {"file": os.path.basename(err.filename),
+                   "line": err.line, "col": err.column}
+            exc = XmlProcessingError(err.message)
+            exc.set_loc(loc)
+            raise exc
+
+        return self._process(doc)
+
+    def _parse(self, path):
+        try:
+            doc = etree.parse(path)
+        except etree.LxmlError as err:
+            # A workaround for cases when lxml (quite strangely)
+            # sets the filename to <string>.
+            if err.error_log[0].filename == "<string>":
+                filename = self._path
+            else:
+                filename = err.error_log[0].filename
+            loc = {"file": os.path.basename(filename),
+                   "line": err.error_log[0].line,
+                   "col": err.error_log[0].column}
+            exc = XmlProcessingError(err.error_log[0].message)
+            exc.set_loc(loc)
+            raise exc
+        except Exception as err:
+            loc = {"file": os.path.basename(self._path),
+                   "line": None,
+                   "col": None}
+            exc = XmlProcessingError(str(err))
+            exc.set_loc(loc)
+            raise exc
+
+        return doc
+
+    def _remove_comments(self, doc):
+        comments = doc.xpath('//comment()')
+        for c in comments:
+            p = c.getparent()
+            if p is not None:
+                p.remove(c)
 
     def _process(self, sm_tag):
-        sm = XmlData(sm_tag)
+        sm = {}
 
         # params
         params_tag = sm_tag.find("params")
@@ -35,7 +95,7 @@ class SlaveMachineParser(XmlParser):
         # interfaces
         interfaces_tag = sm_tag.find("interfaces")
         if interfaces_tag is not None and len(interfaces_tag) > 0:
-            sm["interfaces"] = XmlCollection(interfaces_tag)
+            sm["interfaces"] = []
             for eth_tag in interfaces_tag:
                 interface = self._process_interface(eth_tag)
                 sm["interfaces"].append(interface)
@@ -45,17 +105,17 @@ class SlaveMachineParser(XmlParser):
         return sm
 
     def _process_params(self, params_tag):
-        params = XmlCollection(params_tag)
+        params = []
         if params_tag is not None:
             for param_tag in params_tag:
-                param = XmlData(param_tag)
+                param = {}
                 param["name"] = self._get_attribute(param_tag,
"name")
                 param["value"] = self._get_attribute(param_tag,
"value")
                 params.append(param)
         return params
 
     def _process_interface(self, iface_tag):
-        iface = XmlData(iface_tag)
+        iface = {}
         iface["id"] = self._get_attribute(iface_tag, "id")
         iface["network"] = self._get_attribute(iface_tag, "label")
         iface["type"] = "eth"
@@ -69,7 +129,7 @@ class SlaveMachineParser(XmlParser):
         return iface
 
     def _process_security(self, sec_tag):
-        sec = XmlData(sec_tag)
+        sec = {}
 
         if sec_tag is None:
             sec["auth_type"] = "none"
@@ -95,3 +155,55 @@ class SlaveMachineParser(XmlParser):
             sec["pubkey_path"] = ""
 
         return sec
+
+    def _get_attribute(self, element, attr):
+        return element.attrib[attr].strip()
+
+class XmlProcessingError(ControllerError):
+    """ Exception thrown on parsing errors """
+
+    _filename = None
+    _line = None
+    _col = None
+
+    def __init__(self, msg, obj=None):
+        super(XmlProcessingError, self).__init__()
+        self._msg = msg
+
+        if obj is not None:
+            if hasattr(obj, "loc"):
+                self.set_loc(obj.loc)
+            elif hasattr(obj, "base") and obj.base != None:
+                loc = {}
+                loc["file"] = os.path.basename(obj.base)
+                if hasattr(obj, "sourceline"):
+                    loc["line"] = obj.sourceline
+                self.set_loc(loc)
+
+    def set_loc(self, loc):
+        self._filename = loc["file"]
+        self._line = loc["line"]
+        if "col" in loc:
+            self._col = loc["col"]
+
+    def __str__(self):
+        line = ""
+        col = ""
+        sep = ""
+        loc = ""
+        filename = "<unknown>"
+
+        if self._filename:
+            filename = self._filename
+
+        if self._line:
+            line = "%d" % self._line
+            sep = ":"
+
+        if self._col:
+            col = "%s%d" % (sep, self._col)
+
+        if self._line or self._col:
+            loc = "%s%s:" % (line, col)
+
+        return "Parser error: %s:%s %s" % (filename, loc, self._msg)
-- 
2.13.0

[PATCH v2 30/47] add lnst.Common.InterfaceManagerError

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines the Exception class for errors from the InterfaceManager.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/InterfaceManagerError.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 lnst/Common/InterfaceManagerError.py

diff --git a/lnst/Common/InterfaceManagerError.py b/lnst/Common/InterfaceManagerError.py
new file mode 100644
index 0000000..800e6c0
--- /dev/null
+++ b/lnst/Common/InterfaceManagerError.py
@@ -0,0 +1,16 @@
+"""
+Defines the InterfaceManagerError exception class.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+from lnst.Common.LnstError import LnstError
+
+class InterfaceManagerError(LnstError):
+    pass
-- 
2.13.0

[PATCH v2 31/47] add lnst.Slave.Job

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Defines classes of jobs to be run on a slave. Based on the old
lnst.Common.NetTestCommand code, but simplified to work with newer 'Job'
implementation that started on the Controller.

Defines a Job class that is used as a container object by the Slave to
fork and run a GenericJob object. For now only a ShellExecJob and a
ModuleJob are supported. Previous "control" commands are now only
methods of the Job container object.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Slave/Job.py | 251 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 251 insertions(+)
 create mode 100644 lnst/Slave/Job.py

diff --git a/lnst/Slave/Job.py b/lnst/Slave/Job.py
new file mode 100644
index 0000000..234cd61
--- /dev/null
+++ b/lnst/Slave/Job.py
@@ -0,0 +1,251 @@
+"""
+This module defines classes of jobs to be run on a slave
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import os
+import re
+import sys
+import signal
+import logging
+import multiprocessing
+from lnst.Common.JobError import JobError
+from lnst.Common.ExecCmd import exec_cmd, ExecCmdFail
+from lnst.Common.ConnectionHandler import send_data
+from lnst.Common.Logs import log_exc_traceback
+
+def get_job_class(what):
+    if what["type"] == "shell":
+        return ShellExecJob(what)
+    elif what["type"] == "module":
+        return ModuleJob(what)
+    else:
+        logging.error("Unknown job type \"%s\"" %
what["type"])
+        raise JobError("Unknown command type \"%s\"" %
what["type"])
+
+class JobContext(object):
+    def __init__(self):
+        self._dict = {}
+
+    def add_job(self, job):
+        self._dict[job.get_id()] = job
+
+    def del_job(self, job):
+        del self._dict[job.get_id()]
+
+    def get_job(self, id):
+        if id in self._dict:
+            return self._dict[id]
+        else:
+            return None
+
+    def _kill_all_jobs(self):
+        for id in self._dict:
+            self._dict[id].kill(signal=signal.SIGKILL)
+
+    def cleanup(self):
+        logging.debug("Cleaning up leftover processes.")
+        self._kill_all_jobs()
+        self._dict = {}
+
+    def get_parent_pipes(self):
+        pipes = {}
+        for key in self._dict:
+            pipe = self._dict[key].get_parent_pipe()
+            if pipe != None:
+                pipes[key] = pipe
+        return pipes
+
+class Job(object):
+    def __init__(self, what, log_ctl):
+        self._job_cls = get_job_class(what)
+        self._what = what
+
+        self._id = what["job_id"]
+        self._parent_pipe = None
+        self._child_pipe = None
+        self._process = None
+        self._pid = None
+        self._log_ctl = log_ctl
+        self._finished = False
+
+    def get_id(self):
+        return self._id
+
+    def get_parent_pipe(self):
+        return self._parent_pipe
+
+    def run(self):
+        self._parent_pipe, self._child_pipe = multiprocessing.Pipe()
+        self._process = multiprocessing.Process(target=self._run)
+
+        self._process.daemon = False
+        self._process.start()
+        self._pid = self._process.pid
+
+        logging.info("Running job %d with pid \"%d\"" % (self._id,
self._pid))
+        return True
+
+    def _run(self):
+        os.setpgrp()
+        signal.signal(signal.SIGHUP, signal.SIG_DFL)
+        signal.signal(signal.SIGINT, signal.SIG_DFL)
+        signal.signal(signal.SIGTERM, signal.SIG_DFL)
+
+        self._log_ctl.disable_logging()
+        self._log_ctl.set_connection(self._child_pipe)
+
+        result = {}
+        try:
+            self._job_cls.run()
+        except:
+            log_exc_traceback()
+            type, value, tb = sys.exc_info()
+            data = {"Exception": "%s" % value}
+            # self._job_cls.set_fail(data)
+        finally:
+            res_data = self._job_cls.get_result()
+            result["type"] = "job_finished"
+            result["job_id"] = self._id
+            result["result"] = res_data
+
+        send_data(self._child_pipe, result)
+        self._child_pipe.close()
+
+    def kill(self, signal=signal.SIGKILL):
+        if self._finished:
+            return True
+        try:
+            logging.debug("Sending signal %s to pid %d" % (signal, self._pid))
+            os.kill(self._pid, signal)
+            return True
+        except OSError as exc:
+            logging.error(str(exc))
+            return False
+
+    def join(self):
+        self._process.join()
+
+    def set_finished(self, result):
+        self._finished = True
+        self._result = result
+
+    def get_result(self):
+        return self._result
+
+class GenericJob(object):
+    def __init__(self, what):
+        self._what = what
+        self._result = {"passed": False,
+                        "res_data": None,
+                        "msg": None}
+
+    def run(self):
+        raise JobError("Method run must be defined.")
+
+    def get_result(self):
+        return self._result
+
+    # def format_res_data(self, res_data, level=0):
+        # self._check_res_data(res_data)
+        # formatted_data = ""
+        # if res_data:
+            # max_key_len = 0
+            # for key in res_data.keys():
+                # if len(key) > max_key_len:
+                    # max_key_len = len(key)
+            # for key, value in res_data.iteritems():
+                # if type(value) == dict:
+                    # formatted_data += level*4*" " + str(key) +
":\n"
+                    # formatted_data += self.format_res_data(value, level+1)
+                # if type(value) == list:
+                    # formatted_data += level*4*" " + str(key) +
":\n"
+                    # for i in range(0, len(value)):
+                        # formatted_data += (level+1)*4*" " +\
+                                          # "item %d:" % (i+1) +
"\n"
+                        # formatted_data += self.format_res_data(value[i],
+                                                               # level+2)
+                # else:
+                    # formatted_data += level*4*" " + str(key) + ":"
+ \
+                                      # (max_key_len-len(key))*" " + \
+                                      # "\t" + str(value) + "\n"
+
+        # return formatted_data
+
+    # def _check_res_data(self, res_data):
+        # name_start_char = u":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF"\
+                          # u"\u0370-\u037D\u037F-\u1FFF\u200C-\u200D"\
+                          # u"\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF"\
+                          # u"\uF900-\uFDCF\uFDF0-\uFFFD\U00010000-\U000EFFFF"
+        # name_char = name_start_char +
u"\-\.0-9\xB7\u0300-\u036F\u203F-\u2040"
+        # name = u"[%s]([%s])*$" % (name_start_char, name_char)
+        # char_data = u"[^<&]*"
+        # if isinstance(res_data, dict):
+            # for key in res_data:
+                # if not re.match(name, key, re.UNICODE):
+                    # msg = "'%s' can't be used as an xml element
name!" % key
+                    # raise JobError(msg)
+                # else:
+                    # self._check_res_data(res_data[key])
+        # elif isinstance(res_data, list):
+            # for i in res_data:
+                # self._check_res_data(i)
+        # else:
+            # try:
+                # string = str(res_data)
+            # except:
+                # msg = "res_data can only contain dictionaries, lists or "\
+                      # "stringable objects!"
+                # raise JobError(msg)
+            # if not re.match(char_data, string, re.UNICODE):
+                # msg = "'%s' can't be used as character data in
xml!" % string
+                # raise JobError(msg)
+
+    # def _format_cmd_res_header(self):
+        # if self._what["netns"] != None:
+            # netns = "(%s) " % self._what["netns"]
+        # else:
+            # netns = ""
+
+        # return "%-9s" % (self._what["type"] + netns)
+
+class ShellExecJob(GenericJob):
+    def run(self):
+        try:
+            stdout, stderr = exec_cmd(self._what["command"],
self._what["json"])
+            self._result["passed"] = True
+            self._result["res_data"] = {"stdout": stdout,
"stderr": stderr}
+        except ExecCmdFail as e:
+            self._result["passed"] = False
+            self._result["res_data"] = res_data = {"stdout":
e.get_stdout(),
+                                                   "stderr": e.get_stderr()}
+
+    # def _format_cmd_res_header(self):
+        # cmd_type = self._what["type"]
+        # cmd_val = self._what["command"]
+
+        # if self._what["netns"] != None:
+            # netns = "(%s) " % self._what["netns"]
+        # else:
+            # netns = ""
+
+        # cmd = "%-9scmd: \"%s\"" %(cmd_type + netns, cmd_val)
+        # return cmd
+
+class ModuleJob(GenericJob):
+    def run(self):
+        try:
+            self._result["passed"] = self._what["module"].run()
+            self._result["res_data"] =
self._what["module"].get_res_data()
+        except Exception as e:
+            self._result["passed"] = False
+            self._result["res_data"] = {"exception": str(e)}
+            # self._result["res_data"] = {"stdout": e.get_stdout(),
+                                        # "stderr": e.get_stderr()}
-- 
2.13.0

[PATCH v2 32/47] lnst.Slave.InterfaceManager: heavy reimplementation

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Reimplements the InterfaceManager to work with dynamically provided
Device classes. This means that Device creation/destruction was moved
completely into a Device class implementation and the InterfaceManager
only facilitates this call.

This reimplementation also completely removes the id<->if_id mapping
since this is not used by the Python recipes implementation. All Devices
can now be referred to only by their index (as reported by kernel),
though they can be searched according to other parameters as well.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Slave/InterfaceManager.py | 266 +++++++++++++----------------------------
 1 file changed, 80 insertions(+), 186 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 51aa6cc..da7e712 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -15,12 +15,13 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 import re
 import select
 import logging
-from lnst.Slave.NetConfigDevice import NetConfigDevice
 from lnst.Slave.NetConfigCommon import get_option
 from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.NetUtils import scan_netdevs
 from lnst.Common.ExecCmd import exec_cmd
 from lnst.Common.ConnectionHandler import recv_data
+from lnst.Common.DeviceError import DeviceNotFound
+from lnst.Common.InterfaceManagerError import InterfaceManagerError
 from lnst.Slave.DevlinkManager import DevlinkManager
 from pyroute2 import IPRSocket
 from pyroute2.netlink.rtnl import RTNLGRP_IPV4_IFADDR
@@ -37,51 +38,30 @@ except ImportError:
     from pyroute2.iproute import RTM_NEWADDR
     from pyroute2.iproute import RTM_DELADDR
 
-class IfMgrError(Exception):
-    pass
-
 NL_GROUPS = RTNLGRP_IPV4_IFADDR | RTNLGRP_IPV6_IFADDR | RTNLGRP_LINK
 
 class InterfaceManager(object):
     def __init__(self, server_handler):
+        self._device_classes = {}
+
         self._devices = {} #if_index to device
-        self._id_mapping = {} #id from the ctl to if_index
-        self._tmp_mapping = {} #id from the ctl to newly created device
 
         self._nl_socket = IPRSocket()
         self._nl_socket.bind(groups=NL_GROUPS)
 
         self._dl_manager = DevlinkManager()
 
-        self.rescan_devices()
-
         self._server_handler = server_handler
 
-    def map_if(self, if_id, if_index):
-        if if_id in self._id_mapping:
-            raise IfMgrError("Interface already mapped.")
-        elif if_index not in self._devices:
-            raise IfMgrError("No interface with index %s found." % if_index)
+    def clear_dev_classes(self):
+        self._device_classes = {}
 
-        self._id_mapping[if_id] = if_index
-        return
+    def add_device_class(self, name, cls):
+        if name in self._device_classes:
+            raise InterfaceManagerError("Device class name conflict %s" %
name)
 
-    def unmap_if(self, if_id):
-        if if_id in self._id_mapping:
-            del self._id_mapping[if_id]
-        elif if_id in self._tmp_mapping:
-            del self._tmp_mapping[if_id]
-        else:
-            pass
-
-    def clear_if_mapping(self):
-        self._id_mapping = {}
-
-    def get_id_by_if_index(self, if_index):
-        for if_id, index in self._id_mapping.iteritems():
-            if if_index == index:
-                return if_id
-        return None
+        self._device_classes[name] = cls
+        return cls
 
     def reconnect_netlink(self):
         if self._nl_socket != None:
@@ -100,42 +80,36 @@ class InterfaceManager(object):
         devs = scan_netdevs()
         for dev in devs:
             if dev['index'] not in self._devices:
-                device = None
-                for if_id, d in self._tmp_mapping.items():
-                    d_cfg = d.get_conf_dict()
-                    if d_cfg["name"] == dev["name"]:
-                        device = d
-                        self._id_mapping[if_id] = dev['index']
-                        del self._tmp_mapping[if_id]
-                        break
-                if device == None:
-                    device = Device(self)
-                device.init_netlink(dev['netlink_msg'])
+                device = self._device_classes["Device"](self)
+                device._init_netlink(dev['netlink_msg'])
                 self._devices[dev['index']] = device
+
+                update_msg = {"type": "dev_created",
+                              "dev_data": device._get_if_data()}
+                self._server_handler.send_data_to_ctl(update_msg)
             else:
-               
self._devices[dev['index']].update_netlink(dev['netlink_msg'])
+               
self._devices[dev['index']]._update_netlink(dev['netlink_msg'])
                 devices_to_remove.remove(dev['index'])
 
-            self._devices[dev['index']].clear_ips()
+            self._devices[dev['index']]._clear_ips()
             for addr_msg in dev['ip_addrs']:
-                self._devices[dev['index']].update_netlink(addr_msg)
+                self._devices[dev['index']]._update_netlink(addr_msg)
         for i in devices_to_remove:
-            if self._devices[i].get_netns() != None:
-                continue
-
-            dev_name = self._devices[i].get_name()
+            dev_name = self._devices[i].name
             logging.debug("Deleting Device with if_index %d, name %s because
"\
                           "it doesn't exist anymore." % (i, dev_name))
 
-            del_msg = {"type": "if_deleted",
+            self._devices[i]._deleted = True
+            del self._devices[i]
+
+            del_msg = {"type": "dev_deleted",
                        "if_index": i}
             self._server_handler.send_data_to_ctl(del_msg)
-            del self._devices[i]
 
         self._dl_manager.rescan_ports()
         for device in self._devices.values():
-            dl_port = self._dl_manager.get_port(device.get_name())
-            device.set_devlink(dl_port)
+            dl_port = self._dl_manager.get_port(device.name)
+            device._set_devlink(dl_port)
 
     def handle_netlink_msgs(self, msgs):
         for msg in msgs:
@@ -143,86 +117,61 @@ class InterfaceManager(object):
 
         self._dl_manager.rescan_ports()
         for device in self._devices.values():
-            dl_port = self._dl_manager.get_port(device.get_name())
-            device.set_devlink(dl_port)
+            dl_port = self._dl_manager.get_port(device.name)
+            device._set_devlink(dl_port)
 
     def _handle_netlink_msg(self, msg):
         if msg['header']['type'] in [RTM_NEWLINK, RTM_NEWADDR,
RTM_DELADDR]:
             if msg['index'] in self._devices:
-                update_msg = self._devices[msg['index']].update_netlink(msg)
-                if update_msg != None:
-                    for if_id, if_index in self._id_mapping.iteritems():
-                        if if_index == msg['index']:
-                            update_msg["if_id"] = if_id
-                            break
-                    self._server_handler.send_data_to_ctl(update_msg)
+                self._devices[msg['index']]._update_netlink(msg)
             elif msg['header']['type'] == RTM_NEWLINK:
-                dev = None
-                for if_id, d in self._tmp_mapping.items():
-                    d_cfg = d.get_conf_dict()
-                    if d_cfg["name"] == msg.get_attr("IFLA_IFNAME"):
-                        dev = d
-                        self._id_mapping[if_id] = msg['index']
-                        del self._tmp_mapping[if_id]
-                        break
-                if dev == None:
-                    dev = Device(self)
-                update_msg = dev.init_netlink(msg)
+                dev = self._device_classes["Device"](self)
+                dev._init_netlink(msg)
                 self._devices[msg['index']] = dev
 
-                if update_msg != None:
-                    for if_id, if_index in self._id_mapping.iteritems():
-                        if if_index == msg['index']:
-                            update_msg["if_id"] = if_id
-                            break
-                    self._server_handler.send_data_to_ctl(update_msg)
-
+                update_msg = {"type": "dev_created",
+                              "dev_data": dev._get_if_data()}
+                self._server_handler.send_data_to_ctl(update_msg)
         elif msg['header']['type'] == RTM_DELLINK:
             if msg['index'] in self._devices:
                 dev = self._devices[msg['index']]
-                if dev.get_netns() == None and dev.get_conf_dict() == None:
-                    dev.del_link()
-                    del self._devices[msg['index']]
+                dev._deleted = True
 
-                    del_msg = {"type": "if_deleted",
-                               "if_index": msg['index']}
-                    self._server_handler.send_data_to_ctl(del_msg)
-        else:
-            return
+                del self._devices[msg['index']]
 
-    def get_mapped_device(self, if_id):
-        if if_id in self._id_mapping:
-            if_index = self._id_mapping[if_id]
-            return self._devices[if_index]
-        elif if_id in self._tmp_mapping:
-            return self._tmp_mapping[if_id]
+                del_msg = {"type": "dev_deleted",
+                           "if_index": msg['index']}
+                self._server_handler.send_data_to_ctl(del_msg)
         else:
-            return None
-
-    def get_mapped_devices(self):
-        ret = {}
-        for if_id, if_index in self._id_mapping.iteritems():
-            ret[if_id] = self._devices[if_index]
-        for if_id in self._tmp_mapping:
-            ret[if_id] = self._tmp_mapping[if_id]
-        return ret
+            return
 
     def get_device(self, if_index):
+        self.rescan_devices()
         if if_index in self._devices:
             return self._devices[if_index]
         else:
-            return None
+            raise DeviceNotFound()
 
     def get_devices(self):
+        self.rescan_devices()
         return self._devices.values()
 
     def get_device_by_hwaddr(self, hwaddr):
+        self.rescan_devices()
         for dev in self._devices.values():
-            if dev.get_hwaddr() == hwaddr:
+            if dev.hwaddr == hwaddr:
                 return dev
-        return None
+        raise DeviceNotFound()
+
+    def get_device_by_name(self, name):
+        self.rescan_devices()
+        for dev in self._devices.values():
+            if dev.name == name:
+                return dev
+        raise DeviceNotFound()
 
     def get_device_by_params(self, params):
+        self.rescan_devices()
         matched = None
         for dev in self._devices.values():
             matched = dev
@@ -239,63 +188,44 @@ class InterfaceManager(object):
 
     def deconfigure_all(self):
         for dev in self._devices.itervalues():
-            dev.clear_configuration()
-
-    def create_device_from_config(self, if_id, config):
-        if config["type"] == "eth":
-            raise IfMgrError("Ethernet devices can't be created.")
-
-        config["name"] = self.assign_name(config)
+            pass
+            # dev.clear_configuration()
 
-        device = Device(self)
-        self._tmp_mapping[if_id] = device
+    def create_device(self, clsname, args=[], kwargs={}):
+        devcls = self._device_classes[clsname]
 
-        device.set_configuration(config)
+        device = devcls(self, *args, **kwargs)
         device.create()
 
-        return config["name"]
-
-    def create_device_pair(self, if_id1, config1, if_id2, config2):
-        name1, name2 = self.assign_name(config1)
-        config1["name"] = name1
-        config2["name"] = name2
-        config1["peer_name"] = name2
-        config2["peer_name"] = name1
-
-        device1 = Device(self)
-        device2 = Device(self)
-        self._tmp_mapping[if_id1] = device1
-        self._tmp_mapping[if_id2] = device2
-
-        device1.set_configuration(config1)
-        device2.set_configuration(config2)
-        device1.create()
-
-        device1.set_peer(device2)
-        device2.set_peer(device1)
-        return name1, name2
-
-    def wait_interface_init(self):
-        while len(self._tmp_mapping) > 0:
-            rl, wl, xl = select.select([self._nl_socket], [], [], 1)
+        devs = scan_netdevs()
+        for dev in devs:
+            if dev["name"] == device.name:
+                device._init_netlink(dev['netlink_msg'])
+                self._devices[dev['index']] = device
+                return device
 
-            if len(rl) == 0:
-                continue
+        return None
 
-            msgs = recv_data(self._nl_socket)["data"]
-            self.handle_netlink_msgs(msgs)
+    def replace_dev(self, if_id, dev):
+        del self._devices[if_id]
+        self._devices[if_id] = dev
 
     def _is_name_used(self, name):
         self.rescan_devices()
         for device in self._devices.itervalues():
-            if name == device.get_name():
-                return True
-        for device in self._tmp_mapping.itervalues():
-            if name == device.get_name():
+            if name == device.name:
                 return True
+
+        out, _ = exec_cmd("ovs-vsctl --columns=name list Interface",
+                          log_outputs=False, die_on_err=False)
+        for line in out.split("\n"):
+            m = re.match(r'.*: \"(.*)\"', line)
+            if m is not None:
+                if name == m.group(1):
+                    return True
         return False
 
-    def assign_name_generic(self, prefix):
+    def assign_name(self, prefix):
         index = 0
         while (self._is_name_used(prefix + str(index))):
             index += 1
@@ -311,42 +241,6 @@ class InterfaceManager(object):
             index2 += 1
         return prefix + str(index1), prefix + str(index2)
 
-    def assign_name(self, config):
-        if "name" in config:
-            return config["name"]
-        dev_type = config["type"]
-        if dev_type == "eth":
-            if (not "hwaddr" in config or
-                "name" in config):
-                return
-            hwaddr = normalize_hwaddr(config["hwaddr"])
-            for dev in self._devices:
-                if dev.get_hwaddr() == hwaddr:
-                    return dev.get_name()
-        elif dev_type == "bond":
-            return self.assign_name_generic("t_bond")
-        elif dev_type == "bridge" or dev_type == "ovs_bridge":
-            return self.assign_name_generic("t_br")
-        elif dev_type == "macvlan":
-            return self.assign_name_generic("t_macvlan")
-        elif dev_type == "team":
-            return self.assign_name_generic("t_team")
-        elif dev_type == "vlan":
-            netdev_name =
self.get_mapped_device(config["slaves"][0]).get_name()
-            vlan_tci = get_option(config, "vlan_tci")
-            prefix = "%s.%s_" % (netdev_name, vlan_tci)
-            return self.assign_name_generic(prefix)
-        elif dev_type == "veth":
-            return self._assign_name_pair("veth")
-        elif dev_type == "vti":
-            return self.assign_name_generic("vti")
-        elif dev_type == "vti6":
-            return self.assign_name_generic("t_ip6vti")
-        elif dev_type == "vxlan":
-            return self.assign_name_generic("vxlan")
-        else:
-            return self.assign_name_generic("dev")
-
 class Device(object):
     def __init__(self, if_manager):
         self._initialized = False
-- 
2.13.0

[PATCH v2 33/47] Machine, NetTestSlave: heavy reimplementation

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This commit heavily reimplements the lnst.Controller.Machine and
lnst.Slave.NetTestSlave modules. Since these modules directly depend on
each other, these changes are introduced together in the same commit.

On the Controller side:
* this switches the Machine class implementation from working with the
old XML recipe code to work purely with the Python recipe
implementation, where the Machine class is instantiated by a
SlavePoolManager and is used by the tester through the Host class
providing a tester facing API.

* the Machine class now needs to synchronize different resources to the
slave, namely:
    * Device classes from the lnst.Devices package
    * test module code, which is instantiated on the Controller and send
      to the Slave when required (including all parent classes required
      for the reconstruction of the object on the Slave).
    * In the future this can be extended to also send the
      InterfaceManager implementation, and maybe even SlaveMethods
      implementation.

* includes methods relaying calling the Device methods

* Removes the old Interface implementation since this is all handled by
the new Device classes. And the original simple Device class.

* refactors the rpc_call method to only have one implementation for
everything (including network namespaces, though they're not supported
atm...).

* switches to using the new Job implementation.

On the Slave side:
* accept Device, and test module classes, all the dynamically received
classes are stored in the new Resource cache. If a dynamically received
class is imported during recipe execution it is also cleaned up when the
recipe finishes execution.

* ServerHandler now translates Device objects into Device references
when sending these objects over sockets.

* switches to using the new Job implementation.

* comments out old code that has not been refactored and is not
supported yet with the python recipe implementation - mostly network
namespaces.

* all exceptions are now transferred to the Controller where they'll be
reconstructed and raised so they can be handled there (unless they can
be handled on the Slave)

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * improved class and module synchronization implementation
    * join split lines in deviceref_to_device method
---
 lnst/Controller/Machine.py | 1327 ++++++++++----------------------------------
 lnst/Slave/NetTestSlave.py |  829 +++++++++++++--------------
 2 files changed, 678 insertions(+), 1478 deletions(-)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 3bfcf82..f057390 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -14,17 +14,24 @@ rpazdera(a)redhat.com (Radek Pazdera)
 import logging
 import socket
 import os
+import sys
 import tempfile
 import signal
 from time import sleep
 from xmlrpclib import Binary
-from functools import wraps
 from lnst.Common.NetUtils import normalize_hwaddr
+from lnst.Common.Utils import sha256sum
 from lnst.Common.Utils import wait_for, create_tar_archive
 from lnst.Common.Utils import check_process_running
+from lnst.Common.TestModule import BaseTestModule
 from lnst.Common.NetTestCommand import DEFAULT_TIMEOUT
+from lnst.Common.DeviceError import DeviceDeleted, DeviceNotFound
 from lnst.Controller.Common import ControllerError
 from lnst.Controller.CtlSecSocket import CtlSecSocket
+from lnst.Devices import device_classes
+from lnst.Devices.Device import Device
+from lnst.Devices.RemoteDevice import RemoteDevice
+from lnst.Devices.VirtualDevice import VirtualDevice
 
 # conditional support for libvirt
 if check_process_running("libvirtd"):
@@ -51,7 +58,6 @@ class Machine(object):
         self._ctl_config = ctl_config
         self._slave_desc = None
         self._connection = None
-        self._configured = False
         self._system_config = {}
         self._security = security
         self._security["identity"] =
ctl_config.get_option("security",
@@ -76,8 +82,19 @@ class Machine(object):
         self._interfaces = []
         self._namespaces = []
         self._bg_cmds = {}
+        self._jobs = {}
+        self._job_id_seq = 0
 
         self._device_database = {}
+        self._tmp_device_database = []
+
+        self._init_connection()
+
+    def set_id(self, new_id):
+        self._id = new_id
+
+    def get_id(self):
+        return self._id
 
     def get_configuration(self):
         configuration = {}
@@ -87,150 +104,96 @@ class Machine(object):
         configuration["redhat_release"] =
self._slave_desc["redhat_release"]
 
         configuration["interfaces"] = {}
-        for i in self._interfaces:
-            if not isinstance(i, UnusedInterface):
-                configuration["interface_"+i.get_id()] = i.get_config()
+        for dev in self._device_database.items():
+            configuration["device_"+dev.name] = dev.get_config()
         return configuration
 
-    def _if_id_exists(self, if_id):
-        for iface in self._interfaces:
-            if if_id == iface.get_id():
-                return True
-        return False
+    def add_tmp_device(self, dev):
+        self._tmp_device_database.append(dev)
+
+    def create_remote_device(self, dev):
+        dev_clsname = dev._dev_cls.__name__
+        dev_args = dev._dev_args
+        dev_kwargs = dev._dev_kwargs
+        ret = self.rpc_call("create_device", clsname=dev_clsname,
+                                             args=dev_args,
+                                             kwargs=dev_kwargs)
+        dev.if_index = ret["if_index"]
+        self._device_database[ret["if_index"]] = dev
+
+    def device_created(self, dev_data):
+        if_index = dev_data["if_index"]
+        if if_index not in self._device_database:
+            new_dev = None
+            if len(self._tmp_device_database) > 0:
+                for dev in self._tmp_device_database:
+                    if dev._match_update_data(dev_data):
+                        new_dev = dev
+                        break
 
-    def _generate_if_id(self, if_type):
-        i = 0
-        while True:
-            if_id = "gen_%s_%d" % (if_type, i)
-            if not self._if_id_exists(if_id):
-                break
-            i += 1
-        return if_id
-
-    def _add_interface(self, if_id, if_type, cls):
-        if if_id != None:
-            if self._if_id_exists(if_id):
-                msg = "Interface '%s' already exists on machine
'%s'" \
-                                             % (if_id, self._id)
-                raise MachineError(msg)
-        else:
-            if_id = self._generate_if_id(if_type)
+            if new_dev is None:
+                new_dev = RemoteDevice(Device)
+                new_dev.host = self
+                new_dev.if_index = if_index
+            else:
+                self._tmp_device_database.remove(new_dev)
 
-        iface = cls(self, if_id, if_type)
-        self._interfaces.append(iface)
-        return iface
+                new_dev.if_index = dev_data["if_index"]
 
-    def remove_interface(self, if_id):
-        iface = self.get_interface(if_id)
-        self._interfaces.remove(iface)
+            self._device_database[if_index] = new_dev
 
-    def interface_update(self, if_data):
-        try:
-            iface = self.get_interface(if_data["if_id"])
-        except:
-            iface = None
-        if iface:
-            iface.update(if_data['if_data'])
+    def device_delete(self, dev_data):
+        if dev_data["if_index"] in self._device_database:
+            self._device_database[dev_data["if_index"]].deleted = True
 
-        if if_data["if_data"]["if_index"] in self._device_database:
-            dev =
self._device_database[if_data["if_data"]["if_index"]]
-            dev.update_data(if_data['if_data'])
+    def dev_db_get_if_index(self, if_index):
+        if if_index in self._device_database:
+            return self._device_database[if_index]
         else:
-            dev = Device(if_data["if_data"], self)
-            self._device_database[if_data["if_data"]["if_index"]] =
dev
-
-    def dev_db_delete(self, update_msg):
-        if update_msg["if_index"] in self._device_database:
-            del self._device_database[update_msg["if_index"]]
+            return None
 
     def dev_db_get_name(self, dev_name):
+        #TODO move these to Slave to optimize quering for each device
         for if_index, dev in self._device_database.iteritems():
             if dev.get_name() == dev_name:
                 return dev
         return None
 
-    #
-    # Factory methods for constructing interfaces on this machine. The
-    # types of interfaces are explained with the classes below.
-    #
-    def new_static_interface(self, if_id, if_type):
-        return self._add_interface(if_id, if_type, StaticInterface)
-
-    def new_unused_interface(self, if_type):
-        return self._add_interface(None, if_type, UnusedInterface)
-
-    def new_virtual_interface(self, if_id, if_type):
-        return self._add_interface(if_id, if_type, VirtualInterface)
-
-    def new_soft_interface(self, if_id, if_type):
-        return self._add_interface(if_id, if_type, SoftInterface)
-
-    def new_loopback_interface(self, if_id):
-        return self._add_interface(if_id, 'lo', LoopbackInterface)
-
-    def get_interface(self, if_id):
-        for iface in self._interfaces:
-            if iface.get_id != None and if_id == iface.get_id():
-                return iface
-
-        msg = "Interface '%s' not found on machine '%s'" %
(if_id, self._id)
-        raise MachineError(msg)
-
-    def get_interfaces(self):
-        return self._interfaces
-
-    def get_ordered_interfaces(self):
-        ordered_list = list(self._interfaces)
-        change = True
-        while change:
-            change = False
-            swap = False
-            ind1 = 0
-            ind2 = 0
-            for i in ordered_list:
-                master = i.get_primary_master()
-                if master != None:
-                    ind1 = ordered_list.index(i)
-                    ind2 = ordered_list.index(master)
-                    if ind1 > ind2:
-                        swap = True
-                        break
-            if swap:
-                change = True
-                tmp = ordered_list[ind1]
-                ordered_list[ind1] = ordered_list[ind2]
-                ordered_list[ind2] = tmp
-        return ordered_list
-
-    def _rpc_call(self, method_name, *args):
-        data = {"type": "command", "method_name":
method_name, "args": args}
-
-        self._msg_dispatcher.send_message(self._id, data)
-        result = self._msg_dispatcher.wait_for_result(self._id)
-
-        return result
+    def get_dev_by_hwaddr(self, hwaddr):
+        #TODO move these to Slave to optimize quering for each device
+        for if_index, dev in self._device_database.iteritems():
+            if dev.hwaddr == hwaddr:
+                return dev
+        return None
 
-    def _rpc_call_to_netns(self, netns, method_name, *args):
-        data = {"type": "command", "method_name":
method_name, "args": args}
-        msg = {"type": "to_netns", "netns": netns,
"data": data}
+    def rpc_call(self, method_name, *args, **kwargs):
+        if "netns" in kwargs and kwargs["netns"] is not None:
+            netns = kwargs["netns"]
+            del kwargs["netns"]
+            msg = {"type": "to_netns",
+                   "netns": netns,
+                   "data": {"type": "command",
+                            "method_name": method_name,
+                            "args": args,
+                            "kwargs": kwargs}}
+        else:
+            if "netns" in kwargs:
+                del kwargs["netns"]
+            msg = {"type": "command",
+                   "method_name": method_name,
+                   "args": args,
+                   "kwargs": kwargs}
 
-        self._msg_dispatcher.send_message(self._id, msg)
-        result = self._msg_dispatcher.wait_for_result(self._id)
+        self._msg_dispatcher.send_message(self, msg)
+        result = self._msg_dispatcher.wait_for_result(self)
 
         return result
 
-    def _rpc_call_x(self, netns, method_name, *args):
-        if not netns:
-            return self._rpc_call(method_name, *args)
-        return self._rpc_call_to_netns(netns, method_name, *args)
-
-    def init_connection(self, recipe_name):
+    def _init_connection(self):
         """ Initialize the slave connection
 
-            Calling this method will initialize the rpc connection to the
-            machine and initialize all the interfaces. Note, that it will
-            *not* configure the interfaces. They need to be configured
-            individually later on.
+        This will connect to the Slave, get it's description (should be
+        usable for matching), and checks version compatibility
         """
         hostname = self._hostname
         port = self._port
@@ -242,7 +205,7 @@ class Machine(object):
 
         self._msg_dispatcher.add_slave(self, connection)
 
-        hello, slave_desc = self._rpc_call("hello", recipe_name)
+        hello, slave_desc = self.rpc_call("hello")
         if hello != "hello":
             msg = "Unable to establish RPC connection " \
                   "to machine %s, handshake failed!" % hostname
@@ -266,14 +229,45 @@ class Machine(object):
 
         self._slave_desc = slave_desc
 
-        devices = self._rpc_call("get_devices")
+    def set_recipe(self, recipe_name):
+        """ Reserves the machine for the specified recipe
+
+        Also sends Device classes from the controller and initializes the
+        InterfaceManager on the Slave and builds the device database.
+        """
+        self.rpc_call("set_recipe", recipe_name)
+        self._send_device_classes()
+        self.rpc_call("init_if_manager")
+
+        devices = self.rpc_call("get_devices")
         for if_index, dev in devices.items():
-            self._device_database[if_index] = Device(dev, self)
+            remote_dev = RemoteDevice(Device)
+            remote_dev.host = self
+            remote_dev.if_index = if_index
 
-        for iface in self._interfaces:
-            iface.initialize()
+            self._device_database[if_index] = remote_dev
+
+    def _send_device_classes(self):
+        classes = []
+        for cls_name, cls in device_classes:
+            classes.extend(reversed(self._get_base_classes(cls)))
+
+        for cls in classes:
+            if cls is object:
+                continue
+            module_name = cls.__module__
+            module = sys.modules[module_name]
+            filename = module.__file__
 
-        self._configured = True
+            if filename[-3:] == "pyc":
+                filename = filename[:-1]
+
+            res_hash = self.sync_resource(module_name, filename)
+            self.rpc_call("load_cached_module", module_name, res_hash)
+
+        for cls_name, cls in device_classes:
+            module_name = cls.__module__
+            self.rpc_call("map_device_class", cls_name, module_name)
 
     def is_git_version(self, version):
         try:
@@ -282,10 +276,12 @@ class Machine(object):
         except ValueError:
             return True
 
-    def is_configured(self):
-        """ Test if the machine was configured """
-
-        return self._configured
+    def cleanup_devices(self):
+        self.rpc_call("destroy_devices")
+        for dev in self._device_database.values():
+            if isinstance(dev, VirtualDevice):
+                dev.destroy()
+        self._device_database = {}
 
     def cleanup(self):
         """ Clean the machine up
@@ -295,114 +291,144 @@ class Machine(object):
             all the interfaces that have been configured on the machine,
             and finalize and close the rpc connection to the machine.
         """
-        if not self._configured:
-            return
-
         #connection to the slave was closed
-        if not self._msg_dispatcher.get_connection(self._id):
+        if not self._msg_dispatcher.get_connection(self):
             return
 
-        ordered_ifaces = self.get_ordered_interfaces()
         try:
             #dump statistics
-            for iface in self._interfaces:
-                # Getting stats only from real interfaces
-                if isinstance(iface, UnusedInterface):
-                    continue
-                stats = iface.link_stats()
-                logging.debug("%s:%s:%s: RX:\t bytes: %d\t packets: %d\t dropped:
%d" %
-                              (iface.get_netns(), iface.get_host(), iface.get_id(),
-                              stats["rx_bytes"], stats["rx_packets"],
stats["rx_dropped"]))
-                logging.debug("%s:%s:%s: TX:\t bytes: %d\t packets: %d\t dropped:
%d" %
-                              (iface.get_netns(), iface.get_host(), iface.get_id(),
-                              stats["tx_bytes"], stats["tx_packets"],
stats["tx_dropped"]))
-
-            self._rpc_call("kill_cmds")
+            # for iface in self._interfaces:
+                # # Getting stats only from real interfaces
+                # if isinstance(iface, UnusedInterface):
+                    # continue
+                # stats = iface.link_stats()
+                # logging.debug("%s:%s:%s: RX:\t bytes: %d\t packets: %d\t dropped:
%d" %
+                              # (iface.get_netns(), iface.get_host(), iface.get_id(),
+                              # stats["rx_bytes"],
stats["rx_packets"], stats["rx_dropped"]))
+                # logging.debug("%s:%s:%s: TX:\t bytes: %d\t packets: %d\t dropped:
%d" %
+                              # (iface.get_netns(), iface.get_host(), iface.get_id(),
+                              # stats["tx_bytes"],
stats["tx_packets"], stats["tx_dropped"]))
+
+            self.rpc_call("kill_jobs")
             for netns in self._namespaces:
-                self._rpc_call_to_netns(netns, "kill_cmds")
+                self.rpc_call("kill_jobs", netns=netns)
 
             self.restore_system_config()
-
-            ordered_ifaces.reverse()
-            for iface in ordered_ifaces:
-                iface.deconfigure()
-            for iface in ordered_ifaces:
-                iface.cleanup()
-
-            self.del_namespaces()
-
-            self.restore_nm_option()
-            self._rpc_call("bye")
+            self.cleanup_devices()
+            # self.del_namespaces()
+            # self.restore_nm_option()
+            self.rpc_call("bye")
         except:
             #cleanup is only meaningful on dynamic interfaces, and should
             #always be called when deconfiguration happens- especially
             #when something on the slave breaks during deconfiguration
-            for iface in ordered_ifaces:
-                if not isinstance(iface, VirtualInterface):
-                    continue
-                iface.cleanup()
+            self.cleanup_devices()
             raise
-        finally:
-            self._msg_dispatcher.disconnect_slave(self.get_id())
-
-            self._configured = False
 
     def _timeout_handler(self, signum, frame):
-        msg = "RPC connection to machine %s timed out" % self.get_id()
+        msg = "Timeout expired on machine %s" % self.get_id()
         raise MachineError(msg)
 
-    def run_command(self, command):
-        """ Run a command on the machine """
+    def _get_base_classes(self, cls):
+        new_bases = [cls] + list(cls.__bases__)
+        bases = []
+        while len(new_bases) != len(bases):
+            bases = new_bases
+            new_bases = list(bases)
+            for b in bases:
+                for bs in b.__bases__:
+                    if bs not in new_bases:
+                        new_bases.append(bs)
+        return new_bases
+
+    def run_job(self, job):
+        job.id = self._job_id_seq
+        self._job_id_seq += 1
+        self._jobs[job.id] = job
+
+        if job._type == "module":
+            classes = [job._what]
+            classes.extend(self._get_base_classes(job._what.__class__))
+
+            for cls in reversed(classes):
+                if cls is object or cls is BaseTestModule:
+                    continue
+                m_name = cls.__module__
+                m = sys.modules[m_name]
+                filename = m.__file__
+                if filename[-3:] == "pyc":
+                    filename = filename[:-1]
 
-        prev_handler = signal.signal(signal.SIGALRM, self._timeout_handler)
+                res_hash = self.sync_resource(m_name, filename)
 
-        if 'bg_id' in command:
-            self._bg_cmds[command['bg_id']] = command
-        if command["type"] in ["wait", "intr",
"kill"]:
-            bg_cmd = self._bg_cmds[command["proc_id"]]
-            if bg_cmd["netns"] != None:
-                command["netns"] = bg_cmd["netns"]
-
-        netns = command["netns"]
-        if command["type"] == "wait":
-            logging.debug("Get remaining time of bg process with bg_id == %s"
-                              % command["proc_id"])
-            remaining_time = self._rpc_call_x(netns, "get_remaining_time",
-                                              command["proc_id"])
-            logging.debug("Setting timeout to %d", remaining_time)
-            if remaining_time > 0:
-                signal.alarm(remaining_time)
-            else:
-                # 2 seconds is enough time to do wait via RPC and collect
-                # the result
-                signal.alarm(2)
-        else:
-            if "timeout" in command:
-                timeout = command["timeout"]
-                logging.debug("Setting timeout to \"%d\"", timeout)
-                signal.alarm(timeout)
-            else:
-                logging.debug("Setting default timeout (%ds)." %
DEFAULT_TIMEOUT)
-                signal.alarm(DEFAULT_TIMEOUT)
+                self.rpc_call("load_cached_module", m_name, res_hash)
+
+        logging.info("Host %s executing job %d: %s" %
+                     (self._id, job.id, str(job)))
+        if job._desc is not None:
+            logging.info("Job description: %s" % job._desc)
+
+        return self.rpc_call("run_job", job._to_dict(), netns=job._netns)
+
+    def wait_for_job(self, job, timeout):
+        res = True
+        if job.id not in self._jobs:
+            raise MachineError("No job '%s' running on Machine %s" %
+                               (job.id, self._id))
+
+        prev_handler = signal.signal(signal.SIGALRM, self._timeout_handler)
+        signal.alarm(timeout)
 
         try:
-            cmd_res = self._rpc_call_x(netns, "run_command", command)
+            if timeout > 0:
+                logging.info("Waiting for Job %d on Host %s for %d seconds." %
+                             (job.id, self._id, timeout))
+            elif timeout == 0:
+                logging.info("Waiting for Job %d on Host %s." %
+                             (job.id, self._id))
+            result = self._msg_dispatcher.wait_for_finish(self, job.id)
         except MachineError as exc:
-            if "proc_id" in command:
-                cmd_res = self._rpc_call_x(netns, "kill_command",
-                                           command["proc_id"])
-            else:
-                cmd_res = self._rpc_call_x(netns, "kill_command",
-                                           None)
+            logging.error(str(exc))
+            res = False
 
-            if "killed" in cmd_res and cmd_res["killed"]:
-                cmd_res["passed"] = False
-                cmd_res["msg"] = str(exc)
+        signal.alarm(0)
+        signal.signal(signal.SIGALRM, prev_handler)
+
+        return res
+
+    def wait_for_tmp_devices(self, timeout):
+        res = False
+        prev_handler = signal.signal(signal.SIGALRM, self._timeout_handler)
+        signal.alarm(timeout)
+
+        try:
+            if timeout > 0:
+                logging.info("Waiting for Device creation Host %s for %d
seconds." %
+                             (self._id, timeout))
+            elif timeout == 0:
+                logging.info("Waiting for Device creation on Host %s." %
+                             (self._id))
+
+            while len(self._tmp_device_database) > 0:
+                result = self._msg_dispatcher.handle_messages()
+        except MachineError as exc:
+            logging.error(str(exc))
+            res = False
 
         signal.alarm(0)
         signal.signal(signal.SIGALRM, prev_handler)
+        return res
+
+    def job_finished(self, msg):
+        job_id = msg["job_id"]
+        job = self._jobs[job_id]
+        job._res = msg["result"]
 
-        return cmd_res
+    def kill(self, job, signal):
+        if job.id not in self._jobs:
+            raise MachineError("No job '%s' running on Machine %s" %
+                               (job.id(), self._id))
+        return self.rpc_call("kill_job", job.id, signal, netns=job.netns)
 
     def get_hostname(self):
         """ Get hostname/ip of the machine
@@ -415,13 +441,6 @@ class Machine(object):
     def get_libvirt_domain(self):
         return self._libvirt_domain
 
-    def get_id(self):
-        """ Returns machine's id as defined in the recipe
"""
-        return self._id
-
-    def set_rpc(self, dispatcher):
-        self._msg_dispatcher = dispatcher
-
     def get_mac_pool(self):
         if self._mac_pool:
             return self._mac_pool
@@ -432,9 +451,9 @@ class Machine(object):
         self._mac_pool = mac_pool
 
     def restore_system_config(self):
-        self._rpc_call("restore_system_config")
+        self.rpc_call("restore_system_config")
         for netns in self._namespaces:
-            self._rpc_call_to_netns(netns, "restore_system_config")
+            self.rpc_call("restore_system_config", netns=netns)
         return True
 
     def set_network_bridges(self, bridges):
@@ -459,7 +478,7 @@ class Machine(object):
 
         tmp = {}
         for netns in namespaces:
-            tmp.update(self._rpc_call_x(netns, "start_packet_capture",
""))
+            tmp.update(self.rpc_call("start_packet_capture", "",
netns=netns))
         return tmp
 
     def stop_packet_capture(self):
@@ -468,10 +487,10 @@ class Machine(object):
             namespaces.add(iface.get_netns())
 
         for netns in namespaces:
-            self._rpc_call_x(netns, "stop_packet_capture")
+            self.rpc_call("stop_packet_capture", netns=netns)
 
     def copy_file_to_machine(self, local_path, remote_path=None, netns=None):
-        remote_path = self._rpc_call_x(netns, "start_copy_to", remote_path)
+        remote_path = self.rpc_call("start_copy_to", remote_path, netns=netns)
 
         f = open(local_path, "rb")
 
@@ -480,14 +499,14 @@ class Machine(object):
             if len(data) == 0:
                 break
 
-            self._rpc_call_x(netns, "copy_part_to", remote_path, Binary(data))
+            self.rpc_call("copy_part_to", remote_path, data, netns=netns)
 
-        self._rpc_call_x(netns, "finish_copy_to", remote_path)
+        self.rpc_call("finish_copy_to", remote_path, netns=netns)
 
         return remote_path
 
     def copy_file_from_machine(self, remote_path, local_path):
-        status = self._rpc_call("start_copy_from", remote_path)
+        status = self.rpc_call("start_copy_from", remote_path)
         if not status:
             raise MachineError("The requested file cannot be transfered." \
                        "It does not exist on machine %s" % self.get_id())
@@ -495,836 +514,54 @@ class Machine(object):
         local_file = open(local_path, "wb")
 
         buf_size = 1024*1024 # 1MB buffer
-        binary = "next"
-        while binary != "":
-            binary = self._rpc_call("copy_part_from", remote_path, buf_size)
-            local_file.write(binary.data)
+        while True:
+            data = self.rpc_call("copy_part_from", remote_path, buf_size)
+            if data == "":
+                break
+            local_file.write(data)
 
         local_file.close()
-        self._rpc_call("finish_copy_from", remote_path)
-
-    def sync_resources(self, required):
-        self._rpc_call("clear_resource_table")
-
-        for res_type, resources in required.iteritems():
-            for res_name, res in resources.iteritems():
-                has_resource = self._rpc_call("has_resource",
res["hash"])
-                if not has_resource:
-                    msg = "Transfering %s %s to machine %s" % \
-                            (res_name, res_type, self.get_id())
-                    logging.info(msg)
+        self.rpc_call("finish_copy_from", remote_path)
 
-                    local_path = required[res_type][res_name]["path"]
+    def sync_resource(self, res_name, file_path):
+        digest = sha256sum(file_path)
 
-                    if res_type == "tools":
-                        archive = tempfile.NamedTemporaryFile(delete=False)
-                        archive_path = archive.name
-                        archive.close()
+        if not self.rpc_call("has_resource", digest):
+            msg = "Transfering %s to machine %s as '%s'" % (file_path,
+                                                            self.get_id(),
+                                                            res_name)
+            logging.debug(msg)
 
-                        create_tar_archive(local_path, archive_path, True)
-                        local_path = archive_path
+            remote_path = self.copy_file_to_machine(file_path)
+            self.rpc_call("add_resource_to_cache",
+                           "file", remote_path, res_name)
+        return digest
 
-                    remote_path = self.copy_file_to_machine(local_path)
-                    self._rpc_call("add_resource_to_cache",
res["hash"],
-                                  remote_path, res_name, res["path"],
res_type)
+    # def enable_nm(self):
+        # return self._rpc_call("enable_nm")
 
-                    for ns in self._namespaces:
-                        remote_path = self.copy_file_to_machine(local_path,
-                                          netns=ns)
-                        self._rpc_call_to_netns(ns, "add_resource_to_cache",
-                            res["hash"], remote_path, res_name,
res["path"],
-                            res_type)
+    # def disable_nm(self):
+        # return self._rpc_call("disable_nm")
 
-                    if res_type == "tools":
-                        os.unlink(archive_path)
-
-                self._rpc_call("map_resource", res["hash"], res_type,
res_name)
-                for ns in self._namespaces:
-                    self._rpc_call_to_netns(ns, "map_resource",
res["hash"],
-                        res_type, res_name)
-
-    def enable_nm(self):
-        return self._rpc_call("enable_nm")
-
-    def disable_nm(self):
-        return self._rpc_call("disable_nm")
-
-    def restore_nm_option(self):
-        return self._rpc_call("restore_nm_option")
+    # def restore_nm_option(self):
+        # return self._rpc_call("restore_nm_option")
 
     def __str__(self):
         return "[Machine hostname(%s) libvirt_domain(%s) interfaces(%d)]" % \
                (self._hostname, self._libvirt_domain, len(self._interfaces))
 
-    def add_netns(self, netns):
-        self._namespaces.append(netns)
-        return self._rpc_call("add_namespace", netns)
+    # def add_netns(self, netns):
+        # self._namespaces.append(netns)
+        # return self._rpc_call("add_namespace", netns)
 
-    def del_netns(self, netns):
-        return self._rpc_call("del_namespace", netns)
-
-    def del_namespaces(self):
-        for netns in self._namespaces:
-            self.del_netns(netns)
-        self._namespaces = []
-        return True
+    # def del_netns(self, netns):
+        # return self._rpc_call("del_namespace", netns)
 
-    def wait_interface_init(self):
-        return self._rpc_call("wait_interface_init")
+    # def del_namespaces(self):
+        # for netns in self._namespaces:
+            # self.del_netns(netns)
+        # self._namespaces = []
+        # return True
 
     def get_security(self):
         return self._security
-
-class Interface(object):
-    """ Abstraction of a test network interface on a slave machine
-
-        This is a base class for object that represent test interfaces
-        on a test machine.
-    """
-    def __init__(self, machine, if_id, if_type):
-        self._machine = machine
-        self._configured = False
-
-        self._id = if_id
-        self._type = if_type
-
-        self._hwaddr = None
-        self._devname = None
-        self._network = None
-        self._netem = None
-
-        self._slaves = {}
-        self._slave_options = {}
-        self._addresses = []
-        self._options = []
-
-        self._master = {"primary": None, "other": []}
-
-        self._ovs_conf = None
-
-        self._netns = None
-        self._peer = None
-        self._mtu = None
-        self._driver = None
-        self._devlink = None
-
-    def get_id(self):
-        return self._id
-
-    def get_type(self):
-        return self._type
-
-    def get_driver(self):
-        return self._driver
-
-    def set_hwaddr(self, hwaddr):
-        self._hwaddr = normalize_hwaddr(hwaddr)
-
-    def get_hwaddr(self):
-        if not self._hwaddr:
-            msg = "Hardware address is not available for interface
'%s'" \
-                  % self.get_id()
-            raise MachineError(msg)
-        return self._hwaddr
-
-    def set_devname(self, devname):
-        self._devname = devname
-
-    def get_devname(self):
-        if not self._devname:
-            msg = "Device name is not available for interface '%s'" \
-                  % self.get_id()
-            raise MachineError(msg)
-        return self._devname
-
-    def set_network(self, network):
-        self._network = network
-
-    def get_network(self):
-        if not self._network:
-            msg = "Network segment is not available for interface '%s'"
\
-                  % self.get_id()
-            raise MachineError(msg)
-        return self._network
-
-    def set_option(self, name, value):
-        self._options.append((name, value))
-
-    def set_netem(self, netem):
-        self._netem = netem
-
-    def add_master(self, master, primary=True):
-        if primary and self._master["primary"] != None:
-            msg = "Interface %s already has a primary master."\
-                    % self.get_id()
-            raise MachineError(msg)
-        else:
-            if primary:
-                self._master["primary"] = master
-            else:
-                self._master["other"].append(master)
-
-    def del_master(self, master):
-        if self._master["primary"] is master:
-            self._master["primary"] = None
-        else:
-            self._master["other"].remove(master)
-
-    def get_primary_master(self):
-        return self._master["primary"]
-
-    def add_slave(self, iface):
-        self._slaves[iface.get_id()] = iface
-        if self._type in ["vlan", "vxlan"]:
-            iface.add_master(self, primary=False)
-        else:
-            iface.add_master(self)
-
-    def del_slave(self, iface):
-        iface.del_master(self)
-        del self._slaves[iface.get_id()]
-
-    def set_slave_option(self, slave_id, name, value):
-        if slave_id not in self._slave_options:
-            self._slave_options[slave_id] = []
-        self._slave_options[slave_id].append((name, value))
-
-    def add_address(self, addr):
-        if (type(addr) == type([])):
-            for one_addr in addr:
-                self._addresses.append(one_addr)
-        else:
-            self._addresses.append(addr)
-
-    def get_address(self, num):
-        return self._addresses[num].split('/')[0]
-
-    def get_addresses(self):
-        addrs = []
-        for addr in self._addresses:
-            addrs.append(tuple(addr.split('/')))
-        return addrs
-
-    def set_ovs_conf(self, ovs_conf):
-        self._ovs_conf = ovs_conf
-
-    def get_ovs_conf(self):
-        return self._ovs_conf
-
-    def set_netns(self, netns):
-        self._netns = netns
-
-    def get_netns(self):
-        return self._netns
-
-    def get_host(self):
-        return self._machine.get_id()
-
-    def set_peer(self, peer):
-        self._peer = peer
-
-    def get_peer(self):
-        return self._peer
-
-    def get_prefix(self, num):
-        try:
-            return self._addresses[num].split('/')[1]
-        except IndexError:
-            raise PrefixMissingError
-
-    def get_mtu(self):
-        return self._mtu
-
-    def set_mtu(self, mtu):
-        command = {"type": "config",
-                   "host": self._machine.get_id(),
-                   "persistent": False,
-                   "options":[
-                       {"name": "/sys/class/net/%s/mtu" %
self._devname,
-                        "value": str(mtu)}
-                    ]}
-        command["netns"] = self._netns
-
-        self._machine.run_command(command)
-        self._mtu = mtu
-        return self._mtu
-
-    def link_stats(self):
-        stats = self._machine._rpc_call_x(self._netns, "link_stats",
-                                          self._id)
-        return stats
-
-    def set_addresses(self, ips):
-        self._addresses = ips
-        self._machine._rpc_call_x(self._netns, "set_addresses",
-                                  self._id, ips)
-
-    def add_route(self, dest):
-        self._machine._rpc_call_x(self._netns, "add_route",
-                                  self._id, dest)
-
-    def del_route(self, dest):
-        self._machine._rpc_call_x(self._netns, "del_route",
-                                  self._id, dest)
-
-    def update_from_slave(self):
-        if_data = self._machine._rpc_call_x(self._netns, "get_if_data",
-                                            self._id)
-
-        if if_data is not None:
-            self.update(if_data)
-        return
-
-    def update(self, if_data):
-        self.set_hwaddr(if_data["hwaddr"])
-        self.set_devname(if_data["name"])
-        self._mtu = if_data["mtu"]
-        self._driver = if_data["driver"]
-        self._devlink = if_data["devlink"]
-
-    def get_config(self):
-        config = {"id": self._id,
-                  "hwaddr": self._hwaddr,
-                  "devname": self._devname,
-                  "network_label": self._network,
-                  "type": self._type,
-                  "addresses": self._addresses,
-                  "slaves": self._slaves.keys(),
-                  "options": self._options,
-                  "slave_options": self._slave_options,
-                  "master": None,
-                  "other_masters": [],
-                  "ovs_conf": self._ovs_conf,
-                  "netns": self._netns,
-                  "peer": self._peer,
-                  "netem": self._netem,
-                  "mtu": self._mtu,
-                  "driver": self._driver}
-
-        if self._master["primary"] != None:
-            config["master"] = self._master["primary"].get_id()
-
-        for m in self._master["other"]:
-            config["other_masters"].append(m.get_id())
-
-        return config
-
-    def up(self):
-        self._machine._rpc_call_x(self._netns, "set_device_up", self._id)
-
-    def down(self):
-        self._machine._rpc_call_x(self._netns, "set_device_down", self._id)
-
-    def set_link_up(self):
-        self._machine._rpc_call_x(self._netns, "set_link_up", self._id)
-
-    def set_link_down(self):
-        self._machine._rpc_call_x(self._netns, "set_link_down", self._id)
-
-    def initialize(self):
-        phys_devs = self._machine._rpc_call("map_if_by_hwaddr",
-                                            self._id, self._hwaddr)
-
-        if len(phys_devs) == 1:
-            self.set_devname(phys_devs[0]["name"])
-        elif len(phys_devs) < 1:
-            msg = "Device %s not found on machine %s" \
-                  % (self.get_id(), self._machine.get_id())
-            raise MachineError(msg)
-        elif len(phys_devs) > 1:
-            msg = "More than one device with hwaddr %s found on machine %s" \
-                  % (self._hwaddr, self._machine.get_id())
-            raise MachineError(msg)
-
-        self.down()
-
-    def cleanup(self):
-        self._machine._rpc_call("unmap_if", self._id)
-
-    def configure(self):
-        if self._configured:
-            msg = "Unable to configure interface %s on machine %s. " \
-                  "It has been configured already." % (self.get_id(),
-                  self._machine.get_id())
-            raise MachineError(msg)
-        else:
-            self._configured = True
-
-        logging.info("Configuring interface %s on machine %s", self.get_id(),
-                     self._machine.get_id())
-
-        if self._netns != None:
-            self._machine._rpc_call("set_if_netns", self.get_id(),
self._netns)
-        self._machine._rpc_call_x(self._netns, "configure_interface",
-                                  self.get_id(), self.get_config())
-
-        self.update_from_slave()
-
-    def deconfigure(self):
-        if not self._configured:
-            return
-
-        self._machine._rpc_call_x(self._netns, "deconfigure_interface",
-                                  self.get_id())
-        if self._netns != None:
-            self._machine._rpc_call_to_netns(self._netns,
-                                         "return_if_netns", self.get_id())
-        self._configured = False
-
-    def add_br_vlan(self, br_vlan_info):
-        self._machine._rpc_call_x(self._netns, "add_br_vlan",
-                                  self._id, br_vlan_info)
-
-    def del_br_vlan(self, br_vlan_info):
-        self._machine._rpc_call_x(self._netns, "del_br_vlan",
-                                  self._id, br_vlan_info)
-
-    def get_br_vlans(self):
-        return self._machine._rpc_call_x(self._netns, "get_br_vlans",
self._id)
-
-    def add_br_fdb(self, br_fdb_info):
-        self._machine._rpc_call_x(self._netns, "add_br_fdb",
-                                  self._id, br_fdb_info)
-
-    def del_br_fdb(self, br_fdb_info):
-        self._machine._rpc_call_x(self._netns, "del_br_fdb",
-                                  self._id, br_fdb_info)
-
-    def get_br_fdbs(self):
-        return self._machine._rpc_call_x(self._netns, "get_br_fdbs", self._id)
-
-    def set_br_learning(self, br_learning_info):
-        self._machine._rpc_call_x(self._netns, "set_br_learning", self._id,
-                                  br_learning_info)
-
-    def set_br_learning_sync(self, br_learning_sync_info):
-        self._machine._rpc_call_x(self._netns, "set_br_learning_sync",
self._id,
-                                  br_learning_sync_info)
-
-    def set_br_flooding(self, br_flooding_info):
-        self._machine._rpc_call_x(self._netns, "set_br_flooding", self._id,
-                                  br_flooding_info)
-
-    def set_br_state(self, br_state_info):
-        self._machine._rpc_call_x(self._netns, "set_br_state", self._id,
-                                  br_state_info)
-
-    def set_speed(self, speed):
-        self._machine._rpc_call_x(self._netns, "set_speed", self._id, speed)
-
-    def set_autoneg(self):
-        self._machine._rpc_call_x(self._netns, "set_autoneg", self._id)
-
-    def slave_add(self, if_id):
-        self._machine._rpc_call_x(self._netns, "slave_add", self._id, if_id)
-        self.add_slave(self._machine.get_interface(if_id))
-
-    def slave_del(self, if_id):
-        self.del_slave(self._machine.get_interface(if_id))
-        self._machine._rpc_call_x(self._netns, "slave_del", self._id, if_id)
-
-    def get_devlink_name(self):
-        if self._devlink:
-            return "%s/%s" % (self._devlink["bus_name"],
-                              self._devlink["dev_name"])
-        return None
-
-    def get_devlink_port_name(self):
-        if self._devlink:
-            return "%s/%u" % (self.get_devlink_name(),
-                              self._devlink["port_index"])
-        return None
-
-class StaticInterface(Interface):
-    """ Static interface
-
-        This class represents interfaces that are present on the
-        machine. LNST will only use them for testing without performing
-        any special actions.
-
-        This type is suitable for physical interfaces.
-    """
-    def __init__(self, machine, if_id, if_type):
-        super(StaticInterface, self).__init__(machine, if_id, if_type)
-
-class LoopbackInterface(Interface):
-    """ Static interface
-
-        This class represents interfaces that are present on the
-        machine. LNST will only use them for testing without performing
-        any special actions.
-
-        This type is suitable for physical interfaces.
-    """
-    def __init__(self, machine, if_id, if_type):
-        super(LoopbackInterface, self).__init__(machine, if_id, if_type)
-
-    def initialize(self):
-        pass
-
-    def cleanup(self):
-        pass
-
-    def configure(self):
-        self._hwaddr = '00:00:00:00:00:00'
-        self._driver = 'loopback'
-
-        phys_devs = self._machine._rpc_call_x(self._netns,
-                                              "map_if_by_params", self._id,
-                                              { 'hwaddr': self._hwaddr,
-                                                'driver': self._driver })
-
-        if len(phys_devs) == 1:
-            self.set_devname(phys_devs[0]["name"])
-        elif len(phys_devs) < 1:
-            msg = "Device %s not found on machine %s" \
-                  % (self.get_id(), self._machine.get_id())
-            raise MachineError(msg)
-        elif len(phys_devs) > 1:
-            msg = "More than one device with hwaddr %s found on machine %s" \
-                  % (self._hwaddr, self._machine.get_id())
-            raise MachineError(msg)
-
-        if self._configured:
-            msg = "Unable to configure interface %s on machine %s. " \
-                  "It has been configured already." % (self.get_id(),
-                  self._machine.get_id())
-            raise MachineError(msg)
-
-        logging.info("Configuring interface %s on machine %s", self.get_id(),
-                     self._machine.get_id())
-
-        self._machine._rpc_call_x(self._netns, "configure_interface",
-                                  self.get_id(), self.get_config())
-        self._configured = True
-        self.update_from_slave()
-
-    def deconfigure(self):
-        if not self._configured:
-            return
-
-        self._machine._rpc_call_x(self._netns, "deconfigure_interface",
-                                  self.get_id())
-        self._machine._rpc_call_x(self._netns, "unmap_if", self.get_id())
-        self._configured = False
-
-class VirtualInterface(Interface):
-    """ Dynamically created interface
-
-        This class represents interfaces in libvirt virtual machines
-        that were created dynamically by LNST just for this test.
-
-        This requires some special handling and communication with
-        libvirt.
-    """
-    def __init__(self, machine, if_id, if_type):
-        super(VirtualInterface, self).__init__(machine, if_id, if_type)
-        self._driver = "virtio"
-
-    def set_driver(self, driver):
-        self._driver = driver
-
-    def get_driver(self):
-        return self._driver
-
-    def get_orig_hwaddr(self):
-        if not self._orig_hwaddr:
-            msg = "Hardware address is not available for interface
'%s'" \
-                  % self.get_id()
-            raise MachineError(msg)
-        return self._orig_hwaddr
-
-    def initialize(self):
-        domain_ctl = self._machine.get_domain_ctl()
-
-        if self._hwaddr:
-            query = self._machine._rpc_call('get_devices_by_hwaddr',
-                                           self._hwaddr)
-            if len(query):
-                msg = "Device with hwaddr %s already exists" % self._hwaddr
-                raise MachineError(msg)
-        else:
-            mac_pool = self._machine.get_mac_pool()
-            while True:
-                self._hwaddr = normalize_hwaddr(mac_pool.get_addr())
-                query = self._machine._rpc_call('get_devices_by_hwaddr',
-                                               self._hwaddr)
-                if not len(query):
-                    break
-
-        bridges = self._machine.get_network_bridges()
-        if self._network in bridges:
-            net_ctl = bridges[self._network]
-        else:
-            bridges[self._network] = net_ctl = VirtNetCtl()
-            net_ctl.init()
-
-        net_name = net_ctl.get_name()
-
-        logging.info("Creating interface %s (%s) on machine %s",
-                     self.get_id(), self._hwaddr, self._machine.get_id())
-
-        self._orig_hwaddr = self._hwaddr
-        domain_ctl.attach_interface(self._hwaddr, net_name, self._driver)
-
-
-        # The sleep here is necessary, because udev sometimes renames the
-        # newly created device and if the query for name comes too early,
-        # the controller will then try to configure an nonexistent device
-        sleep(1)
-
-        ready = wait_for(self.is_ready, timeout=10)
-
-        if not ready:
-            msg = "Netdevice initialization failed." \
-                  "Unable to create device %s (%s) on machine %s" \
-                  % (self.get_id(), self._hwaddr, self._machine.get_id())
-            raise MachineError(msg)
-
-        super(VirtualInterface, self).initialize()
-
-    def cleanup(self):
-        self._machine._rpc_call("unmap_if", self._id)
-        domain_ctl = self._machine.get_domain_ctl()
-        domain_ctl.detach_interface(self._orig_hwaddr)
-
-    def is_ready(self):
-        ifaces = self._machine._rpc_call('get_devices_by_hwaddr', self._hwaddr)
-        return len(ifaces) > 0
-
-class SoftInterface(Interface):
-    """ Software interface abstraction
-
-        This type of interface represents interfaces created in the kernel
-        during the runtime. This includes devices such as bonds and teams.
-    """
-
-    def __init__(self, machine, if_id, if_type):
-        super(SoftInterface, self).__init__(machine, if_id, if_type)
-
-    def initialize(self):
-        pass
-
-    def cleanup(self):
-        pass
-
-    def configure(self):
-        if self._configured:
-            return
-        else:
-            self._configured = True
-
-        logging.info("Configuring interface %s on machine %s", self.get_id(),
-                     self._machine.get_id())
-
-        if self._type == "veth":
-            peer_if = self._machine.get_interface(self._peer)
-            peer_config = peer_if.get_config()
-            dev_name, peer_name = self._machine._rpc_call("create_if_pair",
-                                                self._id, self.get_config(),
-                                                self._peer, peer_config)
-            self.set_devname(dev_name)
-            peer_if.set_devname(peer_name)
-            self._configured = True
-            peer_if._configured = True
-            return
-
-        dev_name = self._machine._rpc_call_x(self._netns,
-                                             "create_soft_interface",
-                                             self._id, self.get_config())
-        self.set_devname(dev_name)
-        self.update_from_slave()
-
-    def deconfigure(self):
-        if not self._configured:
-            return
-
-        if self._type == "veth":
-            peer_if = self._machine.get_interface(self._peer)
-
-            self._machine._rpc_call("deconfigure_if_pair", self._id,
self._peer)
-            self._machine._rpc_call("unmap_if", self._id)
-            self._machine._rpc_call("unmap_if", self._peer)
-
-            self._configured = False
-            peer_if._configured = False
-            return
-
-        self._machine._rpc_call_x(self._netns, "deconfigure_interface",
-                                  self.get_id())
-        self._machine._rpc_call_x(self._netns, "unmap_if", self.get_id())
-        self._configured = False
-
-class UnusedInterface(Interface):
-    """ Unused interface for this test
-
-        This class represents interfaces that will not be used in the
-        current test setup. This applies when a slave machine from a
-        pool has more interfaces then the machine it was matched to
-        from the recipe.
-
-        LNST still needs to know about these interfaces so it can turn
-        them off.
-    """
-
-    def __init__(self, machine, if_id, if_type):
-        super(UnusedInterface, self).__init__(machine, if_id, if_type)
-
-    def initialize(self):
-        self._machine._rpc_call('set_unmapped_device_down', self._hwaddr)
-
-    def set_driver(self, driver):
-        pass
-
-    def configure(self):
-        pass
-
-    def deconfigure(self):
-        pass
-
-    def up(self):
-        pass
-
-    def down(self):
-        pass
-
-    def cleanup(self):
-        pass
-
-class Device(object):
-    """ Represents device information received from a
Slave"""
-
-    def pre_call_decorate(func):
-        @wraps(func)
-        def func_wrapper(inst, *args, **kwargs):
-            inst.slave_update()
-            return func(inst, *args, **kwargs)
-        return func_wrapper
-
-    def __init__(self, data, machine):
-        self._if_index = data["if_index"]
-        self._hwaddr = None
-        self._name = None
-        self._ip_addrs = None
-        self._ifi_type = None
-        self._state = None
-        self._master = None
-        self._slaves = None
-        self._netns = None
-        self._peer = None
-        self._mtu = None
-        self._driver = None
-        self._devlink = None
-
-        self._machine = machine
-
-        self.update_data(data)
-
-    def update_data(self, data):
-        if data["if_index"] != self._if_index:
-            return False
-
-        self._hwaddr = data["hwaddr"]
-        self._name = data["name"]
-        self._ip_addrs = data["ip_addrs"]
-        self._ifi_type = data["ifi_type"]
-        self._state = data["state"]
-        self._master = data["master"]
-        self._slaves = data["slaves"]
-        self._netns = data["netns"]
-        self._peer = data["peer"]
-        self._mtu = data["mtu"]
-        self._driver = data["driver"]
-        self._devlink = data["driver"]
-        return True
-
-    def slave_update(self):
-        res = self._machine._rpc_call_x(self._netns,
-                                        "get_device",
-                                        self._if_index)
-        if res:
-            self.update_data(res)
-        return
-
-    def get_if_index(self):
-        return self._if_index
-
-    @pre_call_decorate
-    def get_hwaddr(self):
-        return self._hwaddr
-
-    @pre_call_decorate
-    def get_name(self):
-        return self._name
-
-    @pre_call_decorate
-    def get_ip_addrs(self, selector={}):
-        return [ip["addr"]
-                for ip in self._ip_addrs
-                    if selector.items() <= ip.items()]
-
-    @pre_call_decorate
-    def get_ip_addr(self, num, selector={}):
-        ips = self.get_ip_addrs(selector)
-        return ips[num]
-
-    @pre_call_decorate
-    def get_ifi_type(self):
-        return self._ifi_type
-
-    @pre_call_decorate
-    def get_state(self):
-        return self._state
-
-    @pre_call_decorate
-    def get_master(self):
-        return self._master
-
-    @pre_call_decorate
-    def get_slaves(self):
-        return self._slaves
-
-    @pre_call_decorate
-    def get_netns(self):
-        return self._netns
-
-    @pre_call_decorate
-    def get_peer(self):
-        return self._peer
-
-    @pre_call_decorate
-    def get_mtu(self):
-        return self._mtu
-
-    def set_mtu(self, mtu):
-        command = {"type": "config",
-                   "host": self._machine.get_id(),
-                   "persistent": False,
-                   "options":[
-                       {"name": "/sys/class/net/%s/mtu" %
self._name,
-                        "value": str(mtu)}
-                    ]}
-        command["netns"] = self._netns
-
-        self._machine.run_command(command)
-
-        self.slave_update()
-        return self._mtu
-
-    @pre_call_decorate
-    def get_driver(self):
-        return self._driver
-
-    @pre_call_decorate
-    def get_devlink_name(self):
-        if self._devlink:
-            return "%s/%s" % (self._devlink["bus_name"],
-                              self._devlink["dev_name"])
-        return None
-
-    @pre_call_decorate
-    def get_devlink_port_name(self):
-        if self._devlink:
-            return "%s/%u" % (self.get_devlink_name(),
-                              self._devlink["port_index"])
-        return None
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index b18f47d..0f654e3 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -19,8 +19,10 @@ import datetime
 import socket
 import ctypes
 import multiprocessing
+import imp
+import types
 from time import sleep, time
-from xmlrpclib import Binary
+from inspect import isclass
 from tempfile import NamedTemporaryFile
 from lnst.Common.Logs import log_exc_traceback
 from lnst.Common.PacketCapture import PacketCapture
@@ -34,118 +36,150 @@ from lnst.Common.Utils import check_process_running
 from lnst.Common.Utils import is_installed
 from lnst.Common.ConnectionHandler import send_data
 from lnst.Common.ConnectionHandler import ConnectionHandler
-from lnst.Common.Config import lnst_config
 from lnst.Common.Config import DefaultRPCPort
+from lnst.Common.DeviceRef import DeviceRef
+from lnst.Common.LnstError import LnstError
+from lnst.Common.DeviceError import DeviceDeleted
+from lnst.Common.IpAddress import IpAddress
+from lnst.Slave.Job import Job, JobContext
 from lnst.Slave.InterfaceManager import InterfaceManager
 from lnst.Slave.BridgeTool import BridgeTool
 from lnst.Slave.SlaveSecSocket import SlaveSecSocket, SecSocketException
 
+Devices = types.ModuleType("Devices")
+Devices.__path__ = ["lnst.Devices"]
+
+sys.modules["lnst.Devices"] = Devices
+
 class SlaveMethods:
     '''
     Exported xmlrpc methods
     '''
-    def __init__(self, command_context, log_ctl, if_manager, net_namespaces,
-                 server_handler, slave_server):
+    def __init__(self, job_context, log_ctl, net_namespaces,
+                 server_handler, slave_config, slave_server):
         self._packet_captures = {}
-        self._if_manager = if_manager
-        self._command_context = command_context
+        self._if_manager = None
+        self._job_context = job_context
         self._log_ctl = log_ctl
         self._net_namespaces = net_namespaces
         self._server_handler = server_handler
         self._slave_server = slave_server
+        self._slave_config = slave_config
 
         self._capture_files = {}
         self._copy_targets = {}
         self._copy_sources = {}
         self._system_config = {}
 
-        self._cache = ResourceCache(lnst_config.get_option("cache",
"dir"),
-                        lnst_config.get_option("cache",
"expiration_period"))
+        self._cache = ResourceCache(slave_config.get_option("cache",
"dir"),
+                        slave_config.get_option("cache",
"expiration_period"))
+
+        self._dynamic_modules = {}
+        self._dynamic_classes = {}
+
+        self._bkp_nm_opt_val = slave_config.get_option("environment",
"use_nm")
+
+    def hello(self):
+        logging.info("Recieved a controller connection.")
+
+        slave_desc = {}
+        if check_process_running("NetworkManager"):
+            slave_desc["nm_running"] = True
+        else:
+            slave_desc["nm_running"] = False
 
-        self._resource_table = {'module': {}, 'tools': {}}
+        k_release, _ = exec_cmd("uname -r", False, False, False)
+        r_release, _ = exec_cmd("cat /etc/redhat-release", False, False,
False)
+        slave_desc["kernel_release"] = k_release.strip()
+        slave_desc["redhat_release"] = r_release.strip()
+        slave_desc["lnst_version"] = self._slave_config.version
 
-        self._bkp_nm_opt_val = lnst_config.get_option("environment",
"use_nm")
+        return ("hello", slave_desc)
 
-    def hello(self, recipe_path):
+    def set_recipe(self, recipe_name):
         self.machine_cleanup()
         self.restore_nm_option()
 
-        logging.info("Recieved a controller connection.")
-        self.clear_resource_table()
         self._cache.del_old_entries()
         self.reset_file_transfers()
 
-        self._if_manager.rescan_devices()
-
         date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S")
-        self._log_ctl.set_recipe(recipe_path, expand=date)
+        self._log_ctl.set_recipe(recipe_name, expand=date)
         sleep(1)
 
-        slave_desc = {}
         if check_process_running("NetworkManager"):
             logging.warning("=============================================")
             logging.warning("NetworkManager is running on a slave machine!")
-            if lnst_config.get_option("environment", "use_nm"):
+            if self._slave_config.get_option("environment",
"use_nm"):
                 logging.warning("Support of NM is still experimental!")
             else:
                 logging.warning("Usage of NM is disabled!")
             logging.warning("=============================================")
-            slave_desc["nm_running"] = True
-        else:
-            slave_desc["nm_running"] = False
 
-        k_release, _ = exec_cmd("uname -r", False, False, False)
-        r_release, _ = exec_cmd("cat /etc/redhat-release", False, False,
False)
-        slave_desc["kernel_release"] = k_release.strip()
-        slave_desc["redhat_release"] = r_release.strip()
-        slave_desc["lnst_version"] = lnst_config.version
-
-        return ("hello", slave_desc)
+        return True
 
     def bye(self):
         self.restore_system_config()
-        self.clear_resource_table()
         self._cache.del_old_entries()
         self.reset_file_transfers()
         self._remove_capture_files()
         return "bye"
 
-    def kill_cmds(self):
-        logging.info("Killing all forked processes.")
-        self._command_context.cleanup()
-        return "Commands killed"
+    def map_device_class(self, cls_name, module_name):
+        if cls_name in self._dynamic_classes:
+            return
 
-    def map_if_by_hwaddr(self, if_id, hwaddr):
-        devices = self.map_if_by_params(if_id, {'hwaddr' : hwaddr})
+        module = self._dynamic_modules[module_name]
+        cls = getattr(module, cls_name)
 
-        return devices
+        self._dynamic_classes[cls_name] = cls
 
-    def map_if_by_params(self, if_id, params):
-        devices = self.get_devices_by_params(params)
+        setattr(Devices, cls_name, cls)
 
-        if len(devices) == 1:
-            dev = self._if_manager.get_device_by_params(params)
-            self._if_manager.map_if(if_id, dev.get_if_index())
+    def load_cached_module(self, module_name, res_hash):
+        self._cache.renew_entry(res_hash)
+        if module_name in self._dynamic_modules:
+            return
+        module_path = self._cache.get_path(res_hash)
+        module = imp.load_source(module_name, module_path)
+        self._dynamic_modules[module_name] = module
 
-        return devices
+    def init_if_manager(self):
+        self._if_manager = InterfaceManager(self._server_handler)
+        for cls_name in dir(Devices):
+            cls = getattr(Devices, cls_name)
+            if isclass(cls):
+                self._if_manager.add_device_class(cls_name, cls)
 
-    def unmap_if(self, if_id):
-        self._if_manager.unmap_if(if_id)
+        self._if_manager.rescan_devices()
+        self._server_handler.set_if_manager(self._if_manager)
+        self._server_handler.add_connection('netlink',
+                                            self._if_manager.get_nl_socket())
         return True
 
+    def dev_method(self, if_index, name, args, kwargs):
+        dev = self._if_manager.get_device(if_index)
+        method = getattr(dev, name)
+
+        return method(*args, **kwargs)
+
+    def dev_attr(self, if_index, name):
+        dev = self._if_manager.get_device(if_index)
+        return getattr(dev, name)
+
     def get_devices(self):
         self._if_manager.rescan_devices()
         devices = self._if_manager.get_devices()
         result = {}
         for device in devices:
-            result[device._if_index] = device.get_if_data()
+            result[device.if_index] = device._get_if_data()
         return result
 
     def get_device(self, if_index):
         self._if_manager.rescan_devices()
         device = self._if_manager.get_device(if_index)
         if device:
-            return device.get_if_data()
+            return device._get_if_data()
         else:
             return None
 
@@ -156,7 +190,6 @@ class SlaveMethods:
         for entry in name_scan:
             if entry["name"] == devname:
                 netdevs.append(entry)
-
         return netdevs
 
     def get_devices_by_hwaddr(self, hwaddr):
@@ -167,14 +200,13 @@ class SlaveMethods:
                 entry = {"name": dev.get_name(),
                          "hwaddr": dev.get_hwaddr()}
                 matched.append(entry)
-
         return matched
 
     def get_devices_by_params(self, params):
         devices = self._if_manager.get_devices()
         matched = []
         for dev in devices:
-            dev_data = dev.get_if_data()
+            dev_data = dev._get_if_data()
             entry = {"name": dev.get_name(),
                      "hwaddr": dev.get_hwaddr()}
             for key, value in params.iteritems():
@@ -184,180 +216,121 @@ class SlaveMethods:
 
             if entry is not None:
                 matched.append(entry)
-
         return matched
 
-    def get_if_data(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is None:
-            return None
-        return dev.get_if_data()
-
-    def link_stats(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is None:
-            logging.error("Device with id '%s' not found." % if_id)
-            return None
-        return dev.link_stats()
-
-    def set_addresses(self, if_id, ips):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is None:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        dev.set_addresses(ips)
-        return True
-
-    def add_route(self, if_id, dest):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is None:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        dev.add_route(dest)
-        return True
-
-    def del_route(self, if_id, dest):
-        dev = self._if_manager.get_mapped_device(if_id)
+    def destroy_devices(self):
+        devices = self._if_manager.get_devices()
+        for dev in devices:
+            try:
+                dev.destroy()
+            except DeviceDeleted:
+                pass
+            self._if_manager.rescan_devices()
+
+    # def add_route(self, if_id, dest):
+        # dev = self._if_manager.get_mapped_device(if_id)
+        # if dev is None:
+            # logging.error("Device with id '%s' not found." % if_id)
+            # return False
+        # dev.add_route(dest)
+        # return True
+
+    # def del_route(self, if_id, dest):
+        # dev = self._if_manager.get_mapped_device(if_id)
+        # if dev is None:
+            # logging.error("Device with id '%s' not found." % if_id)
+            # return False
+        # dev.del_route(dest)
+        # return True
+
+    def create_device(self, clsname, args=[], kwargs={}):
+        dev =  self._if_manager.create_device(clsname, args, kwargs)
         if dev is None:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        dev.del_route(dest)
-        return True
-
-    def set_device_up(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        dev.up()
-        return True
-
-    def set_device_down(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.down()
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def set_link_up(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.link_up()
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def set_link_down(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.link_down()
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def set_unmapped_device_down(self, hwaddr):
-        dev = self._if_manager.get_device_by_hwaddr(hwaddr)
-        if dev is not None:
-            dev.down()
-        else:
-            logging.warning("Device with hwaddr '%s' not found." %
hwaddr)
-        return True
-
-    def configure_interface(self, if_id, config):
-        device = self._if_manager.get_mapped_device(if_id)
-        device.set_configuration(config)
-        device.configure()
-        return True
-
-    def create_soft_interface(self, if_id, config):
-        dev_name = self._if_manager.create_device_from_config(if_id, config)
-        dev = self._if_manager.get_mapped_device(if_id)
-        dev.configure()
-        return dev_name
-
-    def create_if_pair(self, if_id1, config1, if_id2, config2):
-        dev_names = self._if_manager.create_device_pair(if_id1, config1,
-                                                        if_id2, config2)
-        dev1 = self._if_manager.get_mapped_device(if_id1)
-        dev2 = self._if_manager.get_mapped_device(if_id2)
-
-        while dev1.get_if_index() == None and dev2.get_if_index() == None:
-            msgs = self._server_handler.get_messages_from_con('netlink')
-            for msg in msgs:
-                self._if_manager.handle_netlink_msgs(msg[1]["data"])
-
-        if config1["netns"] != None:
-            hwaddr = dev1.get_hwaddr()
-            self.set_if_netns(if_id1, config1["netns"])
-
-            msg = {"type": "command", "method_name":
"configure_interface",
-                   "args": [if_id1, config1]}
-            self._server_handler.send_data_to_netns(config1["netns"], msg)
-            result = self._slave_server.wait_for_result(config1["netns"])
-            if result["result"] != True:
-                raise Exception("Configuration failed.")
-        else:
-            dev1.configure()
-        if config2["netns"] != None:
-            hwaddr = dev2.get_hwaddr()
-            self.set_if_netns(if_id2, config2["netns"])
-
-            msg = {"type": "command", "method_name":
"configure_interface",
-                   "args": [if_id2, config2]}
-            self._server_handler.send_data_to_netns(config2["netns"], msg)
-            result = self._slave_server.wait_for_result(config2["netns"])
-            if result["result"] != True:
-                raise Exception("Configuration failed.")
-        else:
-            dev2.configure()
-        return dev_names
-
-    def deconfigure_if_pair(self, if_id1, if_id2):
-        dev1 = self._if_manager.get_mapped_device(if_id1)
-        dev2 = self._if_manager.get_mapped_device(if_id2)
-
-        if dev1.get_netns() == None:
-            dev1.deconfigure()
-        else:
-            netns = dev1.get_netns()
-
-            msg = {"type": "command", "method_name":
"deconfigure_interface",
-                   "args": [if_id1]}
-            self._server_handler.send_data_to_netns(netns, msg)
-            result = self._slave_server.wait_for_result(netns)
-            if result["result"] != True:
-                raise Exception("Deconfiguration failed.")
-
-            self.return_if_netns(if_id1)
-
-        if dev2.get_netns() == None:
-            dev2.deconfigure()
-        else:
-            netns = dev2.get_netns()
-
-            msg = {"type": "command", "method_name":
"deconfigure_interface",
-                   "args": [if_id2]}
-            self._server_handler.send_data_to_netns(netns, msg)
-            result = self._slave_server.wait_for_result(netns)
-            if result["result"] != True:
-                raise Exception("Deconfiguration failed.")
-
-            self.return_if_netns(if_id2)
-
-        dev1.destroy()
-        dev2.destroy()
-        dev1.del_configuration()
-        dev2.del_configuration()
-        return True
-
-    def deconfigure_interface(self, if_id):
-        device = self._if_manager.get_mapped_device(if_id)
-        if device is not None:
-            device.clear_configuration()
-        else:
-            logging.error("No device with id '%s' to deconfigure." %
if_id)
-        return True
+            raise Exception("Device creation failed")
+        return {"if_index": dev.if_index, "name": dev.name}
+
+    # def create_if_pair(self, if_id1, config1, if_id2, config2):
+        # dev_names = self._if_manager.create_device_pair(if_id1, config1,
+                                                        # if_id2, config2)
+        # dev1 = self._if_manager.get_mapped_device(if_id1)
+        # dev2 = self._if_manager.get_mapped_device(if_id2)
+
+        # while dev1.get_if_index() == None and dev2.get_if_index() == None:
+            # msgs = self._server_handler.get_messages_from_con('netlink')
+            # for msg in msgs:
+                # self._if_manager.handle_netlink_msgs(msg[1]["data"])
+
+        # if config1["netns"] != None:
+            # hwaddr = dev1.get_hwaddr()
+            # self.set_if_netns(if_id1, config1["netns"])
+
+            # msg = {"type": "command", "method_name":
"configure_interface",
+                   # "args": [if_id1, config1]}
+            # self._server_handler.send_data_to_netns(config1["netns"], msg)
+            # result = self._slave_server.wait_for_result(config1["netns"])
+            # if result["result"] != True:
+                # raise Exception("Configuration failed.")
+        # else:
+            # dev1.configure()
+        # if config2["netns"] != None:
+            # hwaddr = dev2.get_hwaddr()
+            # self.set_if_netns(if_id2, config2["netns"])
+
+            # msg = {"type": "command", "method_name":
"configure_interface",
+                   # "args": [if_id2, config2]}
+            # self._server_handler.send_data_to_netns(config2["netns"], msg)
+            # result = self._slave_server.wait_for_result(config2["netns"])
+            # if result["result"] != True:
+                # raise Exception("Configuration failed.")
+        # else:
+            # dev2.configure()
+        # return dev_names
+
+    # def deconfigure_if_pair(self, if_id1, if_id2):
+        # dev1 = self._if_manager.get_mapped_device(if_id1)
+        # dev2 = self._if_manager.get_mapped_device(if_id2)
+
+        # if dev1.get_netns() == None:
+            # dev1.deconfigure()
+        # else:
+            # netns = dev1.get_netns()
+
+            # msg = {"type": "command", "method_name":
"deconfigure_interface",
+                   # "args": [if_id1]}
+            # self._server_handler.send_data_to_netns(netns, msg)
+            # result = self._slave_server.wait_for_result(netns)
+            # if result["result"] != True:
+                # raise Exception("Deconfiguration failed.")
+
+            # self.return_if_netns(if_id1)
+
+        # if dev2.get_netns() == None:
+            # dev2.deconfigure()
+        # else:
+            # netns = dev2.get_netns()
+
+            # msg = {"type": "command", "method_name":
"deconfigure_interface",
+                   # "args": [if_id2]}
+            # self._server_handler.send_data_to_netns(netns, msg)
+            # result = self._slave_server.wait_for_result(netns)
+            # if result["result"] != True:
+                # raise Exception("Deconfiguration failed.")
+
+            # self.return_if_netns(if_id2)
+
+        # dev1.destroy()
+        # dev2.destroy()
+        # dev1.del_configuration()
+        # dev2.del_configuration()
+        # return True
+
+    # def deconfigure_interface(self, if_id):
+        # device = self._if_manager.get_mapped_device(if_id)
+        # if device is not None:
+            # device.clear_configuration()
+        # else:
+            # logging.error("No device with id '%s' to deconfigure." %
if_id)
+        # return True
 
     def start_packet_capture(self, filt):
         if not is_installed("tcpdump"):
@@ -448,103 +421,67 @@ class SlaveMethods:
 
         return int(remaining)
 
-    def run_command(self, command):
-        cmd = NetTestCommand(self._command_context, command,
-                                    self._resource_table, self._log_ctl)
+    def run_job(self, job):
+        job_instance = Job(job, self._log_ctl)
+        self._job_context.add_job(job_instance)
 
-        if self._command_context.get_cmd(cmd.get_id()) != None:
-            prev_cmd = self._command_context.get_cmd(cmd.get_id())
-            if not prev_cmd.get_result_sent():
-                if cmd.get_id() is None:
-                    raise Exception("Previous foreground command still "\
-                                    "running!")
-                else:
-                    raise Exception("Different command with id '%s' "\
-                                    "still running!" % cmd.get_id())
-            else:
-                self._command_context.del_cmd(cmd)
-        self._command_context.add_cmd(cmd)
+        res = job_instance.run()
 
-        res = cmd.run()
-        if not cmd.forked():
-            self._command_context.del_cmd(cmd)
+        return res
 
-        if command["type"] == "config":
-            if res["passed"]:
-               
self._update_system_config(res["res_data"]["options"],
-                                           command["persistent"])
-            else:
-                err = "Error occured while setting system "\
-                      "configuration (%s)" %
res["res_data"]["err_msg"]
-                logging.error(err)
+    def kill_job(self, job_id, signal):
+        job = self._job_context.get_job(job_id)
 
-        return res
+        if job is None:
+            logging.error("No job %s found" % job_id)
+            return False
 
-    def kill_command(self, id):
-        cmd = self._command_context.get_cmd(id)
-        if cmd is not None:
-            if not cmd.get_result_sent():
-                cmd.kill(None)
-                result = cmd.get_result()
-                cmd.set_result_sent()
-                return result
-            else:
-                pass
-        else:
-            raise Exception("No command with id '%s'." % id)
+        return job.kill(signal)
+
+    def kill_jobs(self):
+        logging.info("Killing all forked processes.")
+        self._job_context.cleanup()
+        return "Commands killed"
 
     def machine_cleanup(self):
         logging.info("Performing machine cleanup.")
-        self._command_context.cleanup()
+        self._job_context.cleanup()
 
         self.restore_system_config()
 
-        devs = self._if_manager.get_mapped_devices()
-        for if_id, dev in devs.iteritems():
-            peer = dev.get_peer()
-            if peer == None:
-                dev.clear_configuration()
-            else:
-                peer_if_index = peer.get_if_index()
-                peer_id = self._if_manager.get_id_by_if_index(peer_if_index)
-                self.deconfigure_if_pair(if_id, peer_id)
-
-        self._if_manager.deconfigure_all()
+        if self._if_manager is not None:
+            self._if_manager.deconfigure_all()
 
         for netns in self._net_namespaces.keys():
             self.del_namespace(netns)
         self._net_namespaces = {}
 
-        self._if_manager.clear_if_mapping()
+        for cls_name, cls in self._dynamic_classes.items():
+            delattr(Devices, cls_name)
+
+        for module_name, module in self._dynamic_modules.items():
+            del sys.modules[module_name]
+
+        self._dynamic_classes = {}
+        self._dynamic_modules = {}
+        self._if_manager = None
+        self._server_handler.set_if_manager(None)
         self._cache.del_old_entries()
         self._remove_capture_files()
         return True
 
-    def clear_resource_table(self):
-        self._resource_table = {'module': {}, 'tools': {}}
-        return True
-
     def has_resource(self, res_hash):
         if self._cache.query(res_hash):
             return True
 
         return False
 
-    def map_resource(self, res_hash, res_type, res_name):
-        resource_location = self._cache.get_path(res_hash)
-
-        if not res_type in self._resource_table:
-            self._resource_table[res_type] = {}
-
-        self._resource_table[res_type][res_name] = resource_location
-        self._cache.renew_entry(res_hash)
-
-        return True
-
-    def add_resource_to_cache(self, file_hash, local_path, name,
-                                res_hash, res_type):
-        self._cache.add_cache_entry(file_hash, local_path, name, res_type)
-        return True
+    def add_resource_to_cache(self, res_type, local_path, name):
+        if res_type == "file":
+            self._cache.add_file_entry(local_path, name)
+            return True
+        else:
+            raise Exception("Unknown resource type")
 
     def start_copy_to(self, filepath=None):
         if filepath in self._copy_targets:
@@ -559,9 +496,9 @@ class SlaveMethods:
 
         return filepath
 
-    def copy_part_to(self, filepath, binary_data):
+    def copy_part_to(self, filepath, data):
         if self._copy_targets[filepath]:
-            self._copy_targets[filepath].write(binary_data.data)
+            self._copy_targets[filepath].write(data)
             return True
 
         return False
@@ -583,7 +520,7 @@ class SlaveMethods:
         return True
 
     def copy_part_from(self, filepath, buffsize):
-        data = Binary(self._copy_sources[filepath].read(buffsize))
+        data = self._copy_sources[filepath].read(buffsize)
         return data
 
     def finish_copy_from(self, filepath):
@@ -607,26 +544,27 @@ class SlaveMethods:
        
logging.warning("====================================================")
         logging.warning("Enabling use of NetworkManager on controller
request")
        
logging.warning("====================================================")
-        val = lnst_config.get_option("environment", "use_nm")
-        lnst_config.set_option("environment", "use_nm", True)
+        val = self._slave_config.get_option("environment", "use_nm")
+        self._slave_config.set_option("environment", "use_nm", True)
         return val
 
     def disable_nm(self):
        
logging.warning("=====================================================")
         logging.warning("Disabling use of NetworkManager on controller
request")
        
logging.warning("=====================================================")
-        val = lnst_config.get_option("environment", "use_nm")
-        lnst_config.set_option("environment", "use_nm", False)
+        val = self._slave_config.get_option("environment", "use_nm")
+        self._slave_config.set_option("environment", "use_nm",
False)
         return val
 
     def restore_nm_option(self):
-        val = lnst_config.get_option("environment", "use_nm")
+        val = self._slave_config.get_option("environment", "use_nm")
         if val == self._bkp_nm_opt_val:
             return val
         logging.warning("=========================================")
         logging.warning("Restoring use_nm option to original value")
         logging.warning("=========================================")
-        lnst_config.set_option("environment", "use_nm",
self._bkp_nm_opt_val)
+        self._slave_config.set_option("environment", "use_nm",
+                                      self._bkp_nm_opt_val)
         return val
 
     def add_namespace(self, netns):
@@ -724,40 +662,40 @@ class SlaveMethods:
             del self._net_namespaces[netns]
             return True
 
-    def set_if_netns(self, if_id, netns):
-        netns_pid = self._net_namespaces[netns]["pid"]
-
-        device = self._if_manager.get_mapped_device(if_id)
-        dev_name = device.get_name()
-        device.set_netns(netns)
-        hwaddr = device.get_hwaddr()
-
-        exec_cmd("ip link set %s netns %d" % (dev_name, netns_pid))
-        msg = {"type": "command", "method_name":
"map_if_by_hwaddr",
-               "args": [if_id, hwaddr]}
-        self._server_handler.send_data_to_netns(netns, msg)
-        result = self._slave_server.wait_for_result(netns)
-        return result
-
-    def return_if_netns(self, if_id):
-        device = self._if_manager.get_mapped_device(if_id)
-        if device.get_netns() == None:
-            dev_name = device.get_name()
-            ppid = os.getppid()
-            exec_cmd("ip link set %s netns %d" % (dev_name, ppid))
-            self._if_manager.unmap_if(if_id)
-            return True
-        else:
-            netns = device.get_netns()
-            msg = {"type": "command", "method_name":
"return_if_netns",
-                   "args": [if_id]}
-            self._server_handler.send_data_to_netns(netns, msg)
-            result = self._slave_server.wait_for_result(netns)
-            if result["result"] != True:
-                raise Exception("Return from netns failed.")
-
-            device.set_netns(None)
-            return True
+    # def set_if_netns(self, if_id, netns):
+        # netns_pid = self._net_namespaces[netns]["pid"]
+
+        # device = self._if_manager.get_mapped_device(if_id)
+        # dev_name = device.get_name()
+        # device.set_netns(netns)
+        # hwaddr = device.get_hwaddr()
+
+        # exec_cmd("ip link set %s netns %d" % (dev_name, netns_pid))
+        # msg = {"type": "command", "method_name":
"map_if_by_hwaddr",
+               # "args": [if_id, hwaddr]}
+        # self._server_handler.send_data_to_netns(netns, msg)
+        # result = self._slave_server.wait_for_result(netns)
+        # return result
+
+    # def return_if_netns(self, if_id):
+        # device = self._if_manager.get_mapped_device(if_id)
+        # if device.get_netns() == None:
+            # dev_name = device.get_name()
+            # ppid = os.getppid()
+            # exec_cmd("ip link set %s netns %d" % (dev_name, ppid))
+            # self._if_manager.unmap_if(if_id)
+            # return True
+        # else:
+            # netns = device.get_netns()
+            # msg = {"type": "command", "method_name":
"return_if_netns",
+                   # "args": [if_id]}
+            # self._server_handler.send_data_to_netns(netns, msg)
+            # result = self._slave_server.wait_for_result(netns)
+            # if result["result"] != True:
+                # raise Exception("Return from netns failed.")
+
+            # device.set_netns(None)
+            # return True
 
     def add_br_vlan(self, if_id, br_vlan_info):
         dev = self._if_manager.get_mapped_device(if_id)
@@ -847,48 +785,8 @@ class SlaveMethods:
         brt.set_state(br_state_info)
         return True
 
-    def set_speed(self, if_id, speed):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.set_speed(speed)
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def set_autoneg(self, if_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.set_autoneg()
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def wait_interface_init(self):
-        self._if_manager.wait_interface_init()
-        return True
-
-    def slave_add(self, if_id, slave_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.slave_add(slave_id)
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
-    def slave_del(self, if_id, slave_id):
-        dev = self._if_manager.get_mapped_device(if_id)
-        if dev is not None:
-            dev.slave_del(slave_id)
-        else:
-            logging.error("Device with id '%s' not found." % if_id)
-            return False
-        return True
-
 class ServerHandler(ConnectionHandler):
-    def __init__(self, addr):
+    def __init__(self, addr, slave_config):
         super(ServerHandler, self).__init__()
         self._netns_con_mapping = {}
         try:
@@ -902,13 +800,34 @@ class ServerHandler(ConnectionHandler):
 
         self._netns = None
         self._c_socket = None
+        self._c_dev = None
 
         self._if_manager = None
 
-        self._security = lnst_config.get_section_values("security")
+        self._security = slave_config.get_section_values("security")
 
     def set_if_manager(self, if_manager):
         self._if_manager = if_manager
+        self._update_c_dev()
+
+    def _update_c_dev(self):
+        if self._c_dev:
+            self._c_dev.enable()
+            self._c_dev = None
+
+        if self._if_manager is not None:
+            ctl_socket = self.get_ctl_sock()
+            ctl_addr = ctl_socket._socket.getsockname()[0]
+            matched_dev = None
+            for dev in self._if_manager.get_devices():
+                for ip in dev.ips:
+                    if ip.addr == ctl_addr:
+                        matched_dev = dev
+                        break
+                if matched_dev:
+                    break
+            self._c_dev = matched_dev
+            matched_dev.disable()
 
     def accept_connection(self):
         self._c_socket, addr = self._s_socket.accept()
@@ -932,9 +851,9 @@ class ServerHandler(ConnectionHandler):
 
     def set_ctl_sock(self, sock):
         if self._c_socket != None:
-            self._c_socket.close()
-            self._c_socket = None
+            self.close_c_sock()
         self._c_socket = sock
+        self._update_c_dev()
         self.add_connection(self._c_socket[1], self._c_socket[0])
 
     def close_s_sock(self):
@@ -946,9 +865,14 @@ class ServerHandler(ConnectionHandler):
         self.remove_connection(self._c_socket[0])
         self._c_socket = None
 
+        if self._c_dev:
+            self._c_dev.enable()
+            self._c_dev = None
+
     def check_connections(self):
         msgs = super(ServerHandler, self).check_connections()
-        if 'netlink' not in self._connection_mapping:
+        if 'netlink' not in self._connection_mapping and\
+                self._if_manager is not None:
             self._if_manager.reconnect_netlink()
             self.add_connection('netlink', self._if_manager.get_nl_socket())
         return msgs
@@ -970,7 +894,7 @@ class ServerHandler(ConnectionHandler):
         addr = self._c_socket[1]
         if self.get_connection(addr) == None:
             logging.info("Lost controller connection.")
-            self._c_socket = None
+            self.close_c_sock()
         return messages
 
     def get_messages_from_con(self, con_id):
@@ -1026,23 +950,72 @@ class ServerHandler(ConnectionHandler):
             self._connections.remove(con)
         self._netns_con_mapping = {}
 
+
+def device_to_deviceref(obj):
+    try:
+        Device = Devices.Device
+    except:
+        return obj
+
+    if isinstance(obj, Device):
+        dev_ref = DeviceRef(obj.if_index)
+        return dev_ref
+    elif isinstance(obj, dict):
+        new_dict = {}
+        for key, value in obj.items():
+            new_dict[key] = device_to_deviceref(value)
+        return new_dict
+    elif isinstance(obj, list):
+        new_list = []
+        for value in obj:
+            new_list.append(device_to_deviceref(value))
+        return new_list
+    elif isinstance(obj, tuple):
+        new_list = []
+        for value in obj:
+            new_list.append(device_to_deviceref(value))
+        return tuple(new_list)
+    else:
+        return obj
+
+def deviceref_to_device(if_manager, obj):
+    if isinstance(obj, DeviceRef):
+        dev = if_manager.get_device(obj.if_index)
+        return dev
+    elif isinstance(obj, dict):
+        new_dict = {}
+        for key, value in obj.items():
+            new_dict[key] = deviceref_to_device(if_manager, value)
+        return new_dict
+    elif isinstance(obj, list):
+        new_list = []
+        for value in obj:
+            new_list.append(deviceref_to_device(if_manager, value))
+        return new_list
+    elif isinstance(obj, tuple):
+        new_list = []
+        for value in obj:
+            new_list.append(deviceref_to_device(if_manager, value))
+        return tuple(new_list)
+    else:
+        return obj
+
 class NetTestSlave:
-    def __init__(self, log_ctl):
+    def __init__(self, log_ctl, slave_config):
+        self._slave_config = slave_config
         die_when_parent_die()
 
-        self._cmd_context = NetTestCommandContext()
-        port = lnst_config.get_option("environment", "rpcport")
+        self._job_context = JobContext()
+        port = slave_config.get_option("environment", "rpcport")
         logging.info("Using RPC port %d." % port)
-        self._server_handler = ServerHandler(("", port))
-        self._if_manager = InterfaceManager(self._server_handler)
-
-        self._server_handler.set_if_manager(self._if_manager)
+        self._server_handler = ServerHandler(("", port), slave_config)
 
         self._net_namespaces = {}
 
-        self._methods = SlaveMethods(self._cmd_context, log_ctl,
-                                     self._if_manager, self._net_namespaces,
-                                     self._server_handler, self)
+        self._methods = SlaveMethods(self._job_context, log_ctl,
+                                     self._net_namespaces,
+                                     self._server_handler, slave_config,
+                                     self)
 
         self.register_die_signal(signal.SIGHUP)
         self.register_die_signal(signal.SIGINT)
@@ -1052,9 +1025,6 @@ class NetTestSlave:
 
         self._log_ctl = log_ctl
 
-        self._server_handler.add_connection('netlink',
-                                            self._if_manager.get_nl_socket())
-
     def run(self):
         while not self._finished:
             if self._server_handler.get_ctl_sock() == None:
@@ -1063,9 +1033,9 @@ class NetTestSlave:
                     logging.info("Waiting for connection.")
                     self._server_handler.accept_connection()
                 except (socket.error, SecSocketException):
+                    log_exc_traceback()
                     continue
-                self._log_ctl.set_connection(
-                                            self._server_handler.get_ctl_sock())
+                self._log_ctl.set_connection(self._server_handler.get_ctl_sock())
 
             msgs = self._server_handler.get_messages()
 
@@ -1092,24 +1062,29 @@ class NetTestSlave:
         if msg["type"] == "command":
             method = getattr(self._methods, msg["method_name"], None)
             if method != None:
+                if_manager = self._methods._if_manager
+                if if_manager is not None:
+                    args = deviceref_to_device(if_manager, msg["args"])
+                    kwargs = deviceref_to_device(if_manager, msg["kwargs"])
+                else:
+                    args = msg["args"]
+                    kwargs = msg["kwargs"]
+
                 try:
-                    result = method(*msg["args"])
-                except:
+                    result = method(*args, **kwargs)
+                except LnstError as e:
                     log_exc_traceback()
-                    type, value, tb = sys.exc_info()
-                    exc_trace = ''.join(traceback.format_exception(type,
-                                                                   value, tb))
-                    response = {"type": "exception",
"Exception": value}
+                    response = {"type": "exception",
"Exception": e}
 
                     self._server_handler.send_data_to_ctl(response)
                     return
 
-                if result != None:
-                    response = {"type": "result", "result":
result}
-                    self._server_handler.send_data_to_ctl(response)
+                response = {"type": "result", "result":
result}
+                response = device_to_deviceref(response)
+                self._server_handler.send_data_to_ctl(response)
             else:
-                err = "Method '%s' not supported." %
msg["method_name"]
-                response = {"type": "error", "err": err}
+                err = LnstError("Method '%s' not supported." %
msg["method_name"])
+                response = {"type": "exception",
"Exception": err}
                 self._server_handler.send_data_to_ctl(response)
         elif msg["type"] == "log":
             logger = logging.getLogger()
@@ -1122,48 +1097,36 @@ class NetTestSlave:
             else:
                 logging.debug("Recieved an exception from foreground command")
             logging.debug(msg["Exception"])
-            cmd = self._cmd_context.get_cmd(msg["cmd_id"])
-            cmd.join()
-            self._cmd_context.del_cmd(cmd)
+            job = self._job_context.get_cmd(msg["job_id"])
+            job.join()
+            self._job_context.del_cmd(job)
+            self._server_handler.send_data_to_ctl(msg)
+        elif msg["type"] == "job_finished":
+            job = self._job_context.get_job(msg["job_id"])
+            job.join()
+
+            job.set_finished(msg["result"])
             self._server_handler.send_data_to_ctl(msg)
-        elif msg["type"] == "result":
-            if msg["cmd_id"] == None:
-                del msg["cmd_id"]
-                self._server_handler.send_data_to_ctl(msg)
-                cmd = self._cmd_context.get_cmd(None)
-                cmd.join()
-                cmd.set_result_sent()
-            else:
-                cmd = self._cmd_context.get_cmd(msg["cmd_id"])
-                cmd.join()
-                del msg["cmd_id"]
-
-                cmd.set_result(msg["result"])
-                if cmd.finished():
-                    msg["result"] = cmd.get_result()
-                    self._server_handler.send_data_to_ctl(msg)
-                    cmd.set_result_sent()
         elif msg["type"] == "netlink":
-            self._if_manager.handle_netlink_msgs(msg["data"])
+            if_manager = self._methods._if_manager
+            if if_manager is not None:
+                if_manager.handle_netlink_msgs(msg["data"])
         elif msg["type"] == "from_netns":
             self._server_handler.send_data_to_ctl(msg["data"])
         elif msg["type"] == "to_netns":
             netns = msg["netns"]
             try:
                 self._server_handler.send_data_to_netns(netns, msg["data"])
-            except:
+            except LnstError as e:
                 log_exc_traceback()
-                type, value, tb = sys.exc_info()
-                exc_trace = ''.join(traceback.format_exception(type,
-					   value, tb))
-                response = {"type": "exception",
"Exception": value}
+                response = {"type": "exception",
"Exception": e}
 
                 self._server_handler.send_data_to_ctl(response)
                 return
         else:
             raise Exception("Recieved unknown command")
 
-        pipes = self._cmd_context.get_read_pipes()
+        pipes = self._job_context.get_parent_pipes()
         self._server_handler.update_connections(pipes)
 
     def register_die_signal(self, signum):
-- 
2.13.0

[PATCH v2 34/47] lnst.Slave.InterfaceManager: remove Device class implementation

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Not needed anymore as all Device related code is now implemented in it's
own package.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Slave/InterfaceManager.py | 382 -----------------------------------------
 1 file changed, 382 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index da7e712..585adb9 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -240,385 +240,3 @@ class InterfaceManager(object):
         while (self._is_name_used(prefix + str(index2))):
             index2 += 1
         return prefix + str(index1), prefix + str(index2)
-
-class Device(object):
-    def __init__(self, if_manager):
-        self._initialized = False
-        self._configured = False
-        self._created = False
-
-        self._if_index = None
-        self._hwaddr = None
-        self._name = None
-        self._conf = None
-        self._conf_dict = None
-        self._ip_addrs = []
-        self._ifi_type = None
-        self._state = None
-        self._master = {"primary": None, "other": []}
-        self._slaves = []
-        self._netns = None
-        self._peer = None
-        self._mtu = None
-        self._driver = None
-        self._devlink = None
-
-        self._if_manager = if_manager
-
-    def set_devlink(self, devlink_port_data):
-        self._devlink = devlink_port_data
-
-    def init_netlink(self, nl_msg):
-        self._if_index = nl_msg['index']
-        self._ifi_type = nl_msg['ifi_type']
-        self._hwaddr = normalize_hwaddr(nl_msg.get_attr("IFLA_ADDRESS"))
-        self._name = nl_msg.get_attr("IFLA_IFNAME")
-        self._state = nl_msg.get_attr("IFLA_OPERSTATE")
-        self._ip_addrs = []
-        self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
-        self._netns = None
-        self._mtu = nl_msg.get_attr("IFLA_MTU")
-
-        if self._driver is None:
-            self._driver = self._ethtool_get_driver()
-
-        self._initialized = True
-
-        #return an update message that will be sent to the controller
-        return {"type": "if_update",
-                "if_data": self.get_if_data()}
-
-    def update_netlink(self, nl_msg):
-        if self._if_index != nl_msg['index']:
-            return None
-        if nl_msg['header']['type'] == RTM_NEWLINK:
-            self._ifi_type = nl_msg['ifi_type']
-            self._hwaddr = normalize_hwaddr(nl_msg.get_attr("IFLA_ADDRESS"))
-            self._name = nl_msg.get_attr("IFLA_IFNAME")
-            self._state = nl_msg.get_attr("IFLA_OPERSTATE")
-            self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
-            self._mtu = nl_msg.get_attr("IFLA_MTU")
-
-            link = nl_msg.get_attr("IFLA_LINK")
-            if link != None:
-                # IFLA_LINK is an index of device that's closer to physical
-                # interface in the stack, e.g. index of eth0 for eth0.100
-                # so to properly deconfigure the stack we have to save
-                # parent index in the child device; this is the opposite
-                # to IFLA_MASTER
-                link_dev = self._if_manager.get_device(link)
-                if link_dev != None:
-                    link_dev.set_master(self._if_index, primary=False)
-                # This reference shouldn't change - you can't change the realdev
-                # of a vlan, you need to create a new vlan. Therefore the
-                # the following add_slave shouldn't be a problem.
-                self.add_slave(link)
-
-            if self._conf_dict:
-                self._conf_dict["name"] = self._name
-
-            if self._driver is None:
-                self._driver = self._ethtool_get_driver()
-
-            self._initialized = True
-        elif nl_msg['header']['type'] == RTM_NEWADDR:
-            scope = nl_msg['scope']
-            addr_val = nl_msg.get_attr('IFA_ADDRESS')
-            prefix_len = str(nl_msg['prefixlen'])
-            addr = {"addr": addr_val,
-                    "prefix": prefix_len,
-                    "scope": scope}
-            if self.find_addrs(addr) == []:
-                self._ip_addrs.append(addr)
-        elif nl_msg['header']['type'] == RTM_DELADDR:
-            scope = nl_msg['scope']
-            addr_val = nl_msg.get_attr('IFA_ADDRESS')
-            prefix_len = str(nl_msg['prefixlen'])
-            addr = {"addr": addr_val,
-                    "prefix": prefix_len,
-                    "scope": scope}
-            matching_addrs = self.find_addrs(addr)
-            for ip_addr in matching_addrs:
-                self._ip_addrs.remove(ip_addr)
-
-        #return an update message that will be sent to the controller
-        return {"type": "if_update",
-                "if_data": self.get_if_data()}
-
-    def del_link(self):
-        if self._master["primary"]:
-            primary_id = self._master["primary"]
-            primary_dev = self._if_manager.get_device(primary_id)
-            if primary_dev:
-                primary_dev.del_slave(self._if_index)
-
-        for m_id in self._master["other"]:
-            m_dev = self._if_manager.get_device(m_id)
-            if m_dev:
-                m_dev.del_slave(self._if_index)
-
-        for dev_id in self._slaves:
-            dev = self._if_manager.get_device(dev_id)
-            if dev != None:
-                dev.del_master(self._if_index)
-
-    def find_addrs(self, addr_spec):
-        ret = []
-        for addr in self._ip_addrs:
-            if addr_spec.items() <= addr.items():
-                ret.append(addr)
-        return ret
-
-    def get_if_index(self):
-        return self._if_index
-
-    def get_hwaddr(self):
-        return self._hwaddr
-
-    def get_name(self):
-        return self._name
-
-    def get_ips(self):
-        return self._ip_addrs
-
-    def clear_ips(self):
-        self._ip_addrs = []
-
-    def is_configured(self):
-        return self._configured
-
-    def get_conf_dict(self):
-        return self._conf_dict
-
-    def set_peer(self, dev):
-        self._peer = dev
-
-    def get_peer(self):
-        return self._peer
-
-    def set_configuration(self, conf):
-        self.clear_configuration()
-        if "name" not in conf or conf["name"] == None:
-            conf["name"] = self._name
-        self._conf_dict = conf
-        self._conf = NetConfigDevice(conf, self._if_manager)
-
-        if not self._initialized:
-            self._name = conf["name"]
-
-    def get_configuration(self):
-        return self._conf
-
-    def del_configuration(self):
-        self._conf = None
-        self._conf_dict = None
-
-    def _clear_tc_qdisc(self):
-        exec_cmd("tc qdisc replace dev %s root pfifo" % self._name)
-        out, _ = exec_cmd("tc filter show dev %s" % self._name)
-        ingress_handles = re.findall("ingress (\\d+):", out)
-        for ingress_handle in ingress_handles:
-            exec_cmd("tc qdisc del dev %s handle %s: ingress" %
-                     (self._name, ingress_handle))
-        out, _ = exec_cmd("tc qdisc show dev %s" % self._name)
-        ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out)
-        if len(ingress_qdiscs) != 0:
-                exec_cmd("tc qdisc del dev %s ingress" % self._name)
-
-    def _clear_tc_filters(self):
-        out, _ = exec_cmd("tc filter show dev %s" % self._name)
-        egress_prefs = re.findall("pref (\\d+) .* handle", out)
-
-        for egress_pref in egress_prefs:
-            exec_cmd("tc filter del dev %s pref %s" % (self._name,
-                     egress_pref))
-
-    def clear_configuration(self):
-        if self._master["primary"]:
-            primary_id = self._master["primary"]
-            primary_dev = self._if_manager.get_device(primary_id)
-            if primary_dev:
-                primary_dev.clear_configuration()
-
-        for m_id in self._master["other"]:
-            m_dev = self._if_manager.get_device(m_id)
-            if m_dev:
-                m_dev.clear_configuration()
-
-        if self._conf != None:
-            self._clear_tc_qdisc()
-            self._clear_tc_filters()
-            self.down()
-            self.deconfigure()
-            self.destroy()
-            self._conf = None
-            self._conf_dict = None
-
-    def set_master(self, if_index, primary=True):
-        if primary:
-            prev_master_id = self._master["primary"]
-            if prev_master_id != None and if_index != prev_master_id:
-                prev_master_dev = self._if_manager.get_device(prev_master_id)
-                if prev_master_dev != None:
-                    prev_master_dev.del_slave(self._if_index)
-
-            self._master["primary"] = if_index
-            if self._master["primary"] != None:
-                master_id = self._master["primary"]
-                master_dev = self._if_manager.get_device(master_id)
-                if master_dev != None:
-                    master_dev.add_slave(self._if_index)
-        elif if_index not in self._master["other"]:
-            self._master["other"].append(if_index)
-
-    def del_master(self, if_index):
-        if self._master["primary"] == if_index:
-            self._master["primary"] = None
-        elif if_index in self._master["other"]:
-            self._master["other"].remove(if_index)
-
-    def add_slave(self, if_index):
-        if if_index not in self._slaves:
-            self._slaves.append(if_index)
-
-    def del_slave(self, if_index):
-        if if_index in self._slaves:
-            self._slaves.remove(if_index)
-
-    def create(self):
-        if self._conf != None and not self._created:
-            self._conf.create()
-            self._created = True
-            return True
-        return False
-
-    def destroy(self):
-        if self._conf != None and self._created:
-            self._conf.destroy()
-            self._created = False
-            return True
-        return False
-
-    def configure(self):
-        if self._conf != None and not self._configured:
-            self._conf.configure()
-            self._configured = True
-
-    def deconfigure(self):
-        if self._master["primary"]:
-            primary_id = self._master["primary"]
-            primary_dev = self._if_manager.get_device(primary_id)
-            if primary_dev:
-                primary_dev.deconfigure()
-
-        for m_id in self._master["other"]:
-            m_dev = self._if_manager.get_device(m_id)
-            if m_dev:
-                m_dev.deconfigure()
-
-        if self._conf != None and self._configured:
-            self._conf.deconfigure()
-            self._configured = False
-
-    def up(self):
-        if self._conf != None:
-            self._conf.up()
-        else:
-            exec_cmd("ip link set %s up" % self._name)
-
-    def down(self):
-        if self._conf != None:
-            self._conf.down()
-        else:
-            exec_cmd("ip link set %s down" % self._name)
-
-    def link_up(self):
-        exec_cmd("ip link set %s up" % self._name)
-
-    def link_down(self):
-        exec_cmd("ip link set %s down" % self._name)
-
-    def link_stats(self):
-        stats = {"devname": self._name,
-                 "hwaddr": self._hwaddr}
-        out, _ = exec_cmd("ip -s link show %s" % self._name)
-        lines = iter(out.split("\n"))
-        for line in lines:
-            if (len(line.split()) == 0):
-                continue
-            if (line.split()[0] == "RX:"):
-                rx_stats = map(int, lines.next().split())
-                stats.update({"rx_bytes"  : rx_stats[0],
-                              "rx_packets": rx_stats[1],
-                              "rx_errors" : rx_stats[2],
-                              "rx_dropped": rx_stats[3],
-                              "rx_overrun": rx_stats[4],
-                              "rx_mcast"  : rx_stats[5]})
-            if (line.split()[0] == "TX:"):
-                tx_stats = map(int, lines.next().split())
-                stats.update({"tx_bytes"  : tx_stats[0],
-                              "tx_packets": tx_stats[1],
-                              "tx_errors" : tx_stats[2],
-                              "tx_dropped": tx_stats[3],
-                              "tx_carrier": tx_stats[4],
-                              "tx_collsns": tx_stats[5]})
-        return stats
-
-    def set_addresses(self, ips):
-        self._conf.set_addresses(ips)
-        exec_cmd("ip addr flush %s" % self._name)
-        for address in ips:
-            exec_cmd("ip addr add %s dev %s" % (address, self._name))
-
-    def add_route(self, dest):
-        exec_cmd("ip route add %s dev %s" % (dest, self._name))
-
-    def del_route(self, dest):
-        exec_cmd("ip route del %s dev %s" % (dest, self._name))
-
-    def set_netns(self, netns):
-        self._netns = netns
-        return
-
-    def get_netns(self):
-        return self._netns
-
-    def _ethtool_get_driver(self):
-        if self._ifi_type == 772:  #loopback ifi type
-            return 'loopback'
-        out, _ = exec_cmd("ethtool -i %s" % self._name, False, False, False)
-        match = re.search("^driver: (.*)$", out, re.MULTILINE)
-        if match is not None:
-            return match.group(1)
-        else:
-            return None
-
-    def get_if_data(self):
-        if_data = {"if_index": self._if_index,
-                   "hwaddr": self._hwaddr,
-                   "name": self._name,
-                   "ip_addrs": self._ip_addrs,
-                   "ifi_type": self._ifi_type,
-                   "state": self._state,
-                   "master": self._master,
-                   "slaves": self._slaves,
-                   "netns": self._netns,
-                   "peer": self._peer.get_if_index() if self._peer else None,
-                   "mtu": self._mtu,
-                   "driver": self._driver,
-                   "devlink": self._devlink}
-        return if_data
-
-    def set_speed(self, speed):
-        exec_cmd("ethtool -s %s speed %s autoneg off" % (self._name, speed))
-
-    def set_autoneg(self):
-        exec_cmd("ethtool -s %s autoneg on" % self._name)
-
-    def slave_add(self, if_id):
-        if self._conf != None:
-            self._conf.slave_add(if_id)
-
-    def slave_del(self, if_id):
-        if self._conf != None:
-            self._conf.slave_del(if_id)
-- 
2.13.0

[PATCH v2 35/47] add lnst.Controller package imports

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

To export the most commonly tester used classes needed for Recipe
implementation. Mostly just convenience for the tester to shorten the
import lines.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/__init__.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lnst/Controller/__init__.py b/lnst/Controller/__init__.py
index e69de29..1f97b40 100644
--- a/lnst/Controller/__init__.py
+++ b/lnst/Controller/__init__.py
@@ -0,0 +1,3 @@
+from lnst.Controller.Controller import Controller
+from lnst.Controller.Recipe import BaseRecipe
+from lnst.Controller.Requirements import HostReq, DeviceReq
-- 
2.13.0

[PATCH v2 36/47] lnst/__init__.py remove imports

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Remove imports related to the old prototype implementation of PyRecipes.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/__init__.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lnst/__init__.py b/lnst/__init__.py
index 924a8ff..e69de29 100644
--- a/lnst/__init__.py
+++ b/lnst/__init__.py
@@ -1 +0,0 @@
-from lnst.Controller.Task import match, add_host, wait, get_alias, get_module,
breakpoint
-- 
2.13.0

[PATCH v2 37/47] lnst.Controller: remove old modules

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

These are not used anymore in the new Python Recipes implementation.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/NetTestController.py | 620 ---------------------------------
 lnst/Controller/RecipeParser.py      | 572 -------------------------------
 lnst/Controller/SlavePool.py         | 648 -----------------------------------
 lnst/Controller/XmlParser.py         | 188 ----------
 lnst/Controller/XmlProcessing.py     | 235 -------------
 lnst/Controller/XmlTemplates.py      | 438 -----------------------
 6 files changed, 2701 deletions(-)
 delete mode 100644 lnst/Controller/NetTestController.py
 delete mode 100644 lnst/Controller/RecipeParser.py
 delete mode 100644 lnst/Controller/SlavePool.py
 delete mode 100644 lnst/Controller/XmlParser.py
 delete mode 100644 lnst/Controller/XmlProcessing.py
 delete mode 100644 lnst/Controller/XmlTemplates.py

diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py
deleted file mode 100644
index 8f41cd0..0000000
--- a/lnst/Controller/NetTestController.py
+++ /dev/null
@@ -1,620 +0,0 @@
-"""
-This module defines NetTestController class which does the controlling
-part of network testing.
-
-Copyright 2011 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-jpirko(a)redhat.com (Jiri Pirko)
-"""
-
-import logging
-import socket
-import os
-import re
-import cPickle
-import imp
-import copy
-import sys
-from time import sleep
-from lnst.Common.NetUtils import MacPool
-from lnst.Common.Utils import md5sum, dir_md5sum
-from lnst.Common.Utils import check_process_running, bool_it, get_module_tools
-from lnst.Common.NetTestCommand import str_command, CommandException
-from lnst.Controller.RecipeParser import RecipeError
-from lnst.Controller.SlavePool import SlavePool
-from lnst.Controller.Machine import MachineError, VirtualInterface
-from lnst.Controller.Machine import StaticInterface
-from lnst.Controller.CtlSecSocket import CtlSecSocket
-from lnst.Common.SecureSocket import SecSocketException
-from lnst.Common.ConnectionHandler import send_data, recv_data
-from lnst.Common.ConnectionHandler import ConnectionHandler
-from lnst.Common.Config import lnst_config
-from lnst.Common.Path import Path
-from lnst.Common.Colours import decorate_with_preset
-from lnst.Common.NetUtils import test_tcp_connection
-import lnst.Controller.Task as Task
-
-# conditional support for libvirt
-if check_process_running("libvirtd"):
-    from lnst.Controller.VirtUtils import VirtNetCtl, VirtDomainCtl
-
-class NetTestError(Exception):
-    pass
-
-class NoMatchError(NetTestError):
-    pass
-
-def ignore_event(**kwarg):
-    pass
-
-class NetTestController:
-    def __init__(self, recipe_path, log_ctl,
-                 res_serializer=None, pool_checks=True,
-                 packet_capture=False,
-                 defined_aliases=None, reduce_sync=False,
-                 restrict_pools=[], multi_match=False,
-                 breakpoints=False):
-        self._res_serializer = res_serializer
-        self._remote_capture_files = {}
-        self._log_ctl = log_ctl
-        self._recipe_path = Path(None, recipe_path)
-        self._msg_dispatcher = MessageDispatcher(log_ctl)
-        self._packet_capture = packet_capture
-        self._reduce_sync = reduce_sync
-        self._defined_aliases = defined_aliases
-        self._multi_match = multi_match
-
-        self.run_mode = "run"
-        self.breakpoints = breakpoints
-
-        self._machines = {}
-        self._network_bridges = {}
-        self._tasks = []
-
-        mac_pool_range = lnst_config.get_option('environment',
'mac_pool_range')
-        self._mac_pool = MacPool(mac_pool_range[0], mac_pool_range[1])
-
-        conf_pools = lnst_config.get_pools()
-        pools = {}
-        if len(restrict_pools) > 0:
-            for pool_name in restrict_pools:
-                if pool_name in conf_pools:
-                    pools[pool_name] = conf_pools[pool_name]
-                elif len(restrict_pools) == 1 and os.path.isdir(pool_name):
-                    pools = {"cmd_line_pool": pool_name}
-                else:
-                    raise NetTestError("Pool %s does not exist!" % pool_name)
-        else:
-            pools = conf_pools
-
-        sp = SlavePool(pools, pool_checks)
-        self._slave_pool = sp
-
-        modules_dirs = lnst_config.get_option('environment',
'module_dirs')
-        tools_dirs = lnst_config.get_option('environment', 'tool_dirs')
-
-        self._resource_table = {}
-        self._resource_table["module"] = self._load_test_modules(modules_dirs)
-        self._resource_table["tools"] = self._load_test_tools(tools_dirs)
-
-    def _get_machineinfo(self, machine_id):
-        try:
-            info = self._recipe["machines"][machine_id]["params"]
-        except KeyError:
-            msg = "Machine parameters requested, but not yet available"
-            raise NetTestError(msg)
-
-        return info
-
-    @staticmethod
-    def _session_die(session, status):
-        logging.debug("%s terminated with status %s", session.command, status)
-        msg = "SSH session terminated with status %s" % status
-        raise NetTestError(msg)
-
-    def _prepare_network(self, resource_sync=True):
-        mreq = Task.get_mreq()
-
-        machines = self._machines
-        for m_id in machines.keys():
-            self._prepare_machine(m_id, resource_sync)
-
-        for machine_id, machine_data in mreq.iteritems():
-            m_id = machine_id
-            m = machines[m_id]
-            namespaces = set()
-            for if_id, iface_data in machine_data["interfaces"].iteritems():
-                self._prepare_interface(m_id, if_id, iface_data)
-
-                if iface_data["netns"] != None:
-                    namespaces.add(iface_data["netns"])
-
-            if len(namespaces) > 0:
-                m.disable_nm()
-
-            ifaces = m.get_ordered_interfaces()
-            for netns in namespaces:
-                m.add_netns(netns)
-
-            for iface in ifaces:
-                iface.configure()
-                if (m._libvirt_domain is None and
-                        isinstance(iface, StaticInterface)):
-                    driver = iface._driver
-                    if_id = iface._id
-                    mapped_machine = self._slave_pool._map['machines'][m_id]
-                    mapped_machine['interfaces'][if_id]['driver'] =
driver
-            for iface in ifaces:
-                iface.up()
-
-            m.wait_interface_init()
-
-    def set_machine_requirements(self):
-        mreq = Task.get_mreq()
-        sp = self._slave_pool
-        sp.set_machine_requirements(mreq)
-
-    def provision_machines(self):
-        sp = self._slave_pool
-        machines = self._machines
-        if not sp.provision_machines(machines):
-            msg = "This setup cannot be provisioned with the current pool."
-            raise NoMatchError(msg)
-
-    def print_match_description(self):
-        sp = self._slave_pool
-        match = sp.get_match()
-        logging.info("Pool match description:")
-        if sp.is_setup_virtual():
-            logging.info("  Setup is using virtual machines.")
-        for m_id, m in sorted(match["machines"].iteritems()):
-            logging.info("  host \"%s\" uses \"%s\"" %
(m_id, m["target"]))
-            for if_id, match in m["interfaces"].iteritems():
-                pool_id = match["target"]
-                logging.info("    interface \"%s\" matched to
\"%s\"" %\
-                                            (if_id, pool_id))
-
-    def get_pool_match(self):
-        return self._slave_pool.get_match()
-
-    def _prepare_machine(self, m_id, resource_sync=True):
-        machine = self._machines[m_id]
-        address = socket.gethostbyname(machine.get_hostname())
-
-        self._log_ctl.add_slave(m_id)
-        machine.set_rpc(self._msg_dispatcher)
-        machine.set_mac_pool(self._mac_pool)
-        machine.set_network_bridges(self._network_bridges)
-
-        recipe_name = os.path.basename(self._recipe_path.abs_path())
-        machine.init_connection(recipe_name)
-
-    def _prepare_interface(self, m_id, if_id, iface_data):
-        machine = self._machines[m_id]
-
-        iface = machine.get_interface(if_id)
-
-        if iface_data["netns"] != None:
-            iface.set_netns(iface_data["netns"])
-
-    def _prepare_command(self, cmd_data):
-        cmd = {"type": cmd_data["type"]}
-        if "host" in cmd_data:
-            cmd["host"] = cmd_data["host"]
-            if cmd["host"] not in self._machines:
-                msg = "Invalid host id '%s'." % cmd["host"]
-                raise RecipeError(msg, cmd_data)
-
-        if "netns" in cmd_data:
-            cmd["netns"] = cmd_data["netns"]
-
-        if "expect" in cmd_data:
-            expect = cmd_data["expect"]
-            if expect not in ["pass", "fail"]:
-                msg = "Illegal expect attribute value."
-                raise RecipeError(msg, cmd_data)
-            cmd["expect"] = expect == "pass"
-
-        if cmd["type"] == "test":
-            cmd["module"] = cmd_data["module"]
-
-            cmd_opts = {}
-            if "options" in cmd_data:
-                for opt in cmd_data["options"]:
-                    name = opt["name"]
-                    val = opt["value"]
-
-                    if name not in cmd_opts:
-                        cmd_opts[name] = []
-
-                    cmd_opts[name].append({"value": val})
-            cmd["options"] = cmd_opts
-        elif cmd["type"] == "exec":
-            cmd["command"] = cmd_data["command"]
-
-            if "from" in cmd_data:
-                cmd["from"] = cmd_data["from"]
-        elif cmd["type"] in ["wait", "intr",
"kill"]:
-            # 'proc_id' is used to store bg_id for wait/kill/intr
-            # 'bg_id' is used for test/exec
-            # this is used to distinguish between the two in NetTestSlave code
-            cmd["proc_id"] = cmd_data["bg_id"]
-        elif cmd["type"] == "config":
-            cmd["persistent"] = False
-            if "persistent" in cmd_data:
-                cmd["persistent"] = bool_it(cmd_data["persistent"])
-
-            cmd["options"] = []
-            for opt in cmd_data["options"]:
-                name = opt["name"]
-                value = opt["value"]
-                cmd["options"].append({"name": name,
"value": value})
-        elif cmd["type"] == "ctl_wait":
-            cmd["seconds"] = int(cmd_data["seconds"])
-        else:
-            msg = "Unknown command type '%s'" % cmd["type"]
-            raise RecipeError(msg, cmd_data)
-
-
-        if cmd["type"] in ["test", "exec"]:
-            if "bg_id" in cmd_data:
-                cmd["bg_id"] = cmd_data["bg_id"]
-
-            if "timeout" in cmd_data:
-                try:
-                    cmd["timeout"] = int(cmd_data["timeout"])
-                except ValueError:
-                    msg = "Timeout value must be an integer."
-                    raise RecipeError(msg, cmd_data)
-
-        return cmd
-
-    def _check_task(self, task):
-        err = False
-        bg_ids = {}
-        for i, command in enumerate(task["skeleton"]):
-            if command["type"] == "ctl_wait":
-                continue
-
-            machine_id = command["host"]
-            if not machine_id in bg_ids:
-                bg_ids[machine_id] = set()
-
-            cmd_type = command["type"]
-            if cmd_type in ["wait", "intr", "kill"]:
-                bg_id = command["proc_id"]
-                if bg_id in bg_ids[machine_id]:
-                    bg_ids[machine_id].remove(bg_id)
-                else:
-                    logging.error("Found command \"%s\" for bg_id
\"%s\" on "
-                              "host \"%s\" which was not previously
"
-                              "defined", cmd_type, bg_id, machine_id)
-                    err = True
-
-            if "bg_id" in command:
-                bg_id = command["bg_id"]
-                if not bg_id in bg_ids[machine_id]:
-                    bg_ids[machine_id].add(bg_id)
-                else:
-                    logging.error("Command \"%d\" uses bg_id
\"%s\" on host"
-                              "\"%s\" which is already used",
-                                            i, bg_id, machine_id)
-                    err = True
-
-        for machine_id in bg_ids:
-            for bg_id in bg_ids[machine_id]:
-                logging.error("bg_id \"%s\" on host \"%s\" has
no kill/wait "
-                          "command to it", bg_id, machine_id)
-                err = True
-
-        return err
-
-    def _cleanup_slaves(self):
-        if self._machines == None:
-            return
-
-        for machine_id, machine in self._machines.iteritems():
-            if machine.is_configured():
-                try:
-                    machine.cleanup()
-                except:
-                    pass
-
-                #clean-up slave logger
-                self._log_ctl.remove_slave(machine_id)
-
-        for m_id in list(self._machines.keys()):
-            del self._machines[m_id]
-
-        # remove dynamically created bridges
-        for bridge in self._network_bridges.itervalues():
-            bridge.cleanup()
-        self._network_bridges = {}
-
-    def match_setup(self):
-        self.run_mode = "match_setup"
-        res = self._run_python_task()
-        return {"passed": True}
-
-    def run_recipe(self):
-        try:
-            res = self._run_recipe()
-        except Exception as exc:
-            logging.error("Recipe execution terminated by unexpected
exception")
-            raise
-        finally:
-            if self._packet_capture:
-                self._stop_packet_capture()
-                self._gather_capture_files()
-            self._cleanup_slaves()
-
-        return res
-
-    def prepare_test_env(self):
-        try:
-            self.provision_machines()
-            self.print_match_description()
-            if self.run_mode == "match_setup":
-                return True
-            self._prepare_network()
-            Task.ctl.init_hosts(self._machines)
-            return True
-        except (NoMatchError) as exc:
-            self._cleanup_slaves()
-            return False
-        except (KeyboardInterrupt, Exception) as exc:
-            msg = "Exception raised during configuration."
-            logging.error(msg)
-            self._cleanup_slaves()
-            raise
-
-    def _run_recipe(self):
-        overall_res = {"passed": True}
-
-        try:
-            self._res_serializer.add_task()
-            res = self._run_python_task()
-        except CommandException as exc:
-            logging.debug(exc)
-            overall_res["passed"] = False
-            overall_res["err_msg"] = "Command exception raised."
-
-        for machine in self._machines.itervalues():
-            machine.restore_system_config()
-
-        # task failed
-        if not res:
-            overall_res["passed"] = False
-            overall_res["err_msg"] = "At least one command failed."
-
-        return overall_res
-
-    def init_taskapi(self):
-        Task.ctl = Task.ControllerAPI(self)
-
-    def _run_python_task(self):
-        #backup of resource table
-        res_table_bkp = copy.deepcopy(self._resource_table)
-
-        cwd = os.getcwd()
-        task_path = self._recipe_path
-        name = os.path.basename(task_path.abs_path()).split(".")[0]
-        sys.path.append(os.path.dirname(task_path.resolve()))
-        os.chdir(os.path.dirname(task_path.resolve()))
-        imp.load_source(name, task_path.resolve())
-        os.chdir(cwd)
-        sys.path.remove(os.path.dirname(task_path.resolve()))
-
-        #restore resource table
-        self._resource_table = res_table_bkp
-
-        return Task.ctl._result
-
-    def _run_command(self, command):
-        logging.info("Executing command: [%s]", str_command(command))
-
-        if "desc" in command:
-            logging.info("Cmd description: %s", command["desc"])
-
-        if command["type"] == "ctl_wait":
-            sleep(command["seconds"])
-            cmd_res = {"passed": True,
-                       "res_header": "%-9s%ss" %
("ctl_wait",
-                                                   command["seconds"]),
-                       "msg": "",
-                       "res_data": None}
-            if self._res_serializer:
-                self._res_serializer.add_cmd_result(command, cmd_res)
-            return cmd_res
-
-        machine_id = command["host"]
-        machine = self._machines[machine_id]
-
-        try:
-            cmd_res = machine.run_command(command)
-        except Exception as exc:
-            cmd_res = {"passed": False,
-                       "res_data": {"Exception": str(exc)},
-                       "msg": "Exception raised.",
-                       "res_header": "EXCEPTION",
-                       "report": str(exc)}
-            raise
-        finally:
-            if self._res_serializer:
-                self._res_serializer.add_cmd_result(command, cmd_res)
-
-        if cmd_res["passed"]:
-            res_str = decorate_with_preset("PASS", "pass")
-        else:
-            res_str = decorate_with_preset("FAIL", "fail")
-        logging.info("Result: %s" % res_str)
-        if "report" in cmd_res and cmd_res["report"] !=
"":
-            logging.info("Result data:")
-            for line in cmd_res["report"].splitlines():
-                logging.info(4*" " + line)
-        if "msg" in cmd_res and cmd_res["msg"] != "":
-            logging.info("Status message from slave: \"%s\"" %
cmd_res["msg"])
-
-        return cmd_res
-
-    def _start_packet_capture(self):
-        logging.info("Starting packet capture")
-        for machine_id, machine in self._machines.iteritems():
-            capture_files = machine.start_packet_capture()
-            self._remote_capture_files[machine_id] = capture_files
-
-    def _stop_packet_capture(self):
-        logging.info("Stopping packet capture")
-        for machine_id, machine in self._machines.iteritems():
-            machine.stop_packet_capture()
-
-    # TODO: Move this function to logging
-    def _gather_capture_files(self):
-        logging_root = self._log_ctl.get_recipe_log_path()
-        logging_root = os.path.abspath(logging_root)
-        logging.info("Retrieving capture files from slaves")
-        for machine_id, machine in self._machines.iteritems():
-            slave_logging_dir = os.path.join(logging_root, machine_id + "/")
-            try:
-                os.mkdir(slave_logging_dir)
-            except OSError as err:
-                if err.errno != 17:
-                    msg = "Cannot access the logging directory %s" \
-                                            % slave_logging_dir
-                    raise NetTestError(msg)
-
-            capture_files = self._remote_capture_files[machine_id]
-            for if_id, remote_path in capture_files.iteritems():
-                filename = "%s.pcap" % if_id
-                local_path = os.path.join(slave_logging_dir, filename)
-                machine.copy_file_from_machine(remote_path, local_path)
-
-            logging.info("pcap files from machine %s stored at %s",
-                            machine_id, slave_logging_dir)
-
-    def _load_test_modules(self, dirs):
-        modules = {}
-        for dir_name in dirs:
-            files = os.listdir(dir_name)
-            for f in files:
-                test_path = os.path.abspath("%s/%s" % (dir_name, f))
-                if os.path.isfile(test_path):
-                    match = re.match("(.+)\.py$", f)
-                    if match:
-                        test_name = match.group(1)
-                        test_hash = md5sum(test_path)
-
-                        if test_name in modules:
-                            msg = "Overriding previously defined test '%s'
" \
-                                  "from %s with a different one located in " \
-                                  "%s" % (test_name, test_path,
-                                    modules[test_name]["path"])
-                            logging.warn(msg)
-
-                        modules[test_name] = {"path": test_path,
-                                              "hash": test_hash}
-        return modules
-
-    def _load_test_tools(self, dirs):
-        packages = {}
-        for dir_name in dirs:
-            files = os.listdir(dir_name)
-            for f in files:
-                pkg_path = os.path.abspath("%s/%s" % (dir_name, f))
-                if os.path.isdir(pkg_path):
-                    pkg_name = os.path.basename(pkg_path.rstrip("/"))
-                    pkg_hash = dir_md5sum(pkg_path)
-
-                    if pkg_name in packages:
-                        msg = "Overriding previously defined tools " \
-                              "package '%s' from %s with a different "
\
-                              "one located in %s" % (pkg_name, pkg_path,
-                                            packages[pkg_name]["path"])
-                        logging.warn(msg)
-
-                    packages[pkg_name] = {"path": pkg_path,
-                                           "hash": pkg_hash}
-        return packages
-
-    def _get_alias(self, alias):
-        if alias in self._defined_aliases:
-            return self._defined_aliases[alias]
-
-    def _get_aliases(self):
-        return self._defined_aliases
-
-class MessageDispatcher(ConnectionHandler):
-    def __init__(self, log_ctl):
-        super(MessageDispatcher, self).__init__()
-        self._log_ctl = log_ctl
-        self._machines = dict()
-
-    def add_slave(self, machine, connection):
-        machine_id = machine.get_id()
-        self._machines[machine_id] = machine
-        self.add_connection(machine_id, connection)
-
-    def send_message(self, machine_id, data):
-        soc = self.get_connection(machine_id)
-
-        if send_data(soc, data) == False:
-            msg = "Connection error from slave %s" % machine_id
-            raise NetTestError(msg)
-
-    def wait_for_result(self, machine_id):
-        wait = True
-        while wait:
-            connected_slaves = self._connection_mapping.keys()
-
-            messages = self.check_connections()
-
-            remaining_slaves = self._connection_mapping.keys()
-
-            for msg in messages:
-                if msg[1]["type"] == "result" and msg[0] ==
machine_id:
-                    wait = False
-                    result = msg[1]["result"]
-                else:
-                    self._process_message(msg)
-
-            if connected_slaves != remaining_slaves:
-                disconnected_slaves = set(connected_slaves) -\
-                                      set(remaining_slaves)
-                msg = "Slaves " + str(list(disconnected_slaves)) + \
-                      " disconnected from the controller."
-                raise NetTestError(msg)
-
-        return result
-
-    def _process_message(self, message):
-        if message[1]["type"] == "log":
-            record = message[1]["record"]
-            self._log_ctl.add_client_log(message[0], record)
-        elif message[1]["type"] == "result":
-            msg = "Recieved result message from different slave %s" %
message[0]
-            logging.debug(msg)
-        elif message[1]["type"] == "if_update":
-            machine = self._machines[message[0]]
-            machine.interface_update(message[1])
-        elif message[1]["type"] == "if_deleted":
-            machine = self._machines[message[0]]
-            machine.dev_db_delete(message[1])
-        elif message[1]["type"] == "exception":
-            msg = "Slave %s: %s" % (message[0],
message[1]["Exception"])
-            raise CommandException(msg)
-        elif message[1]["type"] == "error":
-            msg = "Recieved an error message from slave %s: %s" %\
-                    (message[0], message[1]["err"])
-            raise CommandException(msg)
-        else:
-            msg = "Unknown message type: %s" % message[1]["type"]
-            raise NetTestError(msg)
-
-    def disconnect_slave(self, machine_id):
-        soc = self.get_connection(machine_id)
-        self.remove_connection(soc)
-        del self._machines[machine_id]
diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py
deleted file mode 100644
index 09233a7..0000000
--- a/lnst/Controller/RecipeParser.py
+++ /dev/null
@@ -1,572 +0,0 @@
-"""
-This module defines RecipeParser class useful to parse xml recipes
-
-Copyright 2013 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import os
-from lnst.Common.Path import Path
-from lnst.Controller.XmlParser import XmlParser
-from lnst.Controller.XmlProcessing import XmlProcessingError, XmlData
-from lnst.Controller.XmlProcessing import XmlCollection
-
-class RecipeError(XmlProcessingError):
-    pass
-
-class RecipeParser(XmlParser):
-    def __init__(self, recipe_path):
-        recipe_path = Path(None, recipe_path).abs_path()
-        super(RecipeParser, self).__init__("schema-recipe.rng", recipe_path)
-
-    def _process(self, lnst_recipe):
-        recipe = XmlData(lnst_recipe)
-
-        # machines
-        machines_tag = lnst_recipe.find("network")
-        if machines_tag is not None:
-            machines = recipe["machines"] = XmlCollection(machines_tag)
-            for machine_tag in machines_tag:
-                machines.append(self._process_machine(machine_tag))
-
-        # tasks
-        tasks = recipe["tasks"] = XmlCollection()
-        task_tags = lnst_recipe.findall("task")
-        for task_tag in task_tags:
-            tasks.append(self._process_task(task_tag))
-
-        return recipe
-
-    def _process_machine(self, machine_tag):
-        machine = XmlData(machine_tag)
-        machine["id"] = self._get_attribute(machine_tag, "id")
-
-        # params
-        params_tag = machine_tag.find("params")
-        params = self._process_params(params_tag)
-        if len(params) > 0:
-            machine["params"] = params
-
-        # interfaces
-        interfaces_tag = machine_tag.find("interfaces")
-        if interfaces_tag is not None and len(interfaces_tag) > 0:
-            machine["interfaces"] = XmlCollection(interfaces_tag)
-
-            lo_netns = []
-            unique_ids = []
-            for interface_tag in interfaces_tag:
-                interfaces = self._process_interface(interface_tag)
-
-                for interface in interfaces:
-                    if interface['id'] in unique_ids:
-                        msg = "Interface with ID \"%s\" has already been
"\
-                              "defined for this machine." %
interface['id']
-                        raise RecipeError(msg, interface_tag)
-                    else:
-                        unique_ids.append(interface['id'])
-
-                    if interface['type'] == 'lo':
-                        if interface['netns'] in lo_netns:
-                            msg = "Only one loopback device per netns "\
-                                  "is allowed."
-                            raise RecipeError(msg, interface_tag)
-                        else:
-                            lo_netns.append(interface['netns'])
-                    elif interface['type'] == "ovs_bridge":
-                        ovs_conf = interface["ovs_conf"]
-                        for i in ovs_conf["tunnels"] +
ovs_conf["internals"]:
-                            if i['id'] in unique_ids:
-                                msg = "Interface with ID \"%s\" has
already "\
-                                      "been defined for this machine." %\
-                                      i['id']
-                                raise RecipeError(msg, i)
-                            else:
-                                unique_ids.append(i['id'])
-
-                machine["interfaces"].extend(interfaces)
-
-        return machine
-
-    def _process_params(self, params_tag):
-        params = XmlCollection(params_tag)
-        if params_tag is not None:
-            for param_tag in params_tag:
-                param = XmlData(param_tag)
-                param["name"] = self._get_attribute(param_tag,
"name")
-                param["value"] = self._get_attribute(param_tag,
"value")
-                params.append(param)
-
-        return params
-
-    def _process_interface(self, iface_tag):
-        iface = XmlData(iface_tag)
-        iface["type"] = iface_tag.tag
-
-        if iface["type"] == "veth_pair":
-            iface = self._process_interface(iface_tag[0])[0]
-            iface2 = self._process_interface(iface_tag[1])[0]
-
-            iface["peer"] = iface2["id"]
-            iface2["peer"] = iface["id"]
-
-            return [iface, iface2]
-
-        iface["id"] = self._get_attribute(iface_tag, "id")
-
-        iface["netns"] = None
-        if self._has_attribute(iface_tag, "netns"):
-            iface["netns"] = self._get_attribute(iface_tag, "netns")
-
-        # netem
-        netem_tag = iface_tag.find("netem")
-        if netem_tag is not None:
-            iface["netem"] = self._process_netem(netem_tag)
-
-        # params
-        params_tag = iface_tag.find("params")
-        params = self._process_params(params_tag)
-        if len(params) > 0:
-            iface["params"] = params
-
-        # addresses
-        addresses_tag = iface_tag.find("addresses")
-        addrs = self._process_addresses(addresses_tag)
-        iface["addresses"] = addrs
-
-        if iface["type"] == "eth":
-            iface["network"] = self._get_attribute(iface_tag,
"label")
-        elif iface["type"] in ["bond", "bridge",
"macvlan", "team"]:
-            # slaves
-            slaves_tag = iface_tag.find("slaves")
-            if slaves_tag is not None and len(slaves_tag) > 0:
-                iface["slaves"] = XmlCollection(slaves_tag)
-                for slave_tag in slaves_tag:
-                    slave = XmlData(slave_tag)
-                    slave["id"] = self._get_attribute(slave_tag,
"id")
-
-                    # slave options
-                    opts_tag = slave_tag.find("options")
-                    opts = self._process_options(opts_tag)
-                    if len(opts) > 0:
-                        slave["options"] = opts
-
-                    iface["slaves"].append(slave)
-
-            # interface options
-            opts_tag = iface_tag.find("options")
-            opts = self._process_options(opts_tag)
-            if len(opts) > 0:
-                iface["options"] = opts
-        elif iface["type"] in ["vti", "vti6"]:
-            # interface options
-            opts_tag = iface_tag.find("options")
-            opts = self._process_options(opts_tag)
-            iface["options"] = opts
-        elif iface["type"] in ["vlan"]:
-            # real_dev of the VLAN interface
-            slaves_tag = iface_tag.find("slaves")
-            if slaves_tag is None or len(slaves_tag) != 1:
-                msg = "VLAN '%s' need exactly one slave definition."\
-                        % iface["id"]
-                raise RecipeError(msg, iface_tag)
-
-            iface["slaves"] = XmlCollection(slaves_tag)
-
-            slave_tag = slaves_tag[0]
-            slave = XmlData(slave_tag)
-            slave["id"] = self._get_attribute(slave_tag, "id")
-
-            iface["slaves"].append(slave)
-
-            # interface options
-            opts_tag = iface_tag.find("options")
-            opts = self._process_options(opts_tag)
-            if len(opts) > 0:
-                iface["options"] = opts
-        elif iface["type"] in ["vxlan"]:
-            # real_dev of the VXLAN interface
-            slaves_tag = iface_tag.find("slaves")
-            if slaves_tag is not  None and len(slaves_tag) > 1:
-                msg = "VXLAN '%s' needs one or no slave definition."\
-                        % iface["id"]
-                raise RecipeError(msg, iface_tag)
-
-            if slaves_tag:
-                iface["slaves"] = XmlCollection(slaves_tag)
-                slave_tag = slaves_tag[0]
-                slave = XmlData(slave_tag)
-                slave["id"] = self._get_attribute(slave_tag, "id")
-                iface["slaves"].append(slave)
-
-            # interface options
-            opts_tag = iface_tag.find("options")
-            opts = self._process_options(opts_tag)
-            if len(opts) > 0:
-                iface["options"] = opts
-        elif iface["type"] == "ovs_bridge":
-            slaves_tag = iface_tag.find("slaves")
-            iface["slaves"] = XmlCollection(slaves_tag)
-            ovsb_slaves = []
-
-            iface["ovs_conf"] = XmlData(slaves_tag)
-            if slaves_tag is not None:
-                for slave_tag in slaves_tag:
-                    slave = XmlData(slave_tag)
-                    slave["id"] = str(self._get_attribute(slave_tag,
"id"))
-                    ovsb_slaves.append(slave["id"])
-
-                    opts_tag = slave_tag.find("options")
-                    opts = self._process_options(opts_tag)
-                    slave["options"] = opts
-
-                    iface["slaves"].append(slave)
-
-            vlan_elems = iface_tag.findall("vlan")
-            vlans = iface["ovs_conf"]["vlans"] = XmlData(slaves_tag)
-            for vlan in vlan_elems:
-                vlan_tag = str(self._get_attribute(vlan, "tag"))
-                if vlan_tag in vlans:
-                    msg = "VLAN '%s' already defined for "\
-                          "this ovs_bridge." % vlan_tag
-                    raise RecipeError(msg, vlan)
-
-                vlans[vlan_tag] = XmlData(vlan)
-                vlans[vlan_tag]["slaves"] = XmlCollection(vlan)
-                vlan_slaves = vlans[vlan_tag]["slaves"]
-
-                slaves_tag = vlan.find("slaves")
-                for slave_tag in slaves_tag:
-                    slave_id = str(self._get_attribute(slave_tag, "id"))
-                    if slave_id not in ovsb_slaves:
-                        msg = "No port with id '%s' defined for "\
-                              "this ovs_bridge." % slave_id
-                        raise RecipeError(msg, slave_tag)
-
-                    if slave_id in vlan_slaves:
-                        msg = "Port '%s' already a member of vlan %s"\
-                              % (slave_id, vlan_tag)
-                        raise RecipeError(msg, slave_tag)
-                    else:
-                        vlan_slaves.append(slave_id)
-
-            bonded_slaves = {}
-            bond_elems = iface_tag.findall("bond")
-            bonds = iface["ovs_conf"]["bonds"] = XmlData(slaves_tag)
-            for bond_tag in bond_elems:
-                bond_id = str(self._get_attribute(bond_tag, "id"))
-                if bond_id in bonds:
-                    msg = "Bond with id '%s' already defined for "\
-                          "this ovs_bridge." % bond_id
-                    raise RecipeError(msg, bond_tag)
-                bonds[bond_id] = XmlData(bond_tag)
-                bond_slaves = bonds[bond_id]["slaves"] =
XmlCollection(bond_tag)
-
-                slaves_tag = bond_tag.find("slaves")
-                for slave_tag in slaves_tag:
-                    slave_id = str(self._get_attribute(slave_tag, "id"))
-                    if slave_id not in ovsb_slaves:
-                        msg = "No port with id '%s' defined for "\
-                              "this ovs_bridge." % slave_id
-                        raise RecipeError(msg, slave_tag)
-
-                    if slave_id in bonded_slaves:
-                        msg = "Port with id '%s' already in bond with id
'%s'"\
-                              % (slave_id, bonded_slaves[slave_id])
-                        raise RecipeError(msg, slave_tag)
-                    else:
-                        bonded_slaves[slave_id] = bond_id
-
-                    bond_slaves.append(slave_id)
-
-                opts_tag = bond_tag.find("options")
-                opts = self._process_options(opts_tag)
-                if len(opts) > 0:
-                    bonds[bond_id]["options"] = opts
-
-            unique_ids = []
-            tunnels = iface["ovs_conf"]["tunnels"] =
XmlCollection(slaves_tag)
-            tunnel_elems = iface_tag.findall("tunnel")
-            for tunnel_elem in tunnel_elems:
-                tunnels.append(XmlData(tunnel_elem))
-                tunnel = tunnels[-1]
-                tunnel["id"] = str(self._get_attribute(tunnel_elem,
"id"))
-                if tunnel["id"] in unique_ids:
-                    msg = "Tunnel with id '%s' already defined for "\
-                          "this ovs_bridge." % tunnel["id"]
-                    raise RecipeError(msg, tunnel_elem)
-                else:
-                    unique_ids.append(tunnel["id"])
-
-                t = str(self._get_attribute(tunnel_elem, "type"))
-                tunnel["type"] = t
-
-                opts_elem = tunnel_elem.find("options")
-                opts = self._process_options(opts_elem)
-                if len(opts) > 0:
-                    tunnel["options"] = opts
-
-                # addresses
-                addresses_tag = tunnel_elem.find("addresses")
-                addrs = self._process_addresses(addresses_tag)
-                tunnel["addresses"] = addrs
-
-            iface["ovs_conf"]["internals"] =
XmlCollection(slaves_tag)
-            internals = iface["ovs_conf"]["internals"]
-            internal_elems = iface_tag.findall("internal")
-            for internal_elem in internal_elems:
-                internals.append(XmlData(internal_elem))
-                internal = internals[-1]
-                internal["id"] = str(self._get_attribute(internal_elem,
"id"))
-                if internal["id"] in unique_ids:
-                    msg = "Internal id '%s' already defined for "\
-                          "this ovs_bridge." % internal["id"]
-                    raise RecipeError(msg, internal_elem)
-                else:
-                    unique_ids.append(internal["id"])
-
-                opts_elem = internal_elem.find("options")
-                opts = self._process_options(opts_elem)
-                if len(opts) > 0:
-                    internal["options"] = opts
-
-                # addresses
-                addresses_tag = internal_elem.find("addresses")
-                addrs = self._process_addresses(addresses_tag)
-                internal["addresses"] = addrs
-
-            iface["ovs_conf"]["flow_entries"] =
XmlCollection(slaves_tag)
-            flow_entries = iface["ovs_conf"]["flow_entries"]
-            flow_elems = iface_tag.findall("flow_entries")
-            if len(flow_elems) == 1:
-                entries = flow_elems[0].findall("entry")
-                for entry in entries:
-                    if self._has_attribute(entry, "value"):
-                        flow_entries.append(self._get_attribute(entry,
-                                                                "value"))
-                    else:
-                        flow_entries.append(self._get_content(entry))
-
-        return [iface]
-
-    def _process_addresses(self, addresses_tag):
-        addresses = XmlCollection(addresses_tag)
-        if addresses_tag is not None and len(addresses_tag) > 0:
-            for addr_tag in addresses_tag:
-                if self._has_attribute(addr_tag, "value"):
-                    addr = self._get_attribute(addr_tag, "value")
-                else:
-                    addr = self._get_content(addr_tag)
-                addresses.append(addr)
-        return addresses
-
-    def _process_options(self, opts_tag):
-        options = XmlCollection(opts_tag)
-        if opts_tag is not None:
-            for opt_tag in opts_tag:
-                opt = XmlData(opt_tag)
-                opt["name"] = self._get_attribute(opt_tag, "name")
-                if self._has_attribute(opt_tag, "value"):
-                    opt["value"] = self._get_attribute(opt_tag,
"value")
-                else:
-                    opt["value"] = self._get_content(opt_tag)
-                options.append(opt)
-
-        return options
-
-    def _validate_netem(self, options, netem_op, netem_tag):
-        if netem_op == "delay":
-            valid = False
-            jitter = False
-            correlation = False
-            distribution = False
-            valid_distributions = ["normal", "uniform",
"pareto", "paretonormal"]
-            for opt in options:
-                if "time" in opt.values():
-                    valid = True
-                elif "distribution" in opt.values():
-                    if opt["value"] not in valid_distributions:
-                        raise RecipeError("netem: invalid distribution type",
netem_tag)
-                    else:
-                        distribution = True
-                elif "jitter" in opt.values():
-                    jitter =  True
-                elif "correlation" in opt.values():
-                    correlation = True
-            if not jitter:
-                if correlation or distribution:
-                    raise RecipeError("netem: jitter option is mandatory when using
<correlation> or <distribution>", netem_tag)
-            if not valid:
-                raise RecipeError("netem: time option is mandatory for
<delay>", netem_tag)
-        elif netem_op == "loss":
-            for opt in options:
-                if "percent" in opt.values():
-                    return
-            raise RecipeError("netem: percent option is mandatory for
<loss>", netem_tag)
-        elif netem_op == "duplication":
-            for opt in options:
-                if "percent" in opt.values():
-                    return
-            raise RecipeError("netem: percent option is mandatory for
<duplication>", netem_tag)
-        elif netem_op == "corrupt":
-            for opt in options:
-                if "percent" in opt.values():
-                    return
-            raise RecipeError("netem: percent option is mandatory for
<corrupt>", netem_tag)
-        elif netem_op == "reordering":
-            for opt in options:
-                if "percent" in opt.values():
-                    return
-            raise RecipeError("netem: percent option is mandatory for
<reordering>", netem_tag)
-
-    def _process_netem(self, netem_tag):
-        interface = XmlData(netem_tag)
-        # params
-        for netem_op in ["delay", "loss", "duplication",
"corrupt", "reordering"]:
-            netem_op_tag = netem_tag.find(netem_op)
-            if netem_op_tag is not None:
-                options_tag = netem_op_tag.find("options")
-                options = self._process_options(options_tag)
-                if len(options) > 0:
-                    self._validate_netem(options, netem_op, netem_tag)
-                    interface[netem_op] = options
-        return interface
-
-    def _process_task(self, task_tag):
-        task = XmlData(task_tag)
-
-        if self._has_attribute(task_tag, "quit_on_fail"):
-            task["quit_on_fail"] = self._get_attribute(task_tag,
"quit_on_fail")
-
-        if self._has_attribute(task_tag, "module_dir"):
-            base_dir = os.path.dirname(task_tag.attrib["__file"])
-            dir_path = str(self._get_attribute(task_tag, "module_dir"))
-            exp_path = os.path.expanduser(dir_path)
-            abs_path = os.path.join(base_dir, exp_path)
-            norm_path = os.path.normpath(abs_path)
-            task["module_dir"] = norm_path
-
-        if self._has_attribute(task_tag, "tools_dir"):
-            base_dir = os.path.dirname(task_tag.attrib["__file"])
-            dir_path = str(self._get_attribute(task_tag, "tools_dir"))
-            exp_path = os.path.expanduser(dir_path)
-            abs_path = os.path.join(base_dir, exp_path)
-            norm_path = os.path.normpath(abs_path)
-            task["tools_dir"] = norm_path
-
-        if self._has_attribute(task_tag, "python"):
-            task["python"] = self._get_attribute(task_tag, "python")
-            return task
-
-        if len(task_tag) > 0:
-            task["commands"] = XmlCollection(task_tag)
-            for cmd_tag in task_tag:
-                if cmd_tag.tag == "run":
-                    cmd = self._process_run_cmd(cmd_tag)
-                elif cmd_tag.tag == "config":
-                    cmd = self._process_config_cmd(cmd_tag)
-                elif cmd_tag.tag == "ctl_wait":
-                    cmd = self._process_ctl_wait_cmd(cmd_tag)
-                elif cmd_tag.tag in ["wait", "intr",
"kill"]:
-                    cmd = self._process_signal_cmd(cmd_tag)
-                else:
-                    msg = "Unknown command '%s'." % cmd_tag.tag
-                    raise RecipeError(msg, cmd_tag)
-
-                task["commands"].append(cmd)
-
-        return task
-
-    def _process_run_cmd(self, cmd_tag):
-        cmd = XmlData(cmd_tag)
-        cmd["host"] = self._get_attribute(cmd_tag, "host")
-
-        cmd["netns"] = None
-        if self._has_attribute(cmd_tag, "netns"):
-            cmd["netns"] = self._get_attribute(cmd_tag, "netns")
-
-        has_module = self._has_attribute(cmd_tag, "module")
-        has_command = self._has_attribute(cmd_tag, "command")
-        has_from = self._has_attribute(cmd_tag, "from")
-
-        if (has_module and has_command) or (has_module and has_from):
-            msg = "Invalid combination of attributes."
-            raise RecipeError(msg, cmd)
-
-        if has_module:
-            cmd["type"] = "test"
-            cmd["module"] = self._get_attribute(cmd_tag, "module")
-
-            # options
-            opts_tag = cmd_tag.find("options")
-            opts = self._process_options(opts_tag)
-            if len(opts) > 0:
-                cmd["options"] = opts
-        elif has_command:
-            cmd["type"] = "exec"
-            cmd["command"] = self._get_attribute(cmd_tag, "command")
-
-            if self._has_attribute(cmd_tag, "from"):
-                cmd["from"] = self._get_attribute(cmd_tag, "from")
-
-        if self._has_attribute(cmd_tag, "bg_id"):
-            cmd["bg_id"] = self._get_attribute(cmd_tag, "bg_id")
-
-        if self._has_attribute(cmd_tag, "timeout"):
-            cmd["timeout"] = self._get_attribute(cmd_tag, "timeout")
-
-        if self._has_attribute(cmd_tag, "expect"):
-            cmd["expect"] = self._get_attribute(cmd_tag, "expect")
-
-        return cmd
-
-    def _process_config_cmd(self, cmd_tag):
-        cmd = XmlData(cmd_tag)
-        cmd["type"] = "config"
-        cmd["host"] = self._get_attribute(cmd_tag, "host")
-
-        cmd["netns"] = None
-        if self._has_attribute(cmd_tag, "netns"):
-            cmd["netns"] = self._get_attribute(cmd_tag, "netns")
-
-        if self._has_attribute(cmd_tag, "persistent"):
-            cmd["persistent"] = self._get_attribute(cmd_tag,
"persistent")
-
-        # inline option
-        if self._has_attribute(cmd_tag, "option"):
-            cmd["options"] = XmlCollection(cmd_tag)
-            if self._has_attribute(cmd_tag, "value"):
-                opt = XmlData(cmd_tag)
-                opt["name"] = self._get_attribute(cmd_tag, "option")
-                opt["value"] = self._get_attribute(cmd_tag, "value")
-
-                cmd["options"] = XmlCollection(cmd_tag)
-                cmd["options"].append(opt)
-            else:
-                raise RecipeError("Missing option value.", cmd)
-        else:
-            # options
-            opts_tag = cmd_tag.find("options")
-            opts = self._process_options(opts_tag)
-            if len(opts) > 0:
-                cmd["options"] = opts
-
-        return cmd
-
-    def _process_ctl_wait_cmd(self, cmd_tag):
-        cmd = XmlData(cmd_tag)
-        cmd["type"] = "ctl_wait"
-        cmd["seconds"] = self._get_attribute(cmd_tag, "seconds")
-        return cmd
-
-    def _process_signal_cmd(self, cmd_tag):
-        cmd = XmlData(cmd_tag)
-        cmd["type"] = cmd_tag.tag
-        cmd["host"] = self._get_attribute(cmd_tag, "host")
-        cmd["bg_id"] = self._get_attribute(cmd_tag, "bg_id")
-        cmd["netns"] = None
-        return cmd
diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py
deleted file mode 100644
index 13cc34e..0000000
--- a/lnst/Controller/SlavePool.py
+++ /dev/null
@@ -1,648 +0,0 @@
-"""
-This module contains implementaion of SlavePool class that
-can be used to maintain a cluster of test machines.
-
-These machines can be provisioned and used in test recipes.
-
-Copyright 2012 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import logging
-import os
-import re
-import socket
-import select
-from lnst.Common.Config import lnst_config
-from lnst.Common.NetUtils import normalize_hwaddr
-from lnst.Controller.Machine import Machine
-from lnst.Controller.SlaveMachineParser import SlaveMachineParser
-from lnst.Controller.SlaveMachineParser import SlaveMachineError
-from lnst.Common.Colours import decorate_with_preset
-from lnst.Common.Utils import check_process_running
-
-class SlavePool:
-    """
-    This class is responsible for managing test machines that
-    are available at the controler and can be used for testing.
-    """
-    def __init__(self, pools, pool_checks=True):
-        self._map = {}
-        self._pools = {}
-        self._pool = {}
-
-        self._machine_matches = []
-        self._network_matches = []
-
-        self._allow_virt = lnst_config.get_option("environment",
-                                                  "allow_virtual")
-        self._allow_virt &= check_process_running("libvirtd")
-        self._pool_checks = pool_checks
-
-        self._mapper = SetupMapper()
-        self._mreqs = None
-
-        logging.info("Checking machine pool availability.")
-        for pool_name, pool_dir in pools.items():
-            self._pools[pool_name] = {}
-            self.add_dir(pool_name, pool_dir)
-            if len(self._pools[pool_name]) == 0:
-                del self._pools[pool_name]
-
-        self._mapper.set_pools(self._pools)
-        logging.info("Finished loading pools.")
-
-    def get_pools(self):
-        return self._pools
-
-    def add_dir(self, pool_name, dir_path):
-        logging.info("Processing pool '%s', directory '%s'" %
(pool_name,
-                                                               dir_path))
-        pool = self._pools[pool_name]
-
-        try:
-            dentries = os.listdir(dir_path)
-        except OSError:
-            logging.warn("Directory '%s' does not exist for pool
'%s'" %
-                                                                  (dir_path,
-                                                                   pool_name))
-            return
-
-        for dirent in dentries:
-            m_id, m = self.add_file(pool_name, dir_path, dirent)
-            if m_id != None and m != None:
-                pool[m_id] = m
-
-        if len(pool) == 0:
-            logging.warn("No machines found in pool '%s', directory
'%s'" %
-                                                                   (pool_name,
-                                                                    dir_path))
-
-        max_len = 0
-        for m_id in pool.keys():
-            if len(m_id) > max_len:
-                max_len = len(m_id)
-
-        if self._pool_checks:
-            check_sockets = {}
-            for m_id, m in sorted(pool.iteritems()):
-                hostname = m["params"]["hostname"]
-                if "rpc_port" in m["params"]:
-                    port = m["params"]["rpc_port"]
-                else:
-                    port = lnst_config.get_option('environment',
'rpcport')
-
-                logging.debug("Querying machine '%s': %s:%s" %\
-                                                (m_id, hostname, port))
-
-                s = socket.socket()
-                s.settimeout(0)
-                try:
-                    s.connect((hostname, port))
-                except:
-                    pass
-                check_sockets[s] = m_id
-
-            while len(check_sockets) > 0:
-                rl, wl, el = select.select([], check_sockets.keys(), [])
-                for s in wl:
-                    err = s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
-                    m_id = check_sockets[s]
-                    if err == 0:
-                        pool[m_id]["available"] = True
-                        s.shutdown(socket.SHUT_RDWR)
-                        s.close()
-                        del check_sockets[s]
-                    else:
-                        pool[m_id]["available"] = False
-                        s.close()
-                        del check_sockets[s]
-        else:
-            for m_id in pool.keys():
-                pool[m_id]["available"] = True
-
-        for m_id in sorted(list(pool.keys())):
-            m = pool[m_id]
-            if m["available"]:
-                if 'libvirt_domain' in m['params']:
-                    libvirt_msg = "   libvirt_domain: %s" %\
-                                        m['params']['libvirt_domain']
-                else:
-                    libvirt_msg = ""
-                msg = "%s%s [%s] %s" % (m_id, (max_len - len(m_id)) * "
",
-                                        decorate_with_preset("UP",
"pass"),
-                                        libvirt_msg)
-            else:
-                msg = "%s%s [%s]" % (m_id, (max_len - len(m_id)) * "
",
-                                     decorate_with_preset("DOWN",
"fail"))
-                del pool[m_id]
-
-            logging.info(msg)
-
-    def add_file(self, pool_name, dir_path, dirent):
-        filepath = dir_path + "/" + dirent
-        pool = self._pools[pool_name]
-        if os.path.isfile(filepath) and re.search("\.xml$", filepath, re.I):
-            dirname, basename = os.path.split(filepath)
-            m_id = re.sub("\.[xX][mM][lL]$", "", basename)
-
-            parser = SlaveMachineParser(filepath)
-            xml_data = parser.parse()
-            machine_spec = self._process_machine_xml_data(m_id, xml_data)
-
-            if 'libvirt_domain' in machine_spec['params'] and \
-               not self._allow_virt:
-                   logging.debug("libvirtd not running or allow_virtual "\
-                                 "disabled. Removing libvirt_domain from "\
-                                 "machine '%s'" % m_id)
-                   del machine_spec['params']['libvirt_domain']
-
-            # Check if there isn't any machine with the same
-            # hostname or libvirt_domain already in the pool
-            for pm_id, m in pool.iteritems():
-                pm = m["params"]
-                rm = machine_spec["params"]
-                if pm["hostname"] == rm["hostname"]:
-                    msg = "You have the same machine listed twice in " \
-                          "your pool ('%s' and '%s')." % (m_id,
pm_id)
-                    raise SlaveMachineError(msg)
-
-                if "libvirt_domain" in rm and "libvirt_domain" in pm
and \
-                   pm["libvirt_domain"] == rm["libvirt_domain"]:
-                    msg = "You have the same libvirt_domain listed twice in "
\
-                          "your pool ('%s' and '%s')." % (m_id,
pm_id)
-                    raise SlaveMachineError(msg)
-
-            return (m_id, machine_spec)
-        return (None, None)
-
-    def _process_machine_xml_data(self, m_id, machine_xml_data):
-        machine_spec = {"interfaces": {}, "params":{},
"security": {}}
-
-        # process parameters
-        if "params" in machine_xml_data:
-            for param in machine_xml_data["params"]:
-                name = str(param["name"])
-                value = str(param["value"])
-
-                if name == "rpc_port":
-                    machine_spec["params"][name] = int(value)
-                else:
-                    machine_spec["params"][name] = value
-
-        mandatory_params = ["hostname"]
-        for p in mandatory_params:
-            if p not in machine_spec["params"]:
-                msg = "Mandatory parameter '%s' missing for machine
%s." \
-                        % (p, m_id)
-                raise SlaveMachineError(msg, machine_xml_data["params"])
-
-        # process interfaces
-        if "interfaces" in machine_xml_data:
-            for iface in machine_xml_data["interfaces"]:
-                if_id = iface["id"]
-                iface_spec = self._process_iface_xml_data(m_id, iface)
-
-                if if_id not in machine_spec["interfaces"]:
-                    machine_spec["interfaces"][if_id] = iface_spec
-                else:
-                    msg = "Duplicate interface id '%s'." % if_id
-                    raise SlaveMachineError(msg, iface)
-        else:
-            if "libvirt_domain" not in machine_spec["params"]:
-                msg = "Machine '%s' has no testing interfaces. " \
-                      "This setup is supported only for virtual slaves." \
-                      % m_id
-                raise SlaveMachineError(msg, machine_xml_data)
-
-        machine_spec["security"] = machine_xml_data["security"]
-
-        return machine_spec
-
-    def _process_iface_xml_data(self, m_id, iface):
-        if_id = iface["id"]
-        iface_spec = {"params": {}}
-        iface_spec["network"] = iface["network"]
-
-        for param in iface["params"]:
-            name = str(param["name"])
-            value = str(param["value"])
-
-            if name == "hwaddr":
-                iface_spec["params"][name] = normalize_hwaddr(value)
-            else:
-                iface_spec["params"][name] = value
-
-        mandatory_params = ["hwaddr"]
-        for p in mandatory_params:
-            if p not in iface_spec["params"]:
-                msg = "Mandatory parameter '%s' missing for machine %s,
" \
-                      "interface '%s'." % (p, m_id, if_id)
-                raise SlaveMachineError(msg, iface["params"])
-
-        return iface_spec
-
-    def set_machine_requirements(self, mreqs):
-        self._mreqs = mreqs
-        self._mapper.set_requirements(mreqs)
-        self._mapper.reset_match_state()
-
-    def provision_machines(self, machines):
-        """
-        This method will try to map a dictionary of machines'
-        requirements to a pool of machines that is available to
-        this instance.
-
-        :param templates: Setup request (dict of required machines)
-        :type templates: dict
-
-        :return: XML machineconfigs of requested machines
-        :rtype: dict
-        """
-        mapper = self._mapper
-        logging.info("Matching machines, without virtuals.")
-        res = mapper.match()
-
-        if not res and not mapper.get_virtual() and self._allow_virt:
-            logging.info("Match failed for normal machines, falling back "\
-                         "to matching virtual machines.")
-            mapper.set_virtual(self._allow_virt)
-            mapper.reset_match_state()
-            res = mapper.match()
-
-        if res:
-            self._map = mapper.get_mapping()
-        else:
-            self._map = {}
-
-        if self._map == {}:
-            self._pool = {}
-            return False
-        else:
-            self._pool = self._pools[self._map["pool_name"]]
-
-        if self._map["virtual"]:
-            mreqs = self._mreqs
-            for m_id in self._map["machines"]:
-                machines[m_id] = self._prepare_virtual_slave(m_id, mreqs[m_id])
-        else:
-            for m_id in self._map["machines"]:
-                machines[m_id] = self._get_mapped_slave(m_id)
-
-        return True
-
-    def is_setup_virtual(self):
-        return self._map["virtual"]
-
-    def get_match(self):
-        return self._map
-
-    def _get_machine_mapping(self, m_id):
-        return self._map["machines"][m_id]["target"]
-
-    def _get_interface_mapping(self, m_id, if_id):
-        return self._map["machines"][m_id]["interfaces"][if_id]
-
-    def _get_network_mapping(self, net_id):
-        return self._map["networks"][net_id]
-
-    def _get_mapped_slave(self, tm_id):
-        pm_id = self._get_machine_mapping(tm_id)
-        pm = self._pool[pm_id]
-
-        hostname = pm["params"]["hostname"]
-
-        rpcport = None
-        if "rpc_port" in pm["params"]:
-            rpcport = pm["params"]["rpc_port"]
-
-        machine = Machine(tm_id, hostname, None, rpcport, pm["security"])
-
-        used = []
-        if_map = self._map["machines"][tm_id]["interfaces"]
-        for t_if, p_if in if_map.iteritems():
-            pool_id = p_if["target"]
-            used.append(pool_id)
-            if_data = pm["interfaces"][pool_id]
-
-            iface = machine.new_static_interface(t_if, "eth")
-            iface.set_hwaddr(if_data["params"]["hwaddr"])
-
-            for t_net, p_net in self._map["networks"].iteritems():
-                if pm["interfaces"][pool_id]["network"] == p_net:
-                    iface.set_network(t_net)
-                    break
-
-        for if_id, if_data in pm["interfaces"].iteritems():
-            if if_id not in used:
-                iface = machine.new_unused_interface("eth")
-                iface.set_hwaddr(if_data["params"]["hwaddr"])
-                iface.set_network(None)
-
-        return machine
-
-    def _prepare_virtual_slave(self, tm_id, tm):
-        pm_id = self._get_machine_mapping(tm_id)
-        pm = self._pool[pm_id]
-
-        hostname = pm["params"]["hostname"]
-        libvirt_domain = pm["params"]["libvirt_domain"]
-
-        rpcport = None
-        if "rpc_port" in pm["params"]:
-            rpcport = pm["params"]["rpc_port"]
-
-        machine = Machine(tm_id, hostname, libvirt_domain, rpcport,
-                          pm["security"])
-
-        # make all the existing unused
-        for if_id, if_data in pm["interfaces"].iteritems():
-            iface = machine.new_unused_interface("eth")
-            iface.set_hwaddr(if_data["params"]["hwaddr"])
-            iface.set_network(None)
-
-        # add all the other devices
-        for if_id, if_data in tm["interfaces"].iteritems():
-            iface = machine.new_virtual_interface(if_id, "eth")
-            iface.set_network(if_data["network"])
-            if "hwaddr" in if_data["params"]:
-                iface.set_hwaddr(if_data["params"]["hwaddr"])
-            if "driver" in if_data["params"]:
-                iface.set_driver(if_data["params"]["driver"])
-
-        return machine
-
-class MapperError(Exception):
-    pass
-
-class SetupMapper(object):
-    def __init__(self):
-        self._pools = {}
-        self._pool_stack = []
-        self._pool = {}
-        self._pool_name = None
-        self._mreqs = {}
-        self._unmatched_req_machines = []
-        self._matched_pool_machines = []
-        self._machine_stack = []
-        self._net_label_mapping = {}
-        self._virtual_matching = False
-
-    def set_requirements(self, mreqs):
-        self._mreqs = mreqs
-
-    def set_pools(self, pools):
-        self._pools = pools
-
-    def set_virtual(self, virt_value):
-        self._virtual_matching = virt_value
-
-        for m_id, m in self._mreqs.iteritems():
-            for if_id, interface in m["interfaces"].iteritems():
-                if "params" in interface:
-                    for name, val in interface["params"].iteritems():
-                        if name not in ["hwaddr", "driver"]:
-                            msg = "Dynamically created interfaces "\
-                                  "only support the 'hwaddr' and
'driver' "\
-                                  "option. '%s=%s' found on machine
'%s' "\
-                                  "interface '%s'" % (name, val,
-                                                      m_id, if_id)
-                            raise MapperError(msg)
-
-    def get_virtual(self):
-        return self._virtual_matching
-
-    def reset_match_state(self):
-        self._net_label_mapping = {}
-        self._machine_stack = []
-        self._unmatched_req_machines = sorted(self._mreqs.keys(), reverse=True)
-
-        self._pool_stack = list(self._pools.keys())
-        if len(self._pool_stack) > 0:
-            self._pool_name = self._pool_stack.pop()
-            self._pool = self._pools[self._pool_name]
-
-        self._unmatched_pool_machines = []
-        for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True):
-            if self._virtual_matching:
-                if "libvirt_domain" in p_machine["params"]:
-                    self._unmatched_pool_machines.append(p_id)
-            else:
-                self._unmatched_pool_machines.append(p_id)
-
-        if len(self._pool) > 0 and len(self._mreqs) > 0:
-            self._push_machine_stack()
-
-    def match(self):
-        logging.info("Trying match with pool: %s" % self._pool_name)
-        while len(self._machine_stack)>0:
-            stack_top = self._machine_stack[-1]
-            if self._virtual_matching and stack_top["virt_matched"]:
-                if stack_top["current_match"] != None:
-                    cur_match = stack_top["current_match"]
-                    self._unmatched_pool_machines.append(cur_match)
-                    stack_top["current_match"] = None
-                stack_top["virt_matched"] = False
-
-            if self._if_match():
-                if len(self._unmatched_req_machines) > 0:
-                    self._push_machine_stack()
-                    continue
-                else:
-                    return True
-            else:
-                #unmap the pool machine
-                if stack_top["current_match"] != None:
-                    cur_match = stack_top["current_match"]
-                    self._unmatched_pool_machines.append(cur_match)
-                stack_top["current_match"] = None
-
-                mreq_m_id = stack_top["m_id"]
-                while len(stack_top["remaining_matches"]) > 0:
-                    pool_m_id = stack_top["remaining_matches"].pop()
-                    if self._check_machine_compatibility(mreq_m_id, pool_m_id):
-                        #map compatible pool machine
-                        stack_top["current_match"] = pool_m_id
-                        stack_top["unmatched_pool_ifs"] = \
-                            sorted(self._pool[pool_m_id]["interfaces"].keys(),
-                                   reverse=True)
-                        self._unmatched_pool_machines.remove(pool_m_id)
-                        break
-
-                if stack_top["current_match"] != None:
-                    #clear if mapping
-                    stack_top["if_stack"] = []
-                    #next iteration will match the interfaces
-                    if not self._virtual_matching:
-                        self._push_if_stack()
-                    continue
-                else:
-                    self._pop_machine_stack()
-                    if len(self._machine_stack) == 0 and\
-                       len(self._pool_stack) > 0:
-                        logging.info("Match with pool %s not found." %
-                                     self._pool_name)
-                        self._pool_name = self._pool_stack.pop()
-                        self._pool = self._pools[self._pool_name]
-                        logging.info("Trying match with pool: %s" %
-                                     self._pool_name)
-
-                        self._unmatched_pool_machines = []
-                        for p_id, p_machine in sorted(self._pool.iteritems(),
reverse=True):
-                            if self._virtual_matching:
-                                if "libvirt_domain" in
p_machine["params"]:
-                                    self._unmatched_pool_machines.append(p_id)
-                            else:
-                                self._unmatched_pool_machines.append(p_id)
-
-                        if len(self._pool) > 0 and len(self._mreqs) > 0:
-                            self._push_machine_stack()
-                    continue
-        return False
-
-    def _if_match(self):
-        m_stack_top = self._machine_stack[-1]
-        if_stack = m_stack_top["if_stack"]
-
-        if self._virtual_matching:
-            if m_stack_top["current_match"] != None:
-                m_stack_top["virt_matched"] = True
-                return True
-            else:
-                return False
-
-        while len(if_stack) > 0:
-            stack_top = if_stack[-1]
-
-            req_m = self._mreqs[m_stack_top["m_id"]]
-            pool_m = self._pool[m_stack_top["current_match"]]
-            req_if = req_m["interfaces"][stack_top["if_id"]]
-            req_net_label = req_if["network"]
-
-            if stack_top["current_match"] != None:
-                cur_match = stack_top["current_match"]
-                m_stack_top["unmatched_pool_ifs"].append(cur_match)
-                pool_if = pool_m["interfaces"][cur_match]
-                pool_net_label = pool_if["network"]
-                net_label_mapping = self._net_label_mapping[req_net_label]
-                if net_label_mapping == (pool_net_label, m_stack_top["m_id"],
-                                         stack_top["if_id"]):
-                    del self._net_label_mapping[req_net_label]
-            stack_top["current_match"] = None
-
-            while len(stack_top["remaining_matches"]) > 0:
-                pool_if_id = stack_top["remaining_matches"].pop()
-                pool_if = pool_m["interfaces"][pool_if_id]
-                if self._check_interface_compatibility(req_if, pool_if):
-                    #map compatible interfaces
-                    stack_top["current_match"] = pool_if_id
-                    if req_net_label not in self._net_label_mapping:
-                        self._net_label_mapping[req_net_label] =\
-                                                   (pool_if["network"],
-                                                   m_stack_top["m_id"],
-                                                   stack_top["if_id"])
-                    m_stack_top["unmatched_pool_ifs"].remove(pool_if_id)
-                    break
-
-            if stack_top["current_match"] != None:
-                if len(m_stack_top["unmatched_ifs"]) > 0:
-                    self._push_if_stack()
-                    continue
-                else:
-                    return True
-            else:
-                self._pop_if_stack()
-                continue
-        return False
-
-    def _push_machine_stack(self):
-        machine_match = {}
-        machine_match["m_id"] = self._unmatched_req_machines.pop()
-        machine_match["current_match"] = None
-        machine_match["remaining_matches"] =
list(self._unmatched_pool_machines)
-        machine_match["if_stack"] = []
-
-        machine = self._mreqs[machine_match["m_id"]]
-        machine_match["unmatched_ifs"] =
sorted(machine["interfaces"].keys(),
-                                                reverse=True)
-        machine_match["unmatched_pool_ifs"] = []
-
-        if self._virtual_matching:
-            machine_match["virt_matched"] = False
-
-        self._machine_stack.append(machine_match)
-
-    def _pop_machine_stack(self):
-        stack_top = self._machine_stack.pop()
-        self._unmatched_req_machines.append(stack_top["m_id"])
-
-    def _push_if_stack(self):
-        m_stack_top = self._machine_stack[-1]
-        if_match = {}
-        if_match["if_id"] = m_stack_top["unmatched_ifs"].pop()
-        if_match["current_match"] = None
-        if_match["remaining_matches"] =
list(m_stack_top["unmatched_pool_ifs"])
-
-        m_stack_top["if_stack"].append(if_match)
-
-    def _pop_if_stack(self):
-        m_stack_top = self._machine_stack[-1]
-        if_stack_top = m_stack_top["if_stack"].pop()
-        m_stack_top["unmatched_ifs"].append(if_stack_top["if_id"])
-
-    def _check_machine_compatibility(self, req_id, pool_id):
-        req_machine = self._mreqs[req_id]
-        pool_machine = self._pool[pool_id]
-        for param, value in req_machine["params"].iteritems():
-            if param not in pool_machine["params"] or\
-               value != pool_machine["params"][param]:
-                return False
-        return True
-
-    def _check_interface_compatibility(self, req_if, pool_if):
-        label_mapping = self._net_label_mapping
-        for req_label, mapping in label_mapping.iteritems():
-            if req_label == req_if["network"] and\
-               mapping[0] != pool_if["network"]:
-                return False
-            if mapping[0] == pool_if["network"] and\
-               req_label != req_if["network"]:
-                return False
-        for param, value in req_if["params"].iteritems():
-            if param not in pool_if["params"] or\
-               value != pool_if["params"][param]:
-                return False
-        return True
-
-    def get_mapping(self):
-        mapping = {"machines": {}, "networks": {},
"virtual": False,
-                   "pool_name": self._pool_name}
-
-        for req_label, label_map in self._net_label_mapping.iteritems():
-            mapping["networks"][req_label] = label_map[0]
-
-        for machine in self._machine_stack:
-            m_map = mapping["machines"][machine["m_id"]] = {}
-
-            m_map["target"] = machine["current_match"]
-
-            hostname =
self._pool[m_map["target"]]["params"]["hostname"]
-            m_map["hostname"] = hostname
-
-            interfaces = m_map["interfaces"] = {}
-            if_stack = machine["if_stack"]
-            for interface in if_stack:
-                i = interfaces[interface["if_id"]] = {}
-                i["target"] = interface["current_match"]
-                pool_if =
self._pool[m_map["target"]]["interfaces"][i["target"]]
-                i["hwaddr"] = pool_if["params"]["hwaddr"]
-
-
-        if self._virtual_matching:
-            mapping["virtual"] = True
-        return mapping
diff --git a/lnst/Controller/XmlParser.py b/lnst/Controller/XmlParser.py
deleted file mode 100644
index 355b5e8..0000000
--- a/lnst/Controller/XmlParser.py
+++ /dev/null
@@ -1,188 +0,0 @@
-"""
-This module contains the XmlParser and LnstParser classes.
-
-Copyright 2013 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import os
-import re
-import sys
-import copy
-from lxml import etree
-from urllib2 import urlopen
-from lnst.Common.Config import lnst_config
-from lnst.Controller.XmlTemplates import XmlTemplates
-from lnst.Controller.XmlProcessing import XmlProcessingError
-
-class XmlParser(object):
-    XINCLUDE_RE = r"\{http\:\/\/www\.w3\.org\/[0-9]{4}\/XInclude\}include"
-
-    def __init__(self, schema_file, xml_path):
-        # locate the schema file
-        # try git path
-        dirname = os.path.dirname(sys.argv[0])
-        schema_path = os.path.join(dirname, schema_file)
-        if not os.path.exists(schema_path):
-            # try configuration
-            res_dir = lnst_config.get_option("environment",
"resource_dir")
-            schema_path = os.path.join(res_dir, schema_file)
-
-        if not os.path.exists(schema_path):
-            raise Exception("The recipe schema file was not found. " + \
-                            "Your LNST installation is corrupt!")
-
-        self._template_proc = XmlTemplates()
-
-        self._path = xml_path
-        relaxng_doc = etree.parse(schema_path)
-        self._schema = etree.RelaxNG(relaxng_doc)
-
-    def parse(self):
-        doc = self._parse(self._path)
-        self._remove_comments(doc)
-
-        # Due to a weird implementation of XInclude in lxml, the
-        # XmlParser resolves included documents on it's own.
-        #
-        # To be able to tell later on where each tag was located
-        # in the XML document, we add a '__file' attribute to
-        # each element of the tree during the parsing.
-        #
-        # However, these special attributes are of course not
-        # valid according to our schemas. To solve this, a copy of
-        # the tree is made and the '__file' attributes are removed
-        # before validation.
-        #
-        # XXX This is a *EXTREMELY* dirty hack. Ideas/proposals
-        # for cleaner solutions are more than welcome!
-        root_tag = self._init_loc(doc.getroot(), self._path)
-        self._expand_xinclude(root_tag, os.path.dirname(self._path))
-
-        self._template_proc.process_aliases(root_tag)
-
-        try:
-            self._validate(doc)
-        except:
-            err = self._schema.error_log[0]
-            loc = {"file": os.path.basename(err.filename),
-                   "line": err.line, "col": err.column}
-            exc = XmlProcessingError(err.message)
-            exc.set_loc(loc)
-            raise exc
-
-        return self._process(root_tag)
-
-    def _parse(self, path):
-        try:
-            if path.startswith('https'):
-                doc = etree.parse(urlopen(path))
-            else:
-                doc = etree.parse(path)
-        except etree.LxmlError as err:
-            # A workaround for cases when lxml (quite strangely)
-            # sets the filename to <string>.
-            if err.error_log[0].filename == "<string>":
-                filename = self._path
-            else:
-                filename = err.error_log[0].filename
-            loc = {"file": os.path.basename(filename),
-                   "line": err.error_log[0].line,
-                   "col": err.error_log[0].column}
-            exc = XmlProcessingError(err.error_log[0].message)
-            exc.set_loc(loc)
-            raise exc
-        except Exception as err:
-            loc = {"file": os.path.basename(self._path),
-                   "line": None,
-                   "col": None}
-            exc = XmlProcessingError(str(err))
-            exc.set_loc(loc)
-            raise exc
-
-        return doc
-
-    def _process(self, root_tag):
-        pass
-
-    def set_machines(self, machines):
-        self._template_proc.set_machines(machines)
-
-    def set_aliases(self, defined, overriden):
-        self._template_proc.set_aliases(defined, overriden)
-
-    def _has_attribute(self, element, attr):
-        return attr in element.attrib
-
-    def _get_attribute(self, element, attr):
-        text = element.attrib[attr].strip()
-        return self._template_proc.expand_functions(text)
-
-    def _get_content(self, element):
-        text = etree.tostring(element, method="text").strip()
-        return self._template_proc.expand_functions(text)
-
-    def _expand_xinclude(self, elem, base_url=""):
-        for e in elem:
-            if re.match(self.XINCLUDE_RE, str(e.tag)):
-                href = os.path.join(base_url, e.get("href"))
-                filename = os.path.basename(href)
-
-                doc = self._parse(href)
-                self._remove_comments(doc)
-                node = doc.getroot()
-
-                node = self._init_loc(node, href)
-
-                if e.tail:
-                    node.tail = (node.tail or "") + e.tail
-                self._expand_xinclude(node, os.path.dirname(href))
-
-                parent = e.getparent()
-                if parent is None:
-                    return node
-
-                parent.replace(e, node)
-            else:
-                self._expand_xinclude(e, base_url)
-        return elem
-
-    def _remove_comments(self, doc):
-        comments = doc.xpath('//comment()')
-        for c in comments:
-            p = c.getparent()
-            if p is not None:
-                p.remove(c)
-
-    def _init_loc(self, elem, filename):
-        """ Remove all coment tags from the tree """
-
-        elem.attrib["__file"] = filename
-        for e in elem:
-            self._init_loc(e, filename)
-
-        return elem
-
-    def _validate(self, original):
-        """
-            Make a copy of the tree, remove the '__file' attributes
-            and validate against the appropriate schema.
-
-            Very unfortunate solution.
-        """
-        doc = copy.deepcopy(original)
-        root = doc.getroot()
-
-        self._prepare_tree_for_validation(root)
-        self._schema.assertValid(doc)
-
-    def _prepare_tree_for_validation(self, elem):
-        if "__file" in elem.attrib:
-            del elem.attrib["__file"]
-        for e in elem:
-            self._prepare_tree_for_validation(e)
diff --git a/lnst/Controller/XmlProcessing.py b/lnst/Controller/XmlProcessing.py
deleted file mode 100644
index b80c3a3..0000000
--- a/lnst/Controller/XmlProcessing.py
+++ /dev/null
@@ -1,235 +0,0 @@
-"""
-This module contains code code for XML parsing and processing.
-
-Copyright 2012 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import os
-
-class XmlProcessingError(Exception):
-    """ Exception thrown on parsing errors """
-
-    _filename = None
-    _line = None
-    _col = None
-
-    def __init__(self, msg, obj=None):
-        super(XmlProcessingError, self).__init__()
-        self._msg = msg
-
-        if obj is not None:
-            if hasattr(obj, "loc"):
-                self.set_loc(obj.loc)
-            elif hasattr(obj, "attrib") and "__file" in obj.attrib:
-                loc = {}
-                loc["file"] = obj.attrib["__file"]
-                if hasattr(obj, "sourceline"):
-                    loc["line"] = obj.sourceline
-                self.set_loc(loc)
-            elif hasattr(obj, "base") and obj.base != None:
-                loc = {}
-                loc["file"] = os.path.basename(obj.base)
-                if hasattr(obj, "sourceline"):
-                    loc["line"] = obj.sourceline
-                self.set_loc(loc)
-
-
-    def set_loc(self, loc):
-        self._filename = loc["file"]
-        self._line = loc["line"]
-        if "col" in loc:
-            self._col = loc["col"]
-
-    def __str__(self):
-        line = ""
-        col = ""
-        sep = ""
-        loc = ""
-        filename = "<unknown>"
-
-        if self._filename:
-            filename = self._filename
-
-        if self._line:
-            line = "%d" % self._line
-            sep = ":"
-
-        if self._col:
-            col = "%s%d" % (sep, self._col)
-
-        if self._line or self._col:
-            loc = "%s%s:" % (line, col)
-
-        return "Parser error: %s:%s %s" % (filename, loc, self._msg)
-
-class XmlDataIterator:
-    def __init__(self, iterator):
-        self._iterator = iterator
-
-    def __iter__(self):
-        return self
-
-    def next(self):
-        n = self._iterator.next()
-
-        # For normal iterators
-        if type(n) == XmlTemplateString:
-            return str(n)
-
-        # For iteritems() iterators
-        if type(n) == tuple and len(n) == 2 and type(n[1]) == XmlTemplateString:
-            return (n[0], str(n[1]))
-
-        return n
-
-class XmlCollection(list):
-    def __init__(self, node=None):
-        super(XmlCollection, self).__init__()
-        if node is not None:
-            if hasattr(node, "loc"):
-                self.loc = node.loc
-            elif "__file" in node.attrib:
-                loc = {}
-                loc["file"] = node.attrib["__file"]
-                if hasattr(node, "sourceline"):
-                    loc["line"] = node.sourceline
-                self.loc = loc
-            elif hasattr(node, "base") and node.base != None:
-                loc = {}
-                loc["file"] = os.path.basename(node.base)
-                if hasattr(node, "sourceline"):
-                    loc["line"] = node.sourceline
-                self.loc = loc
-
-    def __getitem__(self, key):
-        value = super(XmlCollection, self).__getitem__(key)
-        if type(value) == XmlData or type(value) == XmlCollection:
-            return value
-
-        return str(value)
-
-    def __iter__(self):
-        it = super(XmlCollection, self).__iter__()
-        return XmlDataIterator(it)
-
-    def to_list(self):
-        new_list = list()
-        for value in self:
-            if isinstance(value, XmlData):
-                new_val = value.to_dict()
-            elif isinstance(value, XmlCollection):
-                new_val = value.to_list()
-            elif isinstance(value, XmlTemplateString):
-                new_val = str(value)
-            else:
-                new_val = value
-            new_list.append(new_val)
-
-        return new_list
-
-class XmlData(dict):
-    def __init__(self, node=None):
-        super(XmlData, self).__init__()
-        if node is not None:
-            if hasattr(node, "loc"):
-                self.loc = node.loc
-            elif "__file" in node.attrib:
-                loc = {}
-                loc["file"] = node.attrib["__file"]
-                if hasattr(node, "sourceline"):
-                    loc["line"] = node.sourceline
-                self.loc = loc
-            elif hasattr(node, "base") and node.base != None:
-                loc = {}
-                loc["file"] = os.path.basename(node.base)
-                if hasattr(node, "sourceline"):
-                    loc["line"] = node.sourceline
-                self.loc = loc
-
-    def __getitem__(self, key):
-        value = super(XmlData, self).__getitem__(key)
-        if type(value) == XmlData or type(value) == XmlCollection\
-           or value == None:
-            return value
-
-        return str(value)
-
-    def __iter__(self):
-        it = super(XmlData, self).__iter__()
-        return XmlDataIterator(it)
-
-    def iteritems(self):
-        it = super(XmlData, self).iteritems()
-        return XmlDataIterator(it)
-
-    def iterkeys(self):
-        it = super(XmlData, self).iterkeys()
-        return XmlDataIterator(it)
-
-    def itervalues(self):
-        it = super(XmlData, self).itervalues()
-        return XmlDataIterator(it)
-
-    def to_dict(self):
-        new_dict = dict()
-        for key, value in self.iteritems():
-            if isinstance(value, XmlData):
-                new_val = value.to_dict()
-            elif isinstance(value, XmlCollection):
-                new_val = value.to_list()
-            elif isinstance(value, XmlTemplateString):
-                new_val = str(value)
-            else:
-                new_val = value
-            new_dict[key] = new_val
-
-        return new_dict
-
-class XmlTemplateString(object):
-    def __init__(self, param=None, node=None):
-        if type(param) == str:
-            self._parts = [param]
-        elif type(param) == list:
-            self._parts = param
-        else:
-            self._parts = []
-
-        if node and hasattr(node, "loc"):
-            self.loc = node.loc
-
-    def __add__(self, other):
-        if type(other) is str:
-            self.add_part(other)
-        elif type(other) is self.__class__:
-            self._parts += other._parts
-        else:
-            raise XmlProcessingError("Cannot concatenate %s and %s" % \
-                                     str(type(self)), str(type(other)))
-        return self
-
-    def __str__(self):
-        string = ""
-        for part in self._parts:
-            string += str(part)
-        return string
-
-    def __hash__(self):
-        return hash(str(self))
-
-    def __eq__(self, other):
-        return str(self) == str(other)
-
-    def __ne__(self, other):
-        return str(self) != str(other)
-
-    def __len__(self):
-        return len(str(self))
-
-    def add_part(self, part):
-        self._parts.append(part)
diff --git a/lnst/Controller/XmlTemplates.py b/lnst/Controller/XmlTemplates.py
deleted file mode 100644
index a1541db..0000000
--- a/lnst/Controller/XmlTemplates.py
+++ /dev/null
@@ -1,438 +0,0 @@
-"""
-This module contains code to aid processing templates in XML files/recipes
-while they're being parsed.
-
-Templates are strings enclosed in curly braces {} and can be present
-in all text elements of the XML file (this includes tag values or
-attribute values). Templates cannot be used as a stubstitution for tag
-names, attribute names or any other structural elements of the document.
-
-There are two supported types of templates:
-
-    * aliases   - $alias_name
-    * functions - function_name(param1, param2)
-
-Copyright 2012 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import re
-from lxml import etree
-from lnst.Controller.XmlProcessing import XmlTemplateString
-from lnst.Controller.Machine import MachineError, PrefixMissingError
-
-class XmlTemplateError(Exception):
-    pass
-
-class TemplateFunc(object):
-    def __init__(self, args, machines):
-        self._check_args(args)
-        self._args = args
-
-        self._machines = machines
-
-    def __str__(self):
-        return self._implementation()
-
-    def _check_args(self, args):
-        pass
-
-    def _implementation(self):
-        pass
-
-class IpFunc(TemplateFunc):
-    def _check_args(self, args):
-        if len(args) > 3:
-            msg = "Function ip() takes at most 3 arguments, %d passed" \
-                  % len(args)
-            raise XmlTemplateError(msg)
-        if len(args) < 2:
-            msg = "Function ip() must have at least 2 arguments, %d passed" \
-                  % len(args)
-            raise XmlTemplateError(msg)
-
-        if len(args) == 3:
-            try:
-                int(args[2])
-            except ValueError:
-                msg = "The third argument of ip() function must be an integer"
-                raise XmlTemplateError(msg)
-
-    def _implementation(self):
-        m_id = self._args[0]
-        if_id = self._args[1]
-        addr = 0
-        if len(self._args) == 3:
-            addr = self._args[2]
-
-        try:
-            machine = self._machines[m_id]
-        except KeyError:
-            msg = "First parameter of function ip() is invalid: " \
-                  "Machine %s does not exist." % m_id
-            raise XmlTemplateError(msg)
-
-        try:
-            iface = machine.get_interface(if_id)
-        except MachineError:
-            msg = "Second parameter of function ip() is invalid: "\
-                    "Interface %s does not exist." % if_id
-            raise XmlTemplateError(msg)
-
-        try:
-            return iface.get_address(int(addr))
-        except IndexError:
-            msg = "There is no address with index %s on machine %s, " \
-                  "interface %s." % (addr, m_id, if_id)
-            raise XmlTemplateError(msg)
-
-class DevnameFunc(TemplateFunc):
-    def _check_args(self, args):
-        if len(args) != 2:
-            msg = "Function devname() takes 2 arguments, %d passed." %
len(args)
-            raise XmlTemplateError(msg)
-
-    def _implementation(self):
-        m_id = self._args[0]
-        if_id = self._args[1]
-
-        try:
-            machine = self._machines[m_id]
-        except KeyError:
-            msg = "First parameter of function devname() is invalid: " \
-                  "Machine %s does not exist." % m_id
-            raise XmlTemplateError(msg)
-
-        try:
-            iface = machine.get_interface(if_id)
-        except MachineError:
-            msg = "Second parameter of function devname() is invalid: "\
-                    "Interface %s does not exist." % if_id
-            raise XmlTemplateError(msg)
-
-        try:
-            return iface.get_devname()
-        except MachineError:
-            msg = "Devname not availablefor interface '%s' on machine
'%s'." \
-                                                    % (m_id, if_id)
-            raise XmlTemplateError(msg)
-
-class PrefixFunc(TemplateFunc):
-    def _check_args(self, args):
-        if len(args) > 3:
-            msg = "Function prefix() takes at most 3 arguments, %d passed" \
-                  % len(args)
-            raise XmlTemplateError(msg)
-        if len(args) < 2:
-            msg = "Function prefix() must have at least 2 arguments, %d " \
-                  "passed" % len(args)
-            raise XmlTemplateError(msg)
-
-        if len(args) == 3:
-            try:
-                int(args[2])
-            except ValueError:
-                msg = "The third argument of prefix() function must be an " \
-                      "integer"
-                raise XmlTemplateError(msg)
-
-    def _implementation(self):
-        m_id = self._args[0]
-        if_id = self._args[1]
-        addr = 0
-        if len(self._args) == 3:
-            addr = self._args[2]
-
-        try:
-            machine = self._machines[m_id]
-        except KeyError:
-            msg = "First parameter of function prefix() is invalid: " \
-                  "Machine %s does not exist." % m_id
-            raise XmlTemplateError(msg)
-
-        try:
-            iface = machine.get_interface(if_id)
-        except MachineError:
-            msg = "Second parameter of function prefix() is invalid: "\
-                    "Interface %s does not exist." % if_id
-            raise XmlTemplateError(msg)
-
-        try:
-            return iface.get_prefix(int(addr))
-        except IndexError:
-            msg = "There is no address with index %s on machine %s, " \
-                  "interface %s." % (addr, m_id, if_id)
-            raise XmlTemplateError(msg)
-        except PrefixMissingError:
-            msg = "Address with the index %s for the interface %s on machine"
\
-                  "%s does not contain any prefix" % (addr, m_id, if_id)
-
-class HwaddrFunc(TemplateFunc):
-    def _check_args(self, args):
-        if len(args) != 2:
-            msg = "Function hwaddr() takes 2 arguments, %d passed." %
len(args)
-            raise XmlTemplateError(msg)
-
-    def _implementation(self):
-        m_id = self._args[0]
-        if_id = self._args[1]
-
-        try:
-            machine = self._machines[m_id]
-        except KeyError:
-            msg = "First parameter of function hwaddr() is invalid: " \
-                  "Machine %s does not exist." % m_id
-            raise XmlTemplateError(msg)
-
-        try:
-            iface = machine.get_interface(if_id)
-        except MachineError:
-            msg = "Second parameter of function hwaddr() is invalid: "\
-                    "Interface %s does not exist." % if_id
-            raise XmlTemplateError(msg)
-
-        try:
-            return iface.get_hwaddr()
-        except MachineError:
-            msg = "Hwaddr not availablefor interface '%s' on machine
'%s'." \
-                                                    % (m_id, if_id)
-            raise XmlTemplateError(msg)
-
-class XmlTemplates:
-    """ This class serves as template processor """
-
-    _alias_re = "\{\$([a-zA-Z0-9_]+)\}"
-    _func_re  = "\{([a-zA-Z0-9_]+)\(([^\(\)]*)\)\}"
-
-    _func_map = {"ip": IpFunc, "hwaddr": HwaddrFunc,
"devname": DevnameFunc, \
-                 "prefix": PrefixFunc }
-
-    def __init__(self, definitions=None):
-        if definitions:
-            self._definitions = [definitions]
-        else:
-            self._definitions = [{}]
-
-        self._machines = {}
-        self._reserved_aliases = []
-
-    def set_definitions(self, defs):
-        """ Set alias definitions
-
-        All existing definitions and namespace levels are
-        destroyed and replaced with new definitions.
-        """
-        del self._definitions
-        self._definitions = [defs]
-
-    def get_definitions(self):
-        """ Return definitions dict
-
-        Definitions are returned as a single dictionary of
-        all currently defined aliases, regardless the internal
-        division to namespace levels.
-        """
-        defs = {}
-        for level in self._definitions:
-            for name, val in level.iteritems():
-                defs[name] = val
-
-        return defs
-
-    def set_machines(self, machines):
-        """ Assign machine information
-
-        XmlTemplates use these information about the machines
-        to resolve template functions within the recipe.
-        """
-        self._machines = machines
-
-    def set_aliases(self, defined, overriden):
-        """ Set aliases defined or overriden from CLI """
-
-        for name, value in defined.iteritems():
-            self.define_alias(name, value)
-
-        self._overriden_aliases = overriden
-
-    def define_alias(self, name, value):
-        """ Associate an alias name with some value
-
-        The value can be of an atomic type or an array. The
-        definition is added to the current namespace level.
-        """
-
-        if not name in self._reserved_aliases:
-            self._definitions[-1][name] = value
-        else:
-            raise XmlTemplateError("Alias name '%s' is reserved" %
name)
-
-    def add_namespace_level(self):
-        """ Create new namespace level
-
-            This method will create a new level for definitions on
-            the stack. All aliases, that will be defined after this
-            call will be dropped as soon as `drop_namespace_level'
-            is called.
-        """
-        self._definitions.append({})
-
-    def drop_namespace_level(self):
-        """ Remove one namespace level
-
-            This method will erease all defined aliases since the
-            last call of `add_namespace_level' method. All aliases,
-            that were defined beforehand will be kept.
-        """
-        self._definitions.pop()
-
-    def _find_definition(self, name):
-        if name in self._overriden_aliases:
-            return self._overriden_aliases[name]
-
-        for level in reversed(self._definitions):
-            if name in level:
-                return level[name]
-
-        err = "Alias '%s' is not defined here" % name
-        raise XmlTemplateError(err)
-
-    def _dump_definitions(self):
-        dump = self._overriden_aliases.copy()
-
-        for level in self._definitions:
-            for name in level:
-                if not name in dump:
-                    dump[name] = level[name]
-
-        return dump
-
-    def process_aliases(self, element):
-        """ Expand aliases within an element and its children
-
-            This method will iterate through the element tree that is
-            passed and expand aliases in all the text content and
-            attributes.
-        """
-        if element.text != None:
-            element.text = self.expand_aliases(element.text)
-
-        if element.tail != None:
-            element.tail = self.expand_aliases(element.tail)
-
-        for name, value in element.attrib.iteritems():
-            element.set(name, self.expand_aliases(value))
-
-        if element.tag == "define":
-            for alias in element.getchildren():
-                name = alias.attrib["name"].strip()
-                if "value" in alias.attrib:
-                    value = alias.attrib["value"].strip()
-                else:
-                    value = etree.tostring(element, method="text").strip()
-                self.define_alias(name, value)
-            parent = element.getparent()
-            parent.remove(element)
-            return
-
-        self.add_namespace_level()
-
-        for child in element.getchildren():
-            self.process_aliases(child)
-
-        # do not drop alias definitions when at top-level so that python
-        # tasks are able to access them
-        if element.tag != "lnstrecipe":
-            self.drop_namespace_level()
-
-    def expand_aliases(self, string):
-        while True:
-            alias_match = re.search(self._alias_re, string)
-
-            if alias_match:
-                template = alias_match.group(0)
-                result = self._process_alias_template(template)
-                string = string.replace(template, result)
-            else:
-                break
-
-        return string
-
-    def _process_alias_template(self, string):
-        result = None
-
-        alias_match = re.match(self._alias_re, string)
-        if alias_match:
-            alias_name = alias_match.group(1)
-            result = self._find_definition(alias_name)
-
-        return result
-
-    def expand_functions(self, string, node=None):
-        """ Process a string and expand it into a XmlTemplateString
"""
-
-        parts = self._partition_string(string)
-        value = XmlTemplateString(node=node)
-
-        for part in parts:
-            value.add_part(part)
-
-        return value
-
-    def _partition_string(self, string):
-        """ Process templates in a string
-
-            This method will process and expand all template functions
-            in a string.
-
-            The function returns an array of string partitions and
-            unresolved template functions for further processing.
-        """
-
-        result = None
-
-        func_match  = re.search(self._func_re, string)
-        if func_match:
-            prefix = string[0:func_match.start(0)]
-            suffix = string[func_match.end(0):]
-
-            template = func_match.group(0)
-            func = self._process_func_template(template)
-
-            return self._partition_string(prefix) + [func] + \
-                   self._partition_string(suffix)
-
-        return [string]
-
-    def _process_func_template(self, string):
-        func_match = re.match(self._func_re, string)
-        if func_match:
-            func_name = func_match.group(1)
-            func_args = func_match.group(2)
-
-            if func_args == None:
-                func_args = []
-            else:
-                func_args = func_args.split(",")
-
-            param_values = []
-            for param in func_args:
-                param = param.strip()
-                if re.match(self._alias_re, param):
-                    param = self._process_alias_template(param)
-                param_values.append(param)
-
-            if func_name not in self._func_map:
-                msg = "Unknown template function '%s'." % func_name
-                raise XmlTemplateError(msg)
-
-            func = self._func_map[func_name](param_values, self._machines)
-            return func
-        else:
-            msg = "The passed string is not a template function."
-            raise XmlTemplateError(msg)
-- 
2.13.0

[PATCH v2 38/47] setup.py: add new packages

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This adds the lnst.Devices and lnst.Tests packages so that they're
recognized and installed by the setup.py install script.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/setup.py b/setup.py
index 41b86b3..322731f 100755
--- a/setup.py
+++ b/setup.py
@@ -105,7 +105,7 @@ project website <https://fedorahosted.org/lnst>;.
 """
 
 PACKAGES = ["lnst", "lnst.Common", "lnst.Controller",
"lnst.Slave",
-            "lnst.RecipeCommon" ]
+            "lnst.RecipeCommon", "lnst.Devices",
"lnst.Tests" ]
 SCRIPTS = ["lnst-ctl", "lnst-slave", "lnst-pool-wizard"]
 
 RECIPE_FILES = []
-- 
2.13.0

[PATCH v2 39/47] add lnst.Devices.VirtNetCtl

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Moving the VirtNetCtl class to the lnst.Devices package since it is only
used by the VirtualDevice class.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Devices/VirtNetCtl.py    | 85 +++++++++++++++++++++++++++++++++++++++++++
 lnst/Devices/VirtualDevice.py |  2 +-
 2 files changed, 86 insertions(+), 1 deletion(-)
 create mode 100644 lnst/Devices/VirtNetCtl.py

diff --git a/lnst/Devices/VirtNetCtl.py b/lnst/Devices/VirtNetCtl.py
new file mode 100644
index 0000000..2795baf
--- /dev/null
+++ b/lnst/Devices/VirtNetCtl.py
@@ -0,0 +1,85 @@
+"""
+Defines the VirtNetCtl class used for dynamically adding and removing network
+interface cards to libvirt guests.
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
+import logging
+import libvirt
+from libvirt import libvirtError
+from lnst.Common.LnstError import LnstError
+
+#this is a global object because opening the connection to libvirt in every
+#object instance that uses it sometimes fails - the libvirt server probably
+#can't handle that many connections at a time
+_libvirt_conn = None
+
+def init_libvirt_con():
+    global _libvirt_conn
+    if _libvirt_conn is None:
+        _libvirt_conn = libvirt.open(None)
+
+class VirtNetCtlError(LnstError):
+    pass
+
+class VirtNetCtl(object):
+    _network_template = """
+    <network ipv6='yes'>
+        <name>{0}</name>
+        <bridge name='virbr_{0}' stp='off' delay='0' />
+        <domain name='{0}'/>
+    </network>
+    """
+
+    def __init__(self, name=None):
+        init_libvirt_con()
+
+        if not name:
+            name = self._generate_name()
+        self._name = name
+
+    def _generate_name(self):
+        devs = _libvirt_conn.listNetworks()
+
+        index = 0
+        while True:
+            name = "lnst_net%d" % index
+            index += 1
+            if name not in devs:
+                return name
+
+    def get_name(self):
+        return self._name
+
+    def init(self):
+        try:
+            network_xml = self._network_template.format(self._name)
+            _libvirt_conn.networkCreateXML(network_xml)
+            logging.debug("libvirt network '%s' created" % self._name)
+            return True
+        except libvirtError as e:
+            raise VirtNetCtlError(str(e))
+
+    def cleanup(self):
+        try:
+            network = _libvirt_conn.networkLookupByName(self._name)
+            network.destroy()
+            logging.debug("libvirt network '%s' destroyed" %
self._name)
+            return True
+        except libvirtError as e:
+            raise VirtNetCtlError(str(e))
+
+    @classmethod
+    def network_exist(cls, net_name):
+        try:
+            _libvirt_conn.networkLookupByName(net_name)
+            return True
+        except:
+            return False
diff --git a/lnst/Devices/VirtualDevice.py b/lnst/Devices/VirtualDevice.py
index 49fe088..5c41a12 100644
--- a/lnst/Devices/VirtualDevice.py
+++ b/lnst/Devices/VirtualDevice.py
@@ -19,7 +19,7 @@ from lnst.Devices.RemoteDevice import RemoteDevice
 
 # conditional support for libvirt
 if check_process_running("libvirtd"):
-    from lnst.Controller.VirtUtils import VirtNetCtl
+    from lnst.Devices.VirtNetCtl import VirtNetCtl
 
 class VirtualDevice(RemoteDevice):
     """Remote eth device created on the controller through libvirt
-- 
2.13.0

[PATCH v2 40/47] lnst.Controller: move VirtUtils to VirtDomainCtl, remove VirtNetCtl class

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Since the VirtNetCtl class was moved into the lnst.Devices package, it
is not needed in this module anymore. Removing it along with the
BridgeCtl class which wasn't used in a long time (since 2013).

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Machine.py       |   2 +-
 lnst/Controller/VirtDomainCtl.py |  98 ++++++++++++++
 lnst/Controller/VirtUtils.py     | 269 ---------------------------------------
 3 files changed, 99 insertions(+), 270 deletions(-)
 create mode 100644 lnst/Controller/VirtDomainCtl.py
 delete mode 100644 lnst/Controller/VirtUtils.py

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index f057390..2c54f23 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -35,7 +35,7 @@ from lnst.Devices.VirtualDevice import VirtualDevice
 
 # conditional support for libvirt
 if check_process_running("libvirtd"):
-    from lnst.Controller.VirtUtils import VirtNetCtl, VirtDomainCtl
+    from lnst.Controller.VirtDomainCtl import VirtDomainCtl
 
 class MachineError(ControllerError):
     pass
diff --git a/lnst/Controller/VirtDomainCtl.py b/lnst/Controller/VirtDomainCtl.py
new file mode 100644
index 0000000..96c7913
--- /dev/null
+++ b/lnst/Controller/VirtDomainCtl.py
@@ -0,0 +1,98 @@
+"""
+Utilities for manipulating virtualization host, its guests and
+connections between them
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+rpazdera(a)redhat.com (Radek Pazdera)
+"""
+
+import logging
+import libvirt
+from libvirt import libvirtError
+from lnst.Controller.Common import ControllerError
+
+#this is a global object because opening the connection to libvirt in every
+#object instance that uses it sometimes fails - the libvirt server probably
+#can't handle that many connections at a time
+_libvirt_conn = None
+
+def init_libvirt_con():
+    global _libvirt_conn
+    if _libvirt_conn is None:
+        _libvirt_conn = libvirt.open(None)
+
+class VirtDomainCtlError(ControllerError):
+    pass
+
+class VirtDomainCtl(object):
+    _net_device_template = """
+    <interface type='network'>
+        <mac address='{0}'/>
+        <source network='{1}'/>
+        <model type='{2}'/>
+    </interface>
+    """
+    _net_device_bare_template = """
+    <interface>
+        <mac address='{0}'/>
+    </interface>
+    """
+
+    def __init__(self, domain_name):
+        self._name = domain_name
+        self._created_interfaces = {}
+
+        init_libvirt_con()
+
+        try:
+            self._domain = _libvirt_conn.lookupByName(domain_name)
+        except:
+            raise VirtDomainCtlError("Domain '%s' doesn't exist!" %
domain_name)
+
+    def start(self):
+        self._domain.create()
+
+    def stop(self):
+        self._domain.destroy()
+
+    def restart(self):
+        self._domain.reboot()
+
+    def attach_interface(self, hw_addr, net_name, driver="virtio"):
+        try:
+            device_xml = self._net_device_template.format(hw_addr,
+                                                          net_name,
+                                                          driver)
+            self._domain.attachDevice(device_xml)
+            logging.debug("libvirt device with hwaddr '%s' "
+                          "driver '%s' attached" % (hw_addr, driver))
+            self._created_interfaces[hw_addr] = device_xml
+            return True
+        except libvirtError as e:
+            raise VirtDomainCtlError(str(e))
+
+    def detach_interface(self, hw_addr):
+        if hw_addr in self._created_interfaces:
+            device_xml = self._created_interfaces[hw_addr]
+        else:
+            device_xml = self._net_device_bare_template.format(hw_addr)
+
+        try:
+            self._domain.detachDevice(device_xml)
+            logging.debug("libvirt device with hwaddr '%s' detached" %
hw_addr)
+            return True
+        except libvirtError as e:
+            raise VirtDomainCtlError(str(e))
+
+    @classmethod
+    def domain_exist(cls, domain_name):
+        try:
+            _libvirt_conn.lookupByName(domain_name)
+            return True
+        except:
+            return False
diff --git a/lnst/Controller/VirtUtils.py b/lnst/Controller/VirtUtils.py
deleted file mode 100644
index c8da28a..0000000
--- a/lnst/Controller/VirtUtils.py
+++ /dev/null
@@ -1,269 +0,0 @@
-"""
-Utilities for manipulating virtualization host, its guests and
-connections between them
-
-Copyright 2012 Red Hat, Inc.
-Licensed under the GNU General Public License, version 2 as
-published by the Free Software Foundation; see COPYING for details.
-"""
-
-__author__ = """
-rpazdera(a)redhat.com (Radek Pazdera)
-"""
-
-import logging
-import libvirt
-from libvirt import libvirtError
-from lnst.Common.ExecCmd import exec_cmd, ExecCmdFail
-from lnst.Common.NetUtils import scan_netdevs
-from lnst.Controller.Common import ControllerError
-
-#this is a global object because opening the connection to libvirt in every
-#object instance that uses it sometimes fails - the libvirt server probably
-#can't handle that many connections at a time
-_libvirt_conn = None
-
-def init_libvirt_con():
-    global _libvirt_conn
-    if _libvirt_conn is None:
-        _libvirt_conn = libvirt.open(None)
-
-class VirtUtilsError(ControllerError):
-    pass
-
-def _ip(cmd):
-    try:
-        exec_cmd("ip %s" % cmd)
-    except ExecCmdFail as err:
-        raise VirtUtilsError("ip command error: %s" % err)
-
-def _brctl(cmd):
-    try:
-        exec_cmd("brctl %s" % cmd)
-    except ExecCmdFail as err:
-        raise VirtUtilsError("brctl error: %s" % err)
-
-def _iptables(cmd):
-    try:
-        exec_cmd("iptables %s" % cmd)
-    except ExecCmdFail as err:
-        raise VirtUtilsError("iptables error: %s" % err)
-
-def _ip6tables(cmd):
-    try:
-        exec_cmd("ip6tables %s" % cmd)
-    except ExecCmdFail as err:
-        raise VirtUtilsError("ip6tables error: %s" % err)
-
-def _virsh(cmd):
-    try:
-        exec_cmd("virsh %s" % cmd, log_outputs=False)
-    except ExecCmdFail as err:
-        raise VirtUtilsError("virsh error: %s" % err)
-
-class VirtDomainCtl:
-    _net_device_template = """
-    <interface type='network'>
-        <mac address='{0}'/>
-        <source network='{1}'/>
-        <model type='{2}'/>
-    </interface>
-    """
-    _net_device_bare_template = """
-    <interface>
-        <mac address='{0}'/>
-    </interface>
-    """
-
-    def __init__(self, domain_name):
-        self._name = domain_name
-        self._created_interfaces = {}
-
-        init_libvirt_con()
-
-        try:
-            self._domain = _libvirt_conn.lookupByName(domain_name)
-        except:
-            raise VirtUtilsError("Domain '%s' doesn't exist!" %
domain_name)
-
-    def start(self):
-        self._domain.create()
-
-    def stop(self):
-        self._domain.destroy()
-
-    def restart(self):
-        self._domain.reboot()
-
-    def attach_interface(self, hw_addr, net_name, driver="virtio"):
-        try:
-            device_xml = self._net_device_template.format(hw_addr,
-                                                          net_name,
-                                                          driver)
-            self._domain.attachDevice(device_xml)
-            logging.debug("libvirt device with hwaddr '%s' "
-                          "driver '%s' attached" % (hw_addr, driver))
-            self._created_interfaces[hw_addr] = device_xml
-            return True
-        except libvirtError as e:
-            raise VirtUtilsError(str(e))
-
-    def detach_interface(self, hw_addr):
-        if hw_addr in self._created_interfaces:
-            device_xml = self._created_interfaces[hw_addr]
-        else:
-            device_xml = self._net_device_bare_template.format(hw_addr)
-
-        try:
-            self._domain.detachDevice(device_xml)
-            logging.debug("libvirt device with hwaddr '%s' detached" %
hw_addr)
-            return True
-        except libvirtError as e:
-            raise VirtUtilsError(str(e))
-
-    @classmethod
-    def domain_exist(cls, domain_name):
-        try:
-            _libvirt_conn.lookupByName(domain_name)
-            return True
-        except:
-            return False
-
-class NetCtl(object):
-    def __init__(self, name):
-        self._name = name
-
-    def get_name(self):
-        return self._name
-
-    def init(self):
-        pass
-
-    def cleanup(self):
-        pass
-
-class VirtNetCtl(NetCtl):
-    _network_template = """
-    <network ipv6='yes'>
-        <name>{0}</name>
-        <bridge name='virbr_{0}' stp='off' delay='0' />
-        <domain name='{0}'/>
-    </network>
-    """
-
-    def __init__(self, name=None):
-        init_libvirt_con()
-
-        if not name:
-            name = self._generate_name()
-        self._name = name
-
-    def _generate_name(self):
-        devs = _libvirt_conn.listNetworks()
-
-        index = 0
-        while True:
-            name = "lnst_net%d" % index
-            index += 1
-            if name not in devs:
-                return name
-
-    def init(self):
-        try:
-            network_xml = self._network_template.format(self._name)
-            _libvirt_conn.networkCreateXML(network_xml)
-            logging.debug("libvirt network '%s' created" % self._name)
-            return True
-        except libvirtError as e:
-            raise VirtUtilsError(str(e))
-
-    def cleanup(self):
-        try:
-            network = _libvirt_conn.networkLookupByName(self._name)
-            network.destroy()
-            logging.debug("libvirt network '%s' destroyed" %
self._name)
-            return True
-        except libvirtError as e:
-            raise VirtUtilsError(str(e))
-
-    @classmethod
-    def network_exist(cls, net_name):
-        try:
-            _libvirt_conn.networkLookupByName(net_name)
-            return True
-        except:
-            return False
-
-class BridgeCtl(NetCtl):
-    def __init__(self, name=None):
-        if not name:
-            name = self._generate_name()
-
-        self._check_name(name)
-        self._name = name
-        self._remove = False
-
-    def get_name(self):
-        return self._name
-
-    def set_remove(self, remove):
-        self._remove = remove
-
-    @staticmethod
-    def _check_name(name):
-        if len(name) > 16:
-            msg = "Bridge name '%s' longer than 16 characters" % name
-            raise VirtUtilsError(msg)
-
-    @staticmethod
-    def _generate_name():
-        devs = scan_netdevs()
-
-        index = 0
-        while True:
-            name = "lnstbr%d" % index
-            index += 1
-            unique = True
-            for dev in devs:
-                if name == dev["name"]:
-                    unique = False
-                    break
-
-            if unique:
-                return name
-
-    def _exists(self):
-        devs = scan_netdevs()
-        for dev in devs:
-            if self._name == dev["name"]:
-                return True
-
-        return False
-
-    def init(self):
-        if not self._exists():
-            _brctl("addbr %s" % self._name)
-            _iptables("-I FORWARD 1 -j REJECT -i %s -o any" % self._name)
-            _iptables("-I FORWARD 1 -j REJECT -i any -o %s" % self._name)
-            _iptables("-I FORWARD 1 -j ACCEPT -i %s -o %s" %
-                                                    (self._name, self._name))
-            _ip6tables("-I FORWARD 1 -j REJECT -i %s -o any" % self._name)
-            _ip6tables("-I FORWARD 1 -j REJECT -i any -o %s" % self._name)
-            _ip6tables("-I FORWARD 1 -j ACCEPT -i %s -o %s" %
-                                                    (self._name, self._name))
-            self._remove = True
-
-        _ip("link set %s up" % self._name)
-
-    def cleanup(self):
-        if self._remove:
-            _ip("link set %s down" % self._name)
-            _brctl("delbr %s" % self._name)
-            _iptables("-D FORWARD -j REJECT -i %s -o any" % self._name)
-            _iptables("-D FORWARD -j REJECT -i any -o %s" % self._name)
-            _iptables("-D FORWARD -j ACCEPT -i %s -o %s" %
-                                                    (self._name, self._name))
-            _ip6tables("-D FORWARD -j REJECT -i %s -o any" % self._name)
-            _ip6tables("-D FORWARD -j REJECT -i any -o %s" % self._name)
-            _ip6tables("-D FORWARD -j ACCEPT -i %s -o %s" %
-                                                    (self._name, self._name))
-- 
2.13.0

[PATCH v2 41/47] lnst.Common.Parameters: IpParam accepts Device objects

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

You can now set a Device (RemoteDevice on the Controller) objects as
values for IpParameters. This will get the ip address list of the Device
and use the first address as the value for the IpParam object.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;

---
v2:
    * import Device on demand (Device class arrives on Slave later) and
      check isinstance Device instead of RemoteDevice (not available on
      Slave)
    * fix IpParam Exception string
---
 lnst/Common/Parameters.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py
index dcf9bab..16385b4 100644
--- a/lnst/Common/Parameters.py
+++ b/lnst/Common/Parameters.py
@@ -77,12 +77,17 @@ class StrParam(Param):
 class IpParam(Param):
     @Param.val.setter
     def val(self, value):
+        #runtime import this because the Device class arrives on the Slave
+        #during recipe execution, not during Slave init
+        from lnst.Devices.Device import Device
         if isinstance(value, BaseIpAddress):
             self._val = value
         elif isinstance(value, str):
             self._val = IpAddress(value)
+        elif isinstance(value, Device):
+            self.val = value.ips[0]
         else:
-            raise ParamError("Value must be a BaseIpAddress or string object."
+            raise ParamError("Value must be a BaseIpAddress, string or Device
object."
                              "Not {}".format(type(value)))
         self.set = True
 
-- 
2.13.0

[PATCH v2 42/47] add example python_recipe.py script

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This is an example python recipe that can be run as an executable script.
Performs a simple ping between two hosts.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 recipes/examples/python_recipe.py | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100755 recipes/examples/python_recipe.py

diff --git a/recipes/examples/python_recipe.py b/recipes/examples/python_recipe.py
new file mode 100755
index 0000000..b15bc60
--- /dev/null
+++ b/recipes/examples/python_recipe.py
@@ -0,0 +1,31 @@
+#!/bin/python2
+"""
+This is an example python recipe that can be run as an executable script.
+Performs a simple ping between two hosts.
+"""
+
+from lnst.Common.Parameters import IpParam
+from lnst.Common.IpAddress import IpAddress
+from lnst.Controller import Controller
+from lnst.Controller import BaseRecipe
+from lnst.Controller import HostReq, DeviceReq
+
+from lnst.Tests import IcmpPing
+
+class MyRecipe(BaseRecipe):
+    m1 = HostReq()
+    m1.eth0 = DeviceReq(label="net1")
+
+    m2 = HostReq()
+    m2.eth0 = DeviceReq(label="net1")
+
+    def test(self):
+        self.matched.m1.eth0.ip_add(IpAddress("192.168.1.1/24"))
+        self.matched.m2.eth0.ip_add(IpAddress("192.168.1.2/24"))
+        ping_job = self.matched.m1.run(IcmpPing(dst=self.matched.m2.eth0,
+                                                interval=0))
+
+ctl = Controller(debug=1)
+
+r = MyRecipe()
+ctl.run(r, allow_virt=True)
-- 
2.13.0

[PATCH v2 43/47] Controller, Devices: removing unused imports

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Controller/Common.py            | 2 --
 lnst/Controller/Controller.py        | 4 +---
 lnst/Controller/CtlSecSocket.py      | 1 -
 lnst/Controller/Host.py              | 2 --
 lnst/Controller/Job.py               | 1 -
 lnst/Controller/Machine.py           | 8 --------
 lnst/Controller/MessageDispatcher.py | 2 +-
 lnst/Controller/Recipe.py            | 1 -
 lnst/Devices/Device.py               | 2 +-
 lnst/Devices/__init__.py             | 1 -
 10 files changed, 3 insertions(+), 21 deletions(-)

diff --git a/lnst/Controller/Common.py b/lnst/Controller/Common.py
index c9e5771..5e00de6 100644
--- a/lnst/Controller/Common.py
+++ b/lnst/Controller/Common.py
@@ -11,8 +11,6 @@ __author__ = """
 olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
-import os
-import sys
 from lnst.Common.LnstError import LnstError
 
 class ControllerError(LnstError):
diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py
index 7848750..dd0dd41 100644
--- a/lnst/Controller/Controller.py
+++ b/lnst/Controller/Controller.py
@@ -16,19 +16,17 @@ import os
 import sys
 import datetime
 import logging
-import socket
 from lnst.Common.Logs import LoggingCtl
 from lnst.Common.NetUtils import MacPool
 from lnst.Common.LnstError import LnstError
 from lnst.Common.Utils import mkdir_p
-from lnst.Devices import VirtualDevice
+from lnst.Devices.VirtualDevice import VirtualDevice
 from lnst.Controller.Common import ControllerError
 from lnst.Controller.Config import CtlConfig
 from lnst.Controller.MessageDispatcher import MessageDispatcher
 from lnst.Controller.SlavePoolManager import SlavePoolManager
 from lnst.Controller.MachineMapper import MachineMapper
 from lnst.Controller.Host import Hosts, Host
-from lnst.Controller.Requirements import DeviceReq
 from lnst.Controller.Recipe import BaseRecipe
 
 class Controller(object):
diff --git a/lnst/Controller/CtlSecSocket.py b/lnst/Controller/CtlSecSocket.py
index c80917b..0d30a10 100644
--- a/lnst/Controller/CtlSecSocket.py
+++ b/lnst/Controller/CtlSecSocket.py
@@ -13,7 +13,6 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 
 import os
 import hashlib
-import math
 import logging
 from lnst.Common.SecureSocket import SecureSocket
 from lnst.Common.SecureSocket import DH_GROUP, SRP_GROUP
diff --git a/lnst/Controller/Host.py b/lnst/Controller/Host.py
index 23cdadf..09e79f8 100644
--- a/lnst/Controller/Host.py
+++ b/lnst/Controller/Host.py
@@ -12,9 +12,7 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
 import logging
-from lnst.Common.Colours import decorate_with_preset
 from lnst.Common.Parameters import Parameters
-from lnst.Common.TestModule import BaseTestModule
 from lnst.Common.NetTestCommand import DEFAULT_TIMEOUT
 from lnst.Devices import Devices
 from lnst.Devices.VirtualDevice import VirtualDevice
diff --git a/lnst/Controller/Job.py b/lnst/Controller/Job.py
index 985364e..ce323a8 100644
--- a/lnst/Controller/Job.py
+++ b/lnst/Controller/Job.py
@@ -13,7 +13,6 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 
 import logging
 import signal
-import copy_reg
 from lnst.Common.JobError import JobError
 from lnst.Common.TestModule import BaseTestModule
 
diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 2c54f23..dc150d6 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -13,19 +13,11 @@ rpazdera(a)redhat.com (Radek Pazdera)
 
 import logging
 import socket
-import os
 import sys
-import tempfile
 import signal
-from time import sleep
-from xmlrpclib import Binary
-from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.Utils import sha256sum
-from lnst.Common.Utils import wait_for, create_tar_archive
 from lnst.Common.Utils import check_process_running
 from lnst.Common.TestModule import BaseTestModule
-from lnst.Common.NetTestCommand import DEFAULT_TIMEOUT
-from lnst.Common.DeviceError import DeviceDeleted, DeviceNotFound
 from lnst.Controller.Common import ControllerError
 from lnst.Controller.CtlSecSocket import CtlSecSocket
 from lnst.Devices import device_classes
diff --git a/lnst/Controller/MessageDispatcher.py b/lnst/Controller/MessageDispatcher.py
index a6d7f44..0f3d23d 100644
--- a/lnst/Controller/MessageDispatcher.py
+++ b/lnst/Controller/MessageDispatcher.py
@@ -15,7 +15,7 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
 import logging
-from lnst.Common.ConnectionHandler import send_data, recv_data
+from lnst.Common.ConnectionHandler import send_data
 from lnst.Common.ConnectionHandler import ConnectionHandler
 from lnst.Common.DeviceRef import DeviceRef
 from lnst.Controller.Common import ControllerError
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
index d4f33f9..445cb38 100644
--- a/lnst/Controller/Recipe.py
+++ b/lnst/Controller/Recipe.py
@@ -13,7 +13,6 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 import copy
 from lnst.Common.Parameters import Parameters, Param
 from lnst.Controller.Requirements import _Requirements, HostReq
-from lnst.Controller.Host import Hosts, Host
 from lnst.Controller.Common import ControllerError
 
 class RecipeError(ControllerError):
diff --git a/lnst/Devices/Device.py b/lnst/Devices/Device.py
index 8ba59f2..cf7c5ca 100644
--- a/lnst/Devices/Device.py
+++ b/lnst/Devices/Device.py
@@ -16,7 +16,7 @@ from abc import ABCMeta
 from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.ExecCmd import exec_cmd
 from lnst.Common.DeviceError import DeviceError, DeviceDeleted
-from lnst.Common.IpAddress import Ip4Address, Ip6Address, IpAddress
+from lnst.Common.IpAddress import IpAddress
 
 try:
     from pyroute2.netlink.iproute import RTM_NEWLINK
diff --git a/lnst/Devices/__init__.py b/lnst/Devices/__init__.py
index 2f8d4d1..743c8f5 100644
--- a/lnst/Devices/__init__.py
+++ b/lnst/Devices/__init__.py
@@ -9,7 +9,6 @@ from lnst.Devices.VxlanDevice import VxlanDevice
 from lnst.Devices.VtiDevice import VtiDevice, Vti6Device
 from lnst.Devices.VethDevice import VethDevice, PairedVethDevice
 from lnst.Devices.VethPair import VethPair
-from lnst.Devices.VirtualDevice import VirtualDevice
 from lnst.Devices.RemoteDevice import RemoteDevice, remotedev_decorator
 
 device_classes = [
-- 
2.13.0

[PATCH v2 44/47] NetTestSlave: create dynamic lnst.Tests package

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

The lnst.Tests package is present on the Controller and provides
upstream maintained test classes. These can be instantiated and used to
run tests on the Slaves. While doing so they're sent over the network to
the Slaves where they're dynamically imported. Since the lnst.Tests
package doesn't originaly exist on the slave, every imported module
imported this way will result in a python runtime warning of the package
not existing. This doesn't break anything but it does add clutter to the
logs.
Pre-creating the package on the slave will fix the issue.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Slave/NetTestSlave.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index 0f654e3..99914df 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -51,6 +51,11 @@ Devices.__path__ = ["lnst.Devices"]
 
 sys.modules["lnst.Devices"] = Devices
 
+Tests = types.ModuleType("Tests")
+Tests.__path__ = ["lnst.Tests"]
+
+sys.modules["lnst.Tests"] = Tests
+
 class SlaveMethods:
     '''
     Exported xmlrpc methods
-- 
2.13.0

[PATCH v2 45/47] lnst.Tests: add package docstring

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Tests/__init__.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lnst/Tests/__init__.py b/lnst/Tests/__init__.py
index 60da773..69b757a 100644
--- a/lnst/Tests/__init__.py
+++ b/lnst/Tests/__init__.py
@@ -1 +1,17 @@
+"""
+Package for all LNST Test classes. It will contain all Test classes provided
+and maintained by the LNST upstream and later it will also import test classes
+based on user configured directories (from lnst-ctl.conf).
+
+Copyright 2017 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
+olichtne(a)redhat.com (Ondrej Lichtner)
+"""
+
 from lnst.Tests.IcmpPing import IcmpPing
+
+#TODO add support for test classes from lnst-ctl.conf
-- 
2.13.0

[PATCH v2 46/47] lnst.Common.Parameters: add DeviceParam

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

This parameter class accepts a Device or DeviceRef objects as possible
values. Since RemoteDevice is also recognized as a Device instance this
will work on the Controller where the parameter value is set.

DeviceRef is then used by the Controller-Slave communication methods to
transfer the information. For this to work properly the TestModule is
deepcopied and modified (only DeviceParam) before transmission. Since
DeviceParam can refer to a VirtualDevice object it cannot be deepcopied
so we override it's deepcopy operation to only do a shallowcopy.

It's important to note that at the moment this works in just one
direction: Controller -> Slave and not the other way around. However
this shouldn't be a problem since passing Parameters the other way
around doesn't really make sense.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Common/Parameters.py            | 19 +++++++++++++++++++
 lnst/Controller/MessageDispatcher.py | 15 +++++++++++++++
 lnst/Slave/NetTestSlave.py           | 12 ++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py
index 16385b4..e21c8d6 100644
--- a/lnst/Common/Parameters.py
+++ b/lnst/Common/Parameters.py
@@ -14,6 +14,7 @@ __author__ = """
 olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
+from lnst.Common.DeviceRef import DeviceRef
 from lnst.Common.IpAddress import BaseIpAddress, IpAddress
 from lnst.Common.LnstError import LnstError
 
@@ -91,6 +92,24 @@ class IpParam(Param):
                              "Not {}".format(type(value)))
         self.set = True
 
+class DeviceParam(Param):
+    @Param.val.setter
+    def val(self, value):
+        #runtime import this because the Device class arrives on the Slave
+        #during recipe execution, not during Slave init
+        from lnst.Devices.Device import Device
+        if isinstance(value, Device) or isinstance(value, DeviceRef):
+            self._val = value
+        else:
+            raise ParamError("Value must be a Device or DeviceRef object."
+                             "Not {}".format(type(value)))
+        self.set = True
+
+    def __deepcopy__(self, memo):
+        newone = type(self)()
+        newone.__dict__.update(self.__dict__)
+        return newone
+
 class Parameters(object):
     def __getattribute__(self, name):
         """
diff --git a/lnst/Controller/MessageDispatcher.py b/lnst/Controller/MessageDispatcher.py
index 0f3d23d..1609dd5 100644
--- a/lnst/Controller/MessageDispatcher.py
+++ b/lnst/Controller/MessageDispatcher.py
@@ -15,8 +15,11 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
 import logging
+import copy
 from lnst.Common.ConnectionHandler import send_data
 from lnst.Common.ConnectionHandler import ConnectionHandler
+from lnst.Common.TestModule import BaseTestModule
+from lnst.Common.Parameters import Parameters, DeviceParam
 from lnst.Common.DeviceRef import DeviceRef
 from lnst.Controller.Common import ControllerError
 from lnst.Devices.RemoteDevice import RemoteDevice
@@ -64,6 +67,18 @@ def remote_device_to_deviceref(obj):
         for value in obj:
             new_list.append(remote_device_to_deviceref(value))
         return tuple(new_list)
+    elif isinstance(obj, DeviceParam):
+        new_param = DeviceParam()
+        new_param.val = remote_device_to_deviceref(obj.val)
+        return new_param
+    elif isinstance(obj, Parameters):
+        for param_name, param in obj:
+            setattr(obj, param_name, remote_device_to_deviceref(param))
+        return obj
+    elif isinstance(obj, BaseTestModule):
+        new_test = copy.deepcopy(obj)
+        new_test.params = remote_device_to_deviceref(new_test.params)
+        return new_test
     else:
         return obj
 
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index 99914df..9a29808 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -41,6 +41,8 @@ from lnst.Common.DeviceRef import DeviceRef
 from lnst.Common.LnstError import LnstError
 from lnst.Common.DeviceError import DeviceDeleted
 from lnst.Common.IpAddress import IpAddress
+from lnst.Common.TestModule import BaseTestModule
+from lnst.Common.Parameters import Parameters, DeviceParam
 from lnst.Slave.Job import Job, JobContext
 from lnst.Slave.InterfaceManager import InterfaceManager
 from lnst.Slave.BridgeTool import BridgeTool
@@ -1002,6 +1004,16 @@ def deviceref_to_device(if_manager, obj):
         for value in obj:
             new_list.append(deviceref_to_device(if_manager, value))
         return tuple(new_list)
+    elif isinstance(obj, DeviceParam):
+        obj.val = deviceref_to_device(if_manager, obj.val)
+        return obj
+    elif isinstance(obj, Parameters):
+        for param_name, param in obj:
+            deviceref_to_device(if_manager, param)
+        return obj
+    elif isinstance(obj, BaseTestModule):
+        deviceref_to_device(if_manager, obj.params)
+        return obj
     else:
         return obj
 
-- 
2.13.0

[PATCH v2 47/47] lnst.Tests.IcmpPing: use DeviceParam for iface parameter

From: Ondrej Lichtner <olichtne(a)redhat.com&gt;

We can now use the DeviceParam class instead of the generic Param class
and skip additional parameter check implementation by the __init__
method.

This also modifies the example python recipe to show usage of this
parameter.

Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com&gt;
---
 lnst/Tests/IcmpPing.py            | 20 +++-----------------
 recipes/examples/python_recipe.py |  3 ++-
 2 files changed, 5 insertions(+), 18 deletions(-)

diff --git a/lnst/Tests/IcmpPing.py b/lnst/Tests/IcmpPing.py
index 8b07a2c..f28d2cf 100644
--- a/lnst/Tests/IcmpPing.py
+++ b/lnst/Tests/IcmpPing.py
@@ -1,7 +1,6 @@
 import re
 import logging
-from lnst.Devices import Device
-from lnst.Common.Parameters import IntParam, Param, FloatParam, IpParam
+from lnst.Common.Parameters import IntParam, FloatParam, IpParam, DeviceParam
 from lnst.Common.TestModule import BaseTestModule, TestModuleError
 from lnst.Common.ExecCmd import exec_cmd
 
@@ -10,29 +9,16 @@ class IcmpPing(BaseTestModule):
     dst = IpParam(mandatory=True)
     count = IntParam(default=10)
     interval = FloatParam(default=1.0)
-    iface = Param()
+    iface = DeviceParam()
     size = IntParam()
-
     limit_rate = IntParam(default=80)
 
-    def __init__(self, **kwargs):
-        super(IcmpPing, self).__init__(**kwargs)
-
-        if self.iface.set:
-            if not isinstance(self.iface.val, Device) and\
-               not isinstance(self.iface.val, str):
-                   raise TestModuleError("Invalid 'iface' parameter.")
-
     def _compose_cmd(self):
         cmd = "ping %s" % self.params.dst.val
         cmd += " -c %d" % self.params.count
         cmd += " -i %f" % self.params.interval
         if self.params.iface.set:
-            if isinstance(self.params.iface.val, str):
-                cmd += " -I %s" % self.params.iface
-            elif isinstance(self.params.iface.val, Device):
-                pass
-                # cmd += " -I %s" % iface.val.devname
+           cmd += " -I %s" % self.params.iface.val.name
         if self.params.size.set:
             cmd += " -s %d" % self.params.size
         return cmd
diff --git a/recipes/examples/python_recipe.py b/recipes/examples/python_recipe.py
index b15bc60..09b65c4 100755
--- a/recipes/examples/python_recipe.py
+++ b/recipes/examples/python_recipe.py
@@ -23,7 +23,8 @@ class MyRecipe(BaseRecipe):
         self.matched.m1.eth0.ip_add(IpAddress("192.168.1.1/24"))
         self.matched.m2.eth0.ip_add(IpAddress("192.168.1.2/24"))
         ping_job = self.matched.m1.run(IcmpPing(dst=self.matched.m2.eth0,
-                                                interval=0))
+                                                interval=0,
+                                                iface=self.matched.m1.eth0))
 
 ctl = Controller(debug=1)
 
-- 
2.13.0

[Jiri Pirko]

Fri, May 19, 2017 at 12:12:45PM CEST, olichtne(a)redhat.com wrote:
> From: Ondrej Lichtner <olichtne(a)redhat.com&gt;
>
> v2 changes:
>   add lnst.Common.IpAddress
>     * modify exception string.
>   add lnst.Common.Parameters
>     * add docstring explaining the __getattribute__ magic
>     * make val a property
>     * rename x to attr in Parameters dir() iterations
>     * fix IpParam value setter for string values
>     * modify IpParam Exception string
>   add lnst.Devices
>     * make Device a Meta class and register RemoteDevice as implementing
>       the Device interface
>     * removed unnecessary imports
>     * added MasterDevice class for slave* methods
>     * made master_set to be a 'master' property setter
>     * renamed {add, del}_route to route_{add, del}
>     * renamed set_speed to speed_set
>     * renamed set_autoneg to autoneg_set
>     * removed modprobes where unnecessary
>     * link_stats are now retrieved from netlink message
>     * added link_stats64
>     * OvsBridge calls parent _type_init for modprobe
>     * remove SoftDevice import - not required
>   add lnst.Controller.Host
>     * removed debug pprint import
>     * added docstrings to requested methods
>     * Host::run throws an exception when netns set (not supported yet)
>     * added TODO comments to some parts that need to be implemented
>   add lnst.Controller.MachineMapper
>     * docstring now explains that implementing your own MachineMapper is
>       not supported at this moment
>     * removed debug pprint import
>   add lnst.Controller.Recipe
>     * renamed variable 'x' to 'attr'
>   Machine, NetTestSlave: heavy reimplementation
>     * improved class and module synchronization implementation
>     * join split lines in deviceref_to_device method
>   lnst.Common.Parameters: IpParam accepts Device objects
>     * import Device on demand (Device class arrives on Slave later) and
>       check isinstance Device instead of RemoteDevice (not available on
>       Slave)
>     * fix IpParam Exception string
>
> v2 new commits:
>   Controller, Devices: removing unused imports
>   NetTestSlave: create dynamic lnst.Tests package
>   lnst.Tests: add package docstring
>   lnst.Common.Parameters: add DeviceParam
>   lnst.Tests.IcmpPing: use DeviceParam for iface parameter
>  
Looks good on a quick look. Lets apply this and continue in-tree.


>Including the original cover letter as well:
>
>Hi all,
>
>the long avaited patchset for Python Recipes is here.
>
>At this point there's still quite a lot of work left, but since the current
>state is functional and the tester facing API should be mostly stable I think
>it's a good time to get this merged into upstream so that more people can get
>involved with filling in the missing pieces and can slowly start experimenting
>and porting their XML recipes.
>
>This I know are missing:
>* Recipe run summary - how the Summary should look like was described in the
>  running proposal document, but I didn't manage to start with the
>  implementation yet...
>
>* Porting of all the old test_modules into the lnst.Tests package, so far we
>  only have the IcmpPing module that should still be reworked to be more
>  universal (ip4 and ip6 in one class). Porting test_modules should be fairly
>  easy, but at this point there's no guide how to do it so if you're having
>  trouble feel free to contact me either on irc or email.
>
>* test_tools - we haven't even thought of these yet
>
>* network namespaces - also didn't think about them yet, though I'm hoping
this
>  will be simple
>
>* Ip address and network generators as we've discussed them on the upstream
>  meeting
>
>Please review and provide any comments or features I forgot that should be
>added to the above list.
>
>Regards,
>Ondrej
>
>Ondrej Lichtner (47):
>  add lnst.Common.LnstError
>  add lnst.Common.DeviceError
>  add lnst.Common.DeviceRef
>  add lnst.Common.IpAddress
>  add lnst.Common.Parameters
>  add lnst.Common.TestModule
>  add lnst.Common.JobError
>  add lnst.Controller.Common
>  add lnst.Devices
>  add lnst.Controller.Requirements
>  add lnst.Controller.Job
>  add lnst.Controller.Host
>  add lnst.Controller.Config
>  add lnst.Slave.Config
>  add lnst.Controller.MachineMapper
>  lnst.Controller.Machine: change object initialization
>  add lnst.Controller.MessageDispatcher
>  add lnst.Controller.SlavePoolManager
>  add lnst.Controller.Recipe
>  add lnst.Controller.Controller
>  various files: retype exceptions
>  add lnst.Tests package
>  lnst.Common.Config: remove {controller, slave}_init
>  lnst.Common.Config: remove global lnst_config
>  Slave: use a local config object instead of a global one
>  lnst.Common.Utils: add sha256sum function
>  lnst.Common.ResourceCache: simplification
>  lnst.Controller.CtlSecSocket: remove lnst_config import
>  lnst.Controller.SlaveMachineParser: make standalone
>  add lnst.Common.InterfaceManagerError
>  add lnst.Slave.Job
>  lnst.Slave.InterfaceManager: heavy reimplementation
>  Machine, NetTestSlave: heavy reimplementation
>  lnst.Slave.InterfaceManager: remove Device class implementation
>  add lnst.Controller package imports
>  lnst/__init__.py remove imports
>  lnst.Controller: remove old modules
>  setup.py: add new packages
>  add lnst.Devices.VirtNetCtl
>  lnst.Controller: move VirtUtils to VirtDomainCtl, remove VirtNetCtl
>    class
>  lnst.Common.Parameters: IpParam accepts Device objects
>  add example python_recipe.py script
>  Controller, Devices: removing unused imports
>  NetTestSlave: create dynamic lnst.Tests package
>  lnst.Tests: add package docstring
>  lnst.Common.Parameters: add DeviceParam
>  lnst.Tests.IcmpPing: use DeviceParam for iface parameter
>
> lnst-slave                            |   20 +-
> lnst/Common/Config.py                 |  152 +---
> lnst/Common/DeviceError.py            |   22 +
> lnst/Common/DeviceRef.py              |   19 +
> lnst/Common/ExecCmd.py                |    3 +-
> lnst/Common/InterfaceManagerError.py  |   16 +
> lnst/Common/IpAddress.py              |   99 +++
> lnst/Common/JobError.py               |   22 +
> lnst/Common/LnstError.py              |   18 +
> lnst/Common/NetTestCommand.py         |    5 +-
> lnst/Common/Parameters.py             |  157 ++++
> lnst/Common/ResourceCache.py          |  128 ++--
> lnst/Common/SecureSocket.py           |    3 +-
> lnst/Common/ShellProcess.py           |    3 +-
> lnst/Common/TestModule.py             |   67 ++
> lnst/Common/TestsCommon.py            |    3 +-
> lnst/Common/Utils.py                  |   11 +
> lnst/Controller/Common.py             |   17 +
> lnst/Controller/Config.py             |   99 +++
> lnst/Controller/Controller.py         |  218 ++++++
> lnst/Controller/CtlSecSocket.py       |    2 -
> lnst/Controller/Host.py               |  160 ++++
> lnst/Controller/Job.py                |  196 +++++
> lnst/Controller/Machine.py            | 1356 +++++++--------------------------
> lnst/Controller/MachineMapper.py      |  328 ++++++++
> lnst/Controller/MessageDispatcher.py  |  203 +++++
> lnst/Controller/NetTestController.py  |  620 ---------------
> lnst/Controller/Recipe.py             |   97 +++
> lnst/Controller/RecipeParser.py       |  572 --------------
> lnst/Controller/Requirements.py       |  113 +++
> lnst/Controller/SlaveMachineParser.py |  144 +++-
> lnst/Controller/SlavePool.py          |  648 ----------------
> lnst/Controller/SlavePoolManager.py   |  273 +++++++
> lnst/Controller/Task.py               |    4 +-
> lnst/Controller/VirtDomainCtl.py      |   98 +++
> lnst/Controller/VirtUtils.py          |  268 -------
> lnst/Controller/XmlParser.py          |  188 -----
> lnst/Controller/XmlProcessing.py      |  235 ------
> lnst/Controller/XmlTemplates.py       |  438 -----------
> lnst/Controller/__init__.py           |    3 +
> lnst/Devices/BondDevice.py            |   38 +
> lnst/Devices/BridgeDevice.py          |   33 +
> lnst/Devices/Device.py                |  355 +++++++++
> lnst/Devices/MacvlanDevice.py         |   38 +
> lnst/Devices/MasterDevice.py          |   33 +
> lnst/Devices/OvsBridgeDevice.py       |  115 +++
> lnst/Devices/RemoteDevice.py          |   98 +++
> lnst/Devices/SoftDevice.py            |   51 ++
> lnst/Devices/TeamDevice.py            |   60 ++
> lnst/Devices/VethDevice.py            |   58 ++
> lnst/Devices/VethPair.py              |   24 +
> lnst/Devices/VirtNetCtl.py            |   85 +++
> lnst/Devices/VirtualDevice.py         |   98 +++
> lnst/Devices/VlanDevice.py            |   35 +
> lnst/Devices/VtiDevice.py             |   71 ++
> lnst/Devices/VxlanDevice.py           |   65 ++
> lnst/Devices/__init__.py              |   42 +
> lnst/RecipeCommon/ModuleWrap.py       |   33 +-
> lnst/Slave/Config.py                  |   73 ++
> lnst/Slave/InterfaceManager.py        |  648 ++--------------
> lnst/Slave/Job.py                     |  251 ++++++
> lnst/Slave/NetTestSlave.py            |  846 ++++++++++----------
> lnst/Tests/IcmpPing.py                |   62 ++
> lnst/Tests/__init__.py                |   17 +
> lnst/__init__.py                      |    1 -
> recipes/examples/python_recipe.py     |   32 +
> setup.py                              |    2 +-
> 67 files changed, 4991 insertions(+), 5301 deletions(-)
> create mode 100644 lnst/Common/DeviceError.py
> create mode 100644 lnst/Common/DeviceRef.py
> create mode 100644 lnst/Common/InterfaceManagerError.py
> create mode 100644 lnst/Common/IpAddress.py
> create mode 100644 lnst/Common/JobError.py
> create mode 100644 lnst/Common/LnstError.py
> create mode 100644 lnst/Common/Parameters.py
> create mode 100644 lnst/Common/TestModule.py
> create mode 100644 lnst/Controller/Common.py
> create mode 100644 lnst/Controller/Config.py
> create mode 100644 lnst/Controller/Controller.py
> create mode 100644 lnst/Controller/Host.py
> create mode 100644 lnst/Controller/Job.py
> create mode 100644 lnst/Controller/MachineMapper.py
> create mode 100644 lnst/Controller/MessageDispatcher.py
> delete mode 100644 lnst/Controller/NetTestController.py
> create mode 100644 lnst/Controller/Recipe.py
> delete mode 100644 lnst/Controller/RecipeParser.py
> create mode 100644 lnst/Controller/Requirements.py
> delete mode 100644 lnst/Controller/SlavePool.py
> create mode 100644 lnst/Controller/SlavePoolManager.py
> create mode 100644 lnst/Controller/VirtDomainCtl.py
> delete mode 100644 lnst/Controller/VirtUtils.py
> delete mode 100644 lnst/Controller/XmlParser.py
> delete mode 100644 lnst/Controller/XmlProcessing.py
> delete mode 100644 lnst/Controller/XmlTemplates.py
> create mode 100644 lnst/Devices/BondDevice.py
> create mode 100644 lnst/Devices/BridgeDevice.py
> create mode 100644 lnst/Devices/Device.py
> create mode 100644 lnst/Devices/MacvlanDevice.py
> create mode 100644 lnst/Devices/MasterDevice.py
> create mode 100644 lnst/Devices/OvsBridgeDevice.py
> create mode 100644 lnst/Devices/RemoteDevice.py
> create mode 100644 lnst/Devices/SoftDevice.py
> create mode 100644 lnst/Devices/TeamDevice.py
> create mode 100644 lnst/Devices/VethDevice.py
> create mode 100644 lnst/Devices/VethPair.py
> create mode 100644 lnst/Devices/VirtNetCtl.py
> create mode 100644 lnst/Devices/VirtualDevice.py
> create mode 100644 lnst/Devices/VlanDevice.py
> create mode 100644 lnst/Devices/VtiDevice.py
> create mode 100644 lnst/Devices/VxlanDevice.py
> create mode 100644 lnst/Devices/__init__.py
> create mode 100644 lnst/Slave/Config.py
> create mode 100644 lnst/Slave/Job.py
> create mode 100644 lnst/Tests/IcmpPing.py
> create mode 100644 lnst/Tests/__init__.py
> create mode 100755 recipes/examples/python_recipe.py
>
>-- 
>2.13.0
>_______________________________________________
>LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
>To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org

[Ondrej Lichtner]

On Tue, May 23, 2017 at 04:52:55PM +0200, Jiri Pirko wrote:
>  Fri, May 19, 2017 at 12:12:45PM CEST, olichtne(a)redhat.com wrote:
>  >From: Ondrej Lichtner <olichtne(a)redhat.com&gt;
>  >
>  >v2 changes:
>  >  add lnst.Common.IpAddress
>  >    * modify exception string.
>  >  add lnst.Common.Parameters
>  >    * add docstring explaining the __getattribute__ magic
>  >    * make val a property
>  >    * rename x to attr in Parameters dir() iterations
>  >    * fix IpParam value setter for string values
>  >    * modify IpParam Exception string
>  >  add lnst.Devices
>  >    * make Device a Meta class and register RemoteDevice as implementing
>  >      the Device interface
>  >    * removed unnecessary imports
>  >    * added MasterDevice class for slave* methods
>  >    * made master_set to be a 'master' property setter
>  >    * renamed {add, del}_route to route_{add, del}
>  >    * renamed set_speed to speed_set
>  >    * renamed set_autoneg to autoneg_set
>  >    * removed modprobes where unnecessary
>  >    * link_stats are now retrieved from netlink message
>  >    * added link_stats64
>  >    * OvsBridge calls parent _type_init for modprobe
>  >    * remove SoftDevice import - not required
>  >  add lnst.Controller.Host
>  >    * removed debug pprint import
>  >    * added docstrings to requested methods
>  >    * Host::run throws an exception when netns set (not supported yet)
>  >    * added TODO comments to some parts that need to be implemented
>  >  add lnst.Controller.MachineMapper
>  >    * docstring now explains that implementing your own MachineMapper is
>  >      not supported at this moment
>  >    * removed debug pprint import
>  >  add lnst.Controller.Recipe
>  >    * renamed variable 'x' to 'attr'
>  >  Machine, NetTestSlave: heavy reimplementation
>  >    * improved class and module synchronization implementation
>  >    * join split lines in deviceref_to_device method
>  >  lnst.Common.Parameters: IpParam accepts Device objects
>  >    * import Device on demand (Device class arrives on Slave later) and
>  >      check isinstance Device instead of RemoteDevice (not available on
>  >      Slave)
>  >    * fix IpParam Exception string
>  >
>  >v2 new commits:
>  >  Controller, Devices: removing unused imports
>  >  NetTestSlave: create dynamic lnst.Tests package
>  >  lnst.Tests: add package docstring
>  >  lnst.Common.Parameters: add DeviceParam
>  >  lnst.Tests.IcmpPing: use DeviceParam for iface parameter
>  >
>  
>  Looks good on a quick look. Lets apply this and continue in-tree. 
Sorry this took a while, I was busy with certification training this
week, Pushed to the origin/next branch. I also removed the
origin/next_devel branch.

Next on the agenda is to schedule a meeting where we'll discuss the
Device API so that we can settle on something that will not change
anymore. I also have an idea for Host API that I'd like to discuss.

Next week mostly works for me, I'd prefer later than sooner (I have to
do some catchup in school)... What works for you and/or other members of
the mailing list?

-Ondrej

> 
> 
> >Including the original cover letter as well:
> >
> >Hi all,
> >
> >the long avaited patchset for Python Recipes is here.
> >
> >At this point there's still quite a lot of work left, but since the current
> >state is functional and the tester facing API should be mostly stable I think
> >it's a good time to get this merged into upstream so that more people can
get
> >involved with filling in the missing pieces and can slowly start experimenting
> >and porting their XML recipes.
> >
> >This I know are missing:
> >* Recipe run summary - how the Summary should look like was described in the
> >  running proposal document, but I didn't manage to start with the
> >  implementation yet...
> >
> >* Porting of all the old test_modules into the lnst.Tests package, so far we
> >  only have the IcmpPing module that should still be reworked to be more
> >  universal (ip4 and ip6 in one class). Porting test_modules should be fairly
> >  easy, but at this point there's no guide how to do it so if you're
having
> >  trouble feel free to contact me either on irc or email.
> >
> >* test_tools - we haven't even thought of these yet
> >
> >* network namespaces - also didn't think about them yet, though I'm
hoping this
> >  will be simple
> >
> >* Ip address and network generators as we've discussed them on the upstream
> >  meeting
> >
> >Please review and provide any comments or features I forgot that should be
> >added to the above list.
> >
> >Regards,
> >Ondrej
> >
> >Ondrej Lichtner (47):
> >  add lnst.Common.LnstError
> >  add lnst.Common.DeviceError
> >  add lnst.Common.DeviceRef
> >  add lnst.Common.IpAddress
> >  add lnst.Common.Parameters
> >  add lnst.Common.TestModule
> >  add lnst.Common.JobError
> >  add lnst.Controller.Common
> >  add lnst.Devices
> >  add lnst.Controller.Requirements
> >  add lnst.Controller.Job
> >  add lnst.Controller.Host
> >  add lnst.Controller.Config
> >  add lnst.Slave.Config
> >  add lnst.Controller.MachineMapper
> >  lnst.Controller.Machine: change object initialization
> >  add lnst.Controller.MessageDispatcher
> >  add lnst.Controller.SlavePoolManager
> >  add lnst.Controller.Recipe
> >  add lnst.Controller.Controller
> >  various files: retype exceptions
> >  add lnst.Tests package
> >  lnst.Common.Config: remove {controller, slave}_init
> >  lnst.Common.Config: remove global lnst_config
> >  Slave: use a local config object instead of a global one
> >  lnst.Common.Utils: add sha256sum function
> >  lnst.Common.ResourceCache: simplification
> >  lnst.Controller.CtlSecSocket: remove lnst_config import
> >  lnst.Controller.SlaveMachineParser: make standalone
> >  add lnst.Common.InterfaceManagerError
> >  add lnst.Slave.Job
> >  lnst.Slave.InterfaceManager: heavy reimplementation
> >  Machine, NetTestSlave: heavy reimplementation
> >  lnst.Slave.InterfaceManager: remove Device class implementation
> >  add lnst.Controller package imports
> >  lnst/__init__.py remove imports
> >  lnst.Controller: remove old modules
> >  setup.py: add new packages
> >  add lnst.Devices.VirtNetCtl
> >  lnst.Controller: move VirtUtils to VirtDomainCtl, remove VirtNetCtl
> >    class
> >  lnst.Common.Parameters: IpParam accepts Device objects
> >  add example python_recipe.py script
> >  Controller, Devices: removing unused imports
> >  NetTestSlave: create dynamic lnst.Tests package
> >  lnst.Tests: add package docstring
> >  lnst.Common.Parameters: add DeviceParam
> >  lnst.Tests.IcmpPing: use DeviceParam for iface parameter
> >
> > lnst-slave                            |   20 +-
> > lnst/Common/Config.py                 |  152 +---
> > lnst/Common/DeviceError.py            |   22 +
> > lnst/Common/DeviceRef.py              |   19 +
> > lnst/Common/ExecCmd.py                |    3 +-
> > lnst/Common/InterfaceManagerError.py  |   16 +
> > lnst/Common/IpAddress.py              |   99 +++
> > lnst/Common/JobError.py               |   22 +
> > lnst/Common/LnstError.py              |   18 +
> > lnst/Common/NetTestCommand.py         |    5 +-
> > lnst/Common/Parameters.py             |  157 ++++
> > lnst/Common/ResourceCache.py          |  128 ++--
> > lnst/Common/SecureSocket.py           |    3 +-
> > lnst/Common/ShellProcess.py           |    3 +-
> > lnst/Common/TestModule.py             |   67 ++
> > lnst/Common/TestsCommon.py            |    3 +-
> > lnst/Common/Utils.py                  |   11 +
> > lnst/Controller/Common.py             |   17 +
> > lnst/Controller/Config.py             |   99 +++
> > lnst/Controller/Controller.py         |  218 ++++++
> > lnst/Controller/CtlSecSocket.py       |    2 -
> > lnst/Controller/Host.py               |  160 ++++
> > lnst/Controller/Job.py                |  196 +++++
> > lnst/Controller/Machine.py            | 1356 +++++++--------------------------
> > lnst/Controller/MachineMapper.py      |  328 ++++++++
> > lnst/Controller/MessageDispatcher.py  |  203 +++++
> > lnst/Controller/NetTestController.py  |  620 ---------------
> > lnst/Controller/Recipe.py             |   97 +++
> > lnst/Controller/RecipeParser.py       |  572 --------------
> > lnst/Controller/Requirements.py       |  113 +++
> > lnst/Controller/SlaveMachineParser.py |  144 +++-
> > lnst/Controller/SlavePool.py          |  648 ----------------
> > lnst/Controller/SlavePoolManager.py   |  273 +++++++
> > lnst/Controller/Task.py               |    4 +-
> > lnst/Controller/VirtDomainCtl.py      |   98 +++
> > lnst/Controller/VirtUtils.py          |  268 -------
> > lnst/Controller/XmlParser.py          |  188 -----
> > lnst/Controller/XmlProcessing.py      |  235 ------
> > lnst/Controller/XmlTemplates.py       |  438 -----------
> > lnst/Controller/__init__.py           |    3 +
> > lnst/Devices/BondDevice.py            |   38 +
> > lnst/Devices/BridgeDevice.py          |   33 +
> > lnst/Devices/Device.py                |  355 +++++++++
> > lnst/Devices/MacvlanDevice.py         |   38 +
> > lnst/Devices/MasterDevice.py          |   33 +
> > lnst/Devices/OvsBridgeDevice.py       |  115 +++
> > lnst/Devices/RemoteDevice.py          |   98 +++
> > lnst/Devices/SoftDevice.py            |   51 ++
> > lnst/Devices/TeamDevice.py            |   60 ++
> > lnst/Devices/VethDevice.py            |   58 ++
> > lnst/Devices/VethPair.py              |   24 +
> > lnst/Devices/VirtNetCtl.py            |   85 +++
> > lnst/Devices/VirtualDevice.py         |   98 +++
> > lnst/Devices/VlanDevice.py            |   35 +
> > lnst/Devices/VtiDevice.py             |   71 ++
> > lnst/Devices/VxlanDevice.py           |   65 ++
> > lnst/Devices/__init__.py              |   42 +
> > lnst/RecipeCommon/ModuleWrap.py       |   33 +-
> > lnst/Slave/Config.py                  |   73 ++
> > lnst/Slave/InterfaceManager.py        |  648 ++--------------
> > lnst/Slave/Job.py                     |  251 ++++++
> > lnst/Slave/NetTestSlave.py            |  846 ++++++++++----------
> > lnst/Tests/IcmpPing.py                |   62 ++
> > lnst/Tests/__init__.py                |   17 +
> > lnst/__init__.py                      |    1 -
> > recipes/examples/python_recipe.py     |   32 +
> > setup.py                              |    2 +-
> > 67 files changed, 4991 insertions(+), 5301 deletions(-)
> > create mode 100644 lnst/Common/DeviceError.py
> > create mode 100644 lnst/Common/DeviceRef.py
> > create mode 100644 lnst/Common/InterfaceManagerError.py
> > create mode 100644 lnst/Common/IpAddress.py
> > create mode 100644 lnst/Common/JobError.py
> > create mode 100644 lnst/Common/LnstError.py
> > create mode 100644 lnst/Common/Parameters.py
> > create mode 100644 lnst/Common/TestModule.py
> > create mode 100644 lnst/Controller/Common.py
> > create mode 100644 lnst/Controller/Config.py
> > create mode 100644 lnst/Controller/Controller.py
> > create mode 100644 lnst/Controller/Host.py
> > create mode 100644 lnst/Controller/Job.py
> > create mode 100644 lnst/Controller/MachineMapper.py
> > create mode 100644 lnst/Controller/MessageDispatcher.py
> > delete mode 100644 lnst/Controller/NetTestController.py
> > create mode 100644 lnst/Controller/Recipe.py
> > delete mode 100644 lnst/Controller/RecipeParser.py
> > create mode 100644 lnst/Controller/Requirements.py
> > delete mode 100644 lnst/Controller/SlavePool.py
> > create mode 100644 lnst/Controller/SlavePoolManager.py
> > create mode 100644 lnst/Controller/VirtDomainCtl.py
> > delete mode 100644 lnst/Controller/VirtUtils.py
> > delete mode 100644 lnst/Controller/XmlParser.py
> > delete mode 100644 lnst/Controller/XmlProcessing.py
> > delete mode 100644 lnst/Controller/XmlTemplates.py
> > create mode 100644 lnst/Devices/BondDevice.py
> > create mode 100644 lnst/Devices/BridgeDevice.py
> > create mode 100644 lnst/Devices/Device.py
> > create mode 100644 lnst/Devices/MacvlanDevice.py
> > create mode 100644 lnst/Devices/MasterDevice.py
> > create mode 100644 lnst/Devices/OvsBridgeDevice.py
> > create mode 100644 lnst/Devices/RemoteDevice.py
> > create mode 100644 lnst/Devices/SoftDevice.py
> > create mode 100644 lnst/Devices/TeamDevice.py
> > create mode 100644 lnst/Devices/VethDevice.py
> > create mode 100644 lnst/Devices/VethPair.py
> > create mode 100644 lnst/Devices/VirtNetCtl.py
> > create mode 100644 lnst/Devices/VirtualDevice.py
> > create mode 100644 lnst/Devices/VlanDevice.py
> > create mode 100644 lnst/Devices/VtiDevice.py
> > create mode 100644 lnst/Devices/VxlanDevice.py
> > create mode 100644 lnst/Devices/__init__.py
> > create mode 100644 lnst/Slave/Config.py
> > create mode 100644 lnst/Slave/Job.py
> > create mode 100644 lnst/Tests/IcmpPing.py
> > create mode 100644 lnst/Tests/__init__.py
> > create mode 100755 recipes/examples/python_recipe.py
> >
> >-- 
> >2.13.0
> >_______________________________________________
> >LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> >To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org

[Jiri Pirko]

Fri, May 26, 2017 at 03:33:41PM CEST, olichtne(a)redhat.com wrote:
> On Tue, May 23, 2017 at 04:52:55PM +0200, Jiri Pirko wrote:
> > Fri, May 19, 2017 at 12:12:45PM CEST, olichtne(a)redhat.com wrote:
> > >From: Ondrej Lichtner <olichtne(a)redhat.com&gt;
> > >
> > >v2 changes:
> > >  add lnst.Common.IpAddress
> > >    * modify exception string.
> > >  add lnst.Common.Parameters
> > >    * add docstring explaining the __getattribute__ magic
> > >    * make val a property
> > >    * rename x to attr in Parameters dir() iterations
> > >    * fix IpParam value setter for string values
> > >    * modify IpParam Exception string
> > >  add lnst.Devices
> > >    * make Device a Meta class and register RemoteDevice as implementing
> > >      the Device interface
> > >    * removed unnecessary imports
> > >    * added MasterDevice class for slave* methods
> > >    * made master_set to be a 'master' property setter
> > >    * renamed {add, del}_route to route_{add, del}
> > >    * renamed set_speed to speed_set
> > >    * renamed set_autoneg to autoneg_set
> > >    * removed modprobes where unnecessary
> > >    * link_stats are now retrieved from netlink message
> > >    * added link_stats64
> > >    * OvsBridge calls parent _type_init for modprobe
> > >    * remove SoftDevice import - not required
> > >  add lnst.Controller.Host
> > >    * removed debug pprint import
> > >    * added docstrings to requested methods
> > >    * Host::run throws an exception when netns set (not supported yet)
> > >    * added TODO comments to some parts that need to be implemented
> > >  add lnst.Controller.MachineMapper
> > >    * docstring now explains that implementing your own MachineMapper is
> > >      not supported at this moment
> > >    * removed debug pprint import
> > >  add lnst.Controller.Recipe
> > >    * renamed variable 'x' to 'attr'
> > >  Machine, NetTestSlave: heavy reimplementation
> > >    * improved class and module synchronization implementation
> > >    * join split lines in deviceref_to_device method
> > >  lnst.Common.Parameters: IpParam accepts Device objects
> > >    * import Device on demand (Device class arrives on Slave later) and
> > >      check isinstance Device instead of RemoteDevice (not available on
> > >      Slave)
> > >    * fix IpParam Exception string
> > >
> > >v2 new commits:
> > >  Controller, Devices: removing unused imports
> > >  NetTestSlave: create dynamic lnst.Tests package
> > >  lnst.Tests: add package docstring
> > >  lnst.Common.Parameters: add DeviceParam
> > >  lnst.Tests.IcmpPing: use DeviceParam for iface parameter
> > >
> > 
> > Looks good on a quick look. Lets apply this and continue in-tree.
>
> Sorry this took a while, I was busy with certification training this
> week, Pushed to the origin/next branch. I also removed the
> origin/next_devel branch.
>
> Next on the agenda is to schedule a meeting where we'll discuss the
> Device API so that we can settle on something that will not change
> anymore. I also have an idea for Host API that I'd like to discuss.
>
> Next week mostly works for me, I'd prefer later than sooner (I have to
> do some catchup in school)... What works for you and/or other members of
> the mailing list? 
Works for me any time this week, please schedule.