[PATCH 0/7] Machine Provisioning Support

[Radek Pazdera]

This patch series introduces full provisioning support to LNST.

It allows you to create a pool of machines at the controler and then
create tests that can provision machines from that pool and run tests
on them instead of supplying their own configuration for machines.

This idea is useful, because it separates the hardware configuration
from the recipe, so you don't need to rely on any specific hardware
details other than that are important for your test.

To use provisioning, all you need to do is set the 'provisioned' parameter
of the <machineconfig> that you want to have provisioned. It is vital to
point out, that mixing of provisioned machines with your own machineconfigs
is not supported at the moment (and probably will never be). You can chose
either to provision the whole setup or none of it.

Here is an example of provisioned machine's config:

    <machineconfig provisioned="true">
        <netdevices>
            <netdevice network="ttnet" phys_id="1"
type="eth"/>
            <netdevice network="ttnet" phys_id="2"/>
        </netdevices>
    </machineconfig>

This machineconfig indicates, that it wants a machine that has two interfaces
plugged into the same network and at least one needs to be of type ethernet.

    You can specify even more information about the machine, such as,
        * hostname (do this when you want a specific machine from the pool)
        * libvirt_domain
        * device hwaddr
        * device type
    More of these parameters will probably be added in the future.

LNST will take this config, find a suitable provisioner machine and run your
test there.

The pool of machines is a set of directories (can be configured in the
lnst.conf file) with a machineconfig per file.

For instance this is a machine from my pool fedora16-clone.xml:
    <machineconfig>
        <info hostname="192.168.122.10"
libvirt_domain="Fedora16-clone"
               rootpass="redhat"/>
        <netdevices>
            <netdevice network="tnet" phys_id="1"
type="eth" hwaddr="52:54:00:e4:75:16"/>
            <netdevice network="tnet" phys_id="2"
type="eth" hwaddr="52:54:00:91:01:9c"/>
            <netdevice network="tnet" phys_id="3"
type="eth" hwaddr="52:54:00:94:b7:32"/>
            <netdevice network="tnet" phys_id="4"
type="eth" hwaddr="52:54:00:f3:a1:b4"/>
        </netdevices>
    </machineconfig>


I will write some more detailed documentation and how to to the wiki next
week.

I hope I didn't break anything, if so, please let me know and I'll fix it
as soon as I can :).

Cheers,

Radek

Radek Pazdera (7):
  XmlProcessing: Normalize paths in XmlDomTreeInit
  XmlProcessing: Adding possition info to DOM root
  XmlProcessing: Adding default handler to XmlParser
  NetTestSlave: Adding method for turning of ifaces
  MachinePool: Adding provisioning support
  NetTestController: Adding support for provisioning
  NetTestParse: Adding support for provisioning

 Common/XmlProcessing.py      |   16 +-
 NetTest/MachinePool.py       |  488 +++++++++++++++++++++++++++++++++++++++++-
 NetTest/NetTestController.py |   31 +++
 NetTest/NetTestParse.py      |  153 +++++++++++++-
 NetTest/NetTestSlave.py      |    9 +
 5 files changed, 680 insertions(+), 17 deletions(-)

-- 
1.7.7.6

[PATCH 1/7] XmlProcessing: Normalize paths in XmlDomTreeInit

[Radek Pazdera]

Path passed to XmlTreeInit.parse_string() is now stripped
to a filename only, so the error messages don't get too long.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 Common/XmlProcessing.py |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Common/XmlProcessing.py b/Common/XmlProcessing.py
index 2ae5f4c..e225309 100644
--- a/Common/XmlProcessing.py
+++ b/Common/XmlProcessing.py
@@ -111,11 +111,11 @@ class XmlDomTreeInit:
         return self.parse_string(xml_text, filename)
 
     def parse_string(self, xml_text, filename="xml_string"):
-        self._filename = filename
+        self._filename = os.path.basename(filename)
         try:
             dom = parseString(xml_text, self._sax)
         except sax.SAXParseException, err:
-            pos = {"file": filename,
+            pos = {"file": self._filename,
                    "line": err.getLineNumber(),
                    "col": err.getColumnNumber()}
             exc = XmlProcessingError(err.getMessage())
-- 
1.7.7.6

[PATCH 2/7] XmlProcessing: Adding possition info to DOM root

[Radek Pazdera]

Error reporting didn't work correctly when an error happended on
the root level of the tree.

This patch adds the position information to the root as well.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 Common/XmlProcessing.py |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/Common/XmlProcessing.py b/Common/XmlProcessing.py
index e225309..0f1aa0c 100644
--- a/Common/XmlProcessing.py
+++ b/Common/XmlProcessing.py
@@ -122,6 +122,8 @@ class XmlDomTreeInit:
             exc.set_pos(pos)
             raise exc
 
+        pos = {"file": self._filename, "line": 0, "col":
0}
+        dom.parse_position = pos
         return dom
 
 
-- 
1.7.7.6

[PATCH 3/7] XmlProcessing: Adding default handler to XmlParser

[Radek Pazdera]

A new parameter was added to the XmlParser's _process_child_nodes
method. This allows you to specify a handler, that will be called
when no handler was registered for a trigger.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 Common/XmlProcessing.py |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/Common/XmlProcessing.py b/Common/XmlProcessing.py
index 0f1aa0c..dc0fbbf 100644
--- a/Common/XmlProcessing.py
+++ b/Common/XmlProcessing.py
@@ -138,7 +138,8 @@ class XmlParser(object):
         used in error reporting.
     """
 
-    def _process_child_nodes(self, parent, scheme, params=None):
+    def _process_child_nodes(self, parent, scheme, params=None,
+                                    default_handler=None):
         child_nodes = parent.childNodes
 
         if not params:
@@ -153,6 +154,8 @@ class XmlParser(object):
                 if node_name in scheme:
                     handler = scheme[node_name]
                     self._process_node(node, handler, params)
+                elif default_handler:
+                    self._process_node(node, default_handler, params)
                 else:
                     msg = "Unexpected '%s' tag under '%s'" %
(node_name,
                                                         parent.nodeName)
@@ -263,7 +266,7 @@ class RecipeParser(XmlParser):
         handler(**args)
 
     def _process_child_nodes(self, node, scheme, params=None,
-                                    new_ns_level=True):
+                                default_handler=None, new_ns_level=True):
         scheme["define"] = self._define_handler
 
         if not params:
@@ -273,7 +276,8 @@ class RecipeParser(XmlParser):
             self._template_proc.add_namespace_level()
 
         parent = super(RecipeParser, self)
-        result = parent._process_child_nodes(node, scheme, params)
+        result = parent._process_child_nodes(node, scheme, params,
+                                                default_handler)
 
         if new_ns_level:
             self._template_proc.drop_namespace_level()
-- 
1.7.7.6

[PATCH 4/7] NetTestSlave: Adding method for turning of ifaces

[Radek Pazdera]

This patch introduces a new method to the NetTestSlave
class -- set_device_down(hwaddr).

This method can be used for turning off devices through rpc.
It will be useful for initialization of provisioned machines
through machine pool.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 NetTest/NetTestSlave.py |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/NetTest/NetTestSlave.py b/NetTest/NetTestSlave.py
index a375f12..765094f 100644
--- a/NetTest/NetTestSlave.py
+++ b/NetTest/NetTestSlave.py
@@ -23,6 +23,7 @@ from NetConfig.NetConfigDevice import NetConfigDeviceAllCleanup
 from NetTest.NetTestCommand import NetTestCommandContext, NetTestCommand,
CommandException
 from Common.Utils import die_when_parent_die
 from Common.NetUtils import scan_netdevs, test_tcp_connection
+from Common.ExecCmd import exec_cmd
 
 DefaultRPCPort = 9999
 
@@ -60,6 +61,14 @@ class NetTestSlaveXMLRPC:
 
         return netdevs
 
+    def set_device_down(self, hwaddr):
+        devs = self.get_devices_by_hwaddr(hwaddr)
+
+        for dev in devs:
+            exec_cmd("ip link set %s down" % dev["name"])
+
+        return True
+
     def get_interface_info(self, if_id):
         if_config = self._netconfig.get_interface_config(if_id)
         info = {}
-- 
1.7.7.6

[PATCH 5/7] MachinePool: Adding provisioning support

[Radek Pazdera]

The stub implementation for MachinePool class was extended with
a full provisioning support. It can be now used to supply machine
setups for requests from the NetTestController.

A new class -- SetupMapper was added to aid MachinePool in the matter
of matching requested setups against the pool.

SetupMapper is able to find a mapping between a requested topology
and available pool resources. The resulting setup can be then used
to run tests.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 NetTest/MachinePool.py |  488 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 478 insertions(+), 10 deletions(-)

diff --git a/NetTest/MachinePool.py b/NetTest/MachinePool.py
index 2712f09..dd235a1 100644
--- a/NetTest/MachinePool.py
+++ b/NetTest/MachinePool.py
@@ -2,6 +2,8 @@
 This module contains implementaion of MachinePool class that
 can be used to maintain a cluster of test machines.
 
+These machines can be provisioned and used in test recipes.
+
 Copyright 2012 Red Hat, Inc.
 Licensed under the GNU General Public License, version 2 as
 published by the Free Software Foundation; see COPYING for details.
@@ -14,16 +16,22 @@ rpazdera(a)redhat.com (Radek Pazdera)
 import logging
 import os
 import re
+import copy
+from xml.dom import minidom
 from Common.XmlProcessing import XmlDomTreeInit
-from NetTestParse import MachineConfigParse
+from NetTest.NetTestParse import MachineConfigParse
 
 class MachinePool:
-    """ This class is responsible for managing test machines
-        that are available at controler and can be used for
-        testing via template matching
+    """
+    This class is responsible for managing test machines that
+    are available at the controler and can be used for testing.
     """
 
-    _machines = {}
+    _map = {}
+    _pool = {}
+
+    _machine_matches = []
+    _network_matches = []
 
     def __init__(self, pool_dirs):
         for pool_dir in pool_dirs:
@@ -51,11 +59,471 @@ class MachinePool:
             parser.set_machine(machine_id, machine)
 
             machineconfig = dom.getElementsByTagName("machineconfig")[0]
+            machine["dom_node_ref"] = machineconfig
+
             parser.parse(machineconfig)
-            self._machines[machine_id] = machine
+            self._pool[machine_id] = machine
+
+    def provision_setup(self, setup_requirements):
+        """
+        This method will try to map a dictionary of machines'
+        requirements to a pool of machines that is available to
+        this instance.
+
+        :param templates: Setup request (dict of required machines)
+        :type templates: dict
+
+        :return: XML machineconfigs of requested machines
+        :rtype: dict
+        """
+
+        mapper = SetupMapper()
+        self._map = mapper.map_setup(setup_requirements, self._pool)
+
+        if self._map == None:
+            return None
+
+        configs = {}
+        for m_id in self._map["machines"]:
+            configs[m_id] = self._get_mapped_machineconfig_xml(m_id)
+
+        return configs
+
+    def get_provisioner_id(self, m_id):
+        try:
+            return self._get_machine_mapping(m_id)
+        except KeyError:
+            return None
+
+    def get_provisioner(self, m_id):
+        try:
+            p_id = self._get_machine_mapping(m_id)
+            return self._pool[p_id]
+        except KeyError:
+            return None
+
+    def _get_machine_mapping(self, m_id):
+        return self._map["machines"][m_id]["target"]
+
+    def _get_interface_mapping(self, m_id, if_id):
+        return self._map["machines"][m_id]["interfaces"][if_id]
+
+    def _get_network_mapping(self, net_id):
+        return self._map["networks"][net_id]
+
+    def _get_mapped_machineconfig_xml(self, tm_id):
+        pm_id = self._get_machine_mapping(tm_id)
+
+        dom = minidom.Document()
+
+        mcfg = dom.createElement("machineconfig")
+
+        info = dom.createElement("info")
+        supported = ["hostname", "libvirt_domain",
"rpcport", "rootpass"]
+        for attr_name, attr_val in self._pool[pm_id]["info"].iteritems():
+            if attr_name in supported:
+                info.setAttribute(attr_name, attr_val)
+        mcfg.appendChild(info)
+
+        netdevices = dom.createElement("netdevices")
+        mcfg.appendChild(netdevices)
+
+        if_map = self._map["machines"][tm_id]["interfaces"]
+        for t_if, p_if in if_map.iteritems():
+            dev_info = self._pool[pm_id]["netdevices"][p_if]
+
+            dev_node = dom.createElement("netdevice")
+
+            dev_node.setAttribute("phys_id", t_if)
+            dev_node.setAttribute("type", dev_info["type"])
+            dev_node.setAttribute("hwaddr", dev_info["hwaddr"])
+
+            for t_net, p_net in self._map["networks"].iteritems():
+                if dev_info["network"] == p_net:
+                    dev_node.setAttribute("network", t_net)
+                    break
+
+            netdevices.appendChild(dev_node)
+
+        return mcfg.toxml()
+
+
+class SetupMapper:
+    """
+    This class can be used for matching machine setups against
+    a pool of interconnected machines. SetupMapper will search
+    through the pool for suitable matches of the requested
+    setup and return the mapping between the two.
+
+    Here we explain some terminology that is used consistently
+    through the whole class:
+
+        nc = neighbour connection; a 3-tuple that describes connection
+             to an adjacent machine with the information which interface
+             is used.
+
+             (neighbour_id, network_id, iface_id)
+
+        nc_list = list of neighbour connections; it is a building block
+                  of a topology
+
+        topology = dictionary of nc_lists; it is an analogy to a adjacency
+                   list -- represenation of a graph. It is modified so it's
+                   able to represent non-graph structures such as our
+                   topology
+
+        match = a correspondence between a machine, interface or a network
+                from template and from a pool. It's a 2-tuple.
+
+                (template_machine, pool_machine)
+                (template_machines_iface, pool_machines_iface)
+                (template_network, pool_network)
+    """
+
+    _machine_map = None
+    _iface_map = None
+    _network_map = None
+
+    _template_machines = None
+    _pool_machines = None
+
+    @staticmethod
+    def _get_topology(machine_configs):
+        """
+        This function will generate an adjacenty list from machine
+        configuration dictionary. It can handle both machines and
+        templates.
+
+        :param machine_configs: dictionary of machines in the topology
+        :type machines_configs: dict
+
+        :return: Topology - neighbour connection list (adjacency list-like
+            data structure
+        :rtype: dict
+        """
+
+        networks = {}
+        for m_id, m_config in machine_configs.iteritems():
+            for dev_id, dev_info in m_config["netdevices"].iteritems():
+                net = dev_info["network"]
+                if not net in networks:
+                    networks[net] = []
+                networks[net].append((m_id, dev_id))
+
+        topology = {}
+        for m_id, m_config in machine_configs.iteritems():
+            topology[m_id] = []
+            for net_name, net in networks.iteritems():
+                devs_in_net = []
+                for dev_id, dev_info in m_config["netdevices"].iteritems():
+                    if dev_info["network"] == net_name:
+                        devs_in_net.append(dev_id)
+
+                for neighbour in net:
+                    n_m_id = neighbour[0]
+                    n_dev_id = neighbour[1]
+                    if n_m_id != m_id:
+                        for dev_in_net in devs_in_net:
+                            nc = (n_m_id, net_name, dev_in_net)
+                            if not nc in topology[m_id]:
+                                topology[m_id].append(nc)
+
+        return topology
+
+    @staticmethod
+    def _is_match_valid(template_id, pool_id, matches):
+        for match in matches:
+            if (match[0] == template_id and match[1] != pool_id) or \
+               (match[0] != template_id and match[1] == pool_id):
+                return False
+
+        return True
+
+    def _is_machine_match_valid(self, template_id, pool_id):
+        """
+        Method for checking validity of a proposed match between
+        two machines.
+
+        :param template_id: machine id in template setup
+        :type template_id: string
+
+        :param pool_id: machine id in pool setup
+        :type pool_id: string
+
+        :return: True/False indicating the validity of this match
+        :rtype: Bool
+        """
+
+        template_machine = self._template_machines[template_id]
+        pool_machine = self._pool_machines[pool_id]
+
+        # check machine properties
+        properties = ["hostname", "libvirt_domain"]
+        for prop_name, prop_value in template_machine["info"].iteritems():
+            if prop_name in properties:
+                if pool_machine["info"][prop_name] != prop_value:
+                    return False
+
+        # check number of devices
+        tm_ndevs = len(template_machine["netdevices"])
+        pm_ndevs = len(pool_machine["netdevices"])
+        if tm_ndevs > pm_ndevs:
+            return False
+
+        return self._is_match_valid(template_id, pool_id,
+                                    self._machine_map)
+
+    def _is_network_match_valid(self, template_id, pool_id):
+        """
+        Method for checking validity of a proposed match between
+        two network names.
+
+        :param template_id: network id in template setup
+        :type template_id: string
+
+        :param pool_id: network id in pool setup
+        :type pool_id: string
+
+        :return: True/False indicating the validity of this match
+        :rtype: Bool
+        """
+
+        return self._is_match_valid(template_id, pool_id,
+                                    self._network_map)
+
+    def _is_if_match_valid(self, tm_id, t_if_id, pm_id, pm_if_id):
+        """
+        Check if matching of one interface on another is valid.
+        This functions checks the parameters of those two interfaces,
+        such as type, mac address etc.
+
+        :param tm_id: template machine id
+        :type tm_id: string
+
+        :param tm_if_id: template machine's interface id
+        :type tm_if_id: string
+
+        :param pm_id: pool machine id
+        :type tm_id: string
+
+        :param pm_if_id: pool machine's interface id
+        :type pm_if_id: string
+
+        :return: True/False indicating the validity of this match
+        :rtype: Bool
+        """
+
+        t_if = self._template_machines[tm_id]["netdevices"][t_if_id]
+        p_if = self._pool_machines[pm_id]["netdevices"][pm_if_id]
+
+        properties = ["type", "hwaddr"]
+        for prop_name, prop_value in t_if.iteritems():
+            if prop_name in properties:
+                if p_if[prop_name] != prop_value:
+                    return False
+
+        return True
+
+    @staticmethod
+    def _get_node_with_most_neighbours(topology):
+        max_machine = None
+        max_len = 0
+        for machine, nc_list in topology.iteritems():
+            if len(nc_list) > max_len:
+                max_machine = machine
+                max_len = len(nc_list)
+
+        return max_machine
+
+    def _get_possible_matches(self, machine, pool_topology):
+        possible_matches = set(pool_topology.keys())
+        impossible_matches = set()
+
+        for match in self._machine_map:
+            if match[0] == machine and not match[1] in impossible_matches:
+                # in case the machine has already been matched,
+                # return the corresponding match as an only option
+                return set([match[1]])
+            else:
+                # in case the machine has been matched to a different
+                # one in pool, remove it from possible matches
+                impossible_matches.add(match[1])
+
+        return possible_matches - impossible_matches
+
+    def _get_nc_matches(self, tm_id, tm_nc_list, pm_id, pm_nc_list):
+        """
+        Return all possible ways of matching list of neighbours of a template
+        machine on another list of neighbours of a pool machine. This function
+        also keeps in mind what matches already exist and avoids conflicts.
+
+        :param tm_nc_list: short for template machine neighbour connection list
+        :type tm_nc_list: list
+
+        :param pm_nc_list: short for pool machine neighbour connection list
+        :type pm_nc_list: list
+
+        :return: List of all possible mapping updates that are result of a
+            successful matching between the machine's neighbour connections.
+        :rtype: list
+        """
+
+        mmap = self._machine_map
+        nmap = self._network_map
+        mapping_update = []
+
+        # recursion stop condition
+        if len(tm_nc_list) <= 1:
+            for nc in pm_nc_list:
+                if self._is_machine_match_valid(tm_nc_list[0][0], nc[0]) and \
+                   self._is_network_match_valid(tm_nc_list[0][1], nc[1]) and \
+                   self._is_if_match_valid(tm_id, tm_nc_list[0][2],
+                                                pm_id, nc[2]):
+                    mapping = self._get_mapping_update(tm_nc_list[0], nc)
+                    mapping_update.append(mapping)
+
+            return mapping_update
+
+        for nc in pm_nc_list:
+            if self._is_machine_match_valid(tm_nc_list[0][0], nc[0]) and \
+               self._is_network_match_valid(tm_nc_list[0][1], nc[1]) and \
+               self._is_if_match_valid(tm_id, tm_nc_list[0][2], pm_id, nc[2]):
+                recently_added = self._get_mapping_update(tm_nc_list[0], nc)
+                self._save_nc_match(recently_added)
+
+                new_pm_nc_list = copy.deepcopy(pm_nc_list)
+                new_pm_nc_list.remove(nc)
+
+                possible_matches = self._get_nc_matches(tm_id, tm_nc_list[1:],
+                                                        pm_id, new_pm_nc_list)
+                self._revert_nc_match(recently_added)
+                for possible_match in possible_matches:
+                    mapping = (recently_added[0] + possible_match[0],
+                               recently_added[1] + possible_match[1],
+                               recently_added[2] + possible_match[2])
+                    mapping_update.append(mapping)
+
+        return mapping_update
+
+    def _get_mapping_update(self, template_nc, pool_nc):
+        i = [(template_nc[2], pool_nc[2])]
+
+        m = []
+        m_match = (template_nc[0], pool_nc[0])
+        if not m_match in self._machine_map:
+            m.append(m_match)
+
+        n = []
+        n_match = (template_nc[1], pool_nc[1])
+        if not n_match in self._network_map:
+            n.append(n_match)
+
+        return (i, m, n)
+
+    def _save_nc_match(self, nc_match):
+        self._machine_map |= set(nc_match[1])
+        self._network_map |= set(nc_match[2])
+
+    def _revert_nc_match(self, nc_match):
+        self._machine_map -= set(nc_match[1])
+        self._network_map -= set(nc_match[2])
+
+    def _format_map_dict(self, machine_map, network_map):
+        map_dict = {}
+
+        map_dict["machines"] = {}
+        for match in machine_map:
+            if_map = {}
+            for if_match in match[2]:
+                if_map[if_match[0]] = if_match[1]
+            map_dict["machines"][match[0]] = {"target": match[1],
+                                              "interfaces": if_map}
+
+        map_dict["networks"] = {}
+        for match in network_map:
+            map_dict["networks"][match[0]] = match[1]
+
+        return map_dict
+
+    def map_setup(self, template_machines, pool_machines):
+        """
+        Attempt to match template topology to pool topology.
+
+        :param template_topology: dictionary of machine templates to be matched
+            against the pool
+        :type template_topology: dict
+
+        :param pool_topology: dictionary o machine structures that will be used
+            as a pool of available machines
+        :type pool_topology: dict
+
+        :return: 2-tuple (machine_map, network_map). Machine map is a list of
+            3-tuples (template_machine_id, pool_machine_id, iface_map),
+            both iface_map and network_map are list of mappings between
+            matched equivalents in template and pool.
+        :rtype: tuple containing machine and network mappings
+        """
+
+        self._machine_map = set()
+        self._iface_map = {}
+        self._network_map = set()
+
+        self._template_machines = template_machines
+        self._pool_machines = pool_machines
+
+        template_topology = self._get_topology(template_machines)
+        pool_topology = self._get_topology(pool_machines)
+
+        if self._map_setup(template_topology, pool_topology):
+            machine_map = [(tm, pm, self._iface_map[tm]) \
+                            for tm, pm in self._machine_map]
+            network_map = list(self._network_map)
+            return self._format_map_dict(machine_map, network_map)
+        else:
+            return None
+
+    def _map_setup(self, template_topology, pool_topology):
+
+        if len(template_topology) <= 0:
+            return True
+
+        mmap = self._machine_map
+        nmap = self._network_map
+
+        # by choosing to match the biggest nodes in the topology first,
+        # we optimize the amount of time it takes to find out that the
+        # topology cannot be matched (in most cases)
+        machine = self._get_node_with_most_neighbours(template_topology)
+
+        possible_matches = self._get_possible_matches(machine, pool_topology)
+        for possible_match in possible_matches:
+            if self._is_match_valid(machine, possible_match, mmap):
+                mmap.add((machine, possible_match))
+            else:
+                continue
+
+            template_nc_list = template_topology[machine]
+            pool_nc_list = pool_topology[possible_match]
+
+            nc_matches = self._get_nc_matches(machine, template_nc_list,
+                                                possible_match, pool_nc_list)
+            for nc_match in nc_matches:
+                self._save_nc_match(nc_match)
+                self._iface_map[machine] = nc_match[0]
+
+                new_pool = copy.deepcopy(pool_topology)
+                del new_pool[possible_match]
+
+                new_template = copy.deepcopy(template_topology)
+                del new_template[machine]
+
+                if not self._map_setup(new_template, new_pool):
+                    self._revert_nc_match(nc_match)
+                    del self._iface_map[machine]
+                    continue
+                else:
+                    return True
 
-    def get_machines(self):
-        return self._machines
+            mmap.discard((machine, possible_match))
 
-    def match_setup(self, templates):
-        pass
+        return False
-- 
1.7.7.6

[PATCH 6/7] NetTestController: Adding support for provisioning

[Radek Pazdera]

This patch adds some necessary changes to the Controller, so it supports
machine provisioning from the pool.

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 NetTest/NetTestController.py |   31 +++++++++++++++++++++++++++++++
 1 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/NetTest/NetTestController.py b/NetTest/NetTestController.py
index 2629c40..2e67dd6 100644
--- a/NetTest/NetTestController.py
+++ b/NetTest/NetTestController.py
@@ -67,6 +67,10 @@ class NetTestController:
         ntparse.register_event_handler("interface_config_ready",
                                         self._prepare_interface)
 
+        ntparse.register_event_handler("provisioning_requirements_ready",
+                                        self._prepare_provisioning)
+
+
         self._ntparse = ntparse
 
     def _get_machineinfo(self, machine_id):
@@ -93,6 +97,24 @@ class NetTestController:
                                         session.command, status)
         raise Exception("Session Die.")
 
+    def _prepare_provisioning(self):
+        provisioning = self._recipe["provisioning"]
+        if len(provisioning["setup_requirements"]) <= 0:
+            return
+
+        mp = self._machine_pool
+        alloc_machines =
mp.provision_setup(provisioning["setup_requirements"])
+        if alloc_machines == None:
+            msg = "This setup cannot be provisioned with the current pool."
+            raise NetTestError(msg)
+
+        provisioning["allocated_machines"] = alloc_machines
+
+        logging.info("Provisioning initialized")
+        for machine in alloc_machines.keys():
+            provisioner = mp.get_provisioner_id(machine)
+            logging.info("  machine %s uses %s" % (machine, provisioner))
+
     def _prepare_device(self, machine_id, dev_id):
         info = self._get_machineinfo(machine_id)
         rpc = self._get_machinerpc(machine_id)
@@ -220,6 +242,15 @@ class NetTestController:
 
         rpc = self._get_machinerpc(machine_id)
 
+        # Some additional initialization is necessary in case the
+        # underlying machine is provisioned from the pool
+        prov_id = self._machine_pool.get_provisioner_id(machine_id)
+        if prov_id:
+            provisioner = self._machine_pool.get_provisioner(machine_id)
+            logging.info("Initializing provisioned system (%s)" % prov_id)
+            for device in provisioner["netdevices"].itervalues():
+                rpc.set_device_down(device["hwaddr"])
+
         if self._docleanup:
             rpc.machine_cleanup()
 
-- 
1.7.7.6

[PATCH 7/7] NetTestParse: Adding support for provisioning

[Radek Pazdera]

This commit introduces some changes to NetTestParse to aid machine
provisioning from pool. The parsing was split into two passes

FirstPass - ignores everything except the templates and stores them
SecondPass - conventional parsing, as it were before the changes

Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com&gt;
---
 NetTest/NetTestParse.py |  153 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 151 insertions(+), 2 deletions(-)

diff --git a/NetTest/NetTestParse.py b/NetTest/NetTestParse.py
index bdf1721..706d39c 100644
--- a/NetTest/NetTestParse.py
+++ b/NetTest/NetTestParse.py
@@ -35,9 +35,116 @@ class NetTestParse(RecipeParser):
         dom_init = XmlDomTreeInit()
         rp = self._rp
         xml_dom = dom_init.parse_string(rp.to_str(), rp.abs_path())
-        self._parse(xml_dom)
 
-    def _parse(self, xml_dom):
+        first_pass = FirstPass(self)
+        first_pass.parse(xml_dom)
+
+        try:
+            self._trigger_event("provisioning_requirements_ready", {})
+        except Exception, exc:
+            raise XmlProcessingError(str(exc), xml_dom)
+
+        # process templates if used in the recipe before
+        # proceeding to the second pass
+        provisioning = self._recipe["provisioning"]
+
+        dom_init = XmlDomTreeInit()
+        for name, req in provisioning["setup_requirements"].iteritems():
+            xml = provisioning["allocated_machines"][name]
+            dom = dom_init.parse_string(xml, "pool_machine_config")
+
+            # replace the original machineconfig template with
+            # the config recived from MachinePool
+            original_node = req["dom_node_ref"]
+            parent = original_node.parentNode
+            replacement_node = dom.getElementsByTagName("machineconfig")[0]
+
+            parent.replaceChild(replacement_node, original_node)
+
+        second_pass = SecondPass(self)
+        second_pass.parse(xml_dom)
+
+
+class FirstPass(RecipeParser):
+    """
+    Purpose of the first pass through the recipe is to detect
+    machineconfig templates that need to be mapped on pool machines.
+
+    The purpose is generic, but the first pass exist only to
+    detect templates at the moment.
+    """
+
+    _has_nontemplates = False
+
+    def parse(self, node):
+        self._recipe["provisioning"] = {}
+        self._recipe["provisioning"]["setup_requirements"] = {}
+
+        if node.nodeType == node.DOCUMENT_NODE:
+            scheme = {"nettestrecipe": self._nettestrecipe}
+            self._process_child_nodes(node, scheme,
+                        default_handler=self._ignore_tag)
+        else:
+            raise XmlProcessingError("Passed object is not a XML document")
+
+    def _nettestrecipe(self, node, params):
+        scheme = {"machines": self._machines}
+        self._process_child_nodes(node, scheme,
+                    default_handler=self._ignore_tag)
+
+    def _machines(self, node, params):
+        scheme = {"machine": self._machine}
+        self._process_child_nodes(node, scheme,
+                    default_handler=self._ignore_tag)
+
+    def _machine(self, node, params):
+        params = {}
+        params["id"] = self._get_attribute(node, "id")
+
+        scheme = {"machineconfig": self._machineconfig}
+        self._process_child_nodes(node, scheme, params,
+                    default_handler=self._ignore_tag)
+
+    def _machineconfig(self, node, params):
+        machine_req =
self._recipe["provisioning"]["setup_requirements"]
+        is_template = False
+        if self._has_attribute(node, "provisioned"):
+            is_template = self._get_attribute(node, "provisioned", bool_it)
+
+        msg = "Cannot mix non-template machine configs with templates"
+        mixed_err = XmlProcessingError(msg, node)
+
+        if is_template:
+            if self._has_nontemplates:
+                raise mixed_err
+
+            m_id = params["id"]
+            template = {}
+            template["info"] = {}
+            template["netdevices"] = {}
+            machine_req[m_id] = template
+
+            subparser = MachineConfigTemplateParse(self)
+            subparser.set_template(template)
+            subparser.parse(node)
+        else:
+            self._has_nontemplates = True
+            if len(machine_req) > 0:
+                raise mixed_err
+
+    def _ignore_tag(self, node, params):
+        pass
+
+
+class SecondPass(RecipeParser):
+    """
+    Second pass makes sure all recognized values from the recipe
+    are properly saved into the self._recipe.
+
+    This is where the real parsing is done.
+    """
+
+    def parse(self, xml_dom):
         if xml_dom.nodeType == xml_dom.DOCUMENT_NODE:
             scheme = {"nettestrecipe": self._nettestrecipe}
             self._process_child_nodes(xml_dom, scheme)
@@ -112,6 +219,48 @@ class MachineParse(RecipeParser):
         subparser.set_machine(self._id, self._machine)
         subparser.parse(node)
 
+class MachineConfigTemplateParse(RecipeParser):
+    _template = None
+
+    def set_template(self, tmp_dict):
+        self._template = tmp_dict
+
+    def parse(self, node):
+        self._template["dom_node_ref"] = node
+
+        scheme = {"info": self._info,
+                  "netdevices": self._netdevices}
+        self._process_child_nodes(node, scheme)
+
+    def _info(self, node, params):
+        template = self._template
+        info = template["info"]
+
+        if self._has_attribute(node, "hostname"):
+            info["hostname"] = self._get_attribute(node, "hostname")
+
+        if self._has_attribute(node, "libvirt_domain"):
+            info["libvirt_domain"] = self._get_attribute(node,
+                                                "libvirt_domain")
+
+    def _netdevices(self, node, params):
+        scheme = {"netdevice": self._netdevice}
+        self._process_child_nodes(node, scheme)
+
+    def _netdevice(self, node, params):
+        template = self._template
+        phys_id = self._get_attribute(node, "phys_id")
+
+        dev = template["netdevices"][phys_id] = {}
+        dev["network"] = self._get_attribute(node, "network")
+
+        if self._has_attribute(node, "type"):
+            dev["type"] = self._get_attribute(node, "type")
+
+        if self._has_attribute(node, "hwaddr"):
+            hwaddr = self._get_attribute(node, "hwaddr")
+            dev["hwaddr"] = normalize_hwaddr(hwaddr)
+
 
 class MachineConfigParse(RecipeParser):
     _machine_id = None
-- 
1.7.7.6

[Jiri Pirko]

I'm scarred of this :)

I'll apply this over the weekend after I give this some testing.

Thanks!

Thu, Oct 04, 2012 at 02:23:59PM CEST, rpazdera(a)redhat.com wrote:
> This patch series introduces full provisioning support to LNST.
>
> It allows you to create a pool of machines at the controler and then
> create tests that can provision machines from that pool and run tests
> on them instead of supplying their own configuration for machines.
>
> This idea is useful, because it separates the hardware configuration
> from the recipe, so you don't need to rely on any specific hardware
> details other than that are important for your test.
>
> To use provisioning, all you need to do is set the 'provisioned' parameter
> of the <machineconfig> that you want to have provisioned. It is vital to
> point out, that mixing of provisioned machines with your own machineconfigs
> is not supported at the moment (and probably will never be). You can chose
> either to provision the whole setup or none of it.
>
> Here is an example of provisioned machine's config:
>
>     <machineconfig provisioned="true">
>         <netdevices>
>             <netdevice network="ttnet" phys_id="1"
> type="eth"/>
>             <netdevice network="ttnet" phys_id="2"/>
>         </netdevices>
>     </machineconfig>
>
> This machineconfig indicates, that it wants a machine that has two interfaces
> plugged into the same network and at least one needs to be of type ethernet.
>
>     You can specify even more information about the machine, such as,
>         * hostname (do this when you want a specific machine from the pool)
>         * libvirt_domain
>         * device hwaddr
>         * device type
>     More of these parameters will probably be added in the future.
>
> LNST will take this config, find a suitable provisioner machine and run your
> test there.
>
> The pool of machines is a set of directories (can be configured in the
> lnst.conf file) with a machineconfig per file.
>
> For instance this is a machine from my pool fedora16-clone.xml:
>     <machineconfig>
>         <info hostname="192.168.122.10"
> libvirt_domain="Fedora16-clone"
>                rootpass="redhat"/>
>         <netdevices>
>             <netdevice network="tnet" phys_id="1"
> type="eth" hwaddr="52:54:00:e4:75:16"/>
>             <netdevice network="tnet" phys_id="2"
> type="eth" hwaddr="52:54:00:91:01:9c"/>
>             <netdevice network="tnet" phys_id="3"
> type="eth" hwaddr="52:54:00:94:b7:32"/>
>             <netdevice network="tnet" phys_id="4"
> type="eth" hwaddr="52:54:00:f3:a1:b4"/>
>         </netdevices>
>     </machineconfig>
>
>
> I will write some more detailed documentation and how to to the wiki next
> week.
>
> I hope I didn't break anything, if so, please let me know and I'll fix it
> as soon as I can :).
>
> Cheers,
>
> Radek
>
> Radek Pazdera (7):
>   XmlProcessing: Normalize paths in XmlDomTreeInit
>   XmlProcessing: Adding possition info to DOM root
>   XmlProcessing: Adding default handler to XmlParser
>   NetTestSlave: Adding method for turning of ifaces
>   MachinePool: Adding provisioning support
>   NetTestController: Adding support for provisioning
>   NetTestParse: Adding support for provisioning
>
>  Common/XmlProcessing.py      |   16 +-
>  NetTest/MachinePool.py       |  488 +++++++++++++++++++++++++++++++++++++++++-
>  NetTest/NetTestController.py |   31 +++
>  NetTest/NetTestParse.py      |  153 +++++++++++++-
>  NetTest/NetTestSlave.py      |    9 +
>  5 files changed, 680 insertions(+), 17 deletions(-)
>
> -- 
> 1.7.7.6
>
> _______________________________________________
> LNST-developers mailing list
> LNST-developers(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/lnst-developers

[Radek Pazdera]

On Thu, Oct 04, 2012 at 05:58:03PM +0200, Jiri Pirko wrote:
> I'm scarred of this :)
>
> I'll apply this over the weekend after I give this some testing. 
Ok :). If you have any objections, let me know.

One more thing, I gave it a little thought over the weekend and
maybe it would be nice to rename the <machineconfig provisioned="true">
to something like <machinerequires> or <machinerequirements>.

I know we originally agreed on <machineconfig template="true">. I tried
to use it, but it didn't actually capture the purpose. And we already
have one thing associated with "templates" in there (template functions).

> Thanks!
>
> Thu, Oct 04, 2012 at 02:23:59PM CEST, rpazdera(a)redhat.com wrote:
> >This patch series introduces full provisioning support to LNST.
> >
> >It allows you to create a pool of machines at the controler and then
> >create tests that can provision machines from that pool and run tests
> >on them instead of supplying their own configuration for machines.
> >
> >This idea is useful, because it separates the hardware configuration
> >from the recipe, so you don't need to rely on any specific hardware
> >details other than that are important for your test.
> >
> >To use provisioning, all you need to do is set the 'provisioned' parameter
> >of the <machineconfig> that you want to have provisioned. It is vital to
> >point out, that mixing of provisioned machines with your own machineconfigs
> >is not supported at the moment (and probably will never be). You can chose
> >either to provision the whole setup or none of it.
> >
> >Here is an example of provisioned machine's config:
> >
> >    <machineconfig provisioned="true">
> >        <netdevices>
> >            <netdevice network="ttnet" phys_id="1"
> type="eth"/>
> >            <netdevice network="ttnet" phys_id="2"/>
> >        </netdevices>
> >    </machineconfig>
> >
> >This machineconfig indicates, that it wants a machine that has two interfaces
> >plugged into the same network and at least one needs to be of type ethernet.
> >
> >    You can specify even more information about the machine, such as,
> >        * hostname (do this when you want a specific machine from the pool)
> >        * libvirt_domain
> >        * device hwaddr
> >        * device type
> >    More of these parameters will probably be added in the future.
> >
> >LNST will take this config, find a suitable provisioner machine and run your
> >test there.
> >
> >The pool of machines is a set of directories (can be configured in the
> >lnst.conf file) with a machineconfig per file.
> >
> >For instance this is a machine from my pool fedora16-clone.xml:
> >    <machineconfig>
> >        <info hostname="192.168.122.10"
> libvirt_domain="Fedora16-clone"
> >               rootpass="redhat"/>
> >        <netdevices>
> >            <netdevice network="tnet" phys_id="1"
> type="eth" hwaddr="52:54:00:e4:75:16"/>
> >            <netdevice network="tnet" phys_id="2"
> type="eth" hwaddr="52:54:00:91:01:9c"/>
> >            <netdevice network="tnet" phys_id="3"
> type="eth" hwaddr="52:54:00:94:b7:32"/>
> >            <netdevice network="tnet" phys_id="4"
> type="eth" hwaddr="52:54:00:f3:a1:b4"/>
> >        </netdevices>
> >    </machineconfig>
> >
> >
> >I will write some more detailed documentation and how to to the wiki next
> >week.
> >
> >I hope I didn't break anything, if so, please let me know and I'll fix it
> >as soon as I can :).
> >
> >Cheers,
> >
> >Radek
> >
> >Radek Pazdera (7):
> >  XmlProcessing: Normalize paths in XmlDomTreeInit
> >  XmlProcessing: Adding possition info to DOM root
> >  XmlProcessing: Adding default handler to XmlParser
> >  NetTestSlave: Adding method for turning of ifaces
> >  MachinePool: Adding provisioning support
> >  NetTestController: Adding support for provisioning
> >  NetTestParse: Adding support for provisioning
> >
> > Common/XmlProcessing.py      |   16 +-
> > NetTest/MachinePool.py       |  488 +++++++++++++++++++++++++++++++++++++++++-
> > NetTest/NetTestController.py |   31 +++
> > NetTest/NetTestParse.py      |  153 +++++++++++++-
> > NetTest/NetTestSlave.py      |    9 +
> > 5 files changed, 680 insertions(+), 17 deletions(-)
> >
> >-- 
> >1.7.7.6
> >
> >_______________________________________________
> >LNST-developers mailing list
> &gt;LNST-developers(a)lists.fedorahosted.org
> >https://lists.fedorahosted.org/mailman/listinfo/lnst-developers