[PATCH 00/20] phase3 vxlan tests and InterfaceManager improvements

[Ondrej Lichtner]

From: Ondrej Lichtner

Hi,

what follows is a very large patch set with a lot of stuff happening in it.
Normally I would have split it up into at least 2 logical parts, but
unfortunately most of the changes introduced in the patch set is dependent on
each other. The reasoning behind all of the code should be nicely explained in
the commit messages, if you have any additional questions don't hesitate to ask.

One thing that I want to explain is the creation of a whole new concept of
the Controller Device Database, mirroring the one on the Slave so that I could
work on testing OvS vxlans. For now the implementation is just that and nothing
more. At the moment this makes the Controller look very ugly seeing as there's
both an Interface object and a Device object representing the same NIC but with
different information.

In the future I want to merge these two classes, mostly by replacing the current
Interface objects with the new Device objects extended with setters and other
features required for proper configuration. This however is a large
restructuring of the code and it would either depend or heavily conflict with
the implementation of python recipes which is why I chose to just do a very
quick and ugly implementation of a parallel concept that I can use for now while
we wait to merge in the Python recipes implementation.

v2: fixed a traceback when configuring veth devices - reported by jtluka, the 3
updated commits are:
InterfaceManager: fix rescan_devices method
InterfaceManager: extend Device::if_data method
InterfaceManager: add if_deleted update messages

Ondrej Lichtner (20):
  NetConfigDevice: use link_up in Vxlan and Vlan devices
  NetUtils: remove FIXME from scan_netdevs
  NetUtils: add address information to the interface scan
  InterfaceManager: handle RTM_{NEW, DEL}ADDR nl_messages
  InterfaceManager: fix rescan_devices method
  InterfaceManager: extend Device::if_data method
  NetTestSlave: remake get_devices and add get_device
  Machine: construct a Device database reflecting InterfaceManager state
  Task: add DeviceAPI as extension of Machine device database
  InterfaceManager: add if_deleted update messages
  InterfaceManager: add Device objects to tmp_mapping sooner
  InterfaceManager: rescan devices before checking if name used
  InterfaceManager: make assign_name_generic publicly accessible
  NetConfigCommon: add function get_slave_options
  NetConfigDevice: extend NetConfigDeviceOvsBridge
  RecipeParser, schema-recipe: extend OvS bridge XML configuration
  PerfRepoUtils: add perfrepo_baseline_to_dict function
  Netperf: allow bind option in client
  RecipeCommon: add ModuleWrap
  regression_tests: add phase3, starting with vxlan

lnst/Common/NetUtils.py                            |  20 +-
 lnst/Controller/Machine.py                         | 148 +++++++++++-
 lnst/Controller/NetTestController.py               |   3 +
 lnst/Controller/PerfRepoUtils.py                   |  16 ++
 lnst/Controller/RecipeParser.py                    | 123 ++++++++--
 lnst/Controller/Task.py                            |  58 ++++-
 lnst/Controller/Wizard.py                          |  14 +-
 lnst/RecipeCommon/ModuleWrap.py                    | 179 +++++++++++++++
 lnst/Slave/InterfaceManager.py                     | 150 ++++++++++---
 lnst/Slave/NetConfigCommon.py                      |   7 +
 lnst/Slave/NetConfigDevice.py                      | 107 ++++++++-
 lnst/Slave/NetTestSlave.py                         |  13 +-
 .../regression_tests/phase3/2_virt_ovs_vxlan.py    | 247 +++++++++++++++++++++
 .../regression_tests/phase3/2_virt_ovs_vxlan.xml   | 138 ++++++++++++
 .../regression_tests/phase3/novirt_ovs_vxlan.py    | 201 +++++++++++++++++
 .../regression_tests/phase3/novirt_ovs_vxlan.xml   |  86 +++++++
 .../regression_tests/phase3/vxlan_multicast.xml    |  66 ++++++
 recipes/regression_tests/phase3/vxlan_remote.xml   |  67 ++++++
 recipes/regression_tests/phase3/vxlan_test.py      | 247 +++++++++++++++++++++
 schema-recipe.rng                                  |  76 +++++--
 test_modules/Netperf.py                            |   5 +
 21 files changed, 1875 insertions(+), 96 deletions(-)
 create mode 100644 lnst/RecipeCommon/ModuleWrap.py
 create mode 100644 recipes/regression_tests/phase3/2_virt_ovs_vxlan.py
 create mode 100644 recipes/regression_tests/phase3/2_virt_ovs_vxlan.xml
 create mode 100644 recipes/regression_tests/phase3/novirt_ovs_vxlan.py
 create mode 100644 recipes/regression_tests/phase3/novirt_ovs_vxlan.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_multicast.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_remote.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_test.py

[PATCH v2 01/20] NetConfigDevice: use link_up in Vxlan and Vlan devices

[Ondrej Lichtner]

From: Ondrej Lichtner

Calling parent_dev.up() from Vxlan caused crashes due to deadlocks.
Using the recently added method link_up() solves this issue. I modified
this for the Vlan device as well since it's cleaner than explicitly
calling an ip command.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/NetConfigDevice.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py
index dd053a6..6ef3f1c 100644
--- a/lnst/Slave/NetConfigDevice.py
+++ b/lnst/Slave/NetConfigDevice.py
@@ -280,7 +280,7 @@ class NetConfigDeviceVlan(NetConfigDeviceGeneric):
     def up(self):
         parent_id = get_slaves(self._dev_config)[0]
         parent_dev = self._if_manager.get_mapped_device(parent_id)
-        exec_cmd("ip link set %s up" % parent_dev.get_name())
+        parent_dev.link_up()

super(NetConfigDeviceVlan, self).up()

@@ -332,7 +332,7 @@ class NetConfigDeviceVxlan(NetConfigDeviceGeneric):
         if len(slaves) == 1:
             parent_id = get_slaves(self._dev_config)[0]
             parent_dev = self._if_manager.get_mapped_device(parent_id)
-            parent_dev.up()
+            parent_dev.link_up()

super(NetConfigDeviceVxlan, self).up()

[PATCH v2 02/20] NetUtils: remove FIXME from scan_netdevs

[Ondrej Lichtner]

From: Ondrej Lichtner

If there's no IFLA_ADDRESS present in the netlink message, set the
hwaddr to None instead of calling normalize_hwaddr. Removing the FIXME
message.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Common/NetUtils.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/lnst/Common/NetUtils.py b/lnst/Common/NetUtils.py
index 48a82a3..150746c 100644
--- a/lnst/Common/NetUtils.py
+++ b/lnst/Common/NetUtils.py
@@ -33,18 +33,13 @@ def scan_netdevs():
             new_link["netlink_msg"] = part
             new_link["index"] = part["index"]
             new_link["name"] = part.get_attr("IFLA_IFNAME")
-            #
-            # FIXME:
-            #
-            # nlmsg.get_attr() returns None if there is no
-            # such attribute in the NLA chain; if hwaddr is None,
-            # normalize_hwaddr(hwaddr) will raise AttributeError(),
-            # since None has no upper(). The issue is that the
-            # AttributeError() will be a bit unrelated to the
-            # root cause, and since that it will be confusing.
-            #
+
             hwaddr = part.get_attr("IFLA_ADDRESS")
-            new_link["hwaddr"] = normalize_hwaddr(hwaddr)
+            if hwaddr:
+                new_link["hwaddr"] = normalize_hwaddr(hwaddr)
+            else:
+                new_link["hwaddr"] = None
+
             scan.append(new_link)
     except:
         raise

[PATCH v2 03/20] NetUtils: add address information to the interface scan

[Ondrej Lichtner]

From: Ondrej Lichtner

This adds a list of RTM_NEWADDR netlink messages to the interface scan
results, carrying information about the NICs ip addresses.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Common/NetUtils.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lnst/Common/NetUtils.py b/lnst/Common/NetUtils.py
index 150746c..9fd4732 100644
--- a/lnst/Common/NetUtils.py
+++ b/lnst/Common/NetUtils.py
@@ -40,6 +40,9 @@ def scan_netdevs():
             else:
                 new_link["hwaddr"] = None

+            addrs = ipr.get_addr(index=new_link["index"])
+            new_link["ip_addrs"] = addrs
+
             scan.append(new_link)
     except:
         raise

[PATCH v2 04/20] InterfaceManager: handle RTM_{NEW, DEL}ADDR nl_messages

[Ondrej Lichtner]

From: Ondrej Lichtner

InterfaceManager and Device objects are now capable of handling ip
address related netlink messages. The Device object stores addresses in
a list as dictionaries. This should change in the future to either use
the Python ipaddress module or to use our own custom objects for ip
addresses.

Additionally the Device objects now generate an update message for the
controller in the init_netlink method as well and we'll be sending an
update message for all devices, even ones that aren't mapped based on
the currently running recipe.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 76 +++++++++++++++++++++++++++++++++---------
 1 file changed, 60 insertions(+), 16 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index edeb5bc..1999ed8 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -24,9 +24,13 @@ from pyroute2 import IPRSocket
 try:
     from pyroute2.netlink.iproute import RTM_NEWLINK
     from pyroute2.netlink.iproute import RTM_DELLINK
+    from pyroute2.netlink.iproute import RTM_NEWADDR
+    from pyroute2.netlink.iproute import RTM_DELADDR
 except ImportError:
     from pyroute2.iproute import RTM_NEWLINK
     from pyroute2.iproute import RTM_DELLINK
+    from pyroute2.iproute import RTM_NEWADDR
+    from pyroute2.iproute import RTM_DELADDR

class IfMgrError(Exception):
     pass
@@ -95,7 +99,7 @@ class InterfaceManager(object):
             self._handle_netlink_msg(msg)

def _handle_netlink_msg(self, msg):
-        if msg['header']['type'] == RTM_NEWLINK:
+        if msg['header']['type'] in [RTM_NEWLINK, RTM_NEWADDR, RTM_DELADDR]:
             if msg['index'] in self._devices:
                 update_msg = self._devices[msg['index']].update_netlink(msg)
                 if update_msg != None:
@@ -103,9 +107,8 @@ class InterfaceManager(object):
                         if if_index == msg['index']:
                             update_msg["if_id"] = if_id
                             break
-                    if "if_id" in update_msg:
-                        self._server_handler.send_data_to_ctl(update_msg)
-            else:
+                    self._server_handler.send_data_to_ctl(update_msg)
+            elif msg['header']['type'] == RTM_NEWLINK:
                 dev = None
                 for if_id, d in self._tmp_mapping.items():
                     d_cfg = d.get_conf_dict()
@@ -116,8 +119,16 @@ class InterfaceManager(object):
                         break
                 if dev == None:
                     dev = Device(self)
-                dev.init_netlink(msg)
+                update_msg = dev.init_netlink(msg)
                 self._devices[msg['index']] = dev
+
+                if update_msg != None:
+                    for if_id, if_index in self._id_mapping.iteritems():
+                        if if_index == msg['index']:
+                            update_msg["if_id"] = if_id
+                            break
+                    self._server_handler.send_data_to_ctl(update_msg)
+
         elif msg['header']['type'] == RTM_DELLINK:
             if msg['index'] in self._devices:
                 dev = self._devices[msg['index']]
@@ -294,7 +305,7 @@ class Device(object):
         self._name = None
         self._conf = None
         self._conf_dict = None
-        self._ip = None
+        self._ip_addrs = []
         self._ifi_type = None
         self._state = None
         self._master = {"primary": None, "other": []}
@@ -312,7 +323,7 @@ class Device(object):
         self._hwaddr = normalize_hwaddr(nl_msg.get_attr("IFLA_ADDRESS"))
         self._name = nl_msg.get_attr("IFLA_IFNAME")
         self._state = nl_msg.get_attr("IFLA_OPERSTATE")
-        self._ip = None #TODO
+        self._ip_addrs = []
         self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
         self._netns = None
         self._mtu = nl_msg.get_attr("IFLA_MTU")
@@ -322,13 +333,18 @@ class Device(object):

self._initialized = True

+        #return an update message that will be sent to the controller
+        return {"type": "if_update",
+                "if_data": self.get_if_data()}
+
     def update_netlink(self, nl_msg):
-        if self._if_index == nl_msg['index']:
+        if self._if_index != nl_msg['index']:
+            return None
+        if nl_msg['header']['type'] == RTM_NEWLINK:
             self._ifi_type = nl_msg['ifi_type']
             self._hwaddr = normalize_hwaddr(nl_msg.get_attr("IFLA_ADDRESS"))
             self._name = nl_msg.get_attr("IFLA_IFNAME")
             self._state = nl_msg.get_attr("IFLA_OPERSTATE")
-            self._ip = None #TODO
             self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
             self._mtu = nl_msg.get_attr("IFLA_MTU")

@@ -354,11 +370,29 @@ class Device(object):
                 self._driver = self._ethtool_get_driver()

self._initialized = True
-
-            #return an update message that will be sent to the controller
-            return {"type": "if_update",
-                    "if_data": self.get_if_data()}
-        return None
+        elif nl_msg['header']['type'] == RTM_NEWADDR:
+            scope = nl_msg['scope']
+            addr_val = nl_msg.get_attr('IFA_ADDRESS')
+            prefix_len = str(nl_msg['prefixlen'])
+            addr = {"addr": addr_val,
+                    "prefix": prefix_len,
+                    "scope": scope}
+            if self.find_addrs(addr) == []:
+                self._ip_addrs.append(addr)
+        elif nl_msg['header']['type'] == RTM_DELADDR:
+            scope = nl_msg['scope']
+            addr_val = nl_msg.get_attr('IFA_ADDRESS')
+            prefix_len = str(nl_msg['prefixlen'])
+            addr = {"addr": addr_val,
+                    "prefix": prefix_len,
+                    "scope": scope}
+            matching_addrs = self.find_addrs(addr)
+            for ip_addr in matching_addrs:
+                self._ip_addrs.remove(ip_addr)
+
+        #return an update message that will be sent to the controller
+        return {"type": "if_update",
+                "if_data": self.get_if_data()}

def del_link(self):
         if self._master["primary"]:
@@ -377,6 +411,13 @@ class Device(object):
             if dev != None:
                 dev.del_master(self._if_index)

+    def find_addrs(self, addr_spec):
+        ret = []
+        for addr in self._ip_addrs:
+            if addr_spec.viewitems() <= addr.viewitems():
+                ret.append(addr)
+        return ret
+
     def get_if_index(self):
         return self._if_index

@@ -386,8 +427,11 @@ class Device(object):
     def get_name(self):
         return self._name

-    def get_ip_conf(self):
-        return self._ip
+    def get_ips(self):
+        return self._ip_addrs
+
+    def clear_ips(self):
+        self._ip_addrs = []

def is_configured(self):
         return self._configured

[PATCH v2 05/20] InterfaceManager: fix rescan_devices method

[Ondrej Lichtner]

From: Ondrej Lichtner

The rescan_devices method destroyed the entire _devices dictionary and
recreated it from scratch. This caused problems when a Device object was
carying {Net, Nm}ConfigDevice object that's needed for device
deconfiguration.

Instead of recreating the whole dictionary from scratch we now keep it
and just call update_netlink methods for the existing Device objects.
New objects are added as before and Device objects for NICs that don't
exist anymore are removed. No deconfigration is called since there's
nothing to do when the device doesn't exist anymore.

Finally the method now also recognizes RTM_NEWADDR messages received
from the scan_netdevs method and sends them to the Device objects.

v2: don't remove Devices that were moved to a different network
namespace

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 1999ed8..21c4fc0 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -14,6 +14,7 @@ [e-mail] (Ondrej Lichtner)

import re
 import select
+import logging
 from lnst.Slave.NetConfigDevice import NetConfigDevice
 from lnst.Slave.NetConfigCommon import get_option
 from lnst.Common.NetUtils import normalize_hwaddr
@@ -85,14 +86,37 @@ class InterfaceManager(object):
         return self._nl_socket

def rescan_devices(self):
-        self._devices = {}
+        devices_to_remove = self._devices.keys()
         devs = scan_netdevs()
         for dev in devs:
             if dev['index'] not in self._devices:
-                device = Device(self)
+                device = None
+                for if_id, d in self._tmp_mapping.items():
+                    d_cfg = d.get_conf_dict()
+                    if d_cfg["name"] == dev["name"]:
+                        device = d
+                        self._id_mapping[if_id] = dev['index']
+                        del self._tmp_mapping[if_id]
+                        break
+                if device == None:
+                    device = Device(self)
                 device.init_netlink(dev['netlink_msg'])
-
                 self._devices[dev['index']] = device
+            else:
+                self._devices[dev['index']].update_netlink(dev['netlink_msg'])
+                devices_to_remove.remove(dev['index'])
+
+            self._devices[dev['index']].clear_ips()
+            for addr_msg in dev['ip_addrs']:
+                self._devices[dev['index']].update_netlink(addr_msg)
+        for i in devices_to_remove:
+            if self._devices[i].get_netns() != None:
+                continue
+
+            dev_name = self._devices[i].get_name()
+            logging.debug("Deleting Device with if_index %d, name %s because "\
+                          "it doesn't exist anymore." % (i, dev_name))
+            del self._devices[i]

def handle_netlink_msgs(self, msgs):
         for msg in msgs:

[PATCH v2 06/20] InterfaceManager: extend Device::if_data method

[Ondrej Lichtner]

From: Ondrej Lichtner

The if_data method now returns a complete dictionary describing the
Device object.

This includes a nonbackwards compatible change of renaming "devname" to
just "name".

v2: peer attribute is a reference to a Device object. The Controller
can't use it and it's not pickleable. So send just the if_index of the
peer

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Machine.py     |  2 +-
 lnst/Slave/InterfaceManager.py | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 5d8db7d..08df4d8 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -695,7 +695,7 @@ class Interface(object):

def update(self, if_data):
         self.set_hwaddr(if_data["hwaddr"])
-        self.set_devname(if_data["devname"])
+        self.set_devname(if_data["name"])
         self._mtu = if_data["mtu"]
         self._driver = if_data["driver"]

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 21c4fc0..821d8fa 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -607,8 +607,16 @@ class Device(object):
             return None

def get_if_data(self):
-        if_data = {"devname": self._name,
+        if_data = {"if_index": self._if_index,
                    "hwaddr": self._hwaddr,
+                   "name": self._name,
+                   "ip_addrs": self._ip_addrs,
+                   "ifi_type": self._ifi_type,
+                   "state": self._state,
+                   "master": self._master,
+                   "slaves": self._slaves,
+                   "netns": self._netns,
+                   "peer": self._peer.get_if_index() if self._peer else None,
                    "mtu": self._mtu,
                    "driver": self._driver}
         return if_data

[PATCH v2 07/20] NetTestSlave: remake get_devices and add get_device

[Ondrej Lichtner]

From: Ondrej Lichtner

The get_device slave method now returns a list of ALL devices that the
InterfaceManager sees and their data returned by the recently updated
get_if_data method.

This commit also adds a get_device method that returns the result of
get_if_data for one single Device, identified by the interface index in
the kernel.

Additionally this updates the Wizard class as well since it's the only
one that was using the get_devices method. It now uses the
get_devices_by_params method and selects just the DOWN && eth
interfaces.

This is a backwards incompatible change.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Wizard.py  | 14 ++++++++------
 lnst/Slave/NetTestSlave.py | 13 +++++++++----
 2 files changed, 17 insertions(+), 10 deletions(-)

diff --git a/lnst/Controller/Wizard.py b/lnst/Controller/Wizard.py
index 59c3732..dc82c21 100644
--- a/lnst/Controller/Wizard.py
+++ b/lnst/Controller/Wizard.py
@@ -54,7 +54,7 @@ class Wizard:
                 machine_interfaces = self._get_machine_interfaces(sock)
                 sock.close()

-                if machine_interfaces == {}:
+                if machine_interfaces == []:
                     sys.stderr.write("No suitable interfaces found on the host "
                                      "'%s:%s'\n" % (hostname, port))
                 elif machine_interfaces is not None:
@@ -81,7 +81,7 @@ class Wizard:
             machine_interfaces = self._get_machine_interfaces(sock)
             sock.close()

-            if machine_interfaces == {}:
+            if machine_interfaces == []:
                 sys.stderr.write("No suitable interfaces found on the host "
                                  "'%s:%s'\n" % (hostname, port))
             elif machine_interfaces is not None:
@@ -144,7 +144,7 @@ class Wizard:
             machine_interfaces = self._get_machine_interfaces(sock)
             sock.close()

-            if machine_interfaces is {}:
+            if machine_interfaces is []:
                 sys.stderr.write("No suitable interfaces found on the host "
                                  "'%s:%s'\n" % (hostname, port))
                 continue
@@ -280,7 +280,7 @@ class Wizard:
                     pool_dir=None, filename=None, mode=None,
                     port=None, libvirt_domain=None, sec_params=None):
         """ Creates slave machine XML file
-        @param machine_interfaces Dictionary with machine's interfaces
+        @param machine_interfaces List of machine's interfaces
         @param hostname Hostname of the machine
         @param pool_dir Path to directory where XML file will be created
         @param filename Name of the XML file
@@ -312,7 +312,7 @@ class Wizard:
             top_el.appendChild(interfaces_el)

interfaces_added = 0
-            for iface in machine_interfaces.itervalues():
+            for iface in machine_interfaces:
                 if mode == "interactive":
                     msg = "Do you want to add interface '%s' (%s) to the "\
                           "recipe? [Y/n]: " % (iface["name"], iface["hwaddr"])
@@ -390,7 +390,9 @@ class Wizard:
         @param sock Socket used for connecting to machine
         @return Dictionary with machine interfaces or None if RPC call fails
         """
-        msg = {"type": "command", "method_name": "get_devices", "args": {}}
+        msg = {"type": "command",
+               "method_name": "get_devices_by_params",
+               "args": [{"ifi_type": 1, "state": "DOWN"}]}
         if not send_data(sock, msg):
             sys.stderr.write("Could not send request to slave machine\n")
             return None
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index 3b8bf79..1c6f47b 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -136,12 +136,17 @@ class SlaveMethods:
         devices = self._if_manager.get_devices()
         result = {}
         for device in devices:
-            if device._ifi_type == 1 and device._state == 'DOWN':
-                result[device._if_index] = {'name' : device._name,
-                                            'hwaddr' : device._hwaddr,
-                                            'driver' : device._driver}
+            result[device._if_index] = device.get_if_data()
         return result

+    def get_device(self, if_index):
+        self._if_manager.rescan_devices()
+        device = self._if_manager.get_device(if_index)
+        if device:
+            return device.get_if_data()
+        else:
+            return None
+
     def get_devices_by_devname(self, devname):
         name_scan = self._if_manager.get_devices()
         netdevs = []

[PATCH v2 08/20] Machine: construct a Device database reflecting InterfaceManager state

[Ondrej Lichtner]

From: Ondrej Lichtner

This commit adds a new Device class to the Machine module. This class
reflects the Device class from the InterfaceManager wrt. values received
from the kernel and works as a synchronized getter/setter for them.

Right now the Device class only implements the get methods and just the
set_mtu method. This is because the whole idea of this class conflicts
with the Interface classes that we already have and is therefore very
hackish and should be reworked in the future.

This hackish solution is needed here while we still use XML Recipes and
want to test some more complex OvS features, such as VxLan tunneling.

After merging Python recipes this will be completely reworked and Device
objects (similar to how they're now) should completely replace the
Interface objects which are created based on XML requirements.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Machine.py | 146 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 146 insertions(+)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 08df4d8..21b667c 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -18,6 +18,7 @@ import tempfile
 import signal
 from time import sleep
 from xmlrpclib import Binary
+from functools import wraps
 from lnst.Common.Config import lnst_config
 from lnst.Common.NetUtils import normalize_hwaddr
 from lnst.Common.Utils import wait_for, create_tar_archive
@@ -75,6 +76,8 @@ class Machine(object):
         self._namespaces = []
         self._bg_cmds = {}

+        self._device_database = {}
+
     def get_configuration(self):
         configuration = {}
         configuration["id"] = self._id
@@ -128,6 +131,23 @@ class Machine(object):
         if iface:
             iface.update(if_data['if_data'])

+        if if_data["if_data"]["if_index"] in self._device_database:
+            dev = self._device_database[if_data["if_data"]["if_index"]]
+            dev.update_data(if_data['if_data'])
+        else:
+            dev = Device(if_data["if_data"], self)
+            self._device_database[if_data["if_data"]["if_index"]] = dev
+
+    def dev_db_delete(self, update_msg):
+        if update_msg["if_index"] in self._device_database:
+            del self._device_database[update_msg["if_index"]]
+
+    def dev_db_get_name(self, dev_name):
+        for if_index, dev in self._device_database.iteritems():
+            if dev.get_name() == dev_name:
+                return dev
+        return None
+
     #
     # Factory methods for constructing interfaces on this machine. The
     # types of interfaces are explained with the classes below.
@@ -229,6 +249,10 @@ class Machine(object):

self._slave_desc = slave_desc

+        devices = self._rpc_call("get_devices")
+        for if_index, dev in devices.items():
+            self._device_database[if_index] = Device(dev, self)
+
         for iface in self._interfaces:
             iface.initialize()

@@ -1095,3 +1119,125 @@ class UnusedInterface(Interface):

def cleanup(self):
         pass
+
+class Device(object):
+    """ Represents device information received from a Slave"""
+
+    def pre_call_decorate(func):
+        @wraps(func)
+        def func_wrapper(inst, *args, **kwargs):
+            inst.slave_update()
+            return func(inst, *args, **kwargs)
+        return func_wrapper
+
+    def __init__(self, data, machine):
+        self._if_index = data["if_index"]
+        self._hwaddr = None
+        self._name = None
+        self._ip_addrs = None
+        self._ifi_type = None
+        self._state = None
+        self._master = None
+        self._slaves = None
+        self._netns = None
+        self._peer = None
+        self._mtu = None
+        self._driver = None
+
+        self._machine = machine
+
+        self.update_data(data)
+
+    def update_data(self, data):
+        if data["if_index"] != self._if_index:
+            return False
+
+        self._hwaddr = data["hwaddr"]
+        self._name = data["name"]
+        self._ip_addrs = data["ip_addrs"]
+        self._ifi_type = data["ifi_type"]
+        self._state = data["state"]
+        self._master = data["master"]
+        self._slaves = data["slaves"]
+        self._netns = data["netns"]
+        self._peer = data["peer"]
+        self._mtu = data["mtu"]
+        self._driver = data["driver"]
+        return True
+
+    def slave_update(self):
+        res = self._machine._rpc_call_x(self._netns,
+                                        "get_device",
+                                        self._if_index)
+        if res:
+            self.update_data(res)
+        return
+
+    def get_if_index(self):
+        return self._if_index
+
+    @pre_call_decorate
+    def get_hwaddr(self):
+        return self._hwaddr
+
+    @pre_call_decorate
+    def get_name(self):
+        return self._name
+
+    @pre_call_decorate
+    def get_ip_addrs(self, selector={}):
+        return [ip["addr"]
+                for ip in self._ip_addrs
+                    if selector.viewitems() <= ip.viewitems()]
+
+    @pre_call_decorate
+    def get_ip_addr(self, num, selector={}):
+        ips = self.get_ip_addrs(selector)
+        return ips[num]
+
+    @pre_call_decorate
+    def get_ifi_type(self):
+        return self._ifi_type
+
+    @pre_call_decorate
+    def get_state(self):
+        return self._state
+
+    @pre_call_decorate
+    def get_master(self):
+        return self._master
+
+    @pre_call_decorate
+    def get_slaves(self):
+        return self._slaves
+
+    @pre_call_decorate
+    def get_netns(self):
+        return self._netns
+
+    @pre_call_decorate
+    def get_peer(self):
+        return self._peer
+
+    @pre_call_decorate
+    def get_mtu(self):
+        return self._mtu
+
+    def set_mtu(self, mtu):
+        command = {"type": "config",
+                   "host": self._machine.get_id(),
+                   "persistent": False,
+                   "options":[
+                       {"name": "/sys/class/net/%s/mtu" % self._name,
+                        "value": str(mtu)}
+                    ]}
+        command["netns"] = self._netns
+
+        self._machine.run_command(command)
+
+        self.slave_update()
+        return self._mtu
+
+    @pre_call_decorate
+    def get_driver(self):
+        return self._driver

[PATCH v2 09/20] Task: add DeviceAPI as extension of Machine device database

[Ondrej Lichtner]

From: Ondrej Lichtner

The DeviceAPI exports the Device objects stored in the device database
of a machine to the python task. For now it only implements get methods
for Device information received from the Slave, and the set_mtu method.

The implemented methods are compatible with relevant InterfaceAPI
methods.

This patch also adds an optional 'selector' argument to the get_ip and
get_ips methods of InterfaceAPI, this argument is not used and is here
only so that these methods are compatible with DeviceAPI methods.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Task.py | 58 +++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 56 insertions(+), 2 deletions(-)

diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py
index a064086..50523a7 100644
--- a/lnst/Controller/Task.py
+++ b/lnst/Controller/Task.py
@@ -304,6 +304,13 @@ class HostAPI(object):
     def get_interface(self, if_id):
         return self._ifaces[if_id]

+    def get_device(self, name):
+        dev = self._m.dev_db_get_name(name)
+        if dev:
+            return DeviceAPI(self._m.dev_db_get_name(name), self)
+        else:
+            raise TaskError("No device with name '%s' found." % str(name))
+
     @deprecated
     def get_devname(self, if_id):
         """
@@ -476,6 +483,53 @@ class HostAPI(object):

return self._add_iface("vxlan", if_id, netns, ip, options, slaves)

+class DeviceAPI(object):
+    def __init__(self, net_device, host):
+        self._dev = net_device
+        self._host = host
+
+    def get_if_index(self):
+        return self._dev.get_if_index()
+
+    def get_hwaddr(self):
+        return self._dev.get_hwaddr()
+
+    def get_devname(self):
+        return self._dev.get_name()
+
+    def get_ips(self, selector={}):
+        return self._dev.get_ip_addrs(selector)
+
+    def get_ip(self, num, selector={}):
+        return self._dev.get_ip_addr(num, selector)
+
+    def get_ifi_type(self):
+        return self._dev.get_ifi_type()
+
+    def get_state(self):
+        return self._dev.get_state()
+
+    # def get_master(self):
+        # return self._dev.get_master()
+
+    def get_slaves(self):
+        return self._dev.get_slaves()
+
+    def get_netns(self):
+        return self._dev.get_netns()
+
+    # def get_peer(self):
+        # return self._dev.get_peer()
+
+    def get_mtu(self):
+        return self._dev.get_mtu()
+
+    def set_mtu(self, mtu):
+        return self._dev.set_mtu(mtu)
+
+    def get_driver(self):
+        return self._dev.get_driver()
+
 class InterfaceAPI(object):
     def __init__(self, interface, host):
         self._if = interface
@@ -499,10 +553,10 @@ class InterfaceAPI(object):
     def get_hwaddr(self):
         return VolatileValue(self._if.get_hwaddr)

-    def get_ip(self, ip_index=0):
+    def get_ip(self, ip_index=0, selector={}):
         return VolatileValue(self._if.get_address, ip_index)

-    def get_ips(self):
+    def get_ips(self, selector={}):
         return VolatileValue(self._if.get_addresses)

@deprecated

[PATCH v2 10/20] InterfaceManager: add if_deleted update messages

[Ondrej Lichtner]

From: Ondrej Lichtner

The interface manager will now notify the controller when an interface
was removed. The controller will remove the associated Device object
from the device database of a Machine object.

v2: only send the if_deleted message if the Device object was actually
deleted

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/NetTestController.py | 3 +++
 lnst/Slave/InterfaceManager.py       | 8 ++++++++
 2 files changed, 11 insertions(+)

diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py
index 0e9b546..889dfa7 100644
--- a/lnst/Controller/NetTestController.py
+++ b/lnst/Controller/NetTestController.py
@@ -920,6 +920,9 @@ class MessageDispatcher(ConnectionHandler):
         elif message[1]["type"] == "if_update":
             machine = self._machines[message[0]]
             machine.interface_update(message[1])
+        elif message[1]["type"] == "if_deleted":
+            machine = self._machines[message[0]]
+            machine.dev_db_delete(message[1])
         elif message[1]["type"] == "exception":
             msg = "Slave %s: %s" % (message[0], message[1]["Exception"])
             raise CommandException(msg)
diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 821d8fa..c202a3a 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -116,6 +116,10 @@ class InterfaceManager(object):
             dev_name = self._devices[i].get_name()
             logging.debug("Deleting Device with if_index %d, name %s because "\
                           "it doesn't exist anymore." % (i, dev_name))
+
+            del_msg = {"type": "if_deleted",
+                       "if_index": i}
+            self._server_handler.send_data_to_ctl(del_msg)
             del self._devices[i]

def handle_netlink_msgs(self, msgs):
@@ -159,6 +163,10 @@ class InterfaceManager(object):
                 if dev.get_netns() == None and dev.get_conf_dict() == None:
                     dev.del_link()
                     del self._devices[msg['index']]
+
+                    del_msg = {"type": "if_deleted",
+                               "if_index": msg['index']}
+                    self._server_handler.send_data_to_ctl(del_msg)
         else:
             return

[PATCH v2 11/20] InterfaceManager: add Device objects to tmp_mapping sooner

[Ondrej Lichtner]

From: Ondrej Lichtner

This makes sure that the _is_name_used sees the Device object while it
is being created. This is so that if the Device creation actually
creates more than one interface at once it doesn't reuse the same name
for them. An example of such a case is creation of an OvS bridge with
internal interfaces.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index c202a3a..0dec775 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -228,10 +228,11 @@ class InterfaceManager(object):
         config["name"] = self.assign_name(config)

device = Device(self)
+        self._tmp_mapping[if_id] = device
+
         device.set_configuration(config)
         device.create()

-        self._tmp_mapping[if_id] = device
         return config["name"]

def create_device_pair(self, if_id1, config1, if_id2, config2):
@@ -243,6 +244,8 @@ class InterfaceManager(object):

device1 = Device(self)
         device2 = Device(self)
+        self._tmp_mapping[if_id1] = device1
+        self._tmp_mapping[if_id2] = device2

device1.set_configuration(config1)
         device2.set_configuration(config2)
@@ -250,9 +253,6 @@ class InterfaceManager(object):

device1.set_peer(device2)
         device2.set_peer(device1)
-
-        self._tmp_mapping[if_id1] = device1
-        self._tmp_mapping[if_id2] = device2
         return name1, name2

def wait_interface_init(self):

[PATCH v2 12/20] InterfaceManager: rescan devices before checking if name used

[Ondrej Lichtner]

From: Ondrej Lichtner

Similar to the previous patch, this makes sure that the method
_is_name_used checks the most up-to-date state of interface names.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 0dec775..a9f28d7 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -266,6 +266,7 @@ class InterfaceManager(object):
             self.handle_netlink_msgs(msgs)

def _is_name_used(self, name):
+        self.rescan_devices()
         for device in self._devices.itervalues():
             if name == device.get_name():
                 return True

[PATCH v2 13/20] InterfaceManager: make assign_name_generic publicly accessible

[Ondrej Lichtner]

From: Ondrej Lichtner

This renames the _assign_name_generic method to assign_name_generic to
make it publicly accessible. This will be used while creating OvS
bridges with internal devices.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index a9f28d7..62b2805 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -275,7 +275,7 @@ class InterfaceManager(object):
                 return True
         return False

-    def _assign_name_generic(self, prefix):
+    def assign_name_generic(self, prefix):
         index = 0
         while (self._is_name_used(prefix + str(index))):
             index += 1
@@ -304,28 +304,28 @@ class InterfaceManager(object):
                 if dev.get_hwaddr() == hwaddr:
                     return dev.get_name()
         elif dev_type == "bond":
-            return self._assign_name_generic("t_bond")
+            return self.assign_name_generic("t_bond")
         elif dev_type == "bridge" or dev_type == "ovs_bridge":
-            return self._assign_name_generic("t_br")
+            return self.assign_name_generic("t_br")
         elif dev_type == "macvlan":
-            return self._assign_name_generic("t_macvlan")
+            return self.assign_name_generic("t_macvlan")
         elif dev_type == "team":
-            return self._assign_name_generic("t_team")
+            return self.assign_name_generic("t_team")
         elif dev_type == "vlan":
             netdev_name = self.get_mapped_device(config["slaves"][0]).get_name()
             vlan_tci = get_option(config, "vlan_tci")
             prefix = "%s.%s_" % (netdev_name, vlan_tci)
-            return self._assign_name_generic(prefix)
+            return self.assign_name_generic(prefix)
         elif dev_type == "veth":
             return self._assign_name_pair("veth")
         elif dev_type == "vti":
-            return self._assign_name_generic("vti")
+            return self.assign_name_generic("vti")
         elif dev_type == "vti6":
-            return self._assign_name_generic("t_ip6vti")
+            return self.assign_name_generic("t_ip6vti")
         elif dev_type == "vxlan":
-            return self._assign_name_generic("vxlan")
+            return self.assign_name_generic("vxlan")
         else:
-            return self._assign_name_generic("dev")
+            return self.assign_name_generic("dev")

class Device(object):
     def __init__(self, if_manager):

[PATCH v2 14/20] NetConfigCommon: add function get_slave_options

[Ondrej Lichtner]

From: Ondrej Lichtner

Works like the function get_slave_option, but returns the entire option
dictionary for the specified slave instead of just a single option.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/NetConfigCommon.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lnst/Slave/NetConfigCommon.py b/lnst/Slave/NetConfigCommon.py
index ffcb0b4..30517c7 100644
--- a/lnst/Slave/NetConfigCommon.py
+++ b/lnst/Slave/NetConfigCommon.py
@@ -36,6 +36,13 @@ def get_slave_option(netdev, slave_id, opt_name):
             return value
     return None

+def get_slave_options(netdev, slave_id):
+    try:
+        options = netdev["slave_options"][slave_id]
+    except KeyError:
+        return None
+    return options
+
 def get_netem_option(netem_tag, netem_name, opt_name):
     try:
         options = netem_tag[netem_name]

[PATCH v2 15/20] NetConfigDevice: extend NetConfigDeviceOvsBridge

[Ondrej Lichtner]

From: Ondrej Lichtner

The OvS bridge configuration class now supports internal ports, tunnels
and flow entries.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/NetConfigDevice.py | 103 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 100 insertions(+), 3 deletions(-)

diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py
index 6ef3f1c..f6ccd92 100644
--- a/lnst/Slave/NetConfigDevice.py
+++ b/lnst/Slave/NetConfigDevice.py
@@ -14,7 +14,8 @@ [e-mail] (Jiri Pirko)
 import logging
 import re
 from lnst.Common.ExecCmd import exec_cmd
-from lnst.Slave.NetConfigCommon import get_slaves, get_option, get_slave_option, parse_netem
+from lnst.Slave.NetConfigCommon import get_slaves, get_option, get_slave_option
+from lnst.Slave.NetConfigCommon import parse_netem, get_slave_options
 from lnst.Common.Utils import bool_it
 from lnst.Slave.NmConfigDevice import type_class_mapping as nm_type_class_mapping
 from lnst.Slave.NmConfigDevice import is_nm_managed
@@ -421,6 +422,28 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric):
     _modulename = "openvswitch"
     _moduleload = True

+    def up(self):
+        super(NetConfigDeviceOvsBridge, self).up()
+
+        int_ports = self._dev_config["ovs_conf"]["internals"]
+        br_name = self._dev_config["name"]
+        for iport in int_ports:
+            if "addresses" in iport:
+                for address in iport["addresses"]:
+                    exec_cmd("ip addr add %s dev %s" % (address, iport["name"]))
+            exec_cmd("ip link set %s up" % iport["name"])
+
+    def down(self):
+        int_ports = self._dev_config["ovs_conf"]["internals"]
+        br_name = self._dev_config["name"]
+        for iport in int_ports:
+            if "addresses" in iport:
+                for address in iport["addresses"]:
+                    exec_cmd("ip addr del %s dev %s" % (address, iport["name"]))
+            exec_cmd("ip link set %s down" % iport["name"])
+
+        super(NetConfigDeviceOvsBridge, self).down()
+
     @classmethod
     def type_init(self):
         super(NetConfigDeviceOvsBridge, self).type_init()
@@ -447,6 +470,10 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric):
             slave_dev = self._if_manager.get_mapped_device(slave_id)
             slave_name = slave_dev.get_name()

+            options = ""
+            for opt in get_slave_options(self._dev_config, slave_id):
+                options += " %s=%s" % (opt[0], opt[1])
+
             vlan_tags = []
             for tag, vlan in vlans.iteritems():
                 if slave_id in vlan["slaves"]:
@@ -459,6 +486,9 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric):
                 tags = " trunks=" + ",".join(vlan_tags)
             exec_cmd("ovs-vsctl add-port %s %s%s" % (br_name, slave_name, tags))

+            if options != "":
+                exec_cmd("ovs-vsctl set Interface %s%s" % (slave_name, options))
+
     def _del_ports(self):
         slaves = self._dev_config["slaves"]

@@ -477,6 +507,56 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric):

exec_cmd("ovs-vsctl del-port %s %s" % (br_name, slave_name))

+    def _add_internal_ports(self):
+        int_ports = self._dev_config["ovs_conf"]["internals"]
+        br_name = self._dev_config["name"]
+
+        for i in int_ports:
+            i["name"] = self._if_manager.assign_name_generic(prefix="int")
+
+            options = ""
+            for opt in i["options"]:
+                options += " %s=%s" % (opt["name"], opt["value"])
+
+                if opt["name"] == "name":
+                    i["name"] = opt["value"]
+
+            exec_cmd("ovs-vsctl add-port %s %s -- set Interface %s "\
+                     "type=internal %s" % (br_name, i["name"],
+                                           i["name"], options))
+
+    def _del_internal_ports(self):
+        int_ports = self._dev_config["ovs_conf"]["internals"]
+        br_name = self._dev_config["name"]
+
+        for i in int_ports:
+            exec_cmd("ovs-vsctl del-port %s %s" % (br_name, i["name"]))
+
+    def _add_tunnels(self):
+        tunnels = self._dev_config["ovs_conf"]["tunnels"]
+        br_name = self._dev_config["name"]
+
+        for i in tunnels:
+            i["name"] = self._if_manager.assign_name_generic(prefix=i["type"])
+
+            options = ""
+            for opt in i["options"]:
+                options += " %s=%s" % (opt["name"], opt["value"])
+
+                if opt["name"] == "name":
+                    i["name"] = opt["value"]
+
+            exec_cmd("ovs-vsctl add-port %s %s -- set Interface %s "\
+                     "type=%s %s" % (br_name, i["name"], i["name"],
+                                     i["type"], options))
+
+    def _del_tunnels(self):
+        tunnels = self._dev_config["ovs_conf"]["tunnels"]
+        br_name = self._dev_config["name"]
+
+        for i in tunnels:
+            exec_cmd("ovs-vsctl del-port %s %s" % (br_name, i["name"]))
+
     def _add_bonds(self):
         br_name = self._dev_config["name"]

@@ -500,24 +580,41 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric):
         for bond_id, bond in bonds.iteritems():
             exec_cmd("ovs-vsctl del-port %s %s" % (br_name, bond_id))

+    def _add_flow_entries(self):
+        br_name = self._dev_config["name"]
+        entries = self._dev_config["ovs_conf"]["flow_entries"]
+
+        for entry in entries:
+            exec_cmd("ovs-ofctl add-flow %s '%s'" % (br_name, entry))
+
+    def _del_flow_entries(self):
+        br_name = self._dev_config["name"]
+        exec_cmd("ovs-ofctl del-flows %s" % (br_name))
+
     def create(self):
         dev_cfg = self._dev_config
         br_name = dev_cfg["name"]
         exec_cmd("ovs-vsctl add-br %s" % br_name)

+        self._add_internal_ports()
+        self._add_tunnels()
+
     def destroy(self):
+        self._del_tunnels()
+        self._del_internal_ports()
+
         dev_cfg = self._dev_config
         br_name = dev_cfg["name"]
         exec_cmd("ovs-vsctl del-br %s" % br_name)

def configure(self):
         self._add_ports()
-
         self._add_bonds()
+        self._add_flow_entries()

def deconfigure(self):
+        self._del_flow_entries()
         self._del_bonds()
-
         self._del_ports()

class NetConfigDeviceVEth(NetConfigDeviceGeneric):

[PATCH v2 16/20] RecipeParser, schema-recipe: extend OvS bridge XML configuration

[Ondrej Lichtner]

From: Ondrej Lichtner

This extends the LNST supported XML configuration of Open vSwitch
bridges. It now supports internal interfaces, tunnels and flow entries.
An example XML configuration now looks like this:
<ovs_bridge id="bridge1">
    <internal id="int0">
        <addresses>
            <address value="192.168.100.1/24"/>
        </addresses>
        <options>
            <option name="ofport_request" value="5"/>
            <option name="name" value="int0"/>
        </options>
    </internal>
    <tunnel id="vxlan1" type="vxlan">
        <options>
            <option name="option:remote_ip" value="192.168.1.2"/>
            <option name="option:key" value="flow"/>
            <option name="ofport_request" value="10"/>
        </options>
    </tunnel>
    <flow_entries>
        <entry>table=0,in_port=5,actions=set_field:100->tun_id,output:10</entry>
        <entry>table=0,in_port=10,tun_id=100,actions=output:5</entry>
        <entry>table=0,priority=100,actions=drop</entry>
    </flow_entries>
</ovs_bridge>

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/RecipeParser.py | 123 +++++++++++++++++++++++++++++++++-------
 schema-recipe.rng               |  76 ++++++++++++++++++++-----
 2 files changed, 166 insertions(+), 33 deletions(-)

diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py
index 827279f..09233a7 100644
--- a/lnst/Controller/RecipeParser.py
+++ b/lnst/Controller/RecipeParser.py
@@ -70,13 +70,23 @@ class RecipeParser(XmlParser):
                     else:
                         unique_ids.append(interface['id'])

-                    if interface['type'] != 'lo':
-                        continue
-                    elif interface['netns'] in lo_netns:
-                        msg = "Only one loopback device per netns is allowed."
-                        raise RecipeError(msg, interface_tag)
-                    else:
-                        lo_netns.append(interface['netns'])
+                    if interface['type'] == 'lo':
+                        if interface['netns'] in lo_netns:
+                            msg = "Only one loopback device per netns "\
+                                  "is allowed."
+                            raise RecipeError(msg, interface_tag)
+                        else:
+                            lo_netns.append(interface['netns'])
+                    elif interface['type'] == "ovs_bridge":
+                        ovs_conf = interface["ovs_conf"]
+                        for i in ovs_conf["tunnels"] + ovs_conf["internals"]:
+                            if i['id'] in unique_ids:
+                                msg = "Interface with ID "%s" has already "\
+                                      "been defined for this machine." %\
+                                      i['id']
+                                raise RecipeError(msg, i)
+                            else:
+                                unique_ids.append(i['id'])

machine["interfaces"].extend(interfaces)

@@ -125,14 +135,8 @@ class RecipeParser(XmlParser):

# addresses
         addresses_tag = iface_tag.find("addresses")
-        if addresses_tag is not None and len(addresses_tag) > 0:
-            iface["addresses"] = XmlCollection(addresses_tag)
-            for addr_tag in addresses_tag:
-                if self._has_attribute(addr_tag, "value"):
-                    addr = self._get_attribute(addr_tag, "value")
-                else:
-                    addr = self._get_content(addr_tag)
-                iface["addresses"].append(addr)
+        addrs = self._process_addresses(addresses_tag)
+        iface["addresses"] = addrs

if iface["type"] == "eth":
             iface["network"] = self._get_attribute(iface_tag, "label")
@@ -210,12 +214,17 @@ class RecipeParser(XmlParser):
             ovsb_slaves = []

iface["ovs_conf"] = XmlData(slaves_tag)
-            for slave_tag in slaves_tag:
-                slave = XmlData(slave_tag)
-                slave["id"] = str(self._get_attribute(slave_tag, "id"))
-                ovsb_slaves.append(slave["id"])
+            if slaves_tag is not None:
+                for slave_tag in slaves_tag:
+                    slave = XmlData(slave_tag)
+                    slave["id"] = str(self._get_attribute(slave_tag, "id"))
+                    ovsb_slaves.append(slave["id"])

-                iface["slaves"].append(slave)
+                    opts_tag = slave_tag.find("options")
+                    opts = self._process_options(opts_tag)
+                    slave["options"] = opts
+
+                    iface["slaves"].append(slave)

vlan_elems = iface_tag.findall("vlan")
             vlans = iface["ovs_conf"]["vlans"] = XmlData(slaves_tag)
@@ -279,8 +288,82 @@ class RecipeParser(XmlParser):
                 if len(opts) > 0:
                     bonds[bond_id]["options"] = opts

+            unique_ids = []
+            tunnels = iface["ovs_conf"]["tunnels"] = XmlCollection(slaves_tag)
+            tunnel_elems = iface_tag.findall("tunnel")
+            for tunnel_elem in tunnel_elems:
+                tunnels.append(XmlData(tunnel_elem))
+                tunnel = tunnels[-1]
+                tunnel["id"] = str(self._get_attribute(tunnel_elem, "id"))
+                if tunnel["id"] in unique_ids:
+                    msg = "Tunnel with id '%s' already defined for "\
+                          "this ovs_bridge." % tunnel["id"]
+                    raise RecipeError(msg, tunnel_elem)
+                else:
+                    unique_ids.append(tunnel["id"])
+
+                t = str(self._get_attribute(tunnel_elem, "type"))
+                tunnel["type"] = t
+
+                opts_elem = tunnel_elem.find("options")
+                opts = self._process_options(opts_elem)
+                if len(opts) > 0:
+                    tunnel["options"] = opts
+
+                # addresses
+                addresses_tag = tunnel_elem.find("addresses")
+                addrs = self._process_addresses(addresses_tag)
+                tunnel["addresses"] = addrs
+
+            iface["ovs_conf"]["internals"] = XmlCollection(slaves_tag)
+            internals = iface["ovs_conf"]["internals"]
+            internal_elems = iface_tag.findall("internal")
+            for internal_elem in internal_elems:
+                internals.append(XmlData(internal_elem))
+                internal = internals[-1]
+                internal["id"] = str(self._get_attribute(internal_elem, "id"))
+                if internal["id"] in unique_ids:
+                    msg = "Internal id '%s' already defined for "\
+                          "this ovs_bridge." % internal["id"]
+                    raise RecipeError(msg, internal_elem)
+                else:
+                    unique_ids.append(internal["id"])
+
+                opts_elem = internal_elem.find("options")
+                opts = self._process_options(opts_elem)
+                if len(opts) > 0:
+                    internal["options"] = opts
+
+                # addresses
+                addresses_tag = internal_elem.find("addresses")
+                addrs = self._process_addresses(addresses_tag)
+                internal["addresses"] = addrs
+
+            iface["ovs_conf"]["flow_entries"] = XmlCollection(slaves_tag)
+            flow_entries = iface["ovs_conf"]["flow_entries"]
+            flow_elems = iface_tag.findall("flow_entries")
+            if len(flow_elems) == 1:
+                entries = flow_elems[0].findall("entry")
+                for entry in entries:
+                    if self._has_attribute(entry, "value"):
+                        flow_entries.append(self._get_attribute(entry,
+                                                                "value"))
+                    else:
+                        flow_entries.append(self._get_content(entry))
+
         return [iface]

+    def _process_addresses(self, addresses_tag):
+        addresses = XmlCollection(addresses_tag)
+        if addresses_tag is not None and len(addresses_tag) > 0:
+            for addr_tag in addresses_tag:
+                if self._has_attribute(addr_tag, "value"):
+                    addr = self._get_attribute(addr_tag, "value")
+                else:
+                    addr = self._get_content(addr_tag)
+                addresses.append(addr)
+        return addresses
+
     def _process_options(self, opts_tag):
         options = XmlCollection(opts_tag)
         if opts_tag is not None:
diff --git a/schema-recipe.rng b/schema-recipe.rng
index ad841bb..ca06eb8 100644
--- a/schema-recipe.rng
+++ b/schema-recipe.rng
@@ -200,19 +200,24 @@
                     <ref name="define"/>
                 </optional>

-                <element name="slaves">
-                    <interleave>
-                        <optional>
-                            <ref name="define"/>
-                        </optional>
+                <optional>
+                    <element name="slaves">
+                        <interleave>
+                            <optional>
+                                <ref name="define"/>
+                            </optional>

-                        <oneOrMore>
-                            <element name="slave">
-                                <attribute name="id"/>
-                            </element>
-                        </oneOrMore>
-                    </interleave>
-                </element>
+                            <oneOrMore>
+                                <element name="slave">
+                                    <attribute name="id"/>
+                                    <optional>
+                                        <ref name="options"/>
+                                    </optional>
+                                </element>
+                            </oneOrMore>
+                        </interleave>
+                    </element>
+                </optional>

<zeroOrMore>
                     <element name="bond">
@@ -224,7 +229,7 @@
                             </optional>

<optional>
-                            <ref name="options"/>
+                                <ref name="options"/>
                             </optional>

<element name="slaves">
@@ -268,6 +273,51 @@
                         </interleave>
                     </element>
                 </zeroOrMore>
+                <zeroOrMore>
+                    <element name="tunnel">
+                        <attribute name="id"/>
+                        <attribute name="type">
+                            <choice>
+                                <value>vxlan</value>
+                                <value>gre</value>
+                                <value>geneve</value>
+                            </choice>
+                        </attribute>
+                        <interleave>
+                            <optional>
+                                <ref name="options"/>
+                            </optional>
+                            <optional>
+                                <ref name="addresses"/>
+                            </optional>
+                        </interleave>
+                    </element>
+                </zeroOrMore>
+                <zeroOrMore>
+                    <element name="internal">
+                        <attribute name="id"/>
+                        <interleave>
+                            <optional>
+                                <ref name="options"/>
+                            </optional>
+                            <optional>
+                                <ref name="addresses"/>
+                            </optional>
+                        </interleave>
+                    </element>
+                </zeroOrMore>
+                <optional>
+                    <element name="flow_entries">
+                        <oneOrMore>
+                            <element name="entry">
+                                <choice>
+                                    <attribute name="value"/>
+                                    <text/>
+                                </choice>
+                            </element>
+                        </oneOrMore>
+                    </element>
+                </optional>
             </interleave>
         </element>
     </define>

[PATCH v2 17/20] PerfRepoUtils: add perfrepo_baseline_to_dict function

[Ondrej Lichtner]

From: Ondrej Lichtner

This function transforms the PerfRepo TestExecution object representing
a baseline into a dictionary acceptable by the Netperf module.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/PerfRepoUtils.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/lnst/Controller/PerfRepoUtils.py b/lnst/Controller/PerfRepoUtils.py
index 533652b..4c545e4 100644
--- a/lnst/Controller/PerfRepoUtils.py
+++ b/lnst/Controller/PerfRepoUtils.py
@@ -34,6 +34,22 @@ def netperf_baseline_template(module, baseline):
                                'threshold_deviation': '%s bits/sec' % deviation})
     return module

+def perfrepo_baseline_to_dict(baseline):
+    if baseline.get_texec() is None:
+        return {}
+
+    try:
+        throughput = baseline.get_value('throughput')
+        deviation = baseline.get_value('throughput_deviation')
+    except:
+        logging.error("Invalid baseline TestExecution passed.")
+        return {}
+
+    if throughput is not None and deviation is not None:
+        return {'threshold': '%s bits/sec' % throughput,
+                'threshold_deviation': '%s bits/sec' % deviation}
+    return {}
+
 def netperf_result_template(perfrepo_result, netperf_result):
     if isinstance(perfrepo_result, Noop):
         return perfrepo_result

[PATCH v2 18/20] Netperf: allow bind option in client

[Ondrej Lichtner]

From: Ondrej Lichtner

The bind option is translated to a -L command line argument for
netserver, but is ignored for the client. This commit allows us to bind
the client as well as the server.

Signed-off-by: Ondrej Lichtner 
---
 test_modules/Netperf.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/test_modules/Netperf.py b/test_modules/Netperf.py
index 1b9046b..bc6b264 100644
--- a/test_modules/Netperf.py
+++ b/test_modules/Netperf.py
@@ -68,6 +68,11 @@ class Netperf(TestGeneric):
             if self._is_omni():
                 # -P 0 disables banner header of output
                 cmd += " -P 0"
+            if self._bind is not None:
+                """
+                application is bound to this address
+                """
+                cmd += " -L %s" % self._bind
             if self._port is not None:
                 """
                 client connects on this port

[PATCH v2 19/20] RecipeCommon: add ModuleWrap

[Ondrej Lichtner]

From: Ondrej Lichtner

This module defines helper functions for running test modules. Currently
it contains functions for IcmpPing, Icmp6Ping and Netperf modules. The
functions are documented in their respective ___doc___ strings.

Signed-off-by: Ondrej Lichtner 
---
 lnst/RecipeCommon/ModuleWrap.py | 179 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 179 insertions(+)
 create mode 100644 lnst/RecipeCommon/ModuleWrap.py

diff --git a/lnst/RecipeCommon/ModuleWrap.py b/lnst/RecipeCommon/ModuleWrap.py
new file mode 100644
index 0000000..de3babb
--- /dev/null
+++ b/lnst/RecipeCommon/ModuleWrap.py
@@ -0,0 +1,179 @@
+"""
+This module defines helper functions for using test modules from Python Tasks
+
+Copyright 2016 Red Hat, Inc.
+Licensed under the GNU General Public License, version 2 as
+published by the Free Software Foundation; see COPYING for details.
+"""
+
+__author__ = """
[e-mail] (Ondrej Lichtner)
+"""
+
+from lnst.Controller.Task import ctl
+
+def ping(src, dst, options={}, expect="pass"):
+    """ Perform an IcmpPing from source to destination
+
+    Keyword arguments:
+    src -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    dst -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    options -- dictionary of options for the IcmpPing module, can't contain
+        keys 'addr' and 'iface'
+    """
+
+    options = dict(options)
+    if 'addr' in options or 'iface' in options:
+        raise Exception("options can't contain keys 'addr' and 'iface'")
+
+    if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
+        raise Exception('Invalid source specification')
+    try:
+        if len(src) == 2:
+            h1, if1 = src
+            options["iface"] = if1.get_devname()
+        elif len(src) == 3:
+            h1, if1, addr_index1 = src
+            options["iface"] = if1.get_ip(addr_index1)
+        elif len(src) == 4:
+            h1, if1, addr_index1, addr_selector1 = src
+            options["iface"] = if1.get_ip(addr_index1, selector=addr_selector1)
+    except:
+        raise Exception('Invalid source specification')
+
+    if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
+        raise Exception('Invalid destination specification')
+    try:
+        if len(dst) == 3:
+            h2, if2, addr_index2 = dst
+            options["addr"] = if2.get_ip(addr_index2)
+        elif len(dst) == 4:
+            h2, if2, addr_index2, addr_selector2 = dst
+            options["addr"] = if2.get_ip(addr_index2, selector=addr_selector2)
+    except:
+        raise Exception('Invalid destination specification')
+
+    ping_mod = ctl.get_module("IcmpPing",
+                              options = options)
+
+    return h1.run(ping_mod, expect=expect)
+
+def ping6(src, dst, options={}, expect="pass"):
+    """ Perform an Icmp6Ping from source to destination
+
+    Keyword arguments:
+    src -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    dst -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    options -- dictionary of options for the IcmpPing module, can't contain
+        keys 'addr' and 'iface'
+    """
+
+    options = dict(options)
+    if 'addr' in options or 'iface' in options:
+        raise Exception("options can't contain keys 'addr' and 'iface'")
+
+    if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
+        raise Exception('Invalid source specification')
+    try:
+        if len(src) == 2:
+            h1, if1 = src
+            options["iface"] = if1.get_devname()
+        elif len(src) == 3:
+            h1, if1, addr_index1 = src
+            options["iface"] = if1.get_ip(addr_index1)
+        elif len(src) == 4:
+            h1, if1, addr_index1, addr_selector1 = src
+            options["iface"] = if1.get_ip(addr_index1, selector=addr_selector1)
+    except:
+        raise Exception('Invalid source specification')
+
+    if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
+        raise Exception('Invalid destination specification')
+    try:
+        if len(dst) == 3:
+            h2, if2, addr_index2 = dst
+            options["addr"] = if2.get_ip(addr_index2)
+        elif len(dst) == 4:
+            h2, if2, addr_index2, addr_selector2 = dst
+            options["addr"] = if2.get_ip(addr_index2, selector=addr_selector2)
+    except:
+        raise Exception('Invalid destination specification')
+
+    ping_mod = ctl.get_module("Icmp6Ping",
+                              options = options)
+
+    return h1.run(ping_mod, expect=expect)
+
+def netperf(src, dst, server_opts={}, client_opts={}, baseline={}, timeout=60):
+    """ Start a Netserver on the given machine and ip address
+
+    Keyword arguments:
+    src -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    dst -- tuple of (HostAPI, InterfaceAPI/DeviceAPI, ip address index, ip addr selector)
+    server_opts -- dictionary of additional options for the netperf server
+        can't contain 'bind' or 'role'
+    client_opts -- dictionary of additional options for the netperf client
+        can't contain 'bind', 'role', 'netperf_server', 'threshold'
+        or 'threshold_deviation'
+    baseline -- optional dictionary with keys 'threshold' and 'threshold_deviation'
+        that specifies the baseline of the netperf test
+    timeout -- integer number of seconds specifing the maximum amount of time
+        for the test, defaults to 60
+    """
+
+    server_opts = dict(server_opts)
+    if 'bind' in server_opts or 'role' in server_opts:
+        raise Exception("server_opts can't contain keys 'bind' and 'role'")
+
+    client_opts = dict(client_opts)
+    if 'bind' in client_opts or\
+       'role' in client_opts or\
+       'netperf_server' in client_opts:
+        raise Exception("client_opts can't contain keys 'bind', 'role' "\
+                        "and 'netperf_server'")
+
+    if not isinstance(src, tuple) or len(src) < 2 or len(src) > 4:
+        raise Exception('Invalid source specification')
+    try:
+        if len(src) == 3:
+            h1, if1, addr_index1 = src
+            client_ip = if1.get_ip(addr_index1)
+        elif len(src) == 4:
+            h1, if1, addr_index1, addr_selector1 = src
+            client_ip = if1.get_ip(addr_index1, selector=addr_selector1)
+    except:
+        raise Exception('Invalid source specification')
+
+    if not isinstance(dst, tuple) or len(dst) < 3 or len(dst) > 4:
+        raise Exception('Invalid destination specification')
+    try:
+        if len(dst) == 3:
+            h2, if2, addr_index2 = dst
+            server_ip = if2.get_ip(addr_index2)
+        elif len(dst) == 4:
+            h2, if2, addr_index2, addr_selector2 = dst
+            server_ip = if2.get_ip(addr_index2, addr_selector2)
+    except:
+        raise Exception('Invalid destination specification')
+
+    server_opts["role"] = "server"
+    server_opts["bind"] = server_ip
+
+    client_opts["role"] = "client"
+    client_opts["bind"] = client_ip
+    client_opts["netperf_server"] = server_ip
+
+    if "threshold" in baseline:
+        client_opts["threshold"] = baseline["threshold"]
+    if "threshold_deviation" in baseline:
+        client_opts["threshold_deviation"] = baseline["threshold_deviation"]
+
+    netserver_mod = ctl.get_module("Netperf", options=server_opts)
+    netclient_mod = ctl.get_module("Netperf", options=client_opts)
+
+    netserver = h2.run(netserver_mod, bg=True)
+    ctl.wait(2)
+    result = h1.run(netclient_mod, timeout=timeout)
+
+    netserver.intr()
+    return result

[PATCH v2 20/20] regression_tests: add phase3, starting with vxlan

[Ondrej Lichtner]

From: Ondrej Lichtner

This commit adds the first versions of 4 vxlan related recipes. Two of
these recipes:
vxlan_multicast.xml
vxlan_remote.xml

Are simple setups using 2 machines and the kernel VXLAN driver.

The other two:
2_virt_ovs_vxlan.xml
novirt_ovs_vxlan.xml

Uses two Open vSwitch bridges on 2 hosts and connects them through a
VXLAN tunnel using the OvS implementation. The first recipe also uses 4
more guest machines (2 per host) that are separated into two separate
vxlans. The second recipe doesn't use the guest machines and instead
uses internal ports of the OvS bridge.

All 4 recipes perform a TCP_STREAM and UDP_STREAM Netperf for both ipv4
and ipv6 addresses. Same goes for a simple ping check.

In addition to that, the 2_virt_ovs_vxlan recipe also performs all
possible pings between the guests and checks that the vxlan separation
is properly in place.

The recipes use features introduced in the commits directly preceding
this one and are therefore dependent on them. This is a first version of
the recipes, based on other regression_tests recipes, it's ready to be
tested on real hardware but needs further clenups and modifications
before actual deployment.

Signed-off-by: Ondrej Lichtner 
---
 .../regression_tests/phase3/2_virt_ovs_vxlan.py    | 247 +++++++++++++++++++++
 .../regression_tests/phase3/2_virt_ovs_vxlan.xml   | 138 ++++++++++++
 .../regression_tests/phase3/novirt_ovs_vxlan.py    | 201 +++++++++++++++++
 .../regression_tests/phase3/novirt_ovs_vxlan.xml   |  86 +++++++
 .../regression_tests/phase3/vxlan_multicast.xml    |  66 ++++++
 recipes/regression_tests/phase3/vxlan_remote.xml   |  67 ++++++
 recipes/regression_tests/phase3/vxlan_test.py      | 247 +++++++++++++++++++++
 7 files changed, 1052 insertions(+)
 create mode 100644 recipes/regression_tests/phase3/2_virt_ovs_vxlan.py
 create mode 100644 recipes/regression_tests/phase3/2_virt_ovs_vxlan.xml
 create mode 100644 recipes/regression_tests/phase3/novirt_ovs_vxlan.py
 create mode 100644 recipes/regression_tests/phase3/novirt_ovs_vxlan.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_multicast.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_remote.xml
 create mode 100644 recipes/regression_tests/phase3/vxlan_test.py

diff --git a/recipes/regression_tests/phase3/2_virt_ovs_vxlan.py b/recipes/regression_tests/phase3/2_virt_ovs_vxlan.py
new file mode 100644
index 0000000..388b603
--- /dev/null
+++ b/recipes/regression_tests/phase3/2_virt_ovs_vxlan.py
@@ -0,0 +1,247 @@
+from lnst.Controller.Task import ctl
+from lnst.Controller.PerfRepoUtils import perfrepo_baseline_to_dict
+from lnst.Controller.PerfRepoUtils import netperf_result_template
+
+from lnst.RecipeCommon.ModuleWrap import ping, ping6, netperf
+from lnst.RecipeCommon.IRQ import pin_dev_irqs
+from lnst.RecipeCommon.PerfRepo import generate_perfrepo_comment
+
+# ------
+# SETUP
+# ------
+
+mapping_file = ctl.get_alias("mapping_file")
+perf_api = ctl.connect_PerfRepo(mapping_file)
+
+product_name = ctl.get_alias("product_name")
+
+# hosts
+host1 = ctl.get_host("h1")
+host2 = ctl.get_host("h2")
+
+# guest machines
+guest1 = ctl.get_host("test_host1")
+guest2 = ctl.get_host("test_host2")
+guest3 = ctl.get_host("test_host3")
+guest4 = ctl.get_host("test_host4")
+
+for h in [guest1, guest2, guest3, guest4]:
+    h.sync_resources(modules=["IcmpPing", "Icmp6Ping", "Netperf"])
+
+# ------
+# TESTS
+# ------
+
+ipv = ctl.get_alias("ipv")
+mtu = ctl.get_alias("mtu")
+netperf_duration = int(ctl.get_alias("netperf_duration"))
+nperf_reserve = int(ctl.get_alias("nperf_reserve"))
+nperf_confidence = ctl.get_alias("nperf_confidence")
+nperf_max_runs = int(ctl.get_alias("nperf_max_runs"))
+nperf_cpu_util = ctl.get_alias("nperf_cpu_util")
+nperf_mode = ctl.get_alias("nperf_mode")
+nperf_num_parallel = int(ctl.get_alias("nperf_num_parallel"))
+pr_user_comment = ctl.get_alias("perfrepo_comment")
+
+pr_comment = generate_perfrepo_comment([guest1, guest2, guest3, guest4],
+                                       pr_user_comment)
+
+g1_nic = guest1.get_interface("if1")
+g2_nic = guest2.get_interface("if1")
+g3_nic = guest3.get_interface("if1")
+g4_nic = guest4.get_interface("if1")
+
+g1_nic.set_mtu(mtu)
+g2_nic.set_mtu(mtu)
+g3_nic.set_mtu(mtu)
+g4_nic.set_mtu(mtu)
+
+host1.run("service irqbalance stop")
+host2.run("service irqbalance stop")
+guest1.run("service irqbalance stop")
+guest2.run("service irqbalance stop")
+guest3.run("service irqbalance stop")
+guest4.run("service irqbalance stop")
+
+#this will pin devices irqs to cpu #0
+for m, d in [(guest1, g1_nic), (guest2, g2_nic), (guest3, g3_nic), (guest4, g4_nic)]:
+    pin_dev_irqs(m, d, 0)
+
+
+ctl.wait(15)
+
+#pings
+ping_opts = {"count": 100, "interval": 0.1}
+if ipv in ['ipv4', 'both']:
+    ping((guest1, g1_nic, 0),
+         (guest2, g2_nic, 0),
+         options=ping_opts, expect="fail")
+    ping((guest1, g1_nic, 0),
+         (guest3, g3_nic, 0),
+         options=ping_opts)
+    ping((guest1, g1_nic, 0),
+         (guest4, g4_nic, 0),
+         options=ping_opts, expect="fail")
+
+    ping((guest2, g2_nic, 0),
+         (guest3, g3_nic, 0),
+         options=ping_opts, expect="fail")
+    ping((guest2, g2_nic, 0),
+         (guest4, g4_nic, 0),
+         options=ping_opts)
+
+    ping((guest3, g3_nic, 0),
+         (guest4, g4_nic, 0),
+         options=ping_opts, expect="fail")
+
+if ipv in ['ipv6', 'both']:
+    ping6((guest1, g1_nic, 0),
+          (guest2, g2_nic, 1),
+          options=ping_opts, expect="fail")
+    ping6((guest1, g1_nic, 0),
+          (guest3, g3_nic, 1),
+          options=ping_opts)
+    ping6((guest1, g1_nic, 0),
+          (guest4, g4_nic, 1),
+          options=ping_opts, expect="fail")
+
+    ping6((guest2, g2_nic, 0),
+          (guest3, g3_nic, 1),
+          options=ping_opts, expect="fail")
+    ping6((guest2, g2_nic, 0),
+          (guest4, g4_nic, 1),
+          options=ping_opts)
+
+    ping6((guest3, g3_nic, 0),
+          (guest4, g4_nic, 1),
+          options=ping_opts, expect="fail")
+
+# if nperf_mode == "multi":
+    # netperf_cli_tcp.unset_option("confidence")
+    # netperf_cli_udp.unset_option("confidence")
+    # netperf_cli_tcp6.unset_option("confidence")
+    # netperf_cli_udp6.unset_option("confidence")
+
+    # netperf_cli_tcp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_tcp6.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp6.update_options({"num_parallel": nperf_num_parallel})
+
+
+if ipv in [ 'ipv4', 'both' ]:
+    # prepare PerfRepo result for tcp
+    result_tcp = perf_api.new_result("tcp_ipv4_id",
+                                     "tcp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((guest1, g1_nic, 0), (guest3, g3_nic, 0),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    # prepare PerfRepo result for udp
+    result_udp = perf_api.new_result("udp_ipv4_id",
+                                     "udp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((guest1, g1_nic, 0), (guest3, g3_nic, 0),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+if ipv in [ 'ipv6', 'both' ]:
+    # prepare PerfRepo result for tcp ipv6
+    result_tcp = perf_api.new_result("tcp_ipv6_id",
+                                     "tcp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((guest1, g1_nic, 1), (guest3, g3_nic, 1),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    #prepare PerfRepo result for udp ipv6
+    result_udp = perf_api.new_result("udp_ipv6_id",
+                                     "udp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((guest1, g1_nic, 1), (guest3, g3_nic, 1),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+
+host1.run("service irqbalance start")
+host2.run("service irqbalance start")
+guest1.run("service irqbalance start")
+guest2.run("service irqbalance start")
+guest3.run("service irqbalance start")
+guest4.run("service irqbalance start")
diff --git a/recipes/regression_tests/phase3/2_virt_ovs_vxlan.xml b/recipes/regression_tests/phase3/2_virt_ovs_vxlan.xml
new file mode 100644
index 0000000..f34c67f
--- /dev/null
+++ b/recipes/regression_tests/phase3/2_virt_ovs_vxlan.xml
@@ -0,0 +1,138 @@
+<lnstrecipe>
+    <define>
+        <alias name="ipv" value="both" />
+        <alias name="mtu" value="1450" />
+        <alias name="netperf_duration" value="60" />
+        <alias name="nperf_reserve" value="20" />
+        <alias name="nperf_confidence" value="99,5" />
+        <alias name="nperf_max_runs" value="5"/>
+        <alias name="nperf_mode" value="default"/>
+        <alias name="nperf_num_parallel" value="2"/>
+        <alias name="mapping_file" value="2_virt_ovs_vxlan.mapping" />
+        <alias name="net" value="192.168.2"/>
+        <alias name="net6" value="fc00:0:0:0"/>
+        <alias name="vxlan_net" value="192.168.100"/>
+    </define>
+    <network>
+        <host id="h1">
+            <interfaces>
+                <eth id="if1" label="n1">
+                    <addresses>
+                        <address value="{$vxlan_net}.1/24"/>
+                    </addresses>
+                </eth>
+                <eth id="if2" label="to_guest1"/>
+                <eth id="if3" label="to_guest2"/>
+                <ovs_bridge id="ovs1">
+                    <slaves>
+                        <slave id="if2">
+                            <options>
+                                <option name="ofport_request" value="5"/>
+                            </options>
+                        </slave>
+                        <slave id="if3">
+                            <options>
+                                <option name="ofport_request" value="6"/>
+                            </options>
+                        </slave>
+                    </slaves>
+                    <tunnel id="vxlan1" type="vxlan">
+                        <options>
+                            <option name="option:remote_ip" value="{$vxlan_net}.2"/>
+                            <option name="option:key" value="flow"/>
+                            <option name="ofport_request" value="10"/>
+                        </options>
+                    </tunnel>
+                    <flow_entries>
+                        <entry>table=0,in_port=5,actions=set_field:100->tun_id,output:10</entry>
+                        <entry>table=0,in_port=6,actions=set_field:200->tun_id,output:10</entry>
+                        <entry>table=0,in_port=10,tun_id=100,actions=output:5</entry>
+                        <entry>table=0,in_port=10,tun_id=200,actions=output:6</entry>
+                        <entry>table=0,priority=100,actions=drop</entry>
+                    </flow_entries>
+                </ovs_bridge>
+            </interfaces>
+        </host>
+        <host id="test_host1">
+            <interfaces>
+                <eth id="if1" label="to_guest1">
+                    <addresses>
+                        <address value="{$net}.1/24"/>
+                        <address value="{$net6}::1/64"/>
+                    </addresses>
+                </eth>
+            </interfaces>
+        </host>
+        <host id="test_host2">
+            <interfaces>
+                <eth id="if1" label="to_guest2">
+                    <addresses>
+                        <address value="{$net}.2/24"/>
+                        <address value="{$net6}::2/64"/>
+                    </addresses>
+                </eth>
+            </interfaces>
+        </host>
+        <host id="h2">
+            <interfaces>
+                <eth id="if1" label="n1">
+                    <addresses>
+                        <address value="{$vxlan_net}.2/24"/>
+                    </addresses>
+                </eth>
+                <eth id="if2" label="to_guest3"/>
+                <eth id="if3" label="to_guest4"/>
+                <ovs_bridge id="ovs2">
+                    <slaves>
+                        <slave id="if2">
+                            <options>
+                                <option name="ofport_request" value="5"/>
+                            </options>
+                        </slave>
+                        <slave id="if3">
+                            <options>
+                                <option name="ofport_request" value="6"/>
+                            </options>
+                        </slave>
+                    </slaves>
+                    <tunnel id="vxlan1" type="vxlan">
+                        <options>
+                            <option name="option:remote_ip" value="{$vxlan_net}.1"/>
+                            <option name="option:key" value="flow"/>
+                            <option name="ofport_request" value="10"/>
+                        </options>
+                    </tunnel>
+                    <flow_entries>
+                        <entry>table=0,in_port=5,actions=set_field:100->tun_id,output:10</entry>
+                        <entry>table=0,in_port=6,actions=set_field:200->tun_id,output:10</entry>
+                        <entry>table=0,in_port=10,tun_id=100,actions=output:5</entry>
+                        <entry>table=0,in_port=10,tun_id=200,actions=output:6</entry>
+                        <entry>table=0,priority=100,actions=drop</entry>
+                    </flow_entries>
+                </ovs_bridge>
+            </interfaces>
+        </host>
+        <host id="test_host3">
+            <interfaces>
+                <eth id="if1" label="to_guest3">
+                    <addresses>
+                        <address value="{$net}.3/24"/>
+                        <address value="{$net6}::3/64"/>
+                    </addresses>
+                </eth>
+            </interfaces>
+        </host>
+        <host id="test_host4">
+            <interfaces>
+                <eth id="if1" label="to_guest4">
+                    <addresses>
+                        <address value="{$net}.4/24"/>
+                        <address value="{$net6}::4/64"/>
+                    </addresses>
+                </eth>
+            </interfaces>
+        </host>
+    </network>
+
+    <task python="2_virt_ovs_vxlan.py"/>
+</lnstrecipe>
diff --git a/recipes/regression_tests/phase3/novirt_ovs_vxlan.py b/recipes/regression_tests/phase3/novirt_ovs_vxlan.py
new file mode 100644
index 0000000..6114bff
--- /dev/null
+++ b/recipes/regression_tests/phase3/novirt_ovs_vxlan.py
@@ -0,0 +1,201 @@
+from lnst.Controller.Task import ctl
+from lnst.Controller.PerfRepoUtils import perfrepo_baseline_to_dict
+from lnst.Controller.PerfRepoUtils import netperf_result_template
+
+from lnst.RecipeCommon.ModuleWrap import ping, ping6, netperf
+from lnst.RecipeCommon.IRQ import pin_dev_irqs
+from lnst.RecipeCommon.PerfRepo import generate_perfrepo_comment
+
+# ------
+# SETUP
+# ------
+
+mapping_file = ctl.get_alias("mapping_file")
+perf_api = ctl.connect_PerfRepo(mapping_file)
+
+product_name = ctl.get_alias("product_name")
+
+# test hosts
+h1 = ctl.get_host("test_host1")
+h2 = ctl.get_host("test_host2")
+
+for h in [h1, h2]:
+    h.sync_resources(modules=["IcmpPing", "Icmp6Ping", "Netperf"])
+
+# ------
+# TESTS
+# ------
+
+ipv = ctl.get_alias("ipv")
+mtu = ctl.get_alias("mtu")
+netperf_duration = int(ctl.get_alias("netperf_duration"))
+nperf_reserve = int(ctl.get_alias("nperf_reserve"))
+nperf_confidence = ctl.get_alias("nperf_confidence")
+nperf_max_runs = int(ctl.get_alias("nperf_max_runs"))
+nperf_cpu_util = ctl.get_alias("nperf_cpu_util")
+nperf_mode = ctl.get_alias("nperf_mode")
+nperf_num_parallel = int(ctl.get_alias("nperf_num_parallel"))
+pr_user_comment = ctl.get_alias("perfrepo_comment")
+
+pr_comment = generate_perfrepo_comment([h1, h2], pr_user_comment)
+
+h1_nic = h1.get_device("int0")
+h2_nic = h2.get_device("int0")
+
+h1_nic.set_mtu(mtu)
+h2_nic.set_mtu(mtu)
+
+h1.run("service irqbalance stop")
+h2.run("service irqbalance stop")
+
+# this will pin devices irqs to cpu #0
+for m, d in [(h1, h1_nic), (h2, h2_nic)]:
+    pin_dev_irqs(m, d, 0)
+
+# if nperf_mode == "multi":
+    # netperf_cli_tcp.unset_option("confidence")
+    # netperf_cli_udp.unset_option("confidence")
+    # netperf_cli_tcp6.unset_option("confidence")
+    # netperf_cli_udp6.unset_option("confidence")
+
+    # netperf_cli_tcp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_tcp6.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp6.update_options({"num_parallel": nperf_num_parallel})
+
+ctl.wait(15)
+
+#pings
+ping_opts = {"count": 100, "interval": 0.1}
+if ipv in [ 'ipv4', 'both' ]:
+    ping((h1, h1_nic, 0, {"scope": 0}),
+         (h2, h2_nic, 0, {"scope": 0}),
+         options=ping_opts)
+
+if ipv in [ 'ipv6', 'both' ]:
+    ping6((h1, h1_nic, 1, {"scope": 0}),
+          (h2, h2_nic, 1, {"scope": 0}),
+          options=ping_opts)
+
+#netperfs
+if ipv in [ 'ipv4', 'both' ]:
+    ctl.wait(2)
+
+    # prepare PerfRepo result for tcp
+    result_tcp = perf_api.new_result("tcp_ipv4_id",
+                                     "tcp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((h1, h1_nic, 0, {"scope": 0}),
+                           (h2, h2_nic, 0, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    # prepare PerfRepo result for udp
+    result_udp = perf_api.new_result("udp_ipv4_id",
+                                     "udp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((h1, h1_nic, 0, {"scope": 0}),
+                           (h2, h2_nic, 0, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+if ipv in [ 'ipv6', 'both' ]:
+    ctl.wait(2)
+
+    # prepare PerfRepo result for tcp ipv6
+    result_tcp = perf_api.new_result("tcp_ipv6_id",
+                                     "tcp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((h1, h1_nic, 1, {"scope": 0}),
+                           (h2, h2_nic, 1, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    # prepare PerfRepo result for udp ipv6
+    result_udp = perf_api.new_result("udp_ipv6_id",
+                                     "udp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((h1, h1_nic, 1, {"scope": 0}),
+                           (h2, h2_nic, 1, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+
+h1.run("service irqbalance start")
+h2.run("service irqbalance start")
diff --git a/recipes/regression_tests/phase3/novirt_ovs_vxlan.xml b/recipes/regression_tests/phase3/novirt_ovs_vxlan.xml
new file mode 100644
index 0000000..4f6c98d
--- /dev/null
+++ b/recipes/regression_tests/phase3/novirt_ovs_vxlan.xml
@@ -0,0 +1,86 @@
+<lnstrecipe>
+    <define>
+        <alias name="ipv" value="both" />
+        <alias name="mtu" value="1450" />
+        <alias name="netperf_duration" value="60" />
+        <alias name="nperf_reserve" value="20" />
+        <alias name="nperf_confidence" value="99,5" />
+        <alias name="nperf_max_runs" value="5"/>
+        <alias name="nperf_mode" value="default"/>
+        <alias name="nperf_num_parallel" value="2"/>
+        <alias name="mapping_file" value="novirt_ovs_vxlan.mapping" />
+        <alias name="net" value="192.168.2"/>
+        <alias name="net6" value="fc00:0:0:0"/>
+        <alias name="vxlan_net" value="192.168.100"/>
+    </define>
+    <network>
+        <host id="test_host1">
+            <interfaces>
+                <eth id="if1" label="n1">
+                    <addresses>
+                        <address value="{$vxlan_net}.1/24"/>
+                    </addresses>
+                </eth>
+                <ovs_bridge id="ovs1">
+                    <internal id="int0">
+                        <addresses>
+                            <address value="{$net}.1/24"/>
+                            <address value="{$net6}::1/64"/>
+                        </addresses>
+                        <options>
+                            <option name="ofport_request" value="5"/>
+                            <option name="name" value="int0"/>
+                        </options>
+                    </internal>
+                    <tunnel id="vxlan1" type="vxlan">
+                        <options>
+                            <option name="option:remote_ip" value="{$vxlan_net}.2"/>
+                            <option name="option:key" value="flow"/>
+                            <option name="ofport_request" value="10"/>
+                        </options>
+                    </tunnel>
+                    <flow_entries>
+                        <entry>table=0,in_port=5,actions=set_field:100->tun_id,output:10</entry>
+                        <entry>table=0,in_port=10,tun_id=100,actions=output:5</entry>
+                        <entry>table=0,priority=100,actions=drop</entry>
+                    </flow_entries>
+                </ovs_bridge>
+            </interfaces>
+        </host>
+        <host id="test_host2">
+            <interfaces>
+                <eth id="if1" label="n1">
+                    <addresses>
+                        <address value="{$vxlan_net}.2/24"/>
+                    </addresses>
+                </eth>
+                <ovs_bridge id="ovs2">
+                    <internal id="int0">
+                        <options>
+                            <option name="ofport_request" value="5"/>
+                            <option name="name" value="int0"/>
+                        </options>
+                        <addresses>
+                            <address value="{$net}.2/24"/>
+                            <address value="{$net6}::2/24"/>
+                        </addresses>
+                    </internal>
+                    <tunnel id="vxlan1" type="vxlan">
+                        <options>
+                            <option name="option:remote_ip" value="{$vxlan_net}.1"/>
+                            <option name="option:key" value="flow"/>
+                            <option name="ofport_request" value="10"/>
+                        </options>
+                    </tunnel>
+                    <flow_entries>
+                        <entry>table=0,in_port=5,actions=set_field:100->tun_id,output:10</entry>
+                        <entry>table=0,in_port=10,tun_id=100,actions=output:5</entry>
+                        <entry>table=0,priority=100,actions=drop</entry>
+                    </flow_entries>
+                </ovs_bridge>
+            </interfaces>
+        </host>
+    </network>
+
+    <task python="novirt_ovs_vxlan.py"/>
+</lnstrecipe>
diff --git a/recipes/regression_tests/phase3/vxlan_multicast.xml b/recipes/regression_tests/phase3/vxlan_multicast.xml
new file mode 100644
index 0000000..bb603f1
--- /dev/null
+++ b/recipes/regression_tests/phase3/vxlan_multicast.xml
@@ -0,0 +1,66 @@
+<lnstrecipe>
+    <define>
+        <alias name="ipv" value="both" />
+        <alias name="mtu" value="1450" />
+        <alias name="netperf_duration" value="60" />
+        <alias name="nperf_reserve" value="20" />
+        <alias name="nperf_confidence" value="99,5" />
+        <alias name="nperf_max_runs" value="5"/>
+        <alias name="nperf_mode" value="default"/>
+        <alias name="nperf_num_parallel" value="2"/>
+        <alias name="mapping_file" value="vxlan.mapping" />
+        <alias name="net" value="192.168.0"/>
+        <alias name="vxlan_net" value="192.168.100"/>
+        <alias name="vxlan_net6" value="fc01:0:0:0"/>
+    </define>
+    <network>
+        <host id="testmachine1">
+            <interfaces>
+                <eth id="eth" label="tnet">
+                    <addresses>
+                        <address value="{$net}.1/24" />
+                        <address value="fc00:0:0:0::1/64"/>
+                    </addresses>
+                </eth>
+                <vxlan id="test_if">
+                    <options>
+                        <option name="id" value="1"/>
+                        <option name="group_ip" value="239.1.1.1"/>
+                    </options>
+                    <slaves>
+                        <slave id="eth"/>
+                    </slaves>
+                    <addresses>
+                        <address value="{$vxlan_net}.1/24" />
+                        <address value="{$vxlan_net6}::1/64" />
+                    </addresses>
+                </vxlan>
+            </interfaces>
+        </host>
+        <host id="testmachine2">
+            <interfaces>
+                <eth id="eth" label="tnet">
+                    <addresses>
+                        <address value="{$net}.2/24" />
+                        <address value="fc00:0:0:0::2/64"/>
+                    </addresses>
+                </eth>
+                <vxlan id="test_if">
+                    <options>
+                        <option name="id" value="1"/>
+                        <option name="group_ip" value="239.1.1.1"/>
+                    </options>
+                    <slaves>
+                        <slave id="eth"/>
+                    </slaves>
+                    <addresses>
+                        <address value="{$vxlan_net}.2/24" />
+                        <address value="{$vxlan_net6}::2/64" />
+                    </addresses>
+                </vxlan>
+            </interfaces>
+        </host>
+    </network>
+
+    <task python="vxlan_test.py" />
+</lnstrecipe>
diff --git a/recipes/regression_tests/phase3/vxlan_remote.xml b/recipes/regression_tests/phase3/vxlan_remote.xml
new file mode 100644
index 0000000..d04dee2
--- /dev/null
+++ b/recipes/regression_tests/phase3/vxlan_remote.xml
@@ -0,0 +1,67 @@
+<lnstrecipe>
+    <define>
+        <alias name="ipv" value="both" />
+        <alias name="mtu" value="1450" />
+        <alias name="netperf_duration" value="60" />
+        <alias name="nperf_reserve" value="20" />
+        <alias name="nperf_confidence" value="99,5" />
+        <alias name="nperf_max_runs" value="5"/>
+        <alias name="nperf_mode" value="default"/>
+        <alias name="nperf_num_parallel" value="2"/>
+        <alias name="mapping_file" value="vxlan_remote.mapping" />
+        <alias name="net" value="192.168.0"/>
+        <alias name="net6" value="fc00:0:0:0"/>
+        <alias name="vxlan_net" value="192.168.100"/>
+        <alias name="vxlan_net6" value="fc01:0:0:0"/>
+    </define>
+    <network>
+        <host id="testmachine1">
+            <interfaces>
+                <eth id="eth" label="tnet">
+                    <addresses>
+                        <address value="{$net}.1/24" />
+                        <address value="{$net6}::1/64"/>
+                    </addresses>
+                </eth>
+                <vxlan id="test_if">
+                    <options>
+                        <option name="id" value="1"/>
+                        <option name="remote_ip" value="{$net}.2"/>
+                    </options>
+                    <slaves>
+                        <slave id="eth"/>
+                    </slaves>
+                    <addresses>
+                        <address value="{$vxlan_net}.1/24" />
+                        <address value="{$vxlan_net6}::1/64" />
+                    </addresses>
+                </vxlan>
+            </interfaces>
+        </host>
+        <host id="testmachine2">
+            <interfaces>
+                <eth id="eth" label="tnet">
+                    <addresses>
+                        <address value="{$net}.2/24" />
+                        <address value="fc00:0:0:0::2/64"/>
+                    </addresses>
+                </eth>
+                <vxlan id="test_if">
+                    <options>
+                        <option name="id" value="1"/>
+                        <option name="remote_ip" value="{$net}.1"/>
+                    </options>
+                    <slaves>
+                        <slave id="eth"/>
+                    </slaves>
+                    <addresses>
+                        <address value="{$vxlan_net}.2/24" />
+                        <address value="{$vxlan_net6}::2/64" />
+                    </addresses>
+                </vxlan>
+            </interfaces>
+        </host>
+    </network>
+
+    <task python="vxlan_test.py" />
+</lnstrecipe>
diff --git a/recipes/regression_tests/phase3/vxlan_test.py b/recipes/regression_tests/phase3/vxlan_test.py
new file mode 100644
index 0000000..f96fc46
--- /dev/null
+++ b/recipes/regression_tests/phase3/vxlan_test.py
@@ -0,0 +1,247 @@
+from lnst.Controller.Task import ctl
+from lnst.Controller.PerfRepoUtils import perfrepo_baseline_to_dict
+from lnst.Controller.PerfRepoUtils import netperf_result_template
+
+from lnst.RecipeCommon.ModuleWrap import ping, ping6, netperf
+from lnst.RecipeCommon.IRQ import pin_dev_irqs
+from lnst.RecipeCommon.PerfRepo import generate_perfrepo_comment
+
+# ------
+# SETUP
+# ------
+
+mapping_file = ctl.get_alias("mapping_file")
+perf_api = ctl.connect_PerfRepo(mapping_file)
+
+product_name = ctl.get_alias("product_name")
+
+m1 = ctl.get_host("testmachine1")
+m2 = ctl.get_host("testmachine2")
+
+m1.sync_resources(modules=["IcmpPing", "Icmp6Ping", "Netperf"])
+m2.sync_resources(modules=["IcmpPing", "Icmp6Ping", "Netperf"])
+
+
+# ------
+# TESTS
+# ------
+
+# offloads = ["gro", "gso", "tso", "tx", "rx"]
+# offload_settings = [ [("gro", "on"), ("gso", "on"), ("tso", "on"), ("tx", "on"), ("rx", "on")],
+                     # [("gro", "off"), ("gso", "on"), ("tso", "on"), ("tx", "on"), ("rx", "on")],
+                     # [("gro", "on"), ("gso", "off"),  ("tso", "off"), ("tx", "on"), ("rx", "on")],
+                     # [("gro", "on"), ("gso", "on"), ("tso", "off"), ("tx", "off"), ("rx", "on")],
+                     # [("gro", "on"), ("gso", "on"), ("tso", "on"), ("tx", "on"), ("rx", "off")]]
+
+ipv = ctl.get_alias("ipv")
+mtu = ctl.get_alias("mtu")
+netperf_duration = int(ctl.get_alias("netperf_duration"))
+nperf_reserve = int(ctl.get_alias("nperf_reserve"))
+nperf_confidence = ctl.get_alias("nperf_confidence")
+nperf_max_runs = int(ctl.get_alias("nperf_max_runs"))
+nperf_cpupin = ctl.get_alias("nperf_cpupin")
+nperf_cpu_util = ctl.get_alias("nperf_cpu_util")
+nperf_mode = ctl.get_alias("nperf_mode")
+nperf_num_parallel = int(ctl.get_alias("nperf_num_parallel"))
+pr_user_comment = ctl.get_alias("perfrepo_comment")
+
+pr_comment = generate_perfrepo_comment([m1, m2], pr_user_comment)
+
+test_if1 = m1.get_interface("test_if")
+test_if1.set_mtu(mtu)
+test_if2 = m2.get_interface("test_if")
+test_if2.set_mtu(mtu)
+
+if nperf_cpupin:
+    m1.run("service irqbalance stop")
+    m2.run("service irqbalance stop")
+
+    m1_phy1 = m1.get_interface("eth1")
+    m1_phy2 = m1.get_interface("eth2")
+    dev_list = [(m1, m1_phy1), (m1, m1_phy2)]
+
+    if test_if2.get_type() == "bond":
+        m2_phy1 = m2.get_interface("eth1")
+        m2_phy2 = m2.get_interface("eth2")
+        dev_list.extend([(m2, m2_phy1), (m2, m2_phy2)])
+    else:
+        dev_list.append((m2, test_if2))
+
+    # this will pin devices irqs to cpu #0
+    for m, d in dev_list:
+        pin_dev_irqs(m, d, 0)
+
+# p_opts = "-L %s" % (test_if2.get_ip(0))
+# if nperf_cpupin and nperf_mode != "multi":
+    # p_opts += " -T%s,%s" % (nperf_cpupin, nperf_cpupin)
+
+# p_opts6 = "-L %s -6" % (test_if2.get_ip(1))
+# if nperf_cpupin and nperf_mode != "multi":
+    # p_opts6 += " -T%s,%s" % (nperf_cpupin, nperf_cpupin)
+
+# if nperf_mode == "multi":
+    # netperf_cli_tcp.unset_option("confidence")
+    # netperf_cli_udp.unset_option("confidence")
+    # netperf_cli_tcp6.unset_option("confidence")
+    # netperf_cli_udp6.unset_option("confidence")
+
+    # netperf_cli_tcp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_tcp6.update_options({"num_parallel": nperf_num_parallel})
+    # netperf_cli_udp6.update_options({"num_parallel": nperf_num_parallel})
+
+ctl.wait(15)
+
+ping_opts = {"count": 100, "interval": 0.1}
+
+# for setting in offload_settings:
+    # dev_features = ""
+    # for offload in setting:
+        # dev_features += " %s %s" % (offload[0], offload[1])
+    # m1.run("ethtool -K %s %s" % (test_if1.get_devname(), dev_features))
+    # m2.run("ethtool -K %s %s" % (test_if2.get_devname(), dev_features))
+
+if ipv in [ 'ipv4', 'both' ]:
+    ping((m1, test_if1, 0, {"scope": 0}),
+         (m2, test_if2, 0, {"scope": 0}),
+         options=ping_opts)
+
+    ctl.wait(2)
+
+    # prepare PerfRepo result for tcp
+    result_tcp = perf_api.new_result("tcp_ipv4_id",
+                                     "tcp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    # for offload in setting:
+        # result_tcp.set_parameter(offload[0], offload[1])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((m1, test_if1, 0, {"scope": 0}),
+                           (m2, test_if2, 0, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    # prepare PerfRepo result for udp
+    result_udp = perf_api.new_result("udp_ipv4_id",
+                                     "udp_ipv4_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    # for offload in setting:
+        # result_udp.set_parameter(offload[0], offload[1])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((m1, test_if1, 0, {"scope": 0}),
+                           (m2, test_if2, 0, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+
+if ipv in [ 'ipv6', 'both' ]:
+    ping6((m1, test_if1, 1, {"scope": 0}),
+          (m2, test_if2, 1, {"scope": 0}),
+          options=ping_opts)
+
+    # prepare PerfRepo result for tcp ipv6
+    result_tcp = perf_api.new_result("tcp_ipv6_id",
+                                     "tcp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    # for offload in setting:
+        # result_tcp.set_parameter(offload[0], offload[1])
+    result_tcp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_tcp.add_tag("multithreaded")
+        result_tcp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_tcp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    tcp_res_data = netperf((m1, test_if1, 1, {"scope": 0}),
+                           (m2, test_if2, 1, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "TCP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_tcp, tcp_res_data)
+    result_tcp.set_comment(pr_comment)
+    perf_api.save_result(result_tcp)
+
+    # prepare PerfRepo result for udp ipv6
+    result_udp = perf_api.new_result("udp_ipv6_id",
+                                     "udp_ipv6_result",
+                                     hash_ignore=[
+                                         'kernel_release',
+                                         'redhat_release'])
+    # for offload in setting:
+        # result_udp.set_parameter(offload[0], offload[1])
+    result_udp.add_tag(product_name)
+    if nperf_mode == "multi":
+        result_udp.add_tag("multithreaded")
+        result_udp.set_parameter('num_parallel', nperf_num_parallel)
+
+    baseline = perf_api.get_baseline_of_result(result_udp)
+    baseline = perfrepo_baseline_to_dict(baseline)
+
+    udp_res_data = netperf((m1, test_if1, 1, {"scope": 0}),
+                           (m2, test_if2, 1, {"scope": 0}),
+                           client_opts={"duration" : netperf_duration,
+                                        "testname" : "UDP_STREAM",
+                                        "confidence" : nperf_confidence,
+                                        "cpu_util" : nperf_cpu_util,
+                                        "runs": nperf_max_runs,
+                                        "netperf_opts" : "-6"},
+                           baseline = baseline,
+                           timeout = (netperf_duration + nperf_reserve)*nperf_max_runs)
+
+    netperf_result_template(result_udp, udp_res_data)
+    result_udp.set_comment(pr_comment)
+    perf_api.save_result(result_udp)
+
+#reset offload states
+# dev_features = ""
+# for offload in offloads:
+    # dev_features += " %s %s" % (offload, "on")
+# m1.run("ethtool -K %s %s" % (test_if1.get_devname(), dev_features))
+# m2.run("ethtool -K %s %s" % (test_if2.get_devname(), dev_features))
+
+if nperf_cpupin:
+    m1.run("service irqbalance start")
+    m2.run("service irqbalance start")

[Jan Tluka]

Thu, May 05, 2016 at 02:26:46PM CEST, [e-mail] wrote:
> From: Ondrej Lichtner
>
> Hi,
>
> what follows is a very large patch set with a lot of stuff happening in it.
> Normally I would have split it up into at least 2 logical parts, but
> unfortunately most of the changes introduced in the patch set is dependent on
> each other. The reasoning behind all of the code should be nicely explained in
> the commit messages, if you have any additional questions don't hesitate to ask.
>
> One thing that I want to explain is the creation of a whole new concept of
> the Controller Device Database, mirroring the one on the Slave so that I could
> work on testing OvS vxlans. For now the implementation is just that and nothing
> more. At the moment this makes the Controller look very ugly seeing as there's
> both an Interface object and a Device object representing the same NIC but with
> different information.
>
> In the future I want to merge these two classes, mostly by replacing the current
> Interface objects with the new Device objects extended with setters and other
> features required for proper configuration. This however is a large
> restructuring of the code and it would either depend or heavily conflict with
> the implementation of python recipes which is why I chose to just do a very
> quick and ugly implementation of a parallel concept that I can use for now while
> we wait to merge in the Python recipes implementation.
>
> v2: fixed a traceback when configuring veth devices - reported by jtluka, the 3
> updated commits are:
> InterfaceManager: fix rescan_devices method
> InterfaceManager: extend Device::if_data method
> InterfaceManager: add if_deleted update messages

I've checked my recipes with veth devices. The issue has been resolved.
Ack to series.

Acked-by: Jan Tluka