[PATCH 0/8] network namespace support for LNST

[Ondrej Lichtner]

From: Ondrej Lichtner

The following set of patches introduces network namespace support to LNST. This
required some changes to the way the slave server works. Currently there will
be a single "master" process that acts as the main server and manages the root
network namespace (or the namespace that lnst-slave was launched in). This is
the process that the controller connects to and communicates with.

When the controller requests the creation of a new network namespace the server
forks itself and using the unshare syscall creates a new namespace. This new
process will now act as the lnst-slave server that manages a different
namespace (configuration, running commands and cleanup). The new process only
communicates with it's parent through a pipe object, and as far as network
communication is concerned this parent is it's connected controller. At this
point the parent process starts acting like a router resending messages from
the network namespaces to the controller and vice versa.

The configuration itself works in this order:
1. namespace creation
2. move interfaces to their target namespaces
3. configure interfaces (create soft interfaces in their target namespace)

When it comes to recipe XML format, the only thing that changed is that you can
now use the attribute "netns" for devices, and for some commands (where it
makes sense).

Next it will probably be a good idea to add support for veth type devices,
signal commands probably shouldn't require to use the netns attribute, netns
support for python taks.

To finish off I'll add an example recipe:
<lnstrecipe>
    <network>
        <host id="1">
            <interfaces>
                <eth id="t1" netns="xyz" label="ttnet"/>
                <eth id="t2" netns="xyz" label="ttnet"/>
                <bond id="t3" netns="xyz">
                    <addresses>
                        <address value="192.168.100.240/24"/>
                    </addresses>
                    <slaves>
                        <slave id="t1"/>
                        <slave id="t2"/>
                    </slaves>
                </bond>
            </interfaces>
        </host>
        <host id="2">
            <interfaces>
                <eth id="t1" label="ttnet">
                    <addresses>
                        <address value="192.168.100.215/24"/>
                    </addresses>
                </eth>
            </interfaces>
        </host>
    </network>

<task>
        <run host="1" command="ip a"/>
        <run host="1" command="ip a" netns="xyz"/>
        <run host="1" module="IcmpPing" netns="xyz">
            <options>
                <option name="addr" value="{ip(2,t1)}"/>
                <option name="count" value="40"/>
                <option name="interval" value="1"/>
            </options>
        </run>
    </task>
</lnstrecipe>

Ondrej Lichtner (8):
  ConnectioHandler: set select timeout to 0
  NetTestSlave: enable/disable NM on demand
  InterfaceManager: change id to device mapping
  slave: network namespace support
  NetConfigDevice: bonding use iproute2 when possible
  XmlProcessing: don't transform None to string
  recipe: network namespace support
  controller: network namespace support

lnst/Common/ConnectionHandler.py     |   2 +-
 lnst/Controller/Machine.py           | 127 ++++++++++++++++++++++---
 lnst/Controller/NetTestController.py |  19 +++-
 lnst/Controller/RecipeParser.py      |  12 +++
 lnst/Controller/XmlProcessing.py     |   3 +-
 lnst/Slave/InterfaceManager.py       |  60 +++++++++---
 lnst/Slave/NetConfigDevice.py        |  31 +++++--
 lnst/Slave/NetTestSlave.py           | 175 ++++++++++++++++++++++++++++++++++-
 lnst/Slave/NmConfigDevice.py         |  10 +-
 schema-recipe.rng                    |  21 +++++
 10 files changed, 420 insertions(+), 40 deletions(-)

[PATCH 1/8] ConnectioHandler: set select timeout to 0

[Ondrej Lichtner]

From: Ondrej Lichtner

Setting the select timeout to 0 makes the call non-blocking which is
required if we want to use more than one connection handler. This will
be usefull for implementing network namespace support.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Common/ConnectionHandler.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py
index 04eef86..042e78a 100644
--- a/lnst/Common/ConnectionHandler.py
+++ b/lnst/Common/ConnectionHandler.py
@@ -72,7 +72,7 @@ class ConnectionHandler(object):
     def check_connections(self):
         requests = []
         try:
-            rl, wl, xl = select.select(self._connections.values(), [], [])
+            rl, wl, xl = select.select(self._connections.values(), [], [], 0)
         except select.error:
             return []
         for f in rl:

[PATCH 2/8] NetTestSlave: enable/disable NM on demand

[Ondrej Lichtner]

From: Ondrej Lichtner

This commit adds 3 new Slave methods:
enable_nm
disable_nm
restore_nm_option

These can be used to manipulate the value of the "use_nm" option, which
controlls whether the slave will use NM for interface configuration.

The patch also adds these methods to the controller class Machine.

This is usefull for network namespaces since NM doesn't support them and
probably never will.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Machine.py |  9 +++++++++
 lnst/Slave/NetTestSlave.py | 29 +++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 56566a5..7ef3837 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -365,6 +365,15 @@ class Machine(object):

self._rpc_call("map_resource", res["hash"], res_type, res_name)

+    def enable_nm(self):
+        return self._rpc_call("enable_nm")
+
+    def disable_nm(self):
+        return self._rpc_call("disable_nm")
+
+    def restore_nm_option(self):
+        return self._rpc_call("restore_nm_option")
+
     def __str__(self):
         return "[Machine hostname(%s) libvirt_domain(%s) interfaces(%d)]" % \
                (self._hostname, self._libvirt_domain, len(self._interfaces))
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index 0b00a68..2cc0cca 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -59,8 +59,11 @@ class SlaveMethods:

self._resource_table = {'module': {}, 'tools': {}}

+        self._bkp_nm_opt_val = lnst_config.get_option("environment", "use_nm")
+
     def hello(self, recipe_path):
         self.machine_cleanup()
+        self.restore_nm_option()

logging.info("Recieved a controller connection.")
         self.clear_resource_table()
@@ -342,6 +345,32 @@ class SlaveMethods:
             file_handle.close()
         self._copy_sources = {}

+    def enable_nm(self):
+        logging.warning("====================================================")
+        logging.warning("Enabling use of NetworkManager on controller request")
+        logging.warning("====================================================")
+        val = lnst_config.get_option("environment", "use_nm")
+        lnst_config.set_option("environment", "use_nm", True)
+        return val
+
+    def disable_nm(self):
+        logging.warning("=====================================================")
+        logging.warning("Disabling use of NetworkManager on controller request")
+        logging.warning("=====================================================")
+        val = lnst_config.get_option("environment", "use_nm")
+        lnst_config.set_option("environment", "use_nm", False)
+        return val
+
+    def restore_nm_option(self):
+        val = lnst_config.get_option("environment", "use_nm")
+        if val == self._bkp_nm_opt_val:
+            return val
+        logging.warning("=========================================")
+        logging.warning("Restoring use_nm option to original value")
+        logging.warning("=========================================")
+        lnst_config.set_option("environment", "use_nm", self._bkp_nm_opt_val)
+        return val
+
 class ServerHandler(object):
     def __init__(self, addr):
         self._connection_handler = ConnectionHandler()

[PATCH 3/8] InterfaceManager: change id to device mapping

[Ondrej Lichtner]

From: Ondrej Lichtner

This patch replaces the id to device mapping with id to if_index
mapping. This removes the need to update the mapping if a Device object
is removed and recreated for the same device. This can happen when
working with network namespaces - moving a device to a different
namespace deletes it in the current namespace.

Because of this we also need a new attribute - _tmp_mapping that maps id
to device for software devices that were not initialized yet and don't
have an if_index assigned to them.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index d00ee4d..94a5c02 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -27,8 +27,9 @@ class IfMgrError(Exception):

class InterfaceManager(object):
     def __init__(self, server_handler):
-        self._devices = {}
-        self._id_mapping = {} #id from the ctl to device
+        self._devices = {} #if_index to device
+        self._id_mapping = {} #id from the ctl to if_index
+        self._tmp_mapping = {} #id from the ctl to newly created device

self._nl_socket = IPRSocket()
         self._nl_socket.bind()
@@ -43,7 +44,7 @@ class InterfaceManager(object):
         elif if_index not in self._devices:
             raise IfMgrError("No interface with index %s found." % if_index)

-        self._id_mapping[if_id] = self._devices[if_index]
+        self._id_mapping[if_id] = if_index
         return

def clear_if_mapping(self):
@@ -71,20 +72,21 @@ class InterfaceManager(object):
             if msg['index'] in self._devices:
                 update_msg = self._devices[msg['index']].update_netlink(msg)
                 if update_msg != None:
-                    for if_id, dev in self._id_mapping.iteritems():
-                        if dev.get_if_index() == msg['index']:
+                    for if_id, if_index in self._id_mapping.iteritems():
+                        if if_index == msg['index']:
                             update_msg["if_id"] = if_id
                             break
                     if "if_id" in update_msg:
                         self._server_handler.send_data_to_ctl(update_msg)
             else:
                 dev = None
-                for d in self._id_mapping.values():
+                for if_id, d in self._tmp_mapping.items():
                     d_cfg = d.get_conf_dict()
-                    if d.get_if_index() == None and\
-                       d_cfg["name"] == msg.get_attr("IFLA_IFNAME"):
-                            dev = d
-                            break
+                    if d_cfg["name"] == msg.get_attr("IFLA_IFNAME"):
+                        dev = d
+                        self._id_mapping[if_id] = msg['index']
+                        del self._tmp_mapping[if_id]
+                        break
                 if dev == None:
                     dev = Device(self)
                 dev.init_netlink(msg)
@@ -98,12 +100,20 @@ class InterfaceManager(object):

def get_mapped_device(self, if_id):
         if if_id in self._id_mapping:
-            return self._id_mapping[if_id]
+            if_index = self._id_mapping[if_id]
+            return self._devices[if_index]
+        elif if_id in self._tmp_mapping:
+            return self._tmp_mapping[if_id]
         else:
             raise IfMgrError("No device with id %s mapped." % if_id)

def get_mapped_devices(self):
-        return self._id_mapping
+        ret = {}
+        for if_id, if_index in self._id_mapping.iteritems():
+            ret[if_id] = self._devices[if_index]
+        for if_id, dev in self._tmp_mapping:
+            ret[if_id] = self._tmp_mapping[if_id]
+        return ret

def get_device(self, if_index):
         if if_index in self._devices:
@@ -134,7 +144,7 @@ class InterfaceManager(object):
         device.set_configuration(config)
         device.configure()

-        self._id_mapping[if_id] = device
+        self._tmp_mapping[if_id] = device
         return config["name"]

def _is_name_used(self, name):

[PATCH 4/8] slave: network namespace support

[Ondrej Lichtner]

From: Ondrej Lichtner

This patch adds network namespace support to lnst slaves.
NetTestSlave:
* New network namespaces are created using slave methods add_namespace
  and del_namespace. New network namespaces created by forking the
  lnst-slave process and using the unshare system call through libc. The
  namespaces are stored in the _net_namespaces dictionary in the
  SlaveMethods class. The process that manages the new namespaces
  communicates with the parent process through a Pipe object.
* To move an interface to a different network namespace we can use the
  set_if_netns and return_if_netns methods.
* The ServerHandler class now uses two ConnectionHandler objects - one
  for the controller and commands and the new one for communication with
  different network namespaces.
* The ServerHandler class is also capable of sending messages to these
  namespaces by the send_data_to_netns method.
* The Server class now support two new message types: "to_netns",
  "from_netns" that encapsulate a message that should be routed to a
  different process.
InterfaceManager:
* The Device class now has a new attribute _netns that is None if the
  device is in the current network namespace or contains the name of the
  namespace that was created by this process. This is used in order to
  not remove Device objects from the InterfaceManager when we move them
  to different network namespaces.
* I also implemented the reconnect_netlink method that creates a new
  netlink socket so that processes in the new namespace don't use the
  socket inherited from the parent process.
NmConfigDevice:
* Modified the is_nm_managed methods - when working with network
  namespaces NM should not be used for configuration.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/InterfaceManager.py |  24 ++++++-
 lnst/Slave/NetTestSlave.py     | 146 ++++++++++++++++++++++++++++++++++++++++-
 lnst/Slave/NmConfigDevice.py   |  10 ++-
 3 files changed, 174 insertions(+), 6 deletions(-)

diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py
index 94a5c02..78cc46b 100644
--- a/lnst/Slave/InterfaceManager.py
+++ b/lnst/Slave/InterfaceManager.py
@@ -50,6 +50,13 @@ class InterfaceManager(object):
     def clear_if_mapping(self):
         self._id_mapping = {}

+    def reconnect_netlink(self):
+        if self._nl_socket != None:
+            self._nl_socket.close()
+            self._nl_socket = None
+        self._nl_socket = IPRSocket()
+        self._nl_socket.bind()
+
     def get_nl_socket(self):
         return self._nl_socket

@@ -93,8 +100,10 @@ class InterfaceManager(object):
                 self._devices[msg['index']] = dev
         elif msg['header']['type'] == RTM_DELLINK:
             if msg['index'] in self._devices:
-                self._devices[msg['index']].del_link()
-                del self._devices[msg['index']]
+                dev = self._devices[msg['index']]
+                if dev.get_netns() != None:
+                    dev.del_link()
+                    del self._devices[msg['index']]
         else:
             return

@@ -199,6 +208,7 @@ class Device(object):
         self._state = None
         self._master = {"primary": None, "other": []}
         self._slaves = []
+        self._netns = None

self._if_manager = if_manager

@@ -209,6 +219,7 @@ class Device(object):
         self._state = nl_msg.get_attr("IFLA_OPERSTATE")
         self._ip = None #TODO
         self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
+        self._netns = None

self._initialized = True

@@ -216,8 +227,10 @@ class Device(object):
         if self._if_index == nl_msg['index']:
             self._hwaddr = normalize_hwaddr(nl_msg.get_attr("IFLA_ADDRESS"))
             self._name = nl_msg.get_attr("IFLA_IFNAME")
+            self._state = nl_msg.get_attr("IFLA_OPERSTATE")
             self._ip = None #TODO
             self.set_master(nl_msg.get_attr("IFLA_MASTER"), primary=True)
+            self._netns = None

link = nl_msg.get_attr("IFLA_LINK")
             if link != None:
@@ -374,3 +387,10 @@ class Device(object):
             self._conf.down()
         else:
             exec_cmd("ip link set %s down" % self._name)
+
+    def set_netns(self, netns):
+        self._netns = netns
+        return
+
+    def get_netns(self):
+        return self._netns
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index 2cc0cca..780cd9a 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -18,6 +18,8 @@ import sys, traceback
 import datetime
 import socket
 import dbus
+import ctypes
+import multiprocessing
 from time import sleep
 from xmlrpclib import Binary
 from tempfile import NamedTemporaryFile
@@ -43,11 +45,14 @@ class SlaveMethods:
     '''
     Exported xmlrpc methods
     '''
-    def __init__(self, command_context, log_ctl, if_manager):
+    def __init__(self, command_context, log_ctl, if_manager, net_namespaces,
+                 server_handler):
         self._packet_captures = {}
         self._if_manager = if_manager
         self._command_context = command_context
         self._log_ctl = log_ctl
+        self._net_namespaces = net_namespaces
+        self._server_handler = server_handler

self._capture_files = {}
         self._copy_targets = {}
@@ -92,6 +97,10 @@ class SlaveMethods:
         self._cache.del_old_entries()
         self.reset_file_transfers()
         self._remove_capture_files()
+
+        for netns in self._net_namespaces.keys():
+            self.del_namespace(netns)
+        self._net_namespaces = {}
         return "bye"

def kill_cmds(self):
@@ -161,6 +170,8 @@ class SlaveMethods:
     def start_packet_capture(self, filt):
         files = {}
         for if_id, dev in self._if_manager.get_mapped_devices().iteritems():
+            if dev.get_netns() != None:
+                continue
             dev_name = dev.get_name()

df_handle = NamedTemporaryFile(delete=False)
@@ -256,6 +267,11 @@ class SlaveMethods:
         logging.info("Performing machine cleanup.")
         self._command_context.cleanup()
         self._if_manager.deconfigure_all()
+
+        for netns in self._net_namespaces.keys():
+            self.del_namespace(netns)
+        self._net_namespaces = {}
+
         self._if_manager.clear_if_mapping()
         self._cache.del_old_entries()
         self.restore_system_config()
@@ -371,9 +387,83 @@ class SlaveMethods:
         lnst_config.set_option("environment", "use_nm", self._bkp_nm_opt_val)
         return val

+    def add_namespace(self, netns):
+        if netns in self._net_namespaces:
+            logging.debug("Network namespace %s already exists." % netns)
+        else:
+            logging.debug("Creating network namespace %s." % netns)
+            read_pipe, write_pipe = multiprocessing.Pipe()
+            pid = os.fork()
+            if pid != 0:
+                self._net_namespaces[netns] = {"pid": pid,
+                                               "pipe": read_pipe}
+                self._server_handler.add_netns(netns, read_pipe)
+                return None
+            elif pid == 0:
+                #create new network namespace
+                libc_name = ctypes.util.find_library("c")
+                CLONE_NEWNET = 0x40000000 #from sched.h
+                libc = ctypes.CDLL(libc_name)
+                libc.unshare(CLONE_NEWNET)
+
+                #set ctl socket to pipe to main netns
+                self._server_handler.close_s_sock()
+                self._server_handler.close_c_sock()
+                self._server_handler.clear_connections()
+                self._server_handler.clear_netns_connections()
+
+                self._if_manager.reconnect_netlink()
+                self._server_handler.add_connection('netlink',
+                                            self._if_manager.get_nl_socket())
+
+                self._server_handler.set_netns(netns)
+                self._server_handler.set_ctl_sock((write_pipe, "root_netns"))
+
+                self._log_ctl.disable_logging()
+                self._log_ctl.set_connection(write_pipe)
+
+                self._if_manager.rescan_devices()
+                logging.debug("Created network namespace %s" % netns)
+                return True
+            else:
+                raise Exception("Fork failed!")
+
+    def del_namespace(self, netns):
+        if netns not in self._net_namespaces:
+            logging.debug("Network namespace %s doesn't exist." % netns)
+            return False
+        else:
+            netns_pid = self._net_namespaces[netns]["pid"]
+            os.kill(netns_pid, signal.SIGTERM)
+            os.waitpid(netns_pid, 0)
+            logging.debug("Network namespace %s removed." % netns)
+
+            self._net_namespaces[netns]["pipe"].close()
+            self._server_handler.del_netns(netns)
+            del self._net_namespaces[netns]
+            return True
+
+    def set_if_netns(self, if_id, netns):
+        netns_pid = self._net_namespaces[netns]["pid"]
+
+        device = self._if_manager.get_mapped_device(if_id)
+        dev_name = device.get_name()
+        device.set_netns(netns)
+
+        exec_cmd("ip link set %s netns %d" % (dev_name, netns_pid))
+        return True
+
+    def return_if_netns(self, if_id):
+        device = self._if_manager.get_mapped_device(if_id)
+        dev_name = device.get_name()
+        ppid = os.getppid()
+        exec_cmd("ip link set %s netns %d" % (dev_name, ppid))
+        return True
+
 class ServerHandler(object):
     def __init__(self, addr):
         self._connection_handler = ConnectionHandler()
+        self._netns_con_handler = ConnectionHandler()
         try:
             self._s_socket = socket.socket()
             self._s_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
@@ -384,6 +474,7 @@ class ServerHandler(object):
             exit(1)

self._c_socket = None
+        self._netns = None

def accept_connection(self):
         self._c_socket, addr = self._s_socket.accept()
@@ -400,8 +491,26 @@ class ServerHandler(object):
         else:
             return None

+    def set_ctl_sock(self, sock):
+        if self._c_socket != None:
+            self._c_socket.close()
+            self._c_socket = None
+        self._c_socket = sock
+        self._connection_handler.add_connection(self._c_socket[1],
+                                                self._c_socket[0])
+
+    def close_s_sock(self):
+        self._s_socket.close()
+        self._s_socket = None
+
+    def close_c_sock(self):
+        self._c_socket[0].close()
+        self._connection_handler.remove_connection(self._c_socket[0])
+        self._c_socket = None
+
     def get_messages(self):
         messages = self._connection_handler.check_connections()
+        messages += self._netns_con_handler.check_connections()

#push ctl messages to the end of message queue, this ensures that
         #update messages are handled first
@@ -422,10 +531,21 @@ class ServerHandler(object):

def send_data_to_ctl(self, data):
         if self._c_socket != None:
+            if self._netns != None:
+                data = {"type": "from_netns",
+                        "netns": self._netns,
+                        "data": data}
             return send_data(self._c_socket[0], data)
         else:
             return False

+    def send_data_to_netns(self, netns, data):
+        netns_con = self._netns_con_handler.get_connection(netns)
+        if netns_con == None:
+            raise Exception("No such namespace!")
+        else:
+            return send_data(netns_con, data)
+
     def add_connection(self, id, connection):
         self._connection_handler.add_connection(id, connection)

@@ -441,6 +561,19 @@ class ServerHandler(object):
             self.remove_connection(key)
             self.add_connection(key, connection)

+    def set_netns(self, netns):
+        self._netns = netns
+
+    def add_netns(self, netns, connection):
+        self._netns_con_handler.add_connection(netns, connection)
+
+    def del_netns(self, netns):
+        connection = self._netns_con_handler.get_connection(netns)
+        self._netns_con_handler.remove_connection(connection)
+
+    def clear_netns_connections(self):
+        self._netns_con_handler.clear_connections()
+
 class NetTestSlave:
     def __init__(self, log_ctl, port = DefaultRPCPort):
         die_when_parent_die()
@@ -448,8 +581,12 @@ class NetTestSlave:
         self._cmd_context = NetTestCommandContext()
         self._server_handler = ServerHandler(("", port))
         self._if_manager = InterfaceManager(self._server_handler)
+
+        self._net_namespaces = {}
+
         self._methods = SlaveMethods(self._cmd_context, log_ctl,
-                                     self._if_manager)
+                                     self._if_manager, self._net_namespaces,
+                                     self._server_handler)

self.register_die_signal(signal.SIGHUP)
         self.register_die_signal(signal.SIGINT)
@@ -538,6 +675,11 @@ class NetTestSlave:
                     self._cmd_context.del_cmd(cmd)
         elif msg["type"] == "netlink":
             self._if_manager.handle_netlink_msgs(msg["data"])
+        elif msg["type"] == "from_netns":
+            self._server_handler.send_data_to_ctl(msg["data"])
+        elif msg["type"] == "to_netns":
+            netns = msg["netns"]
+            self._server_handler.send_data_to_netns(netns, msg["data"])
         else:
             raise Exception("Recieved unknown command")

diff --git a/lnst/Slave/NmConfigDevice.py b/lnst/Slave/NmConfigDevice.py
index 460fd26..e07c0ba 100644
--- a/lnst/Slave/NmConfigDevice.py
+++ b/lnst/Slave/NmConfigDevice.py
@@ -137,6 +137,8 @@ class NmConfigDeviceGeneric(object):

@classmethod
     def is_nm_managed(cls, dev_config, if_manager):
+        if dev_config["netns"] != None:
+            return False
         return is_nm_managed_by_name(dev_config["name"])

def _wait_for_state(self, new_state, old_state, reason):
@@ -808,5 +810,9 @@ type_class_mapping = {
 }

def is_nm_managed(dev_config, if_manager):
-    return type_class_mapping[dev_config["type"]].is_nm_managed(dev_config,
-                                                                if_manager)
+    if lnst_config.get_option("environment", "use_nm") and\
+       check_process_running("NetworkManager"):
+        return type_class_mapping[dev_config["type"]].is_nm_managed(dev_config,
+                                                                    if_manager)
+    else:
+        return False

[PATCH 5/8] NetConfigDevice: bonding use iproute2 when possible

[Ondrej Lichtner]

From: Ondrej Lichtner

This patch makes the NetConfigDeviceBond class use the ip command for
device creation/destruction and enslavement instead of sysfs. This is
because bond devices created through sysfs are always in the root
network namespace due to a kernel bug.

However on older systems bonding doesn't support netlink so I added an
exception to use sysfs for kernels older than 3.10. I chose 3.10 because
that's the oldest one I tested that worked - it's the current RHEL7
kernel.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Slave/NetConfigDevice.py | 31 +++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py
index 054d15a..338df67 100644
--- a/lnst/Slave/NetConfigDevice.py
+++ b/lnst/Slave/NetConfigDevice.py
@@ -14,6 +14,8 @@ [e-mail] (Jiri Pirko)
 import logging
 import re
 import sys
+from platform import release
+from distutils.version import LooseVersion
 from lnst.Common.ExecCmd import exec_cmd
 from lnst.Slave.NetConfigCommon import get_slaves, get_option, get_slave_option
 from lnst.Common.Utils import kmod_in_use, bool_it
@@ -86,9 +88,16 @@ class NetConfigDeviceBond(NetConfigDeviceGeneric):
     _moduleparams = "max_bonds=0"

def _add_rm_bond(self, mark):
-        bond_masters = "/sys/class/net/bonding_masters"
-        exec_cmd('echo "%s%s" > %s' % (mark, self._dev_config["name"],
-                                       bond_masters))
+        #3.10 works on rhel7, I didn't test which oldest version works...
+        if LooseVersion(release()) > LooseVersion('3.10'):
+            if mark == "+":
+                exec_cmd('ip link add %s type bond' % self._dev_config["name"])
+            elif mark == "-":
+                exec_cmd('ip link del %s' % self._dev_config["name"])
+        else:
+            bond_masters = "/sys/class/net/bonding_masters"
+            exec_cmd('echo "%s%s" > %s' % (mark, self._dev_config["name"],
+                                           bond_masters))

def _get_bond_dir(self):
         return "/sys/class/net/%s/bonding" % self._dev_config["name"]
@@ -114,10 +123,20 @@ class NetConfigDeviceBond(NetConfigDeviceGeneric):
             slave_dev = self._if_manager.get_mapped_device(slave_id)
             slave_conf = slave_dev.get_conf_dict()
             slave_name = slave_dev.get_name()
-            if (mark == "+"):
+            bond_name = self._dev_config["name"]
+            if mark == "+":
                 slave_dev.down()
-            exec_cmd('echo "%s%s" > %s/slaves' % (mark, slave_name,
-                                                  self._get_bond_dir()))
+
+            #3.10 works on rhel7, I didn't test which oldest version works...
+            if LooseVersion(release()) > LooseVersion('3.10'):
+                if mark == "+":
+                    exec_cmd('ip link set %s master %s' % (slave_name,
+                                                           bond_name))
+                elif mark == "-":
+                    exec_cmd('ip link set %s nomaster' % (slave_name))
+            else:
+                exec_cmd('echo "%s%s" > %s/slaves' % (mark, slave_name,
+                                                      self._get_bond_dir()))

def configure(self):
         self._add_rm_bond("+")

[PATCH 6/8] XmlProcessing: don't transform None to string

[Ondrej Lichtner]

From: Ondrej Lichtner

If the XmlData dictionary contained a None value it would be transformed
to a string. This fixes that.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/XmlProcessing.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lnst/Controller/XmlProcessing.py b/lnst/Controller/XmlProcessing.py
index 771ead5..40debae 100644
--- a/lnst/Controller/XmlProcessing.py
+++ b/lnst/Controller/XmlProcessing.py
@@ -140,7 +140,8 @@ class XmlData(dict):

def __getitem__(self, key):
         value = super(XmlData, self).__getitem__(key)
-        if type(value) == XmlData or type(value) == XmlCollection:
+        if type(value) == XmlData or type(value) == XmlCollection\
+           or value == None:
             return value

return str(value)

[PATCH 7/8] recipe: network namespace support

[Ondrej Lichtner]

From: Ondrej Lichtner

This patch extends the recipe XML files to support network namespaces.
This is a pretty simple change:
* all device elemenets (eth, bond, bridge, ...) now support the "netns"
  attribute which can be an arbitrary string
* command elements config, run, kill, intr and wait now also support the
  "netns" attribute

At the moment there are no checks whether or not the value of the
attribute makes sense. The controller will create the namespaces based
on the device configuration so if a command uses a different namespace
it will fail. The same goes for background commands - the corresponding
signal command needs to be run in the same namespace or it will fail.
This probably can be improved in the future.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/RecipeParser.py | 12 ++++++++++++
 schema-recipe.rng               | 21 +++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py
index df0b47a..76da1c6 100644
--- a/lnst/Controller/RecipeParser.py
+++ b/lnst/Controller/RecipeParser.py
@@ -86,6 +86,10 @@ class RecipeParser(XmlParser):
         iface["id"] = self._get_attribute(iface_tag, "id")
         iface["type"] = iface_tag.tag

+        iface["netns"] = None
+        if self._has_attribute(iface_tag, "netns"):
+            iface["netns"] = self._get_attribute(iface_tag, "netns")
+
         # params
         params_tag = iface_tag.find("params")
         params = self._process_params(params_tag)
@@ -283,6 +287,10 @@ class RecipeParser(XmlParser):
         cmd = XmlData(cmd_tag)
         cmd["host"] = self._get_attribute(cmd_tag, "host")

+        cmd["netns"] = None
+        if self._has_attribute(cmd_tag, "netns"):
+            cmd["netns"] = self._get_attribute(cmd_tag, "netns")
+
         has_module = self._has_attribute(cmd_tag, "module")
         has_command = self._has_attribute(cmd_tag, "command")
         has_from = self._has_attribute(cmd_tag, "from")
@@ -323,6 +331,10 @@ class RecipeParser(XmlParser):
         cmd["type"] = "config"
         cmd["host"] = self._get_attribute(cmd_tag, "host")

+        cmd["netns"] = None
+        if self._has_attribute(cmd_tag, "netns"):
+            cmd["netns"] = self._get_attribute(cmd_tag, "netns")
+
         if self._has_attribute(cmd_tag, "persistent"):
             cmd["persistent"] = self._get_attribute(cmd_tag, "persistent")

diff --git a/schema-recipe.rng b/schema-recipe.rng
index b623a99..6795f72 100644
--- a/schema-recipe.rng
+++ b/schema-recipe.rng
@@ -146,6 +146,9 @@
         <element name="eth">
             <attribute name="id"/>
             <attribute name="label"/>
+            <optional>
+                <attribute name="netns"/>
+            </optional>
             <interleave>
                 <optional>
                     <ref name="define"/>
@@ -179,6 +182,9 @@
     <define name="ovs_bridge">
         <element name="ovs_bridge">
             <attribute name="id"/>
+            <optional>
+                <attribute name="netns"/>
+            </optional>

<interleave>
                 <optional>
@@ -275,6 +281,9 @@
     </define>

<define name="softdevice">
+        <optional>
+            <attribute name="netns"/>
+        </optional>
         <interleave>
             <optional>
                 <ref name="define"/>
@@ -383,6 +392,10 @@
             <attribute name="host"/>

<optional>
+                <attribute name="netns"/>
+            </optional>
+
+            <optional>
                 <attribute name="option"/>
                 <attribute name="value"/>
             </optional>
@@ -421,6 +434,10 @@
         <element name="run">
             <attribute name="host"/>

+            <optional>
+                <attribute name="netns"/>
+            </optional>
+
             <choice>
                 <attribute name="module"/>
                 <ref name="run_command"/>
@@ -483,6 +500,10 @@
     <define name="signal_command">
         <attribute name="host"/>
         <attribute name="bg_id"/>
+
+        <optional>
+            <attribute name="netns"/>
+        </optional>
     </define>

<define name="ctl_wait">

[PATCH 8/8] controller: network namespace support

[Ondrej Lichtner]

From: Ondrej Lichtner

This patch adds the rest of the controller support for network
namespaces.
Machine:
* The most important addition is the _rpc_call_to_netns method of the
  Machine class. This method ensures that whent the slave recieves the
  message it will route it's contents to the specified network namespace.
* Packet capture was modified so that we also capture on devices that
  are in different namespaces.
* To add/remove network namespaces we use the add_netns and del_netns
  methods of the Machine class.
Interface:
* If the interface is supposed to be in a non-root network namespace the
  configure method first moves the interface to the correct namespace
  and then configures the interface in the new interface.
  Deconfiguration is the inverse process - first we deconfigure the
  interface and then we return the interface to the previous namespace.
* SoftInterface configuration creates the interfaces in the target
  namespace since it might be impossible to move them to a different
  namespace once they're created.
NetTestController:
* The _prepare_network method finds all required network namespaces
  (based on interface configuration) and takes care of creating these
  devices before configuring the interfaces. It also disables the use of
  network manager on the slave machine since NM doesn't support network
  namespaces.
* In addition to that it takes care of extracting the recipe information
  about network namespaces and storing it in the Interface object, and
  command dictionary.

Signed-off-by: Ondrej Lichtner 
---
 lnst/Controller/Machine.py           | 118 +++++++++++++++++++++++++++++++----
 lnst/Controller/NetTestController.py |  19 +++++-
 2 files changed, 124 insertions(+), 13 deletions(-)

diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index 7ef3837..94d4fbd 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -71,6 +71,7 @@ class Machine(object):
         self._mac_pool = None

self._interfaces = []
+        self._namespaces = []

def _add_interface(self, if_id, if_type, cls):
         if if_id != None:
@@ -148,6 +149,15 @@ class Machine(object):

return result

+    def _rpc_call_to_netns(self, netns, method_name, *args):
+        data = {"type": "command", "method_name": method_name, "args": args}
+        msg = {"type": "to_netns", "netns": netns, "data": data}
+
+        self._msg_dispatcher.send_message(self._id, msg)
+        result = self._msg_dispatcher.wait_for_result(self._id)
+
+        return result
+
     def configure(self, recipe_name):
         """ Prepare the machine

@@ -198,6 +208,8 @@ class Machine(object):
         ordered_ifaces = self.get_ordered_interfaces()
         try:
             self._rpc_call("kill_cmds")
+            for netns in self._namespaces:
+                self._rpc_call_to_netns(netns, "kill_cmds")

if deconfigure:
                 ordered_ifaces.reverse()
@@ -206,6 +218,8 @@ class Machine(object):
                 for iface in ordered_ifaces:
                     iface.cleanup()

+                self.del_namespaces()
+
             self._rpc_call("bye")
         except:
             #cleanup is only meaningful on dynamic interfaces, and should
@@ -217,6 +231,7 @@ class Machine(object):
                 iface.cleanup()
             raise
         finally:
+            self.restore_nm_option()
             self._msg_dispatcher.disconnect_slave(self.get_id())

self._configured = False
@@ -239,7 +254,11 @@ class Machine(object):
             signal.alarm(DEFAULT_TIMEOUT)

try:
-            cmd_res = self._rpc_call("run_command", command)
+            if 'netns' in command and command['netns'] != None:
+                netns = command["netns"]
+                cmd_res = self._rpc_call_to_netns(netns, "run_command", command)
+            else:
+                cmd_res = self._rpc_call("run_command", command)
         except MachineError as exc:
             if "bg_id" in command:
                 cmd_res = self._rpc_call("kill_command", command["bg_id"])
@@ -281,7 +300,10 @@ class Machine(object):
         self._mac_pool = mac_pool

def restore_system_config(self):
-        return self._rpc_call("restore_system_config")
+        self._rpc_call("restore_system_config")
+        for netns in self._namespaces:
+            self._rpc_call_to_netns(netns, "restore_system_config")
+        return True

def set_network_bridges(self, bridges):
         self._network_bridges = bridges
@@ -299,10 +321,28 @@ class Machine(object):
         return self._domain_ctl

def start_packet_capture(self):
-        return self._rpc_call("start_packet_capture", "")
+        namespaces = set()
+        for iface in self._interfaces:
+            namespaces.add(iface.get_netns())
+
+        tmp = {}
+        for netns in namespaces:
+            if netns == None:
+                tmp.update(self._rpc_call("start_packet_capture", ""))
+            else:
+                tmp.update(self._rpc_call_to_netns(netns, "start_packet_capture", ""))
+        return tmp

def stop_packet_capture(self):
-        self._rpc_call("stop_packet_capture")
+        namespaces = set()
+        for iface in self._interfaces:
+            namespaces.add(iface.get_netns())
+
+        for netns in namespaces:
+            if netns == None:
+                self._rpc_call("stop_packet_capture")
+            else:
+                self._rpc_call_to_netns(netns, "stop_packet_capture")

def copy_file_to_machine(self, local_path, remote_path=None):
         remote_path = self._rpc_call("start_copy_to", remote_path)
@@ -378,6 +418,19 @@ class Machine(object):
         return "[Machine hostname(%s) libvirt_domain(%s) interfaces(%d)]" % \
                (self._hostname, self._libvirt_domain, len(self._interfaces))

+    def add_netns(self, netns):
+        self._namespaces.append(netns)
+        return self._rpc_call("add_namespace", netns)
+
+    def del_netns(self, netns):
+        return self._rpc_call("del_namespace", netns)
+
+    def del_namespaces(self):
+        for netns in self._namespaces:
+            self.del_netns(netns)
+        self._namespaces = []
+        return True
+
 class Interface(object):
     """ Abstraction of a test network interface on a slave machine

@@ -404,6 +457,8 @@ class Interface(object):

self._ovs_conf = None

+        self._netns = None
+
     def get_id(self):
         return self._id

@@ -478,6 +533,12 @@ class Interface(object):
     def get_ovs_conf(self):
         return self._ovs_conf

+    def set_netns(self, netns):
+        self._netns = netns
+
+    def get_netns(self):
+        return self._netns
+
     def get_prefix(self, num):
         try:
             return self._addresses[num].split('/')[1]
@@ -490,7 +551,7 @@ class Interface(object):
                   "options": self._options,
                   "slave_options": self._slave_options,
                   "master": None, "other_masters": [],
-                  "ovs_conf": self._ovs_conf}
+                  "ovs_conf": self._ovs_conf, "netns": self._netns}

if self._master["primary"] != None:
             config["master"] = self._master["primary"].get_id()
@@ -501,10 +562,18 @@ class Interface(object):
         return config

def up(self):
-        self._machine._rpc_call("set_device_up", self._id)
+        netns = self._netns
+        if netns != None:
+            self._machine._rpc_call_to_netns(netns, "set_device_up", self._id)
+        else:
+            self._machine._rpc_call("set_device_up", self._id)

def down(self):
-        self._machine._rpc_call("set_device_down", self._id)
+        netns = self._netns
+        if netns != None:
+            self._machine._rpc_call_to_netns(netns, "set_device_down", self._id)
+        else:
+            self._machine._rpc_call("set_device_down", self._id)

def initialize(self):
         phys_devs = self._machine._rpc_call("map_if_by_hwaddr",
@@ -536,15 +605,26 @@ class Interface(object):
         logging.info("Configuring interface %s on machine %s", self.get_id(),
                      self._machine.get_id())

-        self._machine._rpc_call("configure_interface", self.get_id(),
-                               self._get_config())
+        if self._netns != None:
+            self._machine._rpc_call("set_if_netns", self.get_id(), self._netns)
+            self._machine._rpc_call_to_netns(self._netns, "configure_interface",
+                                             self.get_id(), self._get_config())
+        else:
+            self._machine._rpc_call("configure_interface", self.get_id(),
+                                    self._get_config())
         self._configured = True

def deconfigure(self):
         if not self._configured:
             return

-        self._machine._rpc_call("deconfigure_interface", self.get_id())
+        if self._netns != None:
+            self._machine._rpc_call_to_netns(self._netns,
+                                         "deconfigure_interface", self.get_id())
+            self._machine._rpc_call_to_netns(self._netns,
+                                         "return_if_netns", self.get_id())
+        else:
+            self._machine._rpc_call("deconfigure_interface", self.get_id())
         self._configured = False

class StaticInterface(Interface):
@@ -665,11 +745,25 @@ class SoftInterface(Interface):
         logging.info("Configuring interface %s on machine %s", self.get_id(),
                      self._machine.get_id())

-        dev_name = self._machine._rpc_call("create_soft_interface", self._id,
-                                           self._get_config())
+        if self._netns != None:
+            dev_name = self._machine._rpc_call_to_netns(self._netns,
+                          "create_soft_interface", self._id, self._get_config())
+        else:
+            dev_name = self._machine._rpc_call("create_soft_interface",
+                                               self._id, self._get_config())
         self.set_devname(dev_name)
         self._configured = True

+    def deconfigure(self):
+        if not self._configured:
+            return
+
+        if self._netns != None:
+            self._machine._rpc_call_to_netns(self._netns,
+                                         "deconfigure_interface", self.get_id())
+        else:
+            self._machine._rpc_call("deconfigure_interface", self.get_id())
+        self._configured = False

class UnusedInterface(Interface):
     """ Unused interface for this test
diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py
index 890bba7..f62ca45 100644
--- a/lnst/Controller/NetTestController.py
+++ b/lnst/Controller/NetTestController.py
@@ -178,10 +178,21 @@ class NetTestController:

for machine_xml_data in recipe["machines"]:
             m_id = machine_xml_data["id"]
+            m = machines[m_id]
+            namespaces = set()
             for iface_xml_data in machine_xml_data["interfaces"]:
                 self._prepare_interface(m_id, iface_xml_data)

-            ifaces = machines[m_id].get_ordered_interfaces()
+                if iface_xml_data["netns"] != None:
+                    namespaces.add(iface_xml_data["netns"])
+
+            if len(namespaces) > 0:
+                m.disable_nm()
+
+            ifaces = m.get_ordered_interfaces()
+            for netns in namespaces:
+                m.add_netns(netns)
+
             for iface in ifaces:
                 iface.configure()
             for iface in ifaces:
@@ -296,6 +307,9 @@ class NetTestController:
         if "ovs_conf" in iface_xml_data:
             iface.set_ovs_conf(iface_xml_data["ovs_conf"])

+        if iface_xml_data["netns"] != None:
+            iface.set_netns(iface_xml_data["netns"])
+
     def _prepare_tasks(self):
         self._tasks = []
         for task_data in self._recipe["tasks"]:
@@ -354,6 +368,9 @@ class NetTestController:
                 msg = "Invalid host id '%s'." % cmd["host"]
                 raise RecipeError(msg, cmd_data)

+        if "netns" in cmd_data:
+            cmd["netns"] = cmd_data["netns"]
+
         if "expect" in cmd_data:
             expect = cmd_data["expect"]
             if expect not in ["pass", "fail"]:

[Jiri Pirko]

Set applied. Great job!

Wed, Aug 13, 2014 at 06:34:49PM CEST, [e-mail] wrote:
> From: Ondrej Lichtner
>
> The following set of patches introduces network namespace support to LNST. This
> required some changes to the way the slave server works. Currently there will
> be a single "master" process that acts as the main server and manages the root
> network namespace (or the namespace that lnst-slave was launched in). This is
> the process that the controller connects to and communicates with.
>
> When the controller requests the creation of a new network namespace the server
> forks itself and using the unshare syscall creates a new namespace. This new
> process will now act as the lnst-slave server that manages a different
> namespace (configuration, running commands and cleanup). The new process only
> communicates with it's parent through a pipe object, and as far as network
> communication is concerned this parent is it's connected controller. At this
> point the parent process starts acting like a router resending messages from
> the network namespaces to the controller and vice versa.
>
> The configuration itself works in this order:
> namespace creation
> move interfaces to their target namespaces
> configure interfaces (create soft interfaces in their target namespace)
>
> When it comes to recipe XML format, the only thing that changed is that you can
> now use the attribute "netns" for devices, and for some commands (where it
> makes sense).
>
> Next it will probably be a good idea to add support for veth type devices,
> signal commands probably shouldn't require to use the netns attribute, netns
> support for python taks.
>
> To finish off I'll add an example recipe:
>
> <lnstrecipe>
>    <network>
>        <host id="1">
>            <interfaces>
>                <eth id="t1" netns="xyz" label="ttnet"/>
>                <eth id="t2" netns="xyz" label="ttnet"/>
>                <bond id="t3" netns="xyz">
>                    <addresses>
>                        <address value="192.168.100.240/24"/>
>                    </addresses>
>                    <slaves>
>                        <slave id="t1"/>
>                        <slave id="t2"/>
>                    </slaves>
>                </bond>
>            </interfaces>
>        </host>
>        <host id="2">
>            <interfaces>
>                <eth id="t1" label="ttnet">
>                    <addresses>
>                        <address value="192.168.100.215/24"/>
>                    </addresses>
>                </eth>
>            </interfaces>
>        </host>
>    </network>
>
>    <task>
>        <run host="1" command="ip a"/>
>        <run host="1" command="ip a" netns="xyz"/>
>        <run host="1" module="IcmpPing" netns="xyz">
>            <options>
>                <option name="addr" value="{ip(2,t1)}"/>
>                <option name="count" value="40"/>
>                <option name="interval" value="1"/>
>            </options>
>        </run>
>    </task>
> </lnstrecipe>
>
> Ondrej Lichtner (8):
>  ConnectioHandler: set select timeout to 0
>  NetTestSlave: enable/disable NM on demand
>  InterfaceManager: change id to device mapping
>  slave: network namespace support
>  NetConfigDevice: bonding use iproute2 when possible
>  XmlProcessing: don't transform None to string
>  recipe: network namespace support
>  controller: network namespace support
>
> lnst/Common/ConnectionHandler.py     |   2 +-
> lnst/Controller/Machine.py           | 127 ++++++++++++++++++++++---
> lnst/Controller/NetTestController.py |  19 +++-
> lnst/Controller/RecipeParser.py      |  12 +++
> lnst/Controller/XmlProcessing.py     |   3 +-
> lnst/Slave/InterfaceManager.py       |  60 +++++++++---
> lnst/Slave/NetConfigDevice.py        |  31 +++++--
> lnst/Slave/NetTestSlave.py           | 175 ++++++++++++++++++++++++++++++++++-
> lnst/Slave/NmConfigDevice.py         |  10 +-
> schema-recipe.rng                    |  21 +++++
> 10 files changed, 420 insertions(+), 40 deletions(-)
>
> -- 
> 1.9.3
>
> LNST-developers mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/lnst-developers