Hello everyone,
first working draft implementation is completed. Please, read this cover letter, check the patches and let me know what you like, what you don't like, etc. Please, note, that this is not final implementation.
NOTE - patches are not rebased to current HEAD, if you want to try it, please, reset your branch to commit 2398f93.
What doesn't work ================= multi_match - didn't have time and resources to check physical machines, on virt setup it works, but ends with exception due to no res value returned loopbacks - can't be created netem support - not implemented yet
Description of new mechanics ============================ Since we need to run TaskAPI methods to get machine requirements from PyRecipe, methods like provision_machines, prepare_network are run after the python recipe is executed. Methods add_host and add_interface create machine requirements dict in ControllerAPI object. When lnst.match() is called, it runs match algorithm with mreq dict and if match is found, it prepares network and binds Machine and Interface objects with their HostAPI and InterfaceAPI counterparts. When everyting is prepared, execution returns to PyRecipe, where task phase of PyRecipe is executed. Task phase is the same like before, only new method is breakpoint() which will be useful for debugging, as it allows user to pause the execution in whatever part of test. TaskAPI is now used right from lnst module, all callable methods are exported via __init__.py file from lnst/ dir. Folder pyrecipes/ contains few working examples you can try.
TODO ==== * multimatch support * config_only mode should be renamed and polished * TaskAPI create_ovs() method * support for loopbacks * NetEm support * polishing of the code
Jiri Prochazka (5): __init__.py: draft for PyRecipes lnst-ctl: draft for PyRecipes NetTestController: draft for PyRecipes Task: draft for PyRecipes PyRecipes: add example PyRecipes
lnst-ctl | 16 +- lnst/Controller/NetTestController.py | 396 ++++++++++------------------------- lnst/Controller/Task.py | 240 +++++++++++---------- lnst/__init__.py | 1 + pyrecipes/3_vlans.py | 34 +++ pyrecipes/example.py | 33 +++ pyrecipes/ping_flood.py | 48 +++++ 7 files changed, 362 insertions(+), 406 deletions(-) create mode 100644 pyrecipes/3_vlans.py create mode 100644 pyrecipes/example.py create mode 100644 pyrecipes/ping_flood.py
exports TaskAPI methods to lnst module New TaskAPI will be used in this format:
import lnst
m1 = lnst.add_host()
m1.add_interface(label="tnet")
while lnst.match(): mod = lnst.get_module(...) lnst.wait(1) ...
Thus we need to export methods available via TaskAPI to this __init__.py file
Signed-off-by: Jiri Prochazka jprochaz@redhat.com --- lnst/__init__.py | 1 + 1 file changed, 1 insertion(+)
diff --git a/lnst/__init__.py b/lnst/__init__.py index e69de29..924a8ff 100644 --- a/lnst/__init__.py +++ b/lnst/__init__.py @@ -0,0 +1 @@ +from lnst.Controller.Task import match, add_host, wait, get_alias, get_module, breakpoint
Changes: * add multi_match argument for NetTestController __init__ * add init of TaskAPI before executing the test * when folder user in recipe_path, use .py files instead of .xml
Signed-off-by: Jiri Prochazka jprochaz@redhat.com --- lnst-ctl | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/lnst-ctl b/lnst-ctl index 6d656f8..ef70cb1 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -112,7 +112,8 @@ def get_recipe_result(action, file_path, log_ctl, res_serializer, defined_aliases=defined_aliases, overriden_aliases=overriden_aliases, reduce_sync=reduce_sync, - restrict_pools=pools) + restrict_pools=pools, + multi_match=multi_match) except XmlProcessingError as err: log_exc_traceback() logging.error(err) @@ -126,9 +127,13 @@ def get_recipe_result(action, file_path, log_ctl, res_serializer, res = {} if matches == 1: try: - nettestctl.provision_machines() - nettestctl.print_match_description() + # init TaskAPI.Ctl + nettestctl.init_taskapi() res = exec_action(action, nettestctl) + except NoMatchError as err: + no_match = True + log_ctl.unset_recipe() + logging.warning("Match %d not possible." % matches) except Exception as err: no_match = True log_exc_traceback() @@ -138,11 +143,10 @@ def get_recipe_result(action, file_path, log_ctl, res_serializer, retval = RETVAL_ERR elif matches > 1: try: - nettestctl.provision_machines() + nettestctl.init_taskapi() log_ctl.set_recipe(file_path, expand="match_%d" % matches) recipe_head_log_entry(file_path, log_dir, matches) res_serializer.add_recipe(file_path, matches) - nettestctl.print_match_description() res = exec_action(action, nettestctl) except NoMatchError as err: no_match = True @@ -337,7 +341,7 @@ def main(): all_files.sort() for f in all_files: recipe_file = os.path.join(recipe_path, f) - if re.match(r'^.*.xml$', recipe_file): + if re.match(r'^.*.py$', recipe_file): recipe_files.append(recipe_file) else: recipe_files.append(recipe_path)
Changes: * multi_match flag added to NTC attributes * run_mode flag added to NTC attributes * _prepare_network now uses mreq dict instead of parsed XML recipe * new method set_machine_requirements() for SlavePool * removed resource_sync from provision_machines * new method prepare_test_env() which provisions machines, prints match desc, prepares network and initializes HostAPI objects * get_aliases() method reworked
Signed-off-by: Jiri Prochazka jprochaz@redhat.com --- lnst/Controller/NetTestController.py | 396 ++++++++++------------------------- 1 file changed, 108 insertions(+), 288 deletions(-)
diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 8fdd43e..431b909 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -24,7 +24,7 @@ from lnst.Common.NetUtils import MacPool from lnst.Common.Utils import md5sum, dir_md5sum from lnst.Common.Utils import check_process_running, bool_it, get_module_tools from lnst.Common.NetTestCommand import str_command, CommandException -from lnst.Controller.RecipeParser import RecipeParser, RecipeError +from lnst.Controller.RecipeParser import RecipeError from lnst.Controller.SlavePool import SlavePool from lnst.Controller.Machine import MachineError, VirtualInterface from lnst.Controller.Machine import StaticInterface @@ -54,18 +54,21 @@ class NetTestController: res_serializer=None, pool_checks=True, packet_capture=False, defined_aliases=None, overriden_aliases=None, - reduce_sync=False, restrict_pools=[]): + reduce_sync=False, restrict_pools=[], multi_match=False): self._res_serializer = res_serializer self._remote_capture_files = {} self._log_ctl = log_ctl - self._recipe_path = Path(None, recipe_path).abs_path() + self._recipe_path = Path(None, recipe_path) self._msg_dispatcher = MessageDispatcher(log_ctl) self._packet_capture = packet_capture self._reduce_sync = reduce_sync - self._parser = RecipeParser(recipe_path) + self._defined_aliases = defined_aliases + self._multi_match = multi_match
self.remove_saved_machine_config()
+ self.run_mode = "run" + self._machines = {} self._network_bridges = {} self._tasks = [] @@ -73,10 +76,6 @@ class NetTestController: mac_pool_range = lnst_config.get_option('environment', 'mac_pool_range') self._mac_pool = MacPool(mac_pool_range[0], mac_pool_range[1])
- self._parser.set_machines(self._machines) - self._parser.set_aliases(defined_aliases, overriden_aliases) - self._recipe = self._parser.parse() - conf_pools = lnst_config.get_pools() pools = {} if len(restrict_pools) > 0: @@ -93,9 +92,6 @@ class NetTestController: sp = SlavePool(pools, pool_checks) self._slave_pool = sp
- mreq = self._get_machine_requirements() - sp.set_machine_requirements(mreq) - modules_dirs = lnst_config.get_option('environment', 'module_dirs') tools_dirs = lnst_config.get_option('environment', 'tool_dirs')
@@ -103,6 +99,9 @@ class NetTestController: self._resource_table["module"] = self._load_test_modules(modules_dirs) self._resource_table["tools"] = self._load_test_tools(tools_dirs)
+ def _get_run_mode(self): + return self.run_mode + def _get_machineinfo(self, machine_id): try: info = self._recipe["machines"][machine_id]["params"] @@ -118,92 +117,22 @@ class NetTestController: msg = "SSH session terminated with status %s" % status raise NetTestError(msg)
- def _get_machine_requirements(self): - recipe = self._recipe - - # There must be some machines specified in the recipe - if "machines" not in recipe or \ - ("machines" in recipe and len(recipe["machines"]) == 0): - msg = "No hosts specified in the recipe. At least two " \ - "hosts are required to perform a network test." - raise RecipeError(msg, recipe) - - # machine requirements - mreq = {} - for machine in recipe["machines"]: - m_id = machine["id"] - - if m_id in mreq: - msg = "Machine with id='%s' already exists." % m_id - raise RecipeError(msg, machine) - - params = {} - if "params" in machine: - for p in machine["params"]: - if p["name"] in params: - msg = "Parameter '%s' of host %s was specified " \ - "multiple times. Overriding the previous value." \ - % (p["name"], m_id) - logging.warn(RecipeError(msg, p)) - name = p["name"] - val = p["value"] - params[name] = val - - # Each machine must have at least one interface - if "interfaces" not in machine or \ - ("interfaces" in machine and len(machine["interfaces"]) == 0): - msg = "Host '%s' has no interfaces specified." % m_id - raise RecipeError(msg, machine) - - ifaces = {} - for iface in machine["interfaces"]: - if_id = iface["id"] - if if_id in ifaces: - msg = "Interface with id='%s' already exists on host " \ - "'%s'." % (if_id, m_id) - - iface_type = iface["type"] - if iface_type != "eth": - continue - - iface_params = {} - if "params" in iface: - for i in iface["params"]: - if i["name"] in iface_params: - msg = "Parameter '%s' of interface %s of " \ - "host %s was defined multiple times. " \ - "Overriding the previous value." \ - % (i["name"], if_id, m_id) - logging.warn(RecipeError(msg, p)) - name = i["name"] - val = i["value"] - iface_params[name] = val - - ifaces[if_id] = { - "network": iface["network"], - "params": iface_params - } - - mreq[m_id] = {"params": params, "interfaces": ifaces} - - return mreq - def _prepare_network(self, resource_sync=True): - recipe = self._recipe + mreq = Task.get_mreq()
machines = self._machines for m_id in machines.keys(): self._prepare_machine(m_id, resource_sync)
- for machine_xml_data in recipe["machines"]: - m_id = machine_xml_data["id"] + for machine_id, machine_data in mreq.iteritems(): + m_id = machine_id m = machines[m_id] namespaces = set() - for iface_xml_data in machine_xml_data["interfaces"]: - self._prepare_interface(m_id, iface_xml_data) + for if_id, iface_data in machine_data["interfaces"].iteritems(): + self._prepare_interface(m_id, if_id, iface_data)
- if iface_xml_data["netns"] != None: - namespaces.add(iface_xml_data["netns"]) + if iface_data["netns"] != None: + namespaces.add(iface_data["netns"])
if len(namespaces) > 0: m.disable_nm() @@ -225,9 +154,15 @@ class NetTestController:
m.wait_interface_init()
- def provision_machines(self): + + def set_machine_requirements(self): + mreq = Task.get_mreq() sp = self._slave_pool + sp.set_machine_requirements(mreq) + + def provision_machines(self): machines = self._machines + sp = self._slave_pool if not sp.provision_machines(machines): msg = "This setup cannot be provisioned with the current pool." raise NoMatchError(msg) @@ -257,148 +192,51 @@ class NetTestController: machine.set_mac_pool(self._mac_pool) machine.set_network_bridges(self._network_bridges)
- recipe_name = os.path.basename(self._recipe_path) + recipe_name = os.path.basename(self._recipe_path.abs_path()) machine.configure(recipe_name)
- sync_table = {'module': {}, 'tools': {}} - if resource_sync: - for task in self._recipe['tasks']: - res_table = copy.deepcopy(self._resource_table) - if 'module_dir' in task: - modules = self._load_test_modules([task['module_dir']]) - res_table['module'].update(modules) - if 'tools_dir' in task: - tools = self._load_test_tools([task['tools_dir']]) - res_table['tools'].update(tools) - - if 'commands' not in task: - if not self._reduce_sync: - sync_table = res_table - break - else: - continue - for cmd in task['commands']: - if 'host' not in cmd or cmd['host'] != m_id: - continue - if cmd['type'] == 'test': - mod = cmd['module'] - if mod in res_table['module']: - sync_table['module'][mod] = res_table['module'][mod] - # check if test module uses some test tools - mod_path = res_table['module'][mod]["path"] - mod_tools = get_module_tools(mod_path) - for t in mod_tools: - if t in sync_table['tools']: - continue - logging.debug("Adding '%s' tool as "\ - "dependency of %s test module" % (t, mod)) - sync_table['tools'][t] = res_table['tools'][t] - else: - msg = "Module '%s' not found on the controller"\ - % mod - raise RecipeError(msg, cmd) - if cmd['type'] == 'exec' and 'from' in cmd: - tool = cmd['from'] - if tool in res_table['tools']: - sync_table['tools'][tool] = res_table['tools'][tool] - else: - msg = "Tool '%s' not found on the controller" % tool - raise RecipeError(msg, cmd) - machine.sync_resources(sync_table) - - def _prepare_interface(self, m_id, iface_xml_data): + def _prepare_interface(self, m_id, if_id, iface_data): machine = self._machines[m_id] - if_id = iface_xml_data["id"] - if_type = iface_xml_data["type"] - - try: - iface = machine.get_interface(if_id) - except MachineError: - if if_type == 'lo': - iface = machine.new_loopback_interface(if_id) - else: - iface = machine.new_soft_interface(if_id, if_type) - - if "slaves" in iface_xml_data: - for slave in iface_xml_data["slaves"]: - slave_id = slave["id"] - iface.add_slave(machine.get_interface(slave_id)) + #if_type = iface_data["type"]
- # Some soft devices (such as team) use per-slave options - if "options" in slave: - for opt in slave["options"]: - iface.set_slave_option(slave_id, opt["name"], - opt["value"]) + #try: + iface = machine.get_interface(if_id) + #except MachineError: + #if if_type == 'lo': + # iface = machine.new_loopback_interface(if_id) + #else: + # iface = machine.new_soft_interface(if_id, if_type)
- if "addresses" in iface_xml_data: - for addr in iface_xml_data["addresses"]: - iface.add_address(addr) + #if "slaves" in iface_data: + # for slave in iface_data["slaves"]: + # slave_id = slave["id"] + # iface.add_slave(machine.get_interface(slave_id))
- if "options" in iface_xml_data: - for opt in iface_xml_data["options"]: - iface.set_option(opt["name"], opt["value"]) + # # Some soft devices (such as team) use per-slave options + # if "options" in slave: + # for opt in slave["options"]: + # iface.set_slave_option(slave_id, opt["name"], + # opt["value"])
- if "netem" in iface_xml_data: - iface.set_netem(iface_xml_data["netem"].to_dict()) + #if "addresses" in iface_data: + # for addr in iface_data["addresses"]: + # iface.add_address(addr)
- if "ovs_conf" in iface_xml_data: - iface.set_ovs_conf(iface_xml_data["ovs_conf"].to_dict()) + #if "options" in iface_data: + # for opt in iface_data["options"]: + # iface.set_option(opt["name"], opt["value"])
- if iface_xml_data["netns"] != None: - iface.set_netns(iface_xml_data["netns"]) + #if "netem" in iface_data: + # iface.set_netem(iface_data["netem"].to_dict())
- if "peer" in iface_xml_data: - iface.set_peer(iface_xml_data["peer"]) - - def _prepare_tasks(self): - self._tasks = [] - for task_data in self._recipe["tasks"]: - task = {} - task["quit_on_fail"] = False - if "quit_on_fail" in task_data: - task["quit_on_fail"] = bool_it(task_data["quit_on_fail"]) + #if "ovs_conf" in iface_data: + # iface.set_ovs_conf(iface_data["ovs_conf"].to_dict())
- if "module_dir" in task_data: - task["module_dir"] = task_data["module_dir"] + if iface_data["netns"] != None: + iface.set_netns(iface_data["netns"])
- if "tools_dir" in task_data: - task["tools_dir"] = task_data["tools_dir"] - - if "python" in task_data: - root = Path(None, self._recipe_path).get_root() - path = Path(root, task_data["python"]) - - task["python"] = path - if not path.exists(): - msg = "Task file '%s' not found." % path.to_str() - raise RecipeError(msg, task_data) - - self._tasks.append(task) - continue - - task["commands"] = task_data["commands"] - task["skeleton"] = [] - for cmd_data in task["commands"]: - cmd = {"type": cmd_data["type"]} - - if "host" in cmd_data: - cmd["host"] = cmd_data["host"] - if cmd["host"] not in self._machines: - msg = "Invalid host id '%s'." % cmd["host"] - raise RecipeError(msg, cmd_data) - - if cmd["type"] in ["test", "exec"]: - if "bg_id" in cmd_data: - cmd["bg_id"] = cmd_data["bg_id"] - elif cmd["type"] in ["wait", "intr", "kill"]: - cmd["proc_id"] = cmd_data["bg_id"] - - task["skeleton"].append(cmd) - - if self._check_task(task): - raise RecipeError("Incorrect command sequence.", task_data) - - self._tasks.append(task) + #if "peer" in iface_data: + # iface.set_peer(iface_data["peer"])
def _prepare_command(self, cmd_data): cmd = {"type": cmd_data["type"]} @@ -613,36 +451,23 @@ class NetTestController: os.remove("/tmp/.lnst_machine_conf")
def match_setup(self): + self.run_mode = "match_setup" + res = self._run_python_task() return {"passed": True}
def config_only_recipe(self): + self.run_mode = "config_only" try: - self._prepare_network(resource_sync=False) - except (KeyboardInterrupt, Exception) as exc: - msg = "Exception raised during configuration." - logging.error(msg) - self._cleanup_slaves() + res = self._run_recipe() + except Exception as exc: + logging.error("Recipe execution terminated by unexpected exception") raise - - self._save_machine_config() - - self._cleanup_slaves(deconfigure=False) - return {"passed": True} - - def run_recipe(self): - try: - self._prepare_tasks() - self._prepare_network() - except (KeyboardInterrupt, Exception) as exc: - msg = "Exception raised during configuration." - logging.error(msg) + finally: self._cleanup_slaves() - raise
- if self._packet_capture: - self._start_packet_capture() + return res
- err = None + def run_recipe(self): try: res = self._run_recipe() except Exception as exc: @@ -656,46 +481,55 @@ class NetTestController:
return res
+ def prepare_test_env(self): + try: + self.provision_machines() + self.print_match_description() + if self.run_mode == "match_setup": + return True + self._prepare_network() + Task.ctl.init_hosts(self._machines) + return True + except (NoMatchError) as exc: + self._cleanup_slaves() + raise exc + return False + except (KeyboardInterrupt, Exception) as exc: + msg = "Exception raised during configuration." + logging.error(msg) + self._cleanup_slaves() + raise + def _run_recipe(self): overall_res = {"passed": True}
- for task in self._tasks: + try: self._res_serializer.add_task() - try: - res = self._run_task(task) - except CommandException as exc: - logging.debug(exc) - overall_res["passed"] = False - overall_res["err_msg"] = "Command exception raised." - break - - for machine in self._machines.itervalues(): - machine.restore_system_config() - - # task failed, check if we should quit_on_fail - if not res: - overall_res["passed"] = False - overall_res["err_msg"] = "At least one command failed." - if task["quit_on_fail"]: - break + res = self._run_python_task() + except CommandException as exc: + logging.debug(exc) + overall_res["passed"] = False + overall_res["err_msg"] = "Command exception raised." + + for machine in self._machines.itervalues(): + machine.restore_system_config() + + # task failed + if not res: + overall_res["passed"] = False + overall_res["err_msg"] = "At least one command failed."
return overall_res
- def _run_python_task(self, task): + def init_taskapi(self): + Task.ctl = Task.ControllerAPI(self) + + def _run_python_task(self): #backup of resource table res_table_bkp = copy.deepcopy(self._resource_table) - if 'module_dir' in task: - modules = self._load_test_modules([task['module_dir']]) - self._resource_table['module'].update(modules) - if 'tools_dir' in task: - tools = self._load_test_tools([task['tools_dir']]) - self._resource_table['tools'].update(tools) - - # Initialize the API handle - Task.ctl = Task.ControllerAPI(self, self._machines)
cwd = os.getcwd() - task_path = task["python"] + task_path = self._recipe_path name = os.path.basename(task_path.abs_path()).split(".")[0] sys.path.append(os.path.dirname(task_path.resolve())) os.chdir(os.path.dirname(task_path.resolve())) @@ -706,20 +540,7 @@ class NetTestController: #restore resource table self._resource_table = res_table_bkp
- return module.ctl._result - - def _run_task(self, task): - if "python" in task: - return self._run_python_task(task) - - seq_passed = True - for cmd_data in task["commands"]: - cmd = self._prepare_command(cmd_data) - cmd_res = self._run_command(cmd) - if not cmd_res["passed"]: - seq_passed = False - - return seq_passed + return Task.ctl._result
def _run_command(self, command): logging.info("Executing command: [%s]", str_command(command)) @@ -848,12 +669,11 @@ class NetTestController: return packages
def _get_alias(self, alias): - templates = self._parser._template_proc - return templates._find_definition(alias) + if alias in self._defined_aliases: + return self._defined_aliases[alias]
def _get_aliases(self): - templates = self._parser._template_proc - return templates._dump_definitions() + return self._defined_aliases
class MessageDispatcher(ConnectionHandler): def __init__(self, log_ctl):
Changes: * get_alias method now has optional default argument, value is set when alias is not specified in cli args * get_mreq() method returns dict with machine requirements for match * breakpoint() is a new feature for config_only mode, execution is paused until user presses Enter * add_host(), add_interface() new methods for creating mreq dict, also initializes HostAPI and InterfaceAPI methods * match() runs match algorithm and prepares test environment, returns True if task part should be executed * deprecated methods removed * m_id and if_id are automatically generated
Signed-off-by: Jiri Prochazka jprochaz@redhat.com --- lnst/Controller/Task.py | 240 ++++++++++++++++++++++++++---------------------- 1 file changed, 128 insertions(+), 112 deletions(-)
diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py index 5462d44..af9c201 100644 --- a/lnst/Controller/Task.py +++ b/lnst/Controller/Task.py @@ -32,20 +32,105 @@ except: # The handle to be imported from each task ctl = None
+def get_alias(alias, default=None): + return ctl.get_alias(alias, default) + +def get_mreq(): + return ctl.get_mreq() + +def wait(seconds): + return ctl.wait(seconds) + +def get_module(name, options={}): + return ctl.get_module(name, options) + +def breakpoint(): + if ctl.get_run_mode != "config_only": + return + raw_input("Press enter to continue: ") + +def add_host(params=None): + m_id = ctl.gen_m_id() + ctl.mreq[m_id] = {'interfaces' : {}, 'params' : {}} + handle = HostAPI(ctl, m_id) + ctl.add_host(m_id, handle) + return handle + +def match(): + ctl.cleanup_slaves() + + if ctl.first_run: + ctl.first_run = False + ctl.set_machine_requirements() + + if ctl.prepare_test_env(): + if ctl.get_run_mode == "match_setup": + return False + if ctl.packet_capture(): + ctl.start_packet_capture() + return True + else: + if ctl._ctl._multi_match: + if ctl.prepare_test_env(): + if ctl.get_run_mode == "match_setup": + return False + if ctl.packet_capture(): + ctl.start_packet_capture() + return True + else: + return False + else: + return False + + class TaskError(Exception): pass
class ControllerAPI(object): """ An API class representing the controller. """
- def __init__(self, ctl, hosts): + def __init__(self, ctl): self._ctl = ctl + self._run_mode = ctl._get_run_mode() self._result = True + self.first_run = True + self._m_id_seq = 0 + self.mreq = {}
self._perf_repo_api = PerfRepoAPI()
self._hosts = {} + + def get_mreq(self): + return self.mreq + + def cleanup_slaves(self): + self._ctl._cleanup_slaves() + + def get_run_mode(self): + return self._run_mode + + def set_machine_requirements(self): + self._ctl.set_machine_requirements() + + def packet_capture(self): + return self._ctl._packet_capture + + def start_packet_capture(self): + self._ctl._start_packet_capture() + + def prepare_test_env(self): + return self._ctl.prepare_test_env() + + def gen_m_id(self): + self._m_id_seq += 1 + return "m_id_%s" % self._m_id_seq + + def add_host(self, host_id, handle): + self._hosts[host_id] = handle + + def init_hosts(self, hosts): for host_id, host in hosts.iteritems(): - self._hosts[host_id] = HostAPI(self, host_id, host) + self._hosts[host_id].init_host(host)
def _run_command(self, command): """ @@ -58,27 +143,6 @@ class ControllerAPI(object): self._result = self._result and res["passed"] return res
- def get_host(self, host_id): - """ - Get an API handle for the host from the recipe spec with - a specific id. - - :param host_id: id of the host as defined in the recipe - :type host_id: string - - :return: The host handle. - :rtype: HostAPI - - :raises TaskError: If there is no host with such id. - """ - if host_id not in self._hosts: - raise TaskError("Host '%s' not found." % host_id) - - return self._hosts[host_id] - - def get_hosts(self): - return self._hosts - def get_module(self, name, options={}): """ Initialize a module to be run on a host. @@ -104,7 +168,7 @@ class ControllerAPI(object): cmd = {"type": "ctl_wait", "seconds": int(seconds)} return self._ctl._run_command(cmd)
- def get_alias(self, alias): + def get_alias(self, alias, default=None): """ Get the value of user defined alias.
@@ -115,9 +179,13 @@ class ControllerAPI(object): :rtype: string """ try: - return self._ctl._get_alias(alias) + val = self._ctl._get_alias(alias) + if val is None: + return default + else: + return val except XmlTemplateError: - return None + return default
def get_aliases(self): """ @@ -177,21 +245,47 @@ class ControllerAPI(object): class HostAPI(object): """ An API class representing a host machine. """
- def __init__(self, ctl, host_id, host): + def __init__(self, ctl, host_id): self._ctl = ctl self._id = host_id - self._m = host + self._m = None
self._ifaces = {} + self._if_id_seq = 0 + self._bg_id_seq = 0 + + def _gen_if_id(self): + self._if_id_seq += 1 + return "if_id_%s" % self._if_id_seq + + def init_host(self, host): + self._m = host + self.init_ifaces() + + def init_ifaces(self): for interface in self._m.get_interfaces(): if interface.get_id() is None: continue - self._ifaces[interface.get_id()] = InterfaceAPI(interface, self) + self._ifaces[interface.get_id()].init_iface(interface)
- self._bg_id_seq = 0 + def add_interface(self, label, netns=None, params=None): + m_id = self.get_id() + if_id = self._gen_if_id() + + self._ctl.mreq[m_id]['interfaces'][if_id] = {} + self._ctl.mreq[m_id]['interfaces'][if_id]['network'] = label + self._ctl.mreq[m_id]['interfaces'][if_id]['netns'] = netns + + if params: + self._ctl.mreq[m_id]['interfaces'][if_id]['params'] = params + else: + self._ctl.mreq[m_id]['interfaces'][if_id]['params'] = {} + + self._ifaces[if_id] = InterfaceAPI(None, self) + return self._ifaces[if_id]
def get_id(self): - return self._m.get_id() + return self._id
def config(self, option, value, persistent=False, netns=None): """ @@ -287,75 +381,6 @@ class HostAPI(object): cmd_res = self._ctl._run_command(cmd) return ProcessAPI(self._ctl, self._id, cmd_res, bg_id, cmd["netns"])
- def get_interfaces(self): - return self._ifaces - - def get_interface(self, if_id): - return self._ifaces[if_id] - - @deprecated - def get_devname(self, if_id): - """ - Returns devname of the interface. - - :param if_id: which interface - :type if_id: string - - :return: Device name (e.g., eth0). - :rtype: str - """ - iface = self._ifaces[if_id] - return iface.get_devname() - - @deprecated - def get_hwaddr(self, if_id): - """ - Returns hwaddr of the interface. - - :param if_id: which interface - :type if_id: string - - :return: HW address (e.g., 00:11:22:33:44:55:FF). - :rtype: str - """ - iface = self._ifaces[if_id] - return iface.get_hwaddr() - - @deprecated - def get_ip(self, if_id, addr_number=0): - """ - Returns an IP address of the interface. - - :param if_id: which interface - :type if_id: string - - :param addr_number: which address - :type addr_number: int - - :return: IP address (e.g., 192.168.1.10). - :rtype: str - """ - iface = self._ifaces[if_id] - return iface.get_ip_addr(addr_number) - - @deprecated - def get_prefix(self, if_id, addr_number=0): - """ - Returns an IP address prefix (netmask) - of the interface. - - :param if_id: which interface - :type if_id: string - - :param addr_number: which address - :type addr_number: int - - :return: netmask (e.g., 24). - :rtype: str - """ - iface = self._ifaces[if_id] - return iface.get_ip_prefix(addr_number) - def sync_resources(self, modules=[], tools=[]): res_table = self._ctl._ctl._resource_table sync_table = {'module': {}, 'tools': {}} @@ -470,6 +495,9 @@ class InterfaceAPI(object): self._if = interface self._host = host
+ def init_iface(self, interface): + self._if = interface + def get_id(self): return self._if.get_id()
@@ -494,21 +522,9 @@ class InterfaceAPI(object): def get_ips(self): return VolatileValue(self._if.get_addresses)
- @deprecated - def get_ip_addr(self, ip_index=0): - return self.get_ip(ip_index) - - @deprecated - def get_ip_addrs(self): - return self.get_ips() - def get_prefix(self, ip_index=0): return VolatileValue(self._if.get_prefix, ip_index)
- @deprecated - def get_ip_prefix(self, ip_index=0): - return self.get_prefix(ip_index) - def get_mtu(self): return VolatileValue(self._if.get_mtu)
Description: 3_vlans.py: 2 hosts with 1 eth iface 3 VLAN subifaces on both eth ifaces ping from m1_eth.10 to m2_eth.10
example.py: 2 hosts, one with 2 eth ifaces, one with 1 eth iface team created on host ping from m1_team to m2_eth
ping_flood.py: recipe/regression-tests/phase1/ping_flood.* like
Signed-off-by: Jiri Prochazka jprochaz@redhat.com --- pyrecipes/3_vlans.py | 34 ++++++++++++++++++++++++++++++++++ pyrecipes/example.py | 33 +++++++++++++++++++++++++++++++++ pyrecipes/ping_flood.py | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 115 insertions(+) create mode 100644 pyrecipes/3_vlans.py create mode 100644 pyrecipes/example.py create mode 100644 pyrecipes/ping_flood.py
diff --git a/pyrecipes/3_vlans.py b/pyrecipes/3_vlans.py new file mode 100644 index 0000000..067fd33 --- /dev/null +++ b/pyrecipes/3_vlans.py @@ -0,0 +1,34 @@ +import lnst + +def ping_mod_init(if1, if2): + ping_mod = lnst.get_module("IcmpPing", + options={ + "addr": if2.get_ip(0), + "count": 10, + "interval": 1, + "iface" : if1.get_devname()}) + return ping_mod + +m1 = lnst.add_host() +m2 = lnst.add_host() + +m1_eth1 = m1.add_interface(label="tnet") +m2_eth1 = m2.add_interface(label="tnet") + +while lnst.match(): + m1.sync_resources(modules=["IcmpPing"]) + m2.sync_resources(modules=["IcmpPing"]) + + m1_vlan10 = m1.create_vlan(realdev_iface=m1_eth1, vlan_tci="10", ip="192.168.10.1/24") + m1_vlan20 = m1.create_vlan(realdev_iface=m1_eth1, vlan_tci="20", ip="192.168.20.1/24") + m1_vlan30 = m1.create_vlan(realdev_iface=m1_eth1, vlan_tci="30", ip="192.168.30.1/24") + + m2_vlan10 = m2.create_vlan(realdev_iface=m2_eth1, vlan_tci="10", ip="192.168.10.2/24") + m2_vlan20 = m2.create_vlan(realdev_iface=m2_eth1, vlan_tci="20", ip="192.168.20.2/24") + m2_vlan30 = m2.create_vlan(realdev_iface=m2_eth1, vlan_tci="30", ip="192.168.30.2/24") + + ping_mod = ping_mod_init(m1_vlan10, m2_vlan10) + ping_mod_bad = ping_mod_init(m1_vlan10, m2_vlan20) + + m1.run(ping_mod) + #m1.run(ping_mod_bad) diff --git a/pyrecipes/example.py b/pyrecipes/example.py new file mode 100644 index 0000000..32c1b08 --- /dev/null +++ b/pyrecipes/example.py @@ -0,0 +1,33 @@ +import lnst + +# if1 ... src +# if2 ... dst +def ping_mod_init(if1, if2): + ping_mod = lnst.get_module("IcmpPing", + options={ + "addr": if2.get_ip(0), + "count": 10, + "interval": 1, + "iface" : if1.get_devname()}) + return ping_mod + +m1 = lnst.add_host() +m2 = lnst.add_host() + +m1_eth1 = m1.add_interface(label="tnet") +m1_eth2 = m1.add_interface(label="tnet") +m2_eth1 = m2.add_interface(label="tnet") + +while lnst.match(): + m1.sync_resources(modules=["IcmpPing"]) + m2.sync_resources(modules=["IcmpPing"]) + m2_eth1.reset(ip="192.168.0.2/24") + ping = ping_mod_init(m1_eth1, m2_eth1) + #lnst.breakpoint() + team_if = m1.create_team(slaves=[m1_eth1, m1_eth2]) + #lnst.breakpoint() + team_if.reset(ip="192.168.0.1/24") + #lnst.breakpoint() + ping = ping_mod_init(team_if, m2_eth1) + #lnst.breakpoint() + m1.run(ping) diff --git a/pyrecipes/ping_flood.py b/pyrecipes/ping_flood.py new file mode 100644 index 0000000..e65879d --- /dev/null +++ b/pyrecipes/ping_flood.py @@ -0,0 +1,48 @@ +import lnst + +m1 = lnst.add_host() +m2 = lnst.add_host() + +m1_eth1 = m1.add_interface(label="tnet") +m2_eth1 = m2.add_interface(label="tnet") + +while lnst.match(): + m1.sync_resources(modules=["Icmp6Ping", "IcmpPing"]) + m2.sync_resources(modules=["Icmp6Ping", "IcmpPing"]) + + lnst.wait(15) + + ipv = lnst.get_alias("ipv", default="ipv4") + print "ipv" + print ipv + mtu = lnst.get_alias("mtu", default="1500") + print "mtu" + print mtu + + m1_eth1.reset(ip=["192.168.101.10/24", "fc00:0:0:0::1/64"]) + m2_eth1.reset(ip=["192.168.101.11/24", "fc00:0:0:0::2/64"]) + + ping_mod = lnst.get_module("IcmpPing", + options={ + "addr": m2_eth1.get_ip(0), + "count": 10, + "interval": 0.1, + "iface" : m1_eth1.get_devname(), + "limit_rate": 90}) + + ping_mod6 = lnst.get_module("Icmp6Ping", + options={ + "addr": m2_eth1.get_ip(1), + "count": 10, + "interval": 0.1, + "iface" : m1_eth1.get_devname(), + "limit_rate": 90}) + + m1_eth1.set_mtu(mtu) + m2_eth1.set_mtu(mtu) + + if ipv in [ 'ipv6', 'both' ]: + m1.run(ping_mod6) + + if ipv in ['ipv4', 'both' ]: + m1.run(ping_mod)
Wed, Apr 20, 2016 at 02:29:01PM CEST, jprochaz@redhat.com wrote:
Hello everyone,
first working draft implementation is completed. Please, read this cover letter, check the patches and let me know what you like, what you don't like, etc. Please, note, that this is not final implementation.
NOTE - patches are not rebased to current HEAD, if you want to try it, please, reset your branch to commit 2398f93.
What doesn't work
multi_match - didn't have time and resources to check physical machines, on virt setup it works, but ends with exception due to no res value returned loopbacks - can't be created netem support - not implemented yet
Description of new mechanics
Since we need to run TaskAPI methods to get machine requirements from PyRecipe, methods like provision_machines, prepare_network are run after the python recipe is executed. Methods add_host and add_interface create machine requirements dict in ControllerAPI object. When lnst.match() is called, it runs match algorithm with mreq dict and if match is found, it prepares network and binds Machine and Interface objects with their HostAPI and InterfaceAPI counterparts. When everyting is prepared, execution returns to PyRecipe, where task phase of PyRecipe is executed. Task phase is the same like before, only new method is breakpoint() which will be useful for debugging, as it allows user to pause the execution in whatever part of test. TaskAPI is now used right from lnst module, all callable methods are exported via __init__.py file from lnst/ dir. Folder pyrecipes/ contains few working examples you can try.
Patches 3 and 4 are quite hard to review since they are all sqaushed-up. Would it be possible to cut them into multiple patches please?
The example usage looks awesome to me. Nice work!
Thanks!
2016-04-20 23:29 GMT+02:00 Jiri Pirko jiri@resnulli.us:
Patches 3 and 4 are quite hard to review since they are all sqaushed-up. Would it be possible to cut them into multiple patches please?
The example usage looks awesome to me. Nice work!
Thanks!
Hi Jiri, I will send new patches tommorow, sorry for confusion, I tried to deliver the patches as soon as possible.
Thanks for understanding, Jiri
lnst-developers@lists.fedorahosted.org
-
Jiri Pirko -
Jiri Prochazka