LNST-developers September 2013

lnst-developers@lists.fedorahosted.org

6 participants
114 discussions

[PATCH] Fix nested xinclude element bug
by Jan Tluka 17 Sep '13

17 Sep '13

I found a bug where xinclude preprocessing works only for elements that are direct children of lnstrecipe tag. If an xinclude is nested deeper in the structure it won't be resolved. This patch adds one more recursion to process xincludes in deeper levels. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst/Common/XmlParser.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lnst/Common/XmlParser.py b/lnst/Common/XmlParser.py index 143b289..e50a2f2 100644 --- a/lnst/Common/XmlParser.py +++ b/lnst/Common/XmlParser.py @@ -136,6 +136,8 @@ class XmlParser(object): return node parent.replace(e, node) + else: + self._expand_xinclude(e, base_url) return elem def _remove_comments(self, doc): -- 1.8.1.4

1 0

[PATCH 0/3] config_only for virtualized machines
by Ondrej Lichtner 16 Sep '13

16 Sep '13

From: Ondrej Lichtner <olichtne(a)redhat.com> This patchset enables the use of config_only action for virtualized machines. You could still do it previously but it woulnd't work properly as the fynamically created interfaces would get automatically removed. Now the interfaces stay and are deconfigured on the next lnst-ctl recipe run. Information about what to remove is stored in a file /tmp/.lnst_virt_conf using a file in /tmp is ok because rebooting the host machine will have the same effect - everything dynamically created will be removed. The file contains a pickled object describing what was configured so it is not designed to be edited by hand. Ondrej Lichtner (3): Machine: ensure virt device cleanup NetTestController: ensure partial cleanup NetTestController: allow config_only for virtualized machines lnst/Common/VirtUtils.py | 6 +++ lnst/Controller/Machine.py | 8 ++++ lnst/Controller/NetTestController.py | 85 ++++++++++++++++++++++++++++++++++-- 3 files changed, 95 insertions(+), 4 deletions(-) -- 1.8.3.1

1 3

[PATCH] SlavePool: fix copy-paste error
by Ondrej Lichtner 16 Sep '13

16 Sep '13

From: Ondrej Lichtner <olichtne(a)redhat.com> I forgot to change the parameters to a function call that I copied from a different function. This commit fixes that. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> --- lnst/Controller/SlavePool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py index 24724be..d83de40 100644 --- a/lnst/Controller/SlavePool.py +++ b/lnst/Controller/SlavePool.py @@ -725,6 +725,6 @@ class SetupMapper: if self._map_setup_virt(new_template, new_pool): return True - self._machine_map.discard((machine, possible_match)) + self._machine_map.discard((machine_id, pm_id)) return False -- 1.8.3.1

1 0

[PATCH 1/3] lnst-ctl: Rearranging Help
by Radek Pazdera 16 Sep '13

16 Sep '13

The help message was stuffed into a too small space. I made the lines longer and also arranged the options in an alphabetical order. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> --- lnst-ctl | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/lnst-ctl b/lnst-ctl index c01762a..a51b2d6 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -35,20 +35,18 @@ def usage(retval=0): print "ACTION = [ run | config_only | match_setup ]" print "" print "OPTIONS" + print " -a, --define-alias name=value define top-level alias" + print " -A, --override-alias name=value define top-level alias that " \ + "will override any other definitions in the recipe" + print " -c, --config=FILE load additional config file" print " -d, --debug emit debugging messages" - print " -p, --packet-capture capture and log all ongoing\n" \ - " network communication during\n" \ - " the test" print " -h, --help print this message" print " -m, --no-colours disable coloured terminal output" - print " -o, --disable-pool-checks don't check the availability of\n"\ - " machines in the pool" - print " -c, --config=FILE load additional config file\n" + print " -o, --disable-pool-checks don't check the availability of " \ + "machines in the pool" + print " -p, --packet-capture capture and log all ongoing " \ + "network communication during the test" print " -x, --result=FILE file to write xml_result" - print " -a, --define-alias name=value define top-level alias" - print " -A, --override-alias name=value define top-level alias that\n" \ - " will override any other\n"\ - " definitions in the recipe" sys.exit(retval) def store_alias(alias_def, aliases_dict): -- 1.8.3.1

1 2

[lnst] lnst-ctl: Define/override aliases from CLI
by Jiří Pírko 13 Sep '13

13 Sep '13

commit 325d10dc8c4b53b5b7a3b7f47bc659cfdea69ab8 Author: Radek Pazdera <rpazdera(a)redhat.com> Date: Fri Sep 13 14:39:13 2013 +0200 lnst-ctl: Define/override aliases from CLI This commit adds two CLI arguments to lnst-ctl: -a, --define-alias -A, --override-alias Using these you can define or override an alias in the recipe you're just executing. The difference between these two is in their effect. Defied aliases (-a) can be overriden by the aliases that are defined in the recie. The override aliases will replace all definitions of the alias with the name in question within the entire recipe file. Example: lnst-ctl -A http_port=80 recipe.xml run Issue #1. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> lnst-ctl | 50 +++++++++++++++++++++++++++------ lnst/Common/XmlParser.py | 3 ++ lnst/Common/XmlTemplates.py | 13 ++++++++- lnst/Controller/NetTestController.py | 4 ++- 4 files changed, 59 insertions(+), 11 deletions(-) --- diff --git a/lnst-ctl b/lnst-ctl index e71ba83..c01762a 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -32,7 +32,7 @@ def usage(retval=0): """ print "Usage: %s [OPTION...] [RECIPE...] ACTION" % sys.argv[0] print "" - print "ACTION = [run | config_only | match_setup]" + print "ACTION = [ run | config_only | match_setup ]" print "" print "OPTIONS" print " -d, --debug emit debugging messages" @@ -45,13 +45,34 @@ def usage(retval=0): " machines in the pool" print " -c, --config=FILE load additional config file\n" print " -x, --result=FILE file to write xml_result" + print " -a, --define-alias name=value define top-level alias" + print " -A, --override-alias name=value define top-level alias that\n" \ + " will override any other\n"\ + " definitions in the recipe" sys.exit(retval) +def store_alias(alias_def, aliases_dict): + try: + name, value = alias_def.split("=") + except: + msg = "The alias definition '%s' not supported. The proper" + \ + " format is alias_name=alias_value." + raise Exception(msg) + + if name in aliases_dict: + msg = "The same alias %s was defined multiple times through CLI." + logging.warning(msg) + + aliases_dict[name] = value + def process_recipe(action, file_path, res_serializer, - packet_capture, log_ctl, pool_checks): + packet_capture, log_ctl, pool_checks, + defined_aliases, overriden_aliases): nettestctl = NetTestController(file_path, log_ctl, res_serializer=res_serializer, - pool_checks=pool_checks) + pool_checks=pool_checks, + defined_aliases=defined_aliases, + overriden_aliases=overriden_aliases) if action == "run": return nettestctl.run_recipe(packet_capture) elif action == "config_only": @@ -60,7 +81,7 @@ def process_recipe(action, file_path, res_serializer, return nettestctl.match_setup() def get_recipe_result(args, file_path, res_serializer, packet_capture, - log_ctl, pool_checks): + log_ctl, pool_checks, defined_aliases, overriden_aliases): res_serializer.add_recipe(file_path) log_ctl.set_recipe(file_path) @@ -69,7 +90,8 @@ def get_recipe_result(args, file_path, res_serializer, packet_capture, res = {} try: res = process_recipe(args, file_path, res_serializer, - packet_capture, log_ctl, pool_checks) + packet_capture, log_ctl, pool_checks, + defined_aliases, overriden_aliases) except Exception as err: log_exc_traceback() logging.error(err) @@ -92,9 +114,10 @@ def main(): try: opts, args = getopt.getopt( sys.argv[1:], - "dhr:c:x:pom", + "dhr:c:x:poma:A:", ["debug", "help", "recipe=", "config", "result=", - "packet-capture", "disable-pool-checks", "no-colours"] + "packet-capture", "disable-pool-checks", "no-colours", + "define_alias", "override_alias"] ) except getopt.GetoptError as err: print str(err) @@ -124,6 +147,8 @@ def main(): packet_capture = False pool_checks = True coloured_output = True + defined_aliases = {} + overriden_aliases = {} for opt, arg in opts: if opt in ("-d", "--debug"): debug += 1 @@ -139,6 +164,10 @@ def main(): pool_checks = False elif opt in ("-m", "--no-colours"): coloured_output = False + elif opt in ("-a", "--define-alias"): + store_alias(arg, defined_aliases) + elif opt in ("-A", "--override-alias"): + store_alias(arg, overriden_aliases) if config_path is not None: if not os.path.isfile(config_path): @@ -187,12 +216,15 @@ def main(): fp, res, rv = get_recipe_result(action, recipe_file, res_serializer, packet_capture, - log_ctl, pool_checks) + log_ctl, pool_checks, + defined_aliases, + overriden_aliases) else: fp, res, rv = get_recipe_result(action, recipe_path, res_serializer, packet_capture, - log_ctl, pool_checks) + log_ctl, pool_checks, + defined_aliases, overriden_aliases) summary.append((fp, res)) if rv > retval: diff --git a/lnst/Common/XmlParser.py b/lnst/Common/XmlParser.py index c404c0b..143b289 100644 --- a/lnst/Common/XmlParser.py +++ b/lnst/Common/XmlParser.py @@ -102,6 +102,9 @@ class XmlParser(object): def set_machines(self, machines): self._template_proc.set_machines(machines) + def set_aliases(self, defined, overriden): + self._template_proc.set_aliases(defined, overriden) + def _has_attribute(self, element, attr): return attr in element.attrib diff --git a/lnst/Common/XmlTemplates.py b/lnst/Common/XmlTemplates.py index 41dbc0d..a2afd05 100644 --- a/lnst/Common/XmlTemplates.py +++ b/lnst/Common/XmlTemplates.py @@ -250,6 +250,14 @@ class XmlTemplates: """ self._machines = machines + def set_aliases(self, defined, overriden): + """ Set aliases defined or overriden from CLI """ + + for name, value in defined.iteritems(): + self.define_alias(name, value) + + self._overriden_aliases = overriden + def define_alias(self, name, value, skip_reserved_check=False): """ Associate an alias name with some value @@ -283,11 +291,14 @@ class XmlTemplates: self._definitions.pop() def _find_definition(self, name): + if name in self._overriden_aliases: + return self._overriden_aliases[name] + for level in reversed(self._definitions): if name in level: return level[name] - err = "'%s' is not defined here" % name + err = "Alias '%s' is not defined here" % name raise XmlTemplateError(err) def process_aliases(self, element): diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 3379aef..910b40e 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -45,7 +45,8 @@ def ignore_event(**kwarg): class NetTestController: def __init__(self, recipe_path, log_ctl, - res_serializer=None, pool_checks=True): + res_serializer=None, pool_checks=True, + defined_aliases=None, overriden_aliases=None): self._res_serializer = res_serializer self._remote_capture_files = {} self._log_ctl = log_ctl @@ -65,6 +66,7 @@ class NetTestController: parser = RecipeParser(recipe_path) parser.set_machines(self._machines) + parser.set_aliases(defined_aliases, overriden_aliases) self._recipe = parser.parse() modules_dirs = lnst_config.get_option('environment', 'module_dirs')

1 0

[PATCH] lnst-ctl: Define/override aliases from CLI
by Radek Pazdera 13 Sep '13

13 Sep '13

This commit adds two CLI arguments to lnst-ctl: -a, --define-alias -A, --override-alias Using these you can define or override an alias in the recipe you're just executing. The difference between these two is in their effect. Defied aliases (-a) can be overriden by the aliases that are defined in the recie. The override aliases will replace all definitions of the alias with the name in question within the entire recipe file. Example: lnst-ctl -A http_port=80 recipe.xml run Issue #1. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> --- lnst-ctl | 50 +++++++++++++++++++++++++++++------- lnst/Common/XmlParser.py | 3 +++ lnst/Common/XmlTemplates.py | 13 +++++++++- lnst/Controller/NetTestController.py | 4 ++- 4 files changed, 59 insertions(+), 11 deletions(-) diff --git a/lnst-ctl b/lnst-ctl index e71ba83..c01762a 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -32,7 +32,7 @@ def usage(retval=0): """ print "Usage: %s [OPTION...] [RECIPE...] ACTION" % sys.argv[0] print "" - print "ACTION = [run | config_only | match_setup]" + print "ACTION = [ run | config_only | match_setup ]" print "" print "OPTIONS" print " -d, --debug emit debugging messages" @@ -45,13 +45,34 @@ def usage(retval=0): " machines in the pool" print " -c, --config=FILE load additional config file\n" print " -x, --result=FILE file to write xml_result" + print " -a, --define-alias name=value define top-level alias" + print " -A, --override-alias name=value define top-level alias that\n" \ + " will override any other\n"\ + " definitions in the recipe" sys.exit(retval) +def store_alias(alias_def, aliases_dict): + try: + name, value = alias_def.split("=") + except: + msg = "The alias definition '%s' not supported. The proper" + \ + " format is alias_name=alias_value." + raise Exception(msg) + + if name in aliases_dict: + msg = "The same alias %s was defined multiple times through CLI." + logging.warning(msg) + + aliases_dict[name] = value + def process_recipe(action, file_path, res_serializer, - packet_capture, log_ctl, pool_checks): + packet_capture, log_ctl, pool_checks, + defined_aliases, overriden_aliases): nettestctl = NetTestController(file_path, log_ctl, res_serializer=res_serializer, - pool_checks=pool_checks) + pool_checks=pool_checks, + defined_aliases=defined_aliases, + overriden_aliases=overriden_aliases) if action == "run": return nettestctl.run_recipe(packet_capture) elif action == "config_only": @@ -60,7 +81,7 @@ def process_recipe(action, file_path, res_serializer, return nettestctl.match_setup() def get_recipe_result(args, file_path, res_serializer, packet_capture, - log_ctl, pool_checks): + log_ctl, pool_checks, defined_aliases, overriden_aliases): res_serializer.add_recipe(file_path) log_ctl.set_recipe(file_path) @@ -69,7 +90,8 @@ def get_recipe_result(args, file_path, res_serializer, packet_capture, res = {} try: res = process_recipe(args, file_path, res_serializer, - packet_capture, log_ctl, pool_checks) + packet_capture, log_ctl, pool_checks, + defined_aliases, overriden_aliases) except Exception as err: log_exc_traceback() logging.error(err) @@ -92,9 +114,10 @@ def main(): try: opts, args = getopt.getopt( sys.argv[1:], - "dhr:c:x:pom", + "dhr:c:x:poma:A:", ["debug", "help", "recipe=", "config", "result=", - "packet-capture", "disable-pool-checks", "no-colours"] + "packet-capture", "disable-pool-checks", "no-colours", + "define_alias", "override_alias"] ) except getopt.GetoptError as err: print str(err) @@ -124,6 +147,8 @@ def main(): packet_capture = False pool_checks = True coloured_output = True + defined_aliases = {} + overriden_aliases = {} for opt, arg in opts: if opt in ("-d", "--debug"): debug += 1 @@ -139,6 +164,10 @@ def main(): pool_checks = False elif opt in ("-m", "--no-colours"): coloured_output = False + elif opt in ("-a", "--define-alias"): + store_alias(arg, defined_aliases) + elif opt in ("-A", "--override-alias"): + store_alias(arg, overriden_aliases) if config_path is not None: if not os.path.isfile(config_path): @@ -187,12 +216,15 @@ def main(): fp, res, rv = get_recipe_result(action, recipe_file, res_serializer, packet_capture, - log_ctl, pool_checks) + log_ctl, pool_checks, + defined_aliases, + overriden_aliases) else: fp, res, rv = get_recipe_result(action, recipe_path, res_serializer, packet_capture, - log_ctl, pool_checks) + log_ctl, pool_checks, + defined_aliases, overriden_aliases) summary.append((fp, res)) if rv > retval: diff --git a/lnst/Common/XmlParser.py b/lnst/Common/XmlParser.py index c404c0b..143b289 100644 --- a/lnst/Common/XmlParser.py +++ b/lnst/Common/XmlParser.py @@ -102,6 +102,9 @@ class XmlParser(object): def set_machines(self, machines): self._template_proc.set_machines(machines) + def set_aliases(self, defined, overriden): + self._template_proc.set_aliases(defined, overriden) + def _has_attribute(self, element, attr): return attr in element.attrib diff --git a/lnst/Common/XmlTemplates.py b/lnst/Common/XmlTemplates.py index 41dbc0d..a2afd05 100644 --- a/lnst/Common/XmlTemplates.py +++ b/lnst/Common/XmlTemplates.py @@ -250,6 +250,14 @@ class XmlTemplates: """ self._machines = machines + def set_aliases(self, defined, overriden): + """ Set aliases defined or overriden from CLI """ + + for name, value in defined.iteritems(): + self.define_alias(name, value) + + self._overriden_aliases = overriden + def define_alias(self, name, value, skip_reserved_check=False): """ Associate an alias name with some value @@ -283,11 +291,14 @@ class XmlTemplates: self._definitions.pop() def _find_definition(self, name): + if name in self._overriden_aliases: + return self._overriden_aliases[name] + for level in reversed(self._definitions): if name in level: return level[name] - err = "'%s' is not defined here" % name + err = "Alias '%s' is not defined here" % name raise XmlTemplateError(err) def process_aliases(self, element): diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 3379aef..910b40e 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -45,7 +45,8 @@ def ignore_event(**kwarg): class NetTestController: def __init__(self, recipe_path, log_ctl, - res_serializer=None, pool_checks=True): + res_serializer=None, pool_checks=True, + defined_aliases=None, overriden_aliases=None): self._res_serializer = res_serializer self._remote_capture_files = {} self._log_ctl = log_ctl @@ -65,6 +66,7 @@ class NetTestController: parser = RecipeParser(recipe_path) parser.set_machines(self._machines) + parser.set_aliases(defined_aliases, overriden_aliases) self._recipe = parser.parse() modules_dirs = lnst_config.get_option('environment', 'module_dirs') -- 1.8.3.1

1 0

[PATCH 1/2] lnst-ctl: change meaning of -c option
by Ondrej Lichtner 12 Sep '13

12 Sep '13

From: Ondrej Lichtner <olichtne(a)redhat.com> This commit changes the meaning of the -c option. Previously this option was used to perform config cleanup on machines before configuring them. Since slaves are now capable of storing this configuration and perform cleanup independently it is no longer needed. The -c option is now used to specify an additional config file that should be loaded after config files in standard places have been loaded. This is a feature we agreed would be usefull for running automatic tests. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> --- lnst-ctl | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/lnst-ctl b/lnst-ctl index ccfa684..e71ba83 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -43,14 +43,13 @@ def usage(retval=0): print " -m, --no-colours disable coloured terminal output" print " -o, --disable-pool-checks don't check the availability of\n"\ " machines in the pool" - print " -c, --cleanup perform config cleanup\n" \ - " machines" + print " -c, --config=FILE load additional config file\n" print " -x, --result=FILE file to write xml_result" sys.exit(retval) -def process_recipe(action, file_path, cleanup, res_serializer, +def process_recipe(action, file_path, res_serializer, packet_capture, log_ctl, pool_checks): - nettestctl = NetTestController(file_path, log_ctl, cleanup=cleanup, + nettestctl = NetTestController(file_path, log_ctl, res_serializer=res_serializer, pool_checks=pool_checks) if action == "run": @@ -60,7 +59,7 @@ def process_recipe(action, file_path, cleanup, res_serializer, elif action == "match_setup": return nettestctl.match_setup() -def get_recipe_result(args, file_path, cleanup, res_serializer, packet_capture, +def get_recipe_result(args, file_path, res_serializer, packet_capture, log_ctl, pool_checks): res_serializer.add_recipe(file_path) log_ctl.set_recipe(file_path) @@ -69,7 +68,7 @@ def get_recipe_result(args, file_path, cleanup, res_serializer, packet_capture, res = {} try: - res = process_recipe(args, file_path, cleanup, res_serializer, + res = process_recipe(args, file_path, res_serializer, packet_capture, log_ctl, pool_checks) except Exception as err: log_exc_traceback() @@ -93,8 +92,8 @@ def main(): try: opts, args = getopt.getopt( sys.argv[1:], - "dhr:cx:pom", - ["debug", "help", "recipe=", "cleanup", "result=", + "dhr:c:x:pom", + ["debug", "help", "recipe=", "config", "result=", "packet-capture", "disable-pool-checks", "no-colours"] ) except getopt.GetoptError as err: @@ -120,18 +119,18 @@ def main(): debug = 0 recipe_path = None - cleanup = False result_path = None + config_path = None packet_capture = False pool_checks = True - coloured_output = not lnst_config.get_option("colours", "disable_colours") + coloured_output = True for opt, arg in opts: if opt in ("-d", "--debug"): debug += 1 elif opt in ("-h", "--help"): usage(RETVAL_PASS) - elif opt in ("-c", "--cleanup"): - cleanup = True + elif opt in ("-c", "--config"): + config_path = arg elif opt in ("-x", "--result"): result_path = arg elif opt in ("-p", "--packet-capture"): @@ -141,6 +140,18 @@ def main(): elif opt in ("-m", "--no-colours"): coloured_output = False + if config_path is not None: + if not os.path.isfile(config_path): + print "File '%s' doesn't exist!" % config_path + usage(RETVAL_ERR) + else: + print "Loading config file '%s'" % config_path + lnst_config.load_config(config_path) + + if coloured_output: + coloured_output = not lnst_config.get_option("colours", + "disable_colours") + load_presets_from_config(lnst_config) date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S") @@ -174,12 +185,12 @@ def main(): if re.match(r'^.*\.xml$', recipe_file): logging.info("Processing recipe file \"%s\"" % recipe_file) fp, res, rv = get_recipe_result(action, recipe_file, - cleanup, res_serializer, + res_serializer, packet_capture, log_ctl, pool_checks) else: fp, res, rv = get_recipe_result(action, recipe_path, - cleanup, res_serializer, + res_serializer, packet_capture, log_ctl, pool_checks) -- 1.8.3.1

2 3

[lnst] NetTestSlave: cleanup when hello called
by Jiří Pírko 12 Sep '13

12 Sep '13

commit 79e16eb057652439471b515111cfe1f9dacee4ce Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Thu Sep 12 14:25:42 2013 +0200 NetTestSlave: cleanup when hello called This commit moves the call of slave cleanup from the server loop to the rpc method hello() as it makes more sense to do the cleanup whenever a new session is started by the controller. The rest of the patch changes related code on the controller side, this is also related to the previous commit that removed the --cleanup option and replaced it with --config. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> lnst/Controller/Machine.py | 5 +---- lnst/Controller/NetTestController.py | 5 ++--- lnst/Slave/NetTestSlave.py | 12 +++++------- 3 files changed, 8 insertions(+), 14 deletions(-) --- diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py index 96f3657..6a870ef 100644 --- a/lnst/Controller/Machine.py +++ b/lnst/Controller/Machine.py @@ -113,7 +113,7 @@ class Machine(object): return result - def configure(self, recipe_name, do_cleanup=False): + def configure(self, recipe_name): """ Prepare the machine Calling this method will initialize the rpc connection to the @@ -135,9 +135,6 @@ class Machine(object): "to machine %s, handshake failed!" % hostname raise Machine(msg) - if do_cleanup: - self._rpc_call("machine_cleanup") - for iface in self._interfaces: iface.initialize() diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 5404284..3379aef 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -44,9 +44,8 @@ def ignore_event(**kwarg): pass class NetTestController: - def __init__(self, recipe_path, log_ctl, cleanup=False, + def __init__(self, recipe_path, log_ctl, res_serializer=None, pool_checks=True): - self._docleanup = cleanup self._res_serializer = res_serializer self._remote_capture_files = {} self._log_ctl = log_ctl @@ -204,7 +203,7 @@ class NetTestController: machine.set_network_bridges(self._network_bridges) recipe_name = os.path.basename(self._recipe_path) - machine.configure(recipe_name, self._docleanup) + machine.configure(recipe_name) machine.sync_resources(self._resource_table) def _prepare_interface(self, m_id, iface_xml_data): diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index 94a92c0..cbb3958 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -83,11 +83,15 @@ class SlaveMethods: self.ctl_clean_exit = True def hello(self, recipe_path): + if not self.ctl_clean_exit: + self._methods.machine_cleanup() + self._methods.ctl_clean_exit = True + logging.info("Recieved a controller connection.") self.clear_resource_table() self._cache.del_old_entries() self.reset_file_transfers() - self.ctl_clean_exit = False + self._ctl_clean_exit = False date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S") self._log_ctl.set_recipe(recipe_path, expand=date) @@ -451,8 +455,6 @@ class NetTestSlave: self._log_ctl = log_ctl - self._start = True - self._nl_socket = NetlinkSocket(family=NETLINK_ROUTE) self._nl_socket.bind(RTNL_GROUPS) self._server_handler.add_connection('netlink', self._nl_socket) @@ -461,13 +463,9 @@ class NetTestSlave: while not self._finished: if self._server_handler.get_ctl_sock() == None: self._log_ctl.cancel_connection() - if not self._start and not self._methods.ctl_clean_exit: - self._methods.machine_cleanup() - self._methods.ctl_clean_exit = True try: logging.info("Waiting for connection.") self._server_handler.accept_connection() - self._start = False except socket.error: continue self._log_ctl.set_connection(

1 0

[lnst] lnst-ctl: change meaning of -c option
by Jiří Pírko 12 Sep '13

12 Sep '13

commit c6b1c76c7e7e9b802633a97ec90a8d33dd132fda Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Thu Sep 12 14:25:41 2013 +0200 lnst-ctl: change meaning of -c option This commit changes the meaning of the -c option. Previously this option was used to perform config cleanup on machines before configuring them. Since slaves are now capable of storing this configuration and perform cleanup independently it is no longer needed. The -c option is now used to specify an additional config file that should be loaded after config files in standard places have been loaded. This is a feature we agreed would be usefull for running automatic tests. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> lnst-ctl | 39 +++++++++++++++++++++++++-------------- 1 files changed, 25 insertions(+), 14 deletions(-) --- diff --git a/lnst-ctl b/lnst-ctl index ccfa684..e71ba83 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -43,14 +43,13 @@ def usage(retval=0): print " -m, --no-colours disable coloured terminal output" print " -o, --disable-pool-checks don't check the availability of\n"\ " machines in the pool" - print " -c, --cleanup perform config cleanup\n" \ - " machines" + print " -c, --config=FILE load additional config file\n" print " -x, --result=FILE file to write xml_result" sys.exit(retval) -def process_recipe(action, file_path, cleanup, res_serializer, +def process_recipe(action, file_path, res_serializer, packet_capture, log_ctl, pool_checks): - nettestctl = NetTestController(file_path, log_ctl, cleanup=cleanup, + nettestctl = NetTestController(file_path, log_ctl, res_serializer=res_serializer, pool_checks=pool_checks) if action == "run": @@ -60,7 +59,7 @@ def process_recipe(action, file_path, cleanup, res_serializer, elif action == "match_setup": return nettestctl.match_setup() -def get_recipe_result(args, file_path, cleanup, res_serializer, packet_capture, +def get_recipe_result(args, file_path, res_serializer, packet_capture, log_ctl, pool_checks): res_serializer.add_recipe(file_path) log_ctl.set_recipe(file_path) @@ -69,7 +68,7 @@ def get_recipe_result(args, file_path, cleanup, res_serializer, packet_capture, res = {} try: - res = process_recipe(args, file_path, cleanup, res_serializer, + res = process_recipe(args, file_path, res_serializer, packet_capture, log_ctl, pool_checks) except Exception as err: log_exc_traceback() @@ -93,8 +92,8 @@ def main(): try: opts, args = getopt.getopt( sys.argv[1:], - "dhr:cx:pom", - ["debug", "help", "recipe=", "cleanup", "result=", + "dhr:c:x:pom", + ["debug", "help", "recipe=", "config", "result=", "packet-capture", "disable-pool-checks", "no-colours"] ) except getopt.GetoptError as err: @@ -120,18 +119,18 @@ def main(): debug = 0 recipe_path = None - cleanup = False result_path = None + config_path = None packet_capture = False pool_checks = True - coloured_output = not lnst_config.get_option("colours", "disable_colours") + coloured_output = True for opt, arg in opts: if opt in ("-d", "--debug"): debug += 1 elif opt in ("-h", "--help"): usage(RETVAL_PASS) - elif opt in ("-c", "--cleanup"): - cleanup = True + elif opt in ("-c", "--config"): + config_path = arg elif opt in ("-x", "--result"): result_path = arg elif opt in ("-p", "--packet-capture"): @@ -141,6 +140,18 @@ def main(): elif opt in ("-m", "--no-colours"): coloured_output = False + if config_path is not None: + if not os.path.isfile(config_path): + print "File '%s' doesn't exist!" % config_path + usage(RETVAL_ERR) + else: + print "Loading config file '%s'" % config_path + lnst_config.load_config(config_path) + + if coloured_output: + coloured_output = not lnst_config.get_option("colours", + "disable_colours") + load_presets_from_config(lnst_config) date = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M:%S") @@ -174,12 +185,12 @@ def main(): if re.match(r'^.*\.xml$', recipe_file): logging.info("Processing recipe file \"%s\"" % recipe_file) fp, res, rv = get_recipe_result(action, recipe_file, - cleanup, res_serializer, + res_serializer, packet_capture, log_ctl, pool_checks) else: fp, res, rv = get_recipe_result(action, recipe_path, - cleanup, res_serializer, + res_serializer, packet_capture, log_ctl, pool_checks)

1 0

[lnst] XmlParser: Improved XInclude Error Reporting
by Jiří Pírko 12 Sep '13

12 Sep '13

commit 05f9135f63fa736ceeb23480028f959b2f8881a7 Author: Radek Pazdera <rpazdera(a)redhat.com> Date: Thu Sep 12 14:03:30 2013 +0200 XmlParser: Improved XInclude Error Reporting Due to the way XInclude is implemented in lxml, we are unable to get the origin of each tag in the document. This patch provides our own simplified implementation of XInclude resolving that fixes this problem. Signed-off-by: Radek Pazdera <rpazdera(a)redhat.com> Signed-off-by: Jiri Pirko <jiri(a)resnulli.us> lnst/Common/XmlParser.py | 115 +++++++++++++++++++++++++++++++++------ lnst/Common/XmlProcessing.py | 36 +++++++++--- lnst/Controller/RecipeParser.py | 7 --- 3 files changed, 126 insertions(+), 32 deletions(-) --- diff --git a/lnst/Common/XmlParser.py b/lnst/Common/XmlParser.py index 68d3c76..c404c0b 100644 --- a/lnst/Common/XmlParser.py +++ b/lnst/Common/XmlParser.py @@ -11,13 +11,17 @@ rpazdera(a)redhat.com (Radek Pazdera) """ import os +import re import sys import logging +import copy from lxml import etree from lnst.Common.XmlTemplates import XmlTemplates, XmlTemplateError from lnst.Common.XmlProcessing import XmlProcessingError, XmlData class XmlParser(object): + XINCLUDE_RE = r"\{http\:\/\/www\.w3\.org\/[0-9]{4}\/XInclude\}include" + def __init__(self, schema_file, xml_path): # locate the schema file # try git path @@ -39,9 +43,43 @@ class XmlParser(object): self._schema = etree.RelaxNG(relaxng_doc) def parse(self): + doc = self._parse(self._path) + self._remove_comments(doc) + + # Due to a weird implementation of XInclude in lxml, the + # XmlParser resolves included documents on it's own. + # + # To be able to tell later on where each tag was located + # in the XML document, we add a '__file' attribute to + # each element of the tree during the parsing. + # + # However, these special attributes are of course not + # valid according to our schemas. To solve this, a copy of + # the tree is made and the '__file' attributes are removed + # before validation. + # + # XXX This is a *EXTREMELY* dirty hack. Ideas/proposals + # for cleaner solutions are more than welcome! + root_tag = self._init_loc(doc.getroot(), self._path) + self._expand_xinclude(root_tag, os.path.dirname(self._path)) + + self._template_proc.process_aliases(root_tag) + try: - doc = etree.parse(self._path) - doc.xinclude() + self._validate(doc) + except: + err = self._schema.error_log[0] + loc = {"file": os.path.basename(err.filename), + "line": err.line, "col": err.column} + exc = XmlProcessingError(err.message) + exc.set_loc(loc) + raise exc + + return self._process(root_tag) + + def _parse(self, path): + try: + doc = etree.parse(path) except Exception as err: # A workaround for cases when lxml (quite strangely) # sets the filename to <string>. @@ -56,20 +94,7 @@ class XmlParser(object): exc.set_loc(loc) raise exc - root_tag = doc.getroot() - self._template_proc.process_aliases(root_tag) - - try: - self._schema.assertValid(doc) - except: - err = self._schema.error_log[0] - loc = {"file": os.path.basename(err.filename), - "line": err.line, "col": err.column} - exc = XmlProcessingError(err.message) - exc.set_loc(loc) - raise exc - - return self._process(root_tag) + return doc def _process(self, root_tag): pass @@ -86,3 +111,61 @@ class XmlParser(object): def _get_content(self, element): text = etree.tostring(element, method="text") return self._template_proc.expand_functions(text) + + def _expand_xinclude(self, elem, base_url=""): + for e in elem: + if re.match(self.XINCLUDE_RE, str(e.tag)): + href = os.path.join(base_url, e.get("href")) + filename = os.path.basename(href) + + doc = self._parse(href) + self._remove_comments(doc) + node = doc.getroot() + + node = self._init_loc(node, href) + + if e.tail: + node.tail = (node.tail or "") + e.tail + self._expand_xinclude(node, os.path.dirname(href)) + + parent = e.getparent() + if parent is None: + return node + + parent.replace(e, node) + return elem + + def _remove_comments(self, doc): + comments = doc.xpath('//comment()') + for c in comments: + p = c.getparent() + if p is not None: + p.remove(c) + + def _init_loc(self, elem, filename): + """ Remove all coment tags from the tree """ + + elem.attrib["__file"] = filename + for e in elem: + self._init_loc(e, os.path.basename(filename)) + + return elem + + def _validate(self, original): + """ + Make a copy of the tree, remove the '__file' attributes + and validate against the appropriate schema. + + Very unfortunate solution. + """ + doc = copy.deepcopy(original) + root = doc.getroot() + + self._prepare_tree_for_validation(root) + self._schema.assertValid(doc) + + def _prepare_tree_for_validation(self, elem): + if "__file" in elem.attrib: + del elem.attrib["__file"] + for e in elem: + self._prepare_tree_for_validation(e) diff --git a/lnst/Common/XmlProcessing.py b/lnst/Common/XmlProcessing.py index 5ea297d..771ead5 100644 --- a/lnst/Common/XmlProcessing.py +++ b/lnst/Common/XmlProcessing.py @@ -24,15 +24,21 @@ class XmlProcessingError(Exception): super(XmlProcessingError, self).__init__() self._msg = msg - if obj is not None and hasattr(obj, "loc"): - self.set_loc(obj.loc) - - loc = {} - if obj is not None and hasattr(obj, "base") and obj.base != None: - loc["file"] = os.path.basename(obj.base) - if hasattr(obj, "sourceline"): - loc["line"] = obj.sourceline - self.set_loc(loc) + if obj is not None: + if hasattr(obj, "loc"): + self.set_loc(obj.loc) + elif hasattr(obj, "attrib") and "__file" in obj.attrib: + loc = {} + loc["file"] = obj.attrib["__file"] + if hasattr(obj, "sourceline"): + loc["line"] = obj.sourceline + self.set_loc(loc) + elif hasattr(obj, "base") and obj.base != None: + loc = {} + loc["file"] = os.path.basename(obj.base) + if hasattr(obj, "sourceline"): + loc["line"] = obj.sourceline + self.set_loc(loc) def set_loc(self, loc): @@ -89,6 +95,12 @@ class XmlCollection(list): if node is not None: if hasattr(node, "loc"): self.loc = node.loc + elif "__file" in node.attrib: + loc = {} + loc["file"] = node.attrib["__file"] + if hasattr(node, "sourceline"): + loc["line"] = node.sourceline + self.loc = loc elif hasattr(node, "base") and node.base != None: loc = {} loc["file"] = os.path.basename(node.base) @@ -113,6 +125,12 @@ class XmlData(dict): if node is not None: if hasattr(node, "loc"): self.loc = node.loc + elif "__file" in node.attrib: + loc = {} + loc["file"] = node.attrib["__file"] + if hasattr(node, "sourceline"): + loc["line"] = node.sourceline + self.loc = loc elif hasattr(node, "base") and node.base != None: loc = {} loc["file"] = os.path.basename(node.base) diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py index 2bd6c38..2720359 100644 --- a/lnst/Controller/RecipeParser.py +++ b/lnst/Controller/RecipeParser.py @@ -34,13 +34,6 @@ class RecipeParser(XmlParser): def _process(self, lnst_recipe): recipe = XmlData(lnst_recipe) - # remove all comments from the document - comments = lnst_recipe.xpath('//comment()') - for c in comments: - p = c.getparent() - if p is not None: - p.remove(c) - # machines machines_tag = lnst_recipe.find("machines") if machines_tag is not None:

1 0

← Newer
1
...
4
5
6
7
8
9
10
11
12
Older →

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers September 2013