LNST-developers May 2019

lnst-developers@lists.fedorahosted.org

20 participants
34 discussions

[PATCH-next 00/15] log improvements
by olichtne＠redhat.com 11 Jun '19

11 Jun '19

From: Ondrej Lichtner <olichtne(a)redhat.com> Hi all, this patch series starts my effort to improve the logs as provided to the tester. This patchset is centered at removing the need of the tester to use debug=1 logging and still know what's happening wrt. the recipe he's trying to run. This means a significant reduction of logs on the Controller side which in my personal opinion improves usability quite a bit. Next I want to focus on improving the BaseEnrtRecipe hierarchy and include capabilities that provide some additional context as to what stage the test execution is at. -Ondrej Ondrej Lichtner (15): lnst.Common.ExecCmd: fix cmd not being part of the exception description add __repr__ and __str__ implementations of several objects lnst.Controller.RecipeResults: remove useless property redefines lnst.Controller.RecipeResults: add success setter lnst.Controller.Machine: ensure Job{Start, Finish}Result ordering lnst.Devices.RemoteDevice: add _id property lnst.Controller.RecipeResults: add DeviceConfigResult.* classes lnst.Controller.Machine: add stop_recipe lnst.Controller.RunSummaryFormatter: change default source string lnst.Controller.Job: add a 'type' property lnst.Controller.Job: __str__ method format of _what description lnst.Controller.Machine: add device manipulation methods lnst.RecipeCommon.Perf: add flow information to string reports adjust logging levels lnst.Controller.Recipe: log results as they're added lnst/Common/ExecCmd.py | 3 +- lnst/Common/IpAddress.py | 3 + lnst/Controller/Controller.py | 1 + lnst/Controller/Host.py | 1 + lnst/Controller/Job.py | 35 ++--- lnst/Controller/Machine.py | 74 ++++++++-- lnst/Controller/Namespace.py | 10 +- lnst/Controller/Recipe.py | 15 ++ lnst/Controller/RecipeResults.py | 134 ++++++++++++++++-- lnst/Controller/RunSummaryFormatter.py | 6 +- lnst/Devices/RemoteDevice.py | 41 ++++-- .../BaselineFlowAverageEvaluator.py | 3 +- .../Perf/Evaluators/NonzeroFlowEvaluator.py | 5 +- .../Perf/Measurements/BaseFlowMeasurement.py | 28 ++++ .../Perf/Measurements/BaseMeasurement.py | 6 + .../Perf/Measurements/StatCPUMeasurement.py | 9 +- lnst/Recipes/ENRT/BaseEnrtRecipe.py | 22 ++- lnst/Slave/Job.py | 2 +- lnst/Tests/BaseTestModule.py | 12 ++ 19 files changed, 348 insertions(+), 62 deletions(-) -- 2.21.0

3 22

[PATCH-next 1/3] lnst.Common.ExecCmd: decode std{out, err} immediately
by olichtne＠redhat.com 05 Jun '19

05 Jun '19

From: Ondrej Lichtner <olichtne(a)redhat.com> Instead of returning bytes, just decode the data immediately so that it can be worked with as before - strings. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> --- lnst/Common/ExecCmd.py | 6 ++++-- lnst/Common/GitVersion.py | 2 +- lnst/Common/Version.py | 2 +- lnst/Devices/Device.py | 7 +++---- lnst/Slave/InterfaceManager.py | 1 - 5 files changed, 9 insertions(+), 9 deletions(-) diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py index dff5b16..007acbd 100644 --- a/lnst/Common/ExecCmd.py +++ b/lnst/Common/ExecCmd.py @@ -58,6 +58,8 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False, json=F subp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True) (data_stdout, data_stderr) = subp.communicate() + data_stdout = data_stdout.decode() + data_stderr = data_stderr.decode() ''' When we should not die on error, do not print anything and let @@ -65,9 +67,9 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False, json=F ''' if log_outputs: if data_stdout: - log_output(logging.debug, "Stdout", data_stdout.decode()) + log_output(logging.debug, "Stdout", data_stdout) if data_stderr: - log_output(logging.debug, "Stderr", data_stderr.decode()) + log_output(logging.debug, "Stderr", data_stderr) if subp.returncode and die_on_err: err = ExecCmdFail(cmd, subp.returncode, [data_stdout, data_stderr], report_stderr) logging.error(err) diff --git a/lnst/Common/GitVersion.py b/lnst/Common/GitVersion.py index fe088a2..6de3c1e 100644 --- a/lnst/Common/GitVersion.py +++ b/lnst/Common/GitVersion.py @@ -30,6 +30,6 @@ def git_version(): cmd = ['git', 'rev-parse', 'HEAD'] try: proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=null) - return proc.communicate()[0].strip() + return proc.communicate()[0].decode().strip() finally: os.chdir(cwd) diff --git a/lnst/Common/Version.py b/lnst/Common/Version.py index b6a2bc6..c7bacfd 100644 --- a/lnst/Common/Version.py +++ b/lnst/Common/Version.py @@ -21,7 +21,7 @@ def __init__(self): self._version = "14" if git_version: - self._version += "-" + git_version().decode() + self._version += "-" + git_version() @property def version(self): diff --git a/lnst/Devices/Device.py b/lnst/Devices/Device.py index 66ed7fc..30deb25 100644 --- a/lnst/Devices/Device.py +++ b/lnst/Devices/Device.py @@ -232,18 +232,18 @@ def _vars(self): def _clear_tc_qdisc(self): exec_cmd("tc qdisc del dev %s root" % self.name, die_on_err=False) out, _ = exec_cmd("tc filter show dev %s" % self.name) - ingress_handles = re.findall("ingress (\\d+):", out.decode()) + ingress_handles = re.findall("ingress (\\d+):", out) for ingress_handle in ingress_handles: exec_cmd("tc qdisc del dev %s handle %s: ingress" % (self.name, ingress_handle)) out, _ = exec_cmd("tc qdisc show dev %s" % self.name) - ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out.decode()) + ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out) if len(ingress_qdiscs) != 0: exec_cmd("tc qdisc del dev %s ingress" % self.name) def _clear_tc_filters(self): out, _ = exec_cmd("tc filter show dev %s" % self.name) - egress_prefs = re.findall("pref (\\d+) .* handle", out.decode()) + egress_prefs = re.findall("pref (\\d+) .* handle", out) for egress_pref in egress_prefs: exec_cmd("tc filter del dev %s pref %s" % (self.name, egress_pref)) @@ -631,7 +631,6 @@ def autoneg_off(self): def _read_adaptive_coalescing(self): try: res = exec_cmd("ethtool -c %s" % self.name) - res = res.decode() except: logging.debug("Could not read coalescence values of %s." % self.name) return (None, None) diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py index 7411a73..62f9222 100644 --- a/lnst/Slave/InterfaceManager.py +++ b/lnst/Slave/InterfaceManager.py @@ -261,7 +261,6 @@ def _is_name_used(self, name): out, _ = exec_cmd("ovs-vsctl --columns=name list Interface", log_outputs=False, die_on_err=False) - out = out.decode() for line in out.split("\n"): m = re.match(r'.*: \"(.*)\"', line) if m is not None: -- 2.21.0

3 4

[PATCH-next 0/3] Handle DeviceNotFound errors
by csfakian＠redhat.com 30 May '19

30 May '19

From: Christos Sfakianakis <csfakian(a)redhat.com> Hi, this patch handles DeviceNotFound errors, noticed paritcularly with VirtualBridgeVlanInHostRecipe, VirtualBridgeVlanInHostMirroredRecipe, and VirtualBridgeVlansOverBondRecipe tests. When attempting to (directly or indirectly) retrieve information for a device which has been deleted, such errors are thrown. Christos Christos Sfakianakis (3): lnst.Slave.InterfaceManager: add handler for DeviceDeleted and RTM_NEWLINK lnst.Devices.Device: add handlers in _get_if_data lnst.Controller.Machine: add handler in device_created() lnst/Controller/Machine.py | 10 +++++++++- lnst/Devices/Device.py | 31 ++++++++++++++++++++----------- lnst/Slave/InterfaceManager.py | 5 ++++- 3 files changed, 33 insertions(+), 13 deletions(-) -- 2.17.1

2 4

ANNOUNCEMENT next branch is Python3 now
by Ondrej Lichtner 30 May '19

30 May '19

Hi all, just sending an additional email about this to make it a little bit more visible. I just pushed a patch from Jozef Urbanovsky that ports the entirety of the next branch to Python3. There will be no Python2 version of this branch as we've temporarily done for the master branch... Since this was a large patch that change quite a lot of stuff it's expected that there might be some lingering issues that didn't show up during testing. Hopefully we'll resolve those early in the coming weeks. -Ondrej

1 0

[PATCH-next] Port to Python3
by jurbanov＠redhat.com 30 May '19

30 May '19

From: Jozef Urbanovsky <jurbanov(a)redhat.com> Main changes: ================================================ - Print function - PEP 3105[1] - Design of dictionaries, their iteration and respective built-in functions - PEP 469[2], PEP 3106[3] - Division operator - PEP 238[4] - Interrupts of system calls - PEP 475[5] - Socket.accept and interrupts - Docs Python3 Socket[6] Subprocess[7] - Binary data and strings - Docs Python3 porting[8] - Shebangs modified to environmental variable of python3 - Fixed library imports to reflect their respective changes in transition to python3 - multiprocessing - xmlrpc - urllib - thread - pickle lnst/Common/Daemon.py ================================================ - use open instead of deprecated file method lnst/Common/SecureSocket.py ================================================ - Byte string and unicode/ascii string conflict, fixed by using bytes internally and sending messages encoded to ascii - Fixed multiple issues regarding changes to standard python functions, such as encoding to hex, ordinal value of characters, length of different types lnst/Controller/Machine.py ================================================ - module "__builtin__" was renamed in python3 to "builtins" lnst/Controller/Recipe.py ================================================ - valid results are now concatenated using all method instead of deprecated reduce along with list comprehension lnst/Controller/VirtDomainCtl.py ================================================ - HWAddress class implements "__eq__" method, which automatically makes instaces of this class unhashable, which means they can't be stored in dictionaries. As instances of this class might be mutable, it is recommended to leave them unhashable. To ensure storability of instances of this class in dictionary, method "__str__" is used to convert given object to string, which is hashable. lnst/RecipeCommon/Perf/Measurements.py - Fixed issues where some conditions could try to compare NoneType with integer value, resulting into wrong evaluations lnst/Slave/NetTestSlave.py ================================================ - signum 2 (SIGINT) - modified behavior since python 3.5 - Due to changes in interrupting system call, signal receiving and handling required changes to main slave loop - Structure of raising, catching and handling interrupt was changed, to reflect behavior of python 3.5 and higher - Method run is no longer ended by parameter "finished", but rather exception breaks the endless loop lnst/Tests/Netperf.py ================================================ - result["confidence"] was truncating everything after decimal point, therefore division was incorrect for numbers with non-zero decimal part. ================================================ [1] - PEP 3105 https://www.python.org/dev/peps/pep-3105/ [2] - PEP 469 https://www.python.org/dev/peps/pep-0469/ [3] - PEP 3106 https://www.python.org/dev/peps/pep-3106/ [4] - PEP 238 https://legacy.python.org/dev/peps/pep-0238/ [5] - PEP 475 https://www.python.org/dev/peps/pep-0475/ [6] - Docs Python3 Socket https://docs.python.org/3/library/socket.html#socket.socket.accept [7] - Docs Python3 Subprocess https://docs.python.org/3/library/subprocess.html [8] - Docs Python3 Porting https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data Signed-off-by: Jozef Urbanovsky <jurbanov(a)redhat.com> --- lnst-ctl | 90 +++++++++---------- lnst-pool-wizard | 6 +- lnst-slave | 20 ++--- lnst/Common/Colours.py | 12 +-- lnst/Common/Config.py | 8 +- lnst/Common/ConnectionHandler.py | 2 +- lnst/Common/Daemon.py | 10 +-- lnst/Common/ExecCmd.py | 4 +- lnst/Common/LoggingHandler.py | 4 +- lnst/Common/Logs.py | 6 +- lnst/Common/NetTestCommand.py | 18 ++-- lnst/Common/NetUtils.py | 2 +- lnst/Common/Parameters.py | 6 +- lnst/Common/Path.py | 5 +- lnst/Common/ProcessManager.py | 14 +-- lnst/Common/ResourceCache.py | 2 +- lnst/Common/SecureSocket.py | 71 ++++++++------- lnst/Common/ShellProcess.py | 2 +- lnst/Common/Utils.py | 4 +- lnst/Common/Version.py | 2 +- lnst/Controller/Controller.py | 10 +-- lnst/Controller/Host.py | 2 +- lnst/Controller/Machine.py | 10 +-- lnst/Controller/MachineMapper.py | 22 ++--- lnst/Controller/MessageDispatcher.py | 16 ++-- lnst/Controller/Namespace.py | 6 +- lnst/Controller/NetTestResultSerializer.py | 8 +- lnst/Controller/Recipe.py | 4 +- lnst/Controller/Requirements.py | 2 +- lnst/Controller/RunSummaryFormatter.py | 2 +- lnst/Controller/SlavePoolManager.py | 18 ++-- lnst/Controller/Task.py | 18 ++-- lnst/Controller/VirtDomainCtl.py | 6 +- lnst/Controller/Wizard.py | 32 +++---- lnst/Devices/BondDevice.py | 14 +-- lnst/Devices/Device.py | 12 +-- .../Perf/Measurements/BaseCPUMeasurement.py | 2 +- .../Perf/Measurements/IperfFlowMeasurement.py | 10 ++- .../Perf/Measurements/StatCPUMeasurement.py | 10 +-- .../Perf/Measurements/TRexFlowMeasurement.py | 8 +- lnst/RecipeCommon/Perf/Recipe.py | 4 +- lnst/RecipeCommon/PerfRepo/PerfRepo.py | 1 - lnst/RecipeCommon/TestRecipe.py | 2 +- lnst/Recipes/ENRT/BaseEnrtRecipe.py | 8 +- lnst/Recipes/ENRT/PingFloodRecipe.py | 2 +- lnst/Slave/InterfaceManager.py | 17 ++-- lnst/Slave/NetConfigDevice.py | 14 +-- lnst/Slave/NetTestSlave.py | 64 +++++++------ lnst/Tests/BaseTestModule.py | 2 +- lnst/Tests/CPUStatMonitor.py | 2 +- lnst/Tests/Iperf.py | 4 +- lnst/Tests/Netperf.py | 2 +- lnst/Tests/Ping.py | 2 + misc/recipe_conv.py | 8 +- pyrecipes/ping_flood.py | 8 +- recipes/examples/python_recipe.py | 6 +- .../examples/python_recipe_simple_netperf.py | 2 +- recipes/smoke/generate-recipes.py | 48 +++++----- setup.py | 4 +- test_modules/Netperf.py | 2 +- 60 files changed, 361 insertions(+), 341 deletions(-) diff --git a/lnst-ctl b/lnst-ctl index 29d133c..f1c443b 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test controller @@ -34,42 +34,42 @@ def usage(retval=0): """ Print usage of this app """ - print "Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0] - print "" - print "ACTION = [ run | config_only | deconfigure | match_setup | " \ - "list_pools]" - print "" - print "OPTIONS" - print " -A, --override-alias name=value define top-level alias that " \ - "will override any other definitions in the recipe" - print " -a, --define-alias name=value define top-level alias" - print " -b, --breakpoints enable breakpoint feature" - print " -c, --config=FILE load additional config file" - print " -C, --config-override=FILE reset config defaults and load " \ - "the following config file" - print " -d, --debug emit debugging messages" - print " --dump-config dumps the join of all loaded " \ - "configuration files on stdout and exits" - print " -v, --verbose verbose version of list_pools " \ - "command" - print " -h, --help print this message" - print " -m, --no-colours disable coloured terminal output" - print " -o, --disable-pool-checks don't check the availability of " \ - "machines in the pool" - print " -p, --packet-capture capture and log all ongoing " \ - "network communication during the test" - print " --pools=NAME[,...] restricts which pools to use "\ - "for matching, value can be a comma separated list of values or" - print " --pools=PATH a single path to a pool directory" - print " -r, --reduce-sync reduces resource synchronization "\ - "for python tasks, see documentation" - print " -s, --xslt-url=URL URL to a XSLT document that will "\ + print("Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0]) + print("") + print("ACTION = [ run | config_only | deconfigure | match_setup | " \ + "list_pools]") + print("") + print("OPTIONS") + print(" -A, --override-alias name=value define top-level alias that " \ + "will override any other definitions in the recipe") + print(" -a, --define-alias name=value define top-level alias") + print(" -b, --breakpoints enable breakpoint feature") + print(" -c, --config=FILE load additional config file") + print(" -C, --config-override=FILE reset config defaults and load " \ + "the following config file") + print(" -d, --debug emit debugging messages") + print(" --dump-config dumps the join of all loaded " \ + "configuration files on stdout and exits") + print(" -v, --verbose verbose version of list_pools " \ + "command") + print(" -h, --help print this message") + print(" -m, --no-colours disable coloured terminal output") + print(" -o, --disable-pool-checks don't check the availability of " \ + "machines in the pool") + print(" -p, --packet-capture capture and log all ongoing " \ + "network communication during the test") + print(" --pools=NAME[,...] restricts which pools to use "\ + "for matching, value can be a comma separated list of values or") + print(" --pools=PATH a single path to a pool directory") + print(" -r, --reduce-sync reduces resource synchronization "\ + "for python tasks, see documentation") + print(" -s, --xslt-url=URL URL to a XSLT document that will "\ "be used when transforming the xml result file, only useful "\ - "when -t is used as well" - print " -t, --html=FILE generate a formatted result html" - print " -u, --multi-match run each recipe with every "\ - "pool match possible" - print " -x, --result=FILE file to write xml_result" + "when -t is used as well") + print(" -t, --html=FILE generate a formatted result html") + print(" -u, --multi-match run each recipe with every "\ + "pool match possible") + print(" -x, --result=FILE file to write xml_result") sys.exit(retval) def list_pools(restrict_pools, verbose): @@ -91,12 +91,12 @@ def list_pools(restrict_pools, verbose): out = "" # iterate over all pools sp_pools = sp.get_pools() - for pool_name, pool_content in sp_pools.iteritems(): + for pool_name, pool_content in sp_pools.items(): out += "Pool: %s (%s)\n" % (pool_name, pools[pool_name]) # verbose output if verbose: # iterate over all slave machine cfgs - for filename, pool in pool_content.iteritems(): + for filename, pool in pool_content.items(): # print in human-readable format out += 3*" " + filename + ".xml\n" out += 5*" " + "params:\n" @@ -114,9 +114,9 @@ def list_pools(restrict_pools, verbose): out += "\n" # print wihout newlines on the end of string if verbose: - print out[:-2] + print(out[:-2]) else: - print out[:-1] + print(out[:-1]) def store_alias(alias_def, aliases_dict): @@ -267,7 +267,7 @@ def main(): ] ) except getopt.GetoptError as err: - print str(err) + print(str(err)) usage(RETVAL_ERR) lnst_config.controller_init() @@ -315,16 +315,16 @@ def main(): breakpoints = True elif opt in ("-c", "--config"): if not os.path.isfile(arg): - print "File '%s' doesn't exist!" % arg + print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else: lnst_config.load_config(arg) elif opt in ("-C", "--config-override"): if not os.path.isfile(arg): - print "File '%s' doesn't exist!" % arg + print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else: - print >> sys.stderr, "Reloading config defaults!" + print("Reloading config defaults!", file=sys.stderr) lnst_config.controller_init() lnst_config.load_config(arg) elif opt in ("-x", "--result"): @@ -358,7 +358,7 @@ def main(): lnst_config.set_option("environment", "xslt_url", xslt_url) if dump_config: - print lnst_config.dump_config() + print(lnst_config.dump_config()) return RETVAL_PASS if coloured_output: diff --git a/lnst-pool-wizard b/lnst-pool-wizard index ea2849b..f10635e 100755 --- a/lnst-pool-wizard +++ b/lnst-pool-wizard @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Machine pool wizard @@ -19,7 +19,7 @@ RETVAL_ERR = 1 def help(retval=0): - print "Usage:\n"\ + print("Usage:\n"\ " lnst-pool-wizard [mode] [hostname[:port]]\n"\ "\n"\ "Modes:\n"\ @@ -34,7 +34,7 @@ def help(retval=0): " lnst-pool-wizard hostname1:1234 hostname2\n"\ " lnst-pool-wizard --noninteractive 192.168.122.2\n"\ " lnst-pool-wizard -n 192.168.122.2:8888 192.168.122.4\n"\ - " lnst-pool-wizard -p \".pool/\" -n 192.168.1.1:8877 192.168.122.4" + " lnst-pool-wizard -p \".pool/\" -n 192.168.1.1:8877 192.168.122.4") sys.exit(retval) diff --git a/lnst-slave b/lnst-slave index 20696ec..ca8563c 100755 --- a/lnst-slave +++ b/lnst-slave @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test slave @@ -25,14 +25,14 @@ def usage(): """ Print usage of this app """ - print "Usage: %s [OPTION...]" % sys.argv[0] - print "" - print " -d, --debug emit debugging messages" - print " -h, --help print this message" - print " -e, --daemonize go to background after init" - print " -i, --pidfile file to write daemonized process pid" - print " -m, --no-colours disable coloured terminal output" - print " -p, --port xmlrpc port to listen on" + print("Usage: %s [OPTION...]" % sys.argv[0]) + print("") + print(" -d, --debug emit debugging messages") + print(" -h, --help print this message") + print(" -e, --daemonize go to background after init") + print(" -i, --pidfile file to write daemonized process pid") + print(" -m, --no-colours disable coloured terminal output") + print(" -p, --port xmlrpc port to listen on") sys.exit() def main(): @@ -46,7 +46,7 @@ def main(): ["debug", "help", "daemonize", "pidfile=", "port=", "no-colours"] )[0] except getopt.GetoptError as err: - print str(err) + print(str(err)) usage() sys.exit() diff --git a/lnst/Common/Colours.py b/lnst/Common/Colours.py index e57794e..a00e2d2 100644 --- a/lnst/Common/Colours.py +++ b/lnst/Common/Colours.py @@ -52,7 +52,7 @@ def name_to_fg_colour(name): """ Convert name to foreground colour code. Returns None if the colour name isn't supported. """ - if not COLOURS.has_key(name): + if name not in COLOURS: return None return COLOURS[name] @@ -61,7 +61,7 @@ def name_to_bg_colour(name): """ Convert name to background color code. Returns None if the colour name isn't supported. """ - if not COLOURS.has_key(name): + if name not in COLOURS: return None return COLOURS[name] + 10 @@ -136,7 +136,7 @@ def decorate_string(string, fg_colour=None, bg_colour=None, bold=False): raise Exception(msg) else: # Standard definition - if colour_def in COLOURS.keys(): + if colour_def in list(COLOURS.keys()): if fg: colour = name_to_fg_colour(colour_def) else: @@ -165,7 +165,7 @@ def strip_colours(text): def get_preset_conf(preset_name): preset = PRESETS[preset_name] - return map(lambda s: "default" if s == None else str(s), preset) + return ["default" if s == None else str(s) for s in preset] def load_presets_from_config(lnst_config): for preset_name in PRESETS: @@ -178,12 +178,12 @@ def load_presets_from_config(lnst_config): if fg == "default": fg = None - elif not re.match(extended_re, fg) and fg not in COLOURS.keys(): + elif not re.match(extended_re, fg) and fg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % fg) if bg == "default": bg = None - elif not re.match(extended_re, bg) and bg not in COLOURS.keys(): + elif not re.match(extended_re, bg) and bg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % bg) PRESETS[preset_name] = [fg, bg, bool_it(bf)] diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py index c2044f7..7b26d9c 100644 --- a/lnst/Common/Config.py +++ b/lnst/Common/Config.py @@ -62,7 +62,7 @@ class Config(): raise ConfigError(msg) res = {} - for opt_name, opt in self._options[section].items(): + for opt_name, opt in list(self._options[section].items()): res[opt_name] = opt["value"] return res @@ -124,7 +124,7 @@ class Config(): '''Parse and load the config file''' exp_path = os.path.expanduser(path) abs_path = os.path.abspath(exp_path) - print >> sys.stderr, "Loading config file '%s'" % abs_path + print("Loading config file '%s'" % abs_path, file=sys.stderr) sections = self._parse_file(abs_path) self.handleSections(sections, abs_path) @@ -179,7 +179,7 @@ class Config(): def get_pools(self): pools = {} - for pool_name, pool in self._options["pools"].items(): + for pool_name, pool in list(self._options["pools"].items()): pools[pool_name] = pool["value"] return pools @@ -190,7 +190,7 @@ class Config(): return None def _find_option_by_name(self, section, opt_name): - for option in section.itervalues(): + for option in section.values(): if option["name"] == opt_name: return option return None diff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py index 8e1f856..91a94aa 100644 --- a/lnst/Common/ConnectionHandler.py +++ b/lnst/Common/ConnectionHandler.py @@ -13,7 +13,7 @@ olichtne(a)redhat.com (Ondrej Lichtner) import select import socket -from _multiprocessing import Connection +from multiprocessing.connection import Connection from pyroute2 import IPRSocket from lnst.Common.SecureSocket import SecureSocket, SecSocketException diff --git a/lnst/Common/Daemon.py b/lnst/Common/Daemon.py index 6fbf950..23bb77e 100644 --- a/lnst/Common/Daemon.py +++ b/lnst/Common/Daemon.py @@ -25,7 +25,7 @@ class Daemon: def _read_pid(self): try: - handle = file(self._pidfile, "r") + handle = open(self._pidfile, "r") pid = int(handle.read().strip()) handle.close() except IOError: @@ -33,7 +33,7 @@ class Daemon: return pid def _write_pid(self, pid): - handle = file(self._pidfile, "w") + handle = open(self._pidfile, "w") handle.write("%s\n" % str(pid)) handle.close() self._pid_written = True @@ -86,9 +86,9 @@ class Daemon: sys.stdout.flush() sys.stderr.flush() - si = file("/dev/null", 'r') - so = file("/dev/null", 'a+') - se = file("/dev/null", 'a+', 0) + si = open("/dev/null", 'r') + so = open("/dev/null", 'a+') + se = open("/dev/null", 'a+', 0) os.dup2(si.fileno(), sys.stdin.fileno()) os.dup2(so.fileno(), sys.stdout.fileno()) os.dup2(se.fileno(), sys.stderr.fileno()) diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py index 6489766..dff5b16 100644 --- a/lnst/Common/ExecCmd.py +++ b/lnst/Common/ExecCmd.py @@ -65,9 +65,9 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False, json=F ''' if log_outputs: if data_stdout: - log_output(logging.debug, "Stdout", data_stdout) + log_output(logging.debug, "Stdout", data_stdout.decode()) if data_stderr: - log_output(logging.debug, "Stderr", data_stderr) + log_output(logging.debug, "Stderr", data_stderr.decode()) if subp.returncode and die_on_err: err = ExecCmdFail(cmd, subp.returncode, [data_stdout, data_stderr], report_stderr) logging.error(err) diff --git a/lnst/Common/LoggingHandler.py b/lnst/Common/LoggingHandler.py index e2a5dff..b86ff35 100644 --- a/lnst/Common/LoggingHandler.py +++ b/lnst/Common/LoggingHandler.py @@ -16,7 +16,7 @@ olichtne(a)redhat.com (Ondrej Lichtner) import pickle import logging -import xmlrpclib +import xmlrpc.client from lnst.Common.ConnectionHandler import send_data class LogBuffer(logging.Handler): @@ -39,7 +39,7 @@ class LogBuffer(logging.Handler): d['args'] = None d['exc_info'] = None s = pickle.dumps(d, 1) - return xmlrpclib.Binary(s) + return xmlrpc.client.Binary(s) def add_buffer(self, buf): for i in buf: diff --git a/lnst/Common/Logs.py b/lnst/Common/Logs.py index 06c8217..8184f51 100644 --- a/lnst/Common/Logs.py +++ b/lnst/Common/Logs.py @@ -106,7 +106,7 @@ class LoggingCtl: logger = logging.getLogger('') for i in list(logger.handlers): logger.removeHandler(i) - for key, logger in logging.Logger.manager.loggerDict.iteritems(): + for key, logger in list(logging.Logger.manager.loggerDict.items()): if type(logger) != type(logging.Logger): continue for i in list(logger.handlers): @@ -218,7 +218,7 @@ class LoggingCtl: logger = logging.getLogger() logger.addHandler(self.transmit_handler) - for k in self.slaves.keys(): + for k in list(self.slaves.keys()): self.remove_slave(k) def cancel_connection(self): @@ -230,7 +230,7 @@ class LoggingCtl: def disable_logging(self): self.cancel_connection() - for s in self.slaves.keys(): + for s in list(self.slaves.keys()): self.remove_slave(s) self.unset_recipe() diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index ad4fe8f..2a39611 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -326,10 +326,10 @@ class NetTestCommandGeneric(object): formatted_data = "" if res_data: max_key_len = 0 - for key in res_data.keys(): + for key in list(res_data.keys()): if len(key) > max_key_len: max_key_len = len(key) - for key, value in res_data.iteritems(): + for key, value in list(res_data.items()): if type(value) == dict: formatted_data += level*4*" " + str(key) + ":\n" formatted_data += self.format_res_data(value, level+1) @@ -348,13 +348,13 @@ class NetTestCommandGeneric(object): return formatted_data def _check_res_data(self, res_data): - name_start_char = u":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF"\ - u"\u0370-\u037D\u037F-\u1FFF\u200C-\u200D"\ - u"\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF"\ - u"\uF900-\uFDCF\uFDF0-\uFFFD\U00010000-\U000EFFFF" - name_char = name_start_char + u"\-\.0-9\xB7\u0300-\u036F\u203F-\u2040" - name = u"[%s]([%s])*$" % (name_start_char, name_char) - char_data = u"[^<&]*" + name_start_char = ":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\\u02FF"\ + "\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D"\ + "\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF"\ + "\\uF900-\\uFDCF\\uFDF0-\\uFFFD\\U00010000-\\U000EFFFF" + name_char = name_start_char + "\-\.0-9\xB7\\u0300-\\u036F\\u203F-\\u2040" + name = "[%s]([%s])*$" % (name_start_char, name_char) + char_data = "[^<&]*" if isinstance(res_data, dict): for key in res_data: if not re.match(name, key, re.UNICODE): diff --git a/lnst/Common/NetUtils.py b/lnst/Common/NetUtils.py index 8fb177c..5c55263 100644 --- a/lnst/Common/NetUtils.py +++ b/lnst/Common/NetUtils.py @@ -131,7 +131,7 @@ class MacPool(AddressPool): return bs def _byte_string_to_addr(self, byte_string): - return ':'.join(map(lambda x: "%02x" % x, byte_string)) + return ':'.join(["%02x" % x for x in byte_string]) class IpPool(AddressPool): diff --git a/lnst/Common/Parameters.py b/lnst/Common/Parameters.py index b139d6c..b0a84e0 100644 --- a/lnst/Common/Parameters.py +++ b/lnst/Common/Parameters.py @@ -164,18 +164,18 @@ class Parameters(object): return name in self._attrs def __iter__(self): - for attr, val in self._attrs.items(): + for attr, val in list(self._attrs.items()): yield (attr, val) def _to_dict(self): return copy.deepcopy(self._attrs) def _from_dict(self, d): - for name, val in d.items(): + for name, val in list(d.items()): setattr(self, name, copy.deepcopy(val)) def __str__(self): result = "" - for attr, val in self._attrs.items(): + for attr, val in list(self._attrs.items()): result += "%s = %s\n" % (attr, str(val)) return result diff --git a/lnst/Common/Path.py b/lnst/Common/Path.py index ebbbcf8..5fb56ca 100644 --- a/lnst/Common/Path.py +++ b/lnst/Common/Path.py @@ -11,8 +11,9 @@ jtluka(a)redhat.com (Jan Tluka) """ import os -from urlparse import urljoin -from urllib2 import urlopen, HTTPError +from urllib.parse import urljoin +from urllib.request import urlopen +from urllib.error import HTTPError from tempfile import NamedTemporaryFile def get_path_class(root, path): diff --git a/lnst/Common/ProcessManager.py b/lnst/Common/ProcessManager.py index 8c5f2e7..4cd0fdf 100644 --- a/lnst/Common/ProcessManager.py +++ b/lnst/Common/ProcessManager.py @@ -9,18 +9,18 @@ published by the Free Software Foundation; see COPYING for details. __author__ = """ jzupka(a)redhat.com (Jiri Zupka) """ -import os, signal, thread, logging +import os, signal, _thread, logging class ProcessManager: class SubProcess: def __init__(self, pid, handler): self.pid = pid - self.lock = thread.allocate_lock() + self.lock = _thread.allocate_lock() self.lock.acquire() self.handler = handler self.enabled = True self.status = None - thread.start_new_thread(self.waitpid, (self.pid, self.lock, + _thread.start_new_thread(self.waitpid, (self.pid, self.lock, self.handler)) def isAlive(self): @@ -45,12 +45,12 @@ class ProcessManager: logging.error(''.join(traceback.format_exception(type, value, tb))) os.kill(os.getpid(), signal.SIGTERM) else: - print "Process pid %s exit with exitcode %s" % (pid, status) + print("Process pid %s exit with exitcode %s" % (pid, status)) ProcessManager.lock.release() - thread.exit() + _thread.exit() pids = {} - lock = thread.allocate_lock() + lock = _thread.allocate_lock() std_waitpid = None @classmethod @@ -86,7 +86,7 @@ class ProcessManager: return pid, status -lock = thread.allocate_lock() +lock = _thread.allocate_lock() lock.acquire() if os.waitpid != ProcessManager.waitpid: ProcessManager.std_waitpid = os.waitpid diff --git a/lnst/Common/ResourceCache.py b/lnst/Common/ResourceCache.py index 697f22b..43bd3a3 100644 --- a/lnst/Common/ResourceCache.py +++ b/lnst/Common/ResourceCache.py @@ -126,7 +126,7 @@ class ResourceCache(object): rm = [] now = time.time() - for entry_hash, entry in self._index["entries"].iteritems(): + for entry_hash, entry in list(self._index["entries"].items()): if entry["last_used"] <= (now - self._expiration_period): rm.append(entry_hash) diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py index 84ff04a..8fae397 100644 --- a/lnst/Common/SecureSocket.py +++ b/lnst/Common/SecureSocket.py @@ -16,7 +16,7 @@ olichtne(a)redhat.com (Ondrej Lichtner) """ import os -import cPickle +import pickle import hashlib import hmac from lnst.Common.Utils import not_imported @@ -44,11 +44,11 @@ def bit_length(i): except AttributeError: return len(bin(i)) - 2 -DH_GROUP["q"] = (DH_GROUP["p"]-1)/2 -DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])/8 +DH_GROUP["q"] = (DH_GROUP["p"]-1)//2 +DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])//8 if bit_length(DH_GROUP["q"])%8: DH_GROUP["q_size"] += 1 -DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])/8 +DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])//8 if bit_length(DH_GROUP["p"])%8: DH_GROUP["p_size"] += 1 @@ -64,11 +64,11 @@ SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" "DE236D525F54759B65E372FCD68EF20FA7111F9E4AFF73", 16), "g": 2} -SRP_GROUP["q"] = (SRP_GROUP["p"]-1)/2 -SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])/8 +SRP_GROUP["q"] = (SRP_GROUP["p"]-1)//2 +SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])//8 if bit_length(SRP_GROUP["q"])%8: SRP_GROUP["q_size"] += 1 -SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8 +SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])//8 if bit_length(SRP_GROUP["p"])%8: SRP_GROUP["p_size"] += 1 @@ -151,14 +151,14 @@ class SecureSocket(object): "seq_num": 0} def send_msg(self, msg): - pickled_msg = cPickle.dumps(msg) + pickled_msg = pickle.dumps(msg) return self.send(pickled_msg) def recv_msg(self): pickled_msg = self.recv() - if pickled_msg == "": + if pickled_msg == b"": raise SecSocketException("Disconnected") - msg = cPickle.loads(pickled_msg) + msg = pickle.loads(pickled_msg) return msg def _add_mac_sign(self, data): @@ -166,22 +166,26 @@ class SecureSocket(object): return data cryptography_imports() - msg = str(self._current_write_spec["seq_num"]) + str(len(data)) + data + msg = (bytes(str(self._current_write_spec["seq_num"]).encode('ascii')) + + bytes(str(len(data)).encode('ascii')) + + data) signature = hmac.new(self._current_write_spec["mac_key"], msg, hashlib.sha256) signed_msg = {"data": data, "signature": signature.digest()} - return cPickle.dumps(signed_msg) + return pickle.dumps(signed_msg) def _del_mac_sign(self, signed_data): if not self._current_read_spec["mac_key"]: return signed_data cryptography_imports() - signed_msg = cPickle.loads(signed_data) + signed_msg = pickle.loads(signed_data) data = signed_msg["data"] - msg = str(self._current_read_spec["seq_num"]) + str(len(data)) + data + msg = (bytes(str(self._current_read_spec["seq_num"]).encode('ascii')) + + bytes(str(len(data)).encode('ascii')) + + data) signature = hmac.new(self._current_read_spec["mac_key"], msg, @@ -196,9 +200,9 @@ class SecureSocket(object): return data cryptography_imports() - block_size = algorithms.AES.block_size/8 + block_size = algorithms.AES.block_size//8 pad_length = block_size - (len(data) % block_size) - pad_char = ("%02x" % pad_length).decode("hex") + pad_char = bytes([pad_length]) padding = pad_length * pad_char padded_data = data+padding @@ -209,9 +213,9 @@ class SecureSocket(object): return data cryptography_imports() - pad_length = int(data[-1].encode("hex"), 16) + pad_length = ord(data[-1]) for char in data[-pad_length]: - if int(char.encode("hex"), 16) != pad_length: + if ord(char) != pad_length: return None return data[:-pad_length] @@ -221,7 +225,7 @@ class SecureSocket(object): return data cryptography_imports() - iv = os.urandom(algorithms.AES.block_size/8) + iv = os.urandom(algorithms.AES.block_size//8) mode = modes.CBC(iv) key = self._current_write_spec["enc_key"] cipher = Cipher(algorithms.AES(key), mode, default_backend()) @@ -232,14 +236,14 @@ class SecureSocket(object): encrypted_msg = {"iv": iv, "enc_data": encrypted_data} - return cPickle.dumps(encrypted_msg) + return pickle.dumps(encrypted_msg) def _del_encrypt(self, data): if not self._current_read_spec["enc_key"]: return data cryptography_imports() - encrypted_msg = cPickle.loads(data) + encrypted_msg = pickle.loads(data) encrypted_data = encrypted_msg["enc_data"] iv = encrypted_msg["iv"] @@ -277,27 +281,28 @@ class SecureSocket(object): def send(self, data): protected_data = self._protect_data(data) - transmit_data = str(len(protected_data)) + " " + protected_data + transmit_data = bytes(str(len(protected_data)).encode('ascii')) + b" " + protected_data return self._socket.sendall(transmit_data) def recv(self): - length = "" + length = b"" while True: c = self._socket.recv(1) - if c == ' ': - length = int(length) + + if c == b' ': + length = int(length.decode('ascii')) break - elif c == "": - return "" + elif c == b"": + return b"" else: length += c - data = "" + data = b"" while len(data) < length: c = self._socket.recv(length - len(data)) - if c == "": - return "" + if c == b"": + return b"" else: data += c @@ -308,7 +313,7 @@ class SecureSocket(object): def _handle_internal(self, orig_msg): try: - msg = cPickle.loads(orig_msg) + msg = pickle.loads(orig_msg) except: return orig_msg if "type" in msg and msg["type"] == "change_cipher_spec": @@ -349,7 +354,7 @@ class SecureSocket(object): def p_SHA256(self, secret, seed, length): prev_a = seed - result = "" + result = b"" while len(result) < length: a = hmac.new(secret, msg=prev_a, digestmod=hashlib.sha256) prev_a = a.digest() @@ -373,7 +378,7 @@ class SecureSocket(object): raise SecSocketException("Socket without a role!") cryptography_imports() - aes_keysize = max(algorithms.AES.key_sizes)/8 + aes_keysize = max(algorithms.AES.key_sizes)//8 mac_keysize = hashlib.sha256().block_size prf_seq = self.PRF(self._master_secret, diff --git a/lnst/Common/ShellProcess.py b/lnst/Common/ShellProcess.py index 74c1c14..1bd409a 100644 --- a/lnst/Common/ShellProcess.py +++ b/lnst/Common/ShellProcess.py @@ -230,7 +230,7 @@ class ShellProcess: except: return data if r and (r[0][1] & select.POLLIN): - new_data = os.read(fd, 1024) + new_data = os.read(fd, 1024).decode() if not new_data: return data data += new_data diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py index f158ff2..2b29a77 100644 --- a/lnst/Common/Utils.py +++ b/lnst/Common/Utils.py @@ -208,7 +208,7 @@ def get_module_tools(module_path): return tools def recursive_dict_update(original, update): - for key, value in update.iteritems(): + for key, value in list(update.items()): if isinstance(value, collections.Mapping): r = recursive_dict_update(original.get(key, {}), value) original[key] = r @@ -251,7 +251,7 @@ def list_to_dot(original_list, prefix="", key=""): def dict_to_dot(original_dict, prefix=""): return_list = [] - for key, value in original_dict.iteritems(): + for key, value in list(original_dict.items()): if isinstance(value, collections.Mapping): sub_list = dict_to_dot(value, prefix + key + '.') return_list.extend(sub_list) diff --git a/lnst/Common/Version.py b/lnst/Common/Version.py index c7bacfd..b6a2bc6 100644 --- a/lnst/Common/Version.py +++ b/lnst/Common/Version.py @@ -21,7 +21,7 @@ class LNSTVersion(object): self._version = "14" if git_version: - self._version += "-" + git_version() + self._version += "-" + git_version().decode() @property def version(self): diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py index 3e5473f..1166c8b 100644 --- a/lnst/Controller/Controller.py +++ b/lnst/Controller/Controller.py @@ -139,7 +139,7 @@ class Controller(object): self._machines = {} self._hosts = Hosts() pool = self._pools.get_machine_pool(match["pool_name"]) - for m_id, m in match["machines"].items(): + for m_id, m in list(match["machines"].items()): machine = self._machines[m_id] = pool[m["target"]] setattr(self._hosts, m_id, Host(machine)) @@ -149,7 +149,7 @@ class Controller(object): machine.set_mapped(True) self._prepare_machine(machine) - for if_id, i in m["interfaces"].items(): + for if_id, i in list(m["interfaces"].items()): host.map_device(if_id, i) if match["virtual"]: @@ -174,14 +174,14 @@ class Controller(object): if self._machines == None: return - for m_id, machine in self._machines.iteritems(): + for m_id, machine in list(self._machines.items()): try: machine.cleanup() except: #TODO report errors during deconfiguration as FAIL!! log_exc_traceback() finally: - for dev in machine._device_database.values(): + for dev in list(machine._device_database.values()): if isinstance(dev, VirtualDevice): dev._destroy() @@ -192,7 +192,7 @@ class Controller(object): self._machines.clear() # remove dynamically created bridges - for bridge in self._network_bridges.itervalues(): + for bridge in list(self._network_bridges.values()): bridge.cleanup() self._network_bridges = {} diff --git a/lnst/Controller/Host.py b/lnst/Controller/Host.py index ee36803..d19ea67 100644 --- a/lnst/Controller/Host.py +++ b/lnst/Controller/Host.py @@ -50,7 +50,7 @@ class Host(Namespace): Does not include the init namespace (self).""" ret = [] - for x in self._objects.values(): + for x in list(self._objects.values()): if isinstance(x, NetNamespace): ret.append(x) return ret diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py index 6bc82b5..f727062 100644 --- a/lnst/Controller/Machine.py +++ b/lnst/Controller/Machine.py @@ -109,7 +109,7 @@ class Machine(object): configuration["redhat_release"] = self._slave_desc["redhat_release"] configuration["interfaces"] = {} - for dev in self._device_database.items(): + for dev in list(self._device_database.items()): configuration["device_"+dev.name] = dev.get_config() return configuration @@ -166,14 +166,14 @@ class Machine(object): def dev_db_get_name(self, dev_name): #TODO move these to Slave to optimize quering for each device - for dev in self._device_database.values(): + for dev in list(self._device_database.values()): if dev.get_name() == dev_name: return dev return None def get_dev_by_hwaddr(self, hwaddr): #TODO move these to Slave to optimize quering for each device - for dev in self._device_database.values(): + for dev in list(self._device_database.values()): if dev.hwaddr == hwaddr: return dev return None @@ -247,7 +247,7 @@ class Machine(object): self._device_database = {} devices = self.rpc_call("get_devices") - for ifindex, dev in devices.items(): + for ifindex, dev in list(devices.items()): remote_dev = RemoteDevice(Device) remote_dev._machine = self remote_dev.ifindex = ifindex @@ -275,7 +275,7 @@ class Machine(object): for cls in reversed(classes): module_name = cls.__module__ - if module_name == "__builtin__": + if module_name == "builtins": continue module = sys.modules[module_name] diff --git a/lnst/Controller/MachineMapper.py b/lnst/Controller/MachineMapper.py index 2bc6ade..d229bfd 100644 --- a/lnst/Controller/MachineMapper.py +++ b/lnst/Controller/MachineMapper.py @@ -21,9 +21,9 @@ def format_match_description(match): output.append("Pool match description:") if match["virtual"]: output.append(" Setup is using virtual machines.") - for m_id, m in sorted(match["machines"].iteritems()): + for m_id, m in sorted(match["machines"].items()): output.append(" host \"{}\" uses \"{}\"".format(m_id, m["target"])) - for if_id, match in sorted(m["interfaces"].iteritems()): + for if_id, match in sorted(m["interfaces"].items()): pool_id = match["target"] output.append(" interface \"{}\" matched to \"{}\"". format(if_id, pool_id)) @@ -78,7 +78,7 @@ class MachineMapper(object): """resets the state of the backtracking algorithm""" self._net_label_mapping = {} self._machine_stack = [] - self._unmatched_req_machines = sorted(self._mreqs.keys(), reverse=True) + self._unmatched_req_machines = sorted(list(self._mreqs.keys()), reverse=True) self._pool_stack = list(self._pools.keys()) if len(self._pool_stack) > 0: @@ -86,7 +86,7 @@ class MachineMapper(object): self._pool = self._pools[self._pool_name] self._unmatched_pool_machines = [] - for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True): + for p_id, p_machine in sorted(list(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id) @@ -162,7 +162,7 @@ class MachineMapper(object): #map compatible pool machine stack_top["current_match"] = pool_m_id stack_top["unmatched_pool_ifs"] = \ - sorted(self._pool[pool_m_id]["interfaces"].keys(), + sorted(list(self._pool[pool_m_id]["interfaces"].keys()), reverse=True) self._unmatched_pool_machines.remove(pool_m_id) break @@ -186,7 +186,7 @@ class MachineMapper(object): self._pool_name) self._unmatched_pool_machines = [] - for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True): + for p_id, p_machine in sorted(list(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id) @@ -261,7 +261,7 @@ class MachineMapper(object): machine_match["if_stack"] = [] machine = self._mreqs[machine_match["m_id"]] - machine_match["unmatched_ifs"] = sorted(machine["interfaces"].keys(), + machine_match["unmatched_ifs"] = sorted(list(machine["interfaces"].keys()), reverse=True) machine_match["unmatched_pool_ifs"] = [] @@ -291,7 +291,7 @@ class MachineMapper(object): def _check_machine_compatibility(self, req_id, pool_id): req_machine = self._mreqs[req_id] pool_machine = self._pool[pool_id] - for param, value in req_machine["params"].iteritems(): + for param, value in list(req_machine["params"].items()): # skip empty parameters if len(value) == 0: continue @@ -302,14 +302,14 @@ class MachineMapper(object): def _check_interface_compatibility(self, req_if, pool_if): label_mapping = self._net_label_mapping - for req_label, mapping in label_mapping.iteritems(): + for req_label, mapping in list(label_mapping.items()): if req_label == req_if["network"] and\ mapping[0] != pool_if["network"]: return False if mapping[0] == pool_if["network"] and\ req_label != req_if["network"]: return False - for param, value in req_if["params"].iteritems(): + for param, value in list(req_if["params"].items()): # skip empty parameters if len(value) == 0: continue @@ -322,7 +322,7 @@ class MachineMapper(object): mapping = {"machines": {}, "networks": {}, "virtual": False, "pool_name": self._pool_name} - for req_label, label_map in self._net_label_mapping.iteritems(): + for req_label, label_map in list(self._net_label_mapping.items()): mapping["networks"][req_label] = label_map[0] for machine in self._machine_stack: diff --git a/lnst/Controller/MessageDispatcher.py b/lnst/Controller/MessageDispatcher.py index 222bc80..ed6aa51 100644 --- a/lnst/Controller/MessageDispatcher.py +++ b/lnst/Controller/MessageDispatcher.py @@ -31,7 +31,7 @@ def deviceref_to_remote_device(machine, obj): return dev elif isinstance(obj, dict): new_dict = {} - for key, value in obj.items(): + for key, value in list(obj.items()): new_dict[key] = deviceref_to_remote_device(machine, value) return new_dict @@ -55,7 +55,7 @@ def remote_device_to_deviceref(obj): return DeviceRef(obj.ifindex) elif isinstance(obj, dict): new_dict = {} - for key, value in obj.items(): + for key, value in list(obj.items()): new_dict[key] = remote_device_to_deviceref(value) return new_dict elif isinstance(obj, list): @@ -109,7 +109,7 @@ class MessageDispatcher(ConnectionHandler): result = None while True: - connected_slaves = self._connection_mapping.keys() + connected_slaves = list(self._connection_mapping.keys()) messages = self.check_connections() for msg in messages: @@ -122,7 +122,7 @@ class MessageDispatcher(ConnectionHandler): else: self._process_message(msg) - remaining_slaves = self._connection_mapping.keys() + remaining_slaves = list(self._connection_mapping.keys()) if connected_slaves != remaining_slaves: self._handle_disconnects(set(connected_slaves)- set(remaining_slaves)) @@ -147,7 +147,7 @@ class MessageDispatcher(ConnectionHandler): wait = True while wait: - connected_slaves = self._connection_mapping.keys() + connected_slaves = list(self._connection_mapping.keys()) messages = self.check_connections(timeout=1) for msg in messages: try: @@ -160,7 +160,7 @@ class MessageDispatcher(ConnectionHandler): wait = wait and not condition_wrapper() - remaining_slaves = self._connection_mapping.keys() + remaining_slaves = list(self._connection_mapping.keys()) if connected_slaves != remaining_slaves: self._handle_disconnects(set(connected_slaves)- set(remaining_slaves)) @@ -174,14 +174,14 @@ class MessageDispatcher(ConnectionHandler): return res def handle_messages(self): - connected_slaves = self._connection_mapping.keys() + connected_slaves = list(self._connection_mapping.keys()) messages = self.check_connections() for msg in messages: self._process_message(msg) - remaining_slaves = self._connection_mapping.keys() + remaining_slaves = list(self._connection_mapping.keys()) if connected_slaves != remaining_slaves: self._handle_disconnects(set(connected_slaves)- set(remaining_slaves)) diff --git a/lnst/Controller/Namespace.py b/lnst/Controller/Namespace.py index 88f82eb..7ffbaf8 100644 --- a/lnst/Controller/Namespace.py +++ b/lnst/Controller/Namespace.py @@ -62,7 +62,7 @@ class Namespace(object): def devices(self): """List of mapped devices available in the Namespace""" ret = [] - for x in self._objects.values(): + for x in list(self._objects.values()): if isinstance(x, Device) and x.netns == self: ret.append(x) return ret @@ -71,7 +71,7 @@ class Namespace(object): def device_database(self): """List of all devices (including unmapped) available in the Namespace""" ret = [] - for x in self._machine._device_database.values(): + for x in list(self._machine._device_database.values()): if isinstance(x, Device) and x.netns == self: ret.append(x) return ret @@ -197,7 +197,7 @@ class Namespace(object): def _unset(self, value): k_to_del = None - for k, v in self._objects.items(): + for k, v in list(self._objects.items()): if v == value: k_to_del = k break diff --git a/lnst/Controller/NetTestResultSerializer.py b/lnst/Controller/NetTestResultSerializer.py index e4fe086..b6cd5e8 100644 --- a/lnst/Controller/NetTestResultSerializer.py +++ b/lnst/Controller/NetTestResultSerializer.py @@ -94,10 +94,10 @@ class NetTestResultSerializer: "Setup is using virtual machines.", "")) - for m_id, m in sorted(match["machines"].iteritems()): + for m_id, m in sorted(match["machines"].items()): output_pairs.append((4*" " + "host \"%s\" uses \"%s\"" %\ (m_id, m["target"]), "")) - for if_id, pool_if in m["interfaces"].iteritems(): + for if_id, pool_if in list(m["interfaces"].items()): pool_id = pool_if["target"] if "driver" in pool_if: driver = pool_if["driver"] @@ -227,12 +227,12 @@ class NetTestResultSerializer: else: match_el.setAttribute("virtual", "false") - for m_id, m in match["machines"].iteritems(): + for m_id, m in list(match["machines"].items()): m_el = doc.createElement("m_match") m_el.setAttribute("host_id", str(m_id)) m_el.setAttribute("pool_id", str(m["target"])) - for if_id, pool_id in m["interfaces"].iteritems(): + for if_id, pool_id in list(m["interfaces"].items()): if_el = doc.createElement("if_match") if_el.setAttribute("if_id", str(if_id)) if_el.setAttribute("pool_if_id", str(pool_id)) diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py index f38b76b..68c8c37 100644 --- a/lnst/Controller/Recipe.py +++ b/lnst/Controller/Recipe.py @@ -109,7 +109,7 @@ class BaseRecipe(object): setattr(self.req, name, new_val) if len(kwargs): - for key in kwargs.keys(): + for key in list(kwargs.keys()): raise RecipeError("Unknown parameter {}".format(key)) @property @@ -170,4 +170,4 @@ class RecipeRun(object): @property def overall_result(self): - return reduce(lambda x, y: x and y, [i.success for i in self.results]) + return all([i.success for i in self.results]) diff --git a/lnst/Controller/Requirements.py b/lnst/Controller/Requirements.py index 92e1e84..342454f 100644 --- a/lnst/Controller/Requirements.py +++ b/lnst/Controller/Requirements.py @@ -35,7 +35,7 @@ class RecipeParam(Param): class BaseReq(object): def __init__(self, **kwargs): self.params = Parameters() - for name, val in kwargs.items(): + for name, val in list(kwargs.items()): if name == "params": raise RequirementError("'params' is a reserved keyword.") setattr(self.params, name, val) diff --git a/lnst/Controller/RunSummaryFormatter.py b/lnst/Controller/RunSummaryFormatter.py index 2629954..c081abf 100644 --- a/lnst/Controller/RunSummaryFormatter.py +++ b/lnst/Controller/RunSummaryFormatter.py @@ -52,7 +52,7 @@ class RunSummaryFormatter(object): output = [] if data is not None: if isinstance(data, dict): - for key, value in data.items(): + for key, value in list(data.items()): output.append("{pref}{key}:".format(pref=level*prefix, key=key)) nest_res = self._format_data(value, level=level+1) diff --git a/lnst/Controller/SlavePoolManager.py b/lnst/Controller/SlavePoolManager.py index e8ff4a6..3456332 100644 --- a/lnst/Controller/SlavePoolManager.py +++ b/lnst/Controller/SlavePoolManager.py @@ -47,16 +47,16 @@ class SlavePoolManager(object): self._pool_checks = pool_checks logging.info("Checking machine pool availability.") - for pool_name, pool_dir in pools.items(): + for pool_name, pool_dir in list(pools.items()): self._pools[pool_name] = {} self.add_dir(pool_name, pool_dir) if len(self._pools[pool_name]) == 0: del self._pools[pool_name] self._machines = {} - for pool_name, machines in self._pools.items(): + for pool_name, machines in list(self._pools.items()): pool = self._machines[pool_name] = {} - for m_id, m_spec in machines.items(): + for m_id, m_spec in list(machines.items()): params = m_spec["params"] hostname = params["hostname"] @@ -115,13 +115,13 @@ class SlavePoolManager(object): dir_path)) max_len = 0 - for m_id in pool.keys(): + for m_id in list(pool.keys()): if len(m_id) > max_len: max_len = len(m_id) if self._pool_checks: check_sockets = {} - for m_id, m in sorted(pool.iteritems()): + for m_id, m in sorted(pool.items()): hostname = m["params"]["hostname"] if "rpc_port" in m["params"]: port = int(m["params"]["rpc_port"]) @@ -153,7 +153,7 @@ class SlavePoolManager(object): check_sockets[s] = m_id while len(check_sockets) > 0: - rl, wl, el = select.select([], check_sockets.keys(), []) + rl, wl, el = select.select([], list(check_sockets.keys()), []) for s in wl: err = s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) m_id = check_sockets[s] @@ -167,7 +167,7 @@ class SlavePoolManager(object): s.close() del check_sockets[s] else: - for m_id in pool.keys(): + for m_id in list(pool.keys()): pool[m_id]["available"] = True for m_id in sorted(list(pool.keys())): @@ -208,7 +208,7 @@ class SlavePoolManager(object): # Check if there isn't any machine with the same # hostname or libvirt_domain already in the pool - for pm_id, m in pool.iteritems(): + for pm_id, m in pool.items(): pm = m["params"] rm = machine_spec["params"] if pm["hostname"] == rm["hostname"]: @@ -254,7 +254,7 @@ class SlavePoolManager(object): raise PoolManagerError(msg, iface) if_hwaddr = iface_spec["params"]["hwaddr"] - hwaddr_dups = [ k for k, v in machine_spec["interfaces"].iteritems()\ + hwaddr_dups = [ k for k, v in machine_spec["interfaces"].items()\ if v["params"]["hwaddr"] == if_hwaddr ] if len(hwaddr_dups) > 0: msg = "Duplicate MAC address %s for interface '%s' and '%s'."\ diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py index fdb1734..3264406 100644 --- a/lnst/Controller/Task.py +++ b/lnst/Controller/Task.py @@ -49,7 +49,7 @@ def breakpoint(): if not ctl.breakpoints: return logging.info("Breakpoint reached. Press enter to continue.") - raw_input("") + eval(input("")) def add_host(params={}): m_id = ctl.gen_m_id() @@ -130,7 +130,7 @@ class ControllerAPI(object): self._hosts[host_id] = handle def init_hosts(self, hosts): - for host_id, host in hosts.iteritems(): + for host_id, host in list(hosts.items()): self._hosts[host_id].init_host(host) def _run_command(self, command): @@ -229,20 +229,20 @@ class ControllerAPI(object): def get_configuration(self): machines = self._ctl._machines configuration = {} - for m_id, m in machines.items(): + for m_id, m in list(machines.items()): configuration["machine_"+m_id] = m.get_configuration() return configuration def get_mapping(self): match = self._ctl.get_pool_match() mapping = [] - for m_id, m in match["machines"].iteritems(): + for m_id, m in list(match["machines"].items()): machine = {} machine["id"] = m_id machine["pool_id"] = m["target"] machine["hostname"] = m["hostname"] machine["interface"] = [] - for i_id, i in m["interfaces"].iteritems(): + for i_id, i in list(m["interfaces"].items()): interface = {} interface["id"] = i_id interface["pool_id"] = i["target"] @@ -347,7 +347,7 @@ class HostAPI(object): bg_id = None cmd["netns"] = None - for arg, argval in kwargs.iteritems(): + for arg, argval in list(kwargs.items()): if arg == "bg" and argval == True: self._bg_id_seq += 1 cmd["bg_id"] = bg_id = self._bg_id_seq @@ -706,7 +706,7 @@ class ModuleAPI(object): self._name = module_name self._opts = {} - for opt, val in options.iteritems(): + for opt, val in list(options.items()): self._opts[opt] = [] if type(val) == list: for v in val: @@ -719,7 +719,7 @@ class ModuleAPI(object): def set_options(self, options): self._opts = {} - for opt, val in options.iteritems(): + for opt, val in list(options.items()): self._opts[opt] = [] if type(val) == list: for v in val: @@ -728,7 +728,7 @@ class ModuleAPI(object): self._opts[opt].append({"value": str(val)}) def update_options(self, options): - for opt, val in options.iteritems(): + for opt, val in list(options.items()): self._opts[opt] = [] if type(val) == list: for v in val: diff --git a/lnst/Controller/VirtDomainCtl.py b/lnst/Controller/VirtDomainCtl.py index 96c7913..e4b8421 100644 --- a/lnst/Controller/VirtDomainCtl.py +++ b/lnst/Controller/VirtDomainCtl.py @@ -71,14 +71,14 @@ class VirtDomainCtl(object): self._domain.attachDevice(device_xml) logging.debug("libvirt device with hwaddr '%s' " "driver '%s' attached" % (hw_addr, driver)) - self._created_interfaces[hw_addr] = device_xml + self._created_interfaces[str(hw_addr)] = device_xml return True except libvirtError as e: raise VirtDomainCtlError(str(e)) def detach_interface(self, hw_addr): - if hw_addr in self._created_interfaces: - device_xml = self._created_interfaces[hw_addr] + if str(hw_addr) in self._created_interfaces: + device_xml = self._created_interfaces[str(hw_addr)] else: device_xml = self._net_device_bare_template.format(hw_addr) diff --git a/lnst/Controller/Wizard.py b/lnst/Controller/Wizard.py index dc82c21..0862c9c 100644 --- a/lnst/Controller/Wizard.py +++ b/lnst/Controller/Wizard.py @@ -222,8 +222,8 @@ class Wizard: """ while True: if pool_dir is None: - pool_dir = raw_input("Enter path to a pool directory " - "(default: '%s'): " % DefaultPoolDir) + pool_dir = eval(input("Enter path to a pool directory " + "(default: '%s'): " % DefaultPoolDir)) if pool_dir == "": pool_dir = DefaultPoolDir @@ -316,7 +316,7 @@ class Wizard: if mode == "interactive": msg = "Do you want to add interface '%s' (%s) to the "\ "recipe? [Y/n]: " % (iface["name"], iface["hwaddr"]) - answer = raw_input(msg) + answer = eval(input(msg)) if mode == "noninteractive" or answer.lower() == "y"\ or answer == "": interfaces_added += 1 @@ -431,7 +431,7 @@ class Wizard: """ Queries user for adding next machine @return True if user wants to add another machine, False otherwise """ - answer = raw_input("Do you want to add another machine? [Y/n]: ") + answer = eval(input("Do you want to add another machine? [Y/n]: ")) if answer.lower() == "y" or answer == "": return True else: @@ -441,7 +441,7 @@ class Wizard: """ Queries user for creating specified directory @return True if user wants to create the directory, False otherwise """ - answer = raw_input("Create dir '%s'? [Y/n]: " % pool_dir) + answer = eval(input("Create dir '%s'? [Y/n]: " % pool_dir)) if answer.lower() == 'y' or answer == "": return True else: @@ -452,9 +452,9 @@ class Wizard: @hostname Hostname of the machine which is used as default filename @return Name of the file with .xml extension """ - output_file = raw_input("Enter the name of the output .xml file " + output_file = eval(input("Enter the name of the output .xml file " "(without .xml, default is '%s.xml'): " - % hostname) + % hostname)) if output_file == "": return hostname + ".xml" else: @@ -465,7 +465,7 @@ class Wizard: @return Valid (is translatable to an IP address) hostname """ while True: - hostname = raw_input("Enter hostname: ") + hostname = eval(input("Enter hostname: ")) if hostname == "": sys.stderr.write("No hostname entered\n") continue @@ -485,8 +485,8 @@ class Wizard: string representing hostname of the host """ while True: - libvirt_domain = raw_input("Enter libvirt domain " - "of virtual host: ") + libvirt_domain = eval(input("Enter libvirt domain " + "of virtual host: ")) if libvirt_domain == "": sys.stderr.write("No domain entered\n") continue @@ -524,7 +524,7 @@ class Wizard: @return Integer representing port """ while True: - port = raw_input("Enter port (default: %d): " % DefaultRPCPort) + port = eval(input("Enter port (default: %d): " % DefaultRPCPort)) if port == "": return DefaultRPCPort else: @@ -539,7 +539,7 @@ class Wizard: @return Dictionary with the security parameters """ while True: - auth_type = raw_input("Enter authentication type (default: none): ") + auth_type = eval(input("Enter authentication type (default: none): ")) if auth_type == "": auth_type = "none" elif auth_type not in ["none", "no-auth", "password", @@ -555,7 +555,7 @@ class Wizard: return {"auth_type": "ssh"} elif auth_type == "password": while True: - password = raw_input("Enter password: ") + password = eval(input("Enter password: ")) if password == "": sys.stderr.write("Invalid password.") continue @@ -564,19 +564,19 @@ class Wizard: "auth_passwd": password} elif auth_type == "pubkey": while True: - identity = raw_input("Enter identity: ") + identity = eval(input("Enter identity: ")) if identity == "": sys.stderr.write("Invalid identity.") continue break while True: - privkey = raw_input("Enter path to Ctl private key: ") + privkey = eval(input("Enter path to Ctl private key: ")) if privkey == "" or not os.path.isfile(privkey): sys.stderr.write("Invalid path to private key.") continue break while True: - srv_pubkey_path = raw_input("Enter path to Slave public key: ") + srv_pubkey_path = eval(input("Enter path to Slave public key: ")) if srv_pubkey_path == "" or not os.path.isfile(srv_pubkey_path): sys.stderr.write("Invalid path to public key.") continue diff --git a/lnst/Devices/BondDevice.py b/lnst/Devices/BondDevice.py index 99da928..07fd124 100644 --- a/lnst/Devices/BondDevice.py +++ b/lnst/Devices/BondDevice.py @@ -82,7 +82,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_AD_SELECT", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -159,7 +159,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_ARP_VALIDATE", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -177,7 +177,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_ARP_ALL_TARGETS", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -204,7 +204,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_FAIL_OVER_MAC", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -222,7 +222,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_LACP_RATE", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -259,7 +259,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_MODE", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") @@ -317,7 +317,7 @@ class BondDevice(MasterDevice): self._set_linkinfo_data_attr("IFLA_BOND_PRIMARY_RESELECT", val) else: raise DeviceConfigError("Invalid value, must be in {} or {}.". - format(m.keys(), m.values())) + format(list(m.keys()), list(m.values()))) self._nl_sync("set") diff --git a/lnst/Devices/Device.py b/lnst/Devices/Device.py index 3037f45..4d630cd 100644 --- a/lnst/Devices/Device.py +++ b/lnst/Devices/Device.py @@ -35,7 +35,7 @@ except ImportError: from pyroute2.iproute import RTM_NEWADDR from pyroute2.iproute import RTM_DELADDR -class Device(object): +class Device(object, metaclass=ABCMeta): """The base Device class Implemented using the pyroute2 package to access different attributes of @@ -47,7 +47,6 @@ class Device(object): public methods defined in this and derived class are directly available as a tester facing API. """ - __metaclass__ = ABCMeta def __init__(self, if_manager): self.ifindex = None @@ -89,7 +88,7 @@ class Device(object): for k, v in msg.items(): if isinstance(v, dict): processed = self._process_nested_nl_attrs(v["attrs"]) - ret[k] = {"attrs": processed.items()} + ret[k] = {"attrs": list(processed.items())} else: ret[k] = v return ret @@ -238,18 +237,18 @@ class Device(object): def _clear_tc_qdisc(self): exec_cmd("tc qdisc del dev %s root" % self.name, die_on_err=False) out, _ = exec_cmd("tc filter show dev %s" % self.name) - ingress_handles = re.findall("ingress (\\d+):", out) + ingress_handles = re.findall("ingress (\\d+):", out.decode()) for ingress_handle in ingress_handles: exec_cmd("tc qdisc del dev %s handle %s: ingress" % (self.name, ingress_handle)) out, _ = exec_cmd("tc qdisc show dev %s" % self.name) - ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out) + ingress_qdiscs = re.findall("qdisc ingress (\\w+):", out.decode()) if len(ingress_qdiscs) != 0: exec_cmd("tc qdisc del dev %s ingress" % self.name) def _clear_tc_filters(self): out, _ = exec_cmd("tc filter show dev %s" % self.name) - egress_prefs = re.findall("pref (\\d+) .* handle", out) + egress_prefs = re.findall("pref (\\d+) .* handle", out.decode()) for egress_pref in egress_prefs: exec_cmd("tc filter del dev %s pref %s" % (self.name, egress_pref)) @@ -637,6 +636,7 @@ class Device(object): def _read_adaptive_coalescing(self): try: res = exec_cmd("ethtool -c %s" % self.name) + res = res.decode() except: logging.debug("Could not read coalescence values of %s." % self.name) return (None, None) diff --git a/lnst/RecipeCommon/Perf/Measurements/BaseCPUMeasurement.py b/lnst/RecipeCommon/Perf/Measurements/BaseCPUMeasurement.py index 0db5afd..327a54c 100644 --- a/lnst/RecipeCommon/Perf/Measurements/BaseCPUMeasurement.py +++ b/lnst/RecipeCommon/Perf/Measurements/BaseCPUMeasurement.py @@ -64,7 +64,7 @@ class BaseCPUMeasurement(BaseMeasurement): @classmethod def report_results(cls, recipe, results): results_by_host = cls._divide_results_by_host(results) - for host_results in results_by_host.values(): + for host_results in list(results_by_host.values()): cls._report_host_results(recipe, host_results) @classmethod diff --git a/lnst/RecipeCommon/Perf/Measurements/IperfFlowMeasurement.py b/lnst/RecipeCommon/Perf/Measurements/IperfFlowMeasurement.py index 06d38ce..c16eb94 100644 --- a/lnst/RecipeCommon/Perf/Measurements/IperfFlowMeasurement.py +++ b/lnst/RecipeCommon/Perf/Measurements/IperfFlowMeasurement.py @@ -13,6 +13,8 @@ from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import NetworkFlowT from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import BaseFlowMeasurement from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import FlowMeasurementResults +from lnst.RecipeCommon.Perf.Measurements.MeasurementError import MeasurementError + from lnst.Tests.Iperf import IperfClient, IperfServer class IperfFlowMeasurement(BaseFlowMeasurement): @@ -111,13 +113,13 @@ class IperfFlowMeasurement(BaseFlowMeasurement): server_params = dict(bind = ipaddress(flow.receiver_bind), oneoff = True) - if flow.cpupin >= 0: + if flow.cpupin and flow.cpupin >= 0: if flow.parallel_streams == 1: server_params["cpu_bind"] = flow.cpupin else: raise RecipeError("Unsupported combination of single cpupin " "with parallel perf streams.") - elif not flow.cpupin is None: + elif flow.cpupin is not None: raise RecipeError("Negative perf cpupin value provided.") return host.prepare_job(IperfServer(**server_params), @@ -138,13 +140,13 @@ class IperfFlowMeasurement(BaseFlowMeasurement): else: raise RecipeError("Unsupported flow type '{}'".format(flow.type)) - if flow.cpupin >= 0: + if flow.cpupin and flow.cpupin >= 0: if flow.parallel_streams == 1: client_params["cpu_bind"] = flow.cpupin else: raise RecipeError("Unsupported combination of single cpupin " "with parallel perf streams.") - elif not flow.cpupin is None: + elif flow.cpupin is not None: raise RecipeError("Negative perf cpupin value provided.") if flow.parallel_streams > 1: diff --git a/lnst/RecipeCommon/Perf/Measurements/StatCPUMeasurement.py b/lnst/RecipeCommon/Perf/Measurements/StatCPUMeasurement.py index f74f47f..dd2fd64 100644 --- a/lnst/RecipeCommon/Perf/Measurements/StatCPUMeasurement.py +++ b/lnst/RecipeCommon/Perf/Measurements/StatCPUMeasurement.py @@ -14,7 +14,7 @@ class StatCPUMeasurementResults(CPUMeasurementResults): self._data = {} def update_intervals(self, intervals): - for key, interval in intervals.items(): + for key, interval in list(intervals.items()): if key not in self._data: self._data[key] = SequentialPerfResult() self._data[key].append(interval) @@ -68,24 +68,24 @@ class StatCPUMeasurement(BaseCPUMeasurement): for sample in job.result["data"]: parsed_sample = self._parse_sample(sample) - for cpu, cpu_intervals in parsed_sample.items(): + for cpu, cpu_intervals in list(parsed_sample.items()): if cpu not in job_results: job_results[cpu] = StatCPUMeasurementResults(self, host, cpu) cpu_results = job_results[cpu] cpu_results.update_intervals(cpu_intervals) - return job_results.values() + return list(job_results.values()) def _parse_sample(self, sample): result = {} duration = sample["duration"] - for key, value in sample.items(): + for key, value in list(sample.items()): if key.startswith("cpu"): result[key] = self._create_cpu_intervals(duration, value) return result def _create_cpu_intervals(self, duration, cpu_intervals): result = {} - for key, value in cpu_intervals.items(): + for key, value in list(cpu_intervals.items()): result[key] = PerfInterval(value, duration, "time units") return result diff --git a/lnst/RecipeCommon/Perf/Measurements/TRexFlowMeasurement.py b/lnst/RecipeCommon/Perf/Measurements/TRexFlowMeasurement.py index 3c973ca..11449d0 100644 --- a/lnst/RecipeCommon/Perf/Measurements/TRexFlowMeasurement.py +++ b/lnst/RecipeCommon/Perf/Measurements/TRexFlowMeasurement.py @@ -11,6 +11,8 @@ from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import BaseFlowMeas from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import NetworkFlowTest from lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement import FlowMeasurementResults +from lnst.RecipeCommon.Perf.Measurements.MeasurementError import MeasurementError + from lnst.Tests.TRex import TRexServer, TRexClient class TRexFlowMeasurement(BaseFlowMeasurement): @@ -56,7 +58,7 @@ class TRexFlowMeasurement(BaseFlowMeasurement): tests = [] flows_by_generator = self._flows_by_generator(flows) - for generator, flows in flows_by_generator.items(): + for generator, flows in list(flows_by_generator.items()): flow_tuples = [(flow.generator_bind, flow.receiver_bind) for flow in flows] server_job = generator.prepare_job( @@ -67,7 +69,7 @@ class TRexFlowMeasurement(BaseFlowMeasurement): client_job = generator.prepare_job( TRexClient( trex_dir=self._trex_dir, - ports=range(len(flow_tuples)), + ports=list(range(len(flow_tuples))), flows=flow_tuples, duration=flows[0].duration, msg_size=flows[0].msg_size)) @@ -96,7 +98,7 @@ class TRexFlowMeasurement(BaseFlowMeasurement): else: result[flow.generator] = [flow] - for generator, flows in result.items(): + for generator, flows in list(result.items()): for flow in flows: if (flow.duration != flows[0].duration or flow.msg_size != flows[0].msg_size): diff --git a/lnst/RecipeCommon/Perf/Recipe.py b/lnst/RecipeCommon/Perf/Recipe.py index 291e79d..df69b1b 100644 --- a/lnst/RecipeCommon/Perf/Recipe.py +++ b/lnst/RecipeCommon/Perf/Recipe.py @@ -75,7 +75,7 @@ class Recipe(BaseRecipe): self.add_result(False, "No results available to report.") return - for measurement, results in recipe_results.results.items(): + for measurement, results in list(recipe_results.results.items()): measurement.report_results(self, results) def perf_evaluate(self, recipe_results): @@ -85,7 +85,7 @@ class Recipe(BaseRecipe): recipe_conf = recipe_results.recipe_conf - for measurement, results in recipe_results.results.items(): + for measurement, results in list(recipe_results.results.items()): evaluators = recipe_conf.evaluators.get(measurement, []) for evaluator in evaluators: evaluator.evaluate_results(self, results) diff --git a/lnst/RecipeCommon/PerfRepo/PerfRepo.py b/lnst/RecipeCommon/PerfRepo/PerfRepo.py index 9e2af41..fd70c92 100644 --- a/lnst/RecipeCommon/PerfRepo/PerfRepo.py +++ b/lnst/RecipeCommon/PerfRepo/PerfRepo.py @@ -305,7 +305,6 @@ class PerfRepoAPI(object): return True else: return False - return False class PerfRepoResult(object): def __init__(self, test, name, hash_ignore=[]): diff --git a/lnst/RecipeCommon/TestRecipe.py b/lnst/RecipeCommon/TestRecipe.py index 05fc683..360cd2a 100755 --- a/lnst/RecipeCommon/TestRecipe.py +++ b/lnst/RecipeCommon/TestRecipe.py @@ -1,4 +1,4 @@ -#!/bin/python2 +#!/bin/python3 import time import re diff --git a/lnst/Recipes/ENRT/BaseEnrtRecipe.py b/lnst/Recipes/ENRT/BaseEnrtRecipe.py index c886d09..289f2a1 100644 --- a/lnst/Recipes/ENRT/BaseEnrtRecipe.py +++ b/lnst/Recipes/ENRT/BaseEnrtRecipe.py @@ -148,7 +148,7 @@ class BaseEnrtRecipe(PingTestAndEvaluate, PerfRecipe): server_nic.name) ethtool_offload_string = "" - for name, value in sub_config.offload_settings.items(): + for name, value in list(sub_config.offload_settings.items()): ethtool_offload_string += " %s %s" % (name, value) client_netns.run("ethtool -K {} {}".format(client_nic.name, @@ -169,7 +169,7 @@ class BaseEnrtRecipe(PingTestAndEvaluate, PerfRecipe): server_nic.name) ethtool_offload_string = "" - for name, value in sub_config.offload_settings.items(): + for name, value in list(sub_config.offload_settings.items()): ethtool_offload_string += " %s %s" % (name, "on") #set all the offloads back to 'on' state @@ -279,8 +279,8 @@ class BaseEnrtRecipe(PingTestAndEvaluate, PerfRecipe): server_bind = server_nic.ips_filter(family=family)[0] for perf_test in self.params.perf_tests: - offload_values = sub_config.offload_settings.values() - offload_items = sub_config.offload_settings.items() + offload_values = list(sub_config.offload_settings.values()) + offload_items = list(sub_config.offload_settings.items()) if ((perf_test == 'udp_stream' and ('gro', 'off') in offload_items) or (perf_test == 'sctp_stream' and 'off' in offload_values and diff --git a/lnst/Recipes/ENRT/PingFloodRecipe.py b/lnst/Recipes/ENRT/PingFloodRecipe.py index 92fbef3..0b0c574 100644 --- a/lnst/Recipes/ENRT/PingFloodRecipe.py +++ b/lnst/Recipes/ENRT/PingFloodRecipe.py @@ -1,4 +1,4 @@ -#!/bin/python2 +#!/bin/python3 """ Implements scenario similar to regression_tests/phase1/ (ping_flood.xml + simple_ping.py). diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py index b5d0091..e66455a 100644 --- a/lnst/Slave/InterfaceManager.py +++ b/lnst/Slave/InterfaceManager.py @@ -97,7 +97,7 @@ class InterfaceManager(object): #devices with outdated messages self.get_netlink_messages() - devices_to_remove = self._devices.keys() + devices_to_remove = list(self._devices.keys()) devs = scan_netdevs() for dev in devs: if dev['index'] not in self._devices: @@ -199,18 +199,18 @@ class InterfaceManager(object): def get_devices(self): self.rescan_devices() - return self._devices.values() + return list(self._devices.values()) def get_device_by_hwaddr(self, hwaddr): self.rescan_devices() - for dev in self._devices.values(): + for dev in list(self._devices.values()): if dev.hwaddr == hwaddr: return dev raise DeviceNotFound() def get_device_by_name(self, name): self.rescan_devices() - for dev in self._devices.values(): + for dev in list(self._devices.values()): if dev.name == name: return dev raise DeviceNotFound() @@ -218,10 +218,10 @@ class InterfaceManager(object): def get_device_by_params(self, params): self.rescan_devices() matched = None - for dev in self._devices.values(): + for dev in list(self._devices.values()): matched = dev dev_data = dev.get_if_data() - for key, value in params.iteritems(): + for key, value in params.items(): if key not in dev_data or dev_data[key] != value: matched = None break @@ -232,7 +232,7 @@ class InterfaceManager(object): return matched def deconfigure_all(self): - for dev in self._devices.itervalues(): + for dev in self._devices.values(): pass # dev.clear_configuration() @@ -261,12 +261,13 @@ class InterfaceManager(object): def _is_name_used(self, name): self.rescan_devices() - for device in self._devices.itervalues(): + for device in self._devices.values(): if name == device.name: return True out, _ = exec_cmd("ovs-vsctl --columns=name list Interface", log_outputs=False, die_on_err=False) + out = out.decode() for line in out.split("\n"): m = re.match(r'.*: \"(.*)\"', line) if m is not None: diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py index 89d471e..7617767 100644 --- a/lnst/Slave/NetConfigDevice.py +++ b/lnst/Slave/NetConfigDevice.py @@ -226,7 +226,7 @@ class NetConfigDeviceMacvlan(NetConfigDeviceGeneric): _modulename = "macvlan" def create(self): - config = self._dev_config; + config = self._dev_config realdev_id = config["slaves"][0] realdev_name = self._if_manager.get_mapped_device(realdev_id).get_name() dev_name = config["name"] @@ -255,7 +255,7 @@ class NetConfigDeviceVlan(NetConfigDeviceGeneric): return False def _get_vlan_info(self): - config = self._dev_config; + config = self._dev_config realdev_id = get_slaves(config)[0] realdev_name = self._if_manager.get_mapped_device(realdev_id).get_name() dev_name = config["name"] @@ -463,7 +463,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"] bond_ports = [] - for bond in self._dev_config["ovs_conf"]["bonds"].itervalues(): + for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id) @@ -478,7 +478,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): options += " %s=%s" % (opt[0], opt[1]) vlan_tags = [] - for tag, vlan in vlans.iteritems(): + for tag, vlan in vlans.items(): if slave_id in vlan["slaves"]: vlan_tags.append(tag) if len(vlan_tags) == 0: @@ -498,7 +498,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"] bond_ports = [] - for bond in self._dev_config["ovs_conf"]["bonds"].itervalues(): + for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id) @@ -565,7 +565,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"] bonds = self._dev_config["ovs_conf"]["bonds"] - for bond_id, bond in bonds.iteritems(): + for bond_id, bond in bonds.items(): ifaces = "" for slave_id in bond["slaves"]: slave_dev = self._if_manager.get_mapped_device(slave_id) @@ -581,7 +581,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"] bonds = self._dev_config["ovs_conf"]["bonds"] - for bond_id, bond in bonds.iteritems(): + for bond_id, bond in bonds.items(): exec_cmd("ovs-vsctl del-port %s %s" % (br_name, bond_id)) def _add_flow_entries(self): diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index f41091c..363f053 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -68,6 +68,10 @@ RecipeCommon.__path__ = ["lnst.RecipeCommon"] sys.modules["lnst.RecipeCommon"] = RecipeCommon +class SystemCallException(Exception): + """Exception used to handle SIGINT waiting for system calls""" + pass + class SlaveMethods: ''' Exported xmlrpc methods @@ -278,7 +282,7 @@ class SlaveMethods: dev_data = dev._get_if_data() entry = {"name": dev.get_name(), "hwaddr": dev.get_hwaddr()} - for key, value in params.iteritems(): + for key, value in params.items(): if key not in dev_data or dev_data[key] != value: entry = None break @@ -326,7 +330,7 @@ class SlaveMethods: raise Exception("Can't start packet capture, tcpdump not available") files = {} - for if_id, dev in self._if_manager.get_mapped_devices().iteritems(): + for if_id, dev in self._if_manager.get_mapped_devices().items(): if dev.get_netns() != None: continue dev_name = dev.get_name() @@ -352,7 +356,7 @@ class SlaveMethods: if self._packet_captures == None: return True - for ifindex, pcap in self._packet_captures.iteritems(): + for ifindex, pcap in self._packet_captures.items(): pcap.stop() self._packet_captures.clear() @@ -360,7 +364,7 @@ class SlaveMethods: return True def _remove_capture_files(self): - for key, name in self._capture_files.iteritems(): + for key, name in self._capture_files.items(): logging.debug("Removing temporary packet capture file %s", name) os.unlink(name) @@ -383,7 +387,7 @@ class SlaveMethods: def restore_system_config(self): logging.info("Restoring system configuration") - for option, values in self._system_config.iteritems(): + for option, values in self._system_config.items(): try: cmd_str = "echo \"%s\" >%s" % (values["initial_val"], option) (stdout, stderr) = exec_cmd(cmd_str) @@ -441,11 +445,11 @@ class SlaveMethods: if self._if_manager is not None: self._if_manager.deconfigure_all() - for netns in self._net_namespaces.keys(): + for netns in list(self._net_namespaces.keys()): self.del_namespace(netns) self._net_namespaces = {} - for obj_id, obj in self._dynamic_objects.items(): + for obj_id, obj in list(self._dynamic_objects.items()): del obj for cls_name in dir(Devices): @@ -453,7 +457,7 @@ class SlaveMethods: if isclass(cls): delattr(Devices, cls_name) - for module_name, module in self._dynamic_modules.items(): + for module_name, module in list(self._dynamic_modules.items()): del sys.modules[module_name] self._dynamic_objects = {} @@ -527,11 +531,11 @@ class SlaveMethods: return False def reset_file_transfers(self): - for file_handle in self._copy_targets.itervalues(): + for file_handle in self._copy_targets.values(): file_handle.close() self._copy_targets = {} - for file_handle in self._copy_sources.itervalues(): + for file_handle in self._copy_sources.values(): file_handle.close() self._copy_sources = {} @@ -892,7 +896,7 @@ class ServerHandler(ConnectionHandler): self._netns_con_mapping = {} def update_connections(self, connections): - for key, connection in connections.iteritems(): + for key, connection in connections.items(): self.remove_connection_by_id(key) self.add_connection(key, connection) @@ -926,7 +930,7 @@ def device_to_deviceref(obj): return dev_ref elif isinstance(obj, dict): new_dict = {} - for key, value in obj.items(): + for key, value in list(obj.items()): new_dict[key] = device_to_deviceref(value) return new_dict elif isinstance(obj, list): @@ -953,7 +957,7 @@ def deviceref_to_device(if_manager, obj): return dev elif isinstance(obj, dict): new_dict = {} - for key, value in obj.items(): + for key, value in list(obj.items()): new_dict[key] = deviceref_to_device(if_manager, value) return new_dict elif isinstance(obj, list): @@ -1002,21 +1006,23 @@ class NetTestSlave: self._log_ctl = log_ctl def run(self): - while not self._finished: - if self._server_handler.get_ctl_sock() == None: - self._log_ctl.cancel_connection() - try: - logging.info("Waiting for connection.") - self._server_handler.accept_connection() - except (socket.error, SecSocketException): - log_exc_traceback() - continue - self._log_ctl.set_connection(self._server_handler.get_ctl_sock()) - - msgs = self._server_handler.get_messages() - - for msg in msgs: - self._process_msg(msg[1]) + while True: + try: + if self._server_handler.get_ctl_sock() is None: + self._log_ctl.cancel_connection() + try: + logging.info("Waiting for connection.") + self._server_handler.accept_connection() + except (socket.error, SecSocketException): + continue + self._log_ctl.set_connection(self._server_handler.get_ctl_sock()) + + msgs = self._server_handler.get_messages() + + for msg in msgs: + self._process_msg(msg[1]) + except SystemCallException: + break self._methods.machine_cleanup() @@ -1106,7 +1112,7 @@ class NetTestSlave: def _signal_die_handler(self, signum, frame): logging.info("Caught signal %d -> dying" % signum) - self._finished = True + raise SystemCallException() def _parent_resend_signal_handler(self, signum, frame): logging.info("Caught signal %d -> resending to parent" % signum) diff --git a/lnst/Tests/BaseTestModule.py b/lnst/Tests/BaseTestModule.py index d96c1a4..27ba378 100644 --- a/lnst/Tests/BaseTestModule.py +++ b/lnst/Tests/BaseTestModule.py @@ -79,7 +79,7 @@ class BaseTestModule(object): .format(name)) if len(kwargs): - for name in kwargs.keys(): + for name in list(kwargs.keys()): raise TestModuleError("Unknown parameter {}".format(name)) self._res_data = None diff --git a/lnst/Tests/CPUStatMonitor.py b/lnst/Tests/CPUStatMonitor.py index 6e317ea..a010514 100644 --- a/lnst/Tests/CPUStatMonitor.py +++ b/lnst/Tests/CPUStatMonitor.py @@ -58,7 +58,7 @@ class CPUStatMonitor(BaseTestModule): def _subtract_nested_dicts(self, first, second): result = {} - for key, val in first.items(): + for key, val in list(first.items()): if isinstance(val, dict): result[key] = self._subtract_nested_dicts(val, second[key]) else: diff --git a/lnst/Tests/Iperf.py b/lnst/Tests/Iperf.py index 0a14619..c12cde2 100644 --- a/lnst/Tests/Iperf.py +++ b/lnst/Tests/Iperf.py @@ -24,6 +24,8 @@ class IperfBase(BaseTestModule): try: stdout, stderr = server.communicate() + stdout = stdout.decode() + stderr = stderr.decode() except KeyboardInterrupt: pass @@ -102,7 +104,7 @@ class IperfClient(IperfBase): def runtime_estimate(self): _duration_overhead = 5 - return (self.params.duration + _duration_overhead) + return self.params.duration + _duration_overhead def _compose_cmd(self): port = "" diff --git a/lnst/Tests/Netperf.py b/lnst/Tests/Netperf.py index 9e9dd5e..31f54a1 100644 --- a/lnst/Tests/Netperf.py +++ b/lnst/Tests/Netperf.py @@ -477,7 +477,7 @@ class Netperf(BaseTestModule): rate_deviation = 2*res_data["std_deviation"] elif len(rates) == 1 and "confidence" in self.params: result = results[0] - rate_deviation = rate * (result["confidence"][1] / 100) + rate_deviation = rate * (float(result["confidence"][1]) / 100) else: rate_deviation = 0.0 diff --git a/lnst/Tests/Ping.py b/lnst/Tests/Ping.py index a81cf1c..390a8e4 100644 --- a/lnst/Tests/Ping.py +++ b/lnst/Tests/Ping.py @@ -48,6 +48,8 @@ class Ping(BaseTestModule): try: stdout, stderr = ping_process.communicate() + stdout = stdout.decode() + stderr = stderr.decode() except KeyboardInterrupt: pass diff --git a/misc/recipe_conv.py b/misc/recipe_conv.py index a18a0d2..3e133f1 100755 --- a/misc/recipe_conv.py +++ b/misc/recipe_conv.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 """ Recipe converter @@ -106,7 +106,7 @@ def convert_recipe(file_path): writer.close() def usage(): - print "Usage: %s recipe1 recipe2 dir1 ..." % sys.argv[0] + print("Usage: %s recipe1 recipe2 dir1 ..." % sys.argv[0]) def main(): if len(sys.argv) <= 1 or "-h" in sys.argv: @@ -122,10 +122,10 @@ def main(): if re.match(r"^.*\.xml$", file_name): full_path = "%s/%s" % (root, file_name) full_path = os.path.normpath(full_path) - print "Converting %s" % full_path + print("Converting %s" % full_path) convert_recipe(full_path) else: - print "Converting %s" % file_path + print("Converting %s" % file_path) convert_recipe(file_path) return 0 diff --git a/pyrecipes/ping_flood.py b/pyrecipes/ping_flood.py index 35d4741..257f7eb 100644 --- a/pyrecipes/ping_flood.py +++ b/pyrecipes/ping_flood.py @@ -13,11 +13,11 @@ while lnst.match(): lnst.wait(15) ipv = lnst.get_alias("ipv", default="ipv4") - print "ipv" - print ipv + print("ipv") + print(ipv) mtu = lnst.get_alias("mtu", default="1500") - print "mtu" - print mtu + print("mtu") + print(mtu) m1_eth1.reset(ip=["192.168.101.10/24", "fc00:0:0:0::1/64"]) m2_eth1.reset(ip=["192.168.101.11/24", "fc00:0:0:0::2/64"]) diff --git a/recipes/examples/python_recipe.py b/recipes/examples/python_recipe.py index 00dd329..ac323af 100755 --- a/recipes/examples/python_recipe.py +++ b/recipes/examples/python_recipe.py @@ -1,4 +1,4 @@ -#!/bin/python2 +#!/bin/python3 """ This is an example python recipe that can be run as an executable script. Performs a simple ping between two hosts. @@ -20,7 +20,7 @@ from lnst.Devices import VlanDevice from lnst.Devices import VtiDevice from lnst.Devices import VxlanDevice -from lnst.Tests import IcmpPing +from lnst.Tests import Ping from lnst.Tests.Netperf import Netperf, Netserver import signal @@ -40,7 +40,7 @@ class MyRecipe(BaseRecipe): self.matched.m1.eth0.up() self.matched.m2.eth0.ip_add(ipaddress("192.168.1.2/24")) self.matched.m2.eth0.up() - ping_job = self.matched.m1.run(IcmpPing(dst=self.matched.m2.eth0, + ping_job = self.matched.m1.run(Ping(dst=self.matched.m2.eth0, interval=0, iface=self.matched.m1.eth0)) diff --git a/recipes/examples/python_recipe_simple_netperf.py b/recipes/examples/python_recipe_simple_netperf.py index aab1c5b..1f9b400 100755 --- a/recipes/examples/python_recipe_simple_netperf.py +++ b/recipes/examples/python_recipe_simple_netperf.py @@ -1,4 +1,4 @@ -#!/bin/python2 +#!/bin/python3 import time diff --git a/recipes/smoke/generate-recipes.py b/recipes/smoke/generate-recipes.py index 24806ad..064792d 100755 --- a/recipes/smoke/generate-recipes.py +++ b/recipes/smoke/generate-recipes.py @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 # LNST Smoke Tests # Author: Ondrej Lichtner <olichtne(a)redhat.com> @@ -11,27 +11,27 @@ import os import shutil import re -import ConfigParser +import configparser import xml.dom.minidom def print_test_usage(): - print "" + print("") - print "To run these recipes, you need to have a pool prepared with at" - print "least two machines. Both of them must have at least two test" - print "interfaces connected to the same network segment." + print("To run these recipes, you need to have a pool prepared with at") + print("least two machines. Both of them must have at least two test") + print("interfaces connected to the same network segment.") - print "" + print("") - print " +-----------+ +--------+ +-----------+" - print " | |----------| |----------| |" - print " | Machine | | Switch | | Machine |" - print " | 1 |----------| |----------| 2 |" - print " | | +--------+ | |" - print " +-----------+ +-----------+" + print(" +-----------+ +--------+ +-----------+") + print(" | |----------| |----------| |") + print(" | Machine | | Switch | | Machine |") + print(" | 1 |----------| |----------| 2 |") + print(" | | +--------+ | |") + print(" +-----------+ +-----------+") - print "\nYou can execute the set using the following command:" - print " ./lnst-ctl -d run recipes/smoke/tests/" + print("\nYou can execute the set using the following command:") + print(" ./lnst-ctl -d run recipes/smoke/tests/") def replace_variables(recipe, name1, name2, variables): vars = dict(variables.items("defaults")) @@ -41,7 +41,7 @@ def replace_variables(recipe, name1, name2, variables): for name, val in section: vars[name] = val - for name, val in vars.iteritems(): + for name, val in vars.items(): recipe = recipe.replace("#%s#" % name, val) return recipe @@ -50,18 +50,18 @@ def main(): LIB = "../lib/" vars_filename = "%svariables.conf" % LIB - print "[LNST Smoke Tests]" - print "Creating '%s' directory for the recipes..." % DIR, + print("[LNST Smoke Tests]") + print("Creating '%s' directory for the recipes..." % DIR, end=' ') shutil.rmtree(DIR, ignore_errors=True) os.mkdir(DIR) os.chdir(DIR) - print "[DONE]" + print("[DONE]") sequences = "" for seq in os.listdir(LIB): if not re.match("task-.*", seq): continue - print "Found a task: %s%s" % (LIB, seq) + print("Found a task: %s%s" % (LIB, seq)) task_f = open("%s/%s" % (LIB, seq), 'r') task = task_f.read() @@ -70,12 +70,12 @@ def main(): conf_files = [LIB+i for i in os.listdir(LIB) if re.match("conf-.*", i)] for conf in conf_files: - print "Found configuration %s" % conf + print("Found configuration %s" % conf) template_file = open("%s/recipe-temp.xml" % LIB, 'r') template = template_file.read() - variables = ConfigParser.ConfigParser() + variables = configparser.ConfigParser() variables.read(vars_filename) for machine1 in conf_files: @@ -83,7 +83,7 @@ def main(): name1 = re.match(".*conf-(.*)\.xml", machine1).group(1) name2 = re.match(".*conf-(.*)\.xml", machine2).group(1) recipe_name = "recipe-%s-%s.xml" % (name1, name2) - print "Generating %s%s..." % (DIR, recipe_name), + print("Generating %s%s..." % (DIR, recipe_name), end=' ') m1_f = open(machine1, 'r') m1 = m1_f.read() @@ -102,7 +102,7 @@ def main(): recipe_file.write(recipe) recipe_file.close() - print "[DONE]" + print("[DONE]") print_test_usage() diff --git a/setup.py b/setup.py index 3a2c30e..2a6a491 100755 --- a/setup.py +++ b/setup.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 """ Install script for lnst @@ -34,7 +34,7 @@ def process_template(template_path, values): t = open(template_path, "r") f = open(file_path, "w") template = t.read() - for var, value in values.iteritems(): + for var, value in values.items(): template = template.replace("@%s@" % var, value) f.write(template) f.close() diff --git a/test_modules/Netperf.py b/test_modules/Netperf.py index 6950bb8..0c3c8d0 100644 --- a/test_modules/Netperf.py +++ b/test_modules/Netperf.py @@ -469,7 +469,7 @@ class Netperf(TestGeneric): rate_deviation = 2*res_data["std_deviation"] elif len(rates) == 1 and self._confidence is not None: result = results[0] - rate_deviation = rate * (result["confidence"][1] / 100) + rate_deviation = rate * (float(result["confidence"][1]) / 100) else: rate_deviation = 0.0 -- 2.21.0

2 1

[PATCH-next v3 0/5] Remaining phase 3 port
by csfakian＠redhat.com 30 May '19

30 May '19

From: Christos Sfakianakis <csfakian(a)redhat.com> Hi, this is v3 of the patch set. The changes with respect to v2 are: - lnst.Controller.Namespace: Indentation change of the 'else' clause in '__delattr__' in order to avoid no-actions when the attribute to be deleted does not satisfy the first 'if' condition. - Rewording of commit 2 (of 5) Christos Previous messages: Hi, this is v2 of the patch set. Changes include: - lnst.Controller.Namespace: add a method to catch attribute deletion - lnst.Recipes.ENRT.BaseENRTRecipe: change 'perf_msg_size' type to list and update previously ported recipes that relied on it - lnst.Devices.MacsecDevice: remove unused '*args' from intiator - drop the commits relating to lnst.Controller.Host and lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement - lnst.Recipes.ENRT.ShortLivedConnectionsRecipe: remove 'generate_flow_combinations' method after 'perf_msg_sizes' enablement. Removed 'vlan' from comment. Move 'genarate_ping_configurations' method to the bottom - lnst.Recipes.ENRT.SimpleMacsecRecipe: reorder methods for readability, remove 'cond', destroy and delete the macsec devices manually, check encryption against None value explicitly in 'generate_sub_config' Christos Previous message: Hi, this patch includes ports for simple_macsec and short_lived_connections of phase 3. Note that, due to current issue with iperf, ShortLivedConnectionsRecipe cannot be tested yet. In terms of SimpleMacsecRecipe, ip commands were chosen for the creation of the devices, as pyroute2 seems to have issue handling IFLA_MACSEC_ENCRYPT. Christos Christos Sfakianakis (5): lnst.Controller.Namespace: add __delattr__ lnst.Recipes.ENRT.BaseEnrtRecipe: change 'perf_msg_size' name and type lnst.Devices: add MacsecDevice lnst.Recipes.ENRT: add ShortLivedConnectionsRecipe lnst.Recipes.ENRT: add SimpleMacsecRecipe lnst/Controller/Namespace.py | 7 + lnst/Devices/MacsecDevice.py | 64 +++++ lnst/Devices/__init__.py | 4 +- lnst/Recipes/ENRT/BaseEnrtRecipe.py | 37 +-- lnst/Recipes/ENRT/IpsecEspAeadRecipe.py | 33 +-- lnst/Recipes/ENRT/IpsecEspAhCompRecipe.py | 33 +-- .../ENRT/ShortLivedConnectionsRecipe.py | 67 +++++ lnst/Recipes/ENRT/SimpleMacsecRecipe.py | 271 ++++++++++++++++++ 8 files changed, 465 insertions(+), 51 deletions(-) create mode 100644 lnst/Devices/MacsecDevice.py create mode 100644 lnst/Recipes/ENRT/ShortLivedConnectionsRecipe.py create mode 100644 lnst/Recipes/ENRT/SimpleMacsecRecipe.py -- 2.17.1

2 6

绩效考核043
by 陈铭潇 29 May '19

29 May '19

1 0

风险解读44
by 高艳 29 May '19

29 May '19

1 0

[PATCH-next v2 0/5] Remaining phase 3 port
by csfakian＠redhat.com 28 May '19

28 May '19

From: Christos Sfakianakis <csfakian(a)redhat.com> Hi, this is v2 of the patch set. Changes include: - lnst.Controller.Namespace: add a method to catch attribute deletion - lnst.Recipes.ENRT.BaseENRTRecipe: change 'perf_msg_size' type to list and update previously ported recipes that relied on it - lnst.Devices.MacsecDevice: remove unused '*args' from intiator - drop the commits relating to lnst.Controller.Host and lnst.RecipeCommon.Perf.Measurements.BaseFlowMeasurement - lnst.Recipes.ENRT.ShortLivedConnectionsRecipe: remove 'generate_flow_combinations' method after 'perf_msg_sizes' enablement. Removed 'vlan' from comment. Move 'genarate_ping_configurations' method to the bottom - lnst.Recipes.ENRT.SimpleMacsecRecipe: reorder methods for readability, remove 'cond', destroy and delete the macsec devices manually, check encryption against None value explicitly in 'generate_sub_config' Christos Previous message: Hi, this patch includes ports for simple_macsec and short_lived_connections of phase 3. Note that, due to current issue with iperf, ShortLivedConnectionsRecipe cannot be tested yet. In terms of SimpleMacsecRecipe, ip commands were chosen for the creation of the devices, as pyroute2 seems to have issue handling IFLA_MACSEC_ENCRYPT. Christos Christos Sfakianakis (5): lnst.Controller.Namespace: add __delattr__ lnst.Recipes.ENRT.BaseENRTRecipe: change 'perf_msg_size' name and type lnst.Devices: add MacsecDevice lnst.Recipes.ENRT: add ShortLivedConnectionsRecipe lnst.Recipes.ENRT: add SimpleMacsecRecipe lnst/Controller/Namespace.py | 7 + lnst/Devices/MacsecDevice.py | 64 +++++ lnst/Devices/__init__.py | 4 +- lnst/Recipes/ENRT/BaseEnrtRecipe.py | 37 +-- lnst/Recipes/ENRT/IpsecEspAeadRecipe.py | 33 +-- lnst/Recipes/ENRT/IpsecEspAhCompRecipe.py | 33 +-- .../ENRT/ShortLivedConnectionsRecipe.py | 67 +++++ lnst/Recipes/ENRT/SimpleMacsecRecipe.py | 271 ++++++++++++++++++ 8 files changed, 465 insertions(+), 51 deletions(-) create mode 100644 lnst/Devices/MacsecDevice.py create mode 100644 lnst/Recipes/ENRT/ShortLivedConnectionsRecipe.py create mode 100644 lnst/Recipes/ENRT/SimpleMacsecRecipe.py -- 2.17.1

3 8

税务应对21
by 吴永胜 27 May '19

27 May '19

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers May 2019