Main changes: ================================================ - Print function - PEP 3105[1] - Design of dictionaries, their iteration and respective built-in functions - PEP 469[2], PEP 3106[3] - Division operator - PEP 238[4] - Interrupts of system calls - PEP 475[5] - Socket.accept and interrupts - Docs Python3 Socket[6] - Universal newlines - PEP 278[7], PEP 3116[8], Docs Python3 Subprocess[9] - Binary data and strings - Docs Python3 porting[10] - Shebangs modified to environmental variable of python3 - Fixed library imports to reflect their respective changes in transition to python3 - multiprocessing - xmlrpc - urllib - thread - pickle
InterfaceManager.py ================================================ - Byte string and unicode/ascii string conflict, fixed by mostly using a byte string through the project
NetTestSlave.py ================================================ - signum 2 (SIGINT) - modified behavior since python 3.5 - Due to changes in interrupting system call, signal receiving and handling required changes to main slave loop - Structure of raising, catching and handling interrupt was changed, to reflect behavior of python 3.5 and higher - Method run is no longer ended by parameter "finished", but rather exception breaks the endless loop
SecureSocket.py ================================================ - Byte string and unicode/ascii string conflict, fixed by using bytes internally and sending messages encoded to ascii - Fixed multiple issues regarding changes to standard python functions, such as encoding to hex, ordinal value of characters, length of different types
NetTestCommand.py ================================================ - name_start_char is using unicode string, therefore each unicode has to be escaped twice
ExecCmd.py ================================================ - Popen universal newlines - Abstraction to be able to read newlines regardless of system used and its basic encoding
XmlParser.py ================================================ - etree.tostring function behavior changed to generate byte string by default - Unicode encoding is required, as specified by parameter
NmConfigDevice.py ================================================ - Conversion of bytes array to integer value was changed, ord no longer returns plain integer value - Need to convert bytes array to integer - int.from_bytes - Reverse conversion does not need to account for bytes array, therefore it is redundant to convert IP twice
ShellProcess.py ================================================ - Decoding file stream, to use unicode string instead of byte string, as built-in function read returns byte string by default in python3
Netperf.py ================================================ - Changed evaluation of results to account for changes made to base types in python3 - Dictionary comparision with integer by their respective lengths and values
Machine.py ================================================ - List comprehension functionality in python3 was changed - New function to parse queried addresses from slaves - modified code from next branch - get_ip_addr returns a dictionary or IP address objects - fixed to return only IP address portion by getting addr attribute from dictionary
================================================ [1] - PEP 3105 https://www.python.org/dev/peps/pep-3105/ [2] - PEP 469 https://www.python.org/dev/peps/pep-0469/ [3] - PEP 3106 https://www.python.org/dev/peps/pep-3106/ [4] - PEP 238 https://legacy.python.org/dev/peps/pep-0238/ [5] - PEP 475 https://www.python.org/dev/peps/pep-0475/ [6] - Docs Python3 Socket https://docs.python.org/3/library/socket.html#socket.socket.accept [7] - PEP 278 https://www.python.org/dev/peps/pep-0278/ [8] - PEP 3116 https://www.python.org/dev/peps/pep-3116/ [9] - Docs Python3 Subprocess https://docs.python.org/3/library/subprocess.html [10] - Docs Python3 Porting https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data
Signed-off-by: Jozef Urbanovsky jurbanov@redhat.com --- lnst-ctl | 88 +++++++++++----------- lnst-pool-wizard | 6 +- lnst-slave | 20 ++--- lnst/Common/Colours.py | 12 +-- lnst/Common/Config.py | 8 +- lnst/Common/ConnectionHandler.py | 2 +- lnst/Common/ExecCmd.py | 5 +- lnst/Common/LoggingHandler.py | 4 +- lnst/Common/Logs.py | 6 +- lnst/Common/NetTestCommand.py | 18 ++--- lnst/Common/NetUtils.py | 2 +- lnst/Common/Path.py | 5 +- lnst/Common/ProcessManager.py | 14 ++-- lnst/Common/ResourceCache.py | 4 +- lnst/Common/SecureSocket.py | 71 +++++++++-------- lnst/Common/ShellProcess.py | 2 +- lnst/Common/Utils.py | 4 +- lnst/Controller/Machine.py | 30 +++++--- lnst/Controller/NetTestController.py | 36 ++++----- lnst/Controller/NetTestResultSerializer.py | 8 +- lnst/Controller/RecipeParser.py | 16 ++-- lnst/Controller/SlavePool.py | 48 ++++++------ lnst/Controller/Task.py | 16 ++-- lnst/Controller/Wizard.py | 36 ++++----- lnst/Controller/XmlParser.py | 4 +- lnst/Controller/XmlProcessing.py | 12 +-- lnst/Controller/XmlTemplates.py | 6 +- lnst/Slave/InterfaceManager.py | 40 +++++----- lnst/Slave/NetConfigDevice.py | 10 +-- lnst/Slave/NetTestSlave.py | 66 ++++++++-------- lnst/Slave/NmConfigDevice.py | 5 +- setup.py | 4 +- test_modules/Multicast.py | 2 +- test_modules/Netperf.py | 2 +- test_modules/TRexClient.py | 4 +- 35 files changed, 317 insertions(+), 299 deletions(-)
diff --git a/lnst-ctl b/lnst-ctl index d61353d..ed13760 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test controller
@@ -34,41 +34,41 @@ def usage(retval=0): """ Print usage of this app """ - print "Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0] - print "" - print "ACTION = [ run | config_only | deconfigure | match_setup | " \ - "list_pools]" - print "" - print "OPTIONS" - print " -A, --override-alias name=value define top-level alias that " \ - "will override any other definitions in the recipe" - print " -a, --define-alias name=value define top-level alias" - print " -c, --config=FILE load additional config file" - print " -C, --config-override=FILE reset config defaults and load " \ - "the following config file" - print " -d, --debug emit debugging messages" - print " --dump-config dumps the join of all loaded " \ - "configuration files on stdout and exits" - print " -v, --verbose verbose version of list_pools " \ - "command" - print " -h, --help print this message" - print " -m, --no-colours disable coloured terminal output" - print " -o, --disable-pool-checks don't check the availability of " \ - "machines in the pool" - print " -p, --packet-capture capture and log all ongoing " \ - "network communication during the test" - print " --pools=NAME[,...] restricts which pools to use "\ - "for matching, value can be a comma separated list of values or" - print " --pools=PATH a single path to a pool directory" - print " -r, --reduce-sync reduces resource synchronization "\ - "for python tasks, see documentation" - print " -s, --xslt-url=URL URL to a XSLT document that will "\ + print("Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0]) + print("") + print("ACTION = [ run | config_only | deconfigure | match_setup | " \ + "list_pools]") + print("") + print("OPTIONS") + print(" -A, --override-alias name=value define top-level alias that " \ + "will override any other definitions in the recipe") + print(" -a, --define-alias name=value define top-level alias") + print(" -c, --config=FILE load additional config file") + print(" -C, --config-override=FILE reset config defaults and load " \ + "the following config file") + print(" -d, --debug emit debugging messages") + print(" --dump-config dumps the join of all loaded " \ + "configuration files on stdout and exits") + print(" -v, --verbose verbose version of list_pools " \ + "command") + print(" -h, --help print this message") + print(" -m, --no-colours disable coloured terminal output") + print(" -o, --disable-pool-checks don't check the availability of " \ + "machines in the pool") + print(" -p, --packet-capture capture and log all ongoing " \ + "network communication during the test") + print(" --pools=NAME[,...] restricts which pools to use "\ + "for matching, value can be a comma separated list of values or") + print(" --pools=PATH a single path to a pool directory") + print(" -r, --reduce-sync reduces resource synchronization "\ + "for python tasks, see documentation") + print(" -s, --xslt-url=URL URL to a XSLT document that will "\ "be used when transforming the xml result file, only useful "\ - "when -t is used as well" - print " -t, --html=FILE generate a formatted result html" - print " -u, --multi-match run each recipe with every "\ - "pool match possible" - print " -x, --result=FILE file to write xml_result" + "when -t is used as well") + print(" -t, --html=FILE generate a formatted result html") + print(" -u, --multi-match run each recipe with every "\ + "pool match possible") + print(" -x, --result=FILE file to write xml_result") sys.exit(retval)
def list_pools(restrict_pools, verbose): @@ -90,12 +90,12 @@ def list_pools(restrict_pools, verbose): out = "" # iterate over all pools sp_pools = sp.get_pools() - for pool_name, pool_content in sp_pools.iteritems(): + for pool_name, pool_content in sp_pools.items(): out += "Pool: %s (%s)\n" % (pool_name, pools[pool_name]) # verbose output if verbose: # iterate over all slave machine cfgs - for filename, pool in pool_content.iteritems(): + for filename, pool in pool_content.items(): # print in human-readable format out += 3*" " + filename + ".xml\n" out += 5*" " + "params:\n" @@ -113,9 +113,9 @@ def list_pools(restrict_pools, verbose): out += "\n" # print wihout newlines on the end of string if verbose: - print out[:-2] + print(out[:-2]) else: - print out[:-1] + print(out[:-1])
def store_alias(alias_def, aliases_dict): @@ -263,7 +263,7 @@ def main(): ] ) except getopt.GetoptError as err: - print str(err) + print(str(err)) usage(RETVAL_ERR)
lnst_config.controller_init() @@ -308,16 +308,16 @@ def main(): usage(RETVAL_PASS) elif opt in ("-c", "--config"): if not os.path.isfile(arg): - print "File '%s' doesn't exist!" % arg + print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else: lnst_config.load_config(arg) elif opt in ("-C", "--config-override"): if not os.path.isfile(arg): - print "File '%s' doesn't exist!" % arg + print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else: - print >> sys.stderr, "Reloading config defaults!" + print("Reloading config defaults!", file=sys.stderr) lnst_config.controller_init() lnst_config.load_config(arg) elif opt in ("-x", "--result"): @@ -351,7 +351,7 @@ def main(): lnst_config.set_option("environment", "xslt_url", xslt_url)
if dump_config: - print lnst_config.dump_config() + print(lnst_config.dump_config()) return RETVAL_PASS
if coloured_output: diff --git a/lnst-pool-wizard b/lnst-pool-wizard index ea2849b..f10635e 100755 --- a/lnst-pool-wizard +++ b/lnst-pool-wizard @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Machine pool wizard
@@ -19,7 +19,7 @@ RETVAL_ERR = 1
def help(retval=0): - print "Usage:\n"\ + print("Usage:\n"\ " lnst-pool-wizard [mode] [hostname[:port]]\n"\ "\n"\ "Modes:\n"\ @@ -34,7 +34,7 @@ def help(retval=0): " lnst-pool-wizard hostname1:1234 hostname2\n"\ " lnst-pool-wizard --noninteractive 192.168.122.2\n"\ " lnst-pool-wizard -n 192.168.122.2:8888 192.168.122.4\n"\ - " lnst-pool-wizard -p ".pool/" -n 192.168.1.1:8877 192.168.122.4" + " lnst-pool-wizard -p ".pool/" -n 192.168.1.1:8877 192.168.122.4") sys.exit(retval)
diff --git a/lnst-slave b/lnst-slave index ebb4cd3..0f77873 100755 --- a/lnst-slave +++ b/lnst-slave @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test slave
@@ -25,14 +25,14 @@ def usage(): """ Print usage of this app """ - print "Usage: %s [OPTION...]" % sys.argv[0] - print "" - print " -d, --debug emit debugging messages" - print " -h, --help print this message" - print " -e, --daemonize go to background after init" - print " -i, --pidfile file to write daemonized process pid" - print " -m, --no-colours disable coloured terminal output" - print " -p, --port xmlrpc port to listen on" + print("Usage: %s [OPTION...]" % sys.argv[0]) + print("") + print(" -d, --debug emit debugging messages") + print(" -h, --help print this message") + print(" -e, --daemonize go to background after init") + print(" -i, --pidfile file to write daemonized process pid") + print(" -m, --no-colours disable coloured terminal output") + print(" -p, --port xmlrpc port to listen on") sys.exit()
def main(): @@ -46,7 +46,7 @@ def main(): ["debug", "help", "daemonize", "pidfile=", "port=", "no-colours"] )[0] except getopt.GetoptError as err: - print str(err) + print(str(err)) usage() sys.exit()
diff --git a/lnst/Common/Colours.py b/lnst/Common/Colours.py index e57794e..a00e2d2 100644 --- a/lnst/Common/Colours.py +++ b/lnst/Common/Colours.py @@ -52,7 +52,7 @@ def name_to_fg_colour(name): """ Convert name to foreground colour code. Returns None if the colour name isn't supported. """
- if not COLOURS.has_key(name): + if name not in COLOURS: return None
return COLOURS[name] @@ -61,7 +61,7 @@ def name_to_bg_colour(name): """ Convert name to background color code. Returns None if the colour name isn't supported. """
- if not COLOURS.has_key(name): + if name not in COLOURS: return None
return COLOURS[name] + 10 @@ -136,7 +136,7 @@ def decorate_string(string, fg_colour=None, bg_colour=None, bold=False): raise Exception(msg) else: # Standard definition - if colour_def in COLOURS.keys(): + if colour_def in list(COLOURS.keys()): if fg: colour = name_to_fg_colour(colour_def) else: @@ -165,7 +165,7 @@ def strip_colours(text):
def get_preset_conf(preset_name): preset = PRESETS[preset_name] - return map(lambda s: "default" if s == None else str(s), preset) + return ["default" if s == None else str(s) for s in preset]
def load_presets_from_config(lnst_config): for preset_name in PRESETS: @@ -178,12 +178,12 @@ def load_presets_from_config(lnst_config):
if fg == "default": fg = None - elif not re.match(extended_re, fg) and fg not in COLOURS.keys(): + elif not re.match(extended_re, fg) and fg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % fg)
if bg == "default": bg = None - elif not re.match(extended_re, bg) and bg not in COLOURS.keys(): + elif not re.match(extended_re, bg) and bg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % bg)
PRESETS[preset_name] = [fg, bg, bool_it(bf)] diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py index d47f3e2..3aa2820 100644 --- a/lnst/Common/Config.py +++ b/lnst/Common/Config.py @@ -221,7 +221,7 @@ class Config(): raise ConfigError(msg)
res = {} - for opt_name, opt in self._options[section].items(): + for opt_name, opt in list(self._options[section].items()): res[opt_name] = opt["value"] return res
@@ -283,7 +283,7 @@ class Config(): '''Parse and load the config file''' exp_path = os.path.expanduser(path) abs_path = os.path.abspath(exp_path) - print >> sys.stderr, "Loading config file '%s'" % abs_path + print("Loading config file '%s'" % abs_path, file=sys.stderr) sections = self._parse_file(abs_path)
self.handleSections(sections, abs_path) @@ -338,7 +338,7 @@ class Config():
def get_pools(self): pools = {} - for pool_name, pool in self._options["pools"].items(): + for pool_name, pool in list(self._options["pools"].items()): pools[pool_name] = pool["value"] return pools
@@ -349,7 +349,7 @@ class Config(): return None
def _find_option_by_name(self, section, opt_name): - for option in section.itervalues(): + for option in section.values(): if option["name"] == opt_name: return option return None diff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py index 5d3170a..3856917 100644 --- a/lnst/Common/ConnectionHandler.py +++ b/lnst/Common/ConnectionHandler.py @@ -13,7 +13,7 @@ olichtne@redhat.com (Ondrej Lichtner)
import select import socket -from _multiprocessing import Connection +from multiprocessing.connection import Connection from pyroute2 import IPRSocket from lnst.Common.SecureSocket import SecureSocket, SecSocketException
diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py index 24715ae..5327f11 100644 --- a/lnst/Common/ExecCmd.py +++ b/lnst/Common/ExecCmd.py @@ -12,6 +12,7 @@ jpirko@redhat.com (Jiri Pirko)
import logging import subprocess +import sys
class ExecCmdFail(Exception): _cmd = None @@ -55,9 +56,11 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False, json=F cmd = cmd.rstrip(" ") logging.debug("Executing: "%s"" % cmd) subp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, close_fds=True) + stderr=subprocess.PIPE, close_fds=True, universal_newlines=True) (data_stdout, data_stderr) = subp.communicate()
+ #data_stdout = data_stdout.decode(sys.getfilesystemencoding()) + #data_stderr = data_stderr.decode(sys.getfilesystemencoding()) ''' When we should not die on error, do not print anything and let the caller to decide what to do. diff --git a/lnst/Common/LoggingHandler.py b/lnst/Common/LoggingHandler.py index a236922..6acc9bc 100644 --- a/lnst/Common/LoggingHandler.py +++ b/lnst/Common/LoggingHandler.py @@ -16,7 +16,7 @@ olichtne@redhat.com (Ondrej Lichtner)
import pickle import logging -import xmlrpclib +import xmlrpc.client from lnst.Common.ConnectionHandler import send_data
class LogBuffer(logging.Handler): @@ -39,7 +39,7 @@ class LogBuffer(logging.Handler): d['args'] = None d['exc_info'] = None s = pickle.dumps(d, 1) - return xmlrpclib.Binary(s) + return xmlrpc.client.Binary(s)
def add_buffer(self, buf): for i in buf: diff --git a/lnst/Common/Logs.py b/lnst/Common/Logs.py index 36e0580..77c0f14 100644 --- a/lnst/Common/Logs.py +++ b/lnst/Common/Logs.py @@ -106,7 +106,7 @@ class LoggingCtl: logger = logging.getLogger('') for i in list(logger.handlers): logger.removeHandler(i) - for key, logger in logging.Logger.manager.loggerDict.iteritems(): + for key, logger in list(logging.Logger.manager.loggerDict.items()): if type(logger) != type(logging.Logger): continue for i in list(logger.handlers): @@ -218,7 +218,7 @@ class LoggingCtl: logger = logging.getLogger() logger.addHandler(self.transmit_handler)
- for k in self.slaves.keys(): + for k in list(self.slaves.keys()): self.remove_slave(k)
def cancel_connection(self): @@ -230,7 +230,7 @@ class LoggingCtl: def disable_logging(self): self.cancel_connection()
- for s in self.slaves.keys(): + for s in list(self.slaves.keys()): self.remove_slave(s)
self.unset_recipe() diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index b894363..79b5893 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -325,10 +325,10 @@ class NetTestCommandGeneric(object): formatted_data = "" if res_data: max_key_len = 0 - for key in res_data.keys(): + for key in list(res_data.keys()): if len(key) > max_key_len: max_key_len = len(key) - for key, value in res_data.iteritems(): + for key, value in list(res_data.items()): if type(value) == dict: formatted_data += level*4*" " + str(key) + ":\n" formatted_data += self.format_res_data(value, level+1) @@ -347,13 +347,13 @@ class NetTestCommandGeneric(object): return formatted_data
def _check_res_data(self, res_data): - name_start_char = u":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF"\ - u"\u0370-\u037D\u037F-\u1FFF\u200C-\u200D"\ - u"\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF"\ - u"\uF900-\uFDCF\uFDF0-\uFFFD\U00010000-\U000EFFFF" - name_char = name_start_char + u"-.0-9\xB7\u0300-\u036F\u203F-\u2040" - name = u"[%s]([%s])*$" % (name_start_char, name_char) - char_data = u"[^<&]*" + name_start_char = ":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF"\ + "\u0370-\u037D\u037F-\u1FFF\u200C-\u200D"\ + "\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF"\ + "\uF900-\uFDCF\uFDF0-\uFFFD\U00010000-\U000EFFFF" + name_char = name_start_char + "-.0-9\xB7\u0300-\u036F\u203F-\u2040" + name = "[%s]([%s])*$" % (name_start_char, name_char) + char_data = "[^<&]*" if isinstance(res_data, dict): for key in res_data: if not re.match(name, key, re.UNICODE): diff --git a/lnst/Common/NetUtils.py b/lnst/Common/NetUtils.py index 9fd4732..42f377b 100644 --- a/lnst/Common/NetUtils.py +++ b/lnst/Common/NetUtils.py @@ -136,7 +136,7 @@ class MacPool(AddressPool): return bs
def _byte_string_to_addr(self, byte_string): - return ':'.join(map(lambda x: "%02x" % x, byte_string)) + return ':'.join(["%02x" % x for x in byte_string])
class IpPool(AddressPool): diff --git a/lnst/Common/Path.py b/lnst/Common/Path.py index fe7f709..e61b8eb 100644 --- a/lnst/Common/Path.py +++ b/lnst/Common/Path.py @@ -11,8 +11,9 @@ jtluka@redhat.com (Jan Tluka) """
import os -from urlparse import urljoin -from urllib2 import urlopen, HTTPError +from urllib.parse import urljoin +from urllib.request import urlopen +from urllib.error import HTTPError from tempfile import NamedTemporaryFile
def get_path_class(root, path): diff --git a/lnst/Common/ProcessManager.py b/lnst/Common/ProcessManager.py index 1995d15..564b49e 100644 --- a/lnst/Common/ProcessManager.py +++ b/lnst/Common/ProcessManager.py @@ -9,18 +9,18 @@ published by the Free Software Foundation; see COPYING for details. __autor__ = """ jzupka@redhat.com (Jiri Zupka) """ -import os, signal, thread, logging +import os, signal, _thread, logging
class ProcessManager: class SubProcess: def __init__(self, pid, handler): self.pid = pid - self.lock = thread.allocate_lock() + self.lock = _thread.allocate_lock() self.lock.acquire() self.handler = handler self.enabled = True self.status = None - thread.start_new_thread(self.waitpid, (self.pid, self.lock, + _thread.start_new_thread(self.waitpid, (self.pid, self.lock, self.handler))
def isAlive(self): @@ -45,12 +45,12 @@ class ProcessManager: logging.error(''.join(traceback.format_exception(type, value, tb))) os.kill(os.getpid(), signal.SIGTERM) else: - print "Process pid %s exit with exitcode %s" % (pid, status) + print("Process pid %s exit with exitcode %s" % (pid, status)) ProcessManager.lock.release() - thread.exit() + _thread.exit()
pids = {} - lock = thread.allocate_lock() + lock = _thread.allocate_lock() std_waitpid = None
@classmethod @@ -86,7 +86,7 @@ class ProcessManager: return pid, status
-lock = thread.allocate_lock() +lock = _thread.allocate_lock() lock.acquire() if os.waitpid != ProcessManager.waitpid: ProcessManager.std_waitpid = os.waitpid diff --git a/lnst/Common/ResourceCache.py b/lnst/Common/ResourceCache.py index 98558a2..092f974 100644 --- a/lnst/Common/ResourceCache.py +++ b/lnst/Common/ResourceCache.py @@ -65,7 +65,7 @@ class ResourceCache(object): header = "# hash " \ "last_used type name path\n" f.write(header) - for entry_hash, entry in self._entries.iteritems(): + for entry_hash, entry in list(self._entries.items()): values = (entry_hash, entry["last_used"], entry["type"], entry["name"], entry["path"]) line = "%s %d %s %s %s\n" % values @@ -137,7 +137,7 @@ class ResourceCache(object):
rm = [] now = time.time() - for entry_hash, entry in self._entries.iteritems(): + for entry_hash, entry in list(self._entries.items()): if entry["last_used"] <= (now - self._expiration_period): rm.append(entry_hash)
diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py index b856238..349c413 100644 --- a/lnst/Common/SecureSocket.py +++ b/lnst/Common/SecureSocket.py @@ -16,7 +16,7 @@ olichtne@redhat.com (Ondrej Lichtner) """
import os -import cPickle +import pickle import hashlib import hmac from lnst.Common.Utils import not_imported @@ -43,11 +43,11 @@ def bit_length(i): except AttributeError: return len(bin(i)) - 2
-DH_GROUP["q"] = (DH_GROUP["p"]-1)/2 -DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])/8 +DH_GROUP["q"] = (DH_GROUP["p"]-1)//2 +DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])//8 if bit_length(DH_GROUP["q"])%8: DH_GROUP["q_size"] += 1 -DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])/8 +DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])//8 if bit_length(DH_GROUP["p"])%8: DH_GROUP["p_size"] += 1
@@ -63,11 +63,11 @@ SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" "DE236D525F54759B65E372FCD68EF20FA7111F9E4AFF73", 16), "g": 2}
-SRP_GROUP["q"] = (SRP_GROUP["p"]-1)/2 -SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])/8 +SRP_GROUP["q"] = (SRP_GROUP["p"]-1)//2 +SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])//8 if bit_length(SRP_GROUP["q"])%8: SRP_GROUP["q_size"] += 1 -SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8 +SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])//8 if bit_length(SRP_GROUP["p"])%8: SRP_GROUP["p_size"] += 1
@@ -150,14 +150,14 @@ class SecureSocket(object): "seq_num": 0}
def send_msg(self, msg): - pickled_msg = cPickle.dumps(msg) + pickled_msg = pickle.dumps(msg) return self.send(pickled_msg)
def recv_msg(self): pickled_msg = self.recv() - if pickled_msg == "": + if pickled_msg == b"": raise SecSocketException("Disconnected") - msg = cPickle.loads(pickled_msg) + msg = pickle.loads(pickled_msg) return msg
def _add_mac_sign(self, data): @@ -165,22 +165,26 @@ class SecureSocket(object): return data cryptography_imports()
- msg = str(self._current_write_spec["seq_num"]) + str(len(data)) + data + msg = (bytes(str(self._current_write_spec["seq_num"]).encode('ascii')) + + bytes(str(len(data)).encode('ascii')) + + data) signature = hmac.new(self._current_write_spec["mac_key"], msg, hashlib.sha256) signed_msg = {"data": data, "signature": signature.digest()} - return cPickle.dumps(signed_msg) + return pickle.dumps(signed_msg)
def _del_mac_sign(self, signed_data): if not self._current_read_spec["mac_key"]: return signed_data cryptography_imports()
- signed_msg = cPickle.loads(signed_data) + signed_msg = pickle.loads(signed_data) data = signed_msg["data"] - msg = str(self._current_read_spec["seq_num"]) + str(len(data)) + data + msg = (bytes(str(self._current_read_spec["seq_num"]).encode('ascii')) + + bytes(str(len(data)).encode('ascii')) + + data)
signature = hmac.new(self._current_read_spec["mac_key"], msg, @@ -195,9 +199,9 @@ class SecureSocket(object): return data cryptography_imports()
- block_size = algorithms.AES.block_size/8 + block_size = algorithms.AES.block_size//8 pad_length = block_size - (len(data) % block_size) - pad_char = ("%02x" % pad_length).decode("hex") + pad_char = bytes([pad_length]) padding = pad_length * pad_char
padded_data = data+padding @@ -208,9 +212,9 @@ class SecureSocket(object): return data cryptography_imports()
- pad_length = int(data[-1].encode("hex"), 16) + pad_length = ord(data[-1]) for char in data[-pad_length]: - if int(char.encode("hex"), 16) != pad_length: + if ord(char) != pad_length: return None
return data[:-pad_length] @@ -220,7 +224,7 @@ class SecureSocket(object): return data cryptography_imports()
- iv = os.urandom(algorithms.AES.block_size/8) + iv = os.urandom(algorithms.AES.block_size//8) mode = modes.CBC(iv) key = self._current_write_spec["enc_key"] cipher = Cipher(algorithms.AES(key), mode, default_backend()) @@ -231,14 +235,14 @@ class SecureSocket(object): encrypted_msg = {"iv": iv, "enc_data": encrypted_data}
- return cPickle.dumps(encrypted_msg) + return pickle.dumps(encrypted_msg)
def _del_encrypt(self, data): if not self._current_read_spec["enc_key"]: return data cryptography_imports()
- encrypted_msg = cPickle.loads(data) + encrypted_msg = pickle.loads(data) encrypted_data = encrypted_msg["enc_data"]
iv = encrypted_msg["iv"] @@ -276,27 +280,28 @@ class SecureSocket(object): def send(self, data): protected_data = self._protect_data(data)
- transmit_data = str(len(protected_data)) + " " + protected_data + transmit_data = bytes(str(len(protected_data)).encode('ascii')) + b" " + protected_data
return self._socket.sendall(transmit_data)
def recv(self): - length = "" + length = b"" while True: c = self._socket.recv(1) - if c == ' ': - length = int(length) + + if c == b' ': + length = int(length.decode('ascii')) break - elif c == "": - return "" + elif c == b"": + return b"" else: length += c - data = ""
+ data = b"" while len(data) < length: c = self._socket.recv(length - len(data)) - if c == "": - return "" + if c == b"": + return b"" else: data += c
@@ -307,7 +312,7 @@ class SecureSocket(object):
def _handle_internal(self, orig_msg): try: - msg = cPickle.loads(orig_msg) + msg = pickle.loads(orig_msg) except: return orig_msg if "type" in msg and msg["type"] == "change_cipher_spec": @@ -348,7 +353,7 @@ class SecureSocket(object):
def p_SHA256(self, secret, seed, length): prev_a = seed - result = "" + result = b"" while len(result) < length: a = hmac.new(secret, msg=prev_a, digestmod=hashlib.sha256) prev_a = a.digest() @@ -372,7 +377,7 @@ class SecureSocket(object): raise SecSocketException("Socket without a role!") cryptography_imports()
- aes_keysize = max(algorithms.AES.key_sizes)/8 + aes_keysize = max(algorithms.AES.key_sizes)//8 mac_keysize = hashlib.sha256().block_size
prf_seq = self.PRF(self._master_secret, diff --git a/lnst/Common/ShellProcess.py b/lnst/Common/ShellProcess.py index a4928ed..45dedff 100644 --- a/lnst/Common/ShellProcess.py +++ b/lnst/Common/ShellProcess.py @@ -229,7 +229,7 @@ class ShellProcess: except: return data if r and (r[0][1] & select.POLLIN): - new_data = os.read(fd, 1024) + new_data = os.read(fd, 1024).decode() if not new_data: return data data += new_data diff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py index d6d6c57..4369278 100644 --- a/lnst/Common/Utils.py +++ b/lnst/Common/Utils.py @@ -197,7 +197,7 @@ def get_module_tools(module_path): return tools
def recursive_dict_update(original, update): - for key, value in update.iteritems(): + for key, value in list(update.items()): if isinstance(value, collections.Mapping): r = recursive_dict_update(original.get(key, {}), value) original[key] = r @@ -240,7 +240,7 @@ def list_to_dot(original_list, prefix="", key=""):
def dict_to_dot(original_dict, prefix=""): return_list = [] - for key, value in original_dict.iteritems(): + for key, value in list(original_dict.items()): if isinstance(value, collections.Mapping): sub_list = dict_to_dot(value, prefix + key + '.') return_list.extend(sub_list) diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py index b24eb65..2379505 100644 --- a/lnst/Controller/Machine.py +++ b/lnst/Controller/Machine.py @@ -17,7 +17,7 @@ import os import tempfile import signal from time import sleep -from xmlrpclib import Binary +from xmlrpc.client import Binary from functools import wraps from lnst.Common.Config import lnst_config from lnst.Common.NetUtils import normalize_hwaddr @@ -144,7 +144,7 @@ class Machine(object): del self._device_database[update_msg["if_index"]]
def dev_db_get_name(self, dev_name): - for if_index, dev in self._device_database.iteritems(): + for if_index, dev in self._device_database.items(): if dev.get_name() == dev_name: return dev return None @@ -308,7 +308,7 @@ class Machine(object): self._slave_desc = slave_desc
devices = self._rpc_call("get_devices") - for if_index, dev in devices.items(): + for if_index, dev in list(devices.items()): self._device_database[if_index] = Device(dev, self)
for iface in self._interfaces: @@ -552,8 +552,8 @@ class Machine(object): def sync_resources(self, required): self._rpc_call("clear_resource_table")
- for res_type, resources in required.iteritems(): - for res_name, res in resources.iteritems(): + for res_type, resources in required.items(): + for res_name, res in resources.items(): has_resource = self._rpc_call("has_resource", res["hash"]) if not has_resource: msg = "Transfering %s %s to machine %s" % \ @@ -889,7 +889,7 @@ class Interface(object): "network_label": self._network, "type": self._type, "addresses": self._addresses, - "slaves": self._slaves.keys(), + "slaves": list(self._slaves.keys()), "options": self._options, "slave_options": self._slave_options, "master": None, @@ -1422,14 +1422,24 @@ class Device(object):
@pre_call_decorate def get_ip_addrs(self, selector={}): - return [ip["addr"] - for ip in self._ip_addrs - if selector.items() <= ip.items()] + result = [] + for addr in self._ip_addrs: + match = True + for sel_item, value in selector.items(): + try: + if addr.get(sel_item, None) != value: + match = False + break + except: + match = False + if match: + result.append(addr) + return result
@pre_call_decorate def get_ip_addr(self, num, selector={}): ips = self.get_ip_addrs(selector) - return ips[num] + return ips[num].get('addr')
@pre_call_decorate def get_ifi_type(self): diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 860f727..66a5cc2 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -15,7 +15,7 @@ import logging import socket import os import re -import cPickle +import pickle import imp import copy import sys @@ -194,7 +194,7 @@ class NetTestController: recipe = self._recipe
machines = self._machines - for m_id in machines.keys(): + for m_id in list(machines.keys()): self._prepare_machine(m_id, resource_sync)
for machine_xml_data in recipe["machines"]: @@ -243,9 +243,9 @@ class NetTestController: logging.info("Pool match description:") if sp.is_setup_virtual(): logging.info(" Setup is using virtual machines.") - for m_id, m in sorted(match["machines"].iteritems()): + for m_id, m in sorted(match["machines"].items()): logging.info(" host "%s" uses "%s"" % (m_id, m["target"])) - for if_id, match in m["interfaces"].iteritems(): + for if_id, match in m["interfaces"].items(): pool_id = match["target"] logging.info(" interface "%s" matched to "%s"" %\ (if_id, pool_id)) @@ -521,7 +521,7 @@ class NetTestController: if self._machines == None: return
- for machine_id, machine in self._machines.iteritems(): + for machine_id, machine in self._machines.items(): if machine.is_configured(): try: machine.cleanup(deconfigure) @@ -536,7 +536,7 @@ class NetTestController:
# remove dynamically created bridges if deconfigure: - for bridge in self._network_bridges.itervalues(): + for bridge in self._network_bridges.values(): bridge.cleanup() self._network_bridges = {}
@@ -544,7 +544,7 @@ class NetTestController: #saves current virtual configuration to a file, after pickling it config_data = dict() machines = config_data["machines"] = {} - for m in self._machines.itervalues(): + for m in self._machines.values(): machine = machines[m.get_hostname()] = dict()
if m.get_libvirt_domain() != None: @@ -560,12 +560,12 @@ class NetTestController: machine["interfaces"].append(hwaddr)
config_data["bridges"] = bridges = [] - for bridge in self._network_bridges.itervalues(): + for bridge in self._network_bridges.values(): bridges.append(bridge.get_name())
with open("/tmp/.lnst_machine_conf", "wb") as f: os.fchmod(f.fileno(), 0o600) - pickled_data = cPickle.dump(config_data, f) + pickled_data = pickle.dump(config_data, f)
@classmethod def remove_saved_machine_config(cls): @@ -573,7 +573,7 @@ class NetTestController: cfg = None try: with open("/tmp/.lnst_machine_conf", "rb") as f: - cfg = cPickle.load(f) + cfg = pickle.load(f) except: logging.info("No previous configuration found.") return @@ -581,7 +581,7 @@ class NetTestController: if cfg: logging.info("Cleaning up leftover configuration from previous "\ "config_only run.") - for hostname, machine in cfg["machines"].iteritems(): + for hostname, machine in cfg["machines"].items(): port = lnst_config.get_option("environment", "rpcport") if test_tcp_connection(hostname, port): s = socket.create_connection((hostname, port)) @@ -689,7 +689,7 @@ class NetTestController: overall_res["err_msg"] = "Command exception raised." break
- for machine in self._machines.itervalues(): + for machine in self._machines.values(): machine.restore_system_config()
# task failed, check if we should quit_on_fail @@ -790,13 +790,13 @@ class NetTestController:
def _start_packet_capture(self): logging.info("Starting packet capture") - for machine_id, machine in self._machines.iteritems(): + for machine_id, machine in self._machines.items(): capture_files = machine.start_packet_capture() self._remote_capture_files[machine_id] = capture_files
def _stop_packet_capture(self): logging.info("Stopping packet capture") - for machine_id, machine in self._machines.iteritems(): + for machine_id, machine in self._machines.items(): machine.stop_packet_capture()
# TODO: Move this function to logging @@ -804,7 +804,7 @@ class NetTestController: logging_root = self._log_ctl.get_recipe_log_path() logging_root = os.path.abspath(logging_root) logging.info("Retrieving capture files from slaves") - for machine_id, machine in self._machines.iteritems(): + for machine_id, machine in self._machines.items(): slave_logging_dir = os.path.join(logging_root, machine_id + "/") try: os.mkdir(slave_logging_dir) @@ -815,7 +815,7 @@ class NetTestController: raise NetTestError(msg)
capture_files = self._remote_capture_files[machine_id] - for if_id, remote_path in capture_files.iteritems(): + for if_id, remote_path in capture_files.items(): filename = "%s.pcap" % if_id local_path = os.path.join(slave_logging_dir, filename) machine.copy_file_from_machine(remote_path, local_path) @@ -896,11 +896,11 @@ class MessageDispatcher(ConnectionHandler): def wait_for_result(self, machine_id): wait = True while wait: - connected_slaves = self._connection_mapping.keys() + connected_slaves = list(self._connection_mapping.keys())
messages = self.check_connections()
- remaining_slaves = self._connection_mapping.keys() + remaining_slaves = list(self._connection_mapping.keys())
for msg in messages: if msg[1]["type"] == "result" and msg[0] == machine_id: diff --git a/lnst/Controller/NetTestResultSerializer.py b/lnst/Controller/NetTestResultSerializer.py index e4fe086..7439c64 100644 --- a/lnst/Controller/NetTestResultSerializer.py +++ b/lnst/Controller/NetTestResultSerializer.py @@ -94,10 +94,10 @@ class NetTestResultSerializer: "Setup is using virtual machines.", ""))
- for m_id, m in sorted(match["machines"].iteritems()): + for m_id, m in sorted(match["machines"].items()): output_pairs.append((4*" " + "host "%s" uses "%s"" %\ (m_id, m["target"]), "")) - for if_id, pool_if in m["interfaces"].iteritems(): + for if_id, pool_if in m["interfaces"].items(): pool_id = pool_if["target"] if "driver" in pool_if: driver = pool_if["driver"] @@ -227,12 +227,12 @@ class NetTestResultSerializer: else: match_el.setAttribute("virtual", "false")
- for m_id, m in match["machines"].iteritems(): + for m_id, m in match["machines"].items(): m_el = doc.createElement("m_match") m_el.setAttribute("host_id", str(m_id)) m_el.setAttribute("pool_id", str(m["target"]))
- for if_id, pool_id in m["interfaces"].iteritems(): + for if_id, pool_id in m["interfaces"].items(): if_el = doc.createElement("if_match") if_el.setAttribute("if_id", str(if_id)) if_el.setAttribute("pool_if_id", str(pool_id)) diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py index 742c144..d7f70ce 100644 --- a/lnst/Controller/RecipeParser.py +++ b/lnst/Controller/RecipeParser.py @@ -386,16 +386,16 @@ class RecipeParser(XmlParser): distribution = False valid_distributions = ["normal", "uniform", "pareto", "paretonormal"] for opt in options: - if "time" in opt.values(): + if "time" in list(opt.values()): valid = True - elif "distribution" in opt.values(): + elif "distribution" in list(opt.values()): if opt["value"] not in valid_distributions: raise RecipeError("netem: invalid distribution type", netem_tag) else: distribution = True - elif "jitter" in opt.values(): + elif "jitter" in list(opt.values()): jitter = True - elif "correlation" in opt.values(): + elif "correlation" in list(opt.values()): correlation = True if not jitter: if correlation or distribution: @@ -404,22 +404,22 @@ class RecipeParser(XmlParser): raise RecipeError("netem: time option is mandatory for <delay>", netem_tag) elif netem_op == "loss": for opt in options: - if "percent" in opt.values(): + if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <loss>", netem_tag) elif netem_op == "duplication": for opt in options: - if "percent" in opt.values(): + if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <duplication>", netem_tag) elif netem_op == "corrupt": for opt in options: - if "percent" in opt.values(): + if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <corrupt>", netem_tag) elif netem_op == "reordering": for opt in options: - if "percent" in opt.values(): + if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <reordering>", netem_tag)
diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py index 5069f50..8166816 100644 --- a/lnst/Controller/SlavePool.py +++ b/lnst/Controller/SlavePool.py @@ -49,7 +49,7 @@ class SlavePool: self._mreqs = None
logging.info("Checking machine pool availability.") - for pool_name, pool_dir in pools.items(): + for pool_name, pool_dir in list(pools.items()): self._pools[pool_name] = {} self.add_dir(pool_name, pool_dir) if len(self._pools[pool_name]) == 0: @@ -85,13 +85,13 @@ class SlavePool: dir_path))
max_len = 0 - for m_id in pool.keys(): + for m_id in list(pool.keys()): if len(m_id) > max_len: max_len = len(m_id)
if self._pool_checks: check_sockets = {} - for m_id, m in sorted(pool.iteritems()): + for m_id, m in sorted(pool.items()): hostname = m["params"]["hostname"] if "rpc_port" in m["params"]: port = m["params"]["rpc_port"] @@ -123,7 +123,7 @@ class SlavePool: check_sockets[s] = m_id
while len(check_sockets) > 0: - rl, wl, el = select.select([], check_sockets.keys(), []) + rl, wl, el = select.select([], list(check_sockets.keys()), []) for s in wl: err = s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) m_id = check_sockets[s] @@ -137,7 +137,7 @@ class SlavePool: s.close() del check_sockets[s] else: - for m_id in pool.keys(): + for m_id in list(pool.keys()): pool[m_id]["available"] = True
for m_id in sorted(list(pool.keys())): @@ -178,7 +178,7 @@ class SlavePool:
# Check if there isn't any machine with the same # hostname or libvirt_domain already in the pool - for pm_id, m in pool.iteritems(): + for pm_id, m in pool.items(): pm = m["params"] rm = machine_spec["params"] if pm["hostname"] == rm["hostname"]: @@ -228,7 +228,7 @@ class SlavePool: raise SlaveMachineError(msg, iface)
if_hwaddr = iface_spec["params"]["hwaddr"] - hwaddr_dups = [ k for k, v in machine_spec["interfaces"].iteritems()\ + hwaddr_dups = [ k for k, v in machine_spec["interfaces"].items()\ if v["params"]["hwaddr"] == if_hwaddr ] if len(hwaddr_dups) > 0: msg = "Duplicate MAC address %s for interface '%s' and '%s'."\ @@ -348,7 +348,7 @@ class SlavePool:
used = [] if_map = self._map["machines"][tm_id]["interfaces"] - for t_if, p_if in if_map.iteritems(): + for t_if, p_if in if_map.items(): pool_id = p_if["target"] used.append(pool_id) if_data = pm["interfaces"][pool_id] @@ -356,12 +356,12 @@ class SlavePool: iface = machine.new_static_interface(t_if, "eth") iface.set_hwaddr(if_data["params"]["hwaddr"])
- for t_net, p_net in self._map["networks"].iteritems(): + for t_net, p_net in self._map["networks"].items(): if pm["interfaces"][pool_id]["network"] == p_net: iface.set_network(t_net) break
- for if_id, if_data in pm["interfaces"].iteritems(): + for if_id, if_data in pm["interfaces"].items(): if if_id not in used: iface = machine.new_unused_interface("eth") iface.set_hwaddr(if_data["params"]["hwaddr"]) @@ -384,13 +384,13 @@ class SlavePool: pm["security"])
# make all the existing unused - for if_id, if_data in pm["interfaces"].iteritems(): + for if_id, if_data in pm["interfaces"].items(): iface = machine.new_unused_interface("eth") iface.set_hwaddr(if_data["params"]["hwaddr"]) iface.set_network(None)
# add all the other devices - for if_id, if_data in tm["interfaces"].iteritems(): + for if_id, if_data in tm["interfaces"].items(): iface = machine.new_virtual_interface(if_id, "eth") iface.set_network(if_data["network"]) if "hwaddr" in if_data["params"]: @@ -425,10 +425,10 @@ class SetupMapper(object): def set_virtual(self, virt_value): self._virtual_matching = virt_value
- for m_id, m in self._mreqs.iteritems(): - for if_id, interface in m["interfaces"].iteritems(): + for m_id, m in self._mreqs.items(): + for if_id, interface in m["interfaces"].items(): if "params" in interface: - for name, val in interface["params"].iteritems(): + for name, val in interface["params"].items(): if name not in ["hwaddr", "driver"]: msg = "Dynamically created interfaces "\ "only support the 'hwaddr' and 'driver' "\ @@ -443,7 +443,7 @@ class SetupMapper(object): def reset_match_state(self): self._net_label_mapping = {} self._machine_stack = [] - self._unmatched_req_machines = sorted(self._mreqs.keys(), reverse=True) + self._unmatched_req_machines = sorted(list(self._mreqs.keys()), reverse=True)
self._pool_stack = list(self._pools.keys()) if len(self._pool_stack) > 0: @@ -451,7 +451,7 @@ class SetupMapper(object): self._pool = self._pools[self._pool_name]
self._unmatched_pool_machines = [] - for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True): + for p_id, p_machine in sorted(iter(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id) @@ -492,7 +492,7 @@ class SetupMapper(object): #map compatible pool machine stack_top["current_match"] = pool_m_id stack_top["unmatched_pool_ifs"] = \ - sorted(self._pool[pool_m_id]["interfaces"].keys(), + sorted(list(self._pool[pool_m_id]["interfaces"].keys()), reverse=True) self._unmatched_pool_machines.remove(pool_m_id) break @@ -516,7 +516,7 @@ class SetupMapper(object): self._pool_name)
self._unmatched_pool_machines = [] - for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True): + for p_id, p_machine in sorted(iter(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id) @@ -591,7 +591,7 @@ class SetupMapper(object): machine_match["if_stack"] = []
machine = self._mreqs[machine_match["m_id"]] - machine_match["unmatched_ifs"] = sorted(machine["interfaces"].keys(), + machine_match["unmatched_ifs"] = sorted(list(machine["interfaces"].keys()), reverse=True) machine_match["unmatched_pool_ifs"] = []
@@ -621,7 +621,7 @@ class SetupMapper(object): def _check_machine_compatibility(self, req_id, pool_id): req_machine = self._mreqs[req_id] pool_machine = self._pool[pool_id] - for param, value in req_machine["params"].iteritems(): + for param, value in req_machine["params"].items(): # skip empty parameters if len(value) == 0: continue @@ -632,14 +632,14 @@ class SetupMapper(object):
def _check_interface_compatibility(self, req_if, pool_if): label_mapping = self._net_label_mapping - for req_label, mapping in label_mapping.iteritems(): + for req_label, mapping in label_mapping.items(): if req_label == req_if["network"] and\ mapping[0] != pool_if["network"]: return False if mapping[0] == pool_if["network"] and\ req_label != req_if["network"]: return False - for param, value in req_if["params"].iteritems(): + for param, value in req_if["params"].items(): # skip empty parameters if len(value) == 0: continue @@ -652,7 +652,7 @@ class SetupMapper(object): mapping = {"machines": {}, "networks": {}, "virtual": False, "pool_name": self._pool_name}
- for req_label, label_map in self._net_label_mapping.iteritems(): + for req_label, label_map in self._net_label_mapping.items(): mapping["networks"][req_label] = label_map[0]
for machine in self._machine_stack: diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py index bc7670f..e2b98d0 100644 --- a/lnst/Controller/Task.py +++ b/lnst/Controller/Task.py @@ -45,7 +45,7 @@ class ControllerAPI(object): self._perf_repo_api = PerfRepoAPI()
self._hosts = {} - for host_id, host in hosts.iteritems(): + for host_id, host in hosts.items(): self._hosts[host_id] = HostAPI(self, host_id, host)
def _run_command(self, command): @@ -161,20 +161,20 @@ class ControllerAPI(object): def get_configuration(self): machines = self._ctl._machines configuration = {} - for m_id, m in machines.items(): + for m_id, m in list(machines.items()): configuration["machine_"+m_id] = m.get_configuration() return configuration
def get_mapping(self): match = self._ctl.get_pool_match() mapping = [] - for m_id, m in match["machines"].iteritems(): + for m_id, m in match["machines"].items(): machine = {} machine["id"] = m_id machine["pool_id"] = m["target"] machine["hostname"] = m["hostname"] machine["interface"] = [] - for i_id, i in m["interfaces"].iteritems(): + for i_id, i in m["interfaces"].items(): interface = {} interface["id"] = i_id interface["pool_id"] = i["target"] @@ -253,7 +253,7 @@ class HostAPI(object): bg_id = None cmd["netns"] = None
- for arg, argval in kwargs.iteritems(): + for arg, argval in kwargs.items(): if arg == "bg" and argval == True: self._bg_id_seq += 1 cmd["bg_id"] = bg_id = self._bg_id_seq @@ -864,7 +864,7 @@ class ModuleAPI(object): self._name = module_name
self._opts = {} - for opt, val in options.iteritems(): + for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val: @@ -877,7 +877,7 @@ class ModuleAPI(object):
def set_options(self, options): self._opts = {} - for opt, val in options.iteritems(): + for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val: @@ -886,7 +886,7 @@ class ModuleAPI(object): self._opts[opt].append({"value": str(val)})
def update_options(self, options): - for opt, val in options.iteritems(): + for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val: diff --git a/lnst/Controller/Wizard.py b/lnst/Controller/Wizard.py index dc82c21..85fb88b 100644 --- a/lnst/Controller/Wizard.py +++ b/lnst/Controller/Wizard.py @@ -106,7 +106,7 @@ class Wizard:
rv = self._check_path(pool_dir) if rv == PATH_IS_DIR_ACCESSIBLE: - print("Pool directory set to '%s'" % pool_dir) + print(("Pool directory set to '%s'" % pool_dir)) elif rv == PATH_DOES_NOT_EXIST: sys.stderr.write("Path '%s' does not exist\n" % pool_dir) pool_dir = self._create_dir(pool_dir) @@ -124,7 +124,7 @@ class Wizard: return
for host in hostlist: - print("Processing host '%s'" % host) + print(("Processing host '%s'" % host)) hostname, port = self._parse_host(host) if hostname == -1: continue @@ -222,7 +222,7 @@ class Wizard: """ while True: if pool_dir is None: - pool_dir = raw_input("Enter path to a pool directory " + pool_dir = input("Enter path to a pool directory " "(default: '%s'): " % DefaultPoolDir) if pool_dir == "": pool_dir = DefaultPoolDir @@ -230,7 +230,7 @@ class Wizard: pool_dir = os.path.expanduser(pool_dir) rv = self._check_path(pool_dir) if rv == PATH_IS_DIR_ACCESSIBLE: - print("Pool directory set to '%s'" % pool_dir) + print(("Pool directory set to '%s'" % pool_dir)) return pool_dir elif rv == PATH_DOES_NOT_EXIST: sys.stderr.write("Path '%s' does not exist\n" @@ -270,7 +270,7 @@ class Wizard: """ try: mkdir_p(pool_dir) - print("Dir '%s' has been created" % pool_dir) + print(("Dir '%s' has been created" % pool_dir)) return pool_dir except: sys.stderr.write("Failed creating dir\n") @@ -316,7 +316,7 @@ class Wizard: if mode == "interactive": msg = "Do you want to add interface '%s' (%s) to the "\ "recipe? [Y/n]: " % (iface["name"], iface["hwaddr"]) - answer = raw_input(msg) + answer = input(msg) if mode == "noninteractive" or answer.lower() == "y"\ or answer == "": interfaces_added += 1 @@ -359,7 +359,7 @@ class Wizard: pubkey_el.appendChild(pubkey_text)
if self._write_to_file(pool_dir, filename, doc): - print("File '%s/%s' successfuly created." % (pool_dir, filename)) + print(("File '%s/%s' successfuly created." % (pool_dir, filename))) else: sys.stderr.write("File '%s/%s' could not be opened " "or data written.\n" % (pool_dir, filename)) @@ -431,7 +431,7 @@ class Wizard: """ Queries user for adding next machine @return True if user wants to add another machine, False otherwise """ - answer = raw_input("Do you want to add another machine? [Y/n]: ") + answer = input("Do you want to add another machine? [Y/n]: ") if answer.lower() == "y" or answer == "": return True else: @@ -441,7 +441,7 @@ class Wizard: """ Queries user for creating specified directory @return True if user wants to create the directory, False otherwise """ - answer = raw_input("Create dir '%s'? [Y/n]: " % pool_dir) + answer = input("Create dir '%s'? [Y/n]: " % pool_dir) if answer.lower() == 'y' or answer == "": return True else: @@ -452,7 +452,7 @@ class Wizard: @hostname Hostname of the machine which is used as default filename @return Name of the file with .xml extension """ - output_file = raw_input("Enter the name of the output .xml file " + output_file = input("Enter the name of the output .xml file " "(without .xml, default is '%s.xml'): " % hostname) if output_file == "": @@ -465,7 +465,7 @@ class Wizard: @return Valid (is translatable to an IP address) hostname """ while True: - hostname = raw_input("Enter hostname: ") + hostname = input("Enter hostname: ") if hostname == "": sys.stderr.write("No hostname entered\n") continue @@ -485,7 +485,7 @@ class Wizard: string representing hostname of the host """ while True: - libvirt_domain = raw_input("Enter libvirt domain " + libvirt_domain = input("Enter libvirt domain " "of virtual host: ") if libvirt_domain == "": sys.stderr.write("No domain entered\n") @@ -524,7 +524,7 @@ class Wizard: @return Integer representing port """ while True: - port = raw_input("Enter port (default: %d): " % DefaultRPCPort) + port = input("Enter port (default: %d): " % DefaultRPCPort) if port == "": return DefaultRPCPort else: @@ -539,7 +539,7 @@ class Wizard: @return Dictionary with the security parameters """ while True: - auth_type = raw_input("Enter authentication type (default: none): ") + auth_type = input("Enter authentication type (default: none): ") if auth_type == "": auth_type = "none" elif auth_type not in ["none", "no-auth", "password", @@ -555,7 +555,7 @@ class Wizard: return {"auth_type": "ssh"} elif auth_type == "password": while True: - password = raw_input("Enter password: ") + password = input("Enter password: ") if password == "": sys.stderr.write("Invalid password.") continue @@ -564,19 +564,19 @@ class Wizard: "auth_passwd": password} elif auth_type == "pubkey": while True: - identity = raw_input("Enter identity: ") + identity = input("Enter identity: ") if identity == "": sys.stderr.write("Invalid identity.") continue break while True: - privkey = raw_input("Enter path to Ctl private key: ") + privkey = input("Enter path to Ctl private key: ") if privkey == "" or not os.path.isfile(privkey): sys.stderr.write("Invalid path to private key.") continue break while True: - srv_pubkey_path = raw_input("Enter path to Slave public key: ") + srv_pubkey_path = input("Enter path to Slave public key: ") if srv_pubkey_path == "" or not os.path.isfile(srv_pubkey_path): sys.stderr.write("Invalid path to public key.") continue diff --git a/lnst/Controller/XmlParser.py b/lnst/Controller/XmlParser.py index 355b5e8..cbe3569 100644 --- a/lnst/Controller/XmlParser.py +++ b/lnst/Controller/XmlParser.py @@ -15,7 +15,7 @@ import re import sys import copy from lxml import etree -from urllib2 import urlopen +from urllib.request import urlopen from lnst.Common.Config import lnst_config from lnst.Controller.XmlTemplates import XmlTemplates from lnst.Controller.XmlProcessing import XmlProcessingError @@ -124,7 +124,7 @@ class XmlParser(object): return self._template_proc.expand_functions(text)
def _get_content(self, element): - text = etree.tostring(element, method="text").strip() + text = etree.tostring(element, method="text", encoding="unicode").strip() return self._template_proc.expand_functions(text)
def _expand_xinclude(self, elem, base_url=""): diff --git a/lnst/Controller/XmlProcessing.py b/lnst/Controller/XmlProcessing.py index b80c3a3..8e2bb00 100644 --- a/lnst/Controller/XmlProcessing.py +++ b/lnst/Controller/XmlProcessing.py @@ -75,8 +75,8 @@ class XmlDataIterator: def __iter__(self): return self
- def next(self): - n = self._iterator.next() + def __next__(self): + n = next(self._iterator)
# For normal iterators if type(n) == XmlTemplateString: @@ -165,20 +165,20 @@ class XmlData(dict): return XmlDataIterator(it)
def iteritems(self): - it = super(XmlData, self).iteritems() + it = iter(super(XmlData, self).items()) return XmlDataIterator(it)
def iterkeys(self): - it = super(XmlData, self).iterkeys() + it = iter(super(XmlData, self).keys()) return XmlDataIterator(it)
def itervalues(self): - it = super(XmlData, self).itervalues() + it = iter(super(XmlData, self).values()) return XmlDataIterator(it)
def to_dict(self): new_dict = dict() - for key, value in self.iteritems(): + for key, value in self.items(): if isinstance(value, XmlData): new_val = value.to_dict() elif isinstance(value, XmlCollection): diff --git a/lnst/Controller/XmlTemplates.py b/lnst/Controller/XmlTemplates.py index b1bf9a1..4068e85 100644 --- a/lnst/Controller/XmlTemplates.py +++ b/lnst/Controller/XmlTemplates.py @@ -239,7 +239,7 @@ class XmlTemplates: """ defs = {} for level in self._definitions: - for name, val in level.iteritems(): + for name, val in level.items(): defs[name] = val
return defs @@ -255,7 +255,7 @@ class XmlTemplates: def set_aliases(self, defined, overriden): """ Set aliases defined or overriden from CLI """
- for name, value in defined.iteritems(): + for name, value in defined.items(): self.define_alias(name, value)
self._overriden_aliases = overriden @@ -325,7 +325,7 @@ class XmlTemplates: if element.tail != None: element.tail = self.expand_aliases(element.tail)
- for name, value in element.attrib.iteritems(): + for name, value in element.attrib.items(): element.set(name, self.expand_aliases(value))
if element.tag == "define": diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py index a96ddd1..47c1e2a 100644 --- a/lnst/Slave/InterfaceManager.py +++ b/lnst/Slave/InterfaceManager.py @@ -79,7 +79,7 @@ class InterfaceManager(object): self._id_mapping = {}
def get_id_by_if_index(self, if_index): - for if_id, index in self._id_mapping.iteritems(): + for if_id, index in self._id_mapping.items(): if if_index == index: return if_id return None @@ -97,12 +97,12 @@ class InterfaceManager(object): return self._nl_socket
def rescan_devices(self): - devices_to_remove = self._devices.keys() + devices_to_remove = list(self._devices.keys()) devs = scan_netdevs() for dev in devs: if dev['index'] not in self._devices: device = None - for if_id, d in self._tmp_mapping.items(): + for if_id, d in list(self._tmp_mapping.items()): d_cfg = d.get_conf_dict() if d_cfg["name"] == dev["name"]: device = d @@ -134,7 +134,7 @@ class InterfaceManager(object): del self._devices[i]
self._dl_manager.rescan_ports() - for device in self._devices.values(): + for device in list(self._devices.values()): dl_port = self._dl_manager.get_port(device.get_name()) device.set_devlink(dl_port)
@@ -143,7 +143,7 @@ class InterfaceManager(object): self._handle_netlink_msg(msg)
self._dl_manager.rescan_ports() - for device in self._devices.values(): + for device in list(self._devices.values()): dl_port = self._dl_manager.get_port(device.get_name()) device.set_devlink(dl_port)
@@ -152,14 +152,14 @@ class InterfaceManager(object): if msg['index'] in self._devices: update_msg = self._devices[msg['index']].update_netlink(msg) if update_msg != None: - for if_id, if_index in self._id_mapping.iteritems(): + for if_id, if_index in self._id_mapping.items(): if if_index == msg['index']: update_msg["if_id"] = if_id break self._server_handler.send_data_to_ctl(update_msg) elif msg['header']['type'] == RTM_NEWLINK: dev = None - for if_id, d in self._tmp_mapping.items(): + for if_id, d in list(self._tmp_mapping.items()): d_cfg = d.get_conf_dict() if d_cfg["name"] == msg.get_attr("IFLA_IFNAME"): dev = d @@ -172,7 +172,7 @@ class InterfaceManager(object): self._devices[msg['index']] = dev
if update_msg != None: - for if_id, if_index in self._id_mapping.iteritems(): + for if_id, if_index in self._id_mapping.items(): if if_index == msg['index']: update_msg["if_id"] = if_id break @@ -202,7 +202,7 @@ class InterfaceManager(object):
def get_mapped_devices(self): ret = {} - for if_id, if_index in self._id_mapping.iteritems(): + for if_id, if_index in self._id_mapping.items(): ret[if_id] = self._devices[if_index] for if_id in self._tmp_mapping: ret[if_id] = self._tmp_mapping[if_id] @@ -215,20 +215,20 @@ class InterfaceManager(object): return None
def get_devices(self): - return self._devices.values() + return list(self._devices.values())
def get_device_by_hwaddr(self, hwaddr): - for dev in self._devices.values(): + for dev in list(self._devices.values()): if dev.get_hwaddr() == hwaddr: return dev return None
def get_device_by_params(self, params): matched = None - for dev in self._devices.values(): + for dev in list(self._devices.values()): matched = dev dev_data = dev.get_if_data() - for key, value in params.iteritems(): + for key, value in params.items(): if key not in dev_data or dev_data[key] != value: matched = None break @@ -239,7 +239,7 @@ class InterfaceManager(object): return matched
def deconfigure_all(self): - for dev in self._devices.itervalues(): + for dev in self._devices.values(): dev.clear_configuration()
def create_device_from_config(self, if_id, config): @@ -288,10 +288,10 @@ class InterfaceManager(object):
def _is_name_used(self, name): self.rescan_devices() - for device in self._devices.itervalues(): + for device in self._devices.values(): if name == device.get_name(): return True - for device in self._tmp_mapping.itervalues(): + for device in self._tmp_mapping.values(): if name == device.get_name(): return True return False @@ -487,7 +487,7 @@ class Device(object): def find_addrs(self, addr_spec): ret = [] for addr in self._ip_addrs: - if addr_spec.items() <= addr.items(): + if list(addr_spec.items()) <= list(addr.items()): ret.append(addr) return ret
@@ -703,7 +703,7 @@ class Device(object): if (line.split()[0] == 'vf'): break if (line.split()[0] == "RX:"): - rx_stats = map(int, lines.next().split()) + rx_stats = list(map(int, next(lines).split())) stats.update({"rx_bytes" : rx_stats[0], "rx_packets": rx_stats[1], "rx_errors" : rx_stats[2], @@ -711,7 +711,7 @@ class Device(object): "rx_overrun": rx_stats[4], "rx_mcast" : rx_stats[5]}) if (line.split()[0] == "TX:"): - tx_stats = map(int, lines.next().split()) + tx_stats = list(map(int, next(lines).split())) stats.update({"tx_bytes" : tx_stats[0], "tx_packets": tx_stats[1], "tx_errors" : tx_stats[2], @@ -742,7 +742,7 @@ class Device(object): stats_data[i] = stats_data[i].replace("K", "000") stats_data[i] = stats_data[i].replace("M", "000000")
- stats_data = map(int, stats_data) + stats_data = list(map(int, stats_data)) stats["rx_packets"] = stats_data[0] stats["tx_packets"] = stats_data[2] stats["rx_bytes"] = stats_data[4] diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py index 03efd40..dec992f 100644 --- a/lnst/Slave/NetConfigDevice.py +++ b/lnst/Slave/NetConfigDevice.py @@ -585,7 +585,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bond_ports = [] - for bond in self._dev_config["ovs_conf"]["bonds"].itervalues(): + for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id)
@@ -600,7 +600,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): options += " %s=%s" % (opt[0], opt[1])
vlan_tags = [] - for tag, vlan in vlans.iteritems(): + for tag, vlan in vlans.items(): if slave_id in vlan["slaves"]: vlan_tags.append(tag) if len(vlan_tags) == 0: @@ -620,7 +620,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bond_ports = [] - for bond in self._dev_config["ovs_conf"]["bonds"].itervalues(): + for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id)
@@ -687,7 +687,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bonds = self._dev_config["ovs_conf"]["bonds"] - for bond_id, bond in bonds.iteritems(): + for bond_id, bond in bonds.items(): ifaces = "" for slave_id in bond["slaves"]: slave_dev = self._if_manager.get_mapped_device(slave_id) @@ -703,7 +703,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bonds = self._dev_config["ovs_conf"]["bonds"] - for bond_id, bond in bonds.iteritems(): + for bond_id, bond in bonds.items(): exec_cmd("ovs-vsctl del-port %s %s" % (br_name, bond_id))
def _add_flow_entries(self): diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index 662faaf..7c61d23 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -22,7 +22,7 @@ import multiprocessing import re import struct from time import sleep, time -from xmlrpclib import Binary +from xmlrpc.client import Binary from tempfile import NamedTemporaryFile from lnst.Common.Logs import log_exc_traceback from lnst.Common.PacketCapture import PacketCapture @@ -181,7 +181,7 @@ class SlaveMethods: dev_data = dev.get_if_data() entry = {"name": dev.get_name(), "hwaddr": dev.get_hwaddr()} - for key, value in params.iteritems(): + for key, value in params.items(): if key not in dev_data or dev_data[key] != value: entry = None break @@ -405,7 +405,7 @@ class SlaveMethods: raise Exception("Can't start packet capture, tcpdump not available")
files = {} - for if_id, dev in self._if_manager.get_mapped_devices().iteritems(): + for if_id, dev in self._if_manager.get_mapped_devices().items(): if dev.get_netns() != None: continue dev_name = dev.get_name() @@ -431,7 +431,7 @@ class SlaveMethods: if self._packet_captures == None: return True
- for if_index, pcap in self._packet_captures.iteritems(): + for if_index, pcap in self._packet_captures.items(): pcap.stop()
self._packet_captures.clear() @@ -439,7 +439,7 @@ class SlaveMethods: return True
def _remove_capture_files(self): - for key, name in self._capture_files.iteritems(): + for key, name in self._capture_files.items(): logging.debug("Removing temporary packet capture file %s", name) os.unlink(name)
@@ -462,7 +462,7 @@ class SlaveMethods:
def restore_system_config(self): logging.info("Restoring system configuration") - for option, values in self._system_config.iteritems(): + for option, values in self._system_config.items(): try: cmd_str = "echo "%s" >%s" % (values["initial_val"], option) (stdout, stderr) = exec_cmd(cmd_str) @@ -538,7 +538,7 @@ class SlaveMethods: logging.info("Performing machine cleanup.") self._command_context.cleanup()
- for mroute_soc in self.mroute_sockets.values(): + for mroute_soc in list(self.mroute_sockets.values()): mroute_soc.close() del mroute_soc self.mroute_sockets = {} @@ -546,7 +546,7 @@ class SlaveMethods: self.restore_system_config()
devs = self._if_manager.get_mapped_devices() - for if_id, dev in devs.iteritems(): + for if_id, dev in devs.items(): peer = dev.get_peer() if peer == None: dev.clear_configuration() @@ -557,7 +557,7 @@ class SlaveMethods:
self._if_manager.deconfigure_all()
- for netns in self._net_namespaces.keys(): + for netns in list(self._net_namespaces.keys()): self.del_namespace(netns) self._net_namespaces = {}
@@ -641,11 +641,11 @@ class SlaveMethods: return False
def reset_file_transfers(self): - for file_handle in self._copy_targets.itervalues(): + for file_handle in self._copy_targets.values(): file_handle.close() self._copy_targets = {}
- for file_handle in self._copy_sources.itervalues(): + for file_handle in self._copy_sources.values(): file_handle.close() self._copy_sources = {}
@@ -1086,7 +1086,7 @@ class SlaveMethods: return True
def mroute_operation(self, op_type, op, table_id): - if not self.mroute_sockets.has_key(table_id): + if table_id not in self.mroute_sockets: logging.error("mroute %s table was not init", table_id) return False try: @@ -1098,7 +1098,7 @@ class SlaveMethods:
def mroute_init(self, table_id): logging.debug("Initializing mroute socket") - if not self.mroute_sockets.has_key(table_id): + if table_id not in self.mroute_sockets: self.mroute_sockets[table_id] = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IGMP) @@ -1157,7 +1157,7 @@ class SlaveMethods: (source, group, str(out_vifs)))
ttls = [0] * MROUTE.MAX_VIF - for vif, ttl in out_vifs.items(): + for vif, ttl in list(out_vifs.items()): if vif >= MROUTE.MAX_VIF: logging.error("ilegal VIF was asked") return False @@ -1183,7 +1183,7 @@ class SlaveMethods: return self.mroute_operation(op_type, mfc_struct, table_id)
def mroute_get_notif(self, table_id): - if not self.mroute_sockets.has_key(table_id): + if table_id not in self.mroute_sockets: logging.error("mroute table %s was not init", table_id) return False try: @@ -1340,7 +1340,7 @@ class ServerHandler(ConnectionHandler): self._netns_con_mapping = {}
def update_connections(self, connections): - for key, connection in connections.iteritems(): + for key, connection in connections.items(): self.remove_connection_by_id(key) self.add_connection(key, connection)
@@ -1392,21 +1392,23 @@ class NetTestSlave: self._if_manager.get_nl_socket())
def run(self): - while not self._finished: - if self._server_handler.get_ctl_sock() == None: - self._log_ctl.cancel_connection() - try: - logging.info("Waiting for connection.") - self._server_handler.accept_connection() - except (socket.error, SecSocketException): - continue - self._log_ctl.set_connection( - self._server_handler.get_ctl_sock()) - - msgs = self._server_handler.get_messages() - - for msg in msgs: - self._process_msg(msg[1]) + while True: + try: + if self._server_handler.get_ctl_sock() is None: + self._log_ctl.cancel_connection() + try: + logging.info("Waiting for connection.") + self._server_handler.accept_connection() + except (socket.error, SecSocketException): + continue + self._log_ctl.set_connection(self._server_handler.get_ctl_sock()) + + msgs = self._server_handler.get_messages() + + for msg in msgs: + self._process_msg(msg[1]) + except: + break
self._methods.machine_cleanup()
@@ -1507,7 +1509,7 @@ class NetTestSlave:
def _signal_die_handler(self, signum, frame): logging.info("Caught signal %d -> dying" % signum) - self._finished = True + raise Exception("Recieved interrupt to system call")
def _parent_resend_signal_handler(self, signum, frame): logging.info("Caught signal %d -> resending to parent" % signum) diff --git a/lnst/Slave/NmConfigDevice.py b/lnst/Slave/NmConfigDevice.py index 29c2c75..74b4f85 100644 --- a/lnst/Slave/NmConfigDevice.py +++ b/lnst/Slave/NmConfigDevice.py @@ -218,10 +218,7 @@ class NmConfigDeviceGeneric(object): except: #IPv6 conversion into a 16 byte array tmp = socket.inet_pton(socket.AF_INET6, ip) - ip = [] - for i in tmp: - ip.append(ord(i)) - ip = dbus.Array(ip, signature='y') + ip = dbus.Array(tmp, signature='y') def_gateway = dbus.Array([0]*16, signature='y') ipv6s.append(tuple([ip, dbus.UInt32(mask), diff --git a/setup.py b/setup.py index cc5d6f4..ebb05c4 100755 --- a/setup.py +++ b/setup.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 """ Install script for lnst
@@ -34,7 +34,7 @@ def process_template(template_path, values): t = open(template_path, "r") f = open(file_path, "w") template = t.read() - for var, value in values.iteritems(): + for var, value in values.items(): template = template.replace("@%s@" % var, value) f.write(template) f.close() diff --git a/test_modules/Multicast.py b/test_modules/Multicast.py index b366c9b..44a003c 100644 --- a/test_modules/Multicast.py +++ b/test_modules/Multicast.py @@ -61,7 +61,7 @@ class Multicast(TestGeneric):
cmd = "./{0} ".format(setup)
- for optname, optval in opts.iteritems(): + for optname, optval in list(opts.items()): if optval != None: cmd += "--{0} "{1}" ".format(optname, optval)
diff --git a/test_modules/Netperf.py b/test_modules/Netperf.py index 00692e5..7e9fbd0 100644 --- a/test_modules/Netperf.py +++ b/test_modules/Netperf.py @@ -487,7 +487,7 @@ class Netperf(TestGeneric): results.append(result) rates.append(results[-1]["rate"])
- if results > 1: + if len(results) > 1: res_data["results"] = results
if len(rates) > 0: diff --git a/test_modules/TRexClient.py b/test_modules/TRexClient.py index f3ed2ee..60416e8 100644 --- a/test_modules/TRexClient.py +++ b/test_modules/TRexClient.py @@ -19,7 +19,7 @@ class TRexClient(TestGeneric): super(TRexClient, self).__init__(command)
self._trex_path = self.get_mopt("trex_path") - self._ports = map(int, self.get_multi_mopt("ports")) + self._ports = list(map(int, self.get_multi_mopt("ports")))
self._src_macs = [] self._dst_macs = [] @@ -42,7 +42,7 @@ class TRexClient(TestGeneric): for res in results: new_results.append({}) new_res = new_results[-1] - for key, data in res.items(): + for key, data in list(res.items()): if key in self._ports: new_res["port_"+str(key)] = data else:
On Thu, Aug 30, 2018 at 02:27:02PM +0200, Jozef Urbanovsky wrote:
Main changes:
- Print function - PEP 3105[1]
- Design of dictionaries, their iteration and respective built-in functions - PEP 469[2], PEP 3106[3]
- Division operator - PEP 238[4]
- Interrupts of system calls - PEP 475[5]
- Socket.accept and interrupts - Docs Python3 Socket[6]
- Universal newlines - PEP 278[7], PEP 3116[8], Docs Python3 Subprocess[9]
- Binary data and strings - Docs Python3 porting[10]
- Shebangs modified to environmental variable of python3
- Fixed library imports to reflect their respective changes in transition to python3
- multiprocessing
- xmlrpc
- urllib
- thread
- pickle
InterfaceManager.py
- Byte string and unicode/ascii string conflict, fixed by mostly using a byte string through the project
NetTestSlave.py
- signum 2 (SIGINT) - modified behavior since python 3.5
- Due to changes in interrupting system call, signal receiving and handling required changes to main slave loop
- Structure of raising, catching and handling interrupt was changed, to reflect behavior of python 3.5 and higher
- Method run is no longer ended by parameter "finished", but rather exception breaks the endless loop
SecureSocket.py
- Byte string and unicode/ascii string conflict, fixed by using bytes internally and sending messages encoded to ascii
- Fixed multiple issues regarding changes to standard python functions, such as encoding to hex, ordinal value of characters, length of different types
NetTestCommand.py
- name_start_char is using unicode string, therefore each unicode has to be escaped twice
ExecCmd.py
- Popen universal newlines
- Abstraction to be able to read newlines regardless of system used and its basic encoding
XmlParser.py
- etree.tostring function behavior changed to generate byte string by default
- Unicode encoding is required, as specified by parameter
NmConfigDevice.py
- Conversion of bytes array to integer value was changed, ord no longer returns plain integer value
- Need to convert bytes array to integer - int.from_bytes
- Reverse conversion does not need to account for bytes array, therefore it is redundant to convert IP twice
ShellProcess.py
- Decoding file stream, to use unicode string instead of byte string, as built-in function read returns byte string by default in python3
Netperf.py
- Changed evaluation of results to account for changes made to base types in python3
- Dictionary comparision with integer by their respective lengths and values
Machine.py
- List comprehension functionality in python3 was changed
- New function to parse queried addresses from slaves - modified code from next branch
- get_ip_addr returns a dictionary or IP address objects - fixed to return only IP address portion by getting addr attribute from dictionary
================================================ [1] - PEP 3105 https://www.python.org/dev/peps/pep-3105/ [2] - PEP 469 https://www.python.org/dev/peps/pep-0469/ [3] - PEP 3106 https://www.python.org/dev/peps/pep-3106/ [4] - PEP 238 https://legacy.python.org/dev/peps/pep-0238/ [5] - PEP 475 https://www.python.org/dev/peps/pep-0475/ [6] - Docs Python3 Socket https://docs.python.org/3/library/socket.html#socket.socket.accept [7] - PEP 278 https://www.python.org/dev/peps/pep-0278/ [8] - PEP 3116 https://www.python.org/dev/peps/pep-3116/ [9] - Docs Python3 Subprocess https://docs.python.org/3/library/subprocess.html [10] - Docs Python3 Porting https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data
Signed-off-by: Jozef Urbanovsky jurbanov@redhat.com
lnst-ctl | 88 +++++++++++----------- lnst-pool-wizard | 6 +- lnst-slave | 20 ++--- lnst/Common/Colours.py | 12 +-- lnst/Common/Config.py | 8 +- lnst/Common/ConnectionHandler.py | 2 +- lnst/Common/ExecCmd.py | 5 +- lnst/Common/LoggingHandler.py | 4 +- lnst/Common/Logs.py | 6 +- lnst/Common/NetTestCommand.py | 18 ++--- lnst/Common/NetUtils.py | 2 +- lnst/Common/Path.py | 5 +- lnst/Common/ProcessManager.py | 14 ++-- lnst/Common/ResourceCache.py | 4 +- lnst/Common/SecureSocket.py | 71 +++++++++-------- lnst/Common/ShellProcess.py | 2 +- lnst/Common/Utils.py | 4 +- lnst/Controller/Machine.py | 30 +++++--- lnst/Controller/NetTestController.py | 36 ++++----- lnst/Controller/NetTestResultSerializer.py | 8 +- lnst/Controller/RecipeParser.py | 16 ++-- lnst/Controller/SlavePool.py | 48 ++++++------ lnst/Controller/Task.py | 16 ++-- lnst/Controller/Wizard.py | 36 ++++----- lnst/Controller/XmlParser.py | 4 +- lnst/Controller/XmlProcessing.py | 12 +-- lnst/Controller/XmlTemplates.py | 6 +- lnst/Slave/InterfaceManager.py | 40 +++++----- lnst/Slave/NetConfigDevice.py | 10 +-- lnst/Slave/NetTestSlave.py | 66 ++++++++-------- lnst/Slave/NmConfigDevice.py | 5 +- setup.py | 4 +- test_modules/Multicast.py | 2 +- test_modules/Netperf.py | 2 +- test_modules/TRexClient.py | 4 +- 35 files changed, 317 insertions(+), 299 deletions(-)
diff --git a/lnst-ctl b/lnst-ctl index d61353d..ed13760 100755 --- a/lnst-ctl +++ b/lnst-ctl @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test controller
@@ -34,41 +34,41 @@ def usage(retval=0): """ Print usage of this app """
- print "Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0]
- print ""
- print "ACTION = [ run | config_only | deconfigure | match_setup | " \
"list_pools]"- print ""
- print "OPTIONS"
- print " -A, --override-alias name=value define top-level alias that " \
"will override any other definitions in the recipe"- print " -a, --define-alias name=value define top-level alias"
- print " -c, --config=FILE load additional config file"
- print " -C, --config-override=FILE reset config defaults and load " \
"the following config file"- print " -d, --debug emit debugging messages"
- print " --dump-config dumps the join of all loaded " \
"configuration files on stdout and exits"- print " -v, --verbose verbose version of list_pools " \
"command"- print " -h, --help print this message"
- print " -m, --no-colours disable coloured terminal output"
- print " -o, --disable-pool-checks don't check the availability of " \
"machines in the pool"- print " -p, --packet-capture capture and log all ongoing " \
"network communication during the test"- print " --pools=NAME[,...] restricts which pools to use "\
"for matching, value can be a comma separated list of values or"- print " --pools=PATH a single path to a pool directory"
- print " -r, --reduce-sync reduces resource synchronization "\
"for python tasks, see documentation"- print " -s, --xslt-url=URL URL to a XSLT document that will "\
- print("Usage: %s [OPTIONS...] ACTION [RECIPES...]" % sys.argv[0])
- print("")
- print("ACTION = [ run | config_only | deconfigure | match_setup | " \
"list_pools]")- print("")
- print("OPTIONS")
- print(" -A, --override-alias name=value define top-level alias that " \
"will override any other definitions in the recipe")- print(" -a, --define-alias name=value define top-level alias")
- print(" -c, --config=FILE load additional config file")
- print(" -C, --config-override=FILE reset config defaults and load " \
"the following config file")- print(" -d, --debug emit debugging messages")
- print(" --dump-config dumps the join of all loaded " \
"configuration files on stdout and exits")- print(" -v, --verbose verbose version of list_pools " \
"command")- print(" -h, --help print this message")
- print(" -m, --no-colours disable coloured terminal output")
- print(" -o, --disable-pool-checks don't check the availability of " \
"machines in the pool")- print(" -p, --packet-capture capture and log all ongoing " \
"network communication during the test")- print(" --pools=NAME[,...] restricts which pools to use "\
"for matching, value can be a comma separated list of values or")- print(" --pools=PATH a single path to a pool directory")
- print(" -r, --reduce-sync reduces resource synchronization "\
"for python tasks, see documentation")- print(" -s, --xslt-url=URL URL to a XSLT document that will "\ "be used when transforming the xml result file, only useful "\
"when -t is used as well"- print " -t, --html=FILE generate a formatted result html"
- print " -u, --multi-match run each recipe with every "\
"pool match possible"- print " -x, --result=FILE file to write xml_result"
"when -t is used as well")- print(" -t, --html=FILE generate a formatted result html")
- print(" -u, --multi-match run each recipe with every "\
"pool match possible")- print(" -x, --result=FILE file to write xml_result") sys.exit(retval)
def list_pools(restrict_pools, verbose): @@ -90,12 +90,12 @@ def list_pools(restrict_pools, verbose): out = "" # iterate over all pools sp_pools = sp.get_pools()
- for pool_name, pool_content in sp_pools.iteritems():
- for pool_name, pool_content in sp_pools.items(): out += "Pool: %s (%s)\n" % (pool_name, pools[pool_name]) # verbose output if verbose: # iterate over all slave machine cfgs
for filename, pool in pool_content.iteritems():
for filename, pool in pool_content.items(): # print in human-readable format out += 3*" " + filename + ".xml\n" out += 5*" " + "params:\n"@@ -113,9 +113,9 @@ def list_pools(restrict_pools, verbose): out += "\n" # print wihout newlines on the end of string if verbose:
print out[:-2]
print(out[:-2])
print out[:-1]
print(out[:-1])def store_alias(alias_def, aliases_dict): @@ -263,7 +263,7 @@ def main(): ] ) except getopt.GetoptError as err:
print str(err)
print(str(err)) usage(RETVAL_ERR)lnst_config.controller_init()
@@ -308,16 +308,16 @@ def main(): usage(RETVAL_PASS) elif opt in ("-c", "--config"): if not os.path.isfile(arg):
print "File '%s' doesn't exist!" % arg
print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else: lnst_config.load_config(arg) elif opt in ("-C", "--config-override"): if not os.path.isfile(arg):
print "File '%s' doesn't exist!" % arg
print("File '%s' doesn't exist!" % arg) usage(RETVAL_ERR) else:
print >> sys.stderr, "Reloading config defaults!"
print("Reloading config defaults!", file=sys.stderr) lnst_config.controller_init() lnst_config.load_config(arg) elif opt in ("-x", "--result"):@@ -351,7 +351,7 @@ def main(): lnst_config.set_option("environment", "xslt_url", xslt_url)
if dump_config:
print lnst_config.dump_config()
print(lnst_config.dump_config()) return RETVAL_PASSif coloured_output:
diff --git a/lnst-pool-wizard b/lnst-pool-wizard index ea2849b..f10635e 100755 --- a/lnst-pool-wizard +++ b/lnst-pool-wizard @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Machine pool wizard
@@ -19,7 +19,7 @@ RETVAL_ERR = 1
def help(retval=0):
- print "Usage:\n"\
- print("Usage:\n"\ " lnst-pool-wizard [mode] [hostname[:port]]\n"\ "\n"\ "Modes:\n"\
@@ -34,7 +34,7 @@ def help(retval=0): " lnst-pool-wizard hostname1:1234 hostname2\n"\ " lnst-pool-wizard --noninteractive 192.168.122.2\n"\ " lnst-pool-wizard -n 192.168.122.2:8888 192.168.122.4\n"\
" lnst-pool-wizard -p \".pool/\" -n 192.168.1.1:8877 192.168.122.4"
" lnst-pool-wizard -p \".pool/\" -n 192.168.1.1:8877 192.168.122.4")diff --git a/lnst-slave b/lnst-slave index ebb4cd3..0f77873 100755 --- a/lnst-slave +++ b/lnst-slave @@ -1,4 +1,4 @@ -#! /usr/bin/env python2 +#! /usr/bin/env python3 """ Net test slave
@@ -25,14 +25,14 @@ def usage(): """ Print usage of this app """
- print "Usage: %s [OPTION...]" % sys.argv[0]
- print ""
- print " -d, --debug emit debugging messages"
- print " -h, --help print this message"
- print " -e, --daemonize go to background after init"
- print " -i, --pidfile file to write daemonized process pid"
- print " -m, --no-colours disable coloured terminal output"
- print " -p, --port xmlrpc port to listen on"
- print("Usage: %s [OPTION...]" % sys.argv[0])
- print("")
- print(" -d, --debug emit debugging messages")
- print(" -h, --help print this message")
- print(" -e, --daemonize go to background after init")
- print(" -i, --pidfile file to write daemonized process pid")
- print(" -m, --no-colours disable coloured terminal output")
- print(" -p, --port xmlrpc port to listen on") sys.exit()
def main(): @@ -46,7 +46,7 @@ def main(): ["debug", "help", "daemonize", "pidfile=", "port=", "no-colours"] )[0] except getopt.GetoptError as err:
print str(err)
print(str(err)) usage() sys.exit()diff --git a/lnst/Common/Colours.py b/lnst/Common/Colours.py index e57794e..a00e2d2 100644 --- a/lnst/Common/Colours.py +++ b/lnst/Common/Colours.py @@ -52,7 +52,7 @@ def name_to_fg_colour(name): """ Convert name to foreground colour code. Returns None if the colour name isn't supported. """
- if not COLOURS.has_key(name):
if name not in COLOURS: return None
return COLOURS[name]
@@ -61,7 +61,7 @@ def name_to_bg_colour(name): """ Convert name to background color code. Returns None if the colour name isn't supported. """
- if not COLOURS.has_key(name):
if name not in COLOURS: return None
return COLOURS[name] + 10
@@ -136,7 +136,7 @@ def decorate_string(string, fg_colour=None, bg_colour=None, bold=False): raise Exception(msg) else: # Standard definition
if colour_def in COLOURS.keys():
if colour_def in list(COLOURS.keys()): if fg: colour = name_to_fg_colour(colour_def) else:@@ -165,7 +165,7 @@ def strip_colours(text):
def get_preset_conf(preset_name): preset = PRESETS[preset_name]
- return map(lambda s: "default" if s == None else str(s), preset)
- return ["default" if s == None else str(s) for s in preset]
def load_presets_from_config(lnst_config): for preset_name in PRESETS: @@ -178,12 +178,12 @@ def load_presets_from_config(lnst_config):
if fg == "default": fg = None
elif not re.match(extended_re, fg) and fg not in COLOURS.keys():
elif not re.match(extended_re, fg) and fg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % fg) if bg == "default": bg = None
elif not re.match(extended_re, bg) and bg not in COLOURS.keys():
elif not re.match(extended_re, bg) and bg not in list(COLOURS.keys()): raise Exception("Colour '%s' not supported" % bg) PRESETS[preset_name] = [fg, bg, bool_it(bf)]diff --git a/lnst/Common/Config.py b/lnst/Common/Config.py index d47f3e2..3aa2820 100644 --- a/lnst/Common/Config.py +++ b/lnst/Common/Config.py @@ -221,7 +221,7 @@ class Config(): raise ConfigError(msg)
res = {}
for opt_name, opt in self._options[section].items():
for opt_name, opt in list(self._options[section].items()): res[opt_name] = opt["value"] return res@@ -283,7 +283,7 @@ class Config(): '''Parse and load the config file''' exp_path = os.path.expanduser(path) abs_path = os.path.abspath(exp_path)
print >> sys.stderr, "Loading config file '%s'" % abs_path
print("Loading config file '%s'" % abs_path, file=sys.stderr) sections = self._parse_file(abs_path) self.handleSections(sections, abs_path)@@ -338,7 +338,7 @@ class Config():
def get_pools(self): pools = {}
for pool_name, pool in self._options["pools"].items():
for pool_name, pool in list(self._options["pools"].items()): pools[pool_name] = pool["value"] return pools@@ -349,7 +349,7 @@ class Config(): return None
def _find_option_by_name(self, section, opt_name):
for option in section.itervalues():
for option in section.values(): if option["name"] == opt_name: return option return Nonediff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py index 5d3170a..3856917 100644 --- a/lnst/Common/ConnectionHandler.py +++ b/lnst/Common/ConnectionHandler.py @@ -13,7 +13,7 @@ olichtne@redhat.com (Ondrej Lichtner)
import select import socket -from _multiprocessing import Connection +from multiprocessing.connection import Connection from pyroute2 import IPRSocket from lnst.Common.SecureSocket import SecureSocket, SecSocketException
diff --git a/lnst/Common/ExecCmd.py b/lnst/Common/ExecCmd.py index 24715ae..5327f11 100644 --- a/lnst/Common/ExecCmd.py +++ b/lnst/Common/ExecCmd.py @@ -12,6 +12,7 @@ jpirko@redhat.com (Jiri Pirko)
import logging import subprocess +import sys
class ExecCmdFail(Exception): _cmd = None @@ -55,9 +56,11 @@ def exec_cmd(cmd, die_on_err=True, log_outputs=True, report_stderr=False, json=F cmd = cmd.rstrip(" ") logging.debug("Executing: "%s"" % cmd) subp = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, close_fds=True)
stderr=subprocess.PIPE, close_fds=True, universal_newlines=True)(data_stdout, data_stderr) = subp.communicate()
#data_stdout = data_stdout.decode(sys.getfilesystemencoding())
#data_stderr = data_stderr.decode(sys.getfilesystemencoding())
^^^ these are commented out, is that just lefover debug code or is it important and you forgot to uncomment it?
''' When we should not die on error, do not print anything and let the caller to decide what to do.diff --git a/lnst/Common/LoggingHandler.py b/lnst/Common/LoggingHandler.py index a236922..6acc9bc 100644 --- a/lnst/Common/LoggingHandler.py +++ b/lnst/Common/LoggingHandler.py @@ -16,7 +16,7 @@ olichtne@redhat.com (Ondrej Lichtner)
import pickle import logging -import xmlrpclib +import xmlrpc.client from lnst.Common.ConnectionHandler import send_data
class LogBuffer(logging.Handler): @@ -39,7 +39,7 @@ class LogBuffer(logging.Handler): d['args'] = None d['exc_info'] = None s = pickle.dumps(d, 1)
return xmlrpclib.Binary(s)
return xmlrpc.client.Binary(s)def add_buffer(self, buf): for i in buf:
diff --git a/lnst/Common/Logs.py b/lnst/Common/Logs.py index 36e0580..77c0f14 100644 --- a/lnst/Common/Logs.py +++ b/lnst/Common/Logs.py @@ -106,7 +106,7 @@ class LoggingCtl: logger = logging.getLogger('') for i in list(logger.handlers): logger.removeHandler(i)
for key, logger in logging.Logger.manager.loggerDict.iteritems():
for key, logger in list(logging.Logger.manager.loggerDict.items()): if type(logger) != type(logging.Logger): continue for i in list(logger.handlers):@@ -218,7 +218,7 @@ class LoggingCtl: logger = logging.getLogger() logger.addHandler(self.transmit_handler)
for k in self.slaves.keys():
for k in list(self.slaves.keys()): self.remove_slave(k)def cancel_connection(self):
@@ -230,7 +230,7 @@ class LoggingCtl: def disable_logging(self): self.cancel_connection()
for s in self.slaves.keys():
for s in list(self.slaves.keys()): self.remove_slave(s) self.unset_recipe()diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index b894363..79b5893 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -325,10 +325,10 @@ class NetTestCommandGeneric(object): formatted_data = "" if res_data: max_key_len = 0
for key in res_data.keys():
for key in list(res_data.keys()): if len(key) > max_key_len: max_key_len = len(key)
for key, value in res_data.iteritems():
for key, value in list(res_data.items()): if type(value) == dict: formatted_data += level*4*" " + str(key) + ":\n" formatted_data += self.format_res_data(value, level+1)@@ -347,13 +347,13 @@ class NetTestCommandGeneric(object): return formatted_data
def _check_res_data(self, res_data):
name_start_char = u":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF"\u"\u0370-\u037D\u037F-\u1FFF\u200C-\u200D"\u"\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF"\u"\uF900-\uFDCF\uFDF0-\uFFFD\U00010000-\U000EFFFF"name_char = name_start_char + u"\-\.0-9\xB7\u0300-\u036F\u203F-\u2040"name = u"[%s]([%s])*$" % (name_start_char, name_char)char_data = u"[^<&]*"
name_start_char = ":A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\\u02FF"\"\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D"\"\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF"\"\\uF900-\\uFDCF\\uFDF0-\\uFFFD\\U00010000-\\U000EFFFF"name_char = name_start_char + "\-\.0-9\xB7\\u0300-\\u036F\\u203F-\\u2040"name = "[%s]([%s])*$" % (name_start_char, name_char)char_data = "[^<&]*" if isinstance(res_data, dict): for key in res_data: if not re.match(name, key, re.UNICODE):diff --git a/lnst/Common/NetUtils.py b/lnst/Common/NetUtils.py index 9fd4732..42f377b 100644 --- a/lnst/Common/NetUtils.py +++ b/lnst/Common/NetUtils.py @@ -136,7 +136,7 @@ class MacPool(AddressPool): return bs
def _byte_string_to_addr(self, byte_string):
return ':'.join(map(lambda x: "%02x" % x, byte_string))
return ':'.join(["%02x" % x for x in byte_string])class IpPool(AddressPool): diff --git a/lnst/Common/Path.py b/lnst/Common/Path.py index fe7f709..e61b8eb 100644 --- a/lnst/Common/Path.py +++ b/lnst/Common/Path.py @@ -11,8 +11,9 @@ jtluka@redhat.com (Jan Tluka) """
import os -from urlparse import urljoin -from urllib2 import urlopen, HTTPError +from urllib.parse import urljoin +from urllib.request import urlopen +from urllib.error import HTTPError from tempfile import NamedTemporaryFile
def get_path_class(root, path): diff --git a/lnst/Common/ProcessManager.py b/lnst/Common/ProcessManager.py index 1995d15..564b49e 100644 --- a/lnst/Common/ProcessManager.py +++ b/lnst/Common/ProcessManager.py @@ -9,18 +9,18 @@ published by the Free Software Foundation; see COPYING for details. __autor__ = """ jzupka@redhat.com (Jiri Zupka) """ -import os, signal, thread, logging +import os, signal, _thread, logging
class ProcessManager: class SubProcess: def __init__(self, pid, handler): self.pid = pid
self.lock = thread.allocate_lock()
self.lock = _thread.allocate_lock() self.lock.acquire() self.handler = handler self.enabled = True self.status = None
thread.start_new_thread(self.waitpid, (self.pid, self.lock,
_thread.start_new_thread(self.waitpid, (self.pid, self.lock, self.handler)) def isAlive(self):@@ -45,12 +45,12 @@ class ProcessManager: logging.error(''.join(traceback.format_exception(type, value, tb))) os.kill(os.getpid(), signal.SIGTERM) else:
print "Process pid %s exit with exitcode %s" % (pid, status)
print("Process pid %s exit with exitcode %s" % (pid, status)) ProcessManager.lock.release()
thread.exit()
_thread.exit()pids = {}
- lock = thread.allocate_lock()
lock = _thread.allocate_lock() std_waitpid = None
@classmethod
@@ -86,7 +86,7 @@ class ProcessManager: return pid, status
-lock = thread.allocate_lock() +lock = _thread.allocate_lock() lock.acquire() if os.waitpid != ProcessManager.waitpid: ProcessManager.std_waitpid = os.waitpid diff --git a/lnst/Common/ResourceCache.py b/lnst/Common/ResourceCache.py index 98558a2..092f974 100644 --- a/lnst/Common/ResourceCache.py +++ b/lnst/Common/ResourceCache.py @@ -65,7 +65,7 @@ class ResourceCache(object): header = "# hash " \ "last_used type name path\n" f.write(header)
for entry_hash, entry in self._entries.iteritems():
for entry_hash, entry in list(self._entries.items()): values = (entry_hash, entry["last_used"], entry["type"], entry["name"], entry["path"]) line = "%s %d %s %s %s\n" % values@@ -137,7 +137,7 @@ class ResourceCache(object):
rm = [] now = time.time()
for entry_hash, entry in self._entries.iteritems():
for entry_hash, entry in list(self._entries.items()): if entry["last_used"] <= (now - self._expiration_period): rm.append(entry_hash)diff --git a/lnst/Common/SecureSocket.py b/lnst/Common/SecureSocket.py index b856238..349c413 100644 --- a/lnst/Common/SecureSocket.py +++ b/lnst/Common/SecureSocket.py @@ -16,7 +16,7 @@ olichtne@redhat.com (Ondrej Lichtner) """
import os -import cPickle +import pickle import hashlib import hmac from lnst.Common.Utils import not_imported @@ -43,11 +43,11 @@ def bit_length(i): except AttributeError: return len(bin(i)) - 2
-DH_GROUP["q"] = (DH_GROUP["p"]-1)/2 -DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])/8 +DH_GROUP["q"] = (DH_GROUP["p"]-1)//2 +DH_GROUP["q_size"] = bit_length(DH_GROUP["q"])//8 if bit_length(DH_GROUP["q"])%8: DH_GROUP["q_size"] += 1 -DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])/8 +DH_GROUP["p_size"] = bit_length(DH_GROUP["p"])//8 if bit_length(DH_GROUP["p"])%8: DH_GROUP["p_size"] += 1
@@ -63,11 +63,11 @@ SRP_GROUP = {"p": int("0xAC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC" "DE236D525F54759B65E372FCD68EF20FA7111F9E4AFF73", 16), "g": 2}
-SRP_GROUP["q"] = (SRP_GROUP["p"]-1)/2 -SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])/8 +SRP_GROUP["q"] = (SRP_GROUP["p"]-1)//2 +SRP_GROUP["q_size"] = bit_length(SRP_GROUP["q"])//8 if bit_length(SRP_GROUP["q"])%8: SRP_GROUP["q_size"] += 1 -SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])/8 +SRP_GROUP["p_size"] = bit_length(SRP_GROUP["p"])//8 if bit_length(SRP_GROUP["p"])%8: SRP_GROUP["p_size"] += 1
@@ -150,14 +150,14 @@ class SecureSocket(object): "seq_num": 0}
def send_msg(self, msg):
pickled_msg = cPickle.dumps(msg)
pickled_msg = pickle.dumps(msg) return self.send(pickled_msg)def recv_msg(self): pickled_msg = self.recv()
if pickled_msg == "":
if pickled_msg == b"": raise SecSocketException("Disconnected")
msg = cPickle.loads(pickled_msg)
msg = pickle.loads(pickled_msg) return msgdef _add_mac_sign(self, data):
@@ -165,22 +165,26 @@ class SecureSocket(object): return data cryptography_imports()
msg = str(self._current_write_spec["seq_num"]) + str(len(data)) + data
msg = (bytes(str(self._current_write_spec["seq_num"]).encode('ascii'))+ bytes(str(len(data)).encode('ascii'))+ data) signature = hmac.new(self._current_write_spec["mac_key"], msg, hashlib.sha256) signed_msg = {"data": data, "signature": signature.digest()}
return cPickle.dumps(signed_msg)
return pickle.dumps(signed_msg)def _del_mac_sign(self, signed_data): if not self._current_read_spec["mac_key"]: return signed_data cryptography_imports()
signed_msg = cPickle.loads(signed_data)
signed_msg = pickle.loads(signed_data) data = signed_msg["data"]
msg = str(self._current_read_spec["seq_num"]) + str(len(data)) + data
msg = (bytes(str(self._current_read_spec["seq_num"]).encode('ascii'))+ bytes(str(len(data)).encode('ascii'))+ data) signature = hmac.new(self._current_read_spec["mac_key"], msg,@@ -195,9 +199,9 @@ class SecureSocket(object): return data cryptography_imports()
block_size = algorithms.AES.block_size/8
block_size = algorithms.AES.block_size//8 pad_length = block_size - (len(data) % block_size)
pad_char = ("%02x" % pad_length).decode("hex")
pad_char = bytes([pad_length]) padding = pad_length * pad_char padded_data = data+padding@@ -208,9 +212,9 @@ class SecureSocket(object): return data cryptography_imports()
pad_length = int(data[-1].encode("hex"), 16)
pad_length = ord(data[-1]) for char in data[-pad_length]:
if int(char.encode("hex"), 16) != pad_length:
if ord(char) != pad_length: return None return data[:-pad_length]@@ -220,7 +224,7 @@ class SecureSocket(object): return data cryptography_imports()
iv = os.urandom(algorithms.AES.block_size/8)
iv = os.urandom(algorithms.AES.block_size//8) mode = modes.CBC(iv) key = self._current_write_spec["enc_key"] cipher = Cipher(algorithms.AES(key), mode, default_backend())@@ -231,14 +235,14 @@ class SecureSocket(object): encrypted_msg = {"iv": iv, "enc_data": encrypted_data}
return cPickle.dumps(encrypted_msg)
return pickle.dumps(encrypted_msg)def _del_encrypt(self, data): if not self._current_read_spec["enc_key"]: return data cryptography_imports()
encrypted_msg = cPickle.loads(data)
encrypted_msg = pickle.loads(data) encrypted_data = encrypted_msg["enc_data"] iv = encrypted_msg["iv"]@@ -276,27 +280,28 @@ class SecureSocket(object): def send(self, data): protected_data = self._protect_data(data)
transmit_data = str(len(protected_data)) + " " + protected_data
transmit_data = bytes(str(len(protected_data)).encode('ascii')) + b" " + protected_data return self._socket.sendall(transmit_data)def recv(self):
length = ""
length = b"" while True: c = self._socket.recv(1)
if c == ' ':length = int(length)
if c == b' ':length = int(length.decode('ascii')) break
elif c == "":return ""
elif c == b"":return b"" else: length += c
data = ""
data = b"" while len(data) < length: c = self._socket.recv(length - len(data))
if c == "":return ""
if c == b"":return b"" else: data += c@@ -307,7 +312,7 @@ class SecureSocket(object):
def _handle_internal(self, orig_msg): try:
msg = cPickle.loads(orig_msg)
msg = pickle.loads(orig_msg) except: return orig_msg if "type" in msg and msg["type"] == "change_cipher_spec":@@ -348,7 +353,7 @@ class SecureSocket(object):
def p_SHA256(self, secret, seed, length): prev_a = seed
result = ""
result = b"" while len(result) < length: a = hmac.new(secret, msg=prev_a, digestmod=hashlib.sha256) prev_a = a.digest()@@ -372,7 +377,7 @@ class SecureSocket(object): raise SecSocketException("Socket without a role!") cryptography_imports()
aes_keysize = max(algorithms.AES.key_sizes)/8
aes_keysize = max(algorithms.AES.key_sizes)//8 mac_keysize = hashlib.sha256().block_size prf_seq = self.PRF(self._master_secret,diff --git a/lnst/Common/ShellProcess.py b/lnst/Common/ShellProcess.py index a4928ed..45dedff 100644 --- a/lnst/Common/ShellProcess.py +++ b/lnst/Common/ShellProcess.py @@ -229,7 +229,7 @@ class ShellProcess: except: return data if r and (r[0][1] & select.POLLIN):
new_data = os.read(fd, 1024)
new_data = os.read(fd, 1024).decode() if not new_data: return data data += new_datadiff --git a/lnst/Common/Utils.py b/lnst/Common/Utils.py index d6d6c57..4369278 100644 --- a/lnst/Common/Utils.py +++ b/lnst/Common/Utils.py @@ -197,7 +197,7 @@ def get_module_tools(module_path): return tools
def recursive_dict_update(original, update):
- for key, value in update.iteritems():
- for key, value in list(update.items()): if isinstance(value, collections.Mapping): r = recursive_dict_update(original.get(key, {}), value) original[key] = r
@@ -240,7 +240,7 @@ def list_to_dot(original_list, prefix="", key=""):
def dict_to_dot(original_dict, prefix=""): return_list = []
- for key, value in original_dict.iteritems():
- for key, value in list(original_dict.items()): if isinstance(value, collections.Mapping): sub_list = dict_to_dot(value, prefix + key + '.') return_list.extend(sub_list)
diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py index b24eb65..2379505 100644 --- a/lnst/Controller/Machine.py +++ b/lnst/Controller/Machine.py @@ -17,7 +17,7 @@ import os import tempfile import signal from time import sleep -from xmlrpclib import Binary +from xmlrpc.client import Binary from functools import wraps from lnst.Common.Config import lnst_config from lnst.Common.NetUtils import normalize_hwaddr @@ -144,7 +144,7 @@ class Machine(object): del self._device_database[update_msg["if_index"]]
def dev_db_get_name(self, dev_name):
for if_index, dev in self._device_database.iteritems():
for if_index, dev in self._device_database.items(): if dev.get_name() == dev_name: return dev return None@@ -308,7 +308,7 @@ class Machine(object): self._slave_desc = slave_desc
devices = self._rpc_call("get_devices")
for if_index, dev in devices.items():
for if_index, dev in list(devices.items()): self._device_database[if_index] = Device(dev, self) for iface in self._interfaces:@@ -552,8 +552,8 @@ class Machine(object): def sync_resources(self, required): self._rpc_call("clear_resource_table")
for res_type, resources in required.iteritems():for res_name, res in resources.iteritems():
for res_type, resources in required.items():for res_name, res in resources.items(): has_resource = self._rpc_call("has_resource", res["hash"]) if not has_resource: msg = "Transfering %s %s to machine %s" % \@@ -889,7 +889,7 @@ class Interface(object): "network_label": self._network, "type": self._type, "addresses": self._addresses,
"slaves": self._slaves.keys(),
"slaves": list(self._slaves.keys()), "options": self._options, "slave_options": self._slave_options, "master": None,@@ -1422,14 +1422,24 @@ class Device(object):
@pre_call_decorate def get_ip_addrs(self, selector={}):
return [ip["addr"]for ip in self._ip_addrsif selector.items() <= ip.items()]
result = []for addr in self._ip_addrs:match = Truefor sel_item, value in selector.items():try:if addr.get(sel_item, None) != value:match = Falsebreakexcept:match = Falseif match:result.append(addr)return result@pre_call_decorate def get_ip_addr(self, num, selector={}): ips = self.get_ip_addrs(selector)
return ips[num]
return ips[num].get('addr')
I think the .get('addr') should be in the get_ip_addrs method call. I thought that's what we agreed on to keep stability of the API?
@pre_call_decorate def get_ifi_type(self):diff --git a/lnst/Controller/NetTestController.py b/lnst/Controller/NetTestController.py index 860f727..66a5cc2 100644 --- a/lnst/Controller/NetTestController.py +++ b/lnst/Controller/NetTestController.py @@ -15,7 +15,7 @@ import logging import socket import os import re -import cPickle +import pickle import imp import copy import sys @@ -194,7 +194,7 @@ class NetTestController: recipe = self._recipe
machines = self._machines
for m_id in machines.keys():
for m_id in list(machines.keys()): self._prepare_machine(m_id, resource_sync) for machine_xml_data in recipe["machines"]:@@ -243,9 +243,9 @@ class NetTestController: logging.info("Pool match description:") if sp.is_setup_virtual(): logging.info(" Setup is using virtual machines.")
for m_id, m in sorted(match["machines"].iteritems()):
for m_id, m in sorted(match["machines"].items()): logging.info(" host \"%s\" uses \"%s\"" % (m_id, m["target"]))
for if_id, match in m["interfaces"].iteritems():
for if_id, match in m["interfaces"].items(): pool_id = match["target"] logging.info(" interface \"%s\" matched to \"%s\"" %\ (if_id, pool_id))@@ -521,7 +521,7 @@ class NetTestController: if self._machines == None: return
for machine_id, machine in self._machines.iteritems():
for machine_id, machine in self._machines.items(): if machine.is_configured(): try: machine.cleanup(deconfigure)@@ -536,7 +536,7 @@ class NetTestController:
# remove dynamically created bridges if deconfigure:
for bridge in self._network_bridges.itervalues():
for bridge in self._network_bridges.values(): bridge.cleanup() self._network_bridges = {}@@ -544,7 +544,7 @@ class NetTestController: #saves current virtual configuration to a file, after pickling it config_data = dict() machines = config_data["machines"] = {}
for m in self._machines.itervalues():
for m in self._machines.values(): machine = machines[m.get_hostname()] = dict() if m.get_libvirt_domain() != None:@@ -560,12 +560,12 @@ class NetTestController: machine["interfaces"].append(hwaddr)
config_data["bridges"] = bridges = []
for bridge in self._network_bridges.itervalues():
for bridge in self._network_bridges.values(): bridges.append(bridge.get_name()) with open("/tmp/.lnst_machine_conf", "wb") as f: os.fchmod(f.fileno(), 0o600)
pickled_data = cPickle.dump(config_data, f)
pickled_data = pickle.dump(config_data, f)@classmethod def remove_saved_machine_config(cls):
@@ -573,7 +573,7 @@ class NetTestController: cfg = None try: with open("/tmp/.lnst_machine_conf", "rb") as f:
cfg = cPickle.load(f)
cfg = pickle.load(f) except: logging.info("No previous configuration found.") return@@ -581,7 +581,7 @@ class NetTestController: if cfg: logging.info("Cleaning up leftover configuration from previous "\ "config_only run.")
for hostname, machine in cfg["machines"].iteritems():
for hostname, machine in cfg["machines"].items(): port = lnst_config.get_option("environment", "rpcport") if test_tcp_connection(hostname, port): s = socket.create_connection((hostname, port))@@ -689,7 +689,7 @@ class NetTestController: overall_res["err_msg"] = "Command exception raised." break
for machine in self._machines.itervalues():
for machine in self._machines.values(): machine.restore_system_config() # task failed, check if we should quit_on_fail@@ -790,13 +790,13 @@ class NetTestController:
def _start_packet_capture(self): logging.info("Starting packet capture")
for machine_id, machine in self._machines.iteritems():
for machine_id, machine in self._machines.items(): capture_files = machine.start_packet_capture() self._remote_capture_files[machine_id] = capture_filesdef _stop_packet_capture(self): logging.info("Stopping packet capture")
for machine_id, machine in self._machines.iteritems():
for machine_id, machine in self._machines.items(): machine.stop_packet_capture()# TODO: Move this function to logging
@@ -804,7 +804,7 @@ class NetTestController: logging_root = self._log_ctl.get_recipe_log_path() logging_root = os.path.abspath(logging_root) logging.info("Retrieving capture files from slaves")
for machine_id, machine in self._machines.iteritems():
for machine_id, machine in self._machines.items(): slave_logging_dir = os.path.join(logging_root, machine_id + "/") try: os.mkdir(slave_logging_dir)@@ -815,7 +815,7 @@ class NetTestController: raise NetTestError(msg)
capture_files = self._remote_capture_files[machine_id]
for if_id, remote_path in capture_files.iteritems():
for if_id, remote_path in capture_files.items(): filename = "%s.pcap" % if_id local_path = os.path.join(slave_logging_dir, filename) machine.copy_file_from_machine(remote_path, local_path)@@ -896,11 +896,11 @@ class MessageDispatcher(ConnectionHandler): def wait_for_result(self, machine_id): wait = True while wait:
connected_slaves = self._connection_mapping.keys()
connected_slaves = list(self._connection_mapping.keys()) messages = self.check_connections()
remaining_slaves = self._connection_mapping.keys()
remaining_slaves = list(self._connection_mapping.keys()) for msg in messages: if msg[1]["type"] == "result" and msg[0] == machine_id:diff --git a/lnst/Controller/NetTestResultSerializer.py b/lnst/Controller/NetTestResultSerializer.py index e4fe086..7439c64 100644 --- a/lnst/Controller/NetTestResultSerializer.py +++ b/lnst/Controller/NetTestResultSerializer.py @@ -94,10 +94,10 @@ class NetTestResultSerializer: "Setup is using virtual machines.", ""))
for m_id, m in sorted(match["machines"].iteritems()):
for m_id, m in sorted(match["machines"].items()): output_pairs.append((4*" " + "host \"%s\" uses \"%s\"" %\ (m_id, m["target"]), ""))
for if_id, pool_if in m["interfaces"].iteritems():
for if_id, pool_if in m["interfaces"].items(): pool_id = pool_if["target"] if "driver" in pool_if: driver = pool_if["driver"]@@ -227,12 +227,12 @@ class NetTestResultSerializer: else: match_el.setAttribute("virtual", "false")
for m_id, m in match["machines"].iteritems():
for m_id, m in match["machines"].items(): m_el = doc.createElement("m_match") m_el.setAttribute("host_id", str(m_id)) m_el.setAttribute("pool_id", str(m["target"]))
for if_id, pool_id in m["interfaces"].iteritems():
for if_id, pool_id in m["interfaces"].items(): if_el = doc.createElement("if_match") if_el.setAttribute("if_id", str(if_id)) if_el.setAttribute("pool_if_id", str(pool_id))diff --git a/lnst/Controller/RecipeParser.py b/lnst/Controller/RecipeParser.py index 742c144..d7f70ce 100644 --- a/lnst/Controller/RecipeParser.py +++ b/lnst/Controller/RecipeParser.py @@ -386,16 +386,16 @@ class RecipeParser(XmlParser): distribution = False valid_distributions = ["normal", "uniform", "pareto", "paretonormal"] for opt in options:
if "time" in opt.values():
if "time" in list(opt.values()): valid = True
elif "distribution" in opt.values():
elif "distribution" in list(opt.values()): if opt["value"] not in valid_distributions: raise RecipeError("netem: invalid distribution type", netem_tag) else: distribution = True
elif "jitter" in opt.values():
elif "jitter" in list(opt.values()): jitter = True
elif "correlation" in opt.values():
elif "correlation" in list(opt.values()): correlation = True if not jitter: if correlation or distribution:@@ -404,22 +404,22 @@ class RecipeParser(XmlParser): raise RecipeError("netem: time option is mandatory for <delay>", netem_tag) elif netem_op == "loss": for opt in options:
if "percent" in opt.values():
if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <loss>", netem_tag) elif netem_op == "duplication": for opt in options:
if "percent" in opt.values():
if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <duplication>", netem_tag) elif netem_op == "corrupt": for opt in options:
if "percent" in opt.values():
if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <corrupt>", netem_tag) elif netem_op == "reordering": for opt in options:
if "percent" in opt.values():
if "percent" in list(opt.values()): return raise RecipeError("netem: percent option is mandatory for <reordering>", netem_tag)diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py index 5069f50..8166816 100644 --- a/lnst/Controller/SlavePool.py +++ b/lnst/Controller/SlavePool.py @@ -49,7 +49,7 @@ class SlavePool: self._mreqs = None
logging.info("Checking machine pool availability.")
for pool_name, pool_dir in pools.items():
for pool_name, pool_dir in list(pools.items()): self._pools[pool_name] = {} self.add_dir(pool_name, pool_dir) if len(self._pools[pool_name]) == 0:@@ -85,13 +85,13 @@ class SlavePool: dir_path))
max_len = 0
for m_id in pool.keys():
for m_id in list(pool.keys()): if len(m_id) > max_len: max_len = len(m_id) if self._pool_checks: check_sockets = {}
for m_id, m in sorted(pool.iteritems()):
for m_id, m in sorted(pool.items()): hostname = m["params"]["hostname"] if "rpc_port" in m["params"]: port = m["params"]["rpc_port"]@@ -123,7 +123,7 @@ class SlavePool: check_sockets[s] = m_id
while len(check_sockets) > 0:
rl, wl, el = select.select([], check_sockets.keys(), [])
rl, wl, el = select.select([], list(check_sockets.keys()), []) for s in wl: err = s.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) m_id = check_sockets[s]@@ -137,7 +137,7 @@ class SlavePool: s.close() del check_sockets[s] else:
for m_id in pool.keys():
for m_id in list(pool.keys()): pool[m_id]["available"] = True for m_id in sorted(list(pool.keys())):@@ -178,7 +178,7 @@ class SlavePool:
# Check if there isn't any machine with the same # hostname or libvirt_domain already in the pool
for pm_id, m in pool.iteritems():
for pm_id, m in pool.items(): pm = m["params"] rm = machine_spec["params"] if pm["hostname"] == rm["hostname"]:@@ -228,7 +228,7 @@ class SlavePool: raise SlaveMachineError(msg, iface)
if_hwaddr = iface_spec["params"]["hwaddr"]
hwaddr_dups = [ k for k, v in machine_spec["interfaces"].iteritems()\
hwaddr_dups = [ k for k, v in machine_spec["interfaces"].items()\ if v["params"]["hwaddr"] == if_hwaddr ] if len(hwaddr_dups) > 0: msg = "Duplicate MAC address %s for interface '%s' and '%s'."\@@ -348,7 +348,7 @@ class SlavePool:
used = [] if_map = self._map["machines"][tm_id]["interfaces"]
for t_if, p_if in if_map.iteritems():
for t_if, p_if in if_map.items(): pool_id = p_if["target"] used.append(pool_id) if_data = pm["interfaces"][pool_id]@@ -356,12 +356,12 @@ class SlavePool: iface = machine.new_static_interface(t_if, "eth") iface.set_hwaddr(if_data["params"]["hwaddr"])
for t_net, p_net in self._map["networks"].iteritems():
for t_net, p_net in self._map["networks"].items(): if pm["interfaces"][pool_id]["network"] == p_net: iface.set_network(t_net) break
for if_id, if_data in pm["interfaces"].iteritems():
for if_id, if_data in pm["interfaces"].items(): if if_id not in used: iface = machine.new_unused_interface("eth") iface.set_hwaddr(if_data["params"]["hwaddr"])@@ -384,13 +384,13 @@ class SlavePool: pm["security"])
# make all the existing unused
for if_id, if_data in pm["interfaces"].iteritems():
for if_id, if_data in pm["interfaces"].items(): iface = machine.new_unused_interface("eth") iface.set_hwaddr(if_data["params"]["hwaddr"]) iface.set_network(None) # add all the other devices
for if_id, if_data in tm["interfaces"].iteritems():
for if_id, if_data in tm["interfaces"].items(): iface = machine.new_virtual_interface(if_id, "eth") iface.set_network(if_data["network"]) if "hwaddr" in if_data["params"]:@@ -425,10 +425,10 @@ class SetupMapper(object): def set_virtual(self, virt_value): self._virtual_matching = virt_value
for m_id, m in self._mreqs.iteritems():for if_id, interface in m["interfaces"].iteritems():
for m_id, m in self._mreqs.items():for if_id, interface in m["interfaces"].items(): if "params" in interface:
for name, val in interface["params"].iteritems():
for name, val in interface["params"].items(): if name not in ["hwaddr", "driver"]: msg = "Dynamically created interfaces "\ "only support the 'hwaddr' and 'driver' "\@@ -443,7 +443,7 @@ class SetupMapper(object): def reset_match_state(self): self._net_label_mapping = {} self._machine_stack = []
self._unmatched_req_machines = sorted(self._mreqs.keys(), reverse=True)
self._unmatched_req_machines = sorted(list(self._mreqs.keys()), reverse=True) self._pool_stack = list(self._pools.keys()) if len(self._pool_stack) > 0:@@ -451,7 +451,7 @@ class SetupMapper(object): self._pool = self._pools[self._pool_name]
self._unmatched_pool_machines = []
for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True):
for p_id, p_machine in sorted(iter(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id)@@ -492,7 +492,7 @@ class SetupMapper(object): #map compatible pool machine stack_top["current_match"] = pool_m_id stack_top["unmatched_pool_ifs"] = \
sorted(self._pool[pool_m_id]["interfaces"].keys(),
sorted(list(self._pool[pool_m_id]["interfaces"].keys()), reverse=True) self._unmatched_pool_machines.remove(pool_m_id) break@@ -516,7 +516,7 @@ class SetupMapper(object): self._pool_name)
self._unmatched_pool_machines = []
for p_id, p_machine in sorted(self._pool.iteritems(), reverse=True):
for p_id, p_machine in sorted(iter(self._pool.items()), reverse=True): if self._virtual_matching: if "libvirt_domain" in p_machine["params"]: self._unmatched_pool_machines.append(p_id)@@ -591,7 +591,7 @@ class SetupMapper(object): machine_match["if_stack"] = []
machine = self._mreqs[machine_match["m_id"]]
machine_match["unmatched_ifs"] = sorted(machine["interfaces"].keys(),
machine_match["unmatched_ifs"] = sorted(list(machine["interfaces"].keys()), reverse=True) machine_match["unmatched_pool_ifs"] = []@@ -621,7 +621,7 @@ class SetupMapper(object): def _check_machine_compatibility(self, req_id, pool_id): req_machine = self._mreqs[req_id] pool_machine = self._pool[pool_id]
for param, value in req_machine["params"].iteritems():
for param, value in req_machine["params"].items(): # skip empty parameters if len(value) == 0: continue@@ -632,14 +632,14 @@ class SetupMapper(object):
def _check_interface_compatibility(self, req_if, pool_if): label_mapping = self._net_label_mapping
for req_label, mapping in label_mapping.iteritems():
for req_label, mapping in label_mapping.items(): if req_label == req_if["network"] and\ mapping[0] != pool_if["network"]: return False if mapping[0] == pool_if["network"] and\ req_label != req_if["network"]: return False
for param, value in req_if["params"].iteritems():
for param, value in req_if["params"].items(): # skip empty parameters if len(value) == 0: continue@@ -652,7 +652,7 @@ class SetupMapper(object): mapping = {"machines": {}, "networks": {}, "virtual": False, "pool_name": self._pool_name}
for req_label, label_map in self._net_label_mapping.iteritems():
for req_label, label_map in self._net_label_mapping.items(): mapping["networks"][req_label] = label_map[0] for machine in self._machine_stack:diff --git a/lnst/Controller/Task.py b/lnst/Controller/Task.py index bc7670f..e2b98d0 100644 --- a/lnst/Controller/Task.py +++ b/lnst/Controller/Task.py @@ -45,7 +45,7 @@ class ControllerAPI(object): self._perf_repo_api = PerfRepoAPI()
self._hosts = {}
for host_id, host in hosts.iteritems():
for host_id, host in hosts.items(): self._hosts[host_id] = HostAPI(self, host_id, host)def _run_command(self, command):
@@ -161,20 +161,20 @@ class ControllerAPI(object): def get_configuration(self): machines = self._ctl._machines configuration = {}
for m_id, m in machines.items():
for m_id, m in list(machines.items()): configuration["machine_"+m_id] = m.get_configuration() return configurationdef get_mapping(self): match = self._ctl.get_pool_match() mapping = []
for m_id, m in match["machines"].iteritems():
for m_id, m in match["machines"].items(): machine = {} machine["id"] = m_id machine["pool_id"] = m["target"] machine["hostname"] = m["hostname"] machine["interface"] = []
for i_id, i in m["interfaces"].iteritems():
for i_id, i in m["interfaces"].items(): interface = {} interface["id"] = i_id interface["pool_id"] = i["target"]@@ -253,7 +253,7 @@ class HostAPI(object): bg_id = None cmd["netns"] = None
for arg, argval in kwargs.iteritems():
for arg, argval in kwargs.items(): if arg == "bg" and argval == True: self._bg_id_seq += 1 cmd["bg_id"] = bg_id = self._bg_id_seq@@ -864,7 +864,7 @@ class ModuleAPI(object): self._name = module_name
self._opts = {}
for opt, val in options.iteritems():
for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val:@@ -877,7 +877,7 @@ class ModuleAPI(object):
def set_options(self, options): self._opts = {}
for opt, val in options.iteritems():
for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val:@@ -886,7 +886,7 @@ class ModuleAPI(object): self._opts[opt].append({"value": str(val)})
def update_options(self, options):
for opt, val in options.iteritems():
for opt, val in options.items(): self._opts[opt] = [] if type(val) == list: for v in val:diff --git a/lnst/Controller/Wizard.py b/lnst/Controller/Wizard.py index dc82c21..85fb88b 100644 --- a/lnst/Controller/Wizard.py +++ b/lnst/Controller/Wizard.py @@ -106,7 +106,7 @@ class Wizard:
rv = self._check_path(pool_dir) if rv == PATH_IS_DIR_ACCESSIBLE:
print("Pool directory set to '%s'" % pool_dir)
print(("Pool directory set to '%s'" % pool_dir))
^^ i don't think this is necessary?
elif rv == PATH_DOES_NOT_EXIST: sys.stderr.write("Path '%s' does not exist\n" % pool_dir) pool_dir = self._create_dir(pool_dir)@@ -124,7 +124,7 @@ class Wizard: return
for host in hostlist:
print("Processing host '%s'" % host)
print(("Processing host '%s'" % host))
^^ same here
hostname, port = self._parse_host(host) if hostname == -1: continue@@ -222,7 +222,7 @@ class Wizard: """ while True: if pool_dir is None:
pool_dir = raw_input("Enter path to a pool directory "
pool_dir = input("Enter path to a pool directory " "(default: '%s'): " % DefaultPoolDir) if pool_dir == "": pool_dir = DefaultPoolDir@@ -230,7 +230,7 @@ class Wizard: pool_dir = os.path.expanduser(pool_dir) rv = self._check_path(pool_dir) if rv == PATH_IS_DIR_ACCESSIBLE:
print("Pool directory set to '%s'" % pool_dir)
print(("Pool directory set to '%s'" % pool_dir))
^^ same here
return pool_dir elif rv == PATH_DOES_NOT_EXIST: sys.stderr.write("Path '%s' does not exist\n"@@ -270,7 +270,7 @@ class Wizard: """ try: mkdir_p(pool_dir)
print("Dir '%s' has been created" % pool_dir)
print(("Dir '%s' has been created" % pool_dir))
^^ same here
return pool_dir except: sys.stderr.write("Failed creating dir\n")@@ -316,7 +316,7 @@ class Wizard: if mode == "interactive": msg = "Do you want to add interface '%s' (%s) to the "\ "recipe? [Y/n]: " % (iface["name"], iface["hwaddr"])
answer = raw_input(msg)
answer = input(msg) if mode == "noninteractive" or answer.lower() == "y"\ or answer == "": interfaces_added += 1@@ -359,7 +359,7 @@ class Wizard: pubkey_el.appendChild(pubkey_text)
if self._write_to_file(pool_dir, filename, doc):
print("File '%s/%s' successfuly created." % (pool_dir, filename))
print(("File '%s/%s' successfuly created." % (pool_dir, filename)))
^^ same here
else: sys.stderr.write("File '%s/%s' could not be opened " "or data written.\n" % (pool_dir, filename))@@ -431,7 +431,7 @@ class Wizard: """ Queries user for adding next machine @return True if user wants to add another machine, False otherwise """
answer = raw_input("Do you want to add another machine? [Y/n]: ")
answer = input("Do you want to add another machine? [Y/n]: ") if answer.lower() == "y" or answer == "": return True else:@@ -441,7 +441,7 @@ class Wizard: """ Queries user for creating specified directory @return True if user wants to create the directory, False otherwise """
answer = raw_input("Create dir '%s'? [Y/n]: " % pool_dir)
answer = input("Create dir '%s'? [Y/n]: " % pool_dir) if answer.lower() == 'y' or answer == "": return True else:@@ -452,7 +452,7 @@ class Wizard: @hostname Hostname of the machine which is used as default filename @return Name of the file with .xml extension """
output_file = raw_input("Enter the name of the output .xml file "
output_file = input("Enter the name of the output .xml file " "(without .xml, default is '%s.xml'): " % hostname) if output_file == "":@@ -465,7 +465,7 @@ class Wizard: @return Valid (is translatable to an IP address) hostname """ while True:
hostname = raw_input("Enter hostname: ")
hostname = input("Enter hostname: ") if hostname == "": sys.stderr.write("No hostname entered\n") continue@@ -485,7 +485,7 @@ class Wizard: string representing hostname of the host """ while True:
libvirt_domain = raw_input("Enter libvirt domain "
libvirt_domain = input("Enter libvirt domain " "of virtual host: ") if libvirt_domain == "": sys.stderr.write("No domain entered\n")@@ -524,7 +524,7 @@ class Wizard: @return Integer representing port """ while True:
port = raw_input("Enter port (default: %d): " % DefaultRPCPort)
port = input("Enter port (default: %d): " % DefaultRPCPort) if port == "": return DefaultRPCPort else:@@ -539,7 +539,7 @@ class Wizard: @return Dictionary with the security parameters """ while True:
auth_type = raw_input("Enter authentication type (default: none): ")
auth_type = input("Enter authentication type (default: none): ") if auth_type == "": auth_type = "none" elif auth_type not in ["none", "no-auth", "password",@@ -555,7 +555,7 @@ class Wizard: return {"auth_type": "ssh"} elif auth_type == "password": while True:
password = raw_input("Enter password: ")
password = input("Enter password: ") if password == "": sys.stderr.write("Invalid password.") continue@@ -564,19 +564,19 @@ class Wizard: "auth_passwd": password} elif auth_type == "pubkey": while True:
identity = raw_input("Enter identity: ")
identity = input("Enter identity: ") if identity == "": sys.stderr.write("Invalid identity.") continue break while True:
privkey = raw_input("Enter path to Ctl private key: ")
privkey = input("Enter path to Ctl private key: ") if privkey == "" or not os.path.isfile(privkey): sys.stderr.write("Invalid path to private key.") continue break while True:
srv_pubkey_path = raw_input("Enter path to Slave public key: ")
srv_pubkey_path = input("Enter path to Slave public key: ") if srv_pubkey_path == "" or not os.path.isfile(srv_pubkey_path): sys.stderr.write("Invalid path to public key.") continuediff --git a/lnst/Controller/XmlParser.py b/lnst/Controller/XmlParser.py index 355b5e8..cbe3569 100644 --- a/lnst/Controller/XmlParser.py +++ b/lnst/Controller/XmlParser.py @@ -15,7 +15,7 @@ import re import sys import copy from lxml import etree -from urllib2 import urlopen +from urllib.request import urlopen from lnst.Common.Config import lnst_config from lnst.Controller.XmlTemplates import XmlTemplates from lnst.Controller.XmlProcessing import XmlProcessingError @@ -124,7 +124,7 @@ class XmlParser(object): return self._template_proc.expand_functions(text)
def _get_content(self, element):
text = etree.tostring(element, method="text").strip()
text = etree.tostring(element, method="text", encoding="unicode").strip() return self._template_proc.expand_functions(text)def _expand_xinclude(self, elem, base_url=""):
diff --git a/lnst/Controller/XmlProcessing.py b/lnst/Controller/XmlProcessing.py index b80c3a3..8e2bb00 100644 --- a/lnst/Controller/XmlProcessing.py +++ b/lnst/Controller/XmlProcessing.py @@ -75,8 +75,8 @@ class XmlDataIterator: def __iter__(self): return self
- def next(self):
n = self._iterator.next()
def __next__(self):
n = next(self._iterator) # For normal iterators if type(n) == XmlTemplateString:@@ -165,20 +165,20 @@ class XmlData(dict): return XmlDataIterator(it)
def iteritems(self):
it = super(XmlData, self).iteritems()
it = iter(super(XmlData, self).items()) return XmlDataIterator(it)def iterkeys(self):
it = super(XmlData, self).iterkeys()
it = iter(super(XmlData, self).keys()) return XmlDataIterator(it)def itervalues(self):
it = super(XmlData, self).itervalues()
it = iter(super(XmlData, self).values()) return XmlDataIterator(it)def to_dict(self): new_dict = dict()
for key, value in self.iteritems():
for key, value in self.items(): if isinstance(value, XmlData): new_val = value.to_dict() elif isinstance(value, XmlCollection):diff --git a/lnst/Controller/XmlTemplates.py b/lnst/Controller/XmlTemplates.py index b1bf9a1..4068e85 100644 --- a/lnst/Controller/XmlTemplates.py +++ b/lnst/Controller/XmlTemplates.py @@ -239,7 +239,7 @@ class XmlTemplates: """ defs = {} for level in self._definitions:
for name, val in level.iteritems():
for name, val in level.items(): defs[name] = val return defs@@ -255,7 +255,7 @@ class XmlTemplates: def set_aliases(self, defined, overriden): """ Set aliases defined or overriden from CLI """
for name, value in defined.iteritems():
for name, value in defined.items(): self.define_alias(name, value) self._overriden_aliases = overriden@@ -325,7 +325,7 @@ class XmlTemplates: if element.tail != None: element.tail = self.expand_aliases(element.tail)
for name, value in element.attrib.iteritems():
for name, value in element.attrib.items(): element.set(name, self.expand_aliases(value)) if element.tag == "define":diff --git a/lnst/Slave/InterfaceManager.py b/lnst/Slave/InterfaceManager.py index a96ddd1..47c1e2a 100644 --- a/lnst/Slave/InterfaceManager.py +++ b/lnst/Slave/InterfaceManager.py @@ -79,7 +79,7 @@ class InterfaceManager(object): self._id_mapping = {}
def get_id_by_if_index(self, if_index):
for if_id, index in self._id_mapping.iteritems():
for if_id, index in self._id_mapping.items(): if if_index == index: return if_id return None@@ -97,12 +97,12 @@ class InterfaceManager(object): return self._nl_socket
def rescan_devices(self):
devices_to_remove = self._devices.keys()
devices_to_remove = list(self._devices.keys()) devs = scan_netdevs() for dev in devs: if dev['index'] not in self._devices: device = None
for if_id, d in self._tmp_mapping.items():
for if_id, d in list(self._tmp_mapping.items()): d_cfg = d.get_conf_dict() if d_cfg["name"] == dev["name"]: device = d@@ -134,7 +134,7 @@ class InterfaceManager(object): del self._devices[i]
self._dl_manager.rescan_ports()
for device in self._devices.values():
for device in list(self._devices.values()): dl_port = self._dl_manager.get_port(device.get_name()) device.set_devlink(dl_port)@@ -143,7 +143,7 @@ class InterfaceManager(object): self._handle_netlink_msg(msg)
self._dl_manager.rescan_ports()
for device in self._devices.values():
for device in list(self._devices.values()): dl_port = self._dl_manager.get_port(device.get_name()) device.set_devlink(dl_port)@@ -152,14 +152,14 @@ class InterfaceManager(object): if msg['index'] in self._devices: update_msg = self._devices[msg['index']].update_netlink(msg) if update_msg != None:
for if_id, if_index in self._id_mapping.iteritems():
for if_id, if_index in self._id_mapping.items(): if if_index == msg['index']: update_msg["if_id"] = if_id break self._server_handler.send_data_to_ctl(update_msg) elif msg['header']['type'] == RTM_NEWLINK: dev = None
for if_id, d in self._tmp_mapping.items():
for if_id, d in list(self._tmp_mapping.items()): d_cfg = d.get_conf_dict() if d_cfg["name"] == msg.get_attr("IFLA_IFNAME"): dev = d@@ -172,7 +172,7 @@ class InterfaceManager(object): self._devices[msg['index']] = dev
if update_msg != None:
for if_id, if_index in self._id_mapping.iteritems():
for if_id, if_index in self._id_mapping.items(): if if_index == msg['index']: update_msg["if_id"] = if_id break@@ -202,7 +202,7 @@ class InterfaceManager(object):
def get_mapped_devices(self): ret = {}
for if_id, if_index in self._id_mapping.iteritems():
for if_id, if_index in self._id_mapping.items(): ret[if_id] = self._devices[if_index] for if_id in self._tmp_mapping: ret[if_id] = self._tmp_mapping[if_id]@@ -215,20 +215,20 @@ class InterfaceManager(object): return None
def get_devices(self):
return self._devices.values()
return list(self._devices.values())def get_device_by_hwaddr(self, hwaddr):
for dev in self._devices.values():
for dev in list(self._devices.values()): if dev.get_hwaddr() == hwaddr: return dev return Nonedef get_device_by_params(self, params): matched = None
for dev in self._devices.values():
for dev in list(self._devices.values()): matched = dev dev_data = dev.get_if_data()
for key, value in params.iteritems():
for key, value in params.items(): if key not in dev_data or dev_data[key] != value: matched = None break@@ -239,7 +239,7 @@ class InterfaceManager(object): return matched
def deconfigure_all(self):
for dev in self._devices.itervalues():
for dev in self._devices.values(): dev.clear_configuration()def create_device_from_config(self, if_id, config):
@@ -288,10 +288,10 @@ class InterfaceManager(object):
def _is_name_used(self, name): self.rescan_devices()
for device in self._devices.itervalues():
for device in self._devices.values(): if name == device.get_name(): return True
for device in self._tmp_mapping.itervalues():
for device in self._tmp_mapping.values(): if name == device.get_name(): return True return False@@ -487,7 +487,7 @@ class Device(object): def find_addrs(self, addr_spec): ret = [] for addr in self._ip_addrs:
if addr_spec.items() <= addr.items():
if list(addr_spec.items()) <= list(addr.items()): ret.append(addr) return ret@@ -703,7 +703,7 @@ class Device(object): if (line.split()[0] == 'vf'): break if (line.split()[0] == "RX:"):
rx_stats = map(int, lines.next().split())
rx_stats = list(map(int, next(lines).split())) stats.update({"rx_bytes" : rx_stats[0], "rx_packets": rx_stats[1], "rx_errors" : rx_stats[2],@@ -711,7 +711,7 @@ class Device(object): "rx_overrun": rx_stats[4], "rx_mcast" : rx_stats[5]}) if (line.split()[0] == "TX:"):
tx_stats = map(int, lines.next().split())
tx_stats = list(map(int, next(lines).split())) stats.update({"tx_bytes" : tx_stats[0], "tx_packets": tx_stats[1], "tx_errors" : tx_stats[2],@@ -742,7 +742,7 @@ class Device(object): stats_data[i] = stats_data[i].replace("K", "000") stats_data[i] = stats_data[i].replace("M", "000000")
stats_data = map(int, stats_data)
stats_data = list(map(int, stats_data)) stats["rx_packets"] = stats_data[0] stats["tx_packets"] = stats_data[2] stats["rx_bytes"] = stats_data[4]diff --git a/lnst/Slave/NetConfigDevice.py b/lnst/Slave/NetConfigDevice.py index 03efd40..dec992f 100644 --- a/lnst/Slave/NetConfigDevice.py +++ b/lnst/Slave/NetConfigDevice.py @@ -585,7 +585,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bond_ports = []
for bond in self._dev_config["ovs_conf"]["bonds"].itervalues():
for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id)@@ -600,7 +600,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): options += " %s=%s" % (opt[0], opt[1])
vlan_tags = []
for tag, vlan in vlans.iteritems():
for tag, vlan in vlans.items(): if slave_id in vlan["slaves"]: vlan_tags.append(tag) if len(vlan_tags) == 0:@@ -620,7 +620,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bond_ports = []
for bond in self._dev_config["ovs_conf"]["bonds"].itervalues():
for bond in self._dev_config["ovs_conf"]["bonds"].values(): for slave_id in bond["slaves"]: bond_ports.append(slave_id)@@ -687,7 +687,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bonds = self._dev_config["ovs_conf"]["bonds"]
for bond_id, bond in bonds.iteritems():
for bond_id, bond in bonds.items(): ifaces = "" for slave_id in bond["slaves"]: slave_dev = self._if_manager.get_mapped_device(slave_id)@@ -703,7 +703,7 @@ class NetConfigDeviceOvsBridge(NetConfigDeviceGeneric): br_name = self._dev_config["name"]
bonds = self._dev_config["ovs_conf"]["bonds"]
for bond_id, bond in bonds.iteritems():
for bond_id, bond in bonds.items(): exec_cmd("ovs-vsctl del-port %s %s" % (br_name, bond_id))def _add_flow_entries(self):
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index 662faaf..7c61d23 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -22,7 +22,7 @@ import multiprocessing import re import struct from time import sleep, time -from xmlrpclib import Binary +from xmlrpc.client import Binary from tempfile import NamedTemporaryFile from lnst.Common.Logs import log_exc_traceback from lnst.Common.PacketCapture import PacketCapture @@ -181,7 +181,7 @@ class SlaveMethods: dev_data = dev.get_if_data() entry = {"name": dev.get_name(), "hwaddr": dev.get_hwaddr()}
for key, value in params.iteritems():
for key, value in params.items(): if key not in dev_data or dev_data[key] != value: entry = None break@@ -405,7 +405,7 @@ class SlaveMethods: raise Exception("Can't start packet capture, tcpdump not available")
files = {}
for if_id, dev in self._if_manager.get_mapped_devices().iteritems():
for if_id, dev in self._if_manager.get_mapped_devices().items(): if dev.get_netns() != None: continue dev_name = dev.get_name()@@ -431,7 +431,7 @@ class SlaveMethods: if self._packet_captures == None: return True
for if_index, pcap in self._packet_captures.iteritems():
for if_index, pcap in self._packet_captures.items(): pcap.stop() self._packet_captures.clear()@@ -439,7 +439,7 @@ class SlaveMethods: return True
def _remove_capture_files(self):
for key, name in self._capture_files.iteritems():
for key, name in self._capture_files.items(): logging.debug("Removing temporary packet capture file %s", name) os.unlink(name)@@ -462,7 +462,7 @@ class SlaveMethods:
def restore_system_config(self): logging.info("Restoring system configuration")
for option, values in self._system_config.iteritems():
for option, values in self._system_config.items(): try: cmd_str = "echo \"%s\" >%s" % (values["initial_val"], option) (stdout, stderr) = exec_cmd(cmd_str)@@ -538,7 +538,7 @@ class SlaveMethods: logging.info("Performing machine cleanup.") self._command_context.cleanup()
for mroute_soc in self.mroute_sockets.values():
for mroute_soc in list(self.mroute_sockets.values()): mroute_soc.close() del mroute_soc self.mroute_sockets = {}@@ -546,7 +546,7 @@ class SlaveMethods: self.restore_system_config()
devs = self._if_manager.get_mapped_devices()
for if_id, dev in devs.iteritems():
for if_id, dev in devs.items(): peer = dev.get_peer() if peer == None: dev.clear_configuration()@@ -557,7 +557,7 @@ class SlaveMethods:
self._if_manager.deconfigure_all()
for netns in self._net_namespaces.keys():
for netns in list(self._net_namespaces.keys()): self.del_namespace(netns) self._net_namespaces = {}@@ -641,11 +641,11 @@ class SlaveMethods: return False
def reset_file_transfers(self):
for file_handle in self._copy_targets.itervalues():
for file_handle in self._copy_targets.values(): file_handle.close() self._copy_targets = {}
for file_handle in self._copy_sources.itervalues():
for file_handle in self._copy_sources.values(): file_handle.close() self._copy_sources = {}@@ -1086,7 +1086,7 @@ class SlaveMethods: return True
def mroute_operation(self, op_type, op, table_id):
if not self.mroute_sockets.has_key(table_id):
if table_id not in self.mroute_sockets: logging.error("mroute %s table was not init", table_id) return False try:@@ -1098,7 +1098,7 @@ class SlaveMethods:
def mroute_init(self, table_id): logging.debug("Initializing mroute socket")
if not self.mroute_sockets.has_key(table_id):
if table_id not in self.mroute_sockets: self.mroute_sockets[table_id] = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IGMP)@@ -1157,7 +1157,7 @@ class SlaveMethods: (source, group, str(out_vifs)))
ttls = [0] * MROUTE.MAX_VIF
for vif, ttl in out_vifs.items():
for vif, ttl in list(out_vifs.items()): if vif >= MROUTE.MAX_VIF: logging.error("ilegal VIF was asked") return False@@ -1183,7 +1183,7 @@ class SlaveMethods: return self.mroute_operation(op_type, mfc_struct, table_id)
def mroute_get_notif(self, table_id):
if not self.mroute_sockets.has_key(table_id):
if table_id not in self.mroute_sockets: logging.error("mroute table %s was not init", table_id) return False try:@@ -1340,7 +1340,7 @@ class ServerHandler(ConnectionHandler): self._netns_con_mapping = {}
def update_connections(self, connections):
for key, connection in connections.iteritems():
for key, connection in connections.items(): self.remove_connection_by_id(key) self.add_connection(key, connection)@@ -1392,21 +1392,23 @@ class NetTestSlave: self._if_manager.get_nl_socket())
def run(self):
while not self._finished:if self._server_handler.get_ctl_sock() == None:self._log_ctl.cancel_connection()try:logging.info("Waiting for connection.")self._server_handler.accept_connection()except (socket.error, SecSocketException):continueself._log_ctl.set_connection(self._server_handler.get_ctl_sock())msgs = self._server_handler.get_messages()for msg in msgs:self._process_msg(msg[1])
while True:try:if self._server_handler.get_ctl_sock() is None:self._log_ctl.cancel_connection()try:logging.info("Waiting for connection.")self._server_handler.accept_connection()except (socket.error, SecSocketException):continueself._log_ctl.set_connection(self._server_handler.get_ctl_sock())msgs = self._server_handler.get_messages()for msg in msgs:self._process_msg(msg[1])except:break self._methods.machine_cleanup()@@ -1507,7 +1509,7 @@ class NetTestSlave:
def _signal_die_handler(self, signum, frame): logging.info("Caught signal %d -> dying" % signum)
self._finished = True
raise Exception("Recieved interrupt to system call")def _parent_resend_signal_handler(self, signum, frame): logging.info("Caught signal %d -> resending to parent" % signum)
diff --git a/lnst/Slave/NmConfigDevice.py b/lnst/Slave/NmConfigDevice.py index 29c2c75..74b4f85 100644 --- a/lnst/Slave/NmConfigDevice.py +++ b/lnst/Slave/NmConfigDevice.py @@ -218,10 +218,7 @@ class NmConfigDeviceGeneric(object): except: #IPv6 conversion into a 16 byte array tmp = socket.inet_pton(socket.AF_INET6, ip)
ip = []for i in tmp:ip.append(ord(i))ip = dbus.Array(ip, signature='y')
ip = dbus.Array(tmp, signature='y') def_gateway = dbus.Array([0]*16, signature='y') ipv6s.append(tuple([ip, dbus.UInt32(mask),diff --git a/setup.py b/setup.py index cc5d6f4..ebb05c4 100755 --- a/setup.py +++ b/setup.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python2 +#!/usr/bin/env python3 """ Install script for lnst
@@ -34,7 +34,7 @@ def process_template(template_path, values): t = open(template_path, "r") f = open(file_path, "w") template = t.read()
- for var, value in values.iteritems():
- for var, value in values.items(): template = template.replace("@%s@" % var, value) f.write(template) f.close()
diff --git a/test_modules/Multicast.py b/test_modules/Multicast.py index b366c9b..44a003c 100644 --- a/test_modules/Multicast.py +++ b/test_modules/Multicast.py @@ -61,7 +61,7 @@ class Multicast(TestGeneric):
cmd = "./{0} ".format(setup)
for optname, optval in opts.iteritems():
for optname, optval in list(opts.items()): if optval != None: cmd += "--{0} \"{1}\" ".format(optname, optval)diff --git a/test_modules/Netperf.py b/test_modules/Netperf.py index 00692e5..7e9fbd0 100644 --- a/test_modules/Netperf.py +++ b/test_modules/Netperf.py @@ -487,7 +487,7 @@ class Netperf(TestGeneric): results.append(result) rates.append(results[-1]["rate"])
if results > 1:
if len(results) > 1: res_data["results"] = results if len(rates) > 0:diff --git a/test_modules/TRexClient.py b/test_modules/TRexClient.py index f3ed2ee..60416e8 100644 --- a/test_modules/TRexClient.py +++ b/test_modules/TRexClient.py @@ -19,7 +19,7 @@ class TRexClient(TestGeneric): super(TRexClient, self).__init__(command)
self._trex_path = self.get_mopt("trex_path")
self._ports = map(int, self.get_multi_mopt("ports"))
self._ports = list(map(int, self.get_multi_mopt("ports"))) self._src_macs = [] self._dst_macs = []@@ -42,7 +42,7 @@ class TRexClient(TestGeneric): for res in results: new_results.append({}) new_res = new_results[-1]
for key, data in res.items():
for key, data in list(res.items()): if key in self._ports: new_res["port_"+str(key)] = data else:-- 2.17.1 _______________________________________________ LNST-developers mailing list -- lnst-developers@lists.fedorahosted.org To unsubscribe send an email to lnst-developers-leave@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/lnst-developers@lists.fedorahos...
Except for what I commented on inline there's one more thing I noticed. There's a lot of places where dict.items() was replaced with list(dict.items()) in loops where I think it wasn't necessary. This was done automatically by some 2to3 tool correct?
If so then it's probably fine as long as it works...
Overall good work. I'll try this out on real hardware and if everything works I'll apply it. But I think I'll fork it into a separate branch so that we have the python2 version available for now.
-Ondrej
lnst-developers@lists.fedorahosted.org
-
Jozef Urbanovsky -
Ondrej Lichtner