Mon, Feb 06, 2017 at 09:52:57AM CET, olichtne(a)redhat.com wrote:
Hi everyone,
I've got a new version of the API spec file.
ccing Petr.
>
>what changed:
>* I've renamed some stuff - Task -> Recipe (since the class handles both
> requirements and the test execution, calling it task makes no sense to
> me
>* commented out the "def test(m1, m2, m3...)" method of the BaseRecipe
> class, based on upstream discussion this might be confusing, hard to
> implement and if we can always decide later to add it (removing would
> be more problematic)
>* Added 'Controller' to the Tester facing API - provided by the LNST
> library to enable a tester to use the LNST controller from his own
> executable script
>* Host - added some discussion about the name "params"
> - removed 'tool' argument from run(), kept the 'path' argument
> because i didn't know if jpirko meant to remove them both or
> just the second one
>* Job - expanded basic description - how all Jobs will be technically in
> background, and fg/bg handling will be on the Controller.
>* added proposal for changing how the Result summary will look like
>
>I still haven't started working on the Device API due to reworking how
>Hosts will be matched and allocated (this is related to properly
>creating the Device objects...)
>
>Attaching the doc here:
>
>1. test modules
>
>class BaseTestModule:
> def __init__(self, **kwargs):
> #by defaults loads the params into self.params - no checks pseudocode:
> for x in vars(self):
> if isinstance(x, Param):
> param_class = self.getattr(x)
> try:
> val = kwargs[x]
> except KeyError:
> if param_class.is_mandatory():
> raise TestModuleError("Option x is mandatory")
> self.setattr(x.params, param_class.construct(val))
> del kwargs[x]
> for x in kwargs.keys():
> log.error("Undefined parameter x")
> if len(kwargs):
> raise TestModuleError("Undefined TestModule parameters")
>
> #check mandatory parameters
> for name, param in self.params:
> if param.mandatory and not param.set:
> raise TestModuleException("Parameter {} is
mandatory".format(name))
>
> def run():
> #needs to be over-ridden - throw an exception to notify the test developer
>
>
>class MyTest(BaseTestModule):
> param = ParamType(mandatory=True)
> param2 = ParamType2()
> param3 = Multiparam(ParamType())
>
> #optional __init__
> #def __init__(self, **kwargs):
> #super(MyTest).__init__(kwargs)
> #additional tester defined checks
>
> def run():
> #do my test
> #parameters available in self.params
>
> #all imports used by the test module need to happen HERE, reason is
> #that the object is instantiated on the Controller and THEN sent to the
> #slave -> stuff imported on the Controller is not available on the Slave
>
>#in Task:
>import lnst
>#module lnst.tests will dynamically look for module classes in configured
>#locations, similar to how we do it now
>#or you can import BaseTestmodule and define your test module directly in your
>recipe
>
>ping = lnst.modules.Ping(dst=m2.if1.ip[0], count=100, interval=0.1)
>
>m1.run(ping)
>
>================================================
>
>2. Recipe:
>Renamed Task to Recipe... since the class represents the "whole" LNST
workflow
>of setting requirements and defining the test
>
>class BaseRecipe(object):
> def __init__(self):
> #initialize instance specific requirements
> self.requirements = Requirements()
> for x in dir(self):
> val = getattr(self, x)
> if instance(val, Requirement):
> setattr(self.requirements, x, val)
>
> # check types of parameters, the Param class hierarchy is the same as
> # the one used by BaseTestModule
> self.params = object()
> for x in dir(self):
> val = getattr(self, x)
> if instance(val, Param):
> setattr(self.params, x, val)
>
> # check mandatory parameters
> for name, param in self.params:
> if param.mandatory and not param.set:
> raise RecipeException("Parameter {} is
mandatory".format(name))
>
> def test():
> raise Exception("Method test MUST be defined.")
>
>class MyRecipe(lnst.BaseRecipe):
> #class-wide definition of requirements
> m1 = HostReq(arch="x86_64", ...)
> m1.if1 = IfaceReq(label="xyz", driver="mlx", ...)
>
> #m1.params = IfaceReq()
> #m1.devs = IfaceReq()
> #raise Exception("param/devs name is a reserved keyword")
>
> m2 = HostReq(param="val", ...)
> m2.if1 = IfaceReq(label="xyz", param="val", ...)
>
> # idosch requested the ability to "blacklist" parameters, jpirko's
> # suggestion was to use regular expressions in the mapper. This should be
> # easy to do, though it's not entirely "pretty" since regexes are
not very
> # good at doing negative conditions... maybe we can think of something
> # better?
>
> #class-wide definition of Task parameters, with type checking,
> #possibly even type conversions (e.g. string to int), using the same
> #*Type classes as with Module parameters
> mtu = IntType(default=1500)
>
> def __init__(self, **kwargs):
> super(self, lnst.BaseTask).__init__()
>
> #do something with kwargs
> interface_driver = kwargs["driver"]
> self.reqs.m1.if1.driver = interface_driver
> #adjust instance specific requirements
> self.reqs.m3 = HostReq(...)
>
> def test(self):
> self.matched.m1.run(Module)
> self.matched.m1.run("command")
>
> # optionally we can add support for something like this, but at the moment
> # we think it could be confusing and/or complicated to do right
> #def test(m1, m2, m3, m4,...):
> # m1.run(Module)
> # m2.run("command")
> # m1.params.arch == "x86_64"
>
>================================================
>
>3. Running Recipes:
>
>my_test_script.py:
>#!/bin/python
>from MyRecipes import MyRecipe
>from lnst import Controller
>
>recipe_instance = MyRecipe(mtu="5000")
>
>ctl = Controller(config="/etc/lnst-ctl.conf")
>ctl.run(recipe_instance)
>
>chmod +x my_test_script.py
>./my_test_script.py args
>
>OR
>
>lnst-ctl -d run MyRecipe.py -- mtu=8000
># looks for NAME class in the NAME.py file (MyRecipe in this case for which
># the condition "isinstance(NAME, BaseRecipe)" must be True
>
># could also run for all classes in the file where "isinstance(x,
BaseRecipe)"
># is True. with the option to restrict to specific task class... lnst-ctl
># rewritten to do the same as manually running the task from it's own python
># script
>
>Aliases lose meaning - they're parameters passed to the MyTask __init__, when
>using the lnst-ctl CLI, use "-- task_params"?? might not work for multiple
>tasks,
>
>================================================
>
>4. Tester facing API:
>
>#!!!! breakpoints!!!
>
>class Controller:
> # This is the class which serves as an entry point to using LNST in a
> # library-like way to run a controller - as in running a Recipe from it's
> # own executable python script"
> # It's partially based on the old NetTestController class
>
> def __init__(self, debug=0, mapper=MachineMapper, pools=[], pool_checks=True):
> # defines the logging level
> self._debug = debug
> self._mac_pool = MacPool()
> # class controlling logging
> self._log_ctl = LoggingCtl(debug, ...)
> # dictionary of dynamically created networks (libvirt)
> self._network_bridges = {}
>
> # a ConnectionHandler for communicating with slaves
> self._msg_dispatcher = MessageDispatcher(self._log_ctl)
>
> # a Mapper class - handling the matching of requirements to the Hosts
> # available in pools, you can define your own class that supports the
> # API (specified later) to do your own matching API, the basic idea is
> # for the Mapper to get requirements in a certain format and the pools
> # in a certain format and to generate mappings between these two
> self._mapper = mapper()
>
> # pool loading - from a config file, and optionaly restricted from the
> # pools argument, functionality copied from the NetTestController, I'm
> # thinking this could change (pools default value is grabbed from
> # config, so that the user can override them completely?)
> select_pools = {}
> conf_pools = ctl_config.get_pools()
> if len(pools) > 0:
> for pool_name in pools:
> if pool_name in conf_pools:
> select_pools[pool_name] = conf_pools[pool_name]
> elif len(pools) == 1 and os.path.isdir(pool_name):
> select_pools = {"cmd_line_pool": pool_name}
> else:
> raise NetTestError("Pool %s does not exist!" %
pool_name)
> else:
> select_pools = conf_pools
>
> # a PoolManager, that loads the slave description XML files and checks
> # the availability of test hosts. Defines an API to access the available
> # pools usable by the Mapper to match against requirements. I'm thinking
> # this could also be made an argument of the __init__ method when the
> # API is defined, the users could then replace both the Mapper and the
> # PoolManager... although I don't think it would be used very much...
> self._pools = SlavePoolManager(select_pools, pool_checks)
>
> def run(self, recipe, **mapper_kwargs):
> # this method implements the main loop of the recipe execution, the
> # mapper_kwargs are arguments passed to the mapper to influence the
> # mapping process, such as to try all matches, or to enable virtual
> # matches and so on
>
> for match in self._mapper.matches(**mapper_kwargs):
> matched = True
> # uses the current match to map the Machine objects from the
> # PoolManager to create Host objects for the Recipe
> self._map_match(match)
> self._print_match_description(match)
> try:
> recipe.test()
> except Exception as exc:
> logging.error("Recipe execution terminated by unexpected
exception")
> raise
> finally:
> for machine in self._machines.values():
> machine.restore_system_config()
> self._cleanup_slaves()
>
>
>Host objects available in self.matched.selector_name:
>
>class Host: #name can change
> #attributes:
>
> # dynamically filled object of Host attributes such as architecture and
> # so on. Use example in test() would look like this:
> # if host.params.arch == "x86":
> # I separated this into the "params" object so I can overwrite its
> # __getattr__ method and return None/UnknownParam exception for unknown
> # parameters, and to avoid name conflicts with other attributes
> # they should also be iterable by:
> # for param in host.params:
> params = object()
>
> !!! params sounds odd? other possibilities 'property',
'description'?
> !!! m1.property.arch == "x86" ?
> !!! m1.description.arch == "x86" ?
> !!! we've also discussed that it would probably be easiest to just have a
> !!! predefined static set of these attributes that are checked on
> !!! lnst-slave startup and are sent to the Controller on connection.
>
> # dynamically filled object of NetDevice objects accessible directly as the
> # object attributes:
> # host.eth0.set_ip(...)
> # I separated this into the "ifaces" object to avoid name conflicts
with
> # other attributes
> # creation of new NetDevices should be possible through simple assignement:
> # m1.team0 = TeamDevice(...)
> # assignement of an incompatible Type or to an existing Device object will
> # return an exception
> # to deconfigure+remove call m1.team0.remove()
> __devs = object()
>
> # device iterator for the Host object:
> # for dev in host:
>
> def run(what, bg=False, fail=False, timeout=60, path="", json=False,
netns, desc)
> # will run "what" on the remote host
> # "what" is either a Module object, or a string command that will
be
> # executed as a bash command
> # "bg" when True, runs "what" on background - the run()
call
> # immediately returns, and "timeout" is ignored, the background
> # process can be controlled through the returned Job object
> # "fail" if True then the Job is expected to fail, and will be
reported
> # as PASSed if it does
> # "timeout" in seconds, determines how long to block test execution
for
> # before killing the Job. Only when running in foreground
> # "path" changes the current working directory to the specified
path
> # before "what" is executed and changes back after execution is
> # finished.
> # IGNORE FOR NOW
> # "json" if True will attempt to parse the returned stdout of the
Job
> # as json into a dictionary
> # "netns" Job will be run in the specified network namespace
> # "desc" is a description message that will show up in logs and the
> # result summary
> # Returns a Job object
> # !! Exceptions?!?!
>
> # !!!THINK ABOUT THESE...
> def __set(path, value)
> # copied from old API, provides a shortcut for "echo $value #
>/proc/or/sys/path"
> # and returns the original value when the test is finished
>
> def sysfs_set(path, value):
> # check if path starts with "/sys"?
> self.__set(path, value)
>
> def procfs_set(path, value):
> self.__set("/proc/"+path, value)
>
> def send_file(ctl_path="", slave_path="", recursive=False)
> # copies the specified file from the controller to the specified
> # destination path, if recursive == True and srcpath refers to a
> # directory it copies the entire directory
>
> def recv_file(recv_path="", ctl_path="", recursive=False)
>
> def {enable, disable}_service(service)
> # copied from old API, enables or disables the specified service
>
> def del_device(name)
> # removes the specified device, probably easier (more logical?) to do
> # this then "devs.name = None" and "del devs.name" would
be unreliable
> # !!! REMOVE
>
>class Device: #DeviceAPI, InterfaceAPI? name can change...
> # attributes:
>
> # dynamically created Device attributes such as driver and so on. Use
> # example in test() would look like this:
> # if host.devs.eth0.driver == "ixgbe":
> # achieved through rewriting of the __getattr__ method of the Device class
> # should return None or throw UnknownParam exception for unknown parameters
> # this should directly mirror the Device objects that are managed by the
> # InterfaceManager on the Slave
> # !!! predefine the params, assignment will set them, sync on set - wait
> # !!! on the slave for the set to finish and return the current status of
> # !!! everything else as well
> # eg:
> driver = something
> mtu = something
> ips = [IpAddress, ...]
>
>class Job: #ProcessAPI? name can change...
> # The way Jobs are launched is different from how we handled them with XML
> # recipes -> on Slaves, ALL jobs are run in a separate process ("in
> # background"), without a timeout, and they all have an id (previously
> # foreground commands had id == None)
> #
> # The distinction of foreground/background Jobs is handled by the Hosts
> # "run" method -> for background Jobs, the method immediately returns
with
> # the Job handle (instance of this class), for foreground Jobs, the method
> # calls the wait method of this class (default timeout) and if it times out,
> # it calls the kill method of this class (SIGKILL)
> #
> # For now, I'm just considering Module and "Shell command" Jobs,
system
> # configs will probably be handled in a different way...
> #
> # Job results can be picked up at any time the execution is handed over to
> # the LNST library - the message carrying results from the slave will be
> # handled by the connection handler and sent to the correspoding Job
> # object. This is relevant for background Jobs only as foreground Jobs will
> # always be in a finished state after the Host run method returns
>
> #attributes:
>
> # True if the Job finished, False if it's still running in the background
> finished = bool
>
> # contains the result data returned by the Job
> result = object
>
> # contain the stdout and stderr generated by the job, None for Module Jobs
> stdout = ""
> stderr = ""
>
> # simple True/False value indicating success/failure of the Job
> passed = bool
>
> def wait(timeout=0):
> # for background jobs, will wait until the job finished
> # "timeout" in seconds, determines how long to wait for. After
timeout
> # reached, nothing happens, status of the job can be checked with the
> # "finished" attribute. If timeout=0, then wait forever.
>
> def kill(signalnum=signal.SIGKILL):
> # sends the specified signal to the process of the Job running in
> # background
> # in case the Job finished while this method call is executed then the
> # job just finishes normally and the signal isn't sent
> # "signalnum" the signal to be sent, default is SIGKILL
>
> def __str__(self):
> # for easy print m1.run(what)
> return stdout+stderr # what about Modules?
>
>
>================================================
>
>5. Result summary proposal
>
>Since I've changed how Job execution is handled, I've also wrote down a
>proposal to change how we log Recipe results - the RESULTS SUMMARY logs at the
>end of a recipe run. I haven't started working on it yet, I've just wrote an
>example on paper which I'm copying here. Any comments are appreciated.
>
>RESULTS SUMMARY:
>Host m1 Job 1 XYZ PASS/FAIL
> Formatted results:
> ...
>Host m2 Job 1 XYZ started
>Host m1 Job 3 XYZ PASS/FAIL
> Formatted results:
> ...
>Host m2 Job 1 XYZ PASS/FAIL
> Formatted results:
> ...
>Custom summary record.... (optional PASS/FAIL)
> ... optional additional data
> ... i still need to figure out how this will look like
>
>The main difference to the old results summary is that Jobs have numerical ids
>that are unique per host, and you ALWAYS see the id (previously only background
>commands had ids). Since all Jobs "run in background" this will make
matching
>"started" "finished" logs easier. There also won't be any more
"kill cmd" "intr
>cmd" logs here since these commands don't exist anymore.
>
>Since "all Jobs are in background" it means that in reality all of them
>generate a "started" and "finished" log, however, if these are in
a direct
>sequence after each other they get shortened to just the PASS/FAIL log. This
>will also be true for background commands if there were no results to report
>between their start and finish.