[PATCH 0/5] Recipe Run API Fix

From: Perry Gagne <pgagne(a)redhat.com&gt;

These patches rework the API for exporting RecipeRuns.

Rather then using a wrapper object RecipeRunData, we just export the
RecipeRun itself. Also, compression was added to reduce file size.

fixes https://gitlab.cee.redhat.com/lnst-team/lnst/-/issues/29 

Perry Gagne (5):
  RecipeResults.py: Stop excluding RemoteDevices from export
  RecipeRunExport.py: Changed API, added compression, fixedup docs.
  Recipe.py: Made Recipe pickle-able
  Recipe.py: Make export/import of runs based on RecipeRun directly
  Recipe.py: Removed Recipe name property.

 docs/source/recipe_run_export.rst  |  16 +++++
 docs/source/tester_api.rst         |   1 +
 lnst/Common/ConnectionHandler.py   |   7 ++
 lnst/Controller/Controller.py      |   2 +-
 lnst/Controller/Machine.py         |   8 +++
 lnst/Controller/Recipe.py          |  97 ++++++++++++++++++++++++-
 lnst/Controller/RecipeResults.py   |   5 --
 lnst/Controller/RecipeRunExport.py | 112 -----------------------------
 lnst/Devices/RemoteDevice.py       |   2 +
 9 files changed, 131 insertions(+), 119 deletions(-)
 create mode 100644 docs/source/recipe_run_export.rst
 delete mode 100644 lnst/Controller/RecipeRunExport.py

-- 
2.26.2

[PATCH 1/5] RecipeResults.py: Stop excluding RemoteDevices from export

From: Perry Gagne <pgagne(a)redhat.com&gt;

Previously we were excluding RemoteDevice from export due to a number of its attributes
containing references to low level PYCapsule and socket stuff.

I took another look at this to be more precise about what is excluded.

Also, with Ondrej help I was able to find the cause of an issue I a ran into previously
where an infinite loop of calls to RemoteDevice.__setattr__ would occur when trying to
unpickle a exported run.

Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
---
 lnst/Common/ConnectionHandler.py | 7 +++++++
 lnst/Controller/Machine.py       | 8 ++++++++
 lnst/Controller/RecipeResults.py | 5 -----
 lnst/Devices/RemoteDevice.py     | 2 ++
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py
index a74d4a8..04bf7be 100644
--- a/lnst/Common/ConnectionHandler.py
+++ b/lnst/Common/ConnectionHandler.py
@@ -133,3 +133,10 @@ class ConnectionHandler(object):
     def clear_connections(self):
         self._connections = []
         self._connection_mapping = {}
+
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # Remove things that can't be pickled
+        state['_connections'] = []
+        state['_connection_mapping'] = {}
+        return state
\ No newline at end of file
diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
index f63fd6f..7ef3204 100644
--- a/lnst/Controller/Machine.py
+++ b/lnst/Controller/Machine.py
@@ -591,3 +591,11 @@ class Machine(object):
 
     def get_security(self):
         return self._security
+
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        # Remove things that can't be pickled
+        state['_msg_dispatcher'] = None
+        if self.get_libvirt_domain():
+            state['_domain_ctl'] = None
+        return state
\ No newline at end of file
diff --git a/lnst/Controller/RecipeResults.py b/lnst/Controller/RecipeResults.py
index 941ea8d..38a4170 100644
--- a/lnst/Controller/RecipeResults.py
+++ b/lnst/Controller/RecipeResults.py
@@ -118,11 +118,6 @@ class DeviceConfigResult(BaseResult):
     def device(self):
         return self._device
 
-    def __getstate__(self):
-        state = self.__dict__.copy()
-        # Remove things that can't be pickled
-        state['_device'] = None
-        return state
 
 class DeviceCreateResult(DeviceConfigResult):
     @property
diff --git a/lnst/Devices/RemoteDevice.py b/lnst/Devices/RemoteDevice.py
index f11fa65..7891514 100644
--- a/lnst/Devices/RemoteDevice.py
+++ b/lnst/Devices/RemoteDevice.py
@@ -108,6 +108,8 @@ class RemoteDevice(object):
     def __getattr__(self, name):
         if name == "_inited":
             return False
+        if not self._inited:
+            return super(RemoteDevice, self).__getattr__(name)
 
         attr = getattr(self._dev_cls, name)
 
-- 
2.26.2

[PATCH 2/5] RecipeRunExport.py: Changed API, added compression, fixedup docs.

From: Perry Gagne <pgagne(a)redhat.com&gt;

We decided to simplify the API a bit to just have and import and export method. We also
decided to add compression to the exported file.

Updated docs with new API and examples.

Recipe.py: Added recipe reference to RecipeRun so we only have to specify one variable
when exporting. Recipe is still excluded from export.

Also added shortcut for recipe name to fix LNST-project/lnst#194

tester_api.rst: Added link to docs for RecipeRunExport

Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
---
 docs/source/recipe_run_export.rst  |   5 +
 docs/source/tester_api.rst         |   1 +
 lnst/Controller/Controller.py      |   2 +-
 lnst/Controller/Recipe.py          |  16 ++-
 lnst/Controller/RecipeRunExport.py | 161 +++++++++++++++--------------
 5 files changed, 107 insertions(+), 78 deletions(-)
 create mode 100644 docs/source/recipe_run_export.rst

diff --git a/docs/source/recipe_run_export.rst b/docs/source/recipe_run_export.rst
new file mode 100644
index 0000000..6d366c2
--- /dev/null
+++ b/docs/source/recipe_run_export.rst
@@ -0,0 +1,5 @@
+Export/Import of Recipe Runs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: lnst.Controller.RecipeRunExport
+    :members:
\ No newline at end of file
diff --git a/docs/source/tester_api.rst b/docs/source/tester_api.rst
index e7cf743..65a2e1b 100644
--- a/docs/source/tester_api.rst
+++ b/docs/source/tester_api.rst
@@ -8,3 +8,4 @@ Test developer API
    recipe_api
    controller_api
    parameters
+   recipe_run_export
diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py
index 1641ed3..d63ec90 100644
--- a/lnst/Controller/Controller.py
+++ b/lnst/Controller/Controller.py
@@ -156,7 +156,7 @@ class Controller(object):
                 logging.info(line)
             try:
                 self._map_match(match, req, recipe)
-                recipe._init_run(RecipeRun(match,
log_dir=self._log_ctl.get_recipe_log_path()))
+                recipe._init_run(RecipeRun(recipe, match,
log_dir=self._log_ctl.get_recipe_log_path()))
                 recipe.test()
             except Exception as exc:
                 logging.error("Recipe execution terminated by unexpected
exception")
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
index a9f7514..3bb6793 100644
--- a/lnst/Controller/Recipe.py
+++ b/lnst/Controller/Recipe.py
@@ -126,6 +126,10 @@ class BaseRecipe(object):
     def _set_ctl(self, ctl):
         self._ctl = ctl
 
+    @property
+    def name(self):
+        return self.__class__.__name__
+
     @property
     def matched(self):
         if self.ctl is None:
@@ -152,11 +156,12 @@ class BaseRecipe(object):
                                            level, data_level))
 
 class RecipeRun(object):
-    def __init__(self, match, desc=None, log_dir=None):
+    def __init__(self, recipe: BaseRecipe, match, desc=None, log_dir=None):
         self._match = match
         self._desc = desc
         self._results = []
         self._log_dir = log_dir
+        self._recipe = recipe
 
     def add_result(self, result):
         if not isinstance(result, BaseResult):
@@ -196,3 +201,12 @@ class RecipeRun(object):
     @property
     def overall_result(self):
         return all([i.success for i in self.results])
+
+    @property
+    def recipe(self) -> BaseRecipe:
+        return self._recipe
+
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state['_recipe'] = None
+        return state
diff --git a/lnst/Controller/RecipeRunExport.py b/lnst/Controller/RecipeRunExport.py
index cf40ae4..9486fe8 100644
--- a/lnst/Controller/RecipeRunExport.py
+++ b/lnst/Controller/RecipeRunExport.py
@@ -1,20 +1,33 @@
+"""This module provides an API to export data related to a recipe run.
This export can be used to gather data like  CPU and iperf statistic for later analysis,
like when onboarding a new Recipe.
+
+The data that is exported is the instance of :py:class:`lnst.Controller.Recipe.RecipeRun`
that was run.
+
+Data is encapsuled in an :py:class:`RecipeRunData` object which allows us to include
other metadata. The `RecipeRunData` object is "pickled" and compressed with
LZMA/XZ compression using :py:mod:`lzma`.
+
+By default the file will be contain a file extension `.lrc` which stands for "LNST
Run, Compressed".
+
+Use :py:meth:`export_recipe_run` to export and :py:meth:`import_recipe_run` to import.
+"""
+
+__author__ = """
+pgagne(a)redhat.com (Perry Gagne)
+"""
+
 import datetime
 import pickle
 import os
 import logging
-from typing import List, Tuple
-from lnst.Controller.Recipe import BaseRecipe, RecipeRun
+import lzma
+from lnst.Controller.Recipe import RecipeRun
 
 
 class RecipeRunData:
     """
-    Class used to encapsulate a RecipeRun, this is the object
-    that will be pickled and output to a file.
+    Class used to encapsulate a :py:class:`lnst.Controller.Recipe.RecipeRun`, this is the
object
+    that will be pickled, compressed and output to a file.
 
-    :param recipe_cls:
-        class of the Recipe. We do not currently pickle the instance
-        of the recipe itself for ease of exporting.
-    :type recipe_cls: :py:class: `lnst.Controller.Recipe.BaseRecipe`
+    :param recipe_cls: class of the Recipe. We do not currently pickle the instance of
the recipe itself.
+    :type recipe_cls: :py:class:`lnst.Controller.Recipe.BaseRecipe`
 
     :param param: Copy of Recipe parameters.
     :type param: dict
@@ -22,91 +35,87 @@ class RecipeRunData:
     :param req: Copy of Recipe requirements
     :type req: dict
 
-    :param environ: A copy of `os.environ` created when the object is instantiated.
+    :param environ: A copy of :py:data:`os.environ` created when this object is
instantiated.
     :type environ: dict
 
-    :param run: :py:class:`lnst.Controller.Recipe.RecipeRun` instance of the run
+    :param run: instance of the run
     :type run: :py:class:`lnst.Controller.Recipe.RecipeRun`
 
-    :param datetime: A time stamp that is the result of running `datetime.datetime.now()`
during instantiation
+    :param datetime: A time stamp that is the result of running
:py:meth:`datetime.datetime.now` during instantiation
     :type datetime: :py:class:`datetime.datetime`
     """
-
-    def __init__(self, recipe: BaseRecipe, run: RecipeRun):
-        self.recipe_cls = recipe.__class__
-        self.params = recipe.params._to_dict()
-        self.req = recipe.req._to_dict()
+    def __init__(self, run: RecipeRun):
+        self.recipe_cls = run.recipe.__class__
+        self.params = run.recipe.params._to_dict()
+        self.req = run.recipe.req._to_dict()
         self.environ = os.environ.copy()
         self.run = run
         self.datetime = datetime.datetime.now()
 
 
-class RecipeRunExporter:
+def export_recipe_run(run: RecipeRun, export_dir: str = None, name: str = None) ->
str:
     """
-    Class used to export recipe runs.
+    Export a recipe run to a file. Run is encapsulated, pickled and compressed.
+
+    :param run: `RecipeRun` object to export.
+    :type run: :py:class: `lnst.Controller.Recipe.RecipeRun`
+    :param export_dir: Directory to export file to. Defaults to :py:attr:`run.log_dir`
+    :type export_dir: str
+    :param name: Name of output (exclusive of directory). Defaults to
`<recipename>-run-<timestamp>.lrc`.
+    :type name: str
+    :return: Path of output file.
+    :rtype: str
+
+    Example::
+
+        ctl = Controller(...)
+        recipe = BondRecipe(...)
+        ctl.run(recipe)
+        ...
+        >>> path = export_recipe_run(recipe.run[0])
+        2020-10-02 15:20:58 (localhost) - INFO: Exported BondRecipe run data to
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
+        >>> print(path)
+       
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
 
     """
+    run_data = RecipeRunData(run)
+    if not name:
+        name =
f"{run.recipe.name}-run-{run_data.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
+    if not export_dir:
+        export_dir = run.log_dir
 
-    def __init__(self, recipe: BaseRecipe):
-        """
-
-        :param recipe: Recipe
-        :type recipe: :py:class: `lnst.Controller.Recipe.BaseRecipe`
-        """
-        self.recipe = recipe
-        self.recipe_name = self.recipe.__class__.__name__
-
-    def export_run(self, run: RecipeRun, dir=None, name=None) -> str:
-        """
-
-        :param run: The RecipeRun to export
-        :type run: :py:class:`lnst.Controller.Recipe.RecipeRun`
-        :param dir: The path to directory to export to. Does not include file name,
defaults to `run.log_dir`
-        :type dir: str
-        :param name: Name of file to export. Default
`<recipename>-run-<timestamp>.dat'
-        :type name: str
-        :return: Path (dir+filename) of exported run.
-        :rtype:str
-        """
-        data = RecipeRunData(self.recipe, run)
-        if not name:
-            name =
f"{self.recipe_name}-run-{data.datetime:%Y-%m-%d_%H:%M:%S}.dat"
-        if not dir:
-            dir = run.log_dir
-
-        path = os.path.join(dir, name)
-
-        with open(path, 'wb') as f:
-            pickle.dump(data, f)
-
-        logging.info(f"Exported {self.recipe_name} data to {path}")
-        return path
-
-
-def export_recipe_runs(recipe: BaseRecipe) -> List[Tuple[str, RecipeRun]]:
-    """
-    Helper method that exports all runs in a recipe.
-    :param recipe: Recipe to export
-    :type recipe: :py:class: `lnst.Controller.Recipe.BaseRecipe`
-    :return: list of files that contain exported recipe runs
-    :rtype: list
-    """
-    exporter = RecipeRunExporter(recipe)
-    files = []
-    for run in recipe.runs:
-        path = exporter.export_run(run)
-        files.append((run, path))
-    return files
+    path = os.path.join(export_dir, name)
+    with lzma.open(path, 'wb') as f:
+        pickle.dump(run_data, f)
+    logging.info(f"Exported {run.recipe.name} run data to {path}")
+    return path
 
 
 def import_recipe_run(path: str) -> RecipeRunData:
     """
-    Import recipe runs that have been exported using
:py:class:`lnst.Controller.RecipeRunExport.RecipeRunExporter`
-    :param path:  path to file  to import
-    :type path: str
-    :return: `RecipeRun` object containing the run and other metadata.
-    :rtype: :py:class:`lnst.Controller.RecipeRunExport.RecipeRunData`
+    Import a recipe run that was exported using :py:meth:`export_recipe_run`
+
+    :param path: Path to file to import
+    :type path:  str
+    :return: object which contains the imported recipe run
+    :rtype:  :py:class:`RecipeRunData`
+
+    Example::
+
+        >>> run_data =
import_recipe_run("/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc")
+        >>> type(run_data)
+        <class 'lnst.Controller.RecipeRunExport.RecipeRunData'>
+        >>> run_data.recipe_cls
+        <class 'lnst.Recipes.ENRT.BondRecipe.BondRecipe'>
+        >>> run_data.run.results[38]
+        <lnst.Controller.RecipeResults.Result object at 0x7f20727e1e20>
+        >>> run_data.run.results[38].data
+        {'cpu': [[[<lnst.RecipeCommon.Perf.Results.PerfInterval object at
0x7f20727e1e50>,...]]], ... }
+        >>> print(run_data.run.results[38].description)
+        CPU Utilization on host host1:
+        cpu 'cpu': 45.40 +-0.00 time units per second
+        cpu 'cpu0': 45.40 +-0.00 time units per second
     """
-    with open(path, 'rb') as f:
-        data = pickle.load(f)
-    return data
+    with lzma.open(path, 'rb') as f:
+        run_data = pickle.load(f)
+    return run_data
-- 
2.26.2

[PATCH 3/5] Recipe.py: Made Recipe pickle-able

From: Perry Gagne <pgagne(a)redhat.com&gt;

Made recipe pickle-able, export of will still be based on RecipeRuns in the highlevel api
though.

Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
---
 lnst/Controller/Recipe.py          | 10 ++++++----
 lnst/Controller/RecipeRunExport.py |  9 +++++----
 2 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
index 3bb6793..7fb7006 100644
--- a/lnst/Controller/Recipe.py
+++ b/lnst/Controller/Recipe.py
@@ -155,6 +155,12 @@ class BaseRecipe(object):
         self.current_run.add_result(Result(success, description, data,
                                            level, data_level))
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state['_ctl'] = None
+        return state
+
+
 class RecipeRun(object):
     def __init__(self, recipe: BaseRecipe, match, desc=None, log_dir=None):
         self._match = match
@@ -206,7 +212,3 @@ class RecipeRun(object):
     def recipe(self) -> BaseRecipe:
         return self._recipe
 
-    def __getstate__(self):
-        state = self.__dict__.copy()
-        state['_recipe'] = None
-        return state
diff --git a/lnst/Controller/RecipeRunExport.py b/lnst/Controller/RecipeRunExport.py
index 9486fe8..fc27284 100644
--- a/lnst/Controller/RecipeRunExport.py
+++ b/lnst/Controller/RecipeRunExport.py
@@ -1,8 +1,10 @@
-"""This module provides an API to export data related to a recipe run.
This export can be used to gather data like  CPU and iperf statistic for later analysis,
like when onboarding a new Recipe.
+"""This module provides an API to export data related to a recipe run.
+This export can be used to gather data like  CPU and iperf statistic for later analysis,
like when onboarding a new Recipe.
 
 The data that is exported is the instance of :py:class:`lnst.Controller.Recipe.RecipeRun`
that was run.
 
-Data is encapsuled in an :py:class:`RecipeRunData` object which allows us to include
other metadata. The `RecipeRunData` object is "pickled" and compressed with
LZMA/XZ compression using :py:mod:`lzma`.
+Data is encapsuled in an :py:class:`RecipeRunData` object which allows us to include
other metadata.
+The `RecipeRunData` object is "pickled" and compressed with LZMA/XZ compression
using :py:mod:`lzma`.
 
 By default the file will be contain a file extension `.lrc` which stands for "LNST
Run, Compressed".
 
@@ -45,7 +47,6 @@ class RecipeRunData:
     :type datetime: :py:class:`datetime.datetime`
     """
     def __init__(self, run: RecipeRun):
-        self.recipe_cls = run.recipe.__class__
         self.params = run.recipe.params._to_dict()
         self.req = run.recipe.req._to_dict()
         self.environ = os.environ.copy()
@@ -105,7 +106,7 @@ def import_recipe_run(path: str) -> RecipeRunData:
> >> run_data =
> import_recipe_run("/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc")
> >> type(run_data) 
         <class
'lnst.Controller.RecipeRunExport.RecipeRunData'>
-        >>> run_data.recipe_cls
+        >>> run_data.recipe.__class__
         <class 'lnst.Recipes.ENRT.BondRecipe.BondRecipe'>
> >> run_data.run.results[38] 
        
<lnst.Controller.RecipeResults.Result object at 0x7f20727e1e20>
-- 
2.26.2

[PATCH 4/5] Recipe.py: Make export/import of runs based on RecipeRun directly

From: Perry Gagne <pgagne(a)redhat.com&gt;

We decided to export RecipeRun objects directly instead of wrapping them in
RecipeRunData.

Metadata (datetime and environ) was added to RecipeRun, all other attributes are
accessible via the `run.recipe` attribute.

Export/import functions where moved to Recipe.py.

Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
---
 docs/source/recipe_run_export.rst  |  15 +++-
 lnst/Controller/Recipe.py          |  82 +++++++++++++++++++
 lnst/Controller/RecipeRunExport.py | 122 -----------------------------
 3 files changed, 95 insertions(+), 124 deletions(-)
 delete mode 100644 lnst/Controller/RecipeRunExport.py

diff --git a/docs/source/recipe_run_export.rst b/docs/source/recipe_run_export.rst
index 6d366c2..3500655 100644
--- a/docs/source/recipe_run_export.rst
+++ b/docs/source/recipe_run_export.rst
@@ -1,5 +1,16 @@
 Export/Import of Recipe Runs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Recipe runs can be exported to a file for for later analysis.
+For example, to gather data like  CPU and iperf statistic when onboarding a new recipe.
 
-.. automodule:: lnst.Controller.RecipeRunExport
-    :members:
\ No newline at end of file
+The data that is exported is the instance of :py:class:`lnst.Controller.Recipe.RecipeRun`
that was run.
+
+The :py:class:`RecipeRun` object is "pickled" and compressed with LZMA/XZ
compression using :py:mod:`lzma`.
+
+By default the file will be contain a file extension `.lrc` which stands for "LNST
Run, Compressed".
+
+Use :py:meth:`export_recipe_run` to export and :py:meth:`import_recipe_run` to import.
+
+.. autofunction:: lnst.Controller.Recipe.export_recipe_run
+
+.. autofunction:: lnst.Controller.Recipe.import_recipe_run
diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
index 7fb7006..eb9b344 100644
--- a/lnst/Controller/Recipe.py
+++ b/lnst/Controller/Recipe.py
@@ -7,7 +7,11 @@ olichtne(a)redhat.com (Ondrej Lichtner)
 """
 
 import copy
+import datetime
 import logging
+import lzma
+import os
+import pickle
 from lnst.Common.Parameters import Parameters, Param
 from lnst.Common.Colours import decorate_with_preset
 from lnst.Controller.Requirements import _Requirements, HostReq
@@ -168,6 +172,8 @@ class RecipeRun(object):
         self._results = []
         self._log_dir = log_dir
         self._recipe = recipe
+        self._datetime = datetime.datetime.now()
+        self._environ = os.environ.copy()
 
     def add_result(self, result):
         if not isinstance(result, BaseResult):
@@ -212,3 +218,79 @@ class RecipeRun(object):
     def recipe(self) -> BaseRecipe:
         return self._recipe
 
+    @property
+    def datetime(self):
+        return self._datetime
+
+    @property
+    def environ(self):
+        return self._environ
+
+
+def export_recipe_run(run: RecipeRun, export_dir: str = None, name: str = None) ->
str:
+    """
+    Export a recipe run to a file. :py:class:`RecipeRun` is pickled and compressed.
+
+    :param run: `RecipeRun` object to export.
+    :type run: :py:class:`RecipeRun`
+    :param export_dir: Directory to export file to. Defaults to :py:attr:`run.log_dir`
+    :type export_dir: str
+    :param name: Name of output (exclusive of directory). Defaults to
`<recipename>-run-<timestamp>.lrc`.
+    :type name: str
+    :return: Path of output file.
+    :rtype: str
+
+    Example::
+
+        ctl = Controller(...)
+        recipe = BondRecipe(...)
+        ctl.run(recipe)
+        ...
+        >>> from lnst.Controller.Recipe import export_recipe_run
+        >>> path = export_recipe_run(recipe.run[0])
+        2020-10-02 15:20:58 (localhost) - INFO: Exported BondRecipe run data to
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
+        >>> print(path)
+       
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
+
+    """
+    if not name:
+        name = f"{run.recipe.name}-run-{run.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
+    if not export_dir:
+        export_dir = run.log_dir
+
+    path = os.path.join(export_dir, name)
+    with lzma.open(path, 'wb') as f:
+        pickle.dump(run, f)
+    logging.info(f"Exported {run.recipe.name} run to {path}")
+    return path
+
+
+def import_recipe_run(path: str) -> RecipeRun:
+    """
+    Import a recipe run that was exported using :py:meth:`export_recipe_run`
+
+    :param path: Path to file to import
+    :type path:  str
+    :return: object which contains the imported recipe run
+    :rtype:  :py:class:`RecipeRun`
+
+    Example::
+
+        >>> from lnst.Controller.Recipe import import_recipe_run
+        >>> run =
import_recipe_run("/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc")
+        >>> type(run)
+        <class 'lnst.Controller.Recipe.RecipeRun'>
+        >>> run.recipe.__class__
+        <class 'lnst.Recipes.ENRT.BondRecipe.BondRecipe'>
+        >>> run.results[38]
+        <lnst.Controller.RecipeResults.Result object at 0x7f20727e1e20>
+        >>> run.results[38].data
+        {'cpu': [[[<lnst.RecipeCommon.Perf.Results.PerfInterval object at
0x7f20727e1e50>,...]]], ... }
+        >>> print(run.results[38].description)
+        CPU Utilization on host host1:
+        cpu 'cpu': 45.40 +-0.00 time units per second
+        cpu 'cpu0': 45.40 +-0.00 time units per second
+    """
+    with lzma.open(path, 'rb') as f:
+        run = pickle.load(f)
+    return run
diff --git a/lnst/Controller/RecipeRunExport.py b/lnst/Controller/RecipeRunExport.py
deleted file mode 100644
index fc27284..0000000
--- a/lnst/Controller/RecipeRunExport.py
+++ /dev/null
@@ -1,122 +0,0 @@
-"""This module provides an API to export data related to a recipe run.
-This export can be used to gather data like  CPU and iperf statistic for later analysis,
like when onboarding a new Recipe.
-
-The data that is exported is the instance of :py:class:`lnst.Controller.Recipe.RecipeRun`
that was run.
-
-Data is encapsuled in an :py:class:`RecipeRunData` object which allows us to include
other metadata.
-The `RecipeRunData` object is "pickled" and compressed with LZMA/XZ compression
using :py:mod:`lzma`.
-
-By default the file will be contain a file extension `.lrc` which stands for "LNST
Run, Compressed".
-
-Use :py:meth:`export_recipe_run` to export and :py:meth:`import_recipe_run` to import.
-"""
-
-__author__ = """
-pgagne(a)redhat.com (Perry Gagne)
-"""
-
-import datetime
-import pickle
-import os
-import logging
-import lzma
-from lnst.Controller.Recipe import RecipeRun
-
-
-class RecipeRunData:
-    """
-    Class used to encapsulate a :py:class:`lnst.Controller.Recipe.RecipeRun`, this is the
object
-    that will be pickled, compressed and output to a file.
-
-    :param recipe_cls: class of the Recipe. We do not currently pickle the instance of
the recipe itself.
-    :type recipe_cls: :py:class:`lnst.Controller.Recipe.BaseRecipe`
-
-    :param param: Copy of Recipe parameters.
-    :type param: dict
-
-    :param req: Copy of Recipe requirements
-    :type req: dict
-
-    :param environ: A copy of :py:data:`os.environ` created when this object is
instantiated.
-    :type environ: dict
-
-    :param run: instance of the run
-    :type run: :py:class:`lnst.Controller.Recipe.RecipeRun`
-
-    :param datetime: A time stamp that is the result of running
:py:meth:`datetime.datetime.now` during instantiation
-    :type datetime: :py:class:`datetime.datetime`
-    """
-    def __init__(self, run: RecipeRun):
-        self.params = run.recipe.params._to_dict()
-        self.req = run.recipe.req._to_dict()
-        self.environ = os.environ.copy()
-        self.run = run
-        self.datetime = datetime.datetime.now()
-
-
-def export_recipe_run(run: RecipeRun, export_dir: str = None, name: str = None) ->
str:
-    """
-    Export a recipe run to a file. Run is encapsulated, pickled and compressed.
-
-    :param run: `RecipeRun` object to export.
-    :type run: :py:class: `lnst.Controller.Recipe.RecipeRun`
-    :param export_dir: Directory to export file to. Defaults to :py:attr:`run.log_dir`
-    :type export_dir: str
-    :param name: Name of output (exclusive of directory). Defaults to
`<recipename>-run-<timestamp>.lrc`.
-    :type name: str
-    :return: Path of output file.
-    :rtype: str
-
-    Example::
-
-        ctl = Controller(...)
-        recipe = BondRecipe(...)
-        ctl.run(recipe)
-        ...
-        >>> path = export_recipe_run(recipe.run[0])
-        2020-10-02 15:20:58 (localhost) - INFO: Exported BondRecipe run data to
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
-        >>> print(path)
-       
/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
-
-    """
-    run_data = RecipeRunData(run)
-    if not name:
-        name =
f"{run.recipe.name}-run-{run_data.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
-    if not export_dir:
-        export_dir = run.log_dir
-
-    path = os.path.join(export_dir, name)
-    with lzma.open(path, 'wb') as f:
-        pickle.dump(run_data, f)
-    logging.info(f"Exported {run.recipe.name} run data to {path}")
-    return path
-
-
-def import_recipe_run(path: str) -> RecipeRunData:
-    """
-    Import a recipe run that was exported using :py:meth:`export_recipe_run`
-
-    :param path: Path to file to import
-    :type path:  str
-    :return: object which contains the imported recipe run
-    :rtype:  :py:class:`RecipeRunData`
-
-    Example::
-
-        >>> run_data =
import_recipe_run("/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc")
-        >>> type(run_data)
-        <class 'lnst.Controller.RecipeRunExport.RecipeRunData'>
-        >>> run_data.recipe.__class__
-        <class 'lnst.Recipes.ENRT.BondRecipe.BondRecipe'>
-        >>> run_data.run.results[38]
-        <lnst.Controller.RecipeResults.Result object at 0x7f20727e1e20>
-        >>> run_data.run.results[38].data
-        {'cpu': [[[<lnst.RecipeCommon.Perf.Results.PerfInterval object at
0x7f20727e1e50>,...]]], ... }
-        >>> print(run_data.run.results[38].description)
-        CPU Utilization on host host1:
-        cpu 'cpu': 45.40 +-0.00 time units per second
-        cpu 'cpu0': 45.40 +-0.00 time units per second
-    """
-    with lzma.open(path, 'rb') as f:
-        run_data = pickle.load(f)
-    return run_data
-- 
2.26.2

[PATCH 5/5] Recipe.py: Removed Recipe name property.

From: Perry Gagne <pgagne(a)redhat.com&gt;

After feedback from the team, we decided not to have this.

Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
---
 lnst/Controller/Recipe.py | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
index eb9b344..9617495 100644
--- a/lnst/Controller/Recipe.py
+++ b/lnst/Controller/Recipe.py
@@ -130,9 +130,6 @@ class BaseRecipe(object):
     def _set_ctl(self, ctl):
         self._ctl = ctl
 
-    @property
-    def name(self):
-        return self.__class__.__name__
 
     @property
     def matched(self):
@@ -254,14 +251,14 @@ def export_recipe_run(run: RecipeRun, export_dir: str = None, name:
str = None)
 
     """
     if not name:
-        name = f"{run.recipe.name}-run-{run.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
+        name =
f"{run.recipe.__class__.__name__}-run-{run.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
     if not export_dir:
         export_dir = run.log_dir
 
     path = os.path.join(export_dir, name)
     with lzma.open(path, 'wb') as f:
         pickle.dump(run, f)
-    logging.info(f"Exported {run.recipe.name} run to {path}")
+    logging.info(f"Exported {run.recipe.__class__.__name__} run to {path}")
     return path
 
 
-- 
2.26.2

[PATCH 5/5] Recipe.py: Removed Recipe name property.

[Jan Tluka]

Wed, Oct 14, 2020 at 04:17:20PM CEST, pgagne(a)redhat.com wrote:
> From: Perry Gagne <pgagne(a)redhat.com&gt;
>
> After feedback from the team, we decided not to have this.
>  
So why not to merge this into one of the previous commits in the patch
set that added this?

-Jan

> Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
> ---
>  lnst/Controller/Recipe.py | 7 ++-----
>  1 file changed, 2 insertions(+), 5 deletions(-)
>
> diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
> index eb9b344..9617495 100644
> --- a/lnst/Controller/Recipe.py
> +++ b/lnst/Controller/Recipe.py
> @@ -130,9 +130,6 @@ class BaseRecipe(object):
>      def _set_ctl(self, ctl):
>          self._ctl = ctl
>  
> -    @property
> -    def name(self):
> -        return self.__class__.__name__
>  
>      @property
>      def matched(self):
> @@ -254,14 +251,14 @@ def export_recipe_run(run: RecipeRun, export_dir: str = None, name:
> str = None)
>  
>      """
>      if not name:
> -        name = f"{run.recipe.name}-run-{run.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
> +        name =
> f"{run.recipe.__class__.__name__}-run-{run.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
>      if not export_dir:
>          export_dir = run.log_dir
>  
>      path = os.path.join(export_dir, name)
>      with lzma.open(path, 'wb') as f:
>          pickle.dump(run, f)
> -    logging.info(f"Exported {run.recipe.name} run to {path}")
> +    logging.info(f"Exported {run.recipe.__class__.__name__} run to {path}")
>      return path
>  
>  
> -- 
> 2.26.2
> _______________________________________________
> LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...

[PATCH 1/5] RecipeResults.py: Stop excluding RemoteDevices from export

[Jan Tluka]

Wed, Oct 14, 2020 at 04:17:16PM CEST, pgagne(a)redhat.com wrote:
> From: Perry Gagne <pgagne(a)redhat.com&gt;
>
> Previously we were excluding RemoteDevice from export due to a number of its attributes
> containing references to low level PYCapsule and socket stuff.
>
> I took another look at this to be more precise about what is excluded.
>
> Also, with Ondrej help I was able to find the cause of an issue I a ran into previously
> where an infinite loop of calls to RemoteDevice.__setattr__ would occur when trying to
> unpickle a exported run.
>  
Could you please wrap these long lines? Applies to rest of the patches
as well.

Secondly, all patches contain "From: Perry Gagne <pgagne(a)redhat.com&gt;&quot; in
the
patch description. Please fix this.

> Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
> ---
>  lnst/Common/ConnectionHandler.py | 7 +++++++
>  lnst/Controller/Machine.py       | 8 ++++++++
>  lnst/Controller/RecipeResults.py | 5 -----
>  lnst/Devices/RemoteDevice.py     | 2 ++
>  4 files changed, 17 insertions(+), 5 deletions(-)
>
> diff --git a/lnst/Common/ConnectionHandler.py b/lnst/Common/ConnectionHandler.py
> index a74d4a8..04bf7be 100644
> --- a/lnst/Common/ConnectionHandler.py
> +++ b/lnst/Common/ConnectionHandler.py
> @@ -133,3 +133,10 @@ class ConnectionHandler(object):
>      def clear_connections(self):
>          self._connections = []
>          self._connection_mapping = {}
> +
> +    def __getstate__(self):
> +        state = self.__dict__.copy()
> +        # Remove things that can't be pickled
> +        state['_connections'] = []
> +        state['_connection_mapping'] = {}
> +        return state
> \ No newline at end of file
> diff --git a/lnst/Controller/Machine.py b/lnst/Controller/Machine.py
> index f63fd6f..7ef3204 100644
> --- a/lnst/Controller/Machine.py
> +++ b/lnst/Controller/Machine.py
> @@ -591,3 +591,11 @@ class Machine(object):
>  
>      def get_security(self):
>          return self._security
> +
> +    def __getstate__(self):
> +        state = self.__dict__.copy()
> +        # Remove things that can't be pickled
> +        state['_msg_dispatcher'] = None
> +        if self.get_libvirt_domain():
> +            state['_domain_ctl'] = None
> +        return state
> \ No newline at end of file
> diff --git a/lnst/Controller/RecipeResults.py b/lnst/Controller/RecipeResults.py
> index 941ea8d..38a4170 100644
> --- a/lnst/Controller/RecipeResults.py
> +++ b/lnst/Controller/RecipeResults.py
> @@ -118,11 +118,6 @@ class DeviceConfigResult(BaseResult):
>      def device(self):
>          return self._device
>  
> -    def __getstate__(self):
> -        state = self.__dict__.copy()
> -        # Remove things that can't be pickled
> -        state['_device'] = None
> -        return state
>  
>  class DeviceCreateResult(DeviceConfigResult):
>      @property
> diff --git a/lnst/Devices/RemoteDevice.py b/lnst/Devices/RemoteDevice.py
> index f11fa65..7891514 100644
> --- a/lnst/Devices/RemoteDevice.py
> +++ b/lnst/Devices/RemoteDevice.py
> @@ -108,6 +108,8 @@ class RemoteDevice(object):
>      def __getattr__(self, name):
>          if name == "_inited":
>              return False
> +        if not self._inited:
> +            return super(RemoteDevice, self).__getattr__(name)
>  
>          attr = getattr(self._dev_cls, name)
>  
> -- 
> 2.26.2
> _______________________________________________
> LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...

[PATCH 2/5] RecipeRunExport.py: Changed API, added compression, fixedup docs.

[Jan Tluka]

Wed, Oct 14, 2020 at 04:17:17PM CEST, pgagne(a)redhat.com wrote:
> From: Perry Gagne <pgagne(a)redhat.com&gt;
>
> We decided to simplify the API a bit to just have and import and export method. We also
> decided to add compression to the exported file.
>
> Updated docs with new API and examples.
>
> Recipe.py: Added recipe reference to RecipeRun so we only have to specify one variable
> when exporting. Recipe is still excluded from export.
>
> Also added shortcut for recipe name to fix LNST-project/lnst#194
>
> tester_api.rst: Added link to docs for RecipeRunExport
>  
That's three things in one patch. Two of them I consider bigger
features.

Could you split at least the compression and API simplification into two
patches? It makes it much easier to review.

-Jan

> Signed-off-by: Perry Gagne <pgagne(a)redhat.com&gt;
> ---
>  docs/source/recipe_run_export.rst  |   5 +
>  docs/source/tester_api.rst         |   1 +
>  lnst/Controller/Controller.py      |   2 +-
>  lnst/Controller/Recipe.py          |  16 ++-
>  lnst/Controller/RecipeRunExport.py | 161 +++++++++++++++--------------
>  5 files changed, 107 insertions(+), 78 deletions(-)
>  create mode 100644 docs/source/recipe_run_export.rst
>
> diff --git a/docs/source/recipe_run_export.rst b/docs/source/recipe_run_export.rst
> new file mode 100644
> index 0000000..6d366c2
> --- /dev/null
> +++ b/docs/source/recipe_run_export.rst
> @@ -0,0 +1,5 @@
> +Export/Import of Recipe Runs
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +.. automodule:: lnst.Controller.RecipeRunExport
> +    :members:
> \ No newline at end of file
> diff --git a/docs/source/tester_api.rst b/docs/source/tester_api.rst
> index e7cf743..65a2e1b 100644
> --- a/docs/source/tester_api.rst
> +++ b/docs/source/tester_api.rst
> @@ -8,3 +8,4 @@ Test developer API
>     recipe_api
>     controller_api
>     parameters
> +   recipe_run_export
> diff --git a/lnst/Controller/Controller.py b/lnst/Controller/Controller.py
> index 1641ed3..d63ec90 100644
> --- a/lnst/Controller/Controller.py
> +++ b/lnst/Controller/Controller.py
> @@ -156,7 +156,7 @@ class Controller(object):
>                  logging.info(line)
>              try:
>                  self._map_match(match, req, recipe)
> -                recipe._init_run(RecipeRun(match,
> log_dir=self._log_ctl.get_recipe_log_path()))
> +                recipe._init_run(RecipeRun(recipe, match,
> log_dir=self._log_ctl.get_recipe_log_path()))
>                  recipe.test()
>              except Exception as exc:
>                  logging.error("Recipe execution terminated by unexpected
> exception")
> diff --git a/lnst/Controller/Recipe.py b/lnst/Controller/Recipe.py
> index a9f7514..3bb6793 100644
> --- a/lnst/Controller/Recipe.py
> +++ b/lnst/Controller/Recipe.py
> @@ -126,6 +126,10 @@ class BaseRecipe(object):
>      def _set_ctl(self, ctl):
>          self._ctl = ctl
>  
> +    @property
> +    def name(self):
> +        return self.__class__.__name__
> +
>      @property
>      def matched(self):
>          if self.ctl is None:
> @@ -152,11 +156,12 @@ class BaseRecipe(object):
>                                             level, data_level))
>  
>  class RecipeRun(object):
> -    def __init__(self, match, desc=None, log_dir=None):
> +    def __init__(self, recipe: BaseRecipe, match, desc=None, log_dir=None):
>          self._match = match
>          self._desc = desc
>          self._results = []
>          self._log_dir = log_dir
> +        self._recipe = recipe
>  
>      def add_result(self, result):
>          if not isinstance(result, BaseResult):
> @@ -196,3 +201,12 @@ class RecipeRun(object):
>      @property
>      def overall_result(self):
>          return all([i.success for i in self.results])
> +
> +    @property
> +    def recipe(self) -> BaseRecipe:
> +        return self._recipe
> +
> +    def __getstate__(self):
> +        state = self.__dict__.copy()
> +        state['_recipe'] = None
> +        return state
> diff --git a/lnst/Controller/RecipeRunExport.py b/lnst/Controller/RecipeRunExport.py
> index cf40ae4..9486fe8 100644
> --- a/lnst/Controller/RecipeRunExport.py
> +++ b/lnst/Controller/RecipeRunExport.py
> @@ -1,20 +1,33 @@
> +"""This module provides an API to export data related to a recipe run.
> This export can be used to gather data like  CPU and iperf statistic for later analysis,
> like when onboarding a new Recipe.
> +
> +The data that is exported is the instance of :py:class:`lnst.Controller.Recipe.RecipeRun`
> that was run.
> +
> +Data is encapsuled in an :py:class:`RecipeRunData` object which allows us to include
> other metadata. The `RecipeRunData` object is "pickled" and compressed with
> LZMA/XZ compression using :py:mod:`lzma`.
> +
> +By default the file will be contain a file extension `.lrc` which stands for "LNST
> Run, Compressed".
> +
> +Use :py:meth:`export_recipe_run` to export and :py:meth:`import_recipe_run` to import.
> +"""
> +
> +__author__ = """
> +pgagne(a)redhat.com (Perry Gagne)
> +"""
> +
>  import datetime
>  import pickle
>  import os
>  import logging
> -from typing import List, Tuple
> -from lnst.Controller.Recipe import BaseRecipe, RecipeRun
> +import lzma
> +from lnst.Controller.Recipe import RecipeRun
>  
>  
>  class RecipeRunData:
>      """
> -    Class used to encapsulate a RecipeRun, this is the object
> -    that will be pickled and output to a file.
> +    Class used to encapsulate a :py:class:`lnst.Controller.Recipe.RecipeRun`, this is the
> object
> +    that will be pickled, compressed and output to a file.
>  
> -    :param recipe_cls:
> -        class of the Recipe. We do not currently pickle the instance
> -        of the recipe itself for ease of exporting.
> -    :type recipe_cls: :py:class: `lnst.Controller.Recipe.BaseRecipe`
> +    :param recipe_cls: class of the Recipe. We do not currently pickle the instance of
> the recipe itself.
> +    :type recipe_cls: :py:class:`lnst.Controller.Recipe.BaseRecipe`
>  
>      :param param: Copy of Recipe parameters.
>      :type param: dict
> @@ -22,91 +35,87 @@ class RecipeRunData:
>      :param req: Copy of Recipe requirements
>      :type req: dict
>  
> -    :param environ: A copy of `os.environ` created when the object is instantiated.
> +    :param environ: A copy of :py:data:`os.environ` created when this object is
> instantiated.
>      :type environ: dict
>  
> -    :param run: :py:class:`lnst.Controller.Recipe.RecipeRun` instance of the run
> +    :param run: instance of the run
>      :type run: :py:class:`lnst.Controller.Recipe.RecipeRun`
>  
> -    :param datetime: A time stamp that is the result of running `datetime.datetime.now()`
> during instantiation
> +    :param datetime: A time stamp that is the result of running
> :py:meth:`datetime.datetime.now` during instantiation
>      :type datetime: :py:class:`datetime.datetime`
>      """
> -
> -    def __init__(self, recipe: BaseRecipe, run: RecipeRun):
> -        self.recipe_cls = recipe.__class__
> -        self.params = recipe.params._to_dict()
> -        self.req = recipe.req._to_dict()
> +    def __init__(self, run: RecipeRun):
> +        self.recipe_cls = run.recipe.__class__
> +        self.params = run.recipe.params._to_dict()
> +        self.req = run.recipe.req._to_dict()
>          self.environ = os.environ.copy()
>          self.run = run
>          self.datetime = datetime.datetime.now()
>  
>  
> -class RecipeRunExporter:
> +def export_recipe_run(run: RecipeRun, export_dir: str = None, name: str = None) ->
> str:
>      """
> -    Class used to export recipe runs.
> +    Export a recipe run to a file. Run is encapsulated, pickled and compressed.
> +
> +    :param run: `RecipeRun` object to export.
> +    :type run: :py:class: `lnst.Controller.Recipe.RecipeRun`
> +    :param export_dir: Directory to export file to. Defaults to :py:attr:`run.log_dir`
> +    :type export_dir: str
> +    :param name: Name of output (exclusive of directory). Defaults to
> `<recipename>-run-<timestamp>.lrc`.
> +    :type name: str
> +    :return: Path of output file.
> +    :rtype: str
> +
> +    Example::
> +
> +        ctl = Controller(...)
> +        recipe = BondRecipe(...)
> +        ctl.run(recipe)
> +        ...
> +        >>> path = export_recipe_run(recipe.run[0])
> +        2020-10-02 15:20:58 (localhost) - INFO: Exported BondRecipe run data to
> /tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
> +        >>> print(path)
> +       
> /tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc
>  
>      """
> +    run_data = RecipeRunData(run)
> +    if not name:
> +        name =
> f"{run.recipe.name}-run-{run_data.datetime:%Y-%m-%d_%H:%M:%S}.lrc"
> +    if not export_dir:
> +        export_dir = run.log_dir
>  
> -    def __init__(self, recipe: BaseRecipe):
> -        """
> -
> -        :param recipe: Recipe
> -        :type recipe: :py:class: `lnst.Controller.Recipe.BaseRecipe`
> -        """
> -        self.recipe = recipe
> -        self.recipe_name = self.recipe.__class__.__name__
> -
> -    def export_run(self, run: RecipeRun, dir=None, name=None) -> str:
> -        """
> -
> -        :param run: The RecipeRun to export
> -        :type run: :py:class:`lnst.Controller.Recipe.RecipeRun`
> -        :param dir: The path to directory to export to. Does not include file name,
> defaults to `run.log_dir`
> -        :type dir: str
> -        :param name: Name of file to export. Default
> `<recipename>-run-<timestamp>.dat'
> -        :type name: str
> -        :return: Path (dir+filename) of exported run.
> -        :rtype:str
> -        """
> -        data = RecipeRunData(self.recipe, run)
> -        if not name:
> -            name =
> f"{self.recipe_name}-run-{data.datetime:%Y-%m-%d_%H:%M:%S}.dat"
> -        if not dir:
> -            dir = run.log_dir
> -
> -        path = os.path.join(dir, name)
> -
> -        with open(path, 'wb') as f:
> -            pickle.dump(data, f)
> -
> -        logging.info(f"Exported {self.recipe_name} data to {path}")
> -        return path
> -
> -
> -def export_recipe_runs(recipe: BaseRecipe) -> List[Tuple[str, RecipeRun]]:
> -    """
> -    Helper method that exports all runs in a recipe.
> -    :param recipe: Recipe to export
> -    :type recipe: :py:class: `lnst.Controller.Recipe.BaseRecipe`
> -    :return: list of files that contain exported recipe runs
> -    :rtype: list
> -    """
> -    exporter = RecipeRunExporter(recipe)
> -    files = []
> -    for run in recipe.runs:
> -        path = exporter.export_run(run)
> -        files.append((run, path))
> -    return files
> +    path = os.path.join(export_dir, name)
> +    with lzma.open(path, 'wb') as f:
> +        pickle.dump(run_data, f)
> +    logging.info(f"Exported {run.recipe.name} run data to {path}")
> +    return path
>  
>  
>  def import_recipe_run(path: str) -> RecipeRunData:
>      """
> -    Import recipe runs that have been exported using
> :py:class:`lnst.Controller.RecipeRunExport.RecipeRunExporter`
> -    :param path:  path to file  to import
> -    :type path: str
> -    :return: `RecipeRun` object containing the run and other metadata.
> -    :rtype: :py:class:`lnst.Controller.RecipeRunExport.RecipeRunData`
> +    Import a recipe run that was exported using :py:meth:`export_recipe_run`
> +
> +    :param path: Path to file to import
> +    :type path:  str
> +    :return: object which contains the imported recipe run
> +    :rtype:  :py:class:`RecipeRunData`
> +
> +    Example::
> +
> +        >>> run_data =
> import_recipe_run("/tmp/lnst-logs/2020-10-02_15:20:18/BondRecipe_match_0/BondRecipe-run-2020-10-02_15:20:58.lrc")
> +        >>> type(run_data)
> +        <class 'lnst.Controller.RecipeRunExport.RecipeRunData'>
> +        >>> run_data.recipe_cls
> +        <class 'lnst.Recipes.ENRT.BondRecipe.BondRecipe'>
> +        >>> run_data.run.results[38]
> +        <lnst.Controller.RecipeResults.Result object at 0x7f20727e1e20>
> +        >>> run_data.run.results[38].data
> +        {'cpu': [[[<lnst.RecipeCommon.Perf.Results.PerfInterval object at
> 0x7f20727e1e50>,...]]], ... }
> +        >>> print(run_data.run.results[38].description)
> +        CPU Utilization on host host1:
> +        cpu 'cpu': 45.40 +-0.00 time units per second
> +        cpu 'cpu0': 45.40 +-0.00 time units per second
>      """
> -    with open(path, 'rb') as f:
> -        data = pickle.load(f)
> -    return data
> +    with lzma.open(path, 'rb') as f:
> +        run_data = pickle.load(f)
> +    return run_data
> -- 
> 2.26.2
> _______________________________________________
> LNST-developers mailing list -- lnst-developers(a)lists.fedorahosted.org
> To unsubscribe send an email to lnst-developers-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...

[PATCH 1/5] RecipeResults.py: Stop excluding RemoteDevices from export

>  Secondly, all patches contain "From: Perry Gagne
> <pgagne(a)redhat.com&gt;&quot;
>  in the patch description. Please fix this. 
I  think someone may of mentioned this before, and I looked into my
config, but couldn't figure out how to make it not include that. Any
ideas? Here is my git config:

$ git config --global -l
user.name=Perry Gagne
user.email=pgagne(a)redhat.com
sendemail.smtpserver=/usr/bin/msmtp
sendemail.from=pgagne(a)redhat.com
core.editor=vim
format.signoff=true
commit.signoff=true
credential.helper=libsecret

[PATCH 2/5] RecipeRunExport.py: Changed API, added compression, fixedup docs.

>  Could you split at least the compression and API simplification into
>  two patches? It makes it much easier to review. 
Its wasn't really done in a way that I could of done that. Because I
had to change a bunch of small things, I bascially rewrote the
import/export methods from scratch including adding compression at the
same time. 

The only major difference with compression was the use of "lzma.open"
instead of regular "open". 

I might be able to make the patch series a little cleaner though in
otherways

[PATCH 1/5] RecipeResults.py: Stop excluding RemoteDevices from export

On Thu, 2020-10-15 at 10:56 +0200, Jan Tluka wrote:
>  Could you please wrap these long lines? Applies to rest of the
>  patches as well. 
Are you refering to lines in the code or commit messages or both?

[PATCH 1/5] RecipeResults.py: Stop excluding RemoteDevices from export

[Jan Tluka]

Tue, Oct 20, 2020 at 09:59:27PM CEST, pgagne(a)redhat.com wrote:
> On Thu, 2020-10-15 at 10:56 +0200, Jan Tluka wrote:
> > Could you please wrap these long lines? Applies to rest of the
> > patches as well.
>
> Are you refering to lines in the code or commit messages or both?
>  
Just the commit message.