[PATCH 0/4] Fix namespaces
by Jan Tluka
This patchset fixes namespace in LNST that is currently broken. The
fundamental problem was in use of libc calls with improper parameter
types.
Thanks to Ondrej for the additional patch in [1].
[1] https://github.com/LNST-project/lnst/issues/202#issuecomment-820398056
Jan Tluka (4):
Controller.NetNamespace: rename _nsname to _name
lnst.NetTestSlave: fix namespace creation/deletion
Controller.Host: pass NetNamespace to add_netns()
Controller.Machine: update add_namespace and del_namespace arguments
lnst/Controller/Host.py | 2 +-
lnst/Controller/Machine.py | 4 +--
lnst/Controller/NetNamespace.py | 2 +-
lnst/Slave/NetTestSlave.py | 47 +++++++++++++++++++++++++--------
4 files changed, 40 insertions(+), 15 deletions(-)
--
2.26.3
3 years
[PATCH] NeperFlowMeasurement: fix cpu usage PerfInterval calculation
by olichtne＠redhat.com
From: Ondrej Lichtner <olichtne(a)redhat.com>
The original calculation creates the ratio of the "used" cpu time vs the
elapsed real time, however to create a percentage we need to multiply by
100. Without this fix the reported values were always between 0 and 1
would seem confusing in the context of other cpu metrics where a 0 to
100 value is reported.
At the same time, the PerfInterval value should be multiplied by the
actual duration of the interval so that the average calculation makes
sense. This is required for metrics that don't represent a useful
"cumulative" unit/metric - throughput is a metric that accumulates
transferred bytes over time, however percentage cpu utilization doesn't
accumulate over time.
Without this fix, sometimes a short start or end interval (e.g. duration
of 0.00005 seconds) would end up confusing and poisoning the calculation
of the entire Sequence of Intervals.
Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com>
---
lnst/RecipeCommon/Perf/Measurements/NeperFlowMeasurement.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lnst/RecipeCommon/Perf/Measurements/NeperFlowMeasurement.py b/lnst/RecipeCommon/Perf/Measurements/NeperFlowMeasurement.py
index f271e1b3..1e05b8e0 100644
--- a/lnst/RecipeCommon/Perf/Measurements/NeperFlowMeasurement.py
+++ b/lnst/RecipeCommon/Perf/Measurements/NeperFlowMeasurement.py
@@ -220,9 +220,9 @@ def get_interval(s_start: Dict, s_end: Dict, job_start: float,
# unix time using job_start as reference
timestamp = job_start + (s_start_time - neper_start)
duration = s_end_time - s_start_time
- cpu_usage = (utime_delta + stime_delta) / duration
+ cpu_usage = ((utime_delta + stime_delta) / duration) * 100
interval = PerfInterval(transactions, duration, 'transactions', timestamp)
- cpu_interval = PerfInterval(cpu_usage, duration, 'cpu_percent', timestamp)
+ cpu_interval = PerfInterval(cpu_usage * duration, duration, 'cpu_percent', timestamp)
return interval, cpu_interval
--
2.31.1
3 years
[PATCH 0/2] Fixes to remove tcp_crr server side stats
by pgagne＠redhat.com
From: Perry Gagne <pgagne(a)redhat.com>
We decided to ignore tcp_crr stats for now due to a few reasons:
1. It was removed from neper upstream due to memory issues
2. The resulting LRC files where big
3. We were probably not caluclating stats in the first place in the crr
case
4. Neper's stats collection probably needs work to properly collect
stats.
For now we decided to record the server side stats as PerfIntervals of
0.
These patches contain some things to enable that.
Perry Gagne (2):
Utils.py: Added 3.6 backport of contextlib.nullcontext
NeperFlowMeasurement.py: Don't collect tcp_crr server side metric
lnst/Common/Utils.py | 25 ++++++++++++++++
.../Perf/Measurements/NeperFlowMeasurement.py | 8 +++++
lnst/Tests/Neper.py | 29 ++++++++++++++-----
3 files changed, 54 insertions(+), 8 deletions(-)
--
2.30.2
3 years
[DISCUSSION] python version requirement
by Ondrej Lichtner
Hi all,
since we've moved to python3 that is actively developed and versions
move between various long term/short term support cycles, we should also
adapt LNST to this cycle of updating which minimal version of python
LNST requires.
TL;DR: The main questions I'm asking are:
* how do we _implement_ a python version requirement?
* how do we _upgrade_/_migrate_ in the future?
* how do we _document_ a python version requirement?
* which version do we want to use _now_?
More context:
I think at the moment we have a "soft" requirement for python3.6. Soft
because we:
* probably haven't tested on anything older
* it's not explicitly configured/documented anywhere
At the same time, there are now at least two reasons to start thinking
about moving to python3.8:
* I remember Perry asking about the f-string feature introduced in 3.8
* while working with Adrian on the TRex refactoring I started thinking
about a feature for the lnst.Tests package that I've had in mind for a
while, which requires a 3.7 feature
* python3.8 is the current version on Fedora32, is available in RHEL8
(via dnf install python38), and python3.7 was skipped
The lnst.Tests feature I'm thinking of is "lazy" and "dynamic" loading
of BaseTestModule derived modules - for example at the moment, if a
Recipe imports any module from lnst.Tests (e.g. lnst.Tests.Ping), the
entire package is parsed and "loaded", which means that the python
environment will also parse and load lnst.Tests.TRex. This means that a
basic hello world recipe that simply calls Ping, will in some way
require load time dependencies of TRex.
The "lazy" and "dynamic" loading of test modules would ensure that when
a recipe calls:
from lnst.Tests import Ping
Only the Ping module will be parsed, loaded and imported, and nothing
else. And the dynamicity here could mean that we could be able to extend
test modules exported by the lnst.Tests package via the lnst-ctl config
file, for example for user/tester implemented test modules that are not
tracked in the main lnst repository.
I wrote a rough patch to experiment with this:
---
diff --git a/lnst/Tests/__init__.py b/lnst/Tests/__init__.py
index f7c6c90..a39b6f4 100644
--- a/lnst/Tests/__init__.py
+++ b/lnst/Tests/__init__.py
@@ -12,8 +12,26 @@
olichtne(a)redhat.com (Ondrej Lichtner)
"""
-from lnst.Tests.Ping import Ping
-from lnst.Tests.PacketAssert import PacketAssert
-from lnst.Tests.Iperf import IperfClient, IperfServer
+# from lnst.Tests.Ping import Ping
+# from lnst.Tests.PacketAssert import PacketAssert
+# from lnst.Tests.Iperf import IperfClient, IperfServer
+import importlib
+
+lazy_load_modules = {
+ "Ping": "lnst.Tests.Ping",
+ "PacketAssert": "lnst.Tests.PacketAssert",
+ "IperfClient": "lnst.Tests.Iperf",
+ "IperfServer": "lnst.Tests.Iperf",
+}
+
+
+def __getattr__(name):
+ if name not in lazy_load_modules:
+ raise ImportError("Cannot import {}".format(name))
+ mod = importlib.import_module(lazy_load_modules[name])
+ globals()[name] = getattr(mod, name)
+ return globals()[name]
+
+
+# #TODO add support for test classes from lnst-ctl.conf
-#TODO add support for test classes from lnst-ctl.conf
---
However this requires the ability to define __getattr__ for a module,
which is introduced as a python3.7 feature via PEP562 [0].
-Ondrej
[0] https://www.python.org/dev/peps/pep-0562/
3 years
[PATCH] Tests.PacketAssert: continuosly read tcpdump stdout
by olichtne＠redhat.com
From: Ondrej Lichtner <olichtne(a)redhat.com>
When PacketAssert is used for longer or larger streams the buffer for
the pipe file used for stdout can be fully filled and tcpdump will get
stuck on the write syscall. After that when the SIGINT comes from the
recipe this syscall is interrupted and tcpdump quits with:
tcpdump: Unable to write output: Interrupted system call
The fix in this patch is in two parts:
* use the '-U' argument which makes the tcpdump output packet buffered -
fflush is called after each packet is analyzeed and printed.
* rewrite the wait_for_interrupt use with custom SIGINT handling that
continuosly reads lines from stdout or stderr so that the buffer is
never full
Eventually similar functionality may be useful for other Test modules as
well, at that point this should be refactored.
Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com>
---
lnst/Tests/PacketAssert.py | 32 ++++++++++++++++++++++++++------
1 file changed, 26 insertions(+), 6 deletions(-)
diff --git a/lnst/Tests/PacketAssert.py b/lnst/Tests/PacketAssert.py
index 5fd7abfd..9b623b0b 100644
--- a/lnst/Tests/PacketAssert.py
+++ b/lnst/Tests/PacketAssert.py
@@ -1,6 +1,8 @@
import re
import logging
import subprocess
+import signal
+from select import select
from lnst.Common.Parameters import (
StrParam,
ListParam,
@@ -10,10 +12,13 @@
)
from lnst.Devices.Device import Device
from lnst.Common.Utils import is_installed
-from lnst.Tests.BaseTestModule import BaseTestModule
+from lnst.Tests.BaseTestModule import BaseTestModule, InterruptException
from lnst.Common.LnstError import LnstError
+def interrupt_handler(signum, frame):
+ raise InterruptException()
+
class PacketAssert(BaseTestModule):
interface = DeviceParam(mandatory=True)
p_filter = StrParam(default="")
@@ -33,7 +38,7 @@ def _compose_cmd(self):
cmd += " -p"
iface = self.params.interface.name
filt = self.params.p_filter
- cmd += ' -nn -i %s "%s"' % (iface, filt)
+ cmd += ' -nn -U -i %s "%s"' % (iface, filt)
return cmd
@@ -63,12 +68,27 @@ def run(self):
close_fds=True,
)
+ stdout = bytearray()
+ stderr = bytearray()
try:
- self.wait_for_interrupt()
- except:
- raise LnstError("Could not handle interrupt properly.")
+ old_handler = signal.signal(signal.SIGINT, interrupt_handler)
+ # for longer, or larger streams tcpdump can fill the entire io
+ # buffer for the stdout/stderr files, because of that we need to
+ # read the files continuosly
+ while True:
+ rl, wl, xl = select([packet_assert_process.stdout, packet_assert_process.stderr], [], [])
+ if packet_assert_process.stdout in rl:
+ stdout += packet_assert_process.stdout.readline()
+ if packet_assert_process.stderr in rl:
+ stderr += packet_assert_process.stderr.readline()
+ except InterruptException:
+ pass
+ finally:
+ signal.signal(signal.SIGINT, old_handler)
- stdout, stderr = packet_assert_process.communicate()
+ packet_assert_process.wait()
+ stdout += packet_assert_process.stdout.read()
+ stderr += packet_assert_process.stderr.read()
stdout = stdout.decode()
stderr = stderr.decode()
--
2.31.0
3 years
