Change in vdsm[master]: tests: Add time per test if nose-timer is available

Antoni Segura Puimedon has uploaded a new change for review.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

tests: Add time per test if nose-timer is available

This patch adds an output like:

resourceManagerTests.ResourceManagerTests.testStressTest: 12.8313s
    resourceManagerTests.ResourceManagerTests.testResourceAcquireTimeout: 1.0069s
    resourceManagerTests.ResourceManagerTests.testResourceLockSwitch: 0.0108s
    resourceManagerTests.ResourceManagerTests.testResourceAutorelease: 0.0101s
    resourceManagerTests.ResourceManagerTests.testResourceInvalidation: 0.0093s
    resourceManagerTests.ResourceManagerTests.testCrashOnSwitch: 0.0067s
    resourceManagerTests.ResourceManagerTests.testResourceSwitchLockTypeFail:
    0.0062s
    resourceManagerTests.ResourceManagerTests.testCancelExclusiveBetweenShared:
    0.0049s
    resourceManagerTests.ResourceManagerTests.testAcquireResourceExclusive: 0.0042s
    resourceManagerTests.ResourceManagerTests.testCancelRequest: 0.0031s
    resourceManagerTests.ResourceManagerTests.testRequestRefCmp: 0.0030s
    resourceManagerTests.ResourceManagerTests.testRequestRecancel: 0.0030s
    resourceManagerTests.ResourceManagerTests.testResourceStatuses: 0.0029s
    resourceManagerTests.ResourceManagerTests.testAccessAttributeNotExposedByRequestRef:
    0.0029s
    resourceManagerTests.ResourceManagerTests.testFailCreateAfterSwitch: 0.0025s
    resourceManagerTests.ResourceManagerTests.testRequestWithBadCallbackOnCancel:
    0.0025s
    resourceManagerTests.ResourceManagerTests.testAcquireResourceShared: 0.0023s
    resourceManagerTests.ResourceManagerTests.testRereleaseResource: 0.0022s
    resourceManagerTests.ResourceManagerTests.testResourceWrapper: 0.0020s
    resourceManagerTests.ResourceManagerTests.testRequestWithBadCallbackOnGrant:
    0.0020s
    resourceManagerTests.ResourceManagerTests.testRequestRefStr: 0.0020s
    resourceManagerTests.ResourceManagerTests.testAccessAttributeNotExposedByWrapper:
    0.0019s
    resourceManagerTests.ResourceManagerTests.testErrorInFactory: 0.0017s
    resourceManagerTests.ResourceManagerTests.testAcquireNonExistingResource:
    0.0016s
    resourceManagerTests.ResourceManagerTests.testRequestInvalidResource: 0.0015s
    resourceManagerTests.ResourceManagerTests.testRequestRegrant: 0.0015s
    resourceManagerTests.ResourceManagerTests.testResourceAcquireInvalidTimeout:
    0.0014s
    resourceManagerTests.ResourceManagerTests.testReleaseInvalidResource: 0.0014s
    resourceManagerTests.ResourceManagerTests.testSingleton: 0.0014s
    resourceManagerTests.ResourceManagerTests.testForceRegisterNamespace: 0.0013s
    resourceManagerTests.ResourceManagerTests.testListNamespaces: 0.0013s
    resourceManagerTests.ResourceManagerTests.testReregisterNamespace: 0.0012s
    resourceManagerTests.ResourceManagerTests.testRegisterInvalidNamespace: 0.0012s

At the end of the usual tests if nose-timer is detected in the
system. It shows the tests over 500ms in yellow and those over 5s
in red.

If you want to get this output, do:
    pip install nose-timer

Change-Id: I4960b30532a84abd259fb268c7e40a1751847a96
Signed-off-by: Antoni S. Puimedon 
---
M tests/run_tests_local.sh.in
M tests/testrunner.py
2 files changed, 18 insertions(+), 1 deletion(-)

git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/71/30171/1

diff --git a/tests/run_tests_local.sh.in b/tests/run_tests_local.sh.in
index d19f22c..f51473d 100644
--- a/tests/run_tests_local.sh.in
+++ b/tests/run_tests_local.sh.in
@@ -7,4 +7,9 @@
     @top_srcdir@/tests/makecert.sh
 fi

-PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@top_srcdir@/lib:@top_srcdir@/vdsm:@top_srcdir@/client:@top_srcdir@/vdsm_api:$PYTHONPATH" "$PYTHON_EXE" @top_srcdir@/tests/testrunner.py --local-modules $@
+python -c 'import nosetimer' > /dev/null 2>&1
+if [ $? -eq 0 ]; then
+    EXTRA_ARGS='--with-timer'
+fi
+
+PYTHONDONTWRITEBYTECODE=1 LC_ALL=C PYTHONPATH="@top_srcdir@/lib:@top_srcdir@/vdsm:@top_srcdir@/client:@top_srcdir@/vdsm_api:$PYTHONPATH" "$PYTHON_EXE" @top_srcdir@/tests/testrunner.py --local-modules $@ "$EXTRA_ARGS"
diff --git a/tests/testrunner.py b/tests/testrunner.py
index eed743f..ec24c2b 100644
--- a/tests/testrunner.py
+++ b/tests/testrunner.py
@@ -40,6 +40,16 @@
 from nose import config
 from nose import core
 from nose import result
+try:
+    from nosetimer.plugin import TimerPlugin
+except ImportError:
+    timer_plugin = None
+else:
+    timer_plugin = TimerPlugin()
+    timer_plugin.enabled = True
+    timer_plugin.timer_ok = 500  # Okay <= 500ms
+    timer_plugin.timer_warning = 5000  # Warn > 5s
+    timer_plugin.timer_no_color = False  # Enable color output

from testValidation import SlowTestsPlugin, StressTestsPlugin

@@ -378,6 +388,8 @@
                          plugins=core.DefaultPluginManager())
     conf.plugins.addPlugin(SlowTestsPlugin())
     conf.plugins.addPlugin(StressTestsPlugin())
+    if timer_plugin is not None:
+        conf.plugins.addPlugin(timer_plugin)

runner = VdsmTestRunner(stream=conf.stream,
                             verbosity=conf.verbosity,

Francesco Romani has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1: Code-Review+1

nice! IIUC this is optional, so one can avoid to be overwhelmed by output if (s)he likes so.
If so, then +1 from me!

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1:

Build Successful

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/10264/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/11049/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/11206/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_network_functional_tests_gerrit/163... : SUCCESS

Antoni Segura Puimedon has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1: Verified+1

[Nir Soffer]

Nir Soffer has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1:

(1 comment)

http://gerrit.ovirt.org/#/c/30171/1/tests/testrunner.py
File tests/testrunner.py:

Line 48:     timer_plugin = TimerPlugin()
Line 49:     timer_plugin.enabled = True
Line 50:     timer_plugin.timer_ok = 500  # Okay <= 500ms
Line 51:     timer_plugin.timer_warning = 5000  # Warn > 5s
Line 52:     timer_plugin.timer_no_color = False  # Enable color output
These defaults are not correct for all tests.

Some tests are slow because there is no other way to test without waiting, and for these the coloring means nothing.

This should be per test method or per class setting, to allow meaningful output.
Line 53: 
Line 54: from testValidation import SlowTestsPlugin, StressTestsPlugin
Line 55: 
Line 56: import zombiereaper

[Nir Soffer]

Nir Soffer has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1:

(1 comment)

http://gerrit.ovirt.org/#/c/30171/1/tests/run_tests_local.sh.in
File tests/run_tests_local.sh.in:

Line 9: 
Line 10: python -c 'import nosetimer' > /dev/null 2>&1
Line 11: if [ $? -eq 0 ]; then
Line 12:     EXTRA_ARGS='--with-timer'
Line 13: fi
Can we do this in testrunner.py? can be much simpler.
Line 14:

[Dan Kenigsberg]

Dan Kenigsberg has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 1:

(1 comment)

http://gerrit.ovirt.org/#/c/30171/1/tests/testrunner.py
File tests/testrunner.py:

Line 48:     timer_plugin = TimerPlugin()
Line 49:     timer_plugin.enabled = True
Line 50:     timer_plugin.timer_ok = 500  # Okay <= 500ms
Line 51:     timer_plugin.timer_warning = 5000  # Warn > 5s
Line 52:     timer_plugin.timer_no_color = False  # Enable color output
> These defaults are not correct for all tests.

these numbers are indeed a bit arbitrary, but the alternative of tweaking the plugin and running all over our tests defining acceptable times do not sound appealing, either.

Can we drop colours instead?
Line 53: 
Line 54: from testValidation import SlowTestsPlugin, StressTestsPlugin
Line 55: 
Line 56: import zombiereaper

Antoni Segura Puimedon has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 2: Verified+1

[Jenkins CI RO]

oVirt Jenkins CI Server has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 2:

Build Successful

http://jenkins.ovirt.org/job/vdsm_master_unit_tests_gerrit_el/10905/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_unit-tests_created/11847/ : SUCCESS

http://jenkins.ovirt.org/job/vdsm_master_pep8_gerrit/11690/ : SUCCESS

[Nir Soffer]

Nir Soffer has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 2:

(1 comment)

http://gerrit.ovirt.org/#/c/30171/2/tests/run_tests_local.sh.in
File tests/run_tests_local.sh.in:

Line 9: 
Line 10: python -c 'import nosetimer' > /dev/null 2>&1
Line 11: if [ $? -eq 0 ]; then
Line 12:     EXTRA_ARGS='--with-timer'
Line 13: fi
Can we do this inside testlib.py?

I would like to eliminate this wrapper. the tests should run using:

nosetest
Line 14:

[Dan Kenigsberg]

Dan Kenigsberg has abandoned this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Abandoned

[e-mail] has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 2:

* Update tracker::IGNORE, no Bug-Url found

[Jenkins CI RO]

Jenkins CI RO has abandoned this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Abandoned

Abandoned due to no activity - please restore if still relevant

gerrit-hooks has posted comments on this change.

Change subject: tests: Add time per test if nose-timer is available
......................................................................

Patch Set 3:

* Update tracker: IGNORE, no Bug-Url found