Rework oz.git unittests

[James Laska]

Greetings,

I've been looking into integrating existing oz unittests with jenkins.  It
wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
better idea for pass/fail data over time.  One way to do this was to convert
the tests to use py.test (or unittest -- but that's a little crusty).  Py.test
can emit junit XML test output, which is jenkins friendly, and has much less
stock test class/method bloat.  I've modified the existing oz unittests to be
py.test friendly.

Another annoyance with running tests in jenkins is that the nodes need to be
manually setup with appropriate dependencies ahead of time.  While this isn't
hard, it's maintenance and one more thing to forget/fail when testing.  The new
test driver (runtests.sh) will setup a python virtualenv and install required
dependencies there.  This is intended to handle deps installation during
unittest execution.

>   .gitignore                    |    6 +
>   Makefile                      |    6 +-
>   tests/dependencies.txt        |    2 +
>   tests/factory/run.sh          |    5 -
>   tests/factory/test_factory.py |  311 +++++++++++++++++++++--------------------
>   tests/runtests.sh             |   90 ++++++++++++
>   tests/tdl/run.sh              |  128 -----------------
>   tests/tdl/test.cfg            |  135 ++++++++++++++++++
>   tests/tdl/test_tdl.py         |  131 +++++++++++++++++-
>   9 files changed, 524 insertions(+), 290 deletions(-) 
Comments appreciated.

Thanks,
James

[PATCH 1/5] Add .gitignore to ingore .pyc and other files

[James Laska]

---
 .gitignore |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)
 create mode 100644 .gitignore

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..7670203
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,6 @@
+*.gz
+*.pyc
+*.sw[po]
+*~
+tests/test_env
+tests/test_results.xml
-- 
1.7.6

[PATCH 2/5] Replace individual run.sh scripts with a single runtests.sh

[James Laska]

The runtests.sh script is responsible for ...
 * Satisfying test dependencies by creating a virtualenv
 * Running all applicable tests using py.test
---
 Makefile               |    6 +-
 tests/dependencies.txt |    2 +
 tests/factory/run.sh   |    5 --
 tests/runtests.sh      |   87 ++++++++++++++++++++++++++++++++
 tests/tdl/run.sh       |  128 ------------------------------------------------
 5 files changed, 92 insertions(+), 136 deletions(-)
 create mode 100644 tests/dependencies.txt
 delete mode 100755 tests/factory/run.sh
 create mode 100755 tests/runtests.sh
 delete mode 100755 tests/tdl/run.sh

diff --git a/Makefile b/Makefile
index 0957518..d228378 100644
--- a/Makefile
+++ b/Makefile
@@ -28,11 +28,11 @@ man2html:
 	@groff -mandoc man/oz-cleanup-cache.1 -T html > man/oz-cleanup-cache.html
 
 unittests:
-	@cd tests/tdl ; ./run.sh
-	@cd tests/factory ; ./run.sh
+	@bash tests/runtests.sh
+#	@cd tests/factory ; ./run.sh
 
 pylint:
 	pylint --rcfile=pylint.conf oz oz-install oz-customize oz-cleanup-cache
oz-generate-icicle
 
 clean:
-	rm -rf MANIFEST build dist usr *~ oz.spec *.pyc oz/*~ oz/*.pyc examples/*~ oz/auto/*~
man/*~ docs/*~ man/*.html tests/tdl/*~ tests/factory/*~
+	rm -rf MANIFEST build dist usr *~ oz.spec *.pyc oz/*~ oz/*.pyc examples/*~ oz/auto/*~
man/*~ docs/*~ man/*.html tests/tdl/*~ tests/factory/*~ tests/test_results.xml
tests/test_env
diff --git a/tests/dependencies.txt b/tests/dependencies.txt
new file mode 100644
index 0000000..7f0a740
--- /dev/null
+++ b/tests/dependencies.txt
@@ -0,0 +1,2 @@
+pytest>=2.0.1
+libxml2dom
diff --git a/tests/factory/run.sh b/tests/factory/run.sh
deleted file mode 100755
index 0e00333..0000000
--- a/tests/factory/run.sh
+++ /dev/null
@@ -1,5 +0,0 @@
-#!/bin/bash
-
-export PYTHONPATH=../..:$PYTHONPATH
-
-python test_factory.py
diff --git a/tests/runtests.sh b/tests/runtests.sh
new file mode 100755
index 0000000..961d9d4
--- /dev/null
+++ b/tests/runtests.sh
@@ -0,0 +1,87 @@
+#!/bin/bash
+# Test driver responsible for satisfying test dependencies using virtualenv,
+# and running available tests
+
+# Used for the name of the virtualenv
+ENV_PATH='tests/test_env'
+
+# pip-python's requirements
+REQ_FILE='tests/dependencies.txt'
+
+# this is to make sure that pip-python isn't going to touch anything but the
+# virtualenv
+PIP_REQUIRE_VIRTUALENV=1
+
+# this keeps track of the python version since virtualenv will use the system
+# python version by default
+PYTHON_SITELIB=$(python -c "from distutils.sysconfig import get_python_lib; print
get_python_lib(prefix='')")
+
+# Additional py.test command arguments
+PYTEST_ARGS="--junitxml=tests/test_results.xml --ignore=${ENV_PATH}"
+
+# Display basic usage
+usage()
+{
+    cat << EOF
+    usage: $0
+    This script prepares a virtualenv for running the tests (if it doesn't
+    already exist) and runs unit tests in that virtualenv.
+
+    OPTIONS:
+      -h    Show this message
+      -v    Enable Verbosity
+EOF
+    exit 1
+}
+
+# Creates and prepares a virtualenv
+create_env()
+{
+    echo "Virtualenv ($ENV_PATH) does not already exist."
+    echo "Creating virtualenv ... "
+    virtualenv $ENV_PATH
+
+    echo "Installing packages (into $ENV_PATH) required for tests ... "
+    pip-python -E $ENV_PATH install -r $REQ_FILE
+
+    echo "Adding oz to virtualenv site-packages ... "
+    pushd "$ENV_PATH/$PYTHON_SITELIB"
+    ln -sf ../../../../../oz
+    popd
+
+    echo "Virtualenv created and prepared."
+}
+
+# Process arguments
+while getopts "hv" OPTION
+do
+  case ${OPTION} in
+    v ) PYTEST_ARGS="${PYTEST_ARGS} --verbose" ;;
+    \?|h ) usage ;;
+    * ) usage ;;
+  esac
+done
+
+# check for virtualenv
+if [ ! -e $ENV_PATH ]; then
+    create_env
+else
+    echo "Virtualenv ($ENV_PATH) detected."
+fi
+
+# Activate the virtualenv
+source "$ENV_PATH/bin/activate"
+
+# Run the tests
+py.test ${PYTEST_ARGS} tests
+
+TEST_RESULT=$?
+
+# Deactivate the virtualenv
+deactivate
+
+if [ $TEST_RESULT -ne 0 ]; then
+    exit 1
+else
+    exit 0
+fi
diff --git a/tests/tdl/run.sh b/tests/tdl/run.sh
deleted file mode 100755
index f44ce5c..0000000
--- a/tests/tdl/run.sh
+++ /dev/null
@@ -1,128 +0,0 @@
-#!/bin/bash
-
-export PYTHONPATH=../..:$PYTHONPATH
-
-if [ -z "$DEBUG" ]; then
-    DEBUG=0
-fi
-
-SUCCESS=0
-FAIL=0
-
-success() {
-    echo "OK"
-    SUCCESS=$(( $SUCCESS + 1 ))
-}
-
-failure() {
-    echo "FAIL"
-    FAIL=$(( $FAIL + 1 ))
-}
-
-schema_test() {
-    schema="$1"
-    expectsuccess="$2"
-
-    echo -n "Testing Schema $schema..."
-
-    if [ ! -r "$1" ]; then
-	echo "FAIL: File not found"
-	FAIL=$(( $FAIL + 1 ))
-	return
-    fi
-    if [ $DEBUG -eq 0 ]; then
-	xmllint --noout --relaxng ../../docs/tdl.rng "$schema" >& /dev/null
-    else
-	xmllint --relaxng ../../docs/tdl.rng "$schema"
-    fi
-
-    RET=$?
-
-    if [ $RET -eq 0 -a "$expectsuccess" = "true" ] || [ $RET -ne 0 -a
"$expectsuccess" != "true" ]; then
-	success
-    else
-	failure
-    fi
-}
-
-python_test() {
-    parse="$1"
-    expectsuccess="$2"
-
-    echo -n "Testing Parsing $1..."
-    if [ ! -r "$1" ]; then
-	echo "FAIL: File not found"
-	FAIL=$(( $FAIL + 1 ))
-	return
-    fi
-
-    if [ $DEBUG -eq 0 ]; then
-	python test_tdl.py "$1" >& /dev/null
-    else
-	python test_tdl.py "$1"
-    fi
-
-    RET=$?
-
-    if [ $RET -eq 0 -a "$expectsuccess" = "true" ] || [ $RET -ne 0 -a
"$expectsuccess" != "true" ]; then
-	success
-    else
-	failure
-    fi
-}
-
-expect_success() {
-    schema_test "$1" "true"
-    python_test "$1" "true"
-}
-
-expect_fail() {
-    schema_test "$1" "false"
-    python_test "$1" "false"
-}
-
-expect_success test-01-simple-iso.tdl
-expect_success test-02-simple-url.tdl
-expect_fail test-03-empty-template.tdl
-expect_fail test-04-no-os.tdl
-expect_fail test-05-no-name.tdl
-expect_success test-06-simple-iso-description.tdl
-expect_success test-07-packages-no-package.tdl
-expect_success test-08-repositories-no-repository.tdl
-expect_fail test-09-os-invalid-arch.tdl
-expect_fail test-10-os-invalid-install-type.tdl
-expect_success test-11-description-packages-repositories.tdl
-expect_fail test-12-os-no-name.tdl
-expect_fail test-13-os-no-version.tdl
-expect_fail test-14-os-no-arch.tdl
-expect_fail test-15-os-no-install.tdl
-expect_success test-16-signed-repository.tdl
-expect_fail test-17-repo-invalid-signed.tdl
-expect_success test-18-rootpw.tdl
-expect_success test-19-key.tdl
-expect_fail test-20-multiple-install.tdl
-expect_fail test-21-missing-install-type.tdl
-expect_success test-22-md5sum.tdl
-expect_success test-23-sha1sum.tdl
-expect_success test-24-sha256sum.tdl
-expect_fail test-25-md5sum-and-sha1sum.tdl
-expect_fail test-26-md5sum-and-sha256sum.tdl
-expect_fail test-27-sha1sum-and-sha256sum.tdl
-expect_fail test-28-package-no-name.tdl
-expect_success test-29-files.tdl
-expect_fail test-30-file-no-name.tdl
-expect_success test-31-file-raw-type.tdl
-expect_success test-32-file-base64-type.tdl
-expect_fail test-33-file-invalid-type.tdl
-expect_fail test-34-file-invalid-base64.tdl
-expect_fail test-35-repository-no-name.tdl
-expect_fail test-36-repository-no-url.tdl
-expect_success test-37-command.tdl
-expect_fail test-38-command-no-name.tdl
-expect_success test-39-command-raw-type.tdl
-expect_success test-40-command-base64-type.tdl
-expect_fail test-41-command-bogus-base64.tdl
-expect_fail test-42-command-bogus-type.tdl
-
-echo "SUCCESS: $SUCCESS, FAIL: $FAIL"
-exit $FAIL
-- 
1.7.6

[PATCH 3/5] Re-write to comply with py.test format and introduce test.cfg

[James Laska]

 * test_tdl.py - redesigned to comply with py.test format
 * test_tdl.py - use libxml2 directly for xmllint validation
 * test.cfg - Expected test results no longer hard-coded, results are managed
   using ConfigParser-style test.cfg file (see [DEFAULT] for description of
   options)
---
 tests/tdl/test.cfg    |  135 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/tdl/test_tdl.py |  131 ++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 262 insertions(+), 4 deletions(-)
 create mode 100644 tests/tdl/test.cfg

diff --git a/tests/tdl/test.cfg b/tests/tdl/test.cfg
new file mode 100644
index 0000000..f929f04
--- /dev/null
+++ b/tests/tdl/test.cfg
@@ -0,0 +1,135 @@
+# Main config file for oz tdl tests
+
+[DEFAULT]
+# Unique test name, if no name is provided, the section name is used.
+name = %(__name__)s
+
+# Expected overall test result.  When set, individual test results are ignored.
+# If no value is provided, individual results are used.
+result =
+
+# Ozlib test result.  If no value is provided, test is assumed pass.
+ozlib_result =
+
+# xmllint schema test result.  If no value is provided, test is assumed pass.
+schema_result =
+
+# (optional) - Specify exact exception expected from test failure (default:
+# oz.OzException.OzException).  Note, the value used is passed to eval().  Use
+# with caution!!
+# ozlib_exception =
+
+# (optional) - Specify exact exception expected from test failure (default:
+# Exception).  Note, the value used is passed to eval().  Use with caution!!
+# schema_exception =
+
+# All section names must match a .tdl file name
+# Refer to [DEFAULT] for description of available options
+[test-01-simple-iso.tdl]
+
+[test-02-simple-url.tdl]
+
+[test-03-empty-template.tdl]
+result = fail
+
+[test-04-no-os.tdl]
+result = fail
+
+[test-05-no-name.tdl]
+result = fail
+
+[test-06-simple-iso-description.tdl]
+
+[test-07-packages-no-package.tdl]
+
+[test-08-repositories-no-repository.tdl]
+
+[test-09-os-invalid-arch.tdl]
+result = fail
+
+[test-10-os-invalid-install-type.tdl]
+result = fail
+
+[test-11-description-packages-repositories.tdl]
+
+[test-12-os-no-name.tdl]
+result = fail
+
+[test-13-os-no-version.tdl]
+result = fail
+
+[test-14-os-no-arch.tdl]
+result = fail
+
+[test-15-os-no-install.tdl]
+result = fail
+
+[test-16-signed-repository.tdl]
+
+[test-17-repo-invalid-signed.tdl]
+result = fail
+
+[test-18-rootpw.tdl]
+
+[test-19-key.tdl]
+
+[test-20-multiple-install.tdl]
+result = fail
+
+[test-21-missing-install-type.tdl]
+result = fail
+
+[test-22-md5sum.tdl]
+
+[test-23-sha1sum.tdl]
+
+[test-24-sha256sum.tdl]
+
+[test-25-md5sum-and-sha1sum.tdl]
+result = fail
+
+[test-26-md5sum-and-sha256sum.tdl]
+result = fail
+
+[test-27-sha1sum-and-sha256sum.tdl]
+result = fail
+
+[test-28-package-no-name.tdl]
+result = fail
+
+[test-29-files.tdl]
+
+[test-30-file-no-name.tdl]
+result = fail
+
+[test-31-file-raw-type.tdl]
+
+[test-32-file-base64-type.tdl]
+
+[test-33-file-invalid-type.tdl]
+result = fail
+
+[test-34-file-invalid-base64.tdl]
+result = fail
+ozlib_exception = TypeError
+
+[test-35-repository-no-name.tdl]
+result = fail
+
+[test-36-repository-no-url.tdl]
+result = fail
+
+[test-37-command.tdl]
+
+[test-38-command-no-name.tdl]
+result = fail
+
+[test-39-command-raw-type.tdl]
+[test-40-command-base64-type.tdl]
+
+[test-41-command-bogus-base64.tdl]
+result = fail
+ozlib_exception = TypeError
+
+[test-42-command-bogus-type.tdl]
+result = fail
diff --git a/tests/tdl/test_tdl.py b/tests/tdl/test_tdl.py
index 75ea3f8..a2e03c4 100644
--- a/tests/tdl/test_tdl.py
+++ b/tests/tdl/test_tdl.py
@@ -1,8 +1,131 @@
-import oz.TDL
 import sys
+import os
+import glob
+import ConfigParser
 
-if len(sys.argv) != 2:
-    print "Usage: test_tdl.py <tdl>"
+try:
+    import libxml2
+except ImportError:
+    print 'Unable to import libxml2.  Is libxml2-python installed?'
     sys.exit(1)
 
-tdl = oz.TDL.TDL(open(sys.argv[1], 'r').read())
+try:
+    import py.test
+except ImportError:
+    print 'Unable to import py.test.  Is py.test installed?'
+    sys.exit(1)
+
+try:
+    import oz
+    import oz.TDL
+except ImportError:
+    print 'Unable to import oz.  Is oz installed?'
+    sys.exit(1)
+
+# Validate oz handling of tdl file
+def validate_ozlib(tdl_file):
+    xmldata = open(tdl_file, 'r').read()
+    tdl = oz.TDL.TDL(xmldata)
+
+# Validate schema
+def validate_schema(tdl_file):
+
+    # Locate relaxng schema
+    rng_file = None
+    for tryme in ['../../docs/tdl.rng',
+                  '../docs/tdl.rng',
+                  'docs/tdl.rng',
+                  'tdl.rng',]:
+        if os.path.isfile(tryme):
+            rng_file = tryme
+            break
+
+    if rng_file is None:
+        raise Exception('RelaxNG schema file not found: tdl.rng')
+
+    # Load relaxng schema
+    schema = open(rng_file, 'r').read()
+    rngp = libxml2.relaxNGNewMemParserCtxt(schema, len(schema))
+    rngs = rngp.relaxNGParse()
+
+    # Define callback for error handling
+    def error_cb(ctx, str):
+        print "%s: %s" % (ctx, str.strip())
+    libxml2.registerErrorHandler(error_cb, tdl_file)
+
+    # Attempt to validate
+    reader = libxml2.newTextReaderFilename(tdl_file)
+    #reader.SetParserProp(libxml2.PARSER_VALIDATE, 1)
+    reader.RelaxNGSetSchema(rngs)
+    ret = reader.Read()
+    while ret == 1:
+        ret = reader.Read()
+
+    if ret != 0:
+        raise Exception('Error parsing the document: %s' % tdl_file)
+    if reader.IsValid() != 1:
+        raise Exception('Document failed to validate: %s' % tdl_file)
+
+# Test generator that iterates over all .tdl files
+def test_all_tdls():
+
+    # Define a helper to expect an exception
+    def handle_exception(exc, func, *args):
+        with py.test.raises(exc):
+            func(*args)
+
+    # Load test configuration file
+    test_path = os.path.join('tests', 'tdl')
+    cfg = ConfigParser.ConfigParser()
+    cfg.read(os.path.join(test_path, 'test.cfg'))
+
+    # Sanity check to see if any tests are unaccounted for in the config file
+    tdl_files = glob.glob('tests/tdl/*.tdl')
+    for tdl_file in tdl_files:
+        # we want the file, not the full file path
+        tdl_file = os.path.basename(tdl_file)
+        if not cfg.has_section(tdl_file):
+            py.test.fail('Missing test.cfg section for: %s' % tdl_file)
+
+    # Iterate over all expected good tdl files
+    for test_name in cfg.sections():
+
+        test_desc = cfg.get(test_name, 'name')
+        tdl_path = os.path.join(test_path, test_name)
+
+        # load expected results
+        expected_overall = cfg.get(test_name, 'result', '')
+        expected_ozlib = cfg.get(test_name, 'ozlib_result', '')
+        expected_schema = cfg.get(test_name, 'schema_result', '')
+
+        # load possible expected failure conditions
+        ozlib_exception = cfg.has_option(test_name, 'ozlib_exception') \
+            and cfg.get(test_name, 'ozlib_exception') or
oz.OzException.OzException
+        schema_exception = cfg.has_option(test_name, 'schema_exception') \
+            and cfg.get(test_name, 'schema_exception') or Exception
+
+        # Convert an exception string into an exception object
+        if isinstance(ozlib_exception, str):
+            try:
+                ozlib_exception = eval(ozlib_exception)
+            except NameError:
+                py.test.fail('Unknown exception: %s' % ozlib_exception)
+        if isinstance(schema_exception, str):
+            try:
+                schema_exception = eval(schema_exception)
+            except NameError:
+                py.test.fail('Unknown exception: %s' % schema_exception)
+
+        # Run validate_ozlib test
+        if expected_overall in ['','pass'] and expected_ozlib in
['', 'pass']:
+            yield '%s (ozlib)' % test_desc, validate_ozlib, tdl_path
+        else: # expected FAIL
+            yield '%s (ozlib)' % test_desc, handle_exception, \
+                ozlib_exception, validate_ozlib, tdl_path
+
+        # Run validate_schema test
+        if expected_overall in ['','pass'] and expected_schema in
['', 'pass']:
+            yield '%s (schema)' % test_desc, validate_schema, tdl_path
+        else: # expected FAIL
+            yield '%s (schema)' % test_desc, handle_exception, \
+                schema_exception, validate_schema, tdl_path
-- 
1.7.6

[PATCH 4/5] Re-write to comply with py.test format

[James Laska]

---
 tests/factory/test_factory.py |  311 +++++++++++++++++++++--------------------
 1 files changed, 161 insertions(+), 150 deletions(-)

diff --git a/tests/factory/test_factory.py b/tests/factory/test_factory.py
index 934277b..b05ce67 100644
--- a/tests/factory/test_factory.py
+++ b/tests/factory/test_factory.py
@@ -1,13 +1,21 @@
-import oz.TDL
-import oz.GuestFactory
 import ConfigParser
 import StringIO
 import logging
 import os
 import sys
 
-success = 0
-fail = 0
+try:
+    import py.test
+except ImportError:
+    print 'Unable to import py.test.  Is py.test installed?'
+    sys.exit(1)
+
+try:
+    import oz.TDL
+    import oz.GuestFactory
+except ImportError:
+    print 'Unable to import oz.  Is oz installed?'
+    sys.exit(1)
 
 def default_route():
     route_file = "/proc/net/route"
@@ -33,7 +41,7 @@ def default_route():
 # doesn't really matter; it just has to have an IP address
 route = default_route()
 
-def test_distro(distro, version, arch, installtype):
+def runtest(distro, version, arch, installtype):
     global route
 
     tdlxml = """
@@ -63,153 +71,156 @@ def test_distro(distro, version, arch, installtype):
 
     guest = oz.GuestFactory.guest_factory(tdl, config, None)
 
-def runtest(distro, version, arch, installtype, expect_success):
-    global success
-    global fail
-
-    print "Testing %s-%s-%s-%s..." % (distro, version, arch, installtype),
-    try:
-        test_distro(distro, version, arch, installtype)
-        if expect_success:
-            success += 1
-            print "OK"
-        else:
-            fail += 1
-            print "FAIL"
-    except Exception, e:
-        if expect_success:
-            fail += 1
-            print e
-            print "FAIL"
-        else:
-            success += 1
-            print "OK"
-
-def expect_success(distro, version, arch, installtype):
-    return runtest(distro, version, arch, installtype, True)
-
-def expect_fail(distro, version, arch, installtype):
-    return runtest(distro, version, arch, installtype, False)
-
-# bad distro
-expect_fail("foo", "1", "i386", "url")
-# bad installtype
-expect_fail("Fedora", "14", "i386", "dong")
-# bad arch
-expect_fail("Fedora", "14", "ia64", "iso")
-
-# FedoraCore
-for version in ["1", "2", "3", "4",
"5", "6"]:
-    for arch in ["i386", "x86_64"]:
-        for installtype in ["url", "iso"]:
-            expect_success("FedoraCore", version, arch, installtype)
-# bad FedoraCore version
-expect_fail("FedoraCore", "24", "x86_64", "iso")
-
-# Fedora
-for version in ["7", "8", "9", "10",
"11", "12", "13", "14", "15"]:
-    for arch in ["i386", "x86_64"]:
-        for installtype in ["url", "iso"]:
-            expect_success("Fedora", version, arch, installtype)
-# bad Fedora version
-expect_fail("Fedora", "24", "x86_64", "iso")
-
-# RHL
-for version in ["7.0", "7.1", "7.2", "7.3",
"8", "9"]:
-    expect_success("RHL", version, "i386", "url")
-# bad RHL version
-expect_fail("RHL", "10", "i386", "url")
-# bad RHL arch
-expect_fail("RHL", "9", "x86_64", "url")
-# bad RHL installtype
-expect_fail("RHL", "9", "x86_64", "iso")
-
-# RHEL-2.1
-for version in ["GOLD", "U2", "U3", "U4",
"U5", "U6"]:
-    expect_success("RHEL-2.1", version, "i386", "url")
-# bad RHEL-2.1 version
-expect_fail("RHEL-2.1", "U7", "i386", "url")
-# bad RHEL-2.1 arch
-expect_fail("RHEL-2.1", "U6", "x86_64", "url")
-# bad RHEL-2.1 installtype
-expect_fail("RHEL-2.1", "U6", "i386", "iso")
-
-# RHEL-3
-for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7", "U8",
"U9"]:
-    for arch in ["i386", "x86_64"]:
-        expect_success("RHEL-3", version, arch, "url")
-# bad RHEL-3 version
-expect_fail("RHEL-3", "U10", "x86_64", "url")
-# invalid RHEL-3 installtype
-expect_fail("RHEL-3", "U9", "x86_64", "iso")
-
-# RHEL-4/CentOS-4
-for distro in ["RHEL-4", "CentOS-4"]:
-    for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7", "U8",
"U9"]:
+def expect_success(*args):
+    runtest(*args)
+
+#def expect_fail(distro, version, arch, installtype):
+#    return runtest(distro, version, arch, installtype, False)
+
+# Define a helper to expect an exception
+def exception_expected(exc, func, *args):
+    with py.test.raises(exc):
+        runtest(*args)
+
+def expect_fail(*args):
+    with py.test.raises(Exception):
+        runtest(*args)
+
+def test_all():
+
+    # bad distro
+    yield 'bad-distro', expect_fail, "foo", "1",
"i386", "url"
+    # bad installtype
+    yield 'bad-installtype', expect_fail, "Fedora", "14",
"i386", "dong"
+    # bad arch
+    yield 'bad-arch', expect_fail, "Fedora", "14",
"ia64", "iso"
+
+    # FedoraCore
+    for version in ["1", "2", "3", "4",
"5", "6"]:
         for arch in ["i386", "x86_64"]:
             for installtype in ["url", "iso"]:
-                expect_success(distro, version, arch, installtype)
-# bad RHEL-4 version
-expect_fail("RHEL-4", "U10", "x86_64", "url")
+                yield 'FedoraCore-%s-%s-%s' % (version, arch, installtype), \
+                    expect_success, "FedoraCore", version, arch, installtype
+    # bad FedoraCore version
+    yield 'bad-FedoraCore-version', expect_fail, "FedoraCore",
"24", "x86_64", "iso"
 
-# RHEL-5/CentOS-5
-for distro in ["RHEL-5", "CentOS-5"]:
-    for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7"]:
+    # Fedora
+    for version in ["7", "8", "9", "10",
"11", "12", "13", "14", "15"]:
         for arch in ["i386", "x86_64"]:
             for installtype in ["url", "iso"]:
-                expect_success(distro, version, arch, installtype)
-# bad RHEL-5 version
-expect_fail("RHEL-5", "U10", "x86_64", "url")
-
-# RHEL-6
-for version in ["0", "1"]:
-    for arch in ["i386", "x86_64"]:
-        for installtype in ["url", "iso"]:
-            expect_success("RHEL-6", version, arch, installtype)
-# bad RHEL-6 version
-expect_fail("RHEL-6", "U10", "x86_64", "url")
-
-# Debian
-for version in ["5", "6"]:
-    for arch in ["i386", "x86_64"]:
-        expect_success("Debian", version, arch, "iso")
-# bad Debian version
-expect_fail("Debian", "12", "i386", "iso")
-# invalid Debian installtype
-expect_fail("Debian", "6", "x86_64", "url")
-
-# Windows
-expect_success("Windows", "2000", "i386", "iso")
-for version in ["XP", "2003", "2008", "7"]:
-    for arch in ["i386", "x86_64"]:
-        expect_success("Windows", version, arch, "iso")
-# bad Windows 2000 arch
-expect_fail("Windows", "2000", "x86_64", "iso")
-# bad Windows version
-expect_fail("Windows", "1999", "x86_64", "iso")
-# invalid Windows installtype
-expect_fail("Windows", "2008", "x86_64", "url")
-
-# OpenSUSE
-for version in ["11.0", "11.1", "11.2", "11.3",
"11.4"]:
-    for arch in ["i386", "x86_64"]:
-        expect_success("OpenSUSE", version, arch, "iso")
-# bad OpenSUSE version
-expect_fail("OpenSUSE", "16", "x86_64", "iso")
-# invalid OpenSUSE installtype
-expect_fail("OpenSUSE", "11.4", "x86_64", "url")
-
-# Ubuntu
-for version in ["6.06", "6.06.1", "6.06.2",
"6.10", "7.04", "7.10", "8.04",
-                "8.04.1", "8.04.2", "8.04.3",
"8.04.4", "8.10", "9.04", "9.10",
-                "10.04", "10.04.1", "10.04.2",
"10.04.3", "10.10", "11.04"]:
-    for arch in ["i386", "x86_64"]:
-        expect_success("Ubuntu", version, arch, "iso")
-# bad Ubuntu version
-expect_fail("Ubuntu", "10.9", "i386", "iso")
-# bad Ubuntu installtype
-expect_fail("Ubuntu", "10.10", "i386", "url")
-
-print "SUCCESS: %d, FAIL: %d" % (success, fail)
-
-sys.exit(fail)
+                yield 'Fedora-%s-%s-%s' % (version, arch, installtype), \
+                    expect_success, "Fedora", version, arch, installtype
+    # bad Fedora version
+    yield 'bad-Fedora-version', expect_fail, "Fedora", "24",
"x86_64", "iso"
+
+    # RHL
+    for version in ["7.0", "7.1", "7.2", "7.3",
"8", "9"]:
+        yield 'RHL-%s' % version, expect_success, "RHL", version,
"i386", "url"
+
+    # bad RHL version
+    yield 'bad-RHL-version', expect_fail, "RHL", "10",
"i386", "url"
+    # bad RHL arch
+    yield 'bad-RHL-arch', expect_fail, "RHL", "9",
"x86_64", "url"
+    # bad RHL installtype
+    yield 'bad-RHL-installtype', expect_fail, "RHL", "9",
"x86_64", "iso"
+
+    # RHEL-2.1
+    for version in ["GOLD", "U2", "U3", "U4",
"U5", "U6"]:
+        yield 'RHEL21-%s' % version, expect_success, "RHEL-2.1",
version, "i386", "url"
+
+    # bad RHEL-2.1 version
+    yield 'bad-RHEL21-version', expect_fail, "RHEL-2.1",
"U7", "i386", "url"
+    # bad RHEL-2.1 arch
+    yield 'bad-RHEL21-arch', expect_fail, "RHEL-2.1", "U6",
"x86_64", "url"
+    # bad RHEL-2.1 installtype
+    yield 'bad-RHEL21-installtype', expect_fail, "RHEL-2.1",
"U6", "i386", "iso"
+
+    # RHEL-3
+    for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7", "U8",
"U9"]:
+        for arch in ["i386", "x86_64"]:
+            yield 'RHEL-3-%s-%s' % (version, arch), \
+                expect_success, "RHEL-3", version, arch, "url"
+
+    # bad RHEL-3 version
+    yield 'bad-RHEL3-version', expect_fail, "RHEL-3", "U10",
"x86_64", "url"
+    # invalid RHEL-3 installtype
+    yield 'bad-RHEL3-installtype', expect_fail, "RHEL-3",
"U9", "x86_64", "iso"
+
+    # RHEL-4/CentOS-4
+    for distro in ["RHEL-4", "CentOS-4"]:
+        for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7", "U8",
"U9"]:
+            for arch in ["i386", "x86_64"]:
+                for installtype in ["url", "iso"]:
+                    yield '%s-%s-%s-%s' % (distro, version, arch, installtype),
\
+                        expect_success, distro, version, arch, installtype
+    # bad RHEL-4 version
+    yield 'bad-RHEL4-version', expect_fail, "RHEL-4", "U10",
"x86_64", "url"
+
+    # RHEL-5/CentOS-5
+    for distro in ["RHEL-5", "CentOS-5"]:
+        for version in ["GOLD", "U1", "U2", "U3",
"U4", "U5", "U6", "U7"]:
+            for arch in ["i386", "x86_64"]:
+                for installtype in ["url", "iso"]:
+                    yield '%s-%s-%s-%s' % (distro, version, arch, installtype),
\
+                        expect_success, distro, version, arch, installtype
+    # bad RHEL-5 version
+    yield 'bad-RHEL5-version', expect_fail, "RHEL-5", "U10",
"x86_64", "url"
+
+    # RHEL-6
+    for version in ["0", "1"]:
+        for arch in ["i386", "x86_64"]:
+            for installtype in ["url", "iso"]:
+                yield 'RHEL-6-%s-%s-%s' % (version, arch, installtype), \
+                    expect_success, "RHEL-6", version, arch, installtype
+
+    # bad RHEL-6 version
+    yield 'bad-RHEL6-version', expect_fail, "RHEL-6", "U10",
"x86_64", "url"
+
+    # Debian
+    for version in ["5", "6"]:
+        for arch in ["i386", "x86_64"]:
+            yield 'Debian-%s-%s' % (version, arch), \
+                expect_success, "Debian", version, arch, "iso"
+
+    # bad Debian version
+    yield 'bad-debian-version', expect_fail, "Debian", "12",
"i386", "iso"
+    # invalid Debian installtype
+    yield 'bad-debian-installtype', expect_fail, "Debian",
"6", "x86_64", "url"
+
+    # Windows
+    yield 'Windows-2000-i386-iso', expect_success, "Windows",
"2000", "i386", "iso"
+    for version in ["XP", "2003", "2008", "7"]:
+        for arch in ["i386", "x86_64"]:
+            yield 'Windows-%s-%s' % (version, arch), \
+                expect_success, "Windows", version, arch, "iso"
+
+    # bad Windows 2000 arch
+    yield 'bad-windows-arch', expect_fail, "Windows", "2000",
"x86_64", "iso"
+    # bad Windows version
+    yield 'bad-windows-version', expect_fail, "Windows",
"1999", "x86_64", "iso"
+    # invalid Windows installtype
+    yield 'bad-windows-installtype', expect_fail, "Windows",
"2008", "x86_64", "url"
+
+    # OpenSUSE
+    for version in ["11.0", "11.1", "11.2",
"11.3", "11.4"]:
+        for arch in ["i386", "x86_64"]:
+            yield 'OpenSUSE-%s-%s' % (version, arch), \
+                expect_success, "OpenSUSE", version, arch, "iso"
+
+    # bad OpenSUSE version
+    yield 'bad-OpenSUSE-version', expect_fail, "OpenSUSE",
"16", "x86_64", "iso"
+    # invalid OpenSUSE installtype
+    yield 'bad-OpenSUSE-installtype', expect_fail, "OpenSUSE",
"11.4", "x86_64", "url"
+
+    # Ubuntu
+    for version in ["6.06", "6.06.1", "6.06.2",
"6.10", "7.04", "7.10", "8.04",
+                    "8.04.1", "8.04.2", "8.04.3",
"8.04.4", "8.10", "9.04", "9.10",
+                    "10.04", "10.04.1", "10.04.2",
"10.04.3", "10.10", "11.04"]:
+        for arch in ["i386", "x86_64"]:
+            yield 'Ubuntu-%s-%s' % (version, arch), \
+                expect_success, "Ubuntu", version, arch, "iso"
+
+    # bad Ubuntu version
+    yield 'bad-Ubuntu-version', expect_fail, "Ubuntu",
"10.9", "i386", "iso"
+    # bad Ubuntu installtype
+    yield 'bad-Ubuntu-installtype', expect_fail, "Ubuntu",
"10.10", "i386", "url"
-- 
1.7.6

[PATCH 5/5] Allow for command-line test override

[James Laska]

---
 tests/runtests.sh |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/runtests.sh b/tests/runtests.sh
index 961d9d4..8600f72 100755
--- a/tests/runtests.sh
+++ b/tests/runtests.sh
@@ -12,6 +12,9 @@ REQ_FILE='tests/dependencies.txt'
 # virtualenv
 PIP_REQUIRE_VIRTUALENV=1
 
+# Where to look for tests (default: tests)
+RUNTESTS=${@:-tests}
+
 # this keeps track of the python version since virtualenv will use the system
 # python version by default
 PYTHON_SITELIB=$(python -c "from distutils.sysconfig import get_python_lib; print
get_python_lib(prefix='')")
@@ -23,7 +26,7 @@ PYTEST_ARGS="--junitxml=tests/test_results.xml
--ignore=${ENV_PATH}"
 usage()
 {
     cat << EOF
-    usage: $0
+    usage: $0 [tests]
     This script prepares a virtualenv for running the tests (if it doesn't
     already exist) and runs unit tests in that virtualenv.
 
@@ -73,7 +76,7 @@ fi
 source "$ENV_PATH/bin/activate"
 
 # Run the tests
-py.test ${PYTEST_ARGS} tests
+py.test ${PYTEST_ARGS} ${RUNTESTS}
 
 TEST_RESULT=$?
 
-- 
1.7.6

[Chris Lalancette]

On 08/31/11 - 10:49:36AM, James Laska wrote:
>  
>  Greetings,
>  
>  I've been looking into integrating existing oz unittests with jenkins.  It
>  wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
>  better idea for pass/fail data over time.  One way to do this was to convert
>  the tests to use py.test (or unittest -- but that's a little crusty).  Py.test
>  can emit junit XML test output, which is jenkins friendly, and has much less
>  stock test class/method bloat.  I've modified the existing oz unittests to be
>  py.test friendly.
>  
>  Another annoyance with running tests in jenkins is that the nodes need to be
>  manually setup with appropriate dependencies ahead of time.  While this isn't
>  hard, it's maintenance and one more thing to forget/fail when testing.  The new
>  test driver (runtests.sh) will setup a python virtualenv and install required
>  dependencies there.  This is intended to handle deps installation during
>  unittest execution.
>  
>  >  .gitignore                    |    6 +
>  >  Makefile                      |    6 +-
>  >  tests/dependencies.txt        |    2 +
>  >  tests/factory/run.sh          |    5 -
>  >  tests/factory/test_factory.py |  311 +++++++++++++++++++++--------------------
>  >  tests/runtests.sh             |   90 ++++++++++++
>  >  tests/tdl/run.sh              |  128 -----------------
>  >  tests/tdl/test.cfg            |  135 ++++++++++++++++++
>  >  tests/tdl/test_tdl.py         |  131 +++++++++++++++++-
>  >  9 files changed, 524 insertions(+), 290 deletions(-) 
Arg.  I know I said I was willing to entertain these patches, and I was, but
I have to say that I am not a fan.  It turns what is simple bash and python
unittests into a morass of config files, scripts, and magic.  Plus it adds 3
new dependencies to run the unittests, which is an additional barrier to entry
for new contributors.

So there are 3 main problems that you see with the current unittests (the 2
listed above plus one you mentioned on IRC):
1)  It is a sort of odd combination of bash and python, and you need to modify
    the code every time you want to add a new test
2)  You want JUnit XML output to feed to jenkins
3)  You want to have some sort of automatic dependency resolution for jenkins.

I think we can easily handle 1) by re-writing the small portions of this that
are in bash in python, and make it smarter.  For instance, take what
you've done for moving the tests in their own directory, and just have the
python code iterate over those tests.

For 2), after seeing the output of these patches, it seems to me that it should
be easy enough to have the existing python code generate JUnit XML as output.
Maybe we have the code take a flag (--junitxml=output/path), and then we can
build it up ourselves (it is, after all, simple XML).

The third problem you have is a little more difficult.  I want to keep running
the tests as simple as possible, but you want automatic dependency stuff.
I wonder if we can add a new "virtualenv" target to the Makefile, which only
jenkins will call.  Then it could call out to the tests as re-written above.

What do you think about that proposal?

-- 
Chris Lalancette

[James Laska]

On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
>  On 08/31/11 - 10:49:36AM, James Laska wrote:
>  > 
>  > Greetings,
>  > 
>  > I've been looking into integrating existing oz unittests with jenkins.  It
>  > wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
>  > better idea for pass/fail data over time.  One way to do this was to convert
>  > the tests to use py.test (or unittest -- but that's a little crusty).  Py.test
>  > can emit junit XML test output, which is jenkins friendly, and has much less
>  > stock test class/method bloat.  I've modified the existing oz unittests to be
>  > py.test friendly.
>  > 
>  > Another annoyance with running tests in jenkins is that the nodes need to be
>  > manually setup with appropriate dependencies ahead of time.  While this isn't
>  > hard, it's maintenance and one more thing to forget/fail when testing.  The new
>  > test driver (runtests.sh) will setup a python virtualenv and install required
>  > dependencies there.  This is intended to handle deps installation during
>  > unittest execution.
>  > 
>  > >  .gitignore                    |    6 +
>  > >  Makefile                      |    6 +-
>  > >  tests/dependencies.txt        |    2 +
>  > >  tests/factory/run.sh          |    5 -
>  > >  tests/factory/test_factory.py |  311
> +++++++++++++++++++++--------------------
>  > >  tests/runtests.sh             |   90 ++++++++++++
>  > >  tests/tdl/run.sh              |  128 -----------------
>  > >  tests/tdl/test.cfg            |  135 ++++++++++++++++++
>  > >  tests/tdl/test_tdl.py         |  131 +++++++++++++++++-
>  > >  9 files changed, 524 insertions(+), 290 deletions(-)
>  
>  Arg.  I know I said I was willing to entertain these patches, and I was, but
>  I have to say that I am not a fan.  It turns what is simple bash and python
>  unittests into a morass of config files, scripts, and magic.  Plus it adds 3
>  new dependencies to run the unittests, which is an additional barrier to entry
>  for new contributors. 
What are the new dependencies, python-virtualenv and py.test?

I'm not a big fan of writing tests in bash, it's easy to write, and
easier to write wrong and hard to debug.  Plus there are no libs to rely
on, you have to build the custom scaffolding.  But that's just a
personal preference.

I could have gone with unittest, which is built-in to python.  But I've
done it plenty of times and it requires too much scaffolding to do basic
tasks.  I liked py.test since you just write python scripts as your
tests ... but it does add a new dependency.

>  So there are 3 main problems that you see with the current unittests
> (the 2
>  listed above plus one you mentioned on IRC):
>  1)  It is a sort of odd combination of bash and python, and you need to modify
>      the code every time you want to add a new test 
The current proposed patchset requires modifying a config file when
adding new tests.  I didn't consider that code, more config, but it does
require a commit of sorts whenever adding/modifying tests.  A previous
version of the patchset required no modification at all, but offered
less flexibility.  Tests were split into two directories, expected_pass/
and expected_fail/  I can return to this if desired.  I liked the
current .cfg approach since it allowed for customizing the exact failure
case, and I tried to self-document the use of options in the config
file.  If that's overkill, I can definitely return to the simpler
directory-based approach.

>  2)  You want JUnit XML output to feed to jenkins 
Yeah, I think that'd be helpful all around (unless the tests always
pass) :)

>  3)  You want to have some sort of automatic dependency resolution for
> jenkins. 
That'd be nice ... it's also for folks running 'make unittests' from a
git checkout who don't have all the runtime/test dependencies installed.
Rather, the later is how I've used it in the past.

>  I think we can easily handle 1) by re-writing the small portions of
> this that
>  are in bash in python, and make it smarter.  For instance, take what
>  you've done for moving the tests in their own directory, and just have the
>  python code iterate over those tests. 
In the current approach, I left the .tdl test files where they were, but
used a .cfg to define expected outcomes.  Would it be easier to remove
the .cfg and just have tests exist in expected_fail/ expected_pass/ ?

>  For 2), after seeing the output of these patches, it seems to me that
> it should
>  be easy enough to have the existing python code generate JUnit XML as output.
>  Maybe we have the code take a flag (--junitxml=output/path), and then we can
>  build it up ourselves (it is, after all, simple XML). 
My preference would be to avoid custom-built XML generation (that just
seems prone to rot) and let the python framework (py.test) do that for
us.  I figured that would be less to maintain in the long-run.  Let me
know what you think.

>  The third problem you have is a little more difficult.  I want to
> keep running
>  the tests as simple as possible, but you want automatic dependency stuff.
>  I wonder if we can add a new "virtualenv" target to the Makefile, which only
>  jenkins will call.  Then it could call out to the tests as re-written above. 
Oh, I see ... so you'd be required to install the deps manually when
running 'make unittests'.  However, if testing with jenkins, you could
do 'make virtualenv unittests' which would alleviate manual dependency
installation?  That's a neat idea, I could set that up.

>  What do you think about that proposal? 
Thanks for your feedback!  Whatever resolves the 3 problems you
highlighted without introducing too much additional maintenance.  I can
get busy re-working based on your feedback.

Thanks,
James

[Mark McLoughlin]

On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
>  On 08/31/11 - 10:49:36AM, James Laska wrote:
>  > 
>  > Greetings,
>  > 
>  > I've been looking into integrating existing oz unittests with jenkins.  It
>  > wasn't hard to have them run in jenkins as is, but I was looking give jenkins a
>  > better idea for pass/fail data over time.  One way to do this was to convert
>  > the tests to use py.test (or unittest -- but that's a little crusty).  Py.test
>  > can emit junit XML test output, which is jenkins friendly, and has much less
>  > stock test class/method bloat.  I've modified the existing oz unittests to be
>  > py.test friendly.
>  > 
>  > Another annoyance with running tests in jenkins is that the nodes need to be
>  > manually setup with appropriate dependencies ahead of time.  While this isn't
>  > hard, it's maintenance and one more thing to forget/fail when testing.  The new
>  > test driver (runtests.sh) will setup a python virtualenv and install required
>  > dependencies there.  This is intended to handle deps installation during
>  > unittest execution.
>  > 
>  > >  .gitignore                    |    6 +
>  > >  Makefile                      |    6 +-
>  > >  tests/dependencies.txt        |    2 +
>  > >  tests/factory/run.sh          |    5 -
>  > >  tests/factory/test_factory.py |  311
> +++++++++++++++++++++--------------------
>  > >  tests/runtests.sh             |   90 ++++++++++++
>  > >  tests/tdl/run.sh              |  128 -----------------
>  > >  tests/tdl/test.cfg            |  135 ++++++++++++++++++
>  > >  tests/tdl/test_tdl.py         |  131 +++++++++++++++++-
>  > >  9 files changed, 524 insertions(+), 290 deletions(-)
>  
>  Arg.  I know I said I was willing to entertain these patches, and I was, but
>  I have to say that I am not a fan.  It turns what is simple bash and python
>  unittests into a morass of config files, scripts, and magic.  Plus it adds 3
>  new dependencies to run the unittests, which is an additional barrier to entry
>  for new contributors. 
FWIW, I like what OpenStack's glance has done with their unit tests
using virtualenv and nose. Maybe look at that?

https://github.com/openstack/glance

Cheers,
Mark.

[James Laska]

On Fri, 2011-09-02 at 07:12 +0100, Mark McLoughlin wrote:
>  On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
>  > On 08/31/11 - 10:49:36AM, James Laska wrote:
>  > > 
>  > > Greetings,
>  > > 
>  > > I've been looking into integrating existing oz unittests with jenkins.  It
>  > > wasn't hard to have them run in jenkins as is, but I was looking give
> jenkins a
>  > > better idea for pass/fail data over time.  One way to do this was to convert
>  > > the tests to use py.test (or unittest -- but that's a little crusty). 
> Py.test
>  > > can emit junit XML test output, which is jenkins friendly, and has much less
>  > > stock test class/method bloat.  I've modified the existing oz unittests to
> be
>  > > py.test friendly.
>  > > 
>  > > Another annoyance with running tests in jenkins is that the nodes need to be
>  > > manually setup with appropriate dependencies ahead of time.  While this
> isn't
>  > > hard, it's maintenance and one more thing to forget/fail when testing.  The
> new
>  > > test driver (runtests.sh) will setup a python virtualenv and install required
>  > > dependencies there.  This is intended to handle deps installation during
>  > > unittest execution.
>  > > 
>  > > >  .gitignore                    |    6 +
>  > > >  Makefile                      |    6 +-
>  > > >  tests/dependencies.txt        |    2 +
>  > > >  tests/factory/run.sh          |    5 -
>  > > >  tests/factory/test_factory.py |  311
> +++++++++++++++++++++--------------------
>  > > >  tests/runtests.sh             |   90 ++++++++++++
>  > > >  tests/tdl/run.sh              |  128 -----------------
>  > > >  tests/tdl/test.cfg            |  135 ++++++++++++++++++
>  > > >  tests/tdl/test_tdl.py         |  131 +++++++++++++++++-
>  > > >  9 files changed, 524 insertions(+), 290 deletions(-)
>  > 
>  > Arg.  I know I said I was willing to entertain these patches, and I was, but
>  > I have to say that I am not a fan.  It turns what is simple bash and python
>  > unittests into a morass of config files, scripts, and magic.  Plus it adds 3
>  > new dependencies to run the unittests, which is an additional barrier to entry
>  > for new contributors.
>  
>  FWIW, I like what OpenStack's glance has done with their unit tests
>  using virtualenv and nose. Maybe look at that?
>  
>  https://github.com/openstack/glance 
Thanks for the link Mark.  I like their run_tests.sh driver in that it
offers a few additional arguments for managing the virtualenv (--clean,
--force, --no-virtual-env etc...).  This might be helpful for Chris'
suggestion of optionally using virtualenv.

They still use a unittest-style tests.  I don't mind that structure
since it is well documented, it's just extra scaffolding.  All tests
must be importable modules, subclass unittest.TestCase, use unusual (but
documented) assert*() methods for doing normal python things (like
comparisons and asserts) and rely on alphanum sorting to determine test
execution order (which may change as tests mature).

A comparison between nose and py.test ...
https://www.ibm.com/developerworks/aix/library/au-python_test/

Another comparison ...
http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytes...

Using nose would still require an additional dependency (python-nose).
I guess it really comes down to preference on what folks are comfortable
with maintaining/using.  I'm happy to adjust to use unittest or nose if
desired.  My current use of py.test was in part because of built-in
support for --junit output (without requiring packaging another
testrunner like HTMLTestRunner.py [1] or unittest-xml-reporting [2] or
pyjunitxml [3]), getting something going faster using the existing code
Chris developed and of course ... to explore/learn something new.

/me standing by

Thanks,
James

[1] http://tungwaiyip.info/software/HTMLTestRunner.html
[2] https://github.com/danielfm/unittest-xml-reporting
[3] https://launchpad.net/pyjunitxml

[Chris Lalancette]

On 09/01/11 - 09:48:42AM, James Laska wrote:
>  > Arg.  I know I said I was willing to entertain these patches,
> and I was, but
>  > I have to say that I am not a fan.  It turns what is simple bash and python
>  > unittests into a morass of config files, scripts, and magic.  Plus it adds 3
>  > new dependencies to run the unittests, which is an additional barrier to entry
>  > for new contributors.
>  
>  What are the new dependencies, python-virtualenv and py.test? 
And python-pip.

>  
>  I'm not a big fan of writing tests in bash, it's easy to write, and
>  easier to write wrong and hard to debug.  Plus there are no libs to rely
>  on, you have to build the custom scaffolding.  But that's just a
>  personal preference. 
Yeah, I'm with you here.  bash was just easy for me at the time, but I totally
understand converting to python.  Heck, the original version of Oz (before it
was a full project) was all written in bash, and it was a mess :).

>  
>  I could have gone with unittest, which is built-in to python.  But I've
>  done it plenty of times and it requires too much scaffolding to do basic
>  tasks.  I liked py.test since you just write python scripts as your
>  tests ... but it does add a new dependency.
>  
>  > So there are 3 main problems that you see with the current unittests (the 2
>  > listed above plus one you mentioned on IRC):
>  > 1)  It is a sort of odd combination of bash and python, and you need to modify
>  >     the code every time you want to add a new test
>  
>  The current proposed patchset requires modifying a config file when
>  adding new tests.  I didn't consider that code, more config, but it does
>  require a commit of sorts whenever adding/modifying tests.  A previous
>  version of the patchset required no modification at all, but offered
>  less flexibility.  Tests were split into two directories, expected_pass/
>  and expected_fail/  I can return to this if desired.  I liked the
>  current .cfg approach since it allowed for customizing the exact failure
>  case, and I tried to self-document the use of options in the config
>  file.  If that's overkill, I can definitely return to the simpler
>  directory-based approach. 
OK, so you don't care too much about modifying the code.  I also don't care;
you are going to have to commit a patch to add the new test anyway, so adding
one more line to the test controller (run_test or whatever) doesn't seem like
a huge deal.  So the big point on 1) is converting to python.  To that end,
I've come up with the attached patch; let me know what you think.

>  
>  > 2)  You want JUnit XML output to feed to jenkins
>  
>  Yeah, I think that'd be helpful all around (unless the tests always
>  pass) :)
>  
>  > 3)  You want to have some sort of automatic dependency resolution for jenkins.
>  
>  That'd be nice ... it's also for folks running 'make unittests' from a
>  git checkout who don't have all the runtime/test dependencies installed.
>  Rather, the later is how I've used it in the past.
>  
>  > I think we can easily handle 1) by re-writing the small portions of this that
>  > are in bash in python, and make it smarter.  For instance, take what
>  > you've done for moving the tests in their own directory, and just have the
>  > python code iterate over those tests.
>  
>  In the current approach, I left the .tdl test files where they were, but
>  used a .cfg to define expected outcomes.  Would it be easier to remove
>  the .cfg and just have tests exist in expected_fail/ expected_pass/ ?
>  
>  > For 2), after seeing the output of these patches, it seems to me that it should
>  > be easy enough to have the existing python code generate JUnit XML as output.
>  > Maybe we have the code take a flag (--junitxml=output/path), and then we can
>  > build it up ourselves (it is, after all, simple XML).
>  
>  My preference would be to avoid custom-built XML generation (that just
>  seems prone to rot) and let the python framework (py.test) do that for
>  us.  I figured that would be less to maintain in the long-run.  Let me
>  know what you think.
>  
>  > The third problem you have is a little more difficult.  I want to keep running
>  > the tests as simple as possible, but you want automatic dependency stuff.
>  > I wonder if we can add a new "virtualenv" target to the Makefile, which
> only
>  > jenkins will call.  Then it could call out to the tests as re-written above.
>  
>  Oh, I see ... so you'd be required to install the deps manually when
>  running 'make unittests'.  However, if testing with jenkins, you could
>  do 'make virtualenv unittests' which would alleviate manual dependency
>  installation?  That's a neat idea, I could set that up. 
Yeah, if you could look at doing that, that would at least address this part
of the problem.

>  
>  > What do you think about that proposal?
>  
>  Thanks for your feedback!  Whatever resolves the 3 problems you
>  highlighted without introducing too much additional maintenance.  I can
>  get busy re-working based on your feedback. 
So with my proposed patch we've addressed 1) by converting everything to
python.  If you can work on 3) to add a virtualenv Makefile target, that would
be great.  That leaves us with 2); I'm totally with you on avoiding custom code
where we can.  I think it will require a bit more research, but if we can find
something that can generate the junit XML (and only the junit XML), that might
be workable.

-- 
Chris Lalancette

[James Laska]

On Fri, 2011-09-02 at 10:04 -0400, Chris Lalancette wrote:
>  On 09/01/11 - 09:48:42AM, James Laska wrote:
>  > > Arg.  I know I said I was willing to entertain these patches, and I was, but
>  > > I have to say that I am not a fan.  It turns what is simple bash and python
>  > > unittests into a morass of config files, scripts, and magic.  Plus it adds 3
>  > > new dependencies to run the unittests, which is an additional barrier to entry
>  > > for new contributors.
>  > 
>  > What are the new dependencies, python-virtualenv and py.test?
>  
>  And python-pip. 
Oh I forgot, the only hard requirement is python-virtualenv.  Since
anything else is handled magically by virtualenv.  Using the nice idea
of an optional virtualenv, those other requirements would be needed.

>  > 
>  > I'm not a big fan of writing tests in bash, it's easy to write, and
>  > easier to write wrong and hard to debug.  Plus there are no libs to rely
>  > on, you have to build the custom scaffolding.  But that's just a
>  > personal preference.
>  
>  Yeah, I'm with you here.  bash was just easy for me at the time, but I totally
>  understand converting to python.  Heck, the original version of Oz (before it
>  was a full project) was all written in bash, and it was a mess :). 
Haha, I thought after I wrote this that it might come across that your
bash was bad.  Apologies, that was not at all the case.  It was actually
pretty easy to read and you could tell it was done by someone with
development experience.

>  > I could have gone with unittest, which is built-in to python. 
> But I've
>  > done it plenty of times and it requires too much scaffolding to do basic
>  > tasks.  I liked py.test since you just write python scripts as your
>  > tests ... but it does add a new dependency.
>  > 
>  > > So there are 3 main problems that you see with the current unittests (the 2
>  > > listed above plus one you mentioned on IRC):
>  > > 1)  It is a sort of odd combination of bash and python, and you need to modify
>  > >     the code every time you want to add a new test
>  > 
>  > The current proposed patchset requires modifying a config file when
>  > adding new tests.  I didn't consider that code, more config, but it does
>  > require a commit of sorts whenever adding/modifying tests.  A previous
>  > version of the patchset required no modification at all, but offered
>  > less flexibility.  Tests were split into two directories, expected_pass/
>  > and expected_fail/  I can return to this if desired.  I liked the
>  > current .cfg approach since it allowed for customizing the exact failure
>  > case, and I tried to self-document the use of options in the config
>  > file.  If that's overkill, I can definitely return to the simpler
>  > directory-based approach.
>  
>  OK, so you don't care too much about modifying the code.  I also don't care;
>  you are going to have to commit a patch to add the new test anyway, so adding
>  one more line to the test controller (run_test or whatever) doesn't seem like
>  a huge deal.   
Yeah, true.

>  So the big point on 1) is converting to python.  To that end,
>  I've come up with the attached patch; let me know what you think. 
Nice, thank you!  I'll play with this shortly and get some feedback.

>  So with my proposed patch we've addressed 1) by converting
> everything to
>  python.  If you can work on 3) to add a virtualenv Makefile target, that would
>  be great.  That leaves us with 2); I'm totally with you on avoiding custom code
>  where we can.  I think it will require a bit more research, but if we can find
>  something that can generate the junit XML (and only the junit XML), that might
>  be workable. 
I found these two projects that might work ...
https://github.com/danielfm/unittest-xml-reporting
https://launchpad.net/pyjunitxml

I tested unittest-xml-reporting and that seems to work.  I haven't tried
pyjunitxml.  Either would need to be packaged and included in Fedora.
I'll be glad to work that.

Thanks,
James