[kernel-tests] 01/03: Add support for third party module tests

[git repository hosting]

This is an automated email from the git hooks/post-receive script.

jforbes pushed a commit to branch master
in repository kernel-tests.

commit 8f2ea1b2b3eb2660bd85bd7d3b50ba045663a3af
Author: Justin M. Forbes 
Date:   Tue Feb 28 16:49:38 2017 -0600

Add support for third party module tests
---
 config.example    |  3 +++
 documentation.txt | 20 +++++++++++++++++---
 runtests.sh       | 22 +++++++++++++++++++++-
 3 files changed, 41 insertions(+), 4 deletions(-)

diff --git a/config.example b/config.example
index dd76142..6ae342e 100644
--- a/config.example
+++ b/config.example
@@ -14,6 +14,9 @@ submit=none
 # checksig=y
 # validsig="Fedora Secure Boot Signer"

+# Test 3rd Party Modules
+# thirdparty=y
+
 # FAS User credentials.
 # Storing your FAS password here is technically possible, but not advisable
 # for security seasons
diff --git a/documentation.txt b/documentation.txt
index 367d639..5f8c985 100644
--- a/documentation.txt
+++ b/documentation.txt
@@ -30,17 +30,24 @@ It is expected that a basic set of packages are installed to run these tests.
 This includes glibc-devel and gcc.  If those packages are not installed, please
 install them before running the test suite.

+If you wish to test to ensure that 3rd party modules build against the current
+kernel, you can add a 'thirdparty=y' line to your .config.  This will run any
+tests in the thirdparty directory as well. Because these are not upstream
+drivers, a failure of these tests will return 4, the test suite will pass but
+with warning.
+
 === Writing Tests ===

While a test can actually be any sort of executable, it is expected that
 these tests will follow certain basic criteria.  This helps to ensure that
 the test suite is easy to interpret.  The output is controlled by the
-master script, and output is in the form of pass, fail, or skipped. All
-other output is redirected to the log file.
+master script, and output is in the form of pass, fail, warn, or skipped.
+All other output is redirected to the log file.

Return Codes:
 0 - A successful test completion
 3 - Test skipped 
+4 - Warn (this is reserved for things like out of tree modules)
 Anything else is interpreted as fail and the user is asked to check the log
 for more details.

@@ -84,7 +91,6 @@ any necessary source, checking for any specific dependencies, and skipping
 if they are not met. At the completion of the test set, a "Test suite complete"
 is printed with a pass/fail result, and the appropriate return code.

-
 Potential for harm:
 It is expected that these test will be run on real systems.  Any tests
 which have increased risk of data loss or ill effects should be specified
@@ -98,3 +104,11 @@ As a large number of tests are written as simple shell scripts, and many of
 these tests need to perform a series of the same functions, a "library" has
 been created to allow for reuse. source the testutil file as needed.  Any
 functions added to testutil should be clearly commented with purpose and use.
+
+Thirdparty:
+This directory should contain tests for out of tree drivers etc. These tests
+should never return anything other than pass, skip, or warn.  While it is
+handy to know if these things work with the current kernel, as out of tree
+modules, they are not necessarily in-step with upstream development.  To
+return a fail on these tests would be incorrect, but a warn does give a heads
+up so that the upstream for those modules can be contacted.
diff --git a/runtests.sh b/runtests.sh
index f3376df..8820c2b 100755
--- a/runtests.sh
+++ b/runtests.sh
@@ -9,9 +9,11 @@ verbose=n
 testset=default
 cleanrun=PASS
 failedtests=None
+warntests=None
 commit=n
 commithook=/usr/bin/true
 disable_retest=n
+thirdparty=n

if [ -f ./.config ]; then
    source ./.config
@@ -104,6 +106,11 @@ if  [ "$checksig" == "y" ]; then
     dirlist="secureboot $dirlist"
 fi

+# Test Third Party Modules?
+if  [ "$thirdparty" == "y" ]; then
+    dirlist="$dirlist thirdparty"
+fi
+
 #Basic logfile headers
 echo "Date: $(date)" > $logfile
 echo "Test set: $testset" >> $logfile
@@ -142,6 +149,9 @@ do
    	3)
    		result=SKIP
    		;;
+		4)
+			result=WARN
+			;;
    	*)
    		result=FAIL
    	esac
@@ -154,12 +164,22 @@ do
    			failedtests="$failedtests $testname"
    		fi
    	fi
+		if [ "$result" == "WARN" ]; then
+			if [ "$cleanrun" == "PASS" ]; then
+				cleanrun=WARN
+			fi
+			if [ "$warntests" == "None" ]; then
+				warntests="$testname"
+			else
+				warntests="$warntests $testname"
+			fi
+		fi
    	popd &>/dev/null
    done
 done

# Fix up logfile headers
-sed -i "s,RESULTHOLDER,$cleanrun\nFailed Tests: $failedtests,g" $logfile
+sed -i "s,RESULTHOLDER,$cleanrun\nFailed Tests: $failedtests\nWarned Tests: $warntests,g" $logfile
 printf "\n%-65s%-8s\n" "Test suite complete" "$cleanrun"

if [ "$commit" == "y" ]; then