parallel is a nice utility that can save us a lot of work when running tests in parallel. By using parallel we can drop things like creating temporary files with the outputs and failure-flaging.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com --- scripts/githooks/pre-push | 7 +++---- tests/glade/run_glade_tests.sh | 2 +- tests/pylint/pylint-one.sh | 8 ++++---- tests/pylint/runpylint.sh | 22 ++++------------------ 4 files changed, 12 insertions(+), 27 deletions(-)
diff --git a/scripts/githooks/pre-push b/scripts/githooks/pre-push index db8c8ae..ea2e154 100755 --- a/scripts/githooks/pre-push +++ b/scripts/githooks/pre-push @@ -72,11 +72,10 @@ fi
## main ## # fork separate process for every commit, querying bugzilla takes time -num_commits=$(echo ${commits}|wc -w) -echo ${commits}|xargs -n1 --max-procs=${num_commits} .git/hooks/check_commit_msg.sh ${release} -xargs_ret=$? +echo -n ${commits}|parallel -d' ' --gnu -j0 .git/hooks/check_commit_msg.sh ${release} {} +ret=$? if [ ${ACK_FATAL} == "0" ]; then - test ${xargs_ret} -eq 0 + test ${ret} -eq 0 exit $? else exit 0 diff --git a/tests/glade/run_glade_tests.sh b/tests/glade/run_glade_tests.sh index e197180..6c69ad6 100755 --- a/tests/glade/run_glade_tests.sh +++ b/tests/glade/run_glade_tests.sh @@ -22,7 +22,7 @@ fi
status=0 for check in ${srcdir}/*/check_*.py ; do - findtestfiles -name '*.glade' | xargs "${check}" "$@" + findtestfiles -name '*.glade' | parallel --gnu -j0 "${check}" "$@" {} if [ "$?" -ne 0 ]; then status=1 fi diff --git a/tests/pylint/pylint-one.sh b/tests/pylint/pylint-one.sh index 6aed441..8cf4248 100755 --- a/tests/pylint/pylint-one.sh +++ b/tests/pylint/pylint-one.sh @@ -12,8 +12,6 @@ if grep -q '# pylint: skip-file' $1; then exit 0 fi
-file_suffix="$(eval echo $$#|sed s?/?_?g)" - pylint_output="$(pylint \ --msg-template='{msg_id}:{line:3d},{column}: {obj}: {msg}' \ -r n --disable=C,R --rcfile=/dev/null \ @@ -33,6 +31,8 @@ gi.overrides.__path__[0:0] = (os.environ["ANACONDA_WIDGETS_OVERRIDES"].split(":" if [ -n "$(echo "$pylint_output" | fgrep -v '************* Module ')" ]; then # Replace the Module line with the actual filename pylint_output="$(echo "$pylint_output" | sed "s|* Module .*|* Module $(eval echo $$#)|")" - echo "$pylint_output" > pylint-out_$file_suffix - touch "pylint-$file_suffix-failed" + echo "$pylint_output" + exit 1 +else + exit 0 fi diff --git a/tests/pylint/runpylint.sh b/tests/pylint/runpylint.sh index 3615052..6599254 100755 --- a/tests/pylint/runpylint.sh +++ b/tests/pylint/runpylint.sh @@ -114,27 +114,13 @@ if [ -z "$FILES" ]; then egrep -v '(|/)old_tests/') fi
-num_cpus=$(getconf _NPROCESSORS_ONLN) # run pylint in paralel -echo $FILES | xargs --max-procs=$num_cpus -n 1 "$srcdir"/pylint-one.sh $ARGS || exit 1 +output=$(echo -n $FILES | parallel -d' ' --gnu "$srcdir"/pylint-one.sh $ARGS {}) +exit_status=$?
-for file in $(find -name 'pylint-out*'); do - cat "$file" >> pylint-log - rm "$file" -done - -fails=$(find -name 'pylint*failed' -print -exec rm '{}' ;) -if [ -z "$fails" ]; then - exit_status=0 -else - exit_status=1 -fi - -if [ -s pylint-log ]; then +if [ "$output" != '\n' -a "$output" != "" ]; then echo "pylint reports the following issues:" - cat pylint-log -elif [ -e pylint-log ]; then - rm pylint-log + echo "$output" | tee pylint-log fi
exit "$exit_status"
On 07/23/2014 10:35 AM, Vratislav Podzimek wrote:
parallel is a nice utility that can save us a lot of work when running tests in parallel. By using parallel we can drop things like creating temporary files with the outputs and failure-flaging.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
parallel seems to have some quirks of its own, but in all I like this approach a lot better than xargs. Ack.
On Wed, Jul 23, 2014 at 11:35:55AM -0400, David Shea wrote:
On 07/23/2014 10:35 AM, Vratislav Podzimek wrote:
parallel is a nice utility that can save us a lot of work when running tests in parallel. By using parallel we can drop things like creating temporary files with the outputs and failure-flaging.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
parallel seems to have some quirks of its own, but in all I like this approach a lot better than xargs. Ack.
The script should have a test for the presence of the package and fail gracefully. xargs is generally (at least on the systems I have) installed, but parallel is not.
On Wed, 2014-07-23 at 08:57 -0700, Brian C. Lane wrote:
On Wed, Jul 23, 2014 at 11:35:55AM -0400, David Shea wrote:
On 07/23/2014 10:35 AM, Vratislav Podzimek wrote:
parallel is a nice utility that can save us a lot of work when running tests in parallel. By using parallel we can drop things like creating temporary files with the outputs and failure-flaging.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
parallel seems to have some quirks of its own, but in all I like this approach a lot better than xargs. Ack.
The script should have a test for the presence of the package and fail gracefully. xargs is generally (at least on the systems I have) installed, but parallel is not.
Yeah, that's a good point. I wanted to add parallel into BuildRequries, but then I realized we don't even have pylint there.
I'll add that check locally, but maybe we should have some file documenting which packages are required for running tests. Or what about running tests in %check and having those files as BuildRequires?
On Thu, 2014-07-24 at 08:11 +0200, Vratislav Podzimek wrote:
On Wed, 2014-07-23 at 08:57 -0700, Brian C. Lane wrote:
On Wed, Jul 23, 2014 at 11:35:55AM -0400, David Shea wrote:
On 07/23/2014 10:35 AM, Vratislav Podzimek wrote:
parallel is a nice utility that can save us a lot of work when running tests in parallel. By using parallel we can drop things like creating temporary files with the outputs and failure-flaging.
Signed-off-by: Vratislav Podzimek vpodzime@redhat.com
parallel seems to have some quirks of its own, but in all I like this approach a lot better than xargs. Ack.
The script should have a test for the presence of the package and fail gracefully. xargs is generally (at least on the systems I have) installed, but parallel is not.
Yeah, that's a good point. I wanted to add parallel into BuildRequries, but then I realized we don't even have pylint there.
I'll add that check locally, but maybe we should have some file documenting which packages are required for running tests. Or what about running tests in %check and having those files as BuildRequires?
I think there was an idea a while ago to have a makefile target or a script for installing the dependencies needed to run tests & Pylint.
anaconda-patches@lists.fedorahosted.org
-
Brian C. Lane -
David Shea -
Martin Kolman -
Vratislav Podzimek