Hi all,
New set of patches, with suggested fixes, plus few extra things I noticed during work.
Since there is no test suite, I'm leaving RFC in place.
Hubert Kario (6): new routines for socket based synchronisation add synchronisation to system library use internal killtree, to not leave running sleep commands behind add rlWaitForCmd change rlWaitForSocket to use shared code add rlWaitForFile
src/Makefile | 1 + src/beakerlib.sh | 1 + src/synchronisation.sh | 448 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 450 insertions(+) create mode 100644 src/synchronisation.sh
Add new function, rlWaitForSocket, that can wait for a network service to start. To be used instead of `sleep' when testing network-centric services.
Signed-off-by: Hubert Kario hkario@redhat.com --- src/synchronisation.sh | 194 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 src/synchronisation.sh
diff --git a/src/synchronisation.sh b/src/synchronisation.sh new file mode 100644 index 0000000..685a758 --- /dev/null +++ b/src/synchronisation.sh @@ -0,0 +1,194 @@ +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# +# Name: synchronisation.sh - part of the BeakerLib project +# Description: Process synchronisation routines +# +# Author: Hubert Kario hkario@redhat.com +# +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# +# Copyright (c) 2013 Red Hat, Inc. All rights reserved. +# +# This copyrighted material is made available to anyone wishing +# to use, modify, copy, or redistribute it subject to the terms +# and conditions of the GNU General Public License version 2. +# +# This program is distributed in the hope that it will be +# useful, but WITHOUT ANY WARRANTY; without even the implied +# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR +# PURPOSE. See the GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public +# License along with this program; if not, write to the Free +# Software Foundation, Inc., 51 Franklin Street, Fifth Floor, +# Boston, MA 02110-1301, USA. +# +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +getopt -T || ret=$? +if [ ${ret:-0} -ne 4 ]; then + echo "ERROR: Non enhanced getopt version detected" 1>&2 + exit 1 +fi + +: <<'=cut' +=pod + +=head1 NAME + +BeakerLib - synchronisation - Process synchronisation routines + +=head1 DESCRIPTION + +This is a library of helpers for process synchronisation +of applications. + +=head1 FUNCTIONS + +=cut + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# rlWaitForSocket +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +: <<'=cut' +=pod + +=head2 Process Synchronisation + +=head3 rlWaitForSocket + +Pauses script execution until socket starts listening. +Returns 0 if socket started listening, 1 if timeout was reached or PID exited. +Return code is greater than 1 in case of error. + + rlWaitForSocket {port|path} [-p PID] [-t time] + +=over + +=item port|path + +Network port to wait for opening or a path to UNIX socket. +Regular expressions are also supported. + +=item -t time + +Timeout in seconds (optional, default=120). If the socket isn't opened before +the time elapses the command returns 1. + +=item -p PID + +PID of the process that should also be running. If the process exits before +the socket is opened, the command returns with status code of 1. + +=back + +=cut + +rlWaitForSocket(){ + + local timeout=120 + local proc_pid=1 + local socket="" + + # that is the GNU extended getopt syntax! + local TEMP=$(getopt -o t:p: -n 'rlWaitForSocket' -- "$@") + if [[ $? != 0 ]] ; then + rlLogError "rlWaitForSocket: Can't parse command options, terminating..." + return 127 + fi + + eval set -- "$TEMP" + + while true ; do + case "$1" in + -t) timeout="$2"; shift 2 + ;; + -p) proc_pid="$2"; shift 2 + ;; + --) shift 1 + break + ;; + *) rlLogError "rlWaitForSocket: unrecognized option" + return 127 + ;; + esac + done + socket="$1" + + # the case statement is a portable way to check if variable contains only + # digits (regexps are not available in old, RHEL3-era, bash) + case "$timeout" in + ''|*[!0-9]*) rlLogError "rlWaitForSocket: Invalid timeout provided" + return 127 + ;; + esac + case "$proc_pid" in + ''|*[!0-9]*) rlLogError "rlWaitForSocket: Invalid PID provided" + return 127 + ;; + esac + case "$socket" in + *[0-9]) + #socket_type="network" + local grep_opt=":$socket[[:space:]]" + ;; + "") rlLogError "rlWaitForSocket: No socket specified" + return 127 + ;; + *) + #socket_type="unix" + local grep_opt="$socket" + ;; + esac + rlLogInfo "rlWaitForSocket: Waiting max ${timeout}s for socket `$socket' to start listening" + + ( while true ; do + netstat -nl | grep -E "$grep_opt" >/dev/null + if [[ $? -eq 0 ]]; then + exit 0; + else + if [[ ! -e "/proc/$proc_pid" ]]; then + exit 1; + fi + sleep 1 + fi + done ) & + local netstat_pid=$! + + ( sleep $timeout && kill -HUP -$netstat_pid ) 2>/dev/null & + local watcher=$! + + wait $netstat_pid + local ret=$? + if [[ $ret -eq 0 ]]; then + kill -s SIGKILL $watcher 2>/dev/null + rlLogInfo "rlWaitForSocket: Socket opened!" + return 0 + else + if [[ $ret -eq 1 ]]; then + kill -s SIGKILL $watcher 2>/dev/null + rlLogWarning "rlWaitForSocket: PID terminated!" + else + rlLogWarning "rlWaitForSocket: Timeout elapsed" + fi + return 1 + fi +} + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# AUTHORS +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +: <<'=cut' +=pod + +=head1 AUTHORS + +=over + +=item * + +Hubert Kario hkario@redhat.com + +=back + +=cut
Make beakerlib load synchronisation related routines by default and install them.
Signed-off-by: Hubert Kario hkario@redhat.com --- src/Makefile | 1 + src/beakerlib.sh | 1 + 2 files changed, 2 insertions(+)
diff --git a/src/Makefile b/src/Makefile index c5263f7..4379bb0 100644 --- a/src/Makefile +++ b/src/Makefile @@ -33,6 +33,7 @@ MODULES=journal.sh\ performance.sh\ analyze.sh\ libraries.sh\ + synchronisation.sh\ virtualX.sh
FILES=$(MODULES) beakerlib.sh diff --git a/src/beakerlib.sh b/src/beakerlib.sh index 84cfcf0..fe8acbd 100644 --- a/src/beakerlib.sh +++ b/src/beakerlib.sh @@ -287,6 +287,7 @@ export BEAKERLIB=${BEAKERLIB:-"/usr/share/beakerlib"} . $BEAKERLIB/analyze.sh . $BEAKERLIB/performance.sh . $BEAKERLIB/virtualX.sh +. $BEAKERLIB/synchronisation.sh if [ -d $BEAKERLIB/plugins/ ] ; then for source in $BEAKERLIB/plugins/*.sh ; do . $source
Just killing the sub shell running won't send kill signal to running commands, use treekill to kill all processes in process tree. Also gets rid of the ugly bash reports of tasks terminated in background.
Signed-off-by: Hubert Kario hkario@redhat.com --- src/synchronisation.sh | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)
diff --git a/src/synchronisation.sh b/src/synchronisation.sh index 685a758..e782481 100644 --- a/src/synchronisation.sh +++ b/src/synchronisation.sh @@ -31,6 +31,32 @@ if [ ${ret:-0} -ne 4 ]; then exit 1 fi
+# add ability to kill whole process tree +# unfortunately, because we're running inside bash script, we can't +# use the simple solution of process groups and `kill -s SIG -$pid` +# usage: __INTERNAL_killtree PID [SIGNAL] +# returns first failed kill return code or 0 if all returned success +__INTERNAL_killtree() { + local _pid=$1 + if [[ ! -n $_pid ]]; then + return 2 + fi + local _sig=${2:-TERM} + local _ret= + kill -s SIGSTOP ${_pid} || : # prevent parent from forking + local _children=$(pgrep -P ${_pid}) + local _pret=$? + if [[ $_pret -ne 0 && $_pret -ne 1 ]]; then + return 4 + fi + for _child in $_children; do + __INTERNAL_killtree ${_child} ${_sig} || _ret=${_ret:-$?} + done + kill -s ${_sig} ${_pid} || _ret=${_ret:-$?} + kill -s SIGCONT ${_pid} || : # allow for signal delivery to parent + return ${_ret:-0} +} + : <<'=cut' =pod
@@ -155,18 +181,20 @@ rlWaitForSocket(){ done ) & local netstat_pid=$!
- ( sleep $timeout && kill -HUP -$netstat_pid ) 2>/dev/null & + ( sleep $timeout; __INTERNAL_killtree $netstat_pid SIGKILL) 2>/dev/null & local watcher=$!
- wait $netstat_pid + wait $netstat_pid 2> /dev/null local ret=$? if [[ $ret -eq 0 ]]; then - kill -s SIGKILL $watcher 2>/dev/null + __INTERNAL_killtree $watcher SIGKILL 2>/dev/null + wait $watcher 2> /dev/null rlLogInfo "rlWaitForSocket: Socket opened!" return 0 else if [[ $ret -eq 1 ]]; then - kill -s SIGKILL $watcher 2>/dev/null + __INTERNAL_killtree $watcher SIGKILL 2>/dev/null + wait $watcher 2> /dev/null rlLogWarning "rlWaitForSocket: PID terminated!" else rlLogWarning "rlWaitForSocket: Timeout elapsed"
add routine that allows waiting for any command to return specified exit code
Signed-off-by: Hubert Kario hkario@redhat.com --- src/synchronisation.sh | 185 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 184 insertions(+), 1 deletion(-)
diff --git a/src/synchronisation.sh b/src/synchronisation.sh index e782481..2885058 100644 --- a/src/synchronisation.sh +++ b/src/synchronisation.sh @@ -57,6 +57,136 @@ __INTERNAL_killtree() { return ${_ret:-0} }
+# Since all "wait for something to happen" utilities are basically the same, +# use a generic routine that can do all their work +__INTERNAL_wait_for_cmd() { + + # don't wait more than this many seconds + local timeout=120 + # delay between command invocations + local delay=1 + # abort if this process terminates + local proc_pid=1 + # command to run + local cmd + # maximum number of command invocations + local max_invoc="" + # expected return code of command + local exp_retval=0 + # name of routine to return errors for + local routine_name="$1" + shift 1 + + # that is the GNU extended getopt syntax! + local TEMP=$(getopt -o t:p:m:d:r: -n '$routine_name' -- "$@") + if [[ $? != 0 ]] ; then + rlLogError "$routine_name: Can't parse command options, terminating..." + return 127 + fi + + eval set -- "$TEMP" + + while true ; do + case "$1" in + -t) timeout="$2"; shift 2 + ;; + -p) proc_pid="$2"; shift 2 + ;; + -m) max_invoc="$2"; shift 2 + ;; + -d) delay="$2"; shift 2 + ;; + -r) exp_retval="$2"; shift 2 + ;; + --) shift 1 + break + ;; + *) rlLogError "$routine_name: unrecognized option" + return 127 + ;; + esac + done + cmd="$1" + + if [[ $routine_name == "rlWaitForCmd" ]]; then + rlLogInfo "$routine_name: waiting for `$cmd' to return $exp_retval in $timeout seconds" + fi + + # the case statement is a portable way to check if variable contains only + # digits (regexps are not available in old, RHEL3-era, bash) + case "$timeout" in + ''|*[!0-9]*) rlLogError "${routine_name}: Invalid timeout provided" + return 127 + ;; + esac + case "$proc_pid" in + ''|*[!0-9]*) rlLogError "${routine_name}: Invalid PID provided" + return 127 + ;; + esac + if [[ -n "$max_invoc" ]]; then + case "$max_invoc" in + ''|*[!0-9]*) rlLogError "${routine_name}: Invalid maximum number of invocations provided" + return 127 + ;; + esac + fi + # delay can be fractional, so "." is OK + case "$delay" in + ''|*[!0-9.]*) rlLogError "${routine_name}: Invalid delay specified" + return 127 + ;; + esac + case "$exp_retval" in + ''|*[!0-9]*) rlLogError "${routine_name}: Invalid expected command return value provided" + return 127 + ;; + esac + + # we use two child processes to get the timeout and process execution + # one (command_pid) runs the command until it returns expected return value + # the other is just a timout (watcher) + + # run command in loop + ( local i=0 + while [[ -n $max_invoc && $i -lt $max_invoc ]] || [[ ! -n $max_invoc ]]; do + eval $cmd + if [[ $? -eq $exp_retval ]]; then + exit 0; + else + if [[ ! -e "/proc/$proc_pid" ]]; then + exit 1; + fi + sleep $delay + fi + i=$((i+1)) + done + exit 1) & + local command_pid=$! + + # kill command running in background if the timout has elapsed + ( sleep $timeout && __INTERNAL_killtree $command_pid SIGKILL) 2>/dev/null & + local watcher=$! + + wait $command_pid 2> /dev/null + local ret=$? + if [[ $ret -eq 0 ]]; then + __INTERNAL_killtree $watcher SIGKILL 2>/dev/null + wait $watcher 2> /dev/null + rlLogInfo "${routine_name}: Wait successful!" + return 0 + else + if [[ $ret -eq 1 ]]; then + __INTERNAL_killtree $watcher SIGKILL 2>/dev/null + wait $watcher 2> /dev/null + rlLogWarning "${routine_name}: PID terminated!" + else + rlLogWarning "${routine_name}: Timeout elapsed" + fi + return 1 + fi +} + : <<'=cut' =pod
@@ -74,13 +204,66 @@ of applications. =cut
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# rlWaitForSocket +# rlWaitForCmd # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + : <<'=cut' =pod
=head2 Process Synchronisation
+=head3 rlWaitForCmd + +Pauses script execution until command exit status is the expeced value. +Logs a WARNING and returns 1 if the command didn't exit successfully +before timeout elapsed or a maximum number of invocations has been +reached. + + rlWaitForCmd command [-p PID] [-t time] [-n count] [-d delay] [-r retval] + +=over + +=item command + +Command that will be executed until its return code is equal 0 or value +speciefied as option to `-r'. + +=item -t time + +Timeout in seconds, default=120. If the command doesn't return 0 +before time elapses, the command will be killed. + +=item -p PID + +PID of the process to check before running command. If the process +exits before the socket is opened, the command will log a WARNING. + +=item -m count + +Maximum number of `command' executions before continuing anyway. Default is +infite. + +=item -d delay + +Delay between `command' invocations. Default 1. + +=item -r retval + +Expected return value of command. Default 0. + +=back + +=cut +rlWaitForCmd() { + __INTERNAL_wait_for_cmd rlWaitForCmd "$@" +} + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# rlWaitForSocket +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +: <<'=cut' +=pod + =head3 rlWaitForSocket
Pauses script execution until socket starts listening.
remove code duplication, add support for extra option: -d delay
Signed-off-by: Hubert Kario hkario@redhat.com --- src/synchronisation.sh | 55 ++++++++++---------------------------------------- 1 file changed, 11 insertions(+), 44 deletions(-)
diff --git a/src/synchronisation.sh b/src/synchronisation.sh index 2885058..c36110a 100644 --- a/src/synchronisation.sh +++ b/src/synchronisation.sh @@ -270,7 +270,7 @@ Pauses script execution until socket starts listening. Returns 0 if socket started listening, 1 if timeout was reached or PID exited. Return code is greater than 1 in case of error.
- rlWaitForSocket {port|path} [-p PID] [-t time] + rlWaitForSocket {port|path} [-p PID] [-t time] [-d delay]
=over
@@ -289,6 +289,10 @@ the time elapses the command returns 1. PID of the process that should also be running. If the process exits before the socket is opened, the command returns with status code of 1.
+=item -d delay + +Delay between subsequent checks for availability of socket. Default 1. + =back
=cut @@ -297,10 +301,11 @@ rlWaitForSocket(){
local timeout=120 local proc_pid=1 + local delay=1 local socket=""
# that is the GNU extended getopt syntax! - local TEMP=$(getopt -o t:p: -n 'rlWaitForSocket' -- "$@") + local TEMP=$(getopt -o t:p:d: -n 'rlWaitForSocket' -- "$@") if [[ $? != 0 ]] ; then rlLogError "rlWaitForSocket: Can't parse command options, terminating..." return 127 @@ -314,6 +319,8 @@ rlWaitForSocket(){ ;; -p) proc_pid="$2"; shift 2 ;; + -d) delay="$2"; shift 2 + ;; --) shift 1 break ;; @@ -326,16 +333,6 @@ rlWaitForSocket(){
# the case statement is a portable way to check if variable contains only # digits (regexps are not available in old, RHEL3-era, bash) - case "$timeout" in - ''|*[!0-9]*) rlLogError "rlWaitForSocket: Invalid timeout provided" - return 127 - ;; - esac - case "$proc_pid" in - ''|*[!0-9]*) rlLogError "rlWaitForSocket: Invalid PID provided" - return 127 - ;; - esac case "$socket" in *[0-9]) #socket_type="network" @@ -351,39 +348,9 @@ rlWaitForSocket(){ esac rlLogInfo "rlWaitForSocket: Waiting max ${timeout}s for socket `$socket' to start listening"
- ( while true ; do - netstat -nl | grep -E "$grep_opt" >/dev/null - if [[ $? -eq 0 ]]; then - exit 0; - else - if [[ ! -e "/proc/$proc_pid" ]]; then - exit 1; - fi - sleep 1 - fi - done ) & - local netstat_pid=$! + local cmd="netstat -nl | grep -E '$grep_opt' >/dev/null"
- ( sleep $timeout; __INTERNAL_killtree $netstat_pid SIGKILL) 2>/dev/null & - local watcher=$! - - wait $netstat_pid 2> /dev/null - local ret=$? - if [[ $ret -eq 0 ]]; then - __INTERNAL_killtree $watcher SIGKILL 2>/dev/null - wait $watcher 2> /dev/null - rlLogInfo "rlWaitForSocket: Socket opened!" - return 0 - else - if [[ $ret -eq 1 ]]; then - __INTERNAL_killtree $watcher SIGKILL 2>/dev/null - wait $watcher 2> /dev/null - rlLogWarning "rlWaitForSocket: PID terminated!" - else - rlLogWarning "rlWaitForSocket: Timeout elapsed" - fi - return 1 - fi + __INTERNAL_wait_for_cmd "rlWaitForSocket" "${cmd}" -t $timeout -p $proc_pid -d $delay }
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
add synchronisation mechanism that can wait for file creation
Signed-off-by: Hubert Kario hkario@redhat.com --- src/synchronisation.sh | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+)
diff --git a/src/synchronisation.sh b/src/synchronisation.sh index c36110a..4ac3e96 100644 --- a/src/synchronisation.sh +++ b/src/synchronisation.sh @@ -259,6 +259,82 @@ rlWaitForCmd() { }
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# rlWaitForFile +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +: <<'=cut' +=pod + +=head3 rlWaitForFile + +Pauses script execution until specified file or directory starts existing. +Returns 0 if file started existing, 1 if timeout was reached or PID exited. +Return code is greater than 1 in case of error. + + rlWaitForFile path [-p PID] [-t time] [-d delay] + +=over + +=item path + +Path to file that should start existing. + +=item -t time + +Timeout in seconds (optional, default=120). If the file isn't opened before +the time elapses the command returns 1. + +=item -p PID + +PID of the process that should also be running. If the process exits before +the file is created, the command returns with status code of 1. + +=item -d delay + +Delay between subsequent checks for existence of file. Default 1. + +=back +=cut +rlWaitForFile() { + local timeout=120 + local proc_pid=1 + local delay=1 + local file="" + + # that is the GNU extended getopt syntax! + local TEMP=$(getopt -o t:p:d: -n 'rlWaitForFile' -- "$@") + if [[ $? != 0 ]] ; then + rlLogError "rlWaitForSocket: Can't parse command options, terminating..." + return 127 + fi + + eval set -- "$TEMP" + + while true ; do + case "$1" in + -t) timeout="$2"; shift 2 + ;; + -p) proc_pid="$2"; shift 2 + ;; + -d) delay="$2"; shift 2 + ;; + --) shift 1 + break + ;; + *) rlLogError "rlWaitForFile: unrecognized option" + return 127 + ;; + esac + done + file="$1" + + rlLogInfo "rlWaitForFile: Waiting max ${timeout}s for file `$file' to start existing" + + local cmd="[[ -e '$file' ]]" + + __INTERNAL_wait_for_cmd "rlWaitForFile" "${cmd}" -t "$timeout" -p "$proc_pid" -d "$delay" +} + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # rlWaitForSocket # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ : <<'=cut'
beakerlib-devel@lists.fedorahosted.org