[PATCH 1/2] selftest: run-test.sh: wait for subprocess instead of kill it

[Tao Liu]

When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the
crash and dump, server VM will do vmcore saving and if-vmcore-exists
check.

Previously, when client VM finishes running, run-test.sh will kill the lead background
process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string.
However it didn't wait for server VM to finish. As a result, the server VM's final
outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND"
sometimes.

For example, the following is the pstree status of $(jobs -p) before it
gets killed. We can see the server VM is still running:

run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump
  └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
      └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu...
          └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
              ├─{qemu-system-x86},172489
              ├─{qemu-system-x86},172492
              ├─{qemu-system-x86},172493
              ├─{qemu-system-x86},172628
              └─{qemu-system-x86},172629

In this patch, we will wait for $(jobs -p) to finish, in order to get
the complete output of test results.

Signed-off-by: Tao Liu 
---
 tests/scripts/run-test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh
index c3e94a3..1501db4 100755
--- a/tests/scripts/run-test.sh
+++ b/tests/scripts/run-test.sh
@@ -140,7 +140,7 @@ for test_case in $testcases; do

if [ $console -eq 1 ]; then
    	run_test_sync $script | tee $(get_test_console_file $script)
-		[ -n "$(jobs -p)" ] && kill $(jobs -p)
+		[ -n "$(jobs -p)" ] && wait $(jobs -p)
    else
    	$(run_test_sync $script > $(get_test_console_file $script)) &
    	watch_test_outputs $test_outputs

[PATCH 2/2] selftest: Add early kdump test

[Tao Liu]

This patch will introduce early kdump test.

It reuses the code of nfs kdump test, in order to setup 2 seperated VMs,
one(the client) for trigger the early kdump and crash, the other(the server)
for saving vmcore and check if vmcore exists. In order to minimize the
repetted code, a soft link is made to copy the same server side code.

Signed-off-by: Tao Liu 
---
 .../testcases/nfs-early-kdump/0-server.sh     |  1 +
 .../testcases/nfs-early-kdump/1-client.sh     | 46 +++++++++++++++++++
 2 files changed, 47 insertions(+)
 create mode 120000 tests/scripts/testcases/nfs-early-kdump/0-server.sh
 create mode 100755 tests/scripts/testcases/nfs-early-kdump/1-client.sh

diff --git a/tests/scripts/testcases/nfs-early-kdump/0-server.sh b/tests/scripts/testcases/nfs-early-kdump/0-server.sh
new file mode 120000
index 0000000..3f888ce
--- /dev/null
+++ b/tests/scripts/testcases/nfs-early-kdump/0-server.sh
@@ -0,0 +1 @@
+../nfs-kdump/0-server.sh
\ No newline at end of file
diff --git a/tests/scripts/testcases/nfs-early-kdump/1-client.sh b/tests/scripts/testcases/nfs-early-kdump/1-client.sh
new file mode 100755
index 0000000..dfef4d8
--- /dev/null
+++ b/tests/scripts/testcases/nfs-early-kdump/1-client.sh
@@ -0,0 +1,46 @@
+# Executed before VM starts
+on_build() {
+	img_inst_pkg "nfs-utils"
+	img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57"
+}
+
+on_test() {
+	local boot_count=$(get_test_boot_count)
+	local nfs_server=192.168.77.1
+	local earlykdump_path="/usr/lib/dracut/modules.d/99earlykdump/early-kdump.sh"
+	local tmp_file="/tmp/.tmp-file"
+
+	if [[ ! -f $earlykdump_path ]]; then
+		test_failed "early-kdump.sh not exist!"
+	fi
+
+	if [ $boot_count -eq 1 ]; then
+		cat << EOF > /etc/kdump.conf
+nfs $nfs_server:/srv/nfs
+core_collector makedumpfile -l --message-level 7 -d 31
+final_action poweroff
+EOF
+
+		while ! ping -c 1 $nfs_server -W 1; do
+			sleep 1
+		done
+
+		kdumpctl start \
+			|| test_failed "Failed to start kdump"
+		grubby --update-kernel=ALL --args=rd.earlykdump
+
+		cat << EOF > $tmp_file
+echo 1 > /proc/sys/kernel/sysrq
+echo c > /proc/sysrq-trigger
+EOF
+		sed -i "/early_kdump_load$/r $tmp_file" $earlykdump_path
+		dracut -f --add earlykdump
+		kdumpctl restart \
+			|| test_failed "Failed to start earlykdump"
+
+		sync
+		reboot
+	else
+		test_failed "Unexpected reboot"
+	fi
+}

[PATCH 2/2] selftest: Add early kdump test

[Coiby Xu]

Thanks for the patch!

Acked-by: Coiby Xu

On Thu, Dec 30, 2021 at 11:26:43AM +0800, Tao Liu wrote:
> This patch will introduce early kdump test.
>
> It reuses the code of nfs kdump test, in order to setup 2 seperated VMs,
> one(the client) for trigger the early kdump and crash, the other(the server)
> for saving vmcore and check if vmcore exists. In order to minimize the
> repetted code, a soft link is made to copy the same server side code.
> Signed-off-by: Tao Liu
>
> .../testcases/nfs-early-kdump/0-server.sh     |  1 +
> .../testcases/nfs-early-kdump/1-client.sh     | 46 +++++++++++++++++++
> 2 files changed, 47 insertions(+)
> create mode 120000 tests/scripts/testcases/nfs-early-kdump/0-server.sh
> create mode 100755 tests/scripts/testcases/nfs-early-kdump/1-client.sh
>
> diff --git a/tests/scripts/testcases/nfs-early-kdump/0-server.sh b/tests/scripts/testcases/nfs-early-kdump/0-server.sh
> new file mode 120000
> index 0000000..3f888ce
> --- /dev/null
> +++ b/tests/scripts/testcases/nfs-early-kdump/0-server.sh
> @@ -0,0 +1 @@
> +../nfs-kdump/0-server.sh
> \ No newline at end of file
> diff --git a/tests/scripts/testcases/nfs-early-kdump/1-client.sh b/tests/scripts/testcases/nfs-early-kdump/1-client.sh
> new file mode 100755
> index 0000000..dfef4d8
> --- /dev/null
> +++ b/tests/scripts/testcases/nfs-early-kdump/1-client.sh
> @@ -0,0 +1,46 @@
> +# Executed before VM starts
> +on_build() {
> img_inst_pkg "nfs-utils"
> img_add_qemu_cmd "-nic socket,connect=127.0.0.1:8010,mac=52:54:00:12:34:57"
>
> +}
>
> +on_test() {
> local boot_count=$(get_test_boot_count)
> local nfs_server=192.168.77.1
> local earlykdump_path="/usr/lib/dracut/modules.d/99earlykdump/early-kdump.sh"
> local tmp_file="/tmp/.tmp-file"
> if [[ ! -f $earlykdump_path ]]; then
> test_failed "early-kdump.sh not exist!"
> fi
> if [ $boot_count -eq 1 ]; then
> cat << EOF > /etc/kdump.conf
>
> +nfs $nfs_server:/srv/nfs
> +core_collector makedumpfile -l --message-level 7 -d 31
> +final_action poweroff
> +EOF
> while ! ping -c 1 $nfs_server -W 1; do
> 	sleep 1
> done
> kdumpctl start \
> 	|| test_failed "Failed to start kdump"
> grubby --update-kernel=ALL --args=rd.earlykdump
> cat << EOF > $tmp_file
>
> +echo 1 > /proc/sys/kernel/sysrq
> +echo c > /proc/sysrq-trigger
> +EOF
> sed -i "/early_kdump_load$/r $tmp_file" $earlykdump_path
> dracut -f --add earlykdump
> kdumpctl restart \
> 	|| test_failed "Failed to start earlykdump"
> sync
> reboot
> else
> test_failed "Unexpected reboot"
> fi
> +}
>
> 2.33.1

[Coiby Xu]

Hi Tao,

Thanks for the patch!

If we use wait, could the test hang because server VM somehow refuses to
quit?

On Thu, Dec 30, 2021 at 11:26:42AM +0800, Tao Liu wrote:
> When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the
> crash and dump, server VM will do vmcore saving and if-vmcore-exists
> check.
>
> Previously, when client VM finishes running, run-test.sh will kill the lead background
> process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string.
> However it didn't wait for server VM to finish. As a result, the server VM's final
> outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND"
> sometimes.
>
> For example, the following is the pstree status of $(jobs -p) before it
> gets killed. We can see the server VM is still running:
>
> run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump
>  └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
>      └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu...
>          └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
>              ├─{qemu-system-x86},172489
>              ├─{qemu-system-x86},172492
>              ├─{qemu-system-x86},172493
>              ├─{qemu-system-x86},172628
>              └─{qemu-system-x86},172629
>
> In this patch, we will wait for $(jobs -p) to finish, in order to get
> the complete output of test results.
> Signed-off-by: Tao Liu
>
> tests/scripts/run-test.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh
> index c3e94a3..1501db4 100755
> --- a/tests/scripts/run-test.sh
> +++ b/tests/scripts/run-test.sh
> @@ -140,7 +140,7 @@ for test_case in $testcases; do
>
> if [ $console -eq 1 ]; then
>    	run_test_sync $script | tee $(get_test_console_file $script)
> [ -n "$(jobs -p)" ] && kill $(jobs -p)
> [ -n "$(jobs -p)" ] && wait $(jobs -p)
> else
> $(run_test_sync $script > $(get_test_console_file $script)) &
> watch_test_outputs $test_outputs
>
> -- 
> 2.33.1

[Tao Liu]

Hi Coiby,

On Thu, Jan 20, 2022 at 4:06 PM Coiby Xu  wrote:
> Hi Tao,
>
> Thanks for the patch!
>
> If we use wait, could the test hang because server VM somehow refuses to
> quit?

I think it won't hang if VM refuses to quit.  You can see the process
tree of $(jobs -p):

run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh
--console nfs-early-kdump
  └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
      └─timeout,172480 --foreground 10m
/root/kexec-tools/tests/scripts/run-qemu...
          └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
              ├─{qemu-system-x86},172489
              ├─{qemu-system-x86},172492
              ├─{qemu-system-x86},172493
              ├─{qemu-system-x86},172628
              └─{qemu-system-x86},172629

The qemu cmdline is wrapped by timeout. So even if the VM won't exit
itself, it will be killed by timeout 10m later.
So I think waiting for this process group won't hang.

Thanks,
Tao Liu
> On Thu, Dec 30, 2021 at 11:26:42AM +0800, Tao Liu wrote:
> > When run tests with 2 VMs, for example nfs/ssh kdump tests, client VM will do the
> > crash and dump, server VM will do vmcore saving and if-vmcore-exists
> > check.
> >
> > Previously, when client VM finishes running, run-test.sh will kill the lead background
> > process, and then check if server VM has outputted "TEST PASSED" or "TEST FAILED" string.
> > However it didn't wait for server VM to finish. As a result, the server VM's final
> > outputs are not collected and checked, leaving the test result as "TEST RESULT NOT FOUND"
> > sometimes.
> >
> > For example, the following is the pstree status of $(jobs -p) before it
> > gets killed. We can see the server VM is still running:
> >
> > run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh --console nfs-early-kdump
> >  └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
> >      └─timeout,172480 --foreground 10m /root/kexec-tools/tests/scripts/run-qemu...
> >          └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
> >              ├─{qemu-system-x86},172489
> >              ├─{qemu-system-x86},172492
> >              ├─{qemu-system-x86},172493
> >              ├─{qemu-system-x86},172628
> >              └─{qemu-system-x86},172629
> >
> > In this patch, we will wait for $(jobs -p) to finish, in order to get
> > the complete output of test results.
> > Signed-off-by: Tao Liu
> >
> > tests/scripts/run-test.sh | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tests/scripts/run-test.sh b/tests/scripts/run-test.sh
> > index c3e94a3..1501db4 100755
> > --- a/tests/scripts/run-test.sh
> > +++ b/tests/scripts/run-test.sh
> > @@ -140,7 +140,7 @@ for test_case in $testcases; do
> >   if [ $console -eq 1 ]; then
> >           run_test_sync $script | tee $(get_test_console_file $script)
> >          [ -n "$(jobs -p)" ] && kill $(jobs -p)
> >          [ -n "$(jobs -p)" ] && wait $(jobs -p)
> > else
> >         $(run_test_sync $script > $(get_test_console_file $script)) &
> >         watch_test_outputs $test_outputs
> >
> > --
> > 2.33.1
>
> --
> Best regards,
> Coiby
> _______________________________________________
> kexec mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Coiby Xu]

On Thu, Jan 20, 2022 at 04:59:37PM +0800, Tao Liu wrote:
> Hi Coiby,
>
> On Thu, Jan 20, 2022 at 4:06 PM Coiby Xu  wrote:
> > Hi Tao,
> >
> > Thanks for the patch!
> >
> > If we use wait, could the test hang because server VM somehow refuses to
> > quit?
>
> I think it won't hang if VM refuses to quit.  You can see the process
> tree of $(jobs -p):
>
> run-test.sh,172455 /root/kexec-tools/tests/scripts/run-test.sh
> --console nfs-early-kdump
>  └─run-test.sh,172457 /root/kexec-tools/tests/scripts/run-test.sh --console...
>      └─timeout,172480 --foreground 10m
> /root/kexec-tools/tests/scripts/run-qemu...
>          └─qemu-system-x86,172481 -enable-kvm -cpu host -nodefaults...
>              ├─{qemu-system-x86},172489
>              ├─{qemu-system-x86},172492
>              ├─{qemu-system-x86},172493
>              ├─{qemu-system-x86},172628
>              └─{qemu-system-x86},172629
>
> The qemu cmdline is wrapped by timeout. So even if the VM won't exit
> itself, it will be killed by timeout 10m later.
> So I think waiting for this process group won't hang.

Oh, I forgot the timeout wrapper. Thanks for the explanation!

Acked-by: Coiby Xu