Hi, All
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
To solve this problem, we need to separate the error handling code from dracut-pre-pivot hook, and every time when a failure shows up, the separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into shell every time upon a critical failure. It's very convenient for us to re-use the framework of systemd emergency, because we don't have to touch the other parts of systemd. We can use our own script instead of the default one.
This new scheme will overwrite emergency shell and replace with kdump error handling code. And this code will do the error handling as needed. Now, we will not rely on dracut-pre-pivot hook running always. Instead whenever error happens and it is serious enough that emergency shell needed to run, now kdump error handler will run.
This patchset introduce a new kdump emergency service. It will override the existing emergency.service. When fatal error occurs, this emergency service will be triggered and systemd will isolate to emergency path.
This kdump emergency service is a wrapper service and it will isolate to kdump-error-handler service. kdump-error-handler can will read kdump.conf and act according to the configired "default action" (reboot/poweroff/halt/shell/dump_to_rootfs).
Along with this patchset, kdump-capture.service is introduced as a service unit to run kdump.sh. When kdump-capture.service fails, systemd will isolate to kdump emergency service. I copied all the dependencies from dracut-pre-pivot.service to kdump-capture.service so that kdump.sh will be called at the correct time window.
v2: Address several comments from Vivek: - split [patch 1] into 1/6 and 2/6 - remove several unnecessary lines in service unit. - modify the description of kdump-capture.service - introduce another kdump lib kdump-lib-2.sh used in 2nd kernel.
V3: Address several comments from Vivek: - update description of [PATCH 1/5] and [PATCH 5/5] - separate the patch of cleanup to kdump-lib-initramfs.sh
V4: Address several comments from Vivek and Dave: - meld the original [PATCH 3/5] into [PATCH 1/5] - add GPL license for kdump-error-handler.service and kdump-capture.service - separate the "x-initrd.mount" and "mount under /sysroot" patch to another patchset
V5: - Rebase on top of the fadump patchset. - Enable dracut-emergency again, but override it with kdump error handler.
V6: - reboot if entering error handler in error handling path - add comments in dracut-kdump-emergency.service - update changelog
V7: - add wrapper emergency.service to isolate to kdump-error-handler service - remove extra code introduced in v6 for avoiding entering error path again
WANG Chao (2): Introduce kdump error handling service Introduce kdump capture service
dracut-kdump-capture.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump-emergency.service | 27 +++++++++++++++++++++++++++ dracut-kdump-error-handler.service | 34 ++++++++++++++++++++++++++++++++++ dracut-kdump-error-handler.sh | 10 ++++++++++ dracut-kdump.sh | 9 ++------- dracut-module-setup.sh | 11 +++++++++-- kdump-lib-initramfs.sh | 34 ++++++++++++++++++++++++++++++---- kexec-tools.spec | 9 ++++++++- 8 files changed, 150 insertions(+), 14 deletions(-) create mode 100644 dracut-kdump-capture.service create mode 100644 dracut-kdump-emergency.service create mode 100644 dracut-kdump-error-handler.service create mode 100755 dracut-kdump-error-handler.sh
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs. For example:
sysroot-var-crash.mount --> initrd-root-fs.target --> initrd.target \ --> dracut-pre-pivot.service --> kdump.sh
If /sysroot/var/crash mount fails, initrd-root-fs.target will not be reached. And then initrd.target will not be reached, dracut-pre-pivot.service wouldn't run. Finally kdump.sh wouldn't run.
To solve this problem, we need to separate the error handling code from dracut-pre-pivot hook, and every time when a failure shows up, the separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into shell every time upon a critical failure. It's very convenient for us to re-use the framework of systemd emergency, because we don't have to touch the other parts of systemd. We can use our own script instead of the default one.
This new scheme will overwrite emergency shell and replace with kdump error handling code. And this code will do the error handling as needed. Now, we will not rely on dracut-pre-pivot hook running always. Instead whenever error happens and it is serious enough that emergency shell needed to run, now kdump error handler will run.
dracut-emergency is also replaced by kdump error handler and it's enabled again all the way down. So all the failure (including systemd and dracut) in 2nd kernel could be captured, and trigger kdump error handler.
dracut-initqueue is a special case, which calls "systemctl start emergency" directly, not via "OnFailure=emergency". In case of failure, emergency is started, but not in a isolation mode, which means dracut-initqueue is still running. On the other hand, emergency will call dracut-initqueue again when default action is dump_to_rootfs. systemd would block on the last dracut-initqueue, waiting for the first instance to exit, which leaves us hang. It looks like the following:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency (running) --> kdump-error-handler.sh (running) --> call dracut-initqueue: --> blocking and waiting for the original instance to exit.
To fix this, I'd like to introduce a wrapper emergency service. This emegency service will replace both the systemd and dracut emergency. And this service does nothing but to isolate to real kdump error handler service:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency isolate to kdump-error-handler.service --> dracut-emergency and dracut-initqueue will both be stopped and kdump-error-handler.service will run kdump-error-handler.sh.
In a normal failure case, this still works: foo.service fails --> trigger emergency.service --> emergency.service isolates to kdump-error-handler.service --> kdump-error-handler.service will run kdump-error-handler.sh
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-kdump-emergency.service | 27 +++++++++++++++++++++++++++ dracut-kdump-error-handler.service | 34 ++++++++++++++++++++++++++++++++++ dracut-kdump-error-handler.sh | 10 ++++++++++ dracut-kdump.sh | 4 ---- dracut-module-setup.sh | 7 ++++++- kdump-lib-initramfs.sh | 34 ++++++++++++++++++++++++++++++---- kexec-tools.spec | 7 ++++++- 7 files changed, 113 insertions(+), 10 deletions(-) create mode 100644 dracut-kdump-emergency.service create mode 100644 dracut-kdump-error-handler.service create mode 100755 dracut-kdump-error-handler.sh
diff --git a/dracut-kdump-emergency.service b/dracut-kdump-emergency.service new file mode 100644 index 0000000..eef1763 --- /dev/null +++ b/dracut-kdump-emergency.service @@ -0,0 +1,27 @@ +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +# This service will be placed in kdump initramfs and replace both the systemd +# emergency service and dracut emergency shell. IOW, any emergency will be +# kick this service and in turn isolating to kdump error handler. + +[Unit] +Description=Kdump Emergency +DefaultDependencies=no + +[Service] +ExecStart=systemctl --no-block isolate kdump-error-handler.service +Type=oneshot +StandardInput=tty-force +StandardOutput=inherit +StandardError=inherit +KillMode=process +IgnoreSIGPIPE=no + +# Bash ignores SIGTERM, so we send SIGHUP instead, to ensure that bash +# terminates cleanly. +KillSignal=SIGHUP diff --git a/dracut-kdump-error-handler.service b/dracut-kdump-error-handler.service new file mode 100644 index 0000000..13090be --- /dev/null +++ b/dracut-kdump-error-handler.service @@ -0,0 +1,34 @@ +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +# This service will run the real kdump error handler code. Executing the +# default action configured in kdump.conf + +[Unit] +Description=Kdump Error Handler +DefaultDependencies=no +After=systemd-vconsole-setup.service +Wants=systemd-vconsole-setup.service +AllowIsolate=yes + +[Service] +Environment=HOME=/ +Environment=DRACUT_SYSTEMD=1 +Environment=NEWROOT=/sysroot +WorkingDirectory=/ +ExecStart=/bin/kdump-error-handler.sh +ExecStopPost=-/usr/bin/systemctl --fail --no-block default +Type=oneshot +StandardInput=tty-force +StandardOutput=inherit +StandardError=inherit +KillMode=process +IgnoreSIGPIPE=no + +# Bash ignores SIGTERM, so we send SIGHUP instead, to ensure that bash +# terminates cleanly. +KillSignal=SIGHUP diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh new file mode 100755 index 0000000..2f0f1d1 --- /dev/null +++ b/dracut-kdump-error-handler.sh @@ -0,0 +1,10 @@ +#!/bin/sh + +. /lib/kdump-lib-initramfs.sh + +set -o pipefail +export PATH=$PATH:$KDUMP_SCRIPT_DIR + +get_kdump_confs +do_default_action +do_final_action diff --git a/dracut-kdump.sh b/dracut-kdump.sh index a2bfa05..83c1e96 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -9,10 +9,6 @@ exec &> /dev/console . /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
-if [ -f "$initdir/lib/dracut/no-emergency-shell" ]; then - rm -f -- $initdir/lib/dracut/no-emergency-shell -fi - set -o pipefail DUMP_RETVAL=0
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 758420b..9e73f53 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -563,7 +563,6 @@ kdump_install_random_seed() {
install() { kdump_install_conf - >"$initdir/lib/dracut/no-emergency-shell"
if is_ssh_dump_target; then kdump_install_random_seed @@ -581,6 +580,12 @@ install() { inst_hook pre-pivot 9999 "$moddir/kdump.sh" inst "/lib/kdump/kdump-lib.sh" "/lib/kdump-lib.sh" inst "/lib/kdump/kdump-lib-initramfs.sh" "/lib/kdump-lib-initramfs.sh" + inst "$moddir/kdump-error-handler.sh" "/usr/bin/kdump-error-handler.sh" + inst "$moddir/kdump-error-handler.service" "$systemdsystemunitdir/kdump-error-handler.service" + # Replace existing emergency service + cp "$moddir/kdump-emergency.service" "$initdir/$systemdsystemunitdir/emergency.service" + # Also redirect dracut-emergency to kdump error handler + ln_r "$systemdsystemunitdir/emergency.service" "$systemdsystemunitdir/dracut-emergency.service"
# Check for all the devices and if any device is iscsi, bring up iscsi # target. Ideally all this should be pushed into dracut iscsi module diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 9118b64..1517712 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -1,6 +1,5 @@ # These variables and functions are useful in 2nd kernel
-. /lib/dracut-lib.sh . /lib/kdump-lib.sh
KDUMP_PATH="/var/crash" @@ -23,6 +22,7 @@ NEWROOT="/sysroot" get_kdump_confs() { local config_opt config_val + local user_specified_cc
while read config_opt config_val; do @@ -34,6 +34,7 @@ get_kdump_confs() ;; core_collector) [ -n "$config_val" ] && CORE_COLLECTOR="$config_val" + user_specified_cc=yes ;; sshkey) if [ -f "$config_val" ]; then @@ -55,7 +56,7 @@ get_kdump_confs() default) case $config_val in shell) - DEFAULT_ACTION="_emergency_shell kdump" + DEFAULT_ACTION="kdump_emergency_shell" ;; reboot) DEFAULT_ACTION="do_umount; reboot -f" @@ -67,12 +68,19 @@ get_kdump_confs() DEFAULT_ACTION="do_umount; poweroff -f" ;; dump_to_rootfs) - DEFAULT_ACTION="dump_fs $NEWROOT" + DEFAULT_ACTION="dump_to_rootfs" ;; esac ;; esac done < $KDUMP_CONF + + if is_ssh_dump_target || is_raw_dump_target; then + if [ -z "$user_specified_cc" ]; then + CORE_COLLECTOR="$CORE_COLLECTOR -F" + fi + fi + }
# dump_fs <mount point| device> @@ -127,6 +135,24 @@ save_vmcore_dmesg_fs() { fi }
+dump_to_rootfs() +{ + + echo "Kdump: trying to bring up rootfs device" + systemctl start dracut-initqueue + echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds" + systemctl start sysroot.mount + + dump_fs $NEWROOT +} + +kdump_emergency_shell() +{ + echo "PS1="kdump:\${PWD}# "" >/etc/profile + /bin/dracut-emergency + rm -f /etc/profile +} + do_umount() { umount -Rf $NEWROOT @@ -134,7 +160,7 @@ do_umount()
do_default_action() { - wait_for_loginit + echo "Kdump: Executing default action $DEFAULT_ACTION" eval $DEFAULT_ACTION }
diff --git a/kexec-tools.spec b/kexec-tools.spec index b55a7f1..7d64852 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -36,6 +36,9 @@ Source24: kdump-lib-initramfs.sh Source100: dracut-kdump.sh Source101: dracut-module-setup.sh Source102: dracut-monitor_dd_progress +Source103: dracut-kdump-error-handler.sh +Source104: dracut-kdump-emergency.service +Source105: dracut-kdump-error-handler.service
Requires(post): systemd-units Requires(preun): systemd-units @@ -210,7 +213,9 @@ mkdir -p -m755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpba cp %{SOURCE100} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} cp %{SOURCE101} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}} cp %{SOURCE102} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE102}} - +cp %{SOURCE103} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE103}} +cp %{SOURCE104} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE104}} +cp %{SOURCE105} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE105}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}}
On Fri, Aug 01, 2014 at 01:04:54PM +0800, WANG Chao wrote:
[..]
dracut-initqueue is a special case, which calls "systemctl start emergency" directly, not via "OnFailure=emergency". In case of failure, emergency is started, but not in a isolation mode, which means dracut-initqueue is still running. On the other hand, emergency will call dracut-initqueue again when default action is dump_to_rootfs. systemd would block on the last dracut-initqueue, waiting for the first instance to exit, which leaves us hang. It looks like the following:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency (running) --> kdump-error-handler.sh (running) --> call dracut-initqueue: --> blocking and waiting for the original instance to exit.
To fix this, I'd like to introduce a wrapper emergency service. This emegency service will replace both the systemd and dracut emergency. And this service does nothing but to isolate to real kdump error handler service:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency isolate to kdump-error-handler.service --> dracut-emergency and dracut-initqueue will both be stopped and kdump-error-handler.service will run kdump-error-handler.sh.
Chao,
So how does it now work in case of recursion.
I mean say dracut-initqueue failed. We got into kdump-error-hanlder.sh. Which in turn called dracut-initqueue again and it will fail again.
Now dracut-initqueue will call dracut-emergency and dracut-emergency will try to isolate to kdump-error-handler.service. Will it now block because kdump-error-handler is already running or it will stop existing running instance of kdump error handler.
Thanks Vivek
On 08/01/14 at 08:39am, Vivek Goyal wrote:
On Fri, Aug 01, 2014 at 01:04:54PM +0800, WANG Chao wrote:
[..]
dracut-initqueue is a special case, which calls "systemctl start emergency" directly, not via "OnFailure=emergency". In case of failure, emergency is started, but not in a isolation mode, which means dracut-initqueue is still running. On the other hand, emergency will call dracut-initqueue again when default action is dump_to_rootfs. systemd would block on the last dracut-initqueue, waiting for the first instance to exit, which leaves us hang. It looks like the following:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency (running) --> kdump-error-handler.sh (running) --> call dracut-initqueue: --> blocking and waiting for the original instance to exit.
To fix this, I'd like to introduce a wrapper emergency service. This emegency service will replace both the systemd and dracut emergency. And this service does nothing but to isolate to real kdump error handler service:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency isolate to kdump-error-handler.service --> dracut-emergency and dracut-initqueue will both be stopped and kdump-error-handler.service will run kdump-error-handler.sh.
Chao,
So how does it now work in case of recursion.
I mean say dracut-initqueue failed. We got into kdump-error-hanlder.sh. Which in turn called dracut-initqueue again and it will fail again.
Now dracut-initqueue will call dracut-emergency and dracut-emergency will try to isolate to kdump-error-handler.service. Will it now block because kdump-error-handler is already running or it will stop existing running instance of kdump error handler.
dracut-emergency will run: systemctl --no-block isolate kdump-error-handler.service
"--no-block" will means do not block, return rigth away.
So in your case, when kdump-error-handler is running and the second time dracut-initqueue fails, dracut-emergency will try to isolate to kdump-error-handler in a non-block mode and return immediately. And then the second dracut-initqueue will return and kdump-error-handler.sh will continue its code flow.
Thanks WANG Chao
On Sat, Aug 02, 2014 at 12:14:16AM +0800, WANG Chao wrote:
On 08/01/14 at 08:39am, Vivek Goyal wrote:
On Fri, Aug 01, 2014 at 01:04:54PM +0800, WANG Chao wrote:
[..]
dracut-initqueue is a special case, which calls "systemctl start emergency" directly, not via "OnFailure=emergency". In case of failure, emergency is started, but not in a isolation mode, which means dracut-initqueue is still running. On the other hand, emergency will call dracut-initqueue again when default action is dump_to_rootfs. systemd would block on the last dracut-initqueue, waiting for the first instance to exit, which leaves us hang. It looks like the following:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency (running) --> kdump-error-handler.sh (running) --> call dracut-initqueue: --> blocking and waiting for the original instance to exit.
To fix this, I'd like to introduce a wrapper emergency service. This emegency service will replace both the systemd and dracut emergency. And this service does nothing but to isolate to real kdump error handler service:
dracut-initqueue (running) --> call dracut-emergency: --> dracut-emergency isolate to kdump-error-handler.service --> dracut-emergency and dracut-initqueue will both be stopped and kdump-error-handler.service will run kdump-error-handler.sh.
Chao,
So how does it now work in case of recursion.
I mean say dracut-initqueue failed. We got into kdump-error-hanlder.sh. Which in turn called dracut-initqueue again and it will fail again.
Now dracut-initqueue will call dracut-emergency and dracut-emergency will try to isolate to kdump-error-handler.service. Will it now block because kdump-error-handler is already running or it will stop existing running instance of kdump error handler.
dracut-emergency will run: systemctl --no-block isolate kdump-error-handler.service
"--no-block" will means do not block, return rigth away.
So in your case, when kdump-error-handler is running and the second time dracut-initqueue fails, dracut-emergency will try to isolate to kdump-error-handler in a non-block mode and return immediately. And then the second dracut-initqueue will return and kdump-error-handler.sh will continue its code flow.
Ok, thanks for the explanation. This kdump error handling with dracut and systemd is turning out to be very complicated.
Thanks Vivek
This patch introduce a new kdump-capture.service which is used to run kdump.sh.
kdump-capture.service has OnFailure=emergency.target and OnFailureIsolate=yes set. When kdump.sh fails, the kdump emergency service will be triggered and enter the error handling path.
In 2nd kernel, the default target for systemd is initrd.target, so we put kdump-capture.service in initrd.target.wants/ and by that, system will start kdump-capture as part of the boot process.
kdump.sh used to run in dracut-pre-pivot hook. Now kdump-capture.service is placed after dracut-pre-pivot.service and other dependencies are all copied from dracut-pre-pivot.service. So the start point of kdump.sh will be almost the same as it used to be.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-kdump-capture.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump.sh | 5 ++--- dracut-module-setup.sh | 4 +++- kexec-tools.spec | 2 ++ 4 files changed, 37 insertions(+), 4 deletions(-) create mode 100644 dracut-kdump-capture.service
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service new file mode 100644 index 0000000..57139c9 --- /dev/null +++ b/dracut-kdump-capture.service @@ -0,0 +1,30 @@ +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +[Unit] +Description=Kdump Vmcore Save Service +After=initrd.target initrd-parse-etc.service sysroot.mount +After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service +Before=initrd-cleanup.service +ConditionPathExists=/etc/initrd-release +OnFailure=emergency.target +OnFailureIsolate=yes + +[Service] +Environment=DRACUT_SYSTEMD=1 +Environment=NEWROOT=/sysroot +Type=oneshot +ExecStart=/bin/kdump.sh +StandardInput=null +StandardOutput=syslog +StandardError=syslog+console +KillMode=process +RemainAfterExit=yes + +# Bash ignores SIGTERM, so we send SIGHUP instead, to ensure that bash +# terminates cleanly. +KillSignal=SIGHUP diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 83c1e96..600e84e 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -179,8 +179,7 @@ fi get_host_ip if [ $? -ne 0 ]; then echo "kdump: get_host_ip exited with non-zero status!" - do_default_action - do_final_action + exit 1 fi
if [ -z "$DUMP_INSTRUCTION" ]; then @@ -202,7 +201,7 @@ if [ $? -ne 0 ]; then fi
if [ $DUMP_RETVAL -ne 0 ]; then - do_default_action + exit 1 fi
do_final_action diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 9e73f53..365d8f6 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -577,9 +577,11 @@ install() { inst "/bin/cut" "/bin/cut" inst "/sbin/makedumpfile" "/sbin/makedumpfile" inst "/sbin/vmcore-dmesg" "/sbin/vmcore-dmesg" - inst_hook pre-pivot 9999 "$moddir/kdump.sh" inst "/lib/kdump/kdump-lib.sh" "/lib/kdump-lib.sh" inst "/lib/kdump/kdump-lib-initramfs.sh" "/lib/kdump-lib-initramfs.sh" + inst "$moddir/kdump.sh" "/usr/bin/kdump.sh" + inst "$moddir/kdump-capture.service" "$systemdsystemunitdir/kdump-capture.service" + ln_r "$systemdsystemunitdir/kdump-capture.service" "$systemdsystemunitdir/initrd.target.wants/kdump-capture.service" inst "$moddir/kdump-error-handler.sh" "/usr/bin/kdump-error-handler.sh" inst "$moddir/kdump-error-handler.service" "$systemdsystemunitdir/kdump-error-handler.service" # Replace existing emergency service diff --git a/kexec-tools.spec b/kexec-tools.spec index 7d64852..e3774e2 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -39,6 +39,7 @@ Source102: dracut-monitor_dd_progress Source103: dracut-kdump-error-handler.sh Source104: dracut-kdump-emergency.service Source105: dracut-kdump-error-handler.service +Source106: dracut-kdump-capture.service
Requires(post): systemd-units Requires(preun): systemd-units @@ -216,6 +217,7 @@ cp %{SOURCE102} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpb cp %{SOURCE103} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE103}} cp %{SOURCE104} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE104}} cp %{SOURCE105} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE105}} +cp %{SOURCE106} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE106}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}}
On Fri, Aug 01, 2014 at 01:04:53PM +0800, WANG Chao wrote:
Hi, All
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Hi Chao,
I think this series is in a reasonably good shape. It is time to take this in and see how well does it work. I guess that's the only way to sort out rest of the issues, if there are any.
Thanks Chao for this work.
Acked-by: Vivek Goyal vgoyal@redhat.com
Thanks Vivek
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
To solve this problem, we need to separate the error handling code from dracut-pre-pivot hook, and every time when a failure shows up, the separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into shell every time upon a critical failure. It's very convenient for us to re-use the framework of systemd emergency, because we don't have to touch the other parts of systemd. We can use our own script instead of the default one.
This new scheme will overwrite emergency shell and replace with kdump error handling code. And this code will do the error handling as needed. Now, we will not rely on dracut-pre-pivot hook running always. Instead whenever error happens and it is serious enough that emergency shell needed to run, now kdump error handler will run.
This patchset introduce a new kdump emergency service. It will override the existing emergency.service. When fatal error occurs, this emergency service will be triggered and systemd will isolate to emergency path.
This kdump emergency service is a wrapper service and it will isolate to kdump-error-handler service. kdump-error-handler can will read kdump.conf and act according to the configired "default action" (reboot/poweroff/halt/shell/dump_to_rootfs).
Along with this patchset, kdump-capture.service is introduced as a service unit to run kdump.sh. When kdump-capture.service fails, systemd will isolate to kdump emergency service. I copied all the dependencies from dracut-pre-pivot.service to kdump-capture.service so that kdump.sh will be called at the correct time window.
v2: Address several comments from Vivek:
- split [patch 1] into 1/6 and 2/6
- remove several unnecessary lines in service unit.
- modify the description of kdump-capture.service
- introduce another kdump lib kdump-lib-2.sh used in 2nd kernel.
V3: Address several comments from Vivek:
- update description of [PATCH 1/5] and [PATCH 5/5]
- separate the patch of cleanup to kdump-lib-initramfs.sh
V4: Address several comments from Vivek and Dave:
- meld the original [PATCH 3/5] into [PATCH 1/5]
- add GPL license for kdump-error-handler.service and kdump-capture.service
- separate the "x-initrd.mount" and "mount under /sysroot" patch to another patchset
V5:
- Rebase on top of the fadump patchset.
- Enable dracut-emergency again, but override it with kdump error handler.
V6:
- reboot if entering error handler in error handling path
- add comments in dracut-kdump-emergency.service
- update changelog
V7:
- add wrapper emergency.service to isolate to kdump-error-handler service
- remove extra code introduced in v6 for avoiding entering error path again
WANG Chao (2): Introduce kdump error handling service Introduce kdump capture service
dracut-kdump-capture.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump-emergency.service | 27 +++++++++++++++++++++++++++ dracut-kdump-error-handler.service | 34 ++++++++++++++++++++++++++++++++++ dracut-kdump-error-handler.sh | 10 ++++++++++ dracut-kdump.sh | 9 ++------- dracut-module-setup.sh | 11 +++++++++-- kdump-lib-initramfs.sh | 34 ++++++++++++++++++++++++++++++---- kexec-tools.spec | 9 ++++++++- 8 files changed, 150 insertions(+), 14 deletions(-) create mode 100644 dracut-kdump-capture.service create mode 100644 dracut-kdump-emergency.service create mode 100644 dracut-kdump-error-handler.service create mode 100755 dracut-kdump-error-handler.sh
-- 1.9.3
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
-
Vivek Goyal -
WANG Chao