Hi, All
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
To solve this problem, we need to separate the error handling code from dracut-pre-pivot hook, and every time when a failure shows up, the separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into shell every time upon a critical failure. It's very convenient for us to re-use the framework of systemd emergency, because we don't have to touch the other parts of systemd. We can use our own script instead of the default one.
This new scheme will overwrite emergency shell and replace with kdump error handling code. And this code will do the error handling as needed. Now, we will not rely on dracut-pre-pivot hook running always. Instead whenever error happens and it is serious enough that emergency shell needed to run, now kdump error handler will run.
This patchset introduce a new kdump emergency service. It will override the existing emergency.service. When fatal error occurs, this emergency service will be triggered and systemd will isolate to emergency path.
This kdump emergency service can will read kdump.conf and act according to the configired "default action" (reboot/poweroff/halt/shell/dump_to_rootfs).
Along with this patchset, kdump-capture.service is introduced as a service unit to run kdump.sh. When kdump-capture.service fails, systemd will isolate to kdump emergency service. I copied all the dependencies from dracut-pre-pivot.service to kdump-capture.service so that kdump.sh will be called at the correct time window.
v2: Address several comments from Vivek: - split [patch 1] into 1/6 and 2/6 - remove several unnecessary lines in service unit. - modify the description of kdump-capture.service - introduce another kdump lib kdump-lib-2.sh used in 2nd kernel.
V3: Address several comments from Vivek: - update description of [PATCH 1/5] and [PATCH 5/5] - separate the patch of cleanup to kdump-lib-initramfs.sh
V4: Address several comments from Vivek and Dave: - meld the original [PATCH 3/5] into [PATCH 1/5] - add GPL license for kdump-error-handler.service and kdump-capture.service - separate the "x-initrd.mount" and "mount under /sysroot" patch to another patchset
V5: - Rebase on top of the fadump patchset. - Enable dracut-emergency again, but override it with kdump error handler.
WANG Chao (2): Introduce kdump error handling service Introduce kdump capture service
dracut-kdump-capture.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump-emergency.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump-error-handler.sh | 10 ++++++++++ dracut-kdump.sh | 9 ++------- dracut-module-setup.sh | 10 ++++++++-- kdump-lib-initramfs.sh | 34 ++++++++++++++++++++++++++++++---- kexec-tools.spec | 7 ++++++- 7 files changed, 116 insertions(+), 14 deletions(-) create mode 100644 dracut-kdump-capture.service create mode 100644 dracut-kdump-emergency.service create mode 100755 dracut-kdump-error-handler.sh
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
To solve this problem, we need to separate the error handling code from dracut-pre-pivot hook, and every time when a failure shows up, the separated code can be called by the emergency service.
By default systemd provides an emergency service which will drop us into shell every time upon a critical failure. It's very convenient for us to re-use the framework of systemd emergency, because we don't have to touch the other parts of systemd. We can use our own script instead of the default one.
This new scheme will overwrite emergency shell and replace with kdump error handling code. And this code will do the error handling as needed. Now, we will not rely on dracut-pre-pivot hook running always. Instead whenever error happens and it is serious enough that emergency shell needed to run, now kdump error handler will run.
dracut-emergency is also replaced by kdump error handler and it's enabled again all the way down. So all the failure (including systemd and dracut) in 2nd kernel could be captured, and trigger kdump error handler.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-kdump-emergency.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump-error-handler.sh | 10 ++++++++++ dracut-kdump.sh | 4 ---- dracut-module-setup.sh | 6 +++++- kdump-lib-initramfs.sh | 34 ++++++++++++++++++++++++++++++---- kexec-tools.spec | 5 ++++- 6 files changed, 79 insertions(+), 10 deletions(-) create mode 100644 dracut-kdump-emergency.service create mode 100755 dracut-kdump-error-handler.sh
diff --git a/dracut-kdump-emergency.service b/dracut-kdump-emergency.service new file mode 100644 index 0000000..4434b9e --- /dev/null +++ b/dracut-kdump-emergency.service @@ -0,0 +1,30 @@ +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +[Unit] +Description=Kdump Error Handler +DefaultDependencies=no +After=systemd-vconsole-setup.service +Wants=systemd-vconsole-setup.service + +[Service] +Environment=HOME=/ +Environment=DRACUT_SYSTEMD=1 +Environment=NEWROOT=/sysroot +WorkingDirectory=/ +ExecStart=/bin/kdump-error-handler.sh +ExecStopPost=-/usr/bin/systemctl --fail --no-block default +Type=oneshot +StandardInput=tty-force +StandardOutput=inherit +StandardError=inherit +KillMode=process +IgnoreSIGPIPE=no + +# Bash ignores SIGTERM, so we send SIGHUP instead, to ensure that bash +# terminates cleanly. +KillSignal=SIGHUP diff --git a/dracut-kdump-error-handler.sh b/dracut-kdump-error-handler.sh new file mode 100755 index 0000000..2f0f1d1 --- /dev/null +++ b/dracut-kdump-error-handler.sh @@ -0,0 +1,10 @@ +#!/bin/sh + +. /lib/kdump-lib-initramfs.sh + +set -o pipefail +export PATH=$PATH:$KDUMP_SCRIPT_DIR + +get_kdump_confs +do_default_action +do_final_action diff --git a/dracut-kdump.sh b/dracut-kdump.sh index a2bfa05..83c1e96 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -9,10 +9,6 @@ exec &> /dev/console . /lib/dracut-lib.sh . /lib/kdump-lib-initramfs.sh
-if [ -f "$initdir/lib/dracut/no-emergency-shell" ]; then - rm -f -- $initdir/lib/dracut/no-emergency-shell -fi - set -o pipefail DUMP_RETVAL=0
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 758420b..b9a7000 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -563,7 +563,6 @@ kdump_install_random_seed() {
install() { kdump_install_conf - >"$initdir/lib/dracut/no-emergency-shell"
if is_ssh_dump_target; then kdump_install_random_seed @@ -581,6 +580,11 @@ install() { inst_hook pre-pivot 9999 "$moddir/kdump.sh" inst "/lib/kdump/kdump-lib.sh" "/lib/kdump-lib.sh" inst "/lib/kdump/kdump-lib-initramfs.sh" "/lib/kdump-lib-initramfs.sh" + inst "$moddir/kdump-error-handler.sh" "/usr/bin/kdump-error-handler.sh" + # Replace existing emergency service + cp "$moddir/kdump-emergency.service" "$initdir/$systemdsystemunitdir/emergency.service" + # Redirect dracut-emergency to kdump error handler + ln_r "$systemdsystemunitdir/emergency.service" "$systemdsystemunitdir/dracut-emergency.service"
# Check for all the devices and if any device is iscsi, bring up iscsi # target. Ideally all this should be pushed into dracut iscsi module diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh index 9118b64..1517712 100755 --- a/kdump-lib-initramfs.sh +++ b/kdump-lib-initramfs.sh @@ -1,6 +1,5 @@ # These variables and functions are useful in 2nd kernel
-. /lib/dracut-lib.sh . /lib/kdump-lib.sh
KDUMP_PATH="/var/crash" @@ -23,6 +22,7 @@ NEWROOT="/sysroot" get_kdump_confs() { local config_opt config_val + local user_specified_cc
while read config_opt config_val; do @@ -34,6 +34,7 @@ get_kdump_confs() ;; core_collector) [ -n "$config_val" ] && CORE_COLLECTOR="$config_val" + user_specified_cc=yes ;; sshkey) if [ -f "$config_val" ]; then @@ -55,7 +56,7 @@ get_kdump_confs() default) case $config_val in shell) - DEFAULT_ACTION="_emergency_shell kdump" + DEFAULT_ACTION="kdump_emergency_shell" ;; reboot) DEFAULT_ACTION="do_umount; reboot -f" @@ -67,12 +68,19 @@ get_kdump_confs() DEFAULT_ACTION="do_umount; poweroff -f" ;; dump_to_rootfs) - DEFAULT_ACTION="dump_fs $NEWROOT" + DEFAULT_ACTION="dump_to_rootfs" ;; esac ;; esac done < $KDUMP_CONF + + if is_ssh_dump_target || is_raw_dump_target; then + if [ -z "$user_specified_cc" ]; then + CORE_COLLECTOR="$CORE_COLLECTOR -F" + fi + fi + }
# dump_fs <mount point| device> @@ -127,6 +135,24 @@ save_vmcore_dmesg_fs() { fi }
+dump_to_rootfs() +{ + + echo "Kdump: trying to bring up rootfs device" + systemctl start dracut-initqueue + echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds" + systemctl start sysroot.mount + + dump_fs $NEWROOT +} + +kdump_emergency_shell() +{ + echo "PS1="kdump:\${PWD}# "" >/etc/profile + /bin/dracut-emergency + rm -f /etc/profile +} + do_umount() { umount -Rf $NEWROOT @@ -134,7 +160,7 @@ do_umount()
do_default_action() { - wait_for_loginit + echo "Kdump: Executing default action $DEFAULT_ACTION" eval $DEFAULT_ACTION }
diff --git a/kexec-tools.spec b/kexec-tools.spec index b55a7f1..91deade 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -36,6 +36,8 @@ Source24: kdump-lib-initramfs.sh Source100: dracut-kdump.sh Source101: dracut-module-setup.sh Source102: dracut-monitor_dd_progress +Source103: dracut-kdump-error-handler.sh +Source104: dracut-kdump-emergency.service
Requires(post): systemd-units Requires(preun): systemd-units @@ -210,7 +212,8 @@ mkdir -p -m755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpba cp %{SOURCE100} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} cp %{SOURCE101} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}} cp %{SOURCE102} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE102}} - +cp %{SOURCE103} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE103}} +cp %{SOURCE104} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE104}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}}
This patch introduce a new kdump-capture.service which is used to run kdump.sh.
kdump-capture.service has OnFailure=emergency.target and OnFailureIsolate=yes set. When kdump.sh fails, the kdump emergency service will be triggered and enter the error handling path.
In 2nd kernel, the default target for systemd is initrd.target, so we put kdump-capture.service in initrd.target.wants/ and by that, system will start kdump-capture as part of the boot process.
kdump.sh used to run in dracut-pre-pivot hook. Now kdump-capture.service is placed after dracut-pre-pivot.service and other dependencies are all copied from dracut-pre-pivot.service. So the start point of kdump.sh will be almost the same as it used to be.
Signed-off-by: WANG Chao chaowang@redhat.com --- dracut-kdump-capture.service | 30 ++++++++++++++++++++++++++++++ dracut-kdump.sh | 5 ++--- dracut-module-setup.sh | 4 +++- kexec-tools.spec | 2 ++ 4 files changed, 37 insertions(+), 4 deletions(-) create mode 100644 dracut-kdump-capture.service
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service new file mode 100644 index 0000000..57139c9 --- /dev/null +++ b/dracut-kdump-capture.service @@ -0,0 +1,30 @@ +# This file is part of systemd. +# +# systemd is free software; you can redistribute it and/or modify it +# under the terms of the GNU Lesser General Public License as published by +# the Free Software Foundation; either version 2.1 of the License, or +# (at your option) any later version. + +[Unit] +Description=Kdump Vmcore Save Service +After=initrd.target initrd-parse-etc.service sysroot.mount +After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service dracut-pre-pivot.service +Before=initrd-cleanup.service +ConditionPathExists=/etc/initrd-release +OnFailure=emergency.target +OnFailureIsolate=yes + +[Service] +Environment=DRACUT_SYSTEMD=1 +Environment=NEWROOT=/sysroot +Type=oneshot +ExecStart=/bin/kdump.sh +StandardInput=null +StandardOutput=syslog +StandardError=syslog+console +KillMode=process +RemainAfterExit=yes + +# Bash ignores SIGTERM, so we send SIGHUP instead, to ensure that bash +# terminates cleanly. +KillSignal=SIGHUP diff --git a/dracut-kdump.sh b/dracut-kdump.sh index 83c1e96..600e84e 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -179,8 +179,7 @@ fi get_host_ip if [ $? -ne 0 ]; then echo "kdump: get_host_ip exited with non-zero status!" - do_default_action - do_final_action + exit 1 fi
if [ -z "$DUMP_INSTRUCTION" ]; then @@ -202,7 +201,7 @@ if [ $? -ne 0 ]; then fi
if [ $DUMP_RETVAL -ne 0 ]; then - do_default_action + exit 1 fi
do_final_action diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index b9a7000..fe51a02 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -577,9 +577,11 @@ install() { inst "/bin/cut" "/bin/cut" inst "/sbin/makedumpfile" "/sbin/makedumpfile" inst "/sbin/vmcore-dmesg" "/sbin/vmcore-dmesg" - inst_hook pre-pivot 9999 "$moddir/kdump.sh" inst "/lib/kdump/kdump-lib.sh" "/lib/kdump-lib.sh" inst "/lib/kdump/kdump-lib-initramfs.sh" "/lib/kdump-lib-initramfs.sh" + inst "$moddir/kdump.sh" "/usr/bin/kdump.sh" + inst "$moddir/kdump-capture.service" "$systemdsystemunitdir/kdump-capture.service" + ln_r "$systemdsystemunitdir/kdump-capture.service" "$systemdsystemunitdir/initrd.target.wants/kdump-capture.service" inst "$moddir/kdump-error-handler.sh" "/usr/bin/kdump-error-handler.sh" # Replace existing emergency service cp "$moddir/kdump-emergency.service" "$initdir/$systemdsystemunitdir/emergency.service" diff --git a/kexec-tools.spec b/kexec-tools.spec index 91deade..1d94e3d 100644 --- a/kexec-tools.spec +++ b/kexec-tools.spec @@ -38,6 +38,7 @@ Source101: dracut-module-setup.sh Source102: dracut-monitor_dd_progress Source103: dracut-kdump-error-handler.sh Source104: dracut-kdump-emergency.service +Source105: dracut-kdump-capture.service
Requires(post): systemd-units Requires(preun): systemd-units @@ -214,6 +215,7 @@ cp %{SOURCE101} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpb cp %{SOURCE102} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE102}} cp %{SOURCE103} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE103}} cp %{SOURCE104} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE104}} +cp %{SOURCE105} $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE105}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE100}} chmod 755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix %{SOURCE101}}
On Tue, Jul 29, 2014 at 08:57:55PM +0800, WANG Chao wrote:
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
I think it is important to show the systemd dependency graph here.
Just couple of lines.
xyz--->foo.target---->bar.service--->dracut-pre-pivot
And show what target does not reach hence mention that dracut-pre-pivot hook does not run.
[..]
+dump_to_rootfs() +{
- echo "Kdump: trying to bring up rootfs device"
- systemctl start dracut-initqueue
- echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds"
- systemctl start sysroot.mount
Will this ever try to enter emergency shell again (in case of failure?)
Thanks Vivek
On Tue, Jul 29, 2014 at 08:57:56PM +0800, WANG Chao wrote:
This patch introduce a new kdump-capture.service which is used to run kdump.sh.
kdump-capture.service has OnFailure=emergency.target and OnFailureIsolate=yes set. When kdump.sh fails, the kdump emergency service will be triggered and enter the error handling path.
In 2nd kernel, the default target for systemd is initrd.target, so we put kdump-capture.service in initrd.target.wants/ and by that, system will start kdump-capture as part of the boot process.
kdump.sh used to run in dracut-pre-pivot hook. Now kdump-capture.service is placed after dracut-pre-pivot.service and other dependencies are all copied from dracut-pre-pivot.service. So the start point of kdump.sh will be almost the same as it used to be.
I am assuming that this change does not break any of the debug options rd.memdebug or rd.break?
Thanks Vivek
On 07/29/14 at 09:43am, Vivek Goyal wrote:
On Tue, Jul 29, 2014 at 08:57:56PM +0800, WANG Chao wrote:
This patch introduce a new kdump-capture.service which is used to run kdump.sh.
kdump-capture.service has OnFailure=emergency.target and OnFailureIsolate=yes set. When kdump.sh fails, the kdump emergency service will be triggered and enter the error handling path.
In 2nd kernel, the default target for systemd is initrd.target, so we put kdump-capture.service in initrd.target.wants/ and by that, system will start kdump-capture as part of the boot process.
kdump.sh used to run in dracut-pre-pivot hook. Now kdump-capture.service is placed after dracut-pre-pivot.service and other dependencies are all copied from dracut-pre-pivot.service. So the start point of kdump.sh will be almost the same as it used to be.
I am assuming that this change does not break any of the debug options rd.memdebug or rd.break?
rd.break never worked in kdump context, since previously we disable dracut-emergency shell to get us always into kdump.sh with any interrupt.
Now if rd.break is specified, we will kick kdump error handler and do $DEFAULT_ACTION.
User couldn't use rd.debug in the past and now if they really want it they can specify "default shell" in kdump.conf. So I would say now the situation improved a little bit.
rd.memdebug on the other side works just fine.
Thanks WANG Chao
On 07/29/14 at 09:41am, Vivek Goyal wrote:
On Tue, Jul 29, 2014 at 08:57:55PM +0800, WANG Chao wrote:
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
I think it is important to show the systemd dependency graph here.
Just couple of lines.
xyz--->foo.target---->bar.service--->dracut-pre-pivot
And show what target does not reach hence mention that dracut-pre-pivot hook does not run.
[..]
+dump_to_rootfs() +{
- echo "Kdump: trying to bring up rootfs device"
- systemctl start dracut-initqueue
- echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds"
- systemctl start sysroot.mount
Will this ever try to enter emergency shell again (in case of failure?)
sysroot.mount wouldn't, only time out after 90 seconds.
But dracut-initqueue would enter kdump error handler again. I'm not sure which is the best way to deal with dump_to_rootfs case.
a). Calling dracut-initqueue in kdump error handler, would cause a loop of emergency -> dracut-initqueue -> emergency -> dracut-initqueue ... , if something's wrong within dracut-initqueue.
b). Not calling dracut-initqueue in kdump error handler, would cause the root lvm not being brought up, if kdump error handler is triggered early when dracut-initqueue hasn't run yet. In which case, root lvm isn't there thus sysroot.mount would eventually time out.
Between a) and b), I'd prefer b). Because in my experience, most of the user space errors happened during 2nd kernel boot are related to device (disk, network) setting up in dracut-initqueue or something's wrong within kdump.sh.
With b), the trade-off is if kdump error handler runs before dracut-initqueue, we can't dump_to_rootfs because the root device isn't ready. But I would say that if such kind of critical error happens so early, we can't really guarantee a reliable dumping to rootfs.
And if error happens in dracut-initqueue, but the error isn't related to root disk, we can still do dump_to_rootfs, in case of b).
Thanks WANG Chao
On Wed, Jul 30, 2014 at 05:26:57PM +0800, WANG Chao wrote:
[..]
+dump_to_rootfs() +{
- echo "Kdump: trying to bring up rootfs device"
- systemctl start dracut-initqueue
- echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds"
- systemctl start sysroot.mount
Will this ever try to enter emergency shell again (in case of failure?)
sysroot.mount wouldn't, only time out after 90 seconds.
But dracut-initqueue would enter kdump error handler again. I'm not sure which is the best way to deal with dump_to_rootfs case.
a). Calling dracut-initqueue in kdump error handler, would cause a loop of emergency -> dracut-initqueue -> emergency -> dracut-initqueue ... , if something's wrong within dracut-initqueue.
Can we drop a temporary file somewhere which indicates that we already entered the kdump error hanlder. And if another invocation happens, simply exit and do nothing?
So if dracut-initqueue fails and it tries to call kdump error handler, that instance will simply exit.
Thanks Vivek
On 07/29/14 at 09:41am, Vivek Goyal wrote:
On Tue, Jul 29, 2014 at 08:57:55PM +0800, WANG Chao wrote:
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
I think it is important to show the systemd dependency graph here.
Just couple of lines.
xyz--->foo.target---->bar.service--->dracut-pre-pivot
And show what target does not reach hence mention that dracut-pre-pivot hook does not run.
It will be even better to add the graph to the service file as comment. Maybe copy some description from patch description to service file as well.
Thanks Dave
On 07/30/14 at 09:19am, Vivek Goyal wrote:
On Wed, Jul 30, 2014 at 05:26:57PM +0800, WANG Chao wrote:
[..]
+dump_to_rootfs() +{
- echo "Kdump: trying to bring up rootfs device"
- systemctl start dracut-initqueue
- echo "Kdump: waiting for rootfs mount, will timeout after 90 seconds"
- systemctl start sysroot.mount
Will this ever try to enter emergency shell again (in case of failure?)
sysroot.mount wouldn't, only time out after 90 seconds.
But dracut-initqueue would enter kdump error handler again. I'm not sure which is the best way to deal with dump_to_rootfs case.
a). Calling dracut-initqueue in kdump error handler, would cause a loop of emergency -> dracut-initqueue -> emergency -> dracut-initqueue ... , if something's wrong within dracut-initqueue.
Can we drop a temporary file somewhere which indicates that we already entered the kdump error hanlder. And if another invocation happens, simply exit and do nothing?
So if dracut-initqueue fails and it tries to call kdump error handler, that instance will simply exit.
Yeah, right. In any case the error handler shouldn't enter twice. And if that happens, we simply reboot.
On 07/31/14 at 10:41am, Dave Young wrote:
On 07/29/14 at 09:41am, Vivek Goyal wrote:
On Tue, Jul 29, 2014 at 08:57:55PM +0800, WANG Chao wrote:
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
I think it is important to show the systemd dependency graph here.
Just couple of lines.
xyz--->foo.target---->bar.service--->dracut-pre-pivot
And show what target does not reach hence mention that dracut-pre-pivot hook does not run.
It will be even better to add the graph to the service file as comment. Maybe copy some description from patch description to service file as well.
I think Vivek meant for demostrating why dracut-pre-pivot doesn't work. Putting the cause down in the service file doesn't make sense..
On 07/31/14 at 11:31am, WANG Chao wrote:
On 07/31/14 at 10:41am, Dave Young wrote:
On 07/29/14 at 09:41am, Vivek Goyal wrote:
On Tue, Jul 29, 2014 at 08:57:55PM +0800, WANG Chao wrote:
Now upon failure kdump script might not be called at all and it might not be able to execute default action. It results in a hang.
Because we disable emergency shell and rely on kdump.sh being invoked through dracut-pre-pivot hook. But it might happen that we never call into dracut-pre-pivot hook because certain systemd targets could not reach due to failure in their dependencies. In those cases error handling code does not run and system hangs.
I think it is important to show the systemd dependency graph here.
Just couple of lines.
xyz--->foo.target---->bar.service--->dracut-pre-pivot
And show what target does not reach hence mention that dracut-pre-pivot hook does not run.
It will be even better to add the graph to the service file as comment. Maybe copy some description from patch description to service file as well.
I think Vivek meant for demostrating why dracut-pre-pivot doesn't work. Putting the cause down in the service file doesn't make sense..
Hmm, not specific to the graph. For the service itself, add some background and documentation to the service file will make it clear to anyone. So we can avoid to search in git log.
Thanks Dave
-
Dave Young -
Vivek Goyal -
WANG Chao