This patch set implements firmware-assisted dump support for kdump svice. Firmware-assisted dump support depends on existing kdump infrastructure (kdump scripts) present in userland to save dump to the disk. Though existing kdump script will work seemlessly, it still needs to modified to make it aware of presense of firmware- assisted dump feature during service start and stop. These changes are tested successfully on a power box with fedora19.
Changes from v6 to v7: 1. Using node added to device tree by firmware to identify if boot is after crash instead of using "/proc/vmcore". This makes for better error handling. 2. Declared newly introduced variables. 3. Added setup_target_initrd routine to improve readability.
Changes from v5 to v6: 1. Using common global variable TARGET_INITRD for both kdump and fadump modes 2. Reworked on determine_dump_mode routine 3. Using a temporary image for rebuilding
---
Hari Bathini (6): kdump: Modify status routine to check for firmware-assisted dump kdump: Modify kdump script to start the firmware assisted dump. kdump: Modify kdump script to stop firmware assisted dump kdump: Rebuild default initrd for firmware assisted dump kdump: Check whether or not to invoke capturing vmcore kdump: Add firmware-assisted dump howto document
dracut-kdump.sh | 5 + fadump-howto.txt | 250 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ kdumpctl | 217 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 454 insertions(+), 18 deletions(-) create mode 100644 fadump-howto.txt
This patch enables kdump script to check if firmware-assisted dump is enabled or not by reading value from '/sys/kernel/fadump_enabled'. The determine_dump_mode() routine sets dump_mode to 'fadump', if fadump is enabled. By default, dump_mode is set to 'kdump' mode.
Modify status routine to check if firmware assisted dump is registered or not by reading value from '/sys/kernel/fadump_registered' file. If it is set to '1' then return status=0 else return status=1.
0 <= Firmware assisted is enabled and running 1 <= Firmware assisted is enabled but not running
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- kdumpctl | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 3 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 215bbd3..d281abb 100755 --- a/kdumpctl +++ b/kdumpctl @@ -9,6 +9,10 @@ MKDUMPRD="/sbin/mkdumprd -f" SAVE_PATH=/var/crash SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" DUMP_TARGET="" +FADUMP_ENABLED_SYS_NODE="/sys/kernel/fadump_enabled" +FADUMP_REGISTER_SYS_NODE="/sys/kernel/fadump_registered" +#kdump shall be the default dump mode +DEFAULT_DUMP_MODE="kdump"
. /lib/kdump/kdump-lib.sh
@@ -34,6 +38,16 @@ single_instance_lock() done }
+determine_dump_mode() +{ + # Check if firmware-assisted dump is enabled + # if yes, set the dump mode as fadump + if is_fadump_capable; then + echo "Dump mode is fadump" + DEFAULT_DUMP_MODE="fadump" + fi +} + # remove_cmdline_param <kernel cmdline> <param1> [<param2>] ... [<paramN>] # Remove a list of kernel parameters from a given kernel cmdline and print the result. # For each "arg" in the removing params list, "arg" and "arg=xxx" will be removed if exists. @@ -436,6 +450,25 @@ propagate_ssh_key() fi }
+is_fadump_capable() +{ + # Check if firmware-assisted dump is enabled + # if no, fallback to kdump check + if [ -f $FADUMP_ENABLED_SYS_NODE ]; then + rc=`cat $FADUMP_ENABLED_SYS_NODE` + [ $rc -eq 1 ] && return 0 + fi + return 1 +} + +check_current_fadump_status() +{ + # Check if firmware-assisted dump has been registered. + rc=`cat $FADUMP_REGISTER_SYS_NODE` + [ $rc -eq 1 ] && return 0 + return 1 +} + check_current_kdump_status() { rc=`cat /sys/kernel/kexec_crash_loaded` @@ -446,6 +479,17 @@ check_current_kdump_status() fi }
+check_current_status() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + check_current_fadump_status + else + check_current_kdump_status + fi + + return $? +} + save_raw() { local kdump_dir @@ -608,6 +652,16 @@ check_fence_kdump_config() return 0 }
+check_dump_feasibility() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + return 0 + fi + + check_kdump_feasibility + return $? +} + start() { check_config @@ -625,13 +679,13 @@ start() return 1 fi
- check_kdump_feasibility + check_dump_feasibility if [ $? -ne 0 ]; then echo "Starting kdump: [FAILED]" return 1 fi
- check_current_kdump_status + check_current_status if [ $? == 0 ]; then echo "Kdump already running: [WARNING]" return 0 @@ -679,6 +733,9 @@ fi
main () { + # Determine if the dump mode is kdump or fadump + determine_dump_mode + case "$1" in start) if [ -s /proc/vmcore ]; then @@ -693,7 +750,7 @@ main () ;; status) EXIT_CODE=0 - check_current_kdump_status + check_current_status case "$?" in 0) echo "Kdump is operational"
During service kdump start, if firmware assisted dump is not enabled then fallback to starting of existing kexec based kdump. If firmware assisted is enabled but not running, then start firmware assisted dump by echo'ing 1 to '/sys/kernel/fadump_registered' file.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- kdumpctl | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index d281abb..199d2a9 100755 --- a/kdumpctl +++ b/kdumpctl @@ -662,6 +662,29 @@ check_dump_feasibility() return $? }
+start_fadump() +{ + echo 1 > $FADUMP_REGISTER_SYS_NODE + if ! check_current_fadump_status; then + echo "fadump: failed to register" + return 1 + fi + + echo "fadump: registered successfully" + return 0 +} + +start_dump() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + start_fadump + else + load_kdump + fi + + return $? +} + start() { check_config @@ -703,7 +726,8 @@ start() echo "Starting kdump: [FAILED]" return 1 fi - load_kdump + + start_dump if [ $? != 0 ]; then echo "Starting kdump: [FAILED]" return 1
During service kdump stop, if firmware assisted dump is enabled and running, then stop firmware assisted dump by echo'ing 0 to '/sys/kernel/fadump_registered' file.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- kdumpctl | 39 +++++++++++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 6 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 199d2a9..0cfe2b9 100755 --- a/kdumpctl +++ b/kdumpctl @@ -736,18 +736,45 @@ start() echo "Starting kdump: [OK]" }
-stop() +stop_fadump() +{ + echo 0 > $FADUMP_REGISTER_SYS_NODE + if check_current_fadump_status; then + echo "fadump: failed to unregister" + return 1 + fi + + echo "fadump: unregistered successfully" + return 0 +} + +stop_kdump() { $KEXEC -p -u 2>/dev/null - if [ $? == 0 ]; then - echo "kexec: unloaded kdump kernel" - echo "Stopping kdump: [OK]" - return 0 - else + if [ $? != 0 ]; then echo "kexec: failed to unload kdump kernel" + return 1 + fi + + echo "kexec: unloaded kdump kernel" + return 0 +} + +stop() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + stop_fadump + else + stop_kdump + fi + + if [ $? != 0 ]; then echo "Stopping kdump: [FAILED]" return 1 fi + + echo "Stopping kdump: [OK]" + return 0 }
if [ ! -f "$KDUMP_CONFIG_FILE" ]; then
The current kdump infrastructure builds a separate initrd which then gets loaded into memory by kexec-tools for use by kdump kernel. But firmware assisted dump (FADUMP) does not use kexec-based approach. After crash, firmware reboots the partition and loads grub loader like the normal booting process does. Hence in the FADUMP approach, the second kernel (after crash) will always use the default initrd (OS built). So, to support FADUMP, change is required, as in to add dump capturing steps, in this initrd.
The current kdumpctl script implementation already has the code to build initrd using mkdumprd. This patch uses the new '--rebuild' option introduced, in dracut, to incrementally build the initramfs image. Before rebuilding, we may need to probe the initrd image for fadump support, to avoid rebuilding the initrd image multiple times unnecessarily. This can be done using "lsinitrd" tool with the newly proposed '--mod' option & inspecting the presence of "kdumpbase" in the list of modules of default initrd image. We rebuild the image if only "kdumpbase" module is missing in the initrd image. Also, before rebuilding, a backup of default initrd image is taken.
Kexec-tools package in rhel7 is now enhanced to insert a out-of-tree kdump module for dracut, which is responsible for adding vmcore capture steps into initrd, if dracut is invoked with "IN_KDUMP" environment variable set to 1. mkdumprd script exports "IN_KDUMP=1" environment variable before invoking dracut to build kdump initrd. This patch relies on this current mechanism of kdump init script.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- kdumpctl | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 77 insertions(+), 8 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 0cfe2b9..ff09b9a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -9,6 +9,7 @@ MKDUMPRD="/sbin/mkdumprd -f" SAVE_PATH=/var/crash SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" DUMP_TARGET="" +TARGET_INITRD="" FADUMP_ENABLED_SYS_NODE="/sys/kernel/fadump_enabled" FADUMP_REGISTER_SYS_NODE="/sys/kernel/fadump_registered" #kdump shall be the default dump mode @@ -143,13 +144,47 @@ save_core() fi }
-rebuild_initrd() +rebuild_fadump_initrd() +{ + local target_initrd_tmp + + # backup fadump initrd for reference before replacing it + backup_initrd + + target_initrd_tmp="$TARGET_INITRD.tmp" + $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver + if [ $? != 0 ]; then + echo "mkdumprd: failed to rebuild initrd with fadump support" >&2 + return 1 + fi + + # updating fadump initrd + mv $target_initrd_tmp $TARGET_INITRD + sync + + return 0 +} + +rebuild_kdump_initrd() { - $MKDUMPRD $kdump_initrd $kdump_kver + $MKDUMPRD $TARGET_INITRD $kdump_kver if [ $? != 0 ]; then echo "mkdumprd: failed to make kdump initrd" >&2 return 1 fi + + return 0 +} + +rebuild_initrd() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + rebuild_fadump_initrd + else + rebuild_kdump_initrd + fi + + return $? }
#$1: the files to be checked with IFS=' ' @@ -174,6 +209,18 @@ check_executable() done }
+backup_initrd() +{ + local target_initrd_bak + + # Check if backup initrd is already present. + target_initrd_bak="$TARGET_INITRD.bak" + if [ ! -e $target_initrd_bak ];then + echo "Backing up $TARGET_INITRD" + cp $TARGET_INITRD $target_initrd_bak + fi +} + check_config() { local nr @@ -240,10 +287,24 @@ get_pcs_cluster_modified_files() echo $modified_files }
+setup_target_initrd() +{ + if [ $DEFAULT_DUMP_MODE == "fadump" ]; then + TARGET_INITRD="${KDUMP_BOOTDIR}/initramfs-${kdump_kver}.img" + if [ ! -s "$TARGET_INITRD" ]; then + echo "Error: No initrd found to rebuild!" + return 1 + fi + else + TARGET_INITRD="${KDUMP_BOOTDIR}/initramfs-${kdump_kver}kdump.img" + fi +} + check_rebuild() { local extra_modules modified_files="" local _force_rebuild force_rebuild="0" + local initramfs_has_fadump
if [ -z "$KDUMP_KERNELVER" ]; then kdump_kver=`uname -r` @@ -252,7 +313,10 @@ check_rebuild() fi
kdump_kernel="${KDUMP_BOOTDIR}/${KDUMP_IMG}-${kdump_kver}${KDUMP_IMG_EXT}" - kdump_initrd="${KDUMP_BOOTDIR}/initramfs-${kdump_kver}kdump.img" + setup_target_initrd + if [ $? -ne 0 ]; then + return 1 + fi
_force_rebuild=`grep ^force_rebuild $KDUMP_CONFIG_FILE 2>/dev/null` if [ $? -eq 0 ]; then @@ -269,8 +333,8 @@ check_rebuild()
#check to see if dependent files has been modified #since last build of the image file - if [ -f $kdump_initrd ]; then - image_time=`stat -c "%Y" $kdump_initrd 2>/dev/null` + if [ -f $TARGET_INITRD ]; then + image_time=`stat -c "%Y" $TARGET_INITRD 2>/dev/null` else image_time=0 fi @@ -295,10 +359,15 @@ check_rebuild() fi done
+ #check if target initrd has fadump support + initramfs_has_fadump=`lsinitrd -m $TARGET_INITRD | grep ^kdumpbase$ | wc -l` + if [ $image_time -eq 0 ]; then echo -n "No kdump initial ramdisk found."; echo + elif [ $DEFAULT_DUMP_MODE == "fadump" ] && [ "$initramfs_has_fadump" -eq "0" ]; then + echo "$TARGET_INITRD has no fadump support" elif [ "$force_rebuild" != "0" ]; then - echo -n "Force rebuild $kdump_initrd"; echo + echo -n "Force rebuild $TARGET_INITRD"; echo elif [ -n "$modified_files" ]; then echo "Detected change(s) in the following file(s):" echo -n " "; echo "$modified_files" | sed 's/\s/\n /g' @@ -306,7 +375,7 @@ check_rebuild() return 0 fi
- echo "Rebuilding $kdump_initrd" + echo "Rebuilding $TARGET_INITRD" rebuild_initrd return $? } @@ -359,7 +428,7 @@ load_kdump()
$KEXEC $KEXEC_ARGS $standard_kexec_args \ --command-line="$KDUMP_COMMANDLINE" \ - --initrd=$kdump_initrd $kdump_kernel 2>/dev/null + --initrd=$TARGET_INITRD $kdump_kernel 2>/dev/null if [ $? == 0 ]; then echo "kexec: loaded kdump kernel" return 0
The script dracut-kdump.sh is responsible for capturing vmcore during second kernel boot. Currently this script gets installed into kdump initrd as part of kdumpbase dracut module.
With fadump support, 'dracut-kdump.sh' script also gets installed into default initrd to capture vmcore generated by firmware assisted dump. Thus in fadump case, the same initrd is going to be used for normal boot as well as boot after system crash. Hence a check is required to see if it is a normal boot or boot after crash.
A new node "ibm,kernel-dump" is added, to the device tree, by firmware to notify kernel if it is booting after crash. The below patch adds a check for this node before executing steps to capture vmcore. This check will help bypassing the vmcore capture steps during normal boot process.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- dracut-kdump.sh | 5 +++++ kdumpctl | 6 +++++- 2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index cb13d92..9958317 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -1,5 +1,10 @@ #!/bin/sh
+# continue here only if we have to save dump. +if [ -f /etc/fadump.initramfs ] && [ ! -f /proc/device-tree/rtas/ibm,kernel-dump ]; then + return +fi + exec &> /dev/console . /lib/dracut-lib.sh . /lib/kdump-lib.sh diff --git a/kdumpctl b/kdumpctl index ff09b9a..98de747 100755 --- a/kdumpctl +++ b/kdumpctl @@ -151,8 +151,12 @@ rebuild_fadump_initrd() # backup fadump initrd for reference before replacing it backup_initrd
+ # this file tells the initrd is fadump enabled + touch /tmp/fadump.initramfs target_initrd_tmp="$TARGET_INITRD.tmp" - $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver + $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver \ + -i /tmp/fadump.initramfs /etc/fadump.initramfs + rm -f /tmp/fadump.initramfs if [ $? != 0 ]; then echo "mkdumprd: failed to rebuild initrd with fadump support" >&2 return 1
On Fri, Jul 25, 2014 at 12:09:22AM +0530, Hari Bathini wrote:
The script dracut-kdump.sh is responsible for capturing vmcore during second kernel boot. Currently this script gets installed into kdump initrd as part of kdumpbase dracut module.
With fadump support, 'dracut-kdump.sh' script also gets installed into default initrd to capture vmcore generated by firmware assisted dump. Thus in fadump case, the same initrd is going to be used for normal boot as well as boot after system crash. Hence a check is required to see if it is a normal boot or boot after crash.
A new node "ibm,kernel-dump" is added, to the device tree, by firmware to notify kernel if it is booting after crash. The below patch adds a check for this node before executing steps to capture vmcore. This check will help bypassing the vmcore capture steps during normal boot process.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com
dracut-kdump.sh | 5 +++++ kdumpctl | 6 +++++- 2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index cb13d92..9958317 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -1,5 +1,10 @@ #!/bin/sh
+# continue here only if we have to save dump. +if [ -f /etc/fadump.initramfs ] && [ ! -f /proc/device-tree/rtas/ibm,kernel-dump ]; then
- return
+fi
This is much better.
What about the failure case. Say saving vmcore fails. Current default is to "reboot" the system. What are you expecting in case of "fadump"?
I don't think you will like to reboot the system. You probably will want to continue the boot after giving error message.
If yes, then this area is going to need some work. Currently chao is making changes where if an error occurs, then we will be isolated to kdump error handler which will reboot the system.
You and chao will have to discuss how to cater to needs of fadump also.
Thanks Vivek
exec &> /dev/console . /lib/dracut-lib.sh . /lib/kdump-lib.sh diff --git a/kdumpctl b/kdumpctl index ff09b9a..98de747 100755 --- a/kdumpctl +++ b/kdumpctl @@ -151,8 +151,12 @@ rebuild_fadump_initrd() # backup fadump initrd for reference before replacing it backup_initrd
- # this file tells the initrd is fadump enabled
- touch /tmp/fadump.initramfs target_initrd_tmp="$TARGET_INITRD.tmp"
- $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver
- $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver \
-i /tmp/fadump.initramfs /etc/fadump.initramfs- rm -f /tmp/fadump.initramfs if [ $? != 0 ]; then echo "mkdumprd: failed to rebuild initrd with fadump support" >&2 return 1
On 07/24/14 at 02:51pm, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 12:09:22AM +0530, Hari Bathini wrote:
The script dracut-kdump.sh is responsible for capturing vmcore during second kernel boot. Currently this script gets installed into kdump initrd as part of kdumpbase dracut module.
With fadump support, 'dracut-kdump.sh' script also gets installed into default initrd to capture vmcore generated by firmware assisted dump. Thus in fadump case, the same initrd is going to be used for normal boot as well as boot after system crash. Hence a check is required to see if it is a normal boot or boot after crash.
A new node "ibm,kernel-dump" is added, to the device tree, by firmware to notify kernel if it is booting after crash. The below patch adds a check for this node before executing steps to capture vmcore. This check will help bypassing the vmcore capture steps during normal boot process.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com
dracut-kdump.sh | 5 +++++ kdumpctl | 6 +++++- 2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index cb13d92..9958317 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -1,5 +1,10 @@ #!/bin/sh
+# continue here only if we have to save dump. +if [ -f /etc/fadump.initramfs ] && [ ! -f /proc/device-tree/rtas/ibm,kernel-dump ]; then
- return
+fi
This is much better.
What about the failure case. Say saving vmcore fails. Current default is to "reboot" the system. What are you expecting in case of "fadump"?
I don't think you will like to reboot the system. You probably will want to continue the boot after giving error message.
If yes, then this area is going to need some work. Currently chao is making changes where if an error occurs, then we will be isolated to kdump error handler which will reboot the system.
Yes, we will replace the default emergency.service of systemd with our own emergency.service.
I think in case of fadump, we can simply find a way to propagate fadump flag from mkdumprd to dracut-module-setup, then we can skip the replacement and leave the default one still.
W/o the replacement of emergency.service, fadump could work as expected.
Thanks WANG Chao
You and chao will have to discuss how to cater to needs of fadump also.
Thanks Vivek
exec &> /dev/console . /lib/dracut-lib.sh . /lib/kdump-lib.sh diff --git a/kdumpctl b/kdumpctl index ff09b9a..98de747 100755 --- a/kdumpctl +++ b/kdumpctl @@ -151,8 +151,12 @@ rebuild_fadump_initrd() # backup fadump initrd for reference before replacing it backup_initrd
- # this file tells the initrd is fadump enabled
- touch /tmp/fadump.initramfs target_initrd_tmp="$TARGET_INITRD.tmp"
- $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver
- $MKDUMPRD $target_initrd_tmp --rebuild $TARGET_INITRD --kver $kdump_kver \
-i /tmp/fadump.initramfs /etc/fadump.initramfs- rm -f /tmp/fadump.initramfs if [ $? != 0 ]; then echo "mkdumprd: failed to rebuild initrd with fadump support" >&2 return 1
On Fri, Jul 25, 2014 at 02:53:49PM +0800, WANG Chao wrote:
On 07/24/14 at 02:51pm, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 12:09:22AM +0530, Hari Bathini wrote:
The script dracut-kdump.sh is responsible for capturing vmcore during second kernel boot. Currently this script gets installed into kdump initrd as part of kdumpbase dracut module.
With fadump support, 'dracut-kdump.sh' script also gets installed into default initrd to capture vmcore generated by firmware assisted dump. Thus in fadump case, the same initrd is going to be used for normal boot as well as boot after system crash. Hence a check is required to see if it is a normal boot or boot after crash.
A new node "ibm,kernel-dump" is added, to the device tree, by firmware to notify kernel if it is booting after crash. The below patch adds a check for this node before executing steps to capture vmcore. This check will help bypassing the vmcore capture steps during normal boot process.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com
dracut-kdump.sh | 5 +++++ kdumpctl | 6 +++++- 2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index cb13d92..9958317 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -1,5 +1,10 @@ #!/bin/sh
+# continue here only if we have to save dump. +if [ -f /etc/fadump.initramfs ] && [ ! -f /proc/device-tree/rtas/ibm,kernel-dump ]; then
- return
+fi
This is much better.
What about the failure case. Say saving vmcore fails. Current default is to "reboot" the system. What are you expecting in case of "fadump"?
I don't think you will like to reboot the system. You probably will want to continue the boot after giving error message.
If yes, then this area is going to need some work. Currently chao is making changes where if an error occurs, then we will be isolated to kdump error handler which will reboot the system.
Yes, we will replace the default emergency.service of systemd with our own emergency.service.
I think in case of fadump, we can simply find a way to propagate fadump flag from mkdumprd to dracut-module-setup, then we can skip the replacement and leave the default one still.
W/o the replacement of emergency.service, fadump could work as expected.
If saving vmcore failed (after kdump service was started), will that not invoke error handler. I am not sure if that will be kdump one or default one. But that might not be required in fadump case.
Secondly, right now after successful dump , we reboot the machine and fadump will not like that.
I am not sure how is it working for Hari right now. Did I miss something.
Thanks Vivek
On 07/25/2014 05:40 PM, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 02:53:49PM +0800, WANG Chao wrote:
On 07/24/14 at 02:51pm, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 12:09:22AM +0530, Hari Bathini wrote:
The script dracut-kdump.sh is responsible for capturing vmcore during second kernel boot. Currently this script gets installed into kdump initrd as part of kdumpbase dracut module.
With fadump support, 'dracut-kdump.sh' script also gets installed into default initrd to capture vmcore generated by firmware assisted dump. Thus in fadump case, the same initrd is going to be used for normal boot as well as boot after system crash. Hence a check is required to see if it is a normal boot or boot after crash.
A new node "ibm,kernel-dump" is added, to the device tree, by firmware to notify kernel if it is booting after crash. The below patch adds a check for this node before executing steps to capture vmcore. This check will help bypassing the vmcore capture steps during normal boot process.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com
dracut-kdump.sh | 5 +++++ kdumpctl | 6 +++++- 2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh index cb13d92..9958317 100755 --- a/dracut-kdump.sh +++ b/dracut-kdump.sh @@ -1,5 +1,10 @@ #!/bin/sh
+# continue here only if we have to save dump. +if [ -f /etc/fadump.initramfs ] && [ ! -f /proc/device-tree/rtas/ibm,kernel-dump ]; then
- return
+fi
This is much better.
What about the failure case. Say saving vmcore fails. Current default is to "reboot" the system. What are you expecting in case of "fadump"?
I don't think you will like to reboot the system. You probably will want to continue the boot after giving error message.
If yes, then this area is going to need some work. Currently chao is making changes where if an error occurs, then we will be isolated to kdump error handler which will reboot the system.
Yes, we will replace the default emergency.service of systemd with our own emergency.service.
I think in case of fadump, we can simply find a way to propagate fadump flag from mkdumprd to dracut-module-setup, then we can skip the replacement and leave the default one still.
W/o the replacement of emergency.service, fadump could work as expected.
If saving vmcore failed (after kdump service was started), will that not invoke error handler. I am not sure if that will be kdump one or default one. But that might not be required in fadump case.
Secondly, right now after successful dump , we reboot the machine and fadump will not like that.
I am not sure how is it working for Hari right now. Did I miss something.
Thanks Vivek
Vivek, currently after saving dump, we do reboot in fadump case too. "Fadump Operational Flow" section in fadump-howto.txt summarizes the same. In failure case, fadump also depends on existing error handling mechanism in kexec-tools package. So, as of now, all the configuration options in /etc/kdump.conf file apply to fadump as well.
Thanks Hari
On Fri, Jul 25, 2014 at 11:25:25PM +0530, Hari Bathini wrote:
[..]
Vivek, currently after saving dump, we do reboot in fadump case too. "Fadump Operational Flow" section in fadump-howto.txt summarizes the same. In failure case, fadump also depends on existing error handling mechanism in kexec-tools package. So, as of now, all the configuration options in /etc/kdump.conf file apply to fadump as well.
I thought one advantage of fadump was that one does not have to reboot the system and one can continue to boot. (As firmware has taken care of quiescing of the system).
So what's the advantage of fadump if we reboot anyway?
Thanks Vivek
On Fri, Jul 25, 2014 at 11:25:25PM +0530, Hari Bathini wrote:
[..]
Vivek, currently after saving dump, we do reboot in fadump case too. "Fadump Operational Flow" section in fadump-howto.txt summarizes the same. In failure case, fadump also depends on existing error handling mechanism in kexec-tools package. So, as of now, all the configuration options in /etc/kdump.conf file apply to fadump as well.
Documentation/firmware-assisted-dump.txt says following.
Comparing with kdump or other strategies, firmware-assisted dump offers several strong, practical advantages:
-- Unlike kdump, the system has been reset, and loaded with a fresh copy of the kernel. In particular, PCI and I/O devices have been reinitialized and are in a clean, consistent state. -- Once the dump is copied out, the memory that held the dump is immediately available to the running kernel. And therefore, unlike kdump, fadump doesn't need a 2nd reboot to get back the system to the production configuration.
So seocond advantage is no more there in this implementation. The only thing we are looking at is that IO devices have been quiesced by firmware so potentially it can offer more reliability as compared to kdump ?
Thanks Vivek
On 07/25/2014 11:38 PM, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 11:25:25PM +0530, Hari Bathini wrote:
[..]
Vivek, currently after saving dump, we do reboot in fadump case too. "Fadump Operational Flow" section in fadump-howto.txt summarizes the same. In failure case, fadump also depends on existing error handling mechanism in kexec-tools package. So, as of now, all the configuration options in /etc/kdump.conf file apply to fadump as well.
Documentation/firmware-assisted-dump.txt says following.
Comparing with kdump or other strategies, firmware-assisted dump offers several strong, practical advantages:
-- Unlike kdump, the system has been reset, and loaded with a fresh copy of the kernel. In particular, PCI and I/O devices have been reinitialized and are in a clean, consistent state. -- Once the dump is copied out, the memory that held the dump is immediately available to the running kernel. And therefore, unlike kdump, fadump doesn't need a 2nd reboot to get back the system to the production configuration.
So seocond advantage is no more there in this implementation. The only thing we are looking at is that IO devices have been quiesced by firmware so potentially it can offer more reliability as compared to kdump ?
Thanks Vivek
Vivek, both the advantages mentioned above stand true but we are yet to work on advantage two. That is something we plan to do..
Thanks Hari
This patch adds fadump howto document to kexec-tools. The document is prepared in reference to kexec-kdump-howto.txt document.
Signed-off-by: Mahesh Salgaonkar mahesh@linux.vnet.ibm.com Signed-off-by: Hari Bathini hbathini@linux.vnet.ibm.com --- fadump-howto.txt | 250 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 250 insertions(+) create mode 100644 fadump-howto.txt
diff --git a/fadump-howto.txt b/fadump-howto.txt new file mode 100644 index 0000000..e744b8c --- /dev/null +++ b/fadump-howto.txt @@ -0,0 +1,250 @@ +Firmware assisted dump (fadump) HOWTO + +Introduction + +Firmware assisted dump is a new feature in the 3.4 mainline kernel supported +only on powerpc architecture. The goal of firmware-assisted dump is to enable +the dump of a crashed system, and to do so from a fully-reset system, and to +minimize the total elapsed time until the system is back in production use. A +complete documentation on implementation can be found at +Documentation/powerpc/firmware-assisted-dump.txt in upstream linux kernel tree +from 3.4 version and above. + +Please note that the firmware-assisted dump feature is only available on Power6 +and above systems with recent firmware versions. + +Overview + +Fadump + +Fadump is a robust kernel crash dumping mechanism to get reliable kernel crash +dump with assistance from firmware. This approach does not use kexec, instead +firmware assists in booting the kdump kernel while preserving memory contents. +Unlike kdump, the system is fully reset, and loaded with a fresh copy of the +kernel. In particular, PCI and I/O devices are reinitialized and are in a +clean, consistent state. This second kernel, often called a capture kernel, +boots with very little memory and captures the dump image. + +The first kernel registers the sections of memory with the Power firmware for +dump preservation during OS initialization. These registered sections of memory +are reserved by the first kernel during early boot. When a system crashes, the +Power firmware fully resets the system, preserves all the system memory +contents, save the low memory (boot memory of size larger of 5% of system +RAM or 256MB) of RAM to the previous registered region. It will also save +system registers, and hardware PTE's. + +Fadump is supported only on ppc64 platform. The standard kernel and capture +kernel are one and the same on ppc64. + +If you're reading this document, you should already have kexec-tools +installed. If not, you install it via the following command: + + # yum install kexec-tools + +Fadump Operational Flow: + +Like kdump, fadump also exports the ELF formatted kernel crash dump through +/proc/vmcore. Hence existing kdump infrastructure can be used to capture fadump +vmcore. The idea is to keep the functionality transparent to end user. From +user perspective there is no change in the way kdump init script works. + +However, unlike kdump, fadump does not pre-load kdump kernel and initrd into +reserved memory, instead it always uses default OS initrd during second boot +after crash. Hence, for fadump, we rebuild the new kdump initrd and replace it +with default initrd. Before replacing existing default initrd we take a backup +of original default initrd for user's reference. The dracut package has been +enhanced to rebuild the default initrd with vmcore capture steps. The initrd +image is rebuilt as per the configuration in /etc/kdump.conf file. + +The control flow of fadump works as follows: +01. System panics. +02. At the crash, kernel informs power firmware that kernel has crashed. +03. Firmware takes the control and reboots the entire system preserving + only the memory (resets all other devices). +04. The reboot follows the normal booting process (non-kexec). +05. The boot loader loads the default kernel and initrd from /boot +06. The default initrd loads and runs /init +07. dracut-kdump.sh script present in fadump aware default initrd checks if + '/proc/vmcore' file exists before executing steps to capture vmcore. + (This check will help to bypass the vmcore capture steps during normal boot + process.) +09. Captures dump according to /etc/kdump.conf +10. Is dump capture successful (yes goto 12, no goto 11) +11. Perfom the default action specified in /etc/kdump.conf (Default action + is reboot, if unspecified) +12. Reboot + + +How to configure fadump: + +Again, we assume if you're reading this document, you should already have +kexec-tools installed. If not, you install it via the following command: + + # yum install kexec-tools + +To be able to do much of anything interesting in the way of debug analysis, +you'll also need to install the kernel-debuginfo package, of the same arch +as your running kernel, and the crash utility: + + # yum --enablerepo=*debuginfo install kernel-debuginfo.$(uname -m) crash + +Next up, we need to modify some boot parameters to enable firmware assisted +dump. With the help of grubby, it's very easy to append "fadump=on" to the end +of your kernel boot parameters. Optionally, user can also append +'fadump_reserve_mem=X' kernel cmdline to specify size of the memory to reserve +for boot memory dump preservation. + + # grubby --args="fadump=on" --update-kernel=/boot/vmlinuz-`uname -r` + +The term 'boot memory' means size of the low memory chunk that is required for +a kernel to boot successfully when booted with restricted memory. By default, +the boot memory size will be the larger of 5% of system RAM or 256MB. +Alternatively, user can also specify boot memory size through boot parameter +'fadump_reserve_mem=' which will override the default calculated size. Use this +option if default boot memory size is not sufficient for second kernel to boot +successfully. + +After making said changes, reboot your system, so that the specified memory is +reserved and left untouched by the normal system. Take note that the output of +'free -m' will show X MB less memory than without this parameter, which is +expected. If you see OOM (Out Of Memory) error messages while loading capture +kernel, then you should bump up the memory reservation size. + +Now that you've got that reserved memory region set up, you want to turn on +the kdump init script: + + # systemctl enable kdump.service + +Then, start up kdump as well: + + # systemctl start kdump.service + +This should turn on the firmware assisted functionality in kernel by +echo'ing 1 to /sys/kernel/fadump_registered, leaving the system ready +to capture a vmcore upon crashing. To test this out, you can force-crash +your system by echo'ing a c into /proc/sysrq-trigger: + + # echo c > /proc/sysrq-trigger + +You should see some panic output, followed by the system reset and booting into +fresh copy of kernel. When default initrd loads and runs /init, vmcore should +be copied out to disk (by default, in /var/crash/YYYY.MM.DD-HH:MM:SS/vmcore), +then the system rebooted back into your normal kernel. + +Once back to your normal kernel, you can use the previously installed crash +kernel in conjunction with the previously installed kernel-debuginfo to +perform postmortem analysis: + + # crash /usr/lib/debug/lib/modules/2.6.17-1.2621.el5/vmlinux + /var/crash/2006-08-23-15:34/vmcore + + crash> bt + +and so on... + +Saving vmcore-dmesg.txt +---------------------- +Kernel log bufferes are one of the most important information available +in vmcore. Now before saving vmcore, kernel log bufferes are extracted +from /proc/vmcore and saved into a file vmcore-dmesg.txt. After +vmcore-dmesg.txt, vmcore is saved. Destination disk and directory for +vmcore-dmesg.txt is same as vmcore. Note that kernel log buffers will +not be available if dump target is raw device. + +Dump Triggering methods: + +This section talks about the various ways, other than a Kernel Panic, in which +fadump can be triggered. The following methods assume that fadump is configured +on your system, with the scripts enabled as described in the section above. + +1) AltSysRq C + +FAdump can be triggered with the combination of the 'Alt','SysRq' and 'C' +keyboard keys. Please refer to the following link for more details: + +http://kbase.redhat.com/faq/FAQ_43_5559.shtm + +In addition, on PowerPC boxes, fadump can also be triggered via Hardware +Management Console(HMC) using 'Ctrl', 'O' and 'C' keyboard keys. + +2) Kernel OOPs + +If we want to generate a dump everytime the Kernel OOPses, we can achieve this +by setting the 'Panic On OOPs' option as follows: + + # echo 1 > /proc/sys/kernel/panic_on_oops + +3) PowerPC specific methods: + +On IBM PowerPC machines, issuing a soft reset invokes the XMON debugger(if +XMON is configured). To configure XMON one needs to compile the kernel with +the CONFIG_XMON and CONFIG_XMON_DEFAULT options, or by compiling with +CONFIG_XMON and booting the kernel with xmon=on option. + +Following are the ways to remotely issue a soft reset on PowerPC boxes, which +would drop you to XMON. Pressing a 'X' (capital alphabet X) followed by an +'Enter' here will trigger the dump. + +3.1) HMC + +Hardware Management Console(HMC) available on Power4 and Power5 machines allow +partitions to be reset remotely. This is specially useful in hang situations +where the system is not accepting any keyboard inputs. + +Once you have HMC configured, the following steps will enable you to trigger +fadump via a soft reset: + +On Power4 + Using GUI + + * In the right pane, right click on the partition you wish to dump. + * Select "Operating System->Reset". + * Select "Soft Reset". + * Select "Yes". + + Using HMC Commandline + + # reset_partition -m <machine> -p <partition> -t soft + +On Power5 + Using GUI + + * In the right pane, right click on the partition you wish to dump. + * Select "Restart Partition". + * Select "Dump". + * Select "OK". + + Using HMC Commandline + + # chsysstate -m <managed system name> -n <lpar name> -o dumprestart -r lpar + +3.2) Blade Management Console for Blade Center + +To initiate a dump operation, go to Power/Restart option under "Blade Tasks" in +the Blade Management Console. Select the corresponding blade for which you want +to initate the dump and then click "Restart blade with NMI". This issues a +system reset and invokes xmon debugger. + + +Advanced Setups & Default action: + +Kdump and fadump exhibit similar behavior in terms of setup & default action. +For fadump advanced setup related information see section "Advanced Setups" in +"kexec-kdump-howto.txt" document. Refer to "Default action" section in "kexec- +kdump-howto.txt" document for fadump default action related information. + +Compression and filtering + +Refer "Compression and filtering" section in "kexec-kdump-howto.txt" document. +Compression and filtering are same for kdump & fadump. + + +Notes on rootfs mount: +Dracut is designed to mount rootfs by default. If rootfs mounting fails it +will refuse to go on. So fadump leaves rootfs mounting to dracut currently. +We make the assumtion that proper root= cmdline is being passed to dracut +initramfs for the time being. If you need modify "KDUMP_COMMANDLINE=" in +/etc/sysconfig/kdump, you will need to make sure that appropriate root= +options are copied from /proc/cmdline. In general it is best to append +command line options using "KDUMP_COMMANDLINE_APPEND=" instead of replacing +the original command line completely.
On Fri, Jul 25, 2014 at 12:08:22AM +0530, Hari Bathini wrote:
This patch set implements firmware-assisted dump support for kdump svice. Firmware-assisted dump support depends on existing kdump infrastructure (kdump scripts) present in userland to save dump to the disk. Though existing kdump script will work seemlessly, it still needs to modified to make it aware of presense of firmware- assisted dump feature during service start and stop. These changes are tested successfully on a power box with fedora19.
Changes from v6 to v7:
- Using node added to device tree by firmware to identify if boot is after crash instead of using "/proc/vmcore". This makes for better error handling.
- Declared newly introduced variables.
- Added setup_target_initrd routine to improve readability.
Changes from v5 to v6:
- Using common global variable TARGET_INITRD for both kdump and fadump modes
- Reworked on determine_dump_mode routine
- Using a temporary image for rebuilding
I am fine with this patchset. Thanks Hari.
Acked-by: Vivek Goyal vgoyal@redhat.com
Chao, this patchset will conflict with your patchset. May be we can first commit this patchset and then rebase your patches on top of it.
Thanks Vivek
Hari Bathini (6): kdump: Modify status routine to check for firmware-assisted dump kdump: Modify kdump script to start the firmware assisted dump. kdump: Modify kdump script to stop firmware assisted dump kdump: Rebuild default initrd for firmware assisted dump kdump: Check whether or not to invoke capturing vmcore kdump: Add firmware-assisted dump howto document
dracut-kdump.sh | 5 + fadump-howto.txt | 250 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ kdumpctl | 217 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 454 insertions(+), 18 deletions(-) create mode 100644 fadump-howto.txt
--
- Hari
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
On 07/25/14 at 03:48pm, Vivek Goyal wrote:
On Fri, Jul 25, 2014 at 12:08:22AM +0530, Hari Bathini wrote:
This patch set implements firmware-assisted dump support for kdump svice. Firmware-assisted dump support depends on existing kdump infrastructure (kdump scripts) present in userland to save dump to the disk. Though existing kdump script will work seemlessly, it still needs to modified to make it aware of presense of firmware- assisted dump feature during service start and stop. These changes are tested successfully on a power box with fedora19.
Changes from v6 to v7:
- Using node added to device tree by firmware to identify if boot is after crash instead of using "/proc/vmcore". This makes for better error handling.
- Declared newly introduced variables.
- Added setup_target_initrd routine to improve readability.
Changes from v5 to v6:
- Using common global variable TARGET_INITRD for both kdump and fadump modes
- Reworked on determine_dump_mode routine
- Using a temporary image for rebuilding
I am fine with this patchset. Thanks Hari.
Acked-by: Vivek Goyal vgoyal@redhat.com
Chao, this patchset will conflict with your patchset. May be we can first commit this patchset and then rebase your patches on top of it.
Sure. I'll merge these changes.
Thanks WANG Chao
Thanks Vivek
Hari Bathini (6): kdump: Modify status routine to check for firmware-assisted dump kdump: Modify kdump script to start the firmware assisted dump. kdump: Modify kdump script to stop firmware assisted dump kdump: Rebuild default initrd for firmware assisted dump kdump: Check whether or not to invoke capturing vmcore kdump: Add firmware-assisted dump howto document
dracut-kdump.sh | 5 + fadump-howto.txt | 250 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ kdumpctl | 217 +++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 454 insertions(+), 18 deletions(-) create mode 100644 fadump-howto.txt
--
- Hari
kexec mailing list kexec@lists.fedoraproject.org https://lists.fedoraproject.org/mailman/listinfo/kexec
-
Hari Bathini -
Vivek Goyal -
WANG Chao