Block driver detection now doesn't work for raw targets, this will fix it. And watchdog driver detection is broken after we enabled the squash module, this also fixed that problem.
The driver detection code are coupled together so fix them together in this series.
Patch 7/7 will include watchdog-modules by default, but dracut haven't make a release containing this module yet, so the last patch may be merged later.
--
Update from V1: - Fix a few bugs in the patch - Set watchdog's pretimeout to 0 in second kernel
Kairui Song (7): kdumpctl: split the driver detection from fs dection function Remove a redundant nfs check Add a helper for detecting watchdog drivers Fix the watchdog drivers detection code kdump-lib.sh: Use a more generic helper to detect omitted dracut module Set watchdog's pretimeout to zero in kdump kernel Always include watchdog-modules
dracut-module-setup.sh | 1 + kdump-lib.sh | 68 +++++++++++----- kdumpctl | 178 ++++++++++++++--------------------------- 3 files changed, 107 insertions(+), 140 deletions(-)
The driver detection have nothing to do with fs detection, and currently if the dump target is raw, the block driver detection is skipped which is wrong. Just split it out and run the block driver detection when dump target is fs or raw.
Also simplfied the code a bit.
Signed-off-by: Kairui Song kasong@redhat.com --- kdumpctl | 105 ++++++++++++++++++++++++++++--------------------------- 1 file changed, 53 insertions(+), 52 deletions(-)
diff --git a/kdumpctl b/kdumpctl index 94e4f5a..da8069a 100755 --- a/kdumpctl +++ b/kdumpctl @@ -430,14 +430,51 @@ check_files_modified() return 0 }
-check_dump_fs_modified() +check_drivers_modified() +{ + local _target _new_drivers _old_drivers _module_name _module_filename + + # If it's dump target is on block device, detect the block driver + _target=$(get_block_dump_target) + if [[ -n "$_target" ]]; then + _record_block_drivers() { + local _drivers + _drivers=$(udevadm info -a "/dev/block/$1" | sed -n 's/\s*DRIVERS=="(\S+)"/\1/p') + for _driver in $_drivers; do + if ! [[ " $_new_drivers " == *" $_driver "* ]]; then + _new_drivers="$_new_drivers $_driver" + fi + done + + ddebug "MAJ:MIN=$1 drivers='$_drivers'" + } + check_block_and_slaves_all _record_block_drivers "$(get_maj_min "$_target")" + fi + + [ -z "$_new_drivers" ] && return 0 + _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')" + + ddebug "Modules required for dump target: '$_new_drivers'" + ddebug "Modules included in old initramfs: '$_old_drivers'" + for _driver in $_new_drivers; do + # Skip deprecated/invalid driver name or built-in module + _module_name=$(modinfo --set-version "$KDUMP_KERNELVER" -F name $_driver 2>/dev/null) + _module_filename=$(modinfo --set-version "$KDUMP_KERNELVER" -n $_driver 2>/dev/null) + if [ $? -ne 0 ] || [ -z "$_module_name" ] || [[ "$_module_filename" = *"(builtin)"* ]]; then + continue + fi + if ! [[ " $_old_drivers " == *" $_module_name "* ]]; then + dinfo "Detected change in block device driver, new loaded module: $_module_name" + return 1 + fi + done +} + +check_fs_modified() { local _old_dev _old_mntpoint _old_fstype local _new_dev _new_mntpoint _new_fstype local _target _path _dracut_args - local _target_drivers _module_name _module_filename - - local _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')"
# No need to check in case of mount target specified via "dracut_args". if is_mount_in_dracut_args; then @@ -450,56 +487,14 @@ check_dump_fs_modified() return 0 fi
- _target=$(get_user_configured_dump_disk) - - if [[ -n "$_target" ]]; then - _target=$(to_dev_name $_target) - _new_fstype=$(blkid $_target | awk -F"TYPE=" '{print $2}' | cut -d '"' -f 2) - else - _path=$(get_save_path) - _target=$(get_target_from_path $_path) - _target=$(to_dev_name $_target) - _new_fstype=$(get_fs_type_from_target $_target) - if [[ -z "$_target" || -z "$_new_fstype" ]];then - derror "Dump path $_path does not exist" - return 2 - fi + _target=$(get_block_dump_target) + _new_fstype=$(get_fs_type_from_target $_target) + if [[ -z "$_target" ]] || [[ -z "$_new_fstype" ]];then + derror "Dump target is invalid" + return 2 fi
ddebug "_target=$_target _path=$_path _new_fstype=$_new_fstype" - - _record_block_drivers() { - local _drivers - - if [[ -b /dev/block/$1 ]]; then - _drivers=$(udevadm info -a "/dev/block/$1" | sed -n 's/\s*DRIVERS=="(\S+)"/\1/p') - fi - if [[ -b $1 ]]; then - _drivers=$(udevadm info -a "$1" | sed -n 's/\s*DRIVERS=="(\S+)"/\1/p') - fi - for _driver in $_drivers; do - if ! [[ " $_target_drivers " == *" $_driver "* ]]; then - _target_drivers="$_target_drivers $_driver" - fi - done - ddebug "MAJ:MIN=$1 _drivers=$_drivers _target_drivers=$_targer_drivers" - return 1 - } - - check_block_and_slaves_all _record_block_drivers "$(get_maj_min "$_target")" - for _driver in $_target_drivers; do - # Skip deprecated/invalid driver name or built-in module - _module_name=$(modinfo --set-version "$KDUMP_KERNELVER" -F name $_driver 2>/dev/null) - _module_filename=$(modinfo --set-version "$KDUMP_KERNELVER" -n $_driver 2>/dev/null) - if [ $? -ne 0 ] || [ -z "$_module_name" ] || [[ "$_module_filename" = *"(builtin)"* ]]; then - continue - fi - if ! [[ " $_old_drivers " == *" $_module_name "* ]]; then - dinfo "Detected change in block device driver, new loaded module: $_module_name" - return 1 - fi - done - if [[ $(expr substr $_new_fstype 1 3) = "nfs" ]];then _new_dev=$_target else @@ -606,7 +601,13 @@ check_system_modified() return $ret fi
- check_dump_fs_modified + check_fs_modified + ret=$? + if [ $ret -ne 0 ]; then + return $ret + fi + + check_drivers_modified ret=$? if [ $ret -ne 0 ]; then return $ret
In check_fs_modified, is_nfs_dump_target is already called, the dump target can't be nfs. No need to check here.
Signed-off-by: Kairui Song kasong@redhat.com --- kdumpctl | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/kdumpctl b/kdumpctl index da8069a..bf779e1 100755 --- a/kdumpctl +++ b/kdumpctl @@ -495,14 +495,10 @@ check_fs_modified() fi
ddebug "_target=$_target _path=$_path _new_fstype=$_new_fstype" - if [[ $(expr substr $_new_fstype 1 3) = "nfs" ]];then - _new_dev=$_target - else - _new_dev=$(kdump_get_persistent_dev $_target) - if [ -z "$_new_dev" ]; then - derror "Get persistent device name failed" - return 2 - fi + _new_dev=$(kdump_get_persistent_dev $_target) + if [ -z "$_new_dev" ]; then + perror "Get persistent device name failed" + return 2 fi
_new_mntpoint="$(get_kdump_mntpoint_from_target $_target)"
Signed-off-by: Kairui Song kasong@redhat.com --- kdump-lib.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 748419f..43116a9 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -744,6 +744,25 @@ prepare_kdump_bootinfo() fi }
+get_watchdog_drvs() +{ + local _wdtdrvs _drv _dir + + for _dir in /sys/class/watchdog/*; do + # device/modalias will return driver of this device + [[ -f "$_dir/device/modalias" ]] || continue + _drv=$(< "$_dir/device/modalias") + _drv=$(modprobe --set-version "$KDUMP_KERNELVER" -R $_drv 2>/dev/null) + for i in $_drv; do + if ! [[ " $_wdtdrvs " == *" $i "* ]]; then + _wdtdrvs="$_wdtdrvs $i" + fi + done + done + + echo $_wdtdrvs +} + # # prepare_cmdline <commandline> <commandline remove> <commandline append> # This function performs a series of edits on the command line.
Currently the watchdog detection code is broken already, it get the list of active watchdog drivers, then check if they are set in the /etc/cmdline.d/* as preload module. But after we switched to use squash module, /etc/cmdline.d/* is not directly visible.
So just detect whether current needed driver is installed.
Signed-off-by: Kairui Song kasong@redhat.com --- kdumpctl | 67 ++++---------------------------------------------------- 1 file changed, 4 insertions(+), 63 deletions(-)
diff --git a/kdumpctl b/kdumpctl index bf779e1..3577435 100755 --- a/kdumpctl +++ b/kdumpctl @@ -451,10 +451,13 @@ check_drivers_modified() check_block_and_slaves_all _record_block_drivers "$(get_maj_min "$_target")" fi
+ # Include watchdog drivers if watchdog module is not omitted + is_wdt_mod_omitted || _new_drivers+=" $(get_watchdog_drvs)" + [ -z "$_new_drivers" ] && return 0 _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')"
- ddebug "Modules required for dump target: '$_new_drivers'" + ddebug "Modules required for kdump: '$_new_drivers'" ddebug "Modules included in old initramfs: '$_old_drivers'" for _driver in $_new_drivers; do # Skip deprecated/invalid driver name or built-in module @@ -526,62 +529,6 @@ check_fs_modified() return 1 }
-check_wdt_modified() -{ - local -A _drivers - local _alldrivers _active _wdtdrv _wdtppath _dir - local wd_old wd_new - - is_wdt_mod_omitted - [[ $? -eq 0 ]] && return 0 - [[ -d /sys/class/watchdog/ ]] || return 0 - - # Copied logic from dracut 04watchdog/module-setup.sh::installkernel() - for _dir in /sys/class/watchdog/*; do - [[ -d "$_dir" ]] || continue - [[ -f "$_dir/state" ]] || continue - _active=$(< "$_dir/state") - [[ "$_active" = "active" ]] || continue - # device/modalias will return driver of this device - _wdtdrv=$(< "$_dir/device/modalias") - # There can be more than one module represented by same - # modalias. Currently load all of them. - # TODO: Need to find a way to avoid any unwanted module - # represented by modalias - _wdtdrv=$(modprobe --set-version "$KDUMP_KERNELVER" -R $_wdtdrv 2>/dev/null) - if [[ $_wdtdrv ]]; then - for i in $_wdtdrv; do - _drivers[$i]=1 - done - fi - # however in some cases, we also need to check that if there is - # a specific driver for the parent bus/device. In such cases - # we also need to enable driver for parent bus/device. - _wdtppath=$(readlink -f "$_dir/device") - while [[ -d "$_wdtppath" ]] && [[ "$_wdtppath" != "/sys" ]]; do - _wdtppath=$(readlink -f "$_wdtppath/..") - [[ -f "$_wdtppath/modalias" ]] || continue - - _wdtdrv=$(< "$_wdtppath/modalias") - _wdtdrv=$(modprobe --set-version "$KDUMP_KERNELVER" -R $_wdtdrv 2>/dev/null) - if [[ $_wdtdrv ]]; then - for i in $_wdtdrv; do - _drivers[$i]=1 - done - fi - done - done - - # ensure that watchdog module is loaded as early as possible - _alldrivers="${!_drivers[*]}" - [[ $_alldrivers ]] && wd_new="rd.driver.pre=${_alldrivers// /,}" - wd_old=$(lsinitrd $TARGET_INITRD -f etc/cmdline.d/00-watchdog.conf) - - [[ "$wd_old" = "$wd_new" ]] && return 0 - - return 1 -} - # returns 0 if system is not modified # returns 1 if system is modified # returns 2 if system modification is invalid @@ -609,12 +556,6 @@ check_system_modified() return $ret fi
- check_wdt_modified - if [ $? -ne 0 ]; then - dinfo "Detected change in watchdog state" - return 1 - fi - return 0 }
Signed-off-by: Kairui Song kasong@redhat.com --- kdump-lib.sh | 35 ++++++++++++++--------------------- kdumpctl | 2 +- 2 files changed, 15 insertions(+), 22 deletions(-)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 43116a9..6475f52 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -455,28 +455,21 @@ get_ifcfg_filename() { echo -n "${ifcfg_file}" }
-# returns 0 when omission of watchdog module is desired in dracut_args +# returns 0 when omission of a module is desired in dracut_args # returns 1 otherwise -is_wdt_mod_omitted() { - local dracut_args - local ret=1 - - dracut_args=$(grep "^dracut_args" /etc/kdump.conf) - [[ -z $dracut_args ]] && return $ret - - eval set -- $dracut_args - while :; do - [[ -z $1 ]] && break - case $1 in - -o|--omit) - echo $2 | grep -qw "watchdog" - [[ $? == 0 ]] && ret=0 - break - esac - shift - done - - return $ret +is_dracut_mod_omitted() { + local dracut_args dracut_mod=$1 + + set -- $(grep "^dracut_args" /etc/kdump.conf) + while [ $# -gt 0 ]; do + case $1 in + -o|--omit) + [[ " ${2//[^[:alnum:]]/ } " == *" $dracut_mod "* ]] && return 0 + esac + shift + done + + return 1 }
is_wdt_active() { diff --git a/kdumpctl b/kdumpctl index 3577435..097f749 100755 --- a/kdumpctl +++ b/kdumpctl @@ -452,7 +452,7 @@ check_drivers_modified() fi
# Include watchdog drivers if watchdog module is not omitted - is_wdt_mod_omitted || _new_drivers+=" $(get_watchdog_drvs)" + is_dracut_mod_omitted watchdog || _new_drivers+=" $(get_watchdog_drvs)"
[ -z "$_new_drivers" ] && return 0 _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')"
Most watchdogs have a parameter pretimeout, if set to non-zero, it means before the watchdog really reset the system, it will try to panic the kernel first, so kdump could kick in, or, just print a panic stacktrace and then kernel should reset it self.
If we are already in kdump kernel, this is not really helpful, only increase kernel hanging chance. And it also make thing become complex as some watchdog triggers the kernel panic in NMI context, which could also hang the kernel in strange ways, and fail the watchdog it self. So just disable this parameter.
Also for hpwdt, it have another parameter kdumptimeout, which is just designed for first kernel. The default behaviour is the watchdog will simply stop working if timeouted, trigger a panic, and leave the kernel to kdump. Again, if we are already in kdump this is not helpful. So also disable that.
Signed-off-by: Kairui Song kasong@redhat.com --- kdump-lib.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
diff --git a/kdump-lib.sh b/kdump-lib.sh index 6475f52..98ff27c 100755 --- a/kdump-lib.sh +++ b/kdump-lib.sh @@ -797,5 +797,21 @@ prepare_cmdline() if [ ! -z ${id} ] ; then cmdline=$(append_cmdline "${cmdline}" disable_cpu_apicid ${id}) fi + + # If any watchdog is used, set it's pretimeout to 0. pretimeout let + # watchdog panic the kernel first, and reset the system after the + # panic. If the system is already in kdump, panic is not helpful + # and only increase the chance of watchdog failure. + for i in $(get_watchdog_drvs); do + cmdline+=" $i.pretimeout=0" + + if [[ $i == hpwdt ]]; then + # hpwdt have a special parameter kdumptimeout, is's only suppose + # to be set to non-zero in first kernel. In kdump, non-zero + # value could prevent the watchdog from resetting the system. + cmdline+=" $i.kdumptimeout=0" + fi + done + echo ${cmdline} }
Systemd have a RebootWatchdogSec option that use watchdog to prevent reboot hangs. And it can help prevent many kinds of hangs upon kdump reboot, either it's systemd bug or kernel issue.
The overhead of watchdog drivers is trivial, and dracut have this new watchdog-driver module that only load the kernel driver. So alway install this module.
Also install all driver of watchdog instead of just active watchdog, upstream dracut have changed the behaviour so follow up.
Signed-off-by: Kairui Song kasong@redhat.com --- dracut-module-setup.sh | 1 + kdumpctl | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh index 1750c6c..48aca3b 100755 --- a/dracut-module-setup.sh +++ b/dracut-module-setup.sh @@ -40,6 +40,7 @@ depends() { dwarning "Required modules to build a squashed kdump image is missing!" fi
+ add_opt_module watchdog-modules if is_wdt_active; then add_opt_module watchdog fi diff --git a/kdumpctl b/kdumpctl index 097f749..03c9050 100755 --- a/kdumpctl +++ b/kdumpctl @@ -452,7 +452,7 @@ check_drivers_modified() fi
# Include watchdog drivers if watchdog module is not omitted - is_dracut_mod_omitted watchdog || _new_drivers+=" $(get_watchdog_drvs)" + is_dracut_mod_omitted watchdog || is_dracut_mod_omitted watchdog-modules || _new_drivers+=" $(get_watchdog_drvs)"
[ -z "$_new_drivers" ] && return 0 _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/hostonly-kernel-modules.txt | tr '\n' ' ')"
Hi, Kairui
Thanks for this series.
在 2020年11月20日 15:11, Kairui Song 写道:
Block driver detection now doesn't work for raw targets, this will fix it. And watchdog driver detection is broken after we enabled the squash module, this also fixed that problem.
The driver detection code are coupled together so fix them together in this series.
Patch 7/7 will include watchdog-modules by default, but dracut haven't make a release containing this module yet, so the last patch may be merged later.
--
After applied this patch series, I tested it on a dell machine, and got the following errors:
[root@dell-per710-01 kexec-tools]# kdumpctl restart kdump: kexec: unloaded kdump kernel kdump: Stopping kdump: [OK] kdump: Detected change(s) in the following file(s): /etc/kdump.conf /usr/sbin/makedumpfile /etc/kdump/post.d/ /etc/kdump/pre.d/ kdump: Rebuilding /boot/initramfs-5.9.8-100.fc32.x86_64kdump.img kdump: Warning: There might not be enough space to save a vmcore. kdump: The size of /dev/mapper/fedora_dell--per710--01-root should be greater than 24330132 kilo bytes. kdump: mkdumprd: failed to make kdump initrd kdump: Starting kdump: [FAILED]
The mkdumprd always fails to make the kdump initrd, do you happen to run into this issue?
Thanks. Lianbo
Update from V1:
- Fix a few bugs in the patch
- Set watchdog's pretimeout to 0 in second kernel
Kairui Song (7): kdumpctl: split the driver detection from fs dection function Remove a redundant nfs check Add a helper for detecting watchdog drivers Fix the watchdog drivers detection code kdump-lib.sh: Use a more generic helper to detect omitted dracut module Set watchdog's pretimeout to zero in kdump kernel Always include watchdog-modules
dracut-module-setup.sh | 1 + kdump-lib.sh | 68 +++++++++++----- kdumpctl | 178 ++++++++++++++--------------------------- 3 files changed, 107 insertions(+), 140 deletions(-)
Hi Lianbo,
On Sat, Nov 21, 2020 at 11:46 AM lijiang lijiang@redhat.com wrote:
Hi, Kairui
Thanks for this series.
在 2020年11月20日 15:11, Kairui Song 写道:
Block driver detection now doesn't work for raw targets, this will fix it. And watchdog driver detection is broken after we enabled the squash module, this also fixed that problem.
The driver detection code are coupled together so fix them together in this series.
Patch 7/7 will include watchdog-modules by default, but dracut haven't make a release containing this module yet, so the last patch may be merged later.
--
After applied this patch series, I tested it on a dell machine, and got the following errors:
[root@dell-per710-01 kexec-tools]# kdumpctl restart kdump: kexec: unloaded kdump kernel kdump: Stopping kdump: [OK] kdump: Detected change(s) in the following file(s): /etc/kdump.conf /usr/sbin/makedumpfile /etc/kdump/post.d/ /etc/kdump/pre.d/ kdump: Rebuilding /boot/initramfs-5.9.8-100.fc32.x86_64kdump.img kdump: Warning: There might not be enough space to save a vmcore. kdump: The size of /dev/mapper/fedora_dell--per710--01-root should be greater than 24330132 kilo bytes. kdump: mkdumprd: failed to make kdump initrd kdump: Starting kdump: [FAILED]
The mkdumprd always fails to make the kdump initrd, do you happen to run into this issue?
Thanks. Lianbo
You have to drop patch 7/7 for now as it will try to load a still non-exist dracut module "watchdog-module". Dracut will include that module in next release.
Update from V1:
- Fix a few bugs in the patch
- Set watchdog's pretimeout to 0 in second kernel
Kairui Song (7): kdumpctl: split the driver detection from fs dection function Remove a redundant nfs check Add a helper for detecting watchdog drivers Fix the watchdog drivers detection code kdump-lib.sh: Use a more generic helper to detect omitted dracut module Set watchdog's pretimeout to zero in kdump kernel Always include watchdog-modules
dracut-module-setup.sh | 1 + kdump-lib.sh | 68 +++++++++++----- kdumpctl | 178 ++++++++++++++--------------------------- 3 files changed, 107 insertions(+), 140 deletions(-)
在 2020年11月22日 21:02, Kairui Song 写道:
Hi Lianbo,
On Sat, Nov 21, 2020 at 11:46 AM lijiang lijiang@redhat.com wrote:
Hi, Kairui
Thanks for this series.
在 2020年11月20日 15:11, Kairui Song 写道:
Block driver detection now doesn't work for raw targets, this will fix it. And watchdog driver detection is broken after we enabled the squash module, this also fixed that problem.
The driver detection code are coupled together so fix them together in this series.
Patch 7/7 will include watchdog-modules by default, but dracut haven't make a release containing this module yet, so the last patch may be merged later.
--
After applied this patch series, I tested it on a dell machine, and got the following errors:
[root@dell-per710-01 kexec-tools]# kdumpctl restart kdump: kexec: unloaded kdump kernel kdump: Stopping kdump: [OK] kdump: Detected change(s) in the following file(s): /etc/kdump.conf /usr/sbin/makedumpfile /etc/kdump/post.d/ /etc/kdump/pre.d/ kdump: Rebuilding /boot/initramfs-5.9.8-100.fc32.x86_64kdump.img kdump: Warning: There might not be enough space to save a vmcore. kdump: The size of /dev/mapper/fedora_dell--per710--01-root should be greater than 24330132 kilo bytes. kdump: mkdumprd: failed to make kdump initrd kdump: Starting kdump: [FAILED]
The mkdumprd always fails to make the kdump initrd, do you happen to run into this issue?
Thanks. Lianbo
You have to drop patch 7/7 for now as it will try to load a still non-exist dracut module "watchdog-module". Dracut will include that module in next release.
OK. Thanks for the explanation, Kairui.
Maybe it should be good to merge this series after the Dracut incldes the module "watchdog-module".
The v2 looks good to me. Acked-by: Lianbo Jiang lijiang@redhat.com
Update from V1:
- Fix a few bugs in the patch
- Set watchdog's pretimeout to 0 in second kernel
Kairui Song (7): kdumpctl: split the driver detection from fs dection function Remove a redundant nfs check Add a helper for detecting watchdog drivers Fix the watchdog drivers detection code kdump-lib.sh: Use a more generic helper to detect omitted dracut module Set watchdog's pretimeout to zero in kdump kernel Always include watchdog-modules
dracut-module-setup.sh | 1 + kdump-lib.sh | 68 +++++++++++----- kdumpctl | 178 ++++++++++++++--------------------------- 3 files changed, 107 insertions(+), 140 deletions(-)
-
Kairui Song -
lijiang