[PATCH] kexec-kdump-howto.txt: Add document about encrypted targets
by Kairui Song
Currently kdump is not working well with encrypted targets, add document
about this issue.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
kexec-kdump-howto.txt | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/kexec-kdump-howto.txt b/kexec-kdump-howto.txt
index fa01a3d..b4164af 100644
--- a/kexec-kdump-howto.txt
+++ b/kexec-kdump-howto.txt
@@ -684,6 +684,20 @@ a machine with a disk image which have kdump initramfs embedded, you
should rebuild the initramfs using "kdumpctl rebuild" command manually,
or else kdump may not work as expeceted.
+Notes on encrypted dump target:
+
+Currently, kdump is not working well with encrypted dump target.
+First, user have to give the password manually in capture kernel,
+so a working interactive terminal is required in the capture kernel.
+And another major issue is that an OOM problem will occur with certain
+encryption setup. For example, the default setup for LUKS2 will use a
+memory hard key derivation function to mitigate brute force attach,
+it's impossible to reduce the memory usage for mounting the encrypted
+target. In such case, you have to either reserved enough memory for
+crash kernel according, or update your encryption setup.
+It's recommanded to use a non-encrypted target (eg. remote target)
+instead.
+
Parallel Dumping Operation
==========================
Kexec allows kdump using multiple cpus. So parallel feature can accelerate
--
2.20.1
4 years, 11 months
[PATCH] kexec-kdump-howto.txt: Add document about initramfs rebiuld
by Kairui Song
Add some note about the limitation of kdumpctl's auto detect and rebuild
feature, and suggest the user to rebuild the initramfs manually on
major system change, and don't include the initramfs in disk images.
Put the note about system change in front part of the document so user
will less likely to miss it.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
kexec-kdump-howto.txt | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/kexec-kdump-howto.txt b/kexec-kdump-howto.txt
index 12a9450..fa01a3d 100644
--- a/kexec-kdump-howto.txt
+++ b/kexec-kdump-howto.txt
@@ -136,6 +136,22 @@ perform postmortem analysis:
and so on...
+Notes:
+
+When kdump starts, the kdump kernel is loaded together with the kdump
+initramfs. To save memory usage and disk space, the kdump initramfs is
+generated strictly against the system it will run on, and contains the
+minimum set of kernel modules and utilities to boot the machine to a stage
+where the dump target could be mounted.
+
+With kdump service enabled, kdumpctl will try to detect possible system
+change and rebuild the kdump initramfs if needed. But it can not guarantee
+to cover every possible case. So after a hardware change, disk migration,
+storage setup update or any similar system level changes, it's highly
+recommended to rebuild the initramfs manually with following command:
+
+ # kdumpctl rebuild
+
Saving vmcore-dmesg.txt
----------------------
Kernel log bufferes are one of the most important information available
@@ -654,6 +670,20 @@ not been written in watchdog-core framework then this option will not have
any effect and module will not be added. Please note that only systemd
watchdog daemon is supported as watchdog kick application.
+Notes for disk images:
+
+Kdump initramfs is a critical component for capturing the crash dump.
+But it's strictly generated for the machine it will run on, and have
+no generality. If you install a new machine with a previous disk image
+(eg. VMs created with disk image or snapshot), kdump could be broken
+easily due to hardware changes or disk ID changes. So it's strongly
+recommended to not include the kdump initramfs in the disk image in the
+first place, this helps to save space, and kdumpctl will build the
+initramfs automatically if it's missing. If you have already installed
+a machine with a disk image which have kdump initramfs embedded, you
+should rebuild the initramfs using "kdumpctl rebuild" command manually,
+or else kdump may not work as expeceted.
+
Parallel Dumping Operation
==========================
Kexec allows kdump using multiple cpus. So parallel feature can accelerate
--
2.20.1
4 years, 11 months
[PATCH] Get rid of duplicated strip_comments when reading config
by Kairui Song
When reading kdump configs, a single parsing should be enough and this
saves a lot of duplicated striping call which speed up the total load
speed.
Speed up about 2 second when building and 0.1 second for reload in my
tests.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
dracut-kdump.sh | 3 +--
dracut-module-setup.sh | 3 +--
kdump-lib-initramfs.sh | 3 +--
kdump-lib.sh | 7 +++++++
kdumpctl | 8 ++------
mkdumprd | 3 +--
6 files changed, 13 insertions(+), 14 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index b75c2a5..2ae1c7c 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -144,7 +144,6 @@ read_kdump_conf()
while read config_opt config_val;
do
# remove inline comments after the end of a directive.
- config_val=$(strip_comments $config_val)
case "$config_opt" in
dracut_args)
config_val=$(get_dracut_args_target "$config_val")
@@ -160,7 +159,7 @@ read_kdump_conf()
add_dump_code "dump_ssh $SSH_KEY_LOCATION $config_val"
;;
esac
- done < $KDUMP_CONF
+ done <<< "$(read_strip_comments $KDUMP_CONF)"
}
fence_kdump_notify()
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index db7cd23..2998c72 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -501,7 +501,6 @@ kdump_install_conf() {
while read _opt _val;
do
# remove inline comments after the end of a directive.
- _val=$(strip_comments $_val)
case "$_opt" in
raw)
_pdev=$(persistent_policy="by-id" kdump_get_persistent_dev $_val)
@@ -529,7 +528,7 @@ kdump_install_conf() {
dracut_install "${_val%%[[:blank:]]*}"
;;
esac
- done < /etc/kdump.conf
+ done <<< "$(read_strip_comments /etc/kdump.conf)"
default_dump_target_install_conf
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
index d6c01d1..608dc6e 100755
--- a/kdump-lib-initramfs.sh
+++ b/kdump-lib-initramfs.sh
@@ -26,7 +26,6 @@ get_kdump_confs()
while read config_opt config_val;
do
# remove inline comments after the end of a directive.
- config_val=$(strip_comments $config_val)
case "$config_opt" in
path)
KDUMP_PATH="$config_val"
@@ -84,7 +83,7 @@ get_kdump_confs()
esac
;;
esac
- done < $KDUMP_CONF
+ done <<< "$(read_strip_comments $KDUMP_CONF)"
if [ -z "$CORE_COLLECTOR" ]; then
CORE_COLLECTOR="$DEFAULT_CORE_COLLECTOR"
diff --git a/kdump-lib.sh b/kdump-lib.sh
index 3118a4b..95a8e3c 100755
--- a/kdump-lib.sh
+++ b/kdump-lib.sh
@@ -61,6 +61,13 @@ strip_comments()
echo $@ | sed -e 's/\(.*\)#.*/\1/'
}
+# Read from kdump config file stripping all comments
+read_strip_comments()
+{
+ # sed: print all non empty line and ignore any contain after a #
+ sed -n -e "s/^\([^#]\+\).*$/\1/gp" $1
+}
+
# Check if fence kdump is configured in Pacemaker cluster
is_pcs_fence_kdump()
{
diff --git a/kdumpctl b/kdumpctl
index e77877a..8edd58a 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -235,7 +235,6 @@ check_config()
;;
raw|ext2|ext3|ext4|minix|btrfs|xfs|nfs|ssh|sshkey|path|core_collector|kdump_post|kdump_pre|extra_bins|extra_modules|failure_action|default|final_action|force_rebuild|force_no_rebuild|dracut_args|fence_kdump_args|fence_kdump_nodes)
# remove inline comments after the end of a directive.
- config_val=$(strip_comments $config_val)
[ -z "$config_val" ] && {
echo "Invalid kdump config value for option $config_opt."
return 1;
@@ -250,7 +249,7 @@ check_config()
return 1;
;;
esac
- done < $KDUMP_CONFIG_FILE
+ done <<< "$(read_strip_comments $KDUMP_CONFIG_FILE)"
check_failure_action_config || return 1
check_final_action_config || return 1
@@ -695,7 +694,6 @@ check_ssh_config()
case "$config_opt" in
sshkey)
# remove inline comments after the end of a directive.
- config_val=$(strip_comments $config_val)
if [ -f "$config_val" ]; then
# canonicalize the path
SSH_KEY_LOCATION=$(/usr/bin/readlink -m $config_val)
@@ -704,17 +702,15 @@ check_ssh_config()
fi
;;
path)
- config_val=$(strip_comments $config_val)
SAVE_PATH=$config_val
;;
ssh)
- config_val=$(strip_comments $config_val)
DUMP_TARGET=$config_val
;;
*)
;;
esac
- done < $KDUMP_CONFIG_FILE
+ done <<< "$(read_strip_comments $KDUMP_CONFIG_FILE)"
#make sure they've configured kdump.conf for ssh dumps
local SSH_TARGET=`echo -n $DUMP_TARGET | sed -n '/.*@/p'`
diff --git a/mkdumprd b/mkdumprd
index a99d5f1..cf3533f 100644
--- a/mkdumprd
+++ b/mkdumprd
@@ -392,7 +392,6 @@ fi
while read config_opt config_val;
do
# remove inline comments after the end of a directive.
- config_val=$(strip_comments $config_val)
case "$config_opt" in
extra_modules)
extra_modules="$extra_modules $config_val"
@@ -446,7 +445,7 @@ do
*)
;;
esac
-done <<< "$(grep -v -e "^#" -e "^$" $conf_file)"
+done <<< "$(read_strip_comments $conf_file)"
handle_default_dump_target
--
2.20.1
4 years, 11 months
[PATCH V2] earlykdump: provide a prompt message during the rebuild of kdump initramfs.
by Lianbo Jiang
Early kdump inherits the settings of normal kdump, so any changes that
caused normal kdump rebuilding also require rebuilding the system initramfs
to make sure that the changes take effect for early kdump.
Therefore, when the early kdump is enabled, provide a prompt message during
the rebuild of kdump initramfs.
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
kdumpctl | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/kdumpctl b/kdumpctl
index 75eebacef92b..09ede5857fb2 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -549,6 +549,16 @@ check_system_modified()
return 0
}
+check_earlykdump_is_enabled()
+{
+ rc=`grep -w "rd.earlykdump" /proc/cmdline`
+ if [ -n "$rc" ]; then
+ return 1
+ fi
+
+ return 0
+}
+
check_rebuild()
{
local extra_modules
@@ -628,6 +638,10 @@ check_rebuild()
fi
echo "Rebuilding $TARGET_INITRD"
+ check_earlykdump_is_enabled
+ if [ $? -eq 1 ]; then
+ echo "Tips: If early kdump is enabled, also require rebuilding the system initramfs to ensure that the changes take effect for early kdump."
+ fi
rebuild_initrd
return $?
}
@@ -1140,6 +1154,10 @@ rebuild() {
fi
echo "Rebuilding $TARGET_INITRD"
+ check_earlykdump_is_enabled
+ if [ $? -eq 1 ]; then
+ echo "Tips: If early kdump is enabled, also require rebuilding the system initramfs to ensure that the changes take effect for early kdump."
+ fi
rebuild_initrd
return $?
}
--
2.17.1
4 years, 11 months
[PATCH] earlykdump: provide a prompt message during the rebuild of kdump initramfs.
by Lianbo Jiang
Early kdump inherits the settings of normal kdump, so any changes that
caused normal kdump rebuilding also require rebuilding the system initramfs
to make sure that the changes take effect for early kdump.
Signed-off-by: Lianbo Jiang <lijiang(a)redhat.com>
---
kdumpctl | 2 ++
1 file changed, 2 insertions(+)
diff --git a/kdumpctl b/kdumpctl
index 75eebacef92b..7c2f53cabe59 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -628,6 +628,7 @@ check_rebuild()
fi
echo "Rebuilding $TARGET_INITRD"
+ echo "Tips: If early kdump is enabled, also require rebuilding the system initramfs to ensure that the changes take effect for early kdump."
rebuild_initrd
return $?
}
@@ -1140,6 +1141,7 @@ rebuild() {
fi
echo "Rebuilding $TARGET_INITRD"
+ echo "Tips: If early kdump is enabled, also require rebuilding the system initramfs to ensure that the changes take effect for early kdump."
rebuild_initrd
return $?
}
--
2.17.1
4 years, 11 months
[PATCH v2] kdumpctl: Detect block device driver change for initramfs rebuild
by Kairui Song
Previous we rebuild the initramfs when kenrel load module list changed,
but this is not very stable as some async services may load/unload
kernel modules, and cause unnecessary initramfs rebuild.
Instead, it's better to just check if the module required to dump to
the dump target is loaded or not, and rebuild if not loaded. This
avoids most false-positives, and ensure local target change is always
covered.
Currently only local fs dump target is covered, because this check
requires the dump target to be mounted when building the initramfs,
this guarantee that the module is in the loaded kernel module list,
else we may still get some false positive.
dracut-install could be leveraged to combine the modalias list with
kernel loaded module list as a more stable module list in the initramfs,
but upstream dracut change need to be done first.
Passed test on a KVM VM, changing the storage between SATA/USB/VirtIO
will trigger initramfs rebuild and didn't notice any false-positive.
Also passed tests on my laptop and an AWS instance.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
Update from V1:
- Make it work well for most modules with older version of kmod
- Improve some output message
kdumpctl | 41 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/kdumpctl b/kdumpctl
index 3f80ba4..e78ac46 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -357,6 +357,9 @@ check_dump_fs_modified()
local _old_dev _old_mntpoint _old_fstype
local _new_dev _new_mntpoint _new_fstype
local _target _path _dracut_args
+ local _target_drivers _module_name
+
+ local _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/loaded-kernel-modules.txt | tr '\n' ' ')"
# No need to check in case of mount target specified via "dracut_args".
if is_mount_in_dracut_args; then
@@ -385,6 +388,44 @@ check_dump_fs_modified()
fi
fi
+ _record_block_drivers() {
+ local _drivers
+ if [[ -b /dev/block/$1 ]]; then
+ _drivers=$(udevadm info -a "/dev/block/$1" | sed -n 's/\s*DRIVERS=="\(\S\+\)"/\1/p')
+ fi
+ if [[ -b $1 ]]; then
+ _drivers=$(udevadm info -a "$1" | sed -n 's/\s*DRIVERS=="\(\S\+\)"/\1/p')
+ fi
+ for _driver in $_drivers; do
+ if ! [[ " $_target_drivers " == *" $_driver "* ]]; then
+ _target_drivers="$_target_drivers $_driver"
+ fi
+ done
+ return 1
+ }
+
+ check_block_and_slaves_all _record_block_drivers "$(get_maj_min "$_target")"
+ for _driver in $_target_drivers; do
+ # Target is mounted already, if module is not included by current kernel,
+ # could be a deprecated/invalid driver name or a built-in module
+ _module_name=$(modinfo --set-version "$kdump_kver" -F name $_driver 2>/dev/null)
+ if [ $? -ne 0 ]; then
+ continue
+ fi
+ # Older version of kmod util doesn't give module name, so follow Kbuild's
+ # name-fix rule and guess the name
+ if [ -z "$_module_name" ]; then
+ _module_name=$(echo "$_driver" | sed "s/\(,\|-\)/_/g")
+ if ! (grep "\b$_driver\b" /proc/modules -q); then
+ continue
+ fi
+ fi
+ if ! [[ " $_old_drivers " == *" $_module_name "* ]]; then
+ echo "Detected change in block device driver, new loaded module: $_module_name"
+ return 1
+ fi
+ done
+
if [[ $(expr substr $_new_fstype 1 3) = "nfs" ]];then
_new_dev=$_target
else
--
2.20.1
4 years, 11 months
[PATCH] kdumpctl: Detect block device driver change for initramfs rebuild
by Kairui Song
Previous we rebuild the initramfs when kenrel load module list changed,
but this is not very stable as some async services may load/unload
kernel modules, and cause unnecessary initramfs rebuild.
Instead, it's better to just check if the module required to dump to
the dump target is loaded or not, and rebuild if not loaded. This
avoids most false-positives, and ensure local target change is always
covered.
Currently only local fs dump target is covered, because this check
requires the dump target to be mounted when building the initramfs,
this guarantee that the module is in the loaded kernel module list,
else we may still get some false positive.
dracut-install could be leveraged to combine the modalias list with
kernel loaded module list as a more stable module list in the initramfs,
but upstream dracut change need to be done first.
Passed tests on a KVM VM, changing the storage between SATA/USB/VirtIO
will trigger initramfs rebuild when new module is required for target
block device, and didn't notice any false-positive on any machines.
Signed-off-by: Kairui Song <kasong(a)redhat.com>
---
kdumpctl | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/kdumpctl b/kdumpctl
index 3f80ba4..846a45b 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -357,6 +357,9 @@ check_dump_fs_modified()
local _old_dev _old_mntpoint _old_fstype
local _new_dev _new_mntpoint _new_fstype
local _target _path _dracut_args
+ local _target_drivers _module_name
+
+ local _old_drivers="$(lsinitrd $TARGET_INITRD -f /usr/lib/dracut/loaded-kernel-modules.txt | tr '\n' ' ')"
# No need to check in case of mount target specified via "dracut_args".
if is_mount_in_dracut_args; then
@@ -385,6 +388,36 @@ check_dump_fs_modified()
fi
fi
+ _record_block_drivers() {
+ local _drivers
+ if [[ -b /dev/block/$1 ]]; then
+ _drivers=$(udevadm info -a "/dev/block/$1" | sed -n 's/\s*DRIVERS=="\(\S\+\)"/\1/p')
+ fi
+ if [[ -b $1 ]]; then
+ _drivers=$(udevadm info -a "$1" | sed -n 's/\s*DRIVERS=="\(\S\+\)"/\1/p')
+ fi
+ for _driver in $_drivers; do
+ if ! [[ " $_target_drivers " == *" $_driver "* ]]; then
+ _target_drivers="$_target_drivers $_driver"
+ fi
+ done
+ return 1
+ }
+
+ check_block_and_slaves_all _record_block_drivers "$(get_maj_min "$_target")"
+ for _driver in $_target_drivers; do
+ # Target is mounted already, if module is not included by current kernel,
+ # could be a deprecated/invalid driver name
+ _module_name=$(modinfo --set-version "$kdump_kver" -F name $_driver 2>/dev/null)
+ if [ $? -ne 0 ] || [ -z "$_module_name" ]; then
+ continue
+ fi
+ if ! [[ " $_old_drivers " == *" $_module_name "* ]]; then
+ echo "Detected change in block device driver, $_module_name is not included"
+ return 1
+ fi
+ done
+
if [[ $(expr substr $_new_fstype 1 3) = "nfs" ]];then
_new_dev=$_target
else
--
2.20.1
4 years, 11 months
[PATCH 0/2] Check if extra_modules is modified rather then always rebuild
by Kairui Song
First commit make check_files_modified follows symlink, as weak-modules
could be symlink and second commit append the extra_modules to the file
list to be checked.
Update from V1:
- Update check_files_modified instead of introduce a new function
- Give proper message when invalid/builtin module name is given
- Behave properly with weak-modules
Kairui Song (2):
kdumpctl: follow symlink when checking for modified files
kdumpctl: don't always rebuild when extra_modules is set
kdumpctl | 36 +++++++++++++++++++++++++++---------
1 file changed, 27 insertions(+), 9 deletions(-)
--
2.20.1
4 years, 11 months