9:34 a.m.
Thin provision is a mechanism that you can allocate a lvm volume which has
a large virtual size for file systems but actually in a small physical
size. The physical size can be autoextended in use if thin pool reached a
threshold specified in /etc/lvm/lvm.conf.
There are 2 works should be handled when enable lvm2 thinp for kdump:
1) Check if the dump target device or directory is thinp device.
2) Monitor the thin pool and autoextend its size when it reached the threshold
during kdump.
According to my testing, the memory consumption procedure for lvm2 thinp is the thin pool
size-autoextend phase. For fedora and rhel9, the default crashkernel value is enough. But
for rhel8, the default crashkernel value 1G-4G:160M is not enough, so it should
be handled particularly.
v1 -> v2:
1) Modified the usage of lvs cmd when check if target is lvm2 thinp
device.
2) Removed the sync flag way of mounting for lvm2 thinp target
during kdump, use "sync -f vmcore" to force sync data, and handle
the error if fails.
v2 -> v3:
1) Removed "sync -f vmcore" patch out of the patch set, for it is
addressing an issue which is not specifically to lvm2 thinp support for
kdump.
v3 -> v4:
1) Removed lvm2-monitor.service, implemented the monitor service with
a loop function within a shell script instead.
2) Add lvm2 thinp support for dump_raw, for it is addressing the similar
issue as dump_fs.
3) Dave suggested me to implement the lvm2 thin support in dracut
modules instead of kexec-tools. If you are OK with the loop-function-shell-script
technical way, I will give a try to migrate it to dracut.
Tao Liu (4):
Add lvm2 thin provision dump target checker
Add monitor_lvm2_thinp_autoextend script for kdump when lvm2 thinp
enabled
Add lvm2 thinp support for dump_raw
lvm.conf should be check modified if lvm2 thinp enabled
dracut-kdump.sh | 24 +++++++++++++++-----
dracut-module-setup.sh | 11 +++++++++-
dracut-monitor_lvm2_thinp_autoextend | 33 ++++++++++++++++++++++++++++
kdump-lib-initramfs.sh | 10 +++++++++
kdump-lib.sh | 10 +++++++++
kdumpctl | 1 +
kexec-tools.spec | 2 ++
7 files changed, 85 insertions(+), 6 deletions(-)
create mode 100644 dracut-monitor_lvm2_thinp_autoextend
--
2.33.1
9:34 a.m.
New subject: [PATCH v4 1/4] Add lvm2 thin provision dump target checker
We need to check if a directory or a device is lvm2 thinp target.
First, we use get_block_dump_target() to convert dump path into
block device, then we check if the device is lvm2 thinp target by
cmd lvs.
is_lvm2_thinp_device is now located in kdump-lib-initramfs.sh, for it
will be used in 2nd kernel. is_lvm2_thinp_dump_target is located in
kdump-lib.sh, for it is only used in 1st kernel, and it has dependencies
which exist in kdump-lib.sh.
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
kdump-lib-initramfs.sh | 9 +++++++++
kdump-lib.sh | 10 ++++++++++
2 files changed, 19 insertions(+)
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
index 84e6bf7..bcf9927 100755
--- a/kdump-lib-initramfs.sh
+++ b/kdump-lib-initramfs.sh
@@ -131,3 +131,12 @@ is_fs_dump_target()
{
[ -n "$(kdump_get_conf_val "ext[234]\|xfs\|btrfs\|minix")" ]
}
+
+is_lvm2_thinp_device()
+{
+ _device_path=$1
+ _lvm2_thin_device=$(lvm lvs -S 'lv_layout=sparse && lv_layout=thin' \
+ --nosuffix --noheadings -o vg_name,lv_name "$_device_path" 2>/dev/null)
+
+ [ -n "$_lvm2_thin_device" ] && return $?
+}
\ No newline at end of file
diff --git a/kdump-lib.sh b/kdump-lib.sh
index b137c89..534b8b6 100755
--- a/kdump-lib.sh
+++ b/kdump-lib.sh
@@ -117,6 +117,16 @@ is_dump_to_rootfs()
[[ $(kdump_get_conf_val 'failure_action\|default') == dump_to_rootfs ]]
}
+is_lvm2_thinp_dump_target()
+{
+ _target=$(get_block_dump_target)
+ if [ -n "$_target" ]; then
+ is_lvm2_thinp_device "$_target"
+ else
+ return 1
+ fi
+}
+
get_failure_action_target()
{
local _target
--
2.33.1
9:34 a.m.
New subject: [PATCH v4 2/4] Add monitor_lvm2_thinp_autoextend script for kdump when lvm2 thinp enabled
If lvm2 thinp is enabled in kdump, a lvm2 thin pool monitor service
is needed for monitoring and autoextend the size of thin pool when
it reached the threshold.
Usually the monitor service is achieved by enabling lvm2-monitor.service
for systemd, which will start the monitoring with dmeventd. When threshold
reaches, dmeventd will perform the autoextension for thin pool.
However dmeventd is not designed for running in ramdisk environment as
kdump does, for it will mlockall() the whole executable with all libraries
and the memory allocations pinned in RAM, which will consume lot of
memory and unsuitable for kdump use. For example we can notice OOM kill
of dmeventd on rhel8 with crashkernel value 1G-4G:160M when autoextension
starts.
In this patch, we will use a shell script to achieve the same. It
runs in parallel background with vmcore collector and sync, and it calls
lvextend looply. If threshold reaches, autoextend it, if not, just exit.
After kdump finishes, the script will be terminated by TERM signal.
Comparing to lvm2-monitor.service/dmeventd version, this version will
extend thin pool at a fixed time interval(5s for now), it may take a
longer kdump time but without the memory shortage risk of mlockall().
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
dracut-kdump.sh | 9 ++++++++
dracut-module-setup.sh | 5 +++++
dracut-monitor_lvm2_thinp_autoextend | 33 ++++++++++++++++++++++++++++
kexec-tools.spec | 2 ++
4 files changed, 49 insertions(+)
create mode 100644 dracut-monitor_lvm2_thinp_autoextend
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index 4852c01..7ec7ebd 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -164,6 +164,12 @@ dump_fs()
mkdir -p "$_dump_fs_path" || return 1
+ _target=$(get_target_from_path "$_dump_fs_path")
+ if is_lvm2_thinp_device "$_target"; then
+ /kdumpscripts/monitor_lvm2_thinp_autoextend "$_target"
"$CORE_COLLECTOR" "${DMESG_COLLECTOR}" "cp" &
+ _mon_pid=$!
+ fi
+
save_vmcore_dmesg_fs ${DMESG_COLLECTOR} "$_dump_fs_path"
save_opalcore_fs "$_dump_fs_path"
@@ -178,12 +184,15 @@ dump_fs()
_sync_exitcode=$?
if [ $_sync_exitcode -eq 0 ]; then
mv "$_dump_fs_path/vmcore-incomplete" "$_dump_fs_path/vmcore"
+ [ -n "$_mon_pid" ] && kill $_mon_pid
dinfo "saving vmcore complete"
else
+ [ -n "$_mon_pid" ] && kill $_mon_pid
derror "sync vmcore failed, exitcode:$_sync_exitcode"
return 1
fi
else
+ [ -n "$_mon_pid" ] && kill $_mon_pid
derror "saving vmcore failed, exitcode:$_dump_exitcode"
return 1
fi
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index c319fc2..db3d91b 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -1038,7 +1038,9 @@ install() {
fi
dracut_install -o /etc/adjtime /etc/localtime
inst "$moddir/monitor_dd_progress"
"/kdumpscripts/monitor_dd_progress"
+ inst "$moddir/monitor_lvm2_thinp_autoextend"
"/kdumpscripts/monitor_lvm2_thinp_autoextend"
chmod +x "${initdir}/kdumpscripts/monitor_dd_progress"
+ chmod +x "${initdir}/kdumpscripts/monitor_lvm2_thinp_autoextend"
inst "/bin/dd" "/bin/dd"
inst "/bin/tail" "/bin/tail"
inst "/bin/date" "/bin/date"
@@ -1053,6 +1055,9 @@ install() {
inst "/usr/bin/printf" "/sbin/printf"
inst "/usr/bin/logger" "/sbin/logger"
inst "/usr/bin/chmod" "/sbin/chmod"
+ inst "/usr/bin/pidof" "/sbin/pidof"
+ inst "/usr/bin/df" "/sbin/df"
+ inst "/usr/bin/kill" "/sbin/kill"
inst "/lib/kdump/kdump-lib-initramfs.sh"
"/lib/kdump-lib-initramfs.sh"
inst "/lib/kdump/kdump-logger.sh" "/lib/kdump-logger.sh"
inst "$moddir/kdump.sh" "/usr/bin/kdump.sh"
diff --git a/dracut-monitor_lvm2_thinp_autoextend b/dracut-monitor_lvm2_thinp_autoextend
new file mode 100644
index 0000000..5d5c44f
--- /dev/null
+++ b/dracut-monitor_lvm2_thinp_autoextend
@@ -0,0 +1,33 @@
+#!/bin/sh
+
+THIN_LV=$1
+THIN_POOL=$(lvm lvs -S 'lv_layout=sparse && lv_layout=thin' \
+ --nosuffix --noheadings -o vg_name,pool_lv "$THIN_LV" | \
+ awk '{printf("%s/%s",$1,$2);}')
+shift
+COLLECTORS=()
+for collector in "$@"; do
+ COLLECTORS+=($(echo "$collector" | awk '{print $1}'))
+done
+
+while true
+do
+ # Wait when vmcore dumping or syncfs starts.
+ if [ -n "$(pidof sync ${COLLECTORS[@]})" ]; then
+ break
+ fi
+ sleep 0.5
+done
+
+while true
+do
+ # Quit when vmcore dumping and syncfs finishes.
+ if ! [ -n "$(pidof sync ${COLLECTORS[@]})" ]; then
+ break
+ fi
+
+ # Use 'monitoring=0' to override the value in lvm.conf, in case
+ # dmeventd monitoring been started after the calling.
+ lvm lvextend --use-policies --config "activation {monitoring=0}"
"$THIN_POOL"
+ sleep 5
+done
diff --git a/kexec-tools.spec b/kexec-tools.spec
index 6673000..c12f6f6 100644
--- a/kexec-tools.spec
+++ b/kexec-tools.spec
@@ -60,6 +60,7 @@ Source109: dracut-early-kdump-module-setup.sh
Source200: dracut-fadump-init-fadump.sh
Source201: dracut-fadump-module-setup.sh
+Source202: dracut-monitor_lvm2_thinp_autoextend
%ifarch ppc64 ppc64le
Requires(post): servicelog
@@ -240,6 +241,7 @@ cp %{SOURCE102}
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpb
cp %{SOURCE104}
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE104}}
cp %{SOURCE106}
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE106}}
cp %{SOURCE107}
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE107}}
+cp %{SOURCE202}
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE202}}
chmod 755
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE100}}
chmod 755
$RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99kdumpbase/%{remove_dracut_prefix
%{SOURCE101}}
mkdir -p -m755 $RPM_BUILD_ROOT/etc/kdump-adv-conf/kdump_dracut_modules/99earlykdump
--
2.33.1
9:52 a.m.
New subject: [PATCH v4 2/4] Add monitor_lvm2_thinp_autoextend script for
kdump when lvm2 thinp enabled
Now the write mode of thin pool will be: write mode -> (queue IO) mode -> write mode
->
(queue IO) mode -> write mode ... Please see the dmesg log as follows:
[ 3.878793] kdump[523]: saving vmcore
[ 4.452306] device-mapper: thin: 253:2: reached low water mark for data device: sending
event.
[ 4.484872] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO)
mode
[ 4.444225] kdump.sh[524]: Checking for memory holes : [ 0.0
%] / Checking for memory h
oles : [100.0 %] | Excluding unnecessary pages
: [100.0 %] \
[ 4.461538] kdump.sh[527]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds
the size of thin pools and the size of whole
volume group (296.00 MiB).
[ 4.491146] kdump.sh[527]: Size of logical volume vg00/thinpool_tdata changed from
12.00 MiB (3 extents) to 20.00 MiB (5 extents).^
M
[ 5.106573] device-mapper: thin: 253:2: switching pool to write mode
[ 5.115955] device-mapper: thin: 253:2: growing the data device from 192 to 320 blocks
[ 5.141481] device-mapper: thin: 253:2: reached low water mark for data device: sending
event.
[ 4.629948] kdump.sh[527]: Logical volume vg00/thinpool_tdata successfully resized.
[ 5.199211] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO)
mode
[ 9.783912] kdump.sh[524]: Copying data : [ 66.1
%] - eta: 1s
[ 9.793645] kdump.sh[552]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds
the size of thin pools and the size of whole
volume group (296.00 MiB).
[ 9.807207] kdump.sh[552]: Size of logical volume vg00/thinpool_tdata changed from
20.00 MiB (5 extents) to 32.00 MiB (8 extents).^
M
[ 10.388059] device-mapper: thin: 253:2: switching pool to write mode
[ 10.400524] device-mapper: thin: 253:2: growing the data device from 320 to 512 blocks
[ 10.431529] device-mapper: thin: 253:2: reached low water mark for data device: sending
event.
[ 9.919487] kdump.sh[552]: Logical volume vg00/thinpool_tdata successfully resized.
[ 10.494141] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO)
mode
[ 10.067391] kdump.sh[524]: Copying data : [ 96.1
%] / eta: 0sCopying data
: [100.0 %] | eta: 0sCopying data
: [100.0 %] \
eta: 0s
[ 10.087939] kdump.sh[524]: The dumpfile is saved to
/kdumproot/mnt/var/crash/127.0.0.1-2022-06-06-14:30:24//vmcore-incomplete.
[ 10.097779] kdump.sh[524]: makedumpfile Completed.
[ 15.016903] kdump.sh[580]: WARNING: Sum of all thin volume sizes (300.00 MiB) exceeds
the size of thin pools and the size of whole
volume group (296.00 MiB).
[ 15.030921] kdump.sh[580]: Size of logical volume vg00/thinpool_tdata changed from
32.00 MiB (8 extents) to 48.00 MiB (12 extents).
[ 15.591895] device-mapper: thin: 253:2: switching pool to write mode
[ 15.604532] device-mapper: thin: 253:2: growing the data device from 512 to 768 blocks
[ 15.628054] device-mapper: thin: 253:2: reached low water mark for data device: sending
event.
[ 15.124263] kdump.sh[580]: Logical volume vg00/thinpool_tdata successfully resized.
[ 15.221498] kdump[607]: saving vmcore complete
9:34 a.m.
New subject: [PATCH v4 3/4] Add lvm2 thinp support for dump_raw
dump_raw is similar to dump_fs when dumping to lvm2 thinp device.
Except "dd oflag=sync" is used as a substitute of "sync -f vmcore".
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
dracut-kdump.sh | 15 ++++++++++-----
dracut-module-setup.sh | 6 +++++-
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/dracut-kdump.sh b/dracut-kdump.sh
index 7ec7ebd..456e77d 100755
--- a/dracut-kdump.sh
+++ b/dracut-kdump.sh
@@ -389,12 +389,17 @@ dump_raw()
/kdumpscripts/monitor_dd_progress $_src_size_mb &
fi
- dinfo "saving vmcore"
- $CORE_COLLECTOR /proc/vmcore | dd of="$1" bs=$DD_BLKSIZE >>
/tmp/dd_progress_file 2>&1 || return 1
- sync
+ if is_lvm2_thinp_device "$1"; then
+ /kdumpscripts/monitor_lvm2_thinp_autoextend "$1" "$CORE_COLLECTOR"
"dd" &
+ _mon_pid=$!
+ fi
- dinfo "saving vmcore complete"
- return 0
+ dinfo "saving vmcore"
+ $CORE_COLLECTOR /proc/vmcore | dd of="$1" bs=$DD_BLKSIZE oflag=sync >>
/tmp/dd_progress_file 2>&1
+ ret=$?
+ [ -n "$_mon_pid" ] && kill $_mon_pid
+ [ $ret -eq 0 ] && dinfo "saving vmcore complete"
+ return $ret
}
# $1: ssh key file
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index db3d91b..1216c4c 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -670,7 +670,11 @@ kdump_install_conf() {
# remove inline comments after the end of a directive.
case "$_opt" in
raw)
- _pdev=$(persistent_policy="by-id" kdump_get_persistent_dev
"$_val")
+ if is_lvm2_thinp_device "$_val"; then
+ _pdev=$(kdump_get_persistent_dev "$_val")
+ else
+ _pdev=$(persistent_policy="by-id" kdump_get_persistent_dev
"$_val")
+ fi
sed -i -e "s#^${_opt}[[:space:]]\+$_val#$_opt $_pdev#"
"${initdir}/tmp/$$-kdump.conf"
;;
ext[234] | xfs | btrfs | minix)
--
2.33.1
9:34 a.m.
New subject: [PATCH v4 4/4] lvm.conf should be check modified if lvm2 thinp enabled
lvm2 relies on /etc/lvm/lvm.conf to determine its behaviour. The
important configs such as thin_pool_autoextend_threshold and
thin_pool_autoextend_percent will be used during kdump in 2nd
kernel. So if the file is modified, the initramfs should be
rebuild to include the latest.
Signed-off-by: Tao Liu <ltao(a)redhat.com>
---
kdump-lib-initramfs.sh | 1 +
kdumpctl | 1 +
2 files changed, 2 insertions(+)
diff --git a/kdump-lib-initramfs.sh b/kdump-lib-initramfs.sh
index bcf9927..80da64a 100755
--- a/kdump-lib-initramfs.sh
+++ b/kdump-lib-initramfs.sh
@@ -8,6 +8,7 @@ DEFAULT_SSHKEY="/root/.ssh/kdump_id_rsa"
KDUMP_CONFIG_FILE="/etc/kdump.conf"
FENCE_KDUMP_CONFIG_FILE="/etc/sysconfig/fence_kdump"
FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send"
+LVM_CONF="/etc/lvm/lvm.conf"
# Read kdump config in well formated style
kdump_read_conf()
diff --git a/kdumpctl b/kdumpctl
index 6188d47..b157eb8 100755
--- a/kdumpctl
+++ b/kdumpctl
@@ -383,6 +383,7 @@ check_files_modified()
# HOOKS is mandatory and need to check the modification time
files="$files $HOOKS"
+ is_lvm2_thinp_dump_target && files="$files $LVM_CONF"
check_exist "$files" && check_executable "$EXTRA_BINS" ||
return 2
for file in $files; do
--
2.33.1
688
days inactive
688
days old
5 comments
1 participants
participants (1)
-
Tao Liu