Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
Changes since v1: [1] Set default log path to '/var/log/kdump' instead of '/var/crash' [2] Add a timestamp to the debug log messages [3] Print a message to differentiate between the kdumpctl and mkdumprd log
Lianbo Jiang (2): kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file kdumpctl: add the '-d' option to enable the kexec loading debugging messages
kdumpctl | 12 +++++++++++- mkdumprd | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-)
Sometimes, debugging the kdump service failure becomes very challenging because there is no additional debugging information, which requires modification of the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
In view of this, add 'set -x' in the scripts in order to get the complete debugging information.
Signed-off-by: Lianbo Jiang lijiang@redhat.com --- kdumpctl | 10 ++++++++++ mkdumprd | 10 ++++++++++ 2 files changed, 20 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 122e065b8c69..4138f8798d5e 100755 --- a/kdumpctl +++ b/kdumpctl @@ -5,6 +5,7 @@ KDUMP_KERNELVER="" KDUMP_COMMANDLINE="" KEXEC_ARGS="" KDUMP_CONFIG_FILE="/etc/kdump.conf" +KDUMP_LOG_PATH="/var/log" MKDUMPRD="/sbin/mkdumprd -f" DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt" SAVE_PATH=/var/crash @@ -32,6 +33,15 @@ if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump fi
+if [ ! -d $KDUMP_LOG_PATH ]; then + mkdir -p $KDUMP_LOG_PATH +fi + +#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log +export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x + single_instance_lock() { local rc timeout=5 diff --git a/mkdumprd b/mkdumprd index 91ce3d5a0686..f352a026b3a7 100644 --- a/mkdumprd +++ b/mkdumprd @@ -15,6 +15,16 @@ conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" SAVE_PATH=$(get_save_path) OVERRIDE_RESETTABLE=0 +KDUMP_LOG_PATH="/var/log" + +if [ ! -d $KDUMP_LOG_PATH ]; then + mkdir -p $KDUMP_LOG_PATH +fi + +#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log +export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x
extra_modules="" dracut_args="--quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode strict -o "plymouth dash resume ifcfg earlykdump""
On Wed, Aug 12, 2020 at 2:33 PM Lianbo Jiang lijiang@redhat.com wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no additional debugging information, which requires modification of the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
In view of this, add 'set -x' in the scripts in order to get the complete debugging information.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
kdumpctl | 10 ++++++++++ mkdumprd | 10 ++++++++++ 2 files changed, 20 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 122e065b8c69..4138f8798d5e 100755 --- a/kdumpctl +++ b/kdumpctl @@ -5,6 +5,7 @@ KDUMP_KERNELVER="" KDUMP_COMMANDLINE="" KEXEC_ARGS="" KDUMP_CONFIG_FILE="/etc/kdump.conf" +KDUMP_LOG_PATH="/var/log" MKDUMPRD="/sbin/mkdumprd -f" DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt" SAVE_PATH=/var/crash @@ -32,6 +33,15 @@ if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump fi
+if [ ! -d $KDUMP_LOG_PATH ]; then
mkdir -p $KDUMP_LOG_PATH+fi
Hi Lianbo
If KDUMP_LOG_PATH is always "/var/log", I think there is no need to check if it exists.
If /var/log is deleted, the system will be so broken and unusable. Better just error out.
+#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log
If we just redirect the stderr, then some simple error message from kexec or other command will be omitted and redirect to the log file, like error message from kexec that kernel key not available or reserved memory not enough. I think that will make it harder to debug most failures.
Maybe we can use some trick here like: exec >&1 2> >(tee -a $KDUMP_LOG_PATH/kdump.log | grep -v ^+KDUMP_DEBUG >&2) export PS4='+KDUMP_DEBUG $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' set -x
This way any simple error message from kexec will be shown in terminal, and debug logs starting with KDUMP_DEBUG will be redirected to the log file and not spawn the console.
+export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x
single_instance_lock() { local rc timeout=5 diff --git a/mkdumprd b/mkdumprd index 91ce3d5a0686..f352a026b3a7 100644 --- a/mkdumprd +++ b/mkdumprd @@ -15,6 +15,16 @@ conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" SAVE_PATH=$(get_save_path) OVERRIDE_RESETTABLE=0 +KDUMP_LOG_PATH="/var/log"
+if [ ! -d $KDUMP_LOG_PATH ]; then
mkdir -p $KDUMP_LOG_PATH+fi
+#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log
Maybe it's better to use a different log file, like kdump-mkdumprd.log or just mkdumprd.log
+export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x
extra_modules="" dracut_args="--quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode strict -o "plymouth dash resume ifcfg earlykdump"" -- 2.17.1
And as DaveY mentioned in another reply, set -x may have performance issue, I can have an option to enable/disable the debug log.
Hi, Kairui
Thanks for your comment.
在 2020年08月19日 17:48, Kairui Song 写道:
On Wed, Aug 12, 2020 at 2:33 PM Lianbo Jiang lijiang@redhat.com wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no additional debugging information, which requires modification of the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
In view of this, add 'set -x' in the scripts in order to get the complete debugging information.
Signed-off-by: Lianbo Jiang lijiang@redhat.com
kdumpctl | 10 ++++++++++ mkdumprd | 10 ++++++++++ 2 files changed, 20 insertions(+)
diff --git a/kdumpctl b/kdumpctl index 122e065b8c69..4138f8798d5e 100755 --- a/kdumpctl +++ b/kdumpctl @@ -5,6 +5,7 @@ KDUMP_KERNELVER="" KDUMP_COMMANDLINE="" KEXEC_ARGS="" KDUMP_CONFIG_FILE="/etc/kdump.conf" +KDUMP_LOG_PATH="/var/log" MKDUMPRD="/sbin/mkdumprd -f" DRACUT_MODULES_FILE="/usr/lib/dracut/modules.txt" SAVE_PATH=/var/crash @@ -32,6 +33,15 @@ if [ -f /etc/sysconfig/kdump ]; then . /etc/sysconfig/kdump fi
+if [ ! -d $KDUMP_LOG_PATH ]; then
mkdir -p $KDUMP_LOG_PATH+fi
Hi Lianbo
If KDUMP_LOG_PATH is always "/var/log", I think there is no need to check if it exists.
If /var/log is deleted, the system will be so broken and unusable. Better just error out.
The log directory may be deleted mistakenly. But it doesn't cause the system failure because it will recreate the log directory later.
+#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log
If we just redirect the stderr, then some simple error message from kexec or other command will be omitted and redirect to the log file, like error message from kexec that kernel key not available or reserved memory not enough. I think that will make it harder to debug most failures.
Maybe we can use some trick here like: exec >&1 2> >(tee -a $KDUMP_LOG_PATH/kdump.log | grep -v ^+KDUMP_DEBUG >&2) export PS4='+KDUMP_DEBUG $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' set -x
This way any simple error message from kexec will be shown in terminal, and debug logs starting with KDUMP_DEBUG will be redirected to the log file and not spawn the console.
Good idea. Thank you, Kairui.
+export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x
single_instance_lock() { local rc timeout=5 diff --git a/mkdumprd b/mkdumprd index 91ce3d5a0686..f352a026b3a7 100644 --- a/mkdumprd +++ b/mkdumprd @@ -15,6 +15,16 @@ conf_file="/etc/kdump.conf" SSH_KEY_LOCATION="/root/.ssh/kdump_id_rsa" SAVE_PATH=$(get_save_path) OVERRIDE_RESETTABLE=0 +KDUMP_LOG_PATH="/var/log"
+if [ ! -d $KDUMP_LOG_PATH ]; then
mkdir -p $KDUMP_LOG_PATH+fi
+#redirect stderr and apend debug messages to a file +exec 2>> $KDUMP_LOG_PATH/kdump.log
Maybe it's better to use a different log file, like kdump-mkdumprd.log or just mkdumprd.log
The kdumpctl script will call the mkdumprd or some functions in dracut, Seems it's not intuitive to read these message from a different log file.
Thanks. Lianbo
+export PS4='+ $(date "+%Y-%m-%d %H:%M:%S") ${BASH_SOURCE}@${LINENO}: ' +set -x
extra_modules="" dracut_args="--quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode strict -o "plymouth dash resume ifcfg earlykdump"" -- 2.17.1
And as DaveY mentioned in another reply, set -x may have performance issue, I can have an option to enable/disable the debug log.
Currently, the kexec option '--debug/-d' is not enabled by default, which means that users need to set it manually and wait for the next failure to capture the additional information.
Therefore, let's enable the option '-d' for kexec loading by default.
Signed-off-by: Lianbo Jiang lijiang@redhat.com --- kdumpctl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kdumpctl b/kdumpctl index 4138f8798d5e..268bd6bd2419 100755 --- a/kdumpctl +++ b/kdumpctl @@ -24,7 +24,7 @@ image_time=0 . $dracutbasedir/dracut-functions.sh . /lib/kdump/kdump-lib.sh
-standard_kexec_args="-p" +standard_kexec_args="-d -p"
# Some default values in case /etc/sysconfig/kdump doesn't include KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug"
Hi Lianbo, On 08/12/20 at 02:32pm, Lianbo Jiang wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
[2] should be good, but for [1], I'm not sure if this can make kdump service startup even slower. Have you compared the performance with/without the change?
BTW, "set -x" usually means too much noise, we may need to have some loglevel, export fine grained logs if possible by default, the export the whole "set -x" logs when the logleve is set as max.
Another question is there is an option in dracut below: --debug output debug information of the build process
-v, --verbose increase verbosity level (default is info(4))
For /sbin/mkdumprd debuggin, should we just use the dracut option for dracut debug info?
Maybe you have tried to use logger, do you have other reason to drop the usage? With this patches applied, not sure if the original visible logs in systemd journal will disappear or not.
Changes since v1: [1] Set default log path to '/var/log/kdump' instead of '/var/crash' [2] Add a timestamp to the debug log messages [3] Print a message to differentiate between the kdumpctl and mkdumprd log
Lianbo Jiang (2): kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file kdumpctl: add the '-d' option to enable the kexec loading debugging messages
kdumpctl | 12 +++++++++++- mkdumprd | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-)
-- 2.17.1 _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org
Thanks Dave
Hi, Dave
Thanks for the comment.
在 2020年08月19日 16:53, Dave Young 写道:
Hi Lianbo, On 08/12/20 at 02:32pm, Lianbo Jiang wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
[2] should be good, but for [1], I'm not sure if this can make kdump service startup even slower. Have you compared the performance with/without the change?
That will definitely affect performance although I haven't measured it exactly. I will provide the performance data when I post it again.
A good option is that we can accurately output useful messages in the kdump code, which has the least impact on performance.
BTW, "set -x" usually means too much noise, we may need to have some loglevel, export fine grained logs if possible by default, the export the whole "set -x" logs when the logleve is set as max.
For the log level, usually, it doesn't enable the debug log, but it means that we will have to enable it manually and try to reproduce the problem again.
Another question is there is an option in dracut below: --debug output debug information of the build process
-v, --verbose increase verbosity level (default is info(4))For /sbin/mkdumprd debuggin, should we just use the dracut option for dracut debug info?
The debug option(in dracut) also uses the 'set -x' as below: [[ $debug ]] && { export DRACUT_LOG_LEVEL=debug export PS4='${BASH_SOURCE}@${LINENO}(${FUNCNAME[0]}): '; set -x }
There seems to be the same amount of log output.
Maybe you have tried to use logger, do you have other reason to drop the usage? With this patches applied, not sure if the original visible logs in systemd journal will disappear or not.
The logger will output messages into the system log, I think that's fine.
But some users still hope to output the debug messages to a new log file instead of the console or memory.
Thanks. Lianbo
Changes since v1: [1] Set default log path to '/var/log/kdump' instead of '/var/crash' [2] Add a timestamp to the debug log messages [3] Print a message to differentiate between the kdumpctl and mkdumprd log
Lianbo Jiang (2): kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file kdumpctl: add the '-d' option to enable the kexec loading debugging messages
kdumpctl | 12 +++++++++++- mkdumprd | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-)
-- 2.17.1 _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org
Thanks Dave
Hi Lianbo,
On Wed, Aug 19, 2020 at 2:23 PM Dave Young dyoung@redhat.com wrote:
Hi Lianbo, On 08/12/20 at 02:32pm, Lianbo Jiang wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
[2] should be good, but for [1], I'm not sure if this can make kdump service startup even slower. Have you compared the performance with/without the change?
I agree with Dave's comment above. I am particularly worried about the arm64 systems which take a lot of time when MMU is turned off during kdump relocation (for e.g. the thunderx2 systems).
I have a couple of suggestions: - Let's keep a option to enable or disable this logging - not every user would like to enable this 'additional' debug information (log file creation) by default. So how about providing and additional parameter in the kdump configuraiton file(s) to see if this has been enabled explicitly by user. kdumpctl can accordingly enable the '-x' debug flag.
- Also how about doing a simple 'time' comparision with and without the [PATCH 1/2].
Thanks, Bhupesh
BTW, "set -x" usually means too much noise, we may need to have some loglevel, export fine grained logs if possible by default, the export the whole "set -x" logs when the logleve is set as max.
Another question is there is an option in dracut below: --debug output debug information of the build process
-v, --verbose increase verbosity level (default is info(4))For /sbin/mkdumprd debuggin, should we just use the dracut option for dracut debug info?
Maybe you have tried to use logger, do you have other reason to drop the usage? With this patches applied, not sure if the original visible logs in systemd journal will disappear or not.
Changes since v1: [1] Set default log path to '/var/log/kdump' instead of '/var/crash' [2] Add a timestamp to the debug log messages [3] Print a message to differentiate between the kdumpctl and mkdumprd log
Lianbo Jiang (2): kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file kdumpctl: add the '-d' option to enable the kexec loading debugging messages
kdumpctl | 12 +++++++++++- mkdumprd | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-)
-- 2.17.1 _______________________________________________ kexec mailing list -- kexec@lists.fedoraproject.org To unsubscribe send an email to kexec-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org
Thanks Dave
On 8/24/20 10:19 AM, Bhupesh Sharma wrote:
Hi Lianbo,
On Wed, Aug 19, 2020 at 2:23 PM Dave Young dyoung@redhat.com wrote:
Hi Lianbo, On 08/12/20 at 02:32pm, Lianbo Jiang wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
[2] should be good, but for [1], I'm not sure if this can make kdump service startup even slower. Have you compared the performance with/without the change?
I agree with Dave's comment above. I am particularly worried about the arm64 systems which take a lot of time when MMU is turned off during kdump relocation (for e.g. the thunderx2 systems).
I have a couple of suggestions:
- Let's keep a option to enable or disable this logging - not every
user would like to enable this 'additional' debug information (log file creation) by default. So how about providing and additional parameter in the kdump configuraiton file(s) to see if this has been enabled explicitly by user. kdumpctl can accordingly enable the '-x' debug flag.
I like the idea of having an additional parameter in the kdump configuration file to enable and disable the kdumpctl (mkdumprd/kexec load etc.) debug log on-demand. If the test results reveal that the overhead of this debug option is negligible then set it to "enabled" by default else set it to "disabled". If the overhead is significantly high then it may also impact the *overall* system's boot time.
The suggestion to keep the debug log *enabled* by default is to ensure that we capture the addition debug log of any error/problem in the first attempt.
-- Buland
On 8/12/20 12:02 PM, Lianbo Jiang wrote:
Sometimes, debugging the kdump service failure becomes very challenging because there is no complete debugging information, which requires modification of the options or the scripts like kdumpctl, mkdumprd, etc to collect the information for troubleshooting.
That means users have to wait for the next failure so that they can capture the additional information, which could waste valuable time.
This patch series will add the debugging messages and save them to a file. It includes the following patches:
[1] [PATCH 1/2] kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file [2] [PATCH 2/2] kdumpctl: add the '-d' option to enable the kexec loading debugging messages
Changes since v1: [1] Set default log path to '/var/log/kdump' instead of '/var/crash' [2] Add a timestamp to the debug log messages [3] Print a message to differentiate between the kdumpctl and mkdumprd log
Lianbo Jiang (2): kdumpctl/mkdumprd: add 'set -x' to output debugging information to a file kdumpctl: add the '-d' option to enable the kexec loading debugging messages
kdumpctl | 12 +++++++++++- mkdumprd | 10 ++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-)
Hello Lianbo,
As per cover letter the default log path is changed to '/var/log/kdump', however, the 'KDUMP_LOG_PATH' in the actual patch is still set to '/var/log'.
+KDUMP_LOG_PATH="/var/log"
Is it a typo?
-- Buland