Thanks a lot, these info are very helpful.
Better to keep it for debugging for now, and ask users to use it very carefully.
On Tue, Apr 20, 2021 at 3:54 PM Milan Broz <gmazyland(a)gmail.com> wrote:
> TL;DR what you are trying to do is to actually reverse many security measures
> we added. It is perhaps acceptable for debugging but hardly for real generic system.
> - using memory-hard function increases cost of dictionary and brute-force
> You can always decrease amount of memory needed, but you should do it only
> if you know that security margin is ok (like password is randomly generated
> with enough entropy).
> - key is in keyring to remove possibility for normal userspace to receive
> the key from kernel. Moreover, there is no need to retain kernel in keyring once
> dm-crypt device is activated. (It is still in kernel memory but only in crypto
> functions context). (Systemd also uses keyring to cache passphrase but that's
> different thing.)
> You can still use old way for activation with --disable-keyring activation,
> but then you disable this possibility.
> More comments below.
> On 19/04/2021 12:00, Kairui Song wrote:
> > Hi all,
> > I'm currently trying to add kdump support for systemd with full-disk
> > LUKS encryption. vmcores contain sensitive data so they should also be
> > protected, and network dumps sometimes are not available. So kdump has
> > to open the LUKS encrypted device in the kdump environment.
> > I'm using systemd/dracut, my work machine is running Fedora 34, and
> > there are several problems I'm trying to solve:
> > 1. Users have to input the password in the kdump kernel environment.
> > But users often don't have shell access to the kdump environment.
> > (headless server, graphic card not working after kexec, both are very
> > common)
> > 2. LUKS2 prefers Argon2 as the key derivation function, designed to
> > use a lot of memory. kdump is expected to use a minimal amount of
> > memory. Users will have to reserve a huge amount of memory for kdump
> > to work (eg. 1G reserve for kdump with 4G total memory which is not
> > reasonable).
> When I added Argon2 to LUKS2, I actually expected such issues. Despite
> some people beats me that they cannot use arbitrary amount of memory,
> we have some hard limits that were selected that it should work on most recent
> systems. Maybe kdump can live with it.
> - maximum memory cost limit is 4GB, no LUKS2 device can use more for Argon2
> - we never use more than half of available physical memory
> (measured on the host where the device was formatted)
> - required amount of memory is visible in LUKS2 metadata (luksDump)
> for the particular keyslot (Memory: the value is in kB)
> - we use benchmark to calculate memory cost with prefered unlocking
> time 2 seconds (again, on the device where LUKS was formatted)
> Small systems (like RPi2) the uses much smaller acceptable values.
> You can configure all costs (time, memory, threads) during format
> or even set them to predefined values.
> I am sorry, but there is really no way around this - and the requeired
> memory must be physical memory (otherwise it slows down extremely).
> This is a feature, not a bug :-)
> > To fix these problems, I tried to pass the master key to the second
> > kernel directly via initramfs. Kdump will modify the initramfs in
> > ramfs to include the key, kexec_load it, and never write to any actual
> > back storage. This worked with old LUKS configurations.
> Well, passing volume key this way is quite insecure, but perhaps
> acceptable for debugging.
> > But LUKS2/cryptsetup now stores the key in the kernel keyring by
> > default. The key is accessible from userspace.
> If you are talking about volume key (not passsphrase), it is not
> available from userspace. Only reference to it. But you can use
> this reference to construct in-kernel dm-crypt device.
> Please read https://gitlab.com/cryptsetup/cryptsetup/-/blob/master/docs/Keyring.txt
> > Users can enter the password to start kdump manually and then it will
> > work, but usually people expect kdump service to start automatically.
> > (WIP patch series:
> > https://email@example.com...)
> > I've several ideas about how to improve it but not sure which one is
> > better, and if this is a good idea after all:
> > 1. Simply introduce a config to let systemd-cryptsetup disable kernel
> > keyring on setup, there is currently no such option.
> Well, that option could be useful anyway and we have support for it
> in cryptsetup (--disable-keyring CLI option) and libcryptsetup, so why not.
> Just this should not be a default option.
> This is should be patch for systemd-cryptsetup only as libcryptsetup supports it.
> > 2. If we can let the key stay in userspace for a little longer, eg.
> > for systems booted with dracut/systemd, when
> > systemd-cryptsetup(a)%s.service opens the crypt device, keep the key in
> > dm-crypt. And later when services like kdump have finished loading,
> > cryptsetup can refresh the device and store the key in the kernel
> > keyring again.
> We invalidate volume key in keyring after libceyposetup operation
> is finished (and kernel removes the reference once keyring garbage collection
> is run).
> I can imagine to add some option to keep key inside keyring even after
> call is finished, but as said above, this removes some security margin
> we intentionally introduced here.
I agree with your comments, thanks! These two approaches seem not a
good idea now.
How about plan 3 and 4?
> 3. Let kdump use some custom helper/service to load all needed
> resources in the early initrd boot stage, prior to
> systemd-cryptsetup(a)%s.service. It will ask the password / parse the
> keyfile and load kdump, then provide info for systemd-cryptsetup or
> just do the setup. Or maybe let systemd-cryptsetup support some kind
> of "plugins" so other tools can use it.
Some details could be changed/improved, but
systemd-cryptsetup(a)%s.service will prompt for a password or use a
So I think at this point, loading kdump with the volume key should be
safe? At least long as the kdump kernel/environment itself isn't
compromised. Loaded kdump resources can be restricted to be only
accessible from the kernel side.
After panic, kernel kexec jumps to kdump kernel, and that's an
minimized emergency environment that only lives for a very short
> 4. A better and safer solution seems to keep a consistent key ring
> between kexec boots but also more complex and difficult as different
> arch implements kexec differently.
Maybe plan 4 will be a good idea if doable? Since that keeps the key
consistent in the kernel between kexec boots, and cryptsetup can just
This series add an more accurate kdump crashkernel estimation helper.
More details are in PATCH 5/6.
1/6 - 2/6 are required change for the new feature.
3/6 introduce a new file, just to make the code structure more clear.
4/6 fixes a bug I found while composing this patch.
5/6 and 6/6 adds this new command and docs.
Kairui Song (6):
kdumpctl: only acquire the single instance lock when necessary
kdumpctl: allow passing in extra cmdline using env variable
kdump-estiamte.sh: introduce a seperate file
kdump-lib.sh: fix kdump_get_arch_recommend_size
kdump-estimate.sh: add reboot estimation support
.editorconfig | 2 +-
crashkernel-howto.txt | 104 ++++-
dracut-kdump.sh | 15 +
kdump-estimate-cleanup.service | 8 +
kdump-estimate.service | 11 +
kdump-estimate.sh | 778 +++++++++++++++++++++++++++++++++
kdump-lib.sh | 74 +---
kdump.shutdown | 13 +
kdumpctl | 108 +----
kexec-tools.spec | 21 +-
10 files changed, 940 insertions(+), 194 deletions(-)
create mode 100644 kdump-estimate-cleanup.service
create mode 100644 kdump-estimate.service
create mode 100644 kdump-estimate.sh
create mode 100644 kdump.shutdown
When user only provided a "path" value in kdump.conf without explicitly
specify a dump target, covering all cases with "path" config value is
"path" could be a mount point, a dir insidse a mount point, or nested mount,
bind mount etc. kdump need to detect the real dump target based on the
"path" value, so user have to be carefull with it and in charge
of creating it.
But for user specified dump target (dump target explicitly specified with
nfs/ext4/xfs/...), the "path" config have only one sole meaning: the absolute
dump path on a dump target. In this case it's safe for kdump to create
Signed-off-by: Kairui Song <kasong(a)redhat.com>
mkdumprd | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mkdumprd b/mkdumprd
index cc37ae18..0dc4459e 100644
@@ -242,7 +242,8 @@ check_user_configured_target()
# For user configured target, use $SAVE_PATH as the dump path within the target
if [[ ! -d "$_mnt/$SAVE_PATH" ]]; then
- perror_exit "Dump path \"$_mnt/$SAVE_PATH\" does not exist in dump target \"$_target\""
+ dwarn "Dump path \"$SAVE_PATH\" does not exist on dump target \"$_target\", kdump will create this path automatically."
+ mkdir -p "$_mnt/$SAVE_PATH" || perror_exit "Failed to create dump path \"$SAVE_PATH\" on dump target."
check_size fs "$_target"
This mass update is done with the assistant of shellcheck and shfmt.
This patch series fixed most syntax issues found by shellcheck,
tons of corner cases and issues are fixed.
Also simplified a lot of code according to suggestions by shellcheck.
Related shellcheck warn/error/suggest info links are included in the commit
Since shellcheck can help to detect non-POSIX syntax issue in POSIX
script (#!/bin/sh), and the coding style in kexec-tools package is
very messy, this series refactored the code structure in following way:
- kdumpctl, mkdumprd, mkfadumprd, *-module-setup.sh and kdump-lib.sh
is "bash only" script. We rely on bash only syntax heavily, and they
will only be used in the first kernel, so just let them leverage bash
syntax for better maintainability and better performance.
- kdump-lib-initramfs.sh, and dracut-kdump.sh is now POSIX compatible,
and will be used in the second kernel.
This refactoring also enables kdump to use dash or busybox, instead of
bash in the second kernel, which improves performance and memory
usage. Tested with latest dracut on Fedora 34, now "dash" in second
kernel works well, it's faster than bash and save a lot of memory.
We may test dash or busybox as default kdump shell in the future for
Patch 25, 26, 30, 41, 46 is automatic code update using script and
tools like shfmt, other patch may need careful review.
Patch 1 added a editorconfig file for shfmt to work properly, which
is Acked by Pingfan but still posting it to make it easier to review.
Patch 2 - 29 focus on the bash part:
Patch 2, 3, 4, 5, 8 introduced new config file read and format helper,
which unified the config file reading in kexec-tools scripts. So space
in config file should be no longer a problem (previously random problem
may occur since kexec-tools scripts is using many different ways to read
the config file). And this make it much easier to clean up the code.
Patch 6, 7, 9 - 24 fixed many issues found with shellcheck in bash
Some issues are already breaking kdump just no one noticed yet.
Patch 25, 26, 30 is an automatic code update using sed and shfmt, there
should be no code behaviour change.
Patch 27, 28, 29 batch fixed some common bash script issues manually
with hints from shellcheck.
Patch 31 - 48 focus on the POSIX part:
Patch 31 - 33 restructures the code, prepare to make
kdump-lib-initramfs.sh a POSIX script and drop kdump-lib.sh in kdump
Patch 34 - 38 clean up code in dracut-kdump.sh, and fix misc issues
found by shellcheck manually, prepare to make it POSIX compatible.
Patch 39 contains a lot of code refactoring that makes dracut-kdump.sh POSIX
Patch 40 contains a lot of code refactoring that makes kdump-lib-initramfs.sh
Patch 41 is an automatic code update using sed, there should be no code
Patch 42, 43 reworked how nmcli is being called by kexec-tools, fixed
word splitting issue and allow connection name to contain space.
Patch 44 optimized a sed call.
Patch 45 batch fixed some common bash script issues manually
with hints from shellcheck.
Patch 46 is an automatic code update using shfmt, there should be no code
Patch 47, 48 contains a lot of change that makes kdump-logger.sh
Patch 49 allows mkdumprd to use dash instead of bash in second kernel.
Update from V1:
- Fix many typos and code error, and fix a few more existing code issues
found while reviewing the patch, many thanks to Philipp Rudo.
- Add more info in commit message and cover letter.
Kairui Song (49):
Add a .editorconfig file
kdump-lib.sh: add a config format and read helper
kdump-lib.sh: add a config value retrive helper
kdump-lib.sh: use kdump_get_conf_val to read config values
kdumpctl: use kdump_get_conf_val to read config values
kdumpctl: get rid of a `wc` call
kdumpctl: fix fragile loops over find output
mkdumprd: use kdump_get_conf_val to read config values
mkdumprd: make dracut_args an array again
mkdumprd: remove some redundant echo
mkdumprd: fix multiple issues with get_ssh_size
mkdumprd: use array to store ssh arguments in mkdir_save_path_ssh
mkfadumprd: make _dracut_isolate_args an array
dracut-module-setup.sh: rework kdump_get_ip_route_field
dracut-module-setup.sh: remove an unused variable
dracut-module-setup.sh: fix _bondoptions wrong references
dracut-module-setup.sh: use "*" to expend array as string
dracut-module-setup.sh: fix a ambiguous variable reference
dracut-module-setup.sh: fix a loop over ls issue
dracut-module-setup.sh: make iscsi check fail early if cd failed
bash scripts: remove useless cat
bash scripts: get rid of expr and let
bash scripts: get rid of unnessary sed calls
bash scripts: always use "read -r"
bash scripts: use $(...) notation instead of legacy `...`
bash scripts: replace '[ ]' with '[[ ]]' for bash scripts
bash scripts: fix variable quoting issue
bash scripts: fix redundant exit code check
bash scripts: declare and assign separately
bash scripts: reformat with shfmt
Prepare to make kdump-lib-initramfs.sh a POSIX compatible lib
Merge kdump-error-handler.sh into kdump.sh
kdump-lib-initramfs.sh: move dump raleted functions to kdump.sh
dracut-kdump.sh: don't put KDUMP_SCRIPT_DIR in PATH
dracut-kdump.sh: remove add_dump_code
dracut-kdump.sh: simplify dump_ssh
dracut-kdump.sh: Use stat instead of ls to get vmcore size
dracut-kdump.sh: POSIX doesn't support pipefail
dracut-kdump.sh: make it POSIX compatible
kdump-lib-initramfs.sh: make it POSIX compatible
kdump-lib.sh: replace '[ ]' with '[[ ]]' and get rid of legacy ``
kdump-lib.sh: fix and simplify get_nmcli_value_by_field
kdump-lib.sh: fix word splitting of nmcli params
kdump-lib.sh: avoid calling sed multiple times in get_system_size
kdump-lib.sh: batch fix issues found with shellcheck
kdump-lib.sh: reformat with shfmt
kdump-logger.sh: refactor to remove some bash only syntax
kdump-logger.sh: make it POSIX compatible
mkdumprd: allow using dash
.editorconfig | 32 +
dracut-early-kdump-module-setup.sh | 11 +-
dracut-kdump-emergency.service | 2 +-
dracut-kdump-error-handler.sh | 10 -
dracut-kdump.sh | 511 ++++++++---
dracut-module-setup.sh | 668 +++++++-------
kdump-lib-initramfs.sh | 346 ++-----
kdump-lib.sh | 1356 +++++++++++++---------------
kdump-logger.sh | 117 +--
kdumpctl | 604 ++++++-------
kexec-tools.spec | 2 -
mkdumprd | 600 ++++++------
mkfadumprd | 21 +-
13 files changed, 2104 insertions(+), 2176 deletions(-)
create mode 100644 .editorconfig
delete mode 100755 dracut-kdump-error-handler.sh