From: Kairui Song on gitlab.com Merge Request: https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171 NOTE: Truncated patchset due to missing public @redhat.com email address on your GitLab profile at https://gitlab.com/-/profile. Once that is fixed, close and reopen the merge request to retrigger sending the emails.
It has been a long-term demand for having a default crashkernel value for each kernel build. This helps to manage kernel's crashkernel usage by kernel version and provide users a reference default value.
RHEL kernels support a `crashkernel=auto` cmdline param. And kernel have a built-in default value used when `auto` is specified. Many attempts are made to push this upstream. Recent attempt to push this upstream raised a lot of discussions [1].
Upstream is not accepting this `crashkernel=auto` design. So after more discussions, instead of adding a default value in kernel, just add a config that packaged with kernel is more doable. Other packages, like kexec-tools[2], can use this as a reference and update kernel cmdline param.
[1]: https://lore.kernel.org/linux-mm/20210507010432.IN24PudKT%25akpm@linux- foundation.org/ [2]: https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.o rg/thread/MKUSELX3GKKXPFNLEGJMWXSS2LADRPMQ/
Signed-off-by: Kairui Song kasong@redhat.com
--- Documentation/admin-guide/kdump/kdump.rst | 11 ----------- kernel/crash_core.c | 28 ++-------------------------- redhat/Makefile | 1 + redhat/generate_crashkernel_conf.sh | 25 +++++++++++++++++++++++++ redhat/kernel.spec.template | 5 +++++ 5 files changed, 33 insertions(+), 37 deletions(-)
From: Justin M. Forbes on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_5985754...
I am in favor of this, as it changes things back closer to upstream.
From: Patrick Talbert on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_5992831...
Acked-by: Patrick Talbert ptalbert@redhat.com (via approve button)
From: Baoquan He on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6035189...
This series of change looks good to me. While I can't find the approval button to add Acked-by. Is it caused by the failued pipeline?
"Detached merge request pipeline #318739078 failed for 93508176 6 days ago"
From: Baoquan He on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6035962...
And about the crashkernel value, we may need increase the lowest value from 160M to a larger value, e.g 192M. When checked CKI failure of kdump test case, I found the kernel itself has been enlarged a little to 40M+.
And worse, I found since kernel-5.13.0-0.rc4.20210603git324c92e5e0ee.34, bss of kernel increased from 5M to 22M, this surely need be figured out what happened. RHEL9 kernel seems to be larger than rhel8, increasing the crashkernel value a little bit could be safer.
From: Patrick Talbert on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6036438...
The kernel-ark project has a limited number of members on GL and only members can see/use the Approve button. Please simply leave a comment with an 'Acked- by: NAME <EMAIL>' line (or Nacked, etc).
Thank you,
Patrick
From: Baoquan He on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6037217...
Thanks for telling. Kairui told he would update to change the crashkernel value, will add comment with "Acked-by".
From: Kairui Song on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6076991...
Good suggestion, I'll update the crashkernel default values. I tried to figure out what is causing the growth of kernel-5.13.0-0.rc4.20210603git324c92e5e0ee.34, and found a lot of new things enabled in config, kgdb, kfence, and many other debug configs will all cause .bss to grow (switch back to old .config from f33, using the same kernel version and code base the .bss will be ~5M again). I'm not sure if we will turn off some of the debug options in the future but kernel is growing after all.
So for x86 and s390, we can change the crashkernel value to 0G-4G:192M,4G-64G:256M,64G-:512M (previously 1G-4G:160M,4G-64G:192M,64G-1T:256M,1T-:512M).
According to https://access.redhat.com/documentation/en-us/red_hat_enterprise_ linux/8/html/performing_a_standard_rhel_installation/system-requirements- reference_installing-rhel#check-disk-and-memory-requirements_system- requirements-reference, since we already requiring at least 1.5 GiB memory for RHEL, but I'm not sure how things work for VM deployments, will this cause more memory pressure for VMs?
From: Justin M. Forbes on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6082300...
Good suggestion, I'll update the crashkernel default values. I tried to
figure out what is causing the growth of kernel-5.13.0-0.rc4.20210603git324c92e5e0ee.34, and found a lot of new things enabled in config, kgdb, kfence, and many other debug configs will all cause .bss to grow (switch back to old .config from f33, using the same kernel version and code base the .bss will be ~5M again). I'm not sure if we will turn off some of the debug options in the future but kernel is growing after all.
Any rawhide kernel with a git snapshot version after the rcX is built as a debug kernel only. This is a rawhide specific thing, and will nt translate to stable releases.
From: Kairui Song on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6142300...
Thanks for all the useful info!
I just updated the crashkernel value for x86/s390 to `0G-4G:192M,4G-64G:256M,64G-:512M`, and changed the file name to `crashkernel.default`, as suggested by @baoquan_he and @ruyang. Also rebased to latest `os-build` branch.
From: Dave Young on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6159112...
Hi Kairui, it worth to say more about the reason of the update from 1G-4G to 0G-4G:
* originally we have another RHEL only patch to round up the memblock collected system ram size by 128M because the memblock total size usually less than actual dimm size. Moving to 0-4G we can safely work without that patch. * the RHEL installtion minimum mem requirement is 1.5G for RHEL8, so we expect this will not have extra adding impact for small memory systems.
But anyway if virt people can confirm about the minimum memory requirement is
=1G it would be great.
From: David Hildenbrand on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6161249...
With kata-containers we run VMs as small as 256 MiB (or sometimes even smaller). However, in these setups, we run a stripped-down OS, so we can expect to kdump never to be installed/active (and IIUC, that implies that we never do a crashkernel reservation). At least that's what I assume.
Note that [RHEL documents "Recommended minimum RAM" ](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm l/performing_a_standard_rhel_installation/system-requirements- reference_installing-rhel), with a note that "It is possible to complete the installation with less memory than the recommended minimum requirements. The exact requirements depend on your environment and installation path.". So it might work with more or less. I wouldn't be surprised if people are running 1GiB VMs with RHEL8; the question is, if these setups default-install kdump (I assume so?), and if we care for new versions of RHEL.
On small VMs, we end up "wasting" a significant amount of memory on kdump reservations, which makes me believe that kdump isn't actually a good fit for small VMs at all, especially with newer RHEL kernels. So if we assume that people with less than 1.5G don't need kdump, why care at all about "we have another RHEL only patch to round up the memblock collected system ram size by 128M"?
I don't immediately see why the switch from 1G-4G to 0G-4G is actually helpful and I'd just leave it like that and have kdump disabled for machines that have a memblock size < 1 GiB. Can you clarify what I am missing?
From: Dave Young on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6161352...
Hi David,
The details of the round up is like below: For 1G-4G:xM, in kernel code we check if system_ram_size < 1G then kernel does not reserve kdump memory otherwise xM mem is reserved. But here the system_ram_size is not acurate, say a system with 1G memory installed, the system_ram_size could be 900M in kernel point of view. That will cause no kdump memory reserved. It is somehow reasonable but hard to explain to customer because people regard 1G as the system memory size. And it will be hard for QE to test as well. So we had a RHEL only patch which round up system mem size by 128M. This patch will be dropped as well.
Hope this can explain the question. Thanks!
From: David Hildenbrand on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6161523...
Exactly, and as I said, once we're dealing with such small VMs that are below the recommended memory size, why do we even care?
1.5 GiB will work fine, with or without that patch. For 1 GiB VMs crashkernel will not be allocated, which is what we usually want. For example, before this change a 786MiB VM might work just fine. With that change, we might run out of memory during boot because of the crashkernel (consuming 192MiB) still succeeding to get allocated.
Customers that complain that the crashkernel won't get allocated for a 1GiB VM can be pointed at the recommended system size IMHO.
From: Dave Young on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6161569...
Yes, if we regard the 1.5G minimum requirement then it would be good enough as you said. @baoquan_he @kasong , what do you think?
From: David Hildenbrand on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6161598...
Just to be clear, I'm not strongly against this. I just can see it turn some corner case setups broken without any clear benefit if we consider the system requirements a recommendation that customers should stick to until they really know what they are doing (customers will always complain, and if it's that their corner case setup no longer boots due to the changed crashkernel allocation policy).
From: Baoquan He on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6162081...
"1G-4G:192M" looks good to me, and the range is consistent with rhel8.5 crashkernel on the low range.
From: Kairui Song on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6162160...
I agree with "1G-4G:192M", I can update and do a rebase again.
From: Richard W.M. Jones on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6168364...
libguestfs is another "small VM" use case. Our default memory size is 1280M (or 3584M in virt-v2v), so I guess we won't hit this. It might be nice to have a "crashkernel=off" parameter or similar, which we'd use to disable this feature in our appliance.
From: Don Zickus on gitlab.com https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1171#note_6178041...
Good to see this patch always evolving. :-) The current approach is nice because we don't carry an out of tree patch no more as @jmflinuxtx said. I still wish the memory reqs would come down (I know dracut/systemd make it hard). I remember using 32M for rhel-8 testing I think. This would lower the resistance to enabling this more often.
On a side positive side note. From a CI perspective, one of the struggle with using test frameworks with kernel testing is kernel panics. Beaker allows us console access, which is a side channel thing and not the norm. Using kdump and saving dmesg logs on panics, allows tools to easily get at stack traces when a unexpected reboot happens. Not a perfect solution, but something useful when console access isn't always around (or ipmi).
Thanks for this continued effort!
kernel@lists.fedoraproject.org
-
Baoquan He (via Email Bridge) -
Dave Young (via Email Bridge)
-
David Hildenbrand (via Email Bridge)
-
Don Zickus (via Email Bridge)
-
Justin M. Forbes (via Email Bridge)
-
Kairui Song (via Email Bridge)
-
Patrick Talbert (via Email Bridge)
-
Richard W.M. Jones (via Email Bridge)