On 2016/10/10 at 15:43, Xunlei Pang wrote:
The current method for kdump memory debug is to use dracut "rd.memdebug=[0-3]", it is not enough for debugging kernel modules. For example, when we want to find out which kernel module consumes a large amount of memory, "rd.memdebug" won't help too much.
A better way is needed to achieve this requirement, this is very useful for kdump OOM debugging.
The principle of this patch series is to use kernel trace to track slab and buddy allocation calls during kernel module loading(module_init), thus we can analyze all the trace data and get the total memory consumption.
The trace events include memory calls under /sys/kernel/debug/tracing/events: kmem/mm_page_alloc kmem/mm_page_free kmem/kmalloc kmem/kmalloc_node kmem/kmem_cache_alloc kmem/kmem_cache_alloc_node
We also inpect the following events to detect the module loading module/module_load module/module_put
We can get the module name and task pid from "module_load" event which also mark the beginning of the loading, and module_put called by the same task pid implies the end of the loading. So the memory events recorded in between by the same task pid are consumed by this module during loading(i.e. modprobe or module_init()).
With these information, we can record approximately the total memory consumption involved by each kernel module loading.
One major flaw of this method is that the trace ring buffer consumes a lot of memory. If it is too small, old records maybe be overwritten by subsequent
For some large systems, the amount of memory consumption is appreciable.
Does anyone have a good idea to reduce the memory size consumed by tracing?
I previously tried to use trace filter to track specific process doing modprobe, but according to the test results, normally the process name with "systemd-udevd" and "modprobe" will insert the ko, but sometimes I found some unknown process will also insert the ko. So it's unreliable to do so.
I am currently thinking to monitor buddy event only, as all the large slab allocation will actually fall back to buddy to request pages. This will considerably reduce the requirement of trace buffer size.
Regards, Xunlei
records. The trace ring buffer is set to be 10MB by default, but it can be overridden by users via the standard kernel boot parameter "trace_buf_size".
Users should increase the crash kernel memory reservation as needed after setting large trace ring buffer size, in case oom happens during debugging.
Usage: 1)Pass "rd.memdebug" to kdump kernel cmdline using "KDUMP_COMMANDLINE_APPEND" in /etc/sysconfig/kdump. 2)Pass the extra "trace_buf_size=nn[KMG]" to specify trace ring buffer size(per cpu) as needed.
Xunlei Pang (2): memdebug-ko: add dracut-memdebug-ko.sh to debug kernel module memory consumption module-setup: apply kernel module memory debug support
dracut-kdump.sh | 11 ++++ dracut-memdebug-ko.sh | 144 +++++++++++++++++++++++++++++++++++++++++++++++++ dracut-module-setup.sh | 12 +++++ kdumpctl | 14 +++++ kexec-tools.spec | 2 + 5 files changed, 183 insertions(+) create mode 100755 dracut-memdebug-ko.sh