I've got loads of messages like this:
Apr 27 15:31:18 lakshmi kernel: auditd invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=-17 Apr 27 15:31:22 lakshmi kernel: Apr 27 15:31:22 lakshmi kernel: Call Trace: Apr 27 15:31:22 lakshmi kernel: [<ffffffff810756aa>] out_of_memory+0x70/0x2ea Apr 27 15:31:22 lakshmi kernel: [<ffffffff81077af4>] __alloc_pages+0x24a/0x2d3 Apr 27 15:31:22 lakshmi kernel: [<ffffffff8107942d>] __do_page_cache_readahead+0xb3/0x20a Apr 27 15:31:22 lakshmi kernel: [<ffffffff8102e012>] update_curr+0xf8/0x11a Apr 27 15:31:22 lakshmi kernel: [<ffffffff81074b1d>] filemap_fault+0x168/0x327 Apr 27 15:31:22 lakshmi kernel: [<ffffffff8107e310>] __do_fault+0x65/0x489 Apr 27 15:31:22 lakshmi kernel: [<ffffffff81081adf>] handle_mm_fault+0x4ed/0x976 Apr 27 15:31:22 lakshmi kernel: [<ffffffff81052730>] do_futex+0x5a0/0x9d9 Apr 27 15:31:22 lakshmi kernel: [<ffffffff8125eb77>] do_page_fault+0x45b/0x7e4 Apr 27 15:31:22 lakshmi kernel: [<ffffffff8109bcab>] do_readv_writev+0x199/0x1ae Apr 27 15:31:22 lakshmi kernel: [<ffffffff8125d05d>] error_exit+0x0/0x84 Apr 27 15:31:22 lakshmi kernel: Apr 27 15:31:22 lakshmi kernel: Mem-info: Apr 27 15:31:22 lakshmi kernel: Node 0 DMA per-cpu: Apr 27 15:31:22 lakshmi kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA32 per-cpu: Apr 27 15:31:22 lakshmi kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 141 Cold: hi: 62, btch: 15 usd: 14 Apr 27 15:31:22 lakshmi kernel: Active:238680 inactive:247601 dirty:0 writeback:0 unstable:0 Apr 27 15:31:22 lakshmi kernel: free:3428 slab:4875 mapped:573 pagetables:8303 bounce:0 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA free:8036kB min:28kB low:32kB high:40kB active:2624kB inactive:0kB present:10512kB pages_scanned:397855 all_unreclaimable? yes Apr 27 15:31:22 lakshmi kernel: lowmem_reserve[]: 0 2003 2003 2003 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA32 free:5676kB min:5708kB low:7132kB high:8560kB active:952096kB inactive:990404kB present:2051184kB pages_scanned:5302627 all_unreclaimable? yes Apr 27 15:31:22 lakshmi kernel: lowmem_reserve[]: 0 0 0 0 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB Apr 27 15:31:22 lakshmi kernel: Node 0 DMA32: 1*4kB 15*8kB 7*16kB 4*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5676kB Apr 27 15:31:22 lakshmi kernel: Swap cache: add 1268676, delete 1268676, find 941701/1065348, race 0+116 Apr 27 15:31:22 lakshmi kernel: Free swap = 0kB Apr 27 15:31:22 lakshmi kernel: Total swap = 1020088kB Apr 27 15:31:22 lakshmi kernel: Free swap: 0kB Apr 27 15:31:22 lakshmi kernel: 524000 pages of RAM Apr 27 15:31:22 lakshmi kernel: 10614 reserved pages Apr 27 15:31:22 lakshmi kernel: 2369 pages shared Apr 27 15:31:22 lakshmi kernel: 0 pages swap cached Apr 27 15:31:22 lakshmi kernel: rklogd invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0 ...
Neal Becker wrote:
I've got loads of messages like this:
I had some problems with kernel-2.6.25.3-18.fc9.x86_64 the other day that I'm still trying to track down. If your machine is still low on memory, it might be useful to capture /proc/meminfo, /proc/slabinfo, and the output of 'ps auxf'
On Wed, May 21, 2008 at 08:54:50AM -0400, Neal Becker wrote:
I've got loads of messages like this: ... Apr 27 15:31:22 lakshmi kernel: Node 0 DMA free:8036kB min:28kB low:32kB high:40kB active:2624kB inactive:0kB present:10512kB pages_scanned:397855 all_unreclaimable? yes Apr 27 15:31:22 lakshmi kernel: lowmem_reserve[]: 0 2003 2003 2003 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA32 free:5676kB min:5708kB low:7132kB high:8560kB active:952096kB inactive:990404kB present:2051184kB pages_scanned:5302627 all_unreclaimable? yes Apr 27 15:31:22 lakshmi kernel: lowmem_reserve[]: 0 0 0 0 Apr 27 15:31:22 lakshmi kernel: Node 0 DMA: 1*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB Apr 27 15:31:22 lakshmi kernel: Node 0 DMA32: 1*4kB 15*8kB 7*16kB 4*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5676kB Apr 27 15:31:22 lakshmi kernel: Swap cache: add 1268676, delete 1268676, find 941701/1065348, race 0+116 Apr 27 15:31:22 lakshmi kernel: Free swap = 0kB Apr 27 15:31:22 lakshmi kernel: Total swap = 1020088kB Apr 27 15:31:22 lakshmi kernel: Free swap: 0kB Apr 27 15:31:22 lakshmi kernel: 524000 pages of RAM Apr 27 15:31:22 lakshmi kernel: 10614 reserved pages Apr 27 15:31:22 lakshmi kernel: 2369 pages shared Apr 27 15:31:22 lakshmi kernel: 0 pages swap cached Apr 27 15:31:22 lakshmi kernel: rklogd invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
By the looks of things, this is just a case where you filled up RAM, and have no more swap available. There's really not much the kernel can do here. Either add more swap, or find the app that's causing you to eat so much memory, and see if you can somehow convince it to use less. (The latter is probably a better bet, if something is eating up all ram/swap, giving it more will likely just make it consume even more)
Dave