On Mon, Sep 13, 2004 at 11:36:28AM +0200, Axel Thimm wrote:
On Mon, Sep 13, 2004 at 03:05:44AM +0200, Axel Thimm wrote:
On Fri, Sep 03, 2004 at 06:46:18AM -0400, Jakub Jelinek wrote:
On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote:
How can I debug the memory consumption on this box? Which figures are the ones to look for and which ones do accumulate for the OOM killer?
IMHO best would be to install 32-bit and 64-bit httpd side by side,
It turns out that memory gets consumed and not returned back to the system independent of httpd (the oom-killer just strikes there first).
The bug has now been identified as a kernel memory leak in 2.6.8-1.521 and 2.6.8-1.541 on a x86_64 system running ia32 bit apps in IA32_EMULATION mode.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132947
It is easily reproducable, and I wonder why there have been so few reports on the net on this (kernel 2.6.8-1.521 has been released before a month). Probably most x86_64 users are running in 64 bits only and don't observe the leak.
Any references/fixes are more than welcome. :)
On an FC2/x86_64 system (Tyan S2880 with one processor only) with untained 2.6.8-1.521 on 1GB RAM simple compilations can eat up all the memory. I trimmed down such a system up to basic networking to detect which processes were locking the memory, and no userland processes are holding the memory. But almost all memory is flagged as "used" (with negligible size of buffers and cache).
Is this a kernel memory leak? Any other information I should collect?
(I still cannot judge whether the change from kernel 2.4 to 2.6 or the architecture change i386 to x86_64 is responsible for this due to lack of different combinations)
# free total used free shared buffers cached Mem: 1027016 1022600 4416 0 992 7288 -/+ buffers/cache: 1014320 12696 Swap: 2047992 4496 2043496 # vmstat -a procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 0 4496 4352 4548 6556 1 1 399 80 1517 162 2 2 88 8 # cat /proc/meminfo MemTotal: 1027016 kB MemFree: 4352 kB Buffers: 1008 kB Cached: 7316 kB SwapCached: 1148 kB Active: 6528 kB Inactive: 4536 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1027016 kB LowFree: 4352 kB SwapTotal: 2047992 kB SwapFree: 2043496 kB Dirty: 236 kB Writeback: 0 kB Mapped: 5296 kB Slab: 14388 kB Committed_AS: 535496 kB PageTables: 494900 kB VmallocTotal: 536870911 kB VmallocUsed: 1568 kB VmallocChunk: 536869323 kB HugePages_Total: 0 HugePages_Free: 0 Hugepagesize: 2048 kB # ps uaxwwf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 3472 428 ? S Sep12 0:01 init [3] root 2 0.0 0.0 0 0 ? SWN Sep12 0:00 [ksoftirqd/0] root 3 0.0 0.0 0 0 ? SW< Sep12 0:00 [events/0] root 4 0.0 0.0 0 0 ? SW< Sep12 0:00 _ [khelper] root 5 0.0 0.0 0 0 ? SW< Sep12 0:00 _ [kacpid] root 30 0.0 0.0 0 0 ? SW< Sep12 0:00 _ [kblockd/0] root 44 0.0 0.0 0 0 ? SW Sep12 0:00 _ [pdflush] root 45 0.0 0.0 0 0 ? SW Sep12 0:02 _ [pdflush] root 47 0.0 0.0 0 0 ? SW< Sep12 0:00 _ [aio/0] root 186 0.0 0.0 0 0 ? SW< Sep12 0:00 _ [ata/0] root 31 0.0 0.0 0 0 ? SW Sep12 0:00 [khubd] root 46 0.0 0.0 0 0 ? SW Sep12 0:01 [kswapd0] root 151 0.0 0.0 0 0 ? SW Sep12 0:00 [kseriod] root 188 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_0] root 189 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_1] root 204 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] root 339 0.0 0.0 2336 216 ? S< Sep12 0:00 udevd root 896 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] root 897 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] root 898 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] root 899 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald] root 1637 0.0 0.0 0 0 ? SW< Sep12 0:00 [krfcommd] root 1946 0.0 0.0 18104 748 ? S Sep12 0:00 /usr/sbin/sshd root 5189 0.0 0.1 37540 1056 ? S 02:04 0:00 _ sshd: root@pts/0 root 5195 0.0 0.0 45656 1020 pts/0 S 02:04 0:00 | _ -bash root 5255 0.0 0.1 104764 1892 pts/0 S 02:04 0:00 | _ gkrellm root 29075 0.0 0.0 44836 500 pts/0 S 02:38 0:00 | _ sleep 10 root 6119 0.0 0.0 37284 1020 ? S 02:19 0:00 _ sshd: root@pts/1 root 6133 0.0 0.1 45656 1120 pts/1 S 02:19 0:00 | _ -bash root 29079 0.0 0.0 44476 924 pts/1 S 02:38 0:00 | _ /bin/sh ./memory.sh root 29083 0.0 0.0 5228 784 pts/1 R 02:38 0:00 | _ ps uaxwwf root 6193 0.0 0.0 37284 1020 ? S 02:20 0:00 _ sshd: root@pts/2 root 6212 0.0 0.1 45656 1136 pts/2 S 02:20 0:00 | _ -bash root 29077 0.0 0.1 35936 1932 ? S 02:38 0:00 _ sshd: bin [priv] sshd 29078 0.0 0.1 19448 1120 ? S 02:38 0:00 _ sshd: bin [net] root 2542 0.0 0.0 2344 272 tty1 S Sep12 0:00 /sbin/mingetty tty1 root 2543 0.0 0.0 2344 272 tty2 S Sep12 0:00 /sbin/mingetty tty2 root 2544 0.0 0.0 2344 272 tty3 S Sep12 0:00 /sbin/mingetty tty3 root 2545 0.0 0.0 2344 276 tty4 S Sep12 0:00 /sbin/mingetty tty4 root 2546 0.0 0.0 2344 276 tty5 S Sep12 0:00 /sbin/mingetty tty5 root 2547 0.0 0.0 2344 276 tty6 S Sep12 0:00 /sbin/mingetty tty6