2.6.15-1.1955_FC5 memory leak?

[Russell Coker]

I recently setup a server on the Internet in a fairly default FC5T2 
configuration.  I had it running kernel 2.6.15-1.1955_FC5 for 9 days when I 
noticed that it had apparently run out of kernel memory.  All processes were 
highly swapped and the OOM killer was killing processes (by the time I 
noticed the problem most services had been killed).

The machine is a P3 with 256M of RAM, it has full net access with no firewall 
(iptables is used in a fairly default configuration but there is nothing in 
front of the machine protecting it).

Is there a known kernel memory leak in 2.6.15-1.1955_FC5?

The machine in question has BIND, Postfix, Amavis + clamav, and Postgrey 
installed.  However as it doesn't yet have an MX record pointing to is there 
has been little use (some DNS traffic is all that it would get).

I tried rebooting the machine and it crashed.  I guess that the OOM killer 
killed something that was needed for a reboot.  Tomorrow when I get it 
running again I will upgrade it to the latest kernel.  Let me know if there 
are any tests I should perform if this happens again (unfortunately I can't 
get a net-dump server in there but disk-dump is an option I guess).

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[Ivan Gyurdiev]

>  Is there a known kernel memory leak in 2.6.15-1.1955_FC5?
>     
Is this the stock kernel?
Does it have any 3rd party modules installed?

I ask, because I ran into a memory leak with the 1955 LSPP kernel, with 
madwifi and nvidia installed.
Steve Grubb later said he's found some sort of leak, and it will be fixed.
I am now running stock 1977 kernel, will wait and see if it leaks.

[Russell Coker]

On Monday 27 February 2006 02:52, Ivan Gyurdiev <ivg2(a)cornell.edu&gt; wrote:
>  > Is there a known kernel memory leak in 2.6.15-1.1955_FC5?
>
>  Is this the stock kernel?
>  Does it have any 3rd party modules installed? 
Stock kernel with no 3rd party modules.

>  I ask, because I ran into a memory leak with the 1955 LSPP kernel,
> with
>  madwifi and nvidia installed. 
That's a different thing.  My machine in question has no wifi and no X.

>  Steve Grubb later said he's found some sort of leak, and it will
> be fixed.
>  I am now running stock 1977 kernel, will wait and see if it leaks. 
I'm now running that version too.  I guess I'll have an idea of how good it is 
in about 9 days time.  Of course there is the possibility that the memory 
leak was triggered by some sort of DOS attack in which case the attacker 
might just target someone else's machine and mine might stay working even 
with a buggy kernel.  :(

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[Steve Grubb]

On Sunday 26 February 2006 19:29, Russell Coker wrote:
>  > Steve Grubb later said he's found some sort of leak, and it
> will be
>  > fixed. I am now running stock 1977 kernel, will wait and see if it leaks.
>
>  I'm now running that version too.  I guess I'll have an idea of how good it
>  is in about 9 days time.   
Depending on your system you might not make it 9 days. The best way to avoid 
the memory leak is to disable audit (auditctl -e 0)...especially if its a 
remote computer. oom killer might zap sshd.

>  Of course there is the possibility that the memory leak was triggered
> by
>  some sort of DOS attack in which case the attacker might just target someone
>  else's machine and mine might stay working even with a buggy kernel.  :( 
No, it was 2 mallocs not being freed on some syscalls. lspp.10 kernel is 
building right now and I'll update the yum repo tomorrow am. I might put the 
kernel at http://people.redhat.com/sgrubb/files/lspp in a few minutes where 
it can be retrieved and installed manually.

-Steve

[Ivan Gyurdiev]

>  No, it was 2 mallocs not being freed on some syscalls. lspp.10 kernel
> is 
>  building right now and I'll update the yum repo tomorrow am. I might put the 
>  kernel at http://people.redhat.com/sgrubb/files/lspp in a few minutes where 
>  it can be retrieved and installed manually.
>
>     
Allright, I think the stock kernel (+madwifi/nvidia) does not appear to 
leak memory, although Mono does (or whatever mono is running)... it eats 
20% cpu right when I need them most, and then memory usage slowly climbs 
to 880 MB used over a few days. Killing mono fixes the problem. I'm not 
very happy with the resources it takes to run mono...esp since I don't 
really use f-spot or beagle [ people are way too disorganized, why can't 
they just put their files so they can find them? ]

I will re-verify the leak, and check what mono is running exactly...then 
file a bug.
Will test lspp 10 kernel aftwards to see if that leak is fixed.

[Russell Coker]

On Thursday 02 March 2006 12:52, Ivan Gyurdiev <ivg2(a)cornell.edu&gt; wrote:
>  > No, it was 2 mallocs not being freed on some syscalls. lspp.10
> kernel is
>  > building right now and I'll update the yum repo tomorrow am. I might put
>  > the kernel at http://people.redhat.com/sgrubb/files/lspp in a few minutes
>  > where it can be retrieved and installed manually.
>
>  Allright, I think the stock kernel (+madwifi/nvidia) does not appear to
>  leak memory, although Mono does (or whatever mono is running)... it eats 
Below is the output of running uptime immediately after free on two occasions.  
The buffers, cached, and free memory numbers have decreased.  Does this 
indicate a leak?  The machine is totally idle, I have had a single ssh 
session open for all this time, no-one else has logged in, and it's not being 
used for any server tasks apart from light DNS serving.

[root@othello ~]# free
             total       used       free     shared    buffers     cached
Mem:        255196     250824       4372          0      48520      17900
-/+ buffers/cache:     184404      70792
Swap:      1048568        120    1048448
You have new mail in /var/spool/mail/root
[root@othello ~]# uptime
 18:45:13 up 2 days, 21:44,  1 user,  load average: 0.07, 0.03, 0.01
[root@othello ~]# free
             total       used       free     shared    buffers     cached
Mem:        255196     251544       3652          0       8852      13836
-/+ buffers/cache:     228856      26340
Swap:      1048568        120    1048448
You have new mail in /var/spool/mail/root
[root@othello ~]# uptime
 09:12:44 up 3 days, 12:12,  1 user,  load average: 0.18, 0.11, 0.03
[root@othello ~]# uname -a
Linux othello 2.6.15-1.1986.2.1_FC5.lspp.10 #1 Sun Feb 26 19:07:03 EST 2006 
i686 i686 i386 GNU/Linux
[root@othello ~]#

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[Russell Coker]

On Friday 03 March 2006 09:57, Russell Coker <russell(a)coker.com.au&gt; wrote:
>  Below is the output of running uptime immediately after free on two
>  occasions.   The buffers, cached, and free memory numbers have decreased.
>   Does this indicate a leak?  The machine is totally idle, I have had a
>  single ssh session open for all this time, no-one else has logged in, and
>  it's not being used for any server tasks apart from light DNS serving. 
My machine in question has just run out of memory and crashed.  It had been 
operating OK until I tried to copy a set of Fedora ISO files to it's NFS 
share.  I conclude that the bugs which Steve fixed were not the ones that 
afflict my machine.

I'm now upgrading it to 2.6.15-1.2009.4.2_FC5, I'll see if that fixes it.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[Russell Coker]

On Sunday 05 March 2006 13:58, Russell Coker <russell(a)coker.com.au&gt; wrote:
>  On Friday 03 March 2006 09:57, Russell Coker
> <russell(a)coker.com.au&gt; wrote:
>  > Below is the output of running uptime immediately after free on two
>  > occasions.   The buffers, cached, and free memory numbers have decreased.
>  >  Does this indicate a leak?  The machine is totally idle, I have had a
>  > single ssh session open for all this time, no-one else has logged in, and
>  > it's not being used for any server tasks apart from light DNS serving.
>
>  My machine in question has just run out of memory and crashed.  It had been
>  operating OK until I tried to copy a set of Fedora ISO files to it's NFS
>  share.  I conclude that the bugs which Steve fixed were not the ones that
>  afflict my machine.
>
>  I'm now upgrading it to 2.6.15-1.2009.4.2_FC5, I'll see if that fixes it. 
The machine in question has been running 2.6.15-1.2032.2.3_FC5.lspp.12 for 
over 8 days now with no sign of a memory leak (and incidentally the load on 
the machine has increased).  I believe that 2.6.15-1.2032.2.3_FC5.lspp.12 has 
fixed the memory leak problems I experienced.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page