10:23 p.m.
Hi,
On August 8 I added a bugreport for a machine that trashes aprox. every 2
weeks because of the kernel eating memory. You can find it here:
http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=97546
I've also made some graphs available together with some command output
that clearly show it is a kernel problem:
http://dag.wieers.com/rmon-breeg-io-3months-800x120.png
http://dag.wieers.com/rmon-breeg-kernel-3months-800x120.png
http://dag.wieers.com/rmon-breeg-load-3months-800x120.png
http://dag.wieers.com/rmon-breeg-paging-3months-800x120.png
http://dag.wieers.com/rmon-breeg-swap-3months-800x120.png
http://dag.wieers.com/rmon-breeg-mem-3months-800x120.png
The behaviour of the system has been like this since I installed RH9
(kernel 2.4.20-8) and newer kernels (2.4.20-18.9, 2.4.20-19.9 and
2.4.20-20.9) and continues to act like this ;(
I know at least 2 other persons that have the same problems (on different
machines with a similar workload/functionality) so it seems to be a bug in
the kernel on a particular use of the VM.
Anyone else noticed something like this ?
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
10:34 p.m.
On Mon, 8 Sep 2003, Dag Wieers wrote:
On August 8 I added a bugreport for a machine that trashes aprox.
every 2
weeks because of the kernel eating memory. You can find it here:
I know at least 2 other persons that have the same problems (on
different
machines with a similar workload/functionality) so it seems to be a bug in
the kernel on a particular use of the VM.
The thing is, RHL8 and 9 have a completely different VM from
Severn. Also, many people cannot reproduce this problem at
all.
This, together with your graphs, suggests it's more likely
to be a memory leak in some driver then a bug in the core VM
code.
Do you have anything suspicious in /proc/slabinfo when your
system gets close to crashing ?
In the RHL9 and RHL8 kernels, how big is free + active +
inactive* together? How much space is unaccounted for ?
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
11:50 p.m.
On Sun, 7 Sep 2003, Rik van Riel wrote:
On Mon, 8 Sep 2003, Dag Wieers wrote:
> On August 8 I added a bugreport for a machine that trashes aprox. every 2
> weeks because of the kernel eating memory. You can find it here:
> I know at least 2 other persons that have the same problems (on different
> machines with a similar workload/functionality) so it seems to be a bug in
> the kernel on a particular use of the VM.
The thing is, RHL8 and 9 have a completely different VM from
Severn. Also, many people cannot reproduce this problem at
all.
The best thing would be to try out the Severn kernel on RH9 ?
I cannot simply install Severn on this system ;/
This, together with your graphs, suggests it's more likely
to be a memory leak in some driver then a bug in the core VM
code.
With driver you don't mean a loaded module ? Is there some way to find out
what driver/module is allocating this memory ?
Do you have anything suspicious in /proc/slabinfo when your
system gets close to crashing ?
It's now up 7 days (36MB used, 25MB free) so we'll see in another
5 days. These are the higher numbers:
inode_cache 812 1232 480 154 154 1
dentry_cache 572 1380 128 46 46 1
filp 620 630 128 21 21 1
buffer_head 4349 12363 100 162 317 1
mm_struct 6038 6069 224 356 357 1
vm_area_struct 1315 4920 96 38 123 1
pte_chain 729 3277 32 14 29 1
I'm not sure what I have to look for. I guess I better save this also
directly after booting up.
In the RHL9 and RHL8 kernels, how big is free + active +
inactive* together? How much space is unaccounted for ?
I'm not sure how it all adds up. The information I get from ps/top for
all the processes is almost constant (around 12MB) whereas slowly each day
+-3.5MB extra seems to be unaccounted for.
This is after 7 days running:
[root@breeg root]# free
total used free shared buffers cached
Mem: 61676 60648 1028 0 2212 20416
-/+ buffers/cache: 38020 23656
Swap: 457836 8876 448960
[root@breeg root]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 63156224 62107648 1048576 0 2277376 25321472
Swap: 468824064 9084928 459739136
MemTotal: 61676 kB
MemFree: 1024 kB
MemShared: 0 kB
Buffers: 2224 kB
Cached: 20416 kB
SwapCached: 4312 kB
Active: 20688 kB
ActiveAnon: 7660 kB
ActiveCache: 13028 kB
Inact_dirty: 0 kB
Inact_laundry: 7036 kB
Inact_clean: 580 kB
Inact_target: 5660 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 61676 kB
LowFree: 1024 kB
SwapTotal: 457836 kB
SwapFree: 448964 kB
Kind regards,
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
9:09 a.m.
On Mon, 8 Sep 2003, Dag Wieers wrote:
On Sun, 7 Sep 2003, Rik van Riel wrote:
> This, together with your graphs, suggests it's more likely
> to be a memory leak in some driver then a bug in the core VM
> code.
With driver you don't mean a loaded module ? Is there some way to find
out what driver/module is allocating this memory ?
There's no really easy way. The best way would be to look at
which drivers you are using and compare that list with the
drivers being used by the other people who have this problem.
Then look at people who don't have the problem at all. Scratch
the drivers on problemless systems from the list of suspects.
Hopefully, you'll end up with just one or a few suspect drivers...
> Do you have anything suspicious in /proc/slabinfo when your
> system gets close to crashing ?
Hmmm ok, so nothing bad in /proc/slabinfo.
> In the RHL9 and RHL8 kernels, how big is free + active +
> inactive* together? How much space is unaccounted for ?
I'm not sure how it all adds up. The information I get from ps/top for
all the processes is almost constant (around 12MB) whereas slowly each day
+-3.5MB extra seems to be unaccounted for.
This is after 7 days running:
MemTotal: 61676 kB
MemFree: 1024 kB
Active: 20688 kB
Inact_dirty: 0 kB
Inact_laundry: 7036 kB
Inact_clean: 580 kB
OK, that's 1 + 20.5 + 7 + .5 = 29 MB pageable memory.
In other words, 30+ MB is already taken by the kernel!
Definately looks like a memory leak.
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
8 p.m.
On Mon, 8 Sep 2003, Rik van Riel wrote:
On Mon, 8 Sep 2003, Dag Wieers wrote:
> On Sun, 7 Sep 2003, Rik van Riel wrote:
> > This, together with your graphs, suggests it's more likely
> > to be a memory leak in some driver then a bug in the core VM
> > code.
>
> With driver you don't mean a loaded module ? Is there some way to find
> out what driver/module is allocating this memory ?
There's no really easy way. The best way would be to look at
which drivers you are using and compare that list with the
drivers being used by the other people who have this problem.
Ok, this is fairly easy for the modules: ext3 and jbd
Since one of the other machines crashing was a tokenring, I'm sure it's
not the mii module or any of the NIC drivers. And no other modules match
the other systems.
One of the other persons (and I hope they'll join this discussion soon)
reported he already tried running the same setup on three different
systems (completely different HW). And the system trashes every 6 to 8
hours (he now manages to work around it by rebooting often).
Then look at people who don't have the problem at all. Scratch
the drivers on problemless systems from the list of suspects.
Well, I'm almost certain that doing the same stuff on almost any RH9
system will have it trash (either slow or fast depending on the amount of
memory and the IO). So it may be memory-usage related or IO-utilisation
related.
Maybe IDE ? I'm not sure if all the machines are IDE.
Thanks for your time,
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
9:20 p.m.
On Mon, 2003-09-08 at 07:09, Rik van Riel wrote:
> MemTotal: 61676 kB
> MemFree: 1024 kB
> Active: 20688 kB
> Inact_dirty: 0 kB
> Inact_laundry: 7036 kB
> Inact_clean: 580 kB
OK, that's 1 + 20.5 + 7 + .5 = 29 MB pageable memory.
In other words, 30+ MB is already taken by the kernel!
Definately looks like a memory leak.
Would it be normal on a machine with 512MB memory to have a kernel using
this much memory? Mine's ~33MB:
MemTotal: 513284 kB
MemFree: 36460 kB
...
Active: 362564 kB
...
Inact_dirty: 3080 kB
Inact_laundry: 65652 kB
Inact_clean: 10916 kB
...
This is
$ uptime
19:16:56 up 2 days, 1:43, 4 users, load average: 0.20, 0.12, 0.05
$ uname -a
Linux denk.nakedape.priv 2.4.20-19.9 #1 Tue Jul 15 17:18:13 EDT 2003 i686 i686 i386
GNU/Linux
Wil
--
Wil Cooley wcooley(a)nakedape.cc
Naked Ape Consulting http://nakedape.cc
* * * * * * Linux Services for Small Businesses * * * * * *
* Easy, reliable solutions for small businesses *
* Naked Ape Business Server http://nakedape.cc/r/sms *
5:40 p.m.
On Mon, 8 Sep 2003, Rik van Riel wrote:
Definately looks like a memory leak.
Would it help if I say I have a feeling it happens when running out of
physical memory ? Or when allocating/using a lot of memory in a short
timespan and releasing it under this condition.
I removed memory from my desktop system and I seem to suffer from it on my
desktop too (albeit in a less profound way).
I can also add that it wasn't present in 2.4.18-18.7.x (or earlier)
kernels, but it started when I upgraded to RH9 with kernel 2.4.20-8.
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
9:25 p.m.
On Fri, 12 Sep 2003, Dag Wieers wrote:
On Mon, 8 Sep 2003, Rik van Riel wrote:
> Definately looks like a memory leak.
Would it help if I say I have a feeling it happens when running out of
physical memory ? Or when allocating/using a lot of memory in a short
timespan and releasing it under this condition.
This output is generated while it was trashing heavily:
[root@breeg root]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 63156224 62545920 610304 0 733184 6836224
Swap: 468824064 21938176 446885888
MemTotal: 61676 kB
MemFree: 596 kB
MemShared: 0 kB
Buffers: 716 kB
Cached: 4704 kB
SwapCached: 1972 kB
Active: 4912 kB
ActiveAnon: 3728 kB
ActiveCache: 1184 kB
Inact_dirty: 1104 kB
Inact_laundry: 528 kB
Inact_clean: 964 kB
Inact_target: 1500 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 61676 kB
LowFree: 596 kB
SwapTotal: 457836 kB
SwapFree: 436412 kB
Is there anything I can try to help you find the cause ? This machine has
only ext3 and jbd loaded (that match other machines with the same
problems).
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
3:26 p.m.
On Wed, 17 Sep 2003, Dag Wieers wrote:
MemTotal: 61676 kB
MemFree: 596 kB
Active: 4912 kB
Inact_dirty: 1104 kB
Inact_laundry: 528 kB
Inact_clean: 964 kB
Oh dear, that is only 8MB free&pageable memory, out of 61MB
total available memory after bootup...
Is there anything I can try to help you find the cause ? This
machine
has only ext3 and jbd loaded (that match other machines with the same
problems).
Exactly which kernel are you running ?
I wonder if there's some leak in reclaiming ext3 pages
after a truncate. Maybe they get erroneously removed
from the pageout lists and we never find them...
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
12:30 p.m.
On Thu, 18 Sep 2003, Rik van Riel wrote:
On Wed, 17 Sep 2003, Dag Wieers wrote:
> MemTotal: 61676 kB
> MemFree: 596 kB
> Active: 4912 kB
> Inact_dirty: 1104 kB
> Inact_laundry: 528 kB
> Inact_clean: 964 kB
Oh dear, that is only 8MB free&pageable memory, out of 61MB
total available memory after bootup...
> Is there anything I can try to help you find the cause ? This machine
> has only ext3 and jbd loaded (that match other machines with the same
> problems).
Exactly which kernel are you running ?
2.4.20-19.9 currently, but I have the same effect with 2.4.20-18 and
2.4.20-20. The graph is exactly the same as it was weeks ago (every 2
weeks this happens, only this time the system was responsive enough to
return the information ;)
If I go to runlevel 1 when the system isn't trashing (with only bash and
init running) the memory isn't freed (and not claimed by bash or init).
I wonder if there's some leak in reclaiming ext3 pages
after a truncate. Maybe they get erroneously removed
from the pageout lists and we never find them...
Well, the system is running rrdtool for about 100 counters (2 machines)
every 5 minutes. But once every evening (at 22h) it generates graphs and
html-pages of all the data (+- 10months of data) in the rrd's and that
seems to be causing the kernel to leak memory as is clearly shown on these
older graphs:
http://dag.wieers.com/rmon-breeg-mem-2weeks-200x80.png
http://dag.wieers.com/rmon-breeg-mem-3months-800x120.png
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
1:58 p.m.
Hi,
On Fri, 2003-09-19 at 18:30, Dag Wieers wrote:
> Exactly which kernel are you running ?
2.4.20-19.9 currently, but I have the same effect with 2.4.20-18 and
2.4.20-20. The graph is exactly the same as it was weeks ago (every 2
weeks this happens, only this time the system was responsive enough to
return the information ;)
We recently found one problem which could affect the size of the inode
cache. It's not exactly a leak, because the resources can still be
reclaimed, but the inodes were filed away in a place where the VM was
least likely to go looking to reclaim them.
Could you "cat /proc/slabinfo" on the affected systems and see what the
inode_cache entry looks like, please?
Cheers,
Stephen
2:07 p.m.
On Fri, 2003-09-19 at 14:58, Stephen C. Tweedie wrote:
Hi,
On Fri, 2003-09-19 at 18:30, Dag Wieers wrote:
> > Exactly which kernel are you running ?
>
> 2.4.20-19.9 currently, but I have the same effect with 2.4.20-18 and
> 2.4.20-20. The graph is exactly the same as it was weeks ago (every 2
> weeks this happens, only this time the system was responsive enough to
> return the information ;)
We recently found one problem which could affect the size of the inode
cache. It's not exactly a leak, because the resources can still be
reclaimed, but the inodes were filed away in a place where the VM was
least likely to go looking to reclaim them.
Could you "cat /proc/slabinfo" on the affected systems and see what the
inode_cache entry looks like, please?
I'm seeing this behavior under 7.3 using kernel 2.4.20-18.7
here is inode_cache from that machine:
inode_cache 20400 21014 512 3002 3002 1
Is this related to this bug:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=100680
at all?
-sv
2:31 p.m.
Hi,
On Fri, 2003-09-19 at 20:07, seth vidal wrote:
> Could you "cat /proc/slabinfo" on the affected systems
and see what the
> inode_cache entry looks like, please?
I'm seeing this behavior under 7.3 using kernel 2.4.20-18.7
here is inode_cache from that machine:
inode_cache 20400 21014 512 3002 3002 1
64MB ram, and you've got 12MB (4k * 3002) already in the inode cache.
The bug we just fixed could easily be causing your problems.
If those numbers continue to rise directly as performance degrades, then
that's almost certainly the cause.
Is this related to this bug:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=100680
No, I don't think so.
Cheers,
Stephen
2:33 p.m.
On Fri, 2003-09-19 at 15:31, Stephen C. Tweedie wrote:
Hi,
On Fri, 2003-09-19 at 20:07, seth vidal wrote:
> > Could you "cat /proc/slabinfo" on the affected systems and see what
the
> > inode_cache entry looks like, please?
> I'm seeing this behavior under 7.3 using kernel 2.4.20-18.7
> here is inode_cache from that machine:
>
> inode_cache 20400 21014 512 3002 3002 1
64MB ram, and you've got 12MB (4k * 3002) already in the inode cache.
The bug we just fixed could easily be causing your problems.
well on this system - it's got 1GB of ram.
How should I read that line?
If those numbers continue to rise directly as performance degrades,
then
that's almost certainly the cause.
I'll keep an eye on it.
Thanks
-sv
3:20 p.m.
On 19 Sep 2003, Stephen C. Tweedie wrote:
On Fri, 2003-09-19 at 18:30, Dag Wieers wrote:
> > Exactly which kernel are you running ?
>
> 2.4.20-19.9 currently, but I have the same effect with 2.4.20-18 and
> 2.4.20-20. The graph is exactly the same as it was weeks ago (every 2
> weeks this happens, only this time the system was responsive enough to
> return the information ;)
We recently found one problem which could affect the size of the inode
cache. It's not exactly a leak, because the resources can still be
reclaimed, but the inodes were filed away in a place where the VM was
least likely to go looking to reclaim them.
Could you "cat /proc/slabinfo" on the affected systems and see what the
inode_cache entry looks like, please?
The system has been rebooted 2 days ago. The effect will be much bigger if
I at least wait a week.
This is what I pasted earlier on Rik's request:
| > Do you have anything suspicious in /proc/slabinfo when your
| > system gets close to crashing ?
|
| It's now up 7 days (36MB used, 25MB free) so we'll see in another
| 5 days. These are the higher numbers:
|
| inode_cache 812 1232 480 154 154 1
| dentry_cache 572 1380 128 46 46 1
| filp 620 630 128 21 21 1
| buffer_head 4349 12363 100 162 317 1
| mm_struct 6038 6069 224 356 357 1
| vm_area_struct 1315 4920 96 38 123 1
| pte_chain 729 3277 32 14 29 1
|
| I'm not sure what I have to look for. I guess I better save this also
| directly after booting up.
I'll give you some other numbers later. If there's some more stuff you
need from that system just before it is trashing, please tell now ;)
Thanks,
-- dag wieers, dag(a)wieers.com, http://dag.wieers.com/ --
[Any errors in spelling, tact or fact are transmission errors]
9:13 a.m.
FYI: kernel <= 2.4.20-20.9 is not affected by the refile_inode bug
Am Fr, 2003-09-19 um 20.58 schrieb Stephen C. Tweedie:
Hi,
On Fri, 2003-09-19 at 18:30, Dag Wieers wrote:
> > Exactly which kernel are you running ?
>
> 2.4.20-19.9 currently, but I have the same effect with 2.4.20-18 and
> 2.4.20-20. The graph is exactly the same as it was weeks ago (every 2
> weeks this happens, only this time the system was responsive enough to
> return the information ;)
We recently found one problem which could affect the size of the inode
cache. It's not exactly a leak, because the resources can still be
reclaimed, but the inodes were filed away in a place where the VM was
least likely to go looking to reclaim them.
Could you "cat /proc/slabinfo" on the affected systems and see what the
inode_cache entry looks like, please?
Cheers,
Stephen
--
Rhl-devel-list mailing list
Rhl-devel-list(a)redhat.com
http://www.redhat.com/mailman/listinfo/rhl-devel-list
--
Harald Hoyer, Senior Software Engineer Harald.Hoyer(a)redhat.de
Red Hat GmbH Tel. : +49-711-96437-0
D-70178 Stuttgart, Germany Web : http://www.redhat.de
gpg fingerprint E930 20E6 CCF8 C76C 8582 CF9F B7B7 45C2 C557 5542
4:23 p.m.
> Would it help if I say I have a feeling it happens when running
out of
> physical memory ? Or when allocating/using a lot of memory in a short
> timespan and releasing it under this condition.
I am having this exact same problem, with a machine that has 2G of swap
and 768M of physical memory. I filed a bug in Red Hat Bugzilla, and was
told to produce the output of /proc/slabinfo, but it takes the machine a
while to consume all of its resources to the point where everything
essentially is being run out of the swap file.
I will produce the Bug number once Bugzilla is alive again.
--
Michael Lee Yohe <michael.yohe(a)us.army.mil>
U.S. Army Aviation and Missile Command Software Engineering Directorate
7520
days inactive
7536
days old
16 comments
7 participants
participants (7)
-
Dag Wieers -
Harald Hoyer -
Michael Lee Yohe -
Rik van Riel -
seth vidal -
Stephen C. Tweedie -
Wil Cooley