swapping madness, 2.6.8-1.521smp

[Jamie Zawinski]

Since I upgraded from RH9 to FC2, I've been having this problem, 
roughly twice daily: I'll come home, and my machine is swapping 
madly.  Logging in from another machine shows something like:

    top - 11:07:41 up 11:58,  3 users,  load average: 9.64, 9.01, 7.60
    Tasks: 134 total,   2 running, 129 sleeping,   0 stopped,   3 zombie
    Cpu(s): 20.1% us,  5.6% sy,  3.6% ni, 36.7% id, 33.8% wa,  0.0% hi,  0.2% si
    Mem:   1034404k total,  1029672k used,     4732k free,      744k buffers
    Swap:  1028152k total,   267844k used,   760308k free,     8456k cached

Now here's the weird thing: I can't tell which process is holding
on to nearly 1GB of memory.

If I kill X, everything settles down and goes back to normal; so
that suggests that the culprit is the X server or some client of
it.  But I can't tell which.

Top (and ps, and /dev/*/map) says that the X server, in times like
this, is only around 150MB.  It is the largest process.  Number two
is Mozilla, at a svelte 89MB.  

Killing Mozilla alone does not fix matters.  I spent some time 
killing off other X clients one at a time, but I wasn't totally
methodical about it, so I didn't find a culprit that way either.

So, clearly some program is going nuts on me here, but the really
weird thing is that I can't tell who all the memory belongs to.

Someone suggested trying "echo 100 > /proc/sys/vm/swappiness" to
see what happens, but that didn't cause any change that I could see.

Any suggestions for what I should look for the next time this 
happens?  Pretty much guarenteed that it will recur within 12 
or 14 hours.

FC2; I've seen this on both kernel-smp-2.6.7-1.494.2.2 and 
kernel-smp-2.6.8-1.521.  Two "AMD Athlon(tm) MP 2800+" on a
GA-7DPXDW-P mobo.

Thanks,

-- 
Jamie Zawinski
jwz(a)jwz.org             http://www.jwz.org/
jwz(a)dnalounge.com       http://www.dnalounge.com/

[Jason L Tibbitts III]

> >>>> "JZ" == Jamie Zawinski <jwz(a)jwz.org&gt;
> writes: 
JZ> If I kill X, everything settles down and goes back to normal; so
JZ> that suggests that the culprit is the X server or some client of
JZ> it.  But I can't tell which.

Does xrestop give you any useful information?

 - J<

[Jamie Zawinski]

Jason L Tibbitts III wrote:
>  
>  >>>>> "JZ" == Jamie Zawinski <jwz(a)jwz.org&gt; writes:
>  
>  JZ> If I kill X, everything settles down and goes back to normal; so
>  JZ> that suggests that the culprit is the X server or some client of
>  JZ> it.  But I can't tell which.
>  
>  Does xrestop give you any useful information? 
Not that I could tell.  But anyway, that would only show the breakdown
within the X process, which was only around 150MB.

The really weird part here is that the memory usage reported by top does
not add up to 1GB.

-- 
Jamie Zawinski
jwz(a)jwz.org             http://www.jwz.org/
jwz(a)dnalounge.com       http://www.dnalounge.com/

[Alan Cox]

On Fri, Aug 20, 2004 at 03:55:46PM -0700, Jamie Zawinski wrote:
>  Not that I could tell.  But anyway, that would only show the
> breakdown
>  within the X process, which was only around 150MB.
>  
>  The really weird part here is that the memory usage reported by top does
>  not add up to 1GB. 
I'd expect that. X for example has virtual mappings for the video RAM so
can easily be 128Mb of resident pages that are not actually main memory.
Direct render clients can do likewise.