Memory loss after long uptime

[Konstantin Svist]

Hi all,

I have a strange recurring problem on some of my servers, maybe someone 
can help me figure out what might be causing it.

After running a MySQL machine with fairly high load for a while 
(~month), RAM usage in top stops making sense. Normally, RES column 
accounts for everything currently present in RAM (not swap) and 
corresponds pretty well with mem_used-buffers-cache. This is always the 
case after a fresh boot and seems to be the case on most other servers.
But that's not the case here:

59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G) 
supposedly used by all processes
sum(RES column from top) ~= 35127296k (33.5G)

So where's the other 21G??

I'm pretty sure I'm not nitpicking here, this is >30% of total system 
RAM unaccounted for. I've tried stopping all non-OS specific processes 
(and restarting some services that seemed to eat more RAM that they 
should have (irqbalance)) -- and memory is not reclaimed.

Over the few years that I've seen this problem, I've already replaced 
all the hardware and upgraded Fedora from 8 to 14 (currently running 
2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of 
improvement.

Interestingly, another machine with same hardware & OS runs MySQL in 
slave mode to replicate the DB -- that machine has uptime of 134 days 
and does not exhibit the same symptoms. In fact, here is its mem footprint:

63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
sum(RES column from top) ~= 45.5G (which makes sense since a few 
RAM-hungry processes share memory)

Please help!