Memory loss after long uptime

[Rick Stevens]

On 04/27/2012 02:10 PM, Konstantin Svist wrote:
> Hi all,
>
> I have a strange recurring problem on some of my servers, maybe someone
> can help me figure out what might be causing it.
>
> After running a MySQL machine with fairly high load for a while
> (~month), RAM usage in top stops making sense. Normally, RES column
> accounts for everything currently present in RAM (not swap) and
> corresponds pretty well with mem_used-buffers-cache. This is always the
> case after a fresh boot and seems to be the case on most other servers.
> But that's not the case here:
>
> 59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G)
> supposedly used by all processes
> sum(RES column from top) ~= 35127296k (33.5G)
>
> So where's the other 21G??
>
> I'm pretty sure I'm not nitpicking here, this is >30% of total system
> RAM unaccounted for. I've tried stopping all non-OS specific processes
> (and restarting some services that seemed to eat more RAM that they
> should have (irqbalance)) -- and memory is not reclaimed.
>
> Over the few years that I've seen this problem, I've already replaced
> all the hardware and upgraded Fedora from 8 to 14 (currently running
> 2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of
> improvement.
>
> Interestingly, another machine with same hardware & OS runs MySQL in
> slave mode to replicate the DB -- that machine has uptime of 134 days
> and does not exhibit the same symptoms. In fact, here is its mem footprint:
>
> 63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
> sum(RES column from top) ~= 45.5G (which makes sense since a few
> RAM-hungry processes share memory)

We've seen this before in some of our MySQL platforms. It appears that
some of the older MySQL servers hemorrhage shared memory segments which
will not be reclaimed when the process is terminated.

You might try having a look at the output of ipcs after stopping MySQL
and see if your missing memory is in one or more of the shm segments.
If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be
surprised at how many programs don't release IPC resources--especially
if they are rudely terminated (e.g. SIGSEGV or SIGKILL).

Also have a serious look at updating your MySQL platform if possible.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks at alldigital.com -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-     Try to look unimportant.  The bad guys may be low on ammo.     -
----------------------------------------------------------------------