Memory loss after long uptime

[Konstantin Svist]

Hi all,

I have a strange recurring problem on some of my servers, maybe someone 
can help me figure out what might be causing it.

After running a MySQL machine with fairly high load for a while 
(~month), RAM usage in top stops making sense. Normally, RES column 
accounts for everything currently present in RAM (not swap) and 
corresponds pretty well with mem_used-buffers-cache. This is always the 
case after a fresh boot and seems to be the case on most other servers.
But that's not the case here:

59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G) 
supposedly used by all processes
sum(RES column from top) ~= 35127296k (33.5G)

So where's the other 21G??

I'm pretty sure I'm not nitpicking here, this is >30% of total system 
RAM unaccounted for. I've tried stopping all non-OS specific processes 
(and restarting some services that seemed to eat more RAM that they 
should have (irqbalance)) -- and memory is not reclaimed.

Over the few years that I've seen this problem, I've already replaced 
all the hardware and upgraded Fedora from 8 to 14 (currently running 
2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of 
improvement.

Interestingly, another machine with same hardware & OS runs MySQL in 
slave mode to replicate the DB -- that machine has uptime of 134 days 
and does not exhibit the same symptoms. In fact, here is its mem footprint:

63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
sum(RES column from top) ~= 45.5G (which makes sense since a few 
RAM-hungry processes share memory)

Please help!

[Rick Stevens]

On 04/27/2012 02:10 PM, Konstantin Svist wrote:
>  Hi all,
>
>  I have a strange recurring problem on some of my servers, maybe someone
>  can help me figure out what might be causing it.
>
>  After running a MySQL machine with fairly high load for a while
>  (~month), RAM usage in top stops making sense. Normally, RES column
>  accounts for everything currently present in RAM (not swap) and
>  corresponds pretty well with mem_used-buffers-cache. This is always the
>  case after a fresh boot and seems to be the case on most other servers.
>  But that's not the case here:
>
>  59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G)
>  supposedly used by all processes
>  sum(RES column from top) ~= 35127296k (33.5G)
>
>  So where's the other 21G??
>
>  I'm pretty sure I'm not nitpicking here, this is >30% of total system
>  RAM unaccounted for. I've tried stopping all non-OS specific processes
>  (and restarting some services that seemed to eat more RAM that they
>  should have (irqbalance)) -- and memory is not reclaimed.
>
>  Over the few years that I've seen this problem, I've already replaced
>  all the hardware and upgraded Fedora from 8 to 14 (currently running
>  2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of
>  improvement.
>
>  Interestingly, another machine with same hardware & OS runs MySQL in
>  slave mode to replicate the DB -- that machine has uptime of 134 days
>  and does not exhibit the same symptoms. In fact, here is its mem footprint:
>
>  63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
>  sum(RES column from top) ~= 45.5G (which makes sense since a few
>  RAM-hungry processes share memory) 
We've seen this before in some of our MySQL platforms. It appears that
some of the older MySQL servers hemorrhage shared memory segments which
will not be reclaimed when the process is terminated.

You might try having a look at the output of ipcs after stopping MySQL
and see if your missing memory is in one or more of the shm segments.
If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be
surprised at how many programs don't release IPC resources--especially
if they are rudely terminated (e.g. SIGSEGV or SIGKILL).

Also have a serious look at updating your MySQL platform if possible.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
-                                                                    -
-     Try to look unimportant.  The bad guys may be low on ammo.     -
----------------------------------------------------------------------

[Konstantin Svist]

On 04/27/2012 03:52 PM, Rick Stevens wrote:
>  On 04/27/2012 02:10 PM, Konstantin Svist wrote:
> > Hi all,
> >
> > I have a strange recurring problem on some of my servers, maybe someone
> > can help me figure out what might be causing it.
> >
> > After running a MySQL machine with fairly high load for a while
> > (~month), RAM usage in top stops making sense. Normally, RES column
> > accounts for everything currently present in RAM (not swap) and
> > corresponds pretty well with mem_used-buffers-cache. This is always the
> > case after a fresh boot and seems to be the case on most other servers.
> > But that's not the case here:
> >
> > 59098540k used - 1633832k cached - 25824k buffers = 57438884 (54.7G)
> > supposedly used by all processes
> > sum(RES column from top) ~= 35127296k (33.5G)
> >
> > So where's the other 21G??
> >
> > I'm pretty sure I'm not nitpicking here, this is >30% of total system
> > RAM unaccounted for. I've tried stopping all non-OS specific processes
> > (and restarting some services that seemed to eat more RAM that they
> > should have (irqbalance)) -- and memory is not reclaimed.
> >
> > Over the few years that I've seen this problem, I've already replaced
> > all the hardware and upgraded Fedora from 8 to 14 (currently running
> > 2.6.35.14-95.fc14.x86_64) and MySQL and other code without any sign of
> > improvement.
> >
> > Interestingly, another machine with same hardware & OS runs MySQL in
> > slave mode to replicate the DB -- that machine has uptime of 134 days
> > and does not exhibit the same symptoms. In fact, here is its mem 
> > footprint:
> >
> > 63166496k used - 19786032k cached - 2881484k buffers = 40498980k (38.6G)
> > sum(RES column from top) ~= 45.5G (which makes sense since a few
> > RAM-hungry processes share memory)
>
>  We've seen this before in some of our MySQL platforms. It appears that
>  some of the older MySQL servers hemorrhage shared memory segments which
>  will not be reclaimed when the process is terminated.
>
>  You might try having a look at the output of ipcs after stopping MySQL
>  and see if your missing memory is in one or more of the shm segments.
>  If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be
>  surprised at how many programs don't release IPC resources--especially
>  if they are rudely terminated (e.g. SIGSEGV or SIGKILL).
>
>  Also have a serious look at updating your MySQL platform if possible.
>  ----------------------------------------------------------------------
>  - Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
>  - AIM/Skype: therps2        ICQ: 22643734            Yahoo: origrps2 -
>  -                                                                    -
>  -     Try to look unimportant.  The bad guys may be low on ammo.     -
>  ---------------------------------------------------------------------- 

Thanks Rick,

I tried ipcs on a few machines with same symptoms, but it doesn't look 
like there's a lot of shared memory mentioned there.
The MySQL machine had already been rebooted, so I'll keep checking it... 
but doesn't look promising so far.

[Chris Adams]

Once upon a time, Rick Stevens <ricks(a)alldigital.com&gt; said:
>  You might try having a look at the output of ipcs after stopping
> MySQL
>  and see if your missing memory is in one or more of the shm segments.
>  If so, you can reclaim it by using "ipcrm -m <shmid>". You'd be
>  surprised at how many programs don't release IPC resources--especially
>  if they are rudely terminated (e.g. SIGSEGV or SIGKILL). 
Are you sure?  I've been running MySQL for a long time, and I don't
remember it ever using SysV IPC.  I certainly don't see it using that
now, even on an old version I still have running (much older than the
OP's F14).

To the OP: if you think you are seeing RAM in use that isn't reflected
when comparing the output of "free" to "ps"/"top" process
usage, it
could be in other kernel buffers.  Check out "slabtop" (has to run as
root); there are other kernel caches that "free" doesn't know about,
especially the dentry and inode caches.  These will show up as just more
kernel RAM in use, but really they are caches that should be discarded
as needed (just like the old buffers/cache lines in "free").
-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Konstantin Svist]

On 04/27/2012 10:02 PM, Chris Adams wrote:
>  Once upon a time, Rick Stevens<ricks(a)alldigital.com>  said:
> > You might try having a look at the output of ipcs after stopping MySQL
> > and see if your missing memory is in one or more of the shm segments.
> > If so, you can reclaim it by using "ipcrm -m<shmid>". You'd be
> > surprised at how many programs don't release IPC resources--especially
> > if they are rudely terminated (e.g. SIGSEGV or SIGKILL).
>  Are you sure?  I've been running MySQL for a long time, and I don't
>  remember it ever using SysV IPC.  I certainly don't see it using that
>  now, even on an old version I still have running (much older than the
>  OP's F14).
>
>  To the OP: if you think you are seeing RAM in use that isn't reflected
>  when comparing the output of "free" to "ps"/"top" process
> usage, it
>  could be in other kernel buffers.  Check out "slabtop" (has to run as
>  root); there are other kernel caches that "free" doesn't know about,
>  especially the dentry and inode caches.  These will show up as just more
>  kernel RAM in use, but really they are caches that should be discarded
>  as needed (just like the old buffers/cache lines in "free"). 

Here's slabtop from one machine that's using too much RAM:

  Active / Total Objects (% used)    : 8516783 / 8967394 (95.0%)
  Active / Total Slabs (% used)      : 270701 / 270701 (100.0%)
  Active / Total Caches (% used)     : 68 / 101 (67.3%)
  Active / Total Size (% used)       : 5113995.32K / 5341810.93K (95.7%)
  Minimum / Average / Maximum Object : 0.01K / 0.59K / 8.00K

   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
3836868 3691157  96%    1.00K 119903       32   3836896K kmalloc-1024
3624368 3573469  98%    0.25K 113262       32    906096K kmalloc-256
570560 553390  96%    0.50K  17830       32    285280K kmalloc-512
197526 196494  99%    0.55K   3468       57    110976K radix_tree_node
140992 114638  81%    0.12K   4406       32     17624K kmalloc-128
113856  29876  26%    0.06K   1779       64      7116K kmalloc-64
  40448  40445  99%    0.01K     79      512       316K kmalloc-8
  35904  35485  98%    0.08K    704       51      2816K sysfs_dir_cache
  32724  26778  81%    1.69K   1818       18     58176K TCP
  29190  16929  57%    0.19K    695       42      5560K dentry
  26112  16386  62%    0.02K    102      256       408K ext4_io_page
...


Looks like the kmalloc-##s are taking up most of the space.
Question is, what (if anything) can be done about this on a running 
system? Any way to reclaim that memory?