Growing shared memory leak?

[Jordan Metzmeier]

I am trying to determine if a problem I am experiencing is a bug or if
there is something I am missing. Last week I found that 15GB of my
32GB RAM was used by shared memory, causing my system to fill swap.
After about 1hr of trying to discover the reason for this, the only
thing I could find was that `df` was indicating that space was being
used by tmpfs mounted on /tmp, but I could not find files that came
anywhere close to this size.

Now I am inspecting my system after a few days of uptime and have
found that shared memory is slowly growing again. It's currently up to
almost 4GB:

jmetzmeier@localhost:/tmp $ free -m
              total        used        free      shared  buff/cache   available
Mem:           32065        9493        2185        3920       20386       18204
Swap:           8191        2114        6077

When I check df, what's reported as used on /tmp is pretty close to my
shared memory usage:

jmetzmeier@localhost:/tmp $ df -hT /tmp
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs   16G  3.5G   13G  22% /tmp

However, `du` does not show 3.5GB of files inside:

jmetzmeier@localhost:/tmp $ sudo du -sh ./
32K     ./

Does anyone know what could be happening here? Does writing a file to
tmpfs and then deleting it cause the tmpfs to retain that allocated
memory until reboot?

My system is running Fedora 34. I have ensured nothing is mounted over
top of /tmp and hiding files.

Thanks,
Jordan Metzmeier

[Joe Zeff]

On 10/26/21 12:23 PM, Jordan Metzmeier wrote:
> Does anyone know what could be happening here? Does writing a file to
> tmpfs and then deleting it cause the tmpfs to retain that allocated
> memory until reboot?

Wikipedia tells us that shared memory is used to communicate between 
programs and between threads in a program.  It's created by shm_open and 
then goes on to say this:

The shared memory created by shm_open is persistent. It stays in the 
system until explicitly removed by a process. This has a drawback that 
if the process crashes and fails to clean up shared memory it will stay 
until system shutdown.

My guess is that one or more of the programs you're using is either 
crashing or is otherwise ill-behaved and not cleaning up after itself.

[Todd Zullinger]

Hi,

I trimmed a lot of useful context only because I don't have
anything terribly useful to say about it.

Jordan Metzmeier wrote:
> jmetzmeier@localhost:/tmp $ df -hT /tmp
> Filesystem     Type   Size  Used Avail Use% Mounted on
> tmpfs          tmpfs   16G  3.5G   13G  22% /tmp
>
> However, `du` does not show 3.5GB of files inside:
>
> jmetzmeier@localhost:/tmp $ sudo du -sh ./
> 32K     ./
>
> Does anyone know what could be happening here? Does writing a file to
> tmpfs and then deleting it cause the tmpfs to retain that allocated
> memory until reboot?
>
> My system is running Fedora 34. I have ensured nothing is mounted over
> top of /tmp and hiding files.

I don't know if there are subtle issues because this is a
memory-based tmpfs filesystem.  But in general, you can see
something like this if a process opens a file and keeps it
open after it's deleted.  The space used by that file isn't
freed until the process which has it open exits (or closes
the file).

You can look for that sort of thing with the lsof command:

sudo lsof -a +L1 /tmp

Maybe that will point to what's taking up the space.

[Jordan Metzmeier]

On Tue, Oct 26, 2021 at 2:21 PM Todd Zullinger  wrote:
> Hi,
>
> I trimmed a lot of useful context only because I don't have
> anything terribly useful to say about it.
>
> Jordan Metzmeier wrote:
> > jmetzmeier@localhost:/tmp $ df -hT /tmp
> > Filesystem     Type   Size  Used Avail Use% Mounted on
> > tmpfs          tmpfs   16G  3.5G   13G  22% /tmp
> >
> > However, `du` does not show 3.5GB of files inside:
> >
> > jmetzmeier@localhost:/tmp $ sudo du -sh ./
> > 32K     ./
> >
> > Does anyone know what could be happening here? Does writing a file to
> > tmpfs and then deleting it cause the tmpfs to retain that allocated
> > memory until reboot?
> >
> > My system is running Fedora 34. I have ensured nothing is mounted over
> > top of /tmp and hiding files.
>
> I don't know if there are subtle issues because this is a
> memory-based tmpfs filesystem.  But in general, you can see
> something like this if a process opens a file and keeps it
> open after it's deleted.  The space used by that file isn't
> freed until the process which has it open exits (or closes
> the file).
>
> You can look for that sort of thing with the lsof command:
> sudo lsof -a +L1 /tmp
>
> Maybe that will point to what's taking up the space.
>
> --
> Todd

Deleted files were the culprit, thanks!

[Roger Heflin]

On Tue, Oct 26, 2021 at 2:46 PM Jordan Metzmeier  wrote:
>
> Deleted files were the culprit, thanks!
> _______________________________________________

Not sure why it is this way (counted as shared) but this seems to
confirm that tmpfs is counted as shared:
from the /proc/meminfo documentation:

Shmem

Total memory used by shared memory (shmem) and tmpfs

ShmemHugePages

Memory used by shared memory (shmem) and tmpfs allocated with huge pages

I have been reading the meminfo documentation trying to try to account
for all of the memory usage on a system.