sssd_nss writing 25 MiB to /var/lib/sss/mc every time it starts up

[D M]

Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs.
We're running sssd 2.3.3.

The people who run the underlying disks for our VM hosts have been complaining about a
spike in disk write latency every 10min.
After investigation, I figured out the issue.

By default, every 10 mins Debian systems run a cronjob, which kicks off a script called
debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.

But running this cronjob (which runs as root by default) causes sssd_nss to get started up
(on Ubuntu, it's socket activated, as you probably well know).

When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
For us, it writes about 25 MiB of data. (Is that a lot?)

By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info,
sssd_nss shuts down.

So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands
of VMs, this adds up to a lot of writes!

Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of
just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time,
which is fine.
It'll prevent all those disk writes.

However, I wanted to see if you had any other suggestions. Thanks!

[Alexey Tikhonov]

Hi,

On Thu, Jan 18, 2024 at 4:16 PM D M <dmazin(a)drwuk.com&gt; wrote:
>  Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs.
> We're running sssd 2.3.3.
>
>  The people who run the underlying disks for our VM hosts have been complaining about a
> spike in disk write latency every 10min.
>  After investigation, I figured out the issue.
>
>  By default, every 10 mins Debian systems run a cronjob, which kicks off a script called
> debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.
>
>  But running this cronjob (which runs as root by default) causes sssd_nss to get started
> up (on Ubuntu, it's socket activated, as you probably well know).
>
>  When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
>  For us, it writes about 25 MiB of data. (Is that a lot?) 
This should be a sum of 'memcache_size_*' sssd.conf options (man sssd.conf)
Just zero initialization.

>  By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info,
> sssd_nss shuts down. 
This can be tuned using 'client_idle_timeout'

>  So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands
> of VMs, this adds up to a lot of writes!
>
>  Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of
> just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time,
> which is fine. 
Just disable systemd socket and add 'nss' to sssd.conf::services

>  It'll prevent all those disk writes.
>
>  However, I wanted to see if you had any other suggestions. Thanks!
>  --
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Alexey Tikhonov]

On Thu, Jan 18, 2024 at 5:06 PM Alexey Tikhonov <atikhono(a)redhat.com&gt; wrote:
>  Hi,
>
>  On Thu, Jan 18, 2024 at 4:16 PM D M <dmazin(a)drwuk.com&gt; wrote:
>  >
>  > Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs.
> We're running sssd 2.3.3.
>  >
>  > The people who run the underlying disks for our VM hosts have been complaining about
> a spike in disk write latency every 10min.
>  > After investigation, I figured out the issue.
>  >
>  > By default, every 10 mins Debian systems run a cronjob, which kicks off a script
> called debian-sa1. By default, this script is basically a no-op. It doesn't write to
> disk.
>  >
>  > But running this cronjob (which runs as root by default) causes sssd_nss to get
> started up (on Ubuntu, it's socket activated, as you probably well know).
>  >
>  > When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
>  > For us, it writes about 25 MiB of data. (Is that a lot?)
>
>  This should be a sum of 'memcache_size_*' sssd.conf options (man sssd.conf)
>  Just zero initialization.
>
>  >
>  > By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user
> info, sssd_nss shuts down.
>
>  This can be tuned using 'client_idle_timeout' 
Sorry, it's not 'client_idle_timeout' but 'responder_idle_timeout'
(and if set to 0 then automatic shut down is disabled).

>  >
>  > So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many
> thousands of VMs, this adds up to a lot of writes!
>  >
>  > Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking
> of just doing the same on Ubuntu. This will keep the memory-mapped cache active all the
> time, which is fine.
>
>  Just disable systemd socket and add 'nss' to sssd.conf::services
>
>  > It'll prevent all those disk writes.
>  >
>  > However, I wanted to see if you had any other suggestions. Thanks!
>  > --
>  > _______________________________________________
>  > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Dmitry Mazin]

Thank you for the helpful response. We'll proceed with our original plan of disabling
the socket.

Still, I was a little surprised to find nothing on the web about this. The idle timeout is
5 min by default; sssd_nss is socket-activated by default on Ubuntu; and debian-sa1
cronjob runs every 10 mins by default on Ubuntu. Does no one else run into this issue? I
am thinking that perhaps we have too much data in one domain or something.