sssd_nss writing 25 MiB to /var/lib/sss/mc every time it starts up

[D M]

Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs. We're running sssd 2.3.3.

The people who run the underlying disks for our VM hosts have been complaining about a spike in disk write latency every 10min.
After investigation, I figured out the issue.

By default, every 10 mins Debian systems run a cronjob, which kicks off a script called debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.

But running this cronjob (which runs as root by default) causes sssd_nss to get started up (on Ubuntu, it's socket activated, as you probably well know).

When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
For us, it writes about 25 MiB of data. (Is that a lot?)

By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info, sssd_nss shuts down.

So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands of VMs, this adds up to a lot of writes!

Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time, which is fine.
It'll prevent all those disk writes.

However, I wanted to see if you had any other suggestions. Thanks!

[Alexey Tikhonov]

Hi,

On Thu, Jan 18, 2024 at 4:16 PM D M  wrote:
> Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs. We're running sssd 2.3.3.
>
> The people who run the underlying disks for our VM hosts have been complaining about a spike in disk write latency every 10min.
> After investigation, I figured out the issue.
>
> By default, every 10 mins Debian systems run a cronjob, which kicks off a script called debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.
>
> But running this cronjob (which runs as root by default) causes sssd_nss to get started up (on Ubuntu, it's socket activated, as you probably well know).
>
> When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
> For us, it writes about 25 MiB of data. (Is that a lot?)

This should be a sum of 'memcache_size_*' sssd.conf options (man sssd.conf)
Just zero initialization.
> By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info, sssd_nss shuts down.

This can be tuned using 'client_idle_timeout'
> So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands of VMs, this adds up to a lot of writes!
>
> Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time, which is fine.

Just disable systemd socket and add 'nss' to sssd.conf::services
> It'll prevent all those disk writes.
> However, I wanted to see if you had any other suggestions. Thanks!
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Alexey Tikhonov]

On Thu, Jan 18, 2024 at 5:06 PM Alexey Tikhonov  wrote:
> Hi,
>
> On Thu, Jan 18, 2024 at 4:16 PM D M  wrote:
> > Hi folks! I'm chasing down some interesting behavior on our Ubuntu 20.04 VMs. We're running sssd 2.3.3.
> >
> > The people who run the underlying disks for our VM hosts have been complaining about a spike in disk write latency every 10min.
> > After investigation, I figured out the issue.
> >
> > By default, every 10 mins Debian systems run a cronjob, which kicks off a script called debian-sa1. By default, this script is basically a no-op. It doesn't write to disk.
> >
> > But running this cronjob (which runs as root by default) causes sssd_nss to get started up (on Ubuntu, it's socket activated, as you probably well know).
> >
> > When sssd_nss starts up, it writes the memory-mapped cache in /var/lib/sss/mc/.
> > For us, it writes about 25 MiB of data. (Is that a lot?)
>
> This should be a sum of 'memcache_size_*' sssd.conf options (man sssd.conf)
> Just zero initialization.
> > By default, sssd_nss has a 5-minute timeout, so, if nothing else looks up any user info, sssd_nss shuts down.
>
> This can be tuned using 'client_idle_timeout'

Sorry, it's not 'client_idle_timeout' but 'responder_idle_timeout'
(and if set to 0 then automatic shut down is disabled).
> > So, basically, every 10 mins, sssd_nss wakes up and writes 25 MiB. Across many thousands of VMs, this adds up to a lot of writes!
> >
> > Oh RHEL, seems like sssd_nss runs continuously, not socket-activated. I am thinking of just doing the same on Ubuntu. This will keep the memory-mapped cache active all the time, which is fine.
>
> Just disable systemd socket and add 'nss' to sssd.conf::services
> > It'll prevent all those disk writes.
> > However, I wanted to see if you had any other suggestions. Thanks!
> >
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Dmitry Mazin]

Thank you for the helpful response. We'll proceed with our original plan of disabling the socket.

Still, I was a little surprised to find nothing on the web about this. The idle timeout is 5 min by default; sssd_nss is socket-activated by default on Ubuntu; and debian-sa1 cronjob runs every 10 mins by default on Ubuntu. Does no one else run into this issue? I am thinking that perhaps we have too much data in one domain or something.