[SSSD] Some performance ideas related to running sssd on cluster nodes

[Jakub Hrozek]

Hi,

I was helping analyze poor performance and server-side load spikes in an
environment where cluster nodes running sssd were all booted up at the
same time.

It turned out that this meant cache entries were expiring at the same
time and also the LDAP connection was expiring and reconnecting at the
same time. There are some tickets we filed (the ideas were mostly
William's) and I wanted to discuss them here.

https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
object hadn't changed in a long time
    - I think this the most controversial and as we discussed a bit on
      our phone call this is probably too dangerous to do by default.
      Nonetheless, for resolving identity requests, it might be a
      tunable that might provide a nice performance benefit.

https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by a
couple of percent
    - What the title says. This might prevent hammering the servers in
      case the cluster nodes went up at the same time and had the same
      expire timestamps for all objects. Again, I'm not sure if this
      makes sense by default, because it adds a bit of a fuzzy
      behaviour, but I think it makes sense as a configurable.

https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks use
randomization
    - The be_ptask API already supports a bit of randomization, but
      we're not really using it. I guess the review should be
      case-by-case, but at least for ptasks that fetch any data from the
      back end, I would even just randomize a bit by default.

https://pagure.io/SSSD/sssd/issue/3630 - Randomize
ldap_connection_expire_timeout either by default or w/ a configure option
    - Again, if all connections expire and are reconnected at the same
      time, the servers suffer.

Does anyone have an opinion on the issues? I think at least the
connection timeout is something we should look at, because IIRC that was
causing the most issues on the IDM servers. The other tickets are IMO
less important and I'm even not sure if we should implement them by
default.