Some performance ideas related to running sssd on cluster nodes

[Jakub Hrozek]

Hi,

I was helping analyze poor performance and server-side load spikes in an
environment where cluster nodes running sssd were all booted up at the
same time.

It turned out that this meant cache entries were expiring at the same
time and also the LDAP connection was expiring and reconnecting at the
same time. There are some tickets we filed (the ideas were mostly
William's) and I wanted to discuss them here.

https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
object hadn't changed in a long time
    - I think this the most controversial and as we discussed a bit on
      our phone call this is probably too dangerous to do by default.
      Nonetheless, for resolving identity requests, it might be a
      tunable that might provide a nice performance benefit.

https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by a
couple of percent
    - What the title says. This might prevent hammering the servers in
      case the cluster nodes went up at the same time and had the same
      expire timestamps for all objects. Again, I'm not sure if this
      makes sense by default, because it adds a bit of a fuzzy
      behaviour, but I think it makes sense as a configurable.

https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks use
randomization
    - The be_ptask API already supports a bit of randomization, but
      we're not really using it. I guess the review should be
      case-by-case, but at least for ptasks that fetch any data from the
      back end, I would even just randomize a bit by default.

https://pagure.io/SSSD/sssd/issue/3630 - Randomize
ldap_connection_expire_timeout either by default or w/ a configure option
    - Again, if all connections expire and are reconnected at the same
      time, the servers suffer.

Does anyone have an opinion on the issues? I think at least the
connection timeout is something we should look at, because IIRC that was
causing the most issues on the IDM servers. The other tickets are IMO
less important and I'm even not sure if we should implement them by
default.

[Pavel Březina]

On 02/05/2018 03:38 PM, Jakub Hrozek wrote:
>  Hi,
>
>  I was helping analyze poor performance and server-side load spikes in an
>  environment where cluster nodes running sssd were all booted up at the
>  same time.
>
>  It turned out that this meant cache entries were expiring at the same
>  time and also the LDAP connection was expiring and reconnecting at the
>  same time. There are some tickets we filed (the ideas were mostly
>  William's) and I wanted to discuss them here.
>
>  https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
>  object hadn't changed in a long time
>      - I think this the most controversial and as we discussed a bit on
>        our phone call this is probably too dangerous to do by default.
>        Nonetheless, for resolving identity requests, it might be a
>        tunable that might provide a nice performance benefit. 
If configurable, it can be done. We may try to be clever here, e.g. 
refresh user if group changed.

>  https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by
> a
>  couple of percent
>      - What the title says. This might prevent hammering the servers in
>        case the cluster nodes went up at the same time and had the same
>        expire timestamps for all objects. Again, I'm not sure if this
>        makes sense by default, because it adds a bit of a fuzzy
>        behaviour, but I think it makes sense as a configurable. 
Yes. This might help a lot in this scenario.

>  https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks
> use
>  randomization
>      - The be_ptask API already supports a bit of randomization, but
>        we're not really using it. I guess the review should be
>        case-by-case, but at least for ptasks that fetch any data from the
>        back end, I would even just randomize a bit by default. 
It can be done directly in ptask, enabled by configuration option. 
However we must make sure that randomized schedules still make sense 
(e.g. sudo smart refresh inteval << sudo full refresh interval).

>  https://pagure.io/SSSD/sssd/issue/3630 - Randomize
>  ldap_connection_expire_timeout either by default or w/ a configure option
>      - Again, if all connections expire and are reconnected at the same
>        time, the servers suffer. 
Agree.

>  Does anyone have an opinion on the issues? I think at least the
>  connection timeout is something we should look at, because IIRC that was
>  causing the most issues on the IDM servers. The other tickets are IMO
>  less important and I'm even not sure if we should implement them by
>  default. 
There are also two more things we can done:
a) Implement "throttling" behaviour on periodical refresh of expired 
object. We can update the object in several batches scheduled over large 
(and random) period of time.

b) I do not remember the exact name but we wanted to implement a 389ds 
mechanism that would push changes to clients instead of clients pulling 
for them. We may want to revisit.

[Jakub Hrozek]

On Tue, Feb 06, 2018 at 09:55:00AM +0100, Pavel Březina wrote:
>  On 02/05/2018 03:38 PM, Jakub Hrozek wrote:
>  > Hi,
>  > 
>  > I was helping analyze poor performance and server-side load spikes in an
>  > environment where cluster nodes running sssd were all booted up at the
>  > same time.
>  > 
>  > It turned out that this meant cache entries were expiring at the same
>  > time and also the LDAP connection was expiring and reconnecting at the
>  > same time. There are some tickets we filed (the ideas were mostly
>  > William's) and I wanted to discuss them here.
>  > 
>  > https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
>  > object hadn't changed in a long time
>  >     - I think this the most controversial and as we discussed a bit on
>  >       our phone call this is probably too dangerous to do by default.
>  >       Nonetheless, for resolving identity requests, it might be a
>  >       tunable that might provide a nice performance benefit.
>  
>  If configurable, it can be done. We may try to be clever here, e.g. refresh
>  user if group changed. 
The important thing here would be to ignore this optimization for
authentication requests and set a sane upper limit. Otherwise it might
be frustrating to admins to see that they configured e.g. an hour cache,
but changes they did on the server side don't propagate to the clients
for several hours.

>  
>  > https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by a
>  > couple of percent
>  >     - What the title says. This might prevent hammering the servers in
>  >       case the cluster nodes went up at the same time and had the same
>  >       expire timestamps for all objects. Again, I'm not sure if this
>  >       makes sense by default, because it adds a bit of a fuzzy
>  >       behaviour, but I think it makes sense as a configurable.
>  
>  Yes. This might help a lot in this scenario.
>  
>  > https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks use
>  > randomization
>  >     - The be_ptask API already supports a bit of randomization, but
>  >       we're not really using it. I guess the review should be
>  >       case-by-case, but at least for ptasks that fetch any data from the
>  >       back end, I would even just randomize a bit by default.
>  
>  It can be done directly in ptask, enabled by configuration option. However
>  we must make sure that randomized schedules still make sense (e.g. sudo
>  smart refresh inteval << sudo full refresh interval). 
OK, good idea, I added it to the ticket.

>  
>  > https://pagure.io/SSSD/sssd/issue/3630 - Randomize
>  > ldap_connection_expire_timeout either by default or w/ a configure option
>  >     - Again, if all connections expire and are reconnected at the same
>  >       time, the servers suffer.
>  
>  Agree.
>  
>  > Does anyone have an opinion on the issues? I think at least the
>  > connection timeout is something we should look at, because IIRC that was
>  > causing the most issues on the IDM servers. The other tickets are IMO
>  > less important and I'm even not sure if we should implement them by
>  > default.
>  
>  There are also two more things we can done:
>  a) Implement "throttling" behaviour on periodical refresh of expired object.
>  We can update the object in several batches scheduled over large (and
>  random) period of time.
>  
>  b) I do not remember the exact name but we wanted to implement a 389ds
>  mechanism that would push changes to clients instead of clients pulling for
>  them. We may want to revisit. 
That's the Persistent Search control. It would be possible to use it for
sssd instances running on the IPA server itself, but not in general,
because each client then requires a thread on the server side..

[Pavel Březina]

On 02/06/2018 10:46 AM, Jakub Hrozek wrote:
>  On Tue, Feb 06, 2018 at 09:55:00AM +0100, Pavel Březina wrote:
> > On 02/05/2018 03:38 PM, Jakub Hrozek wrote:
> >> Hi,
> >>
> >> I was helping analyze poor performance and server-side load spikes in an
> >> environment where cluster nodes running sssd were all booted up at the
> >> same time.
> >>
> >> It turned out that this meant cache entries were expiring at the same
> >> time and also the LDAP connection was expiring and reconnecting at the
> >> same time. There are some tickets we filed (the ideas were mostly
> >> William's) and I wanted to discuss them here.
> >>
> >> https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
> >> object hadn't changed in a long time
> >>     - I think this the most controversial and as we discussed a bit on
> >>       our phone call this is probably too dangerous to do by default.
> >>       Nonetheless, for resolving identity requests, it might be a
> >>       tunable that might provide a nice performance benefit.
> >
> > If configurable, it can be done. We may try to be clever here, e.g. refresh
> > user if group changed.
>
>  The important thing here would be to ignore this optimization for
>  authentication requests and set a sane upper limit. Otherwise it might
>  be frustrating to admins to see that they configured e.g. an hour cache,
>  but changes they did on the server side don't propagate to the clients
>  for several hours.
>
> >
> >> https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by a
> >> couple of percent
> >>     - What the title says. This might prevent hammering the servers in
> >>       case the cluster nodes went up at the same time and had the same
> >>       expire timestamps for all objects. Again, I'm not sure if this
> >>       makes sense by default, because it adds a bit of a fuzzy
> >>       behaviour, but I think it makes sense as a configurable.
> >
> > Yes. This might help a lot in this scenario.
> >
> >> https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks use
> >> randomization
> >>     - The be_ptask API already supports a bit of randomization, but
> >>       we're not really using it. I guess the review should be
> >>       case-by-case, but at least for ptasks that fetch any data from the
> >>       back end, I would even just randomize a bit by default.
> >
> > It can be done directly in ptask, enabled by configuration option. However
> > we must make sure that randomized schedules still make sense (e.g. sudo
> > smart refresh inteval << sudo full refresh interval).
>
>  OK, good idea, I added it to the ticket.
>
> >
> >> https://pagure.io/SSSD/sssd/issue/3630 - Randomize
> >> ldap_connection_expire_timeout either by default or w/ a configure option
> >>     - Again, if all connections expire and are reconnected at the same
> >>       time, the servers suffer.
> >
> > Agree.
> >
> >> Does anyone have an opinion on the issues? I think at least the
> >> connection timeout is something we should look at, because IIRC that was
> >> causing the most issues on the IDM servers. The other tickets are IMO
> >> less important and I'm even not sure if we should implement them by
> >> default.
> >
> > There are also two more things we can done:
> > a) Implement "throttling" behaviour on periodical refresh of expired
> object.
> > We can update the object in several batches scheduled over large (and
> > random) period of time.
> >
> > b) I do not remember the exact name but we wanted to implement a 389ds
> > mechanism that would push changes to clients instead of clients pulling for
> > them. We may want to revisit.
>
>  That's the Persistent Search control. It would be possible to use it for
>  sssd instances running on the IPA server itself, but not in general,
>  because each client then requires a thread on the server side.. 
A way to go would be create a similar plugin. A plugin that would keep 
track of all registered SSSD clients and send information about changes 
to all of them in batches. The client would then interpret the change 
locally. This is out of scope for the moment but we can consider this 
for some future time.