On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote:

On 07/27/2016 03:36 PM, Jakub Hrozek wrote:

On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote:

On 07/27/2016 01:56 PM, Jakub Hrozek wrote:

On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote:

On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote:

On (27/07/16 12:08), Jakub Hrozek wrote:

On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote:

On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote:

ehlo,

attached patch fixes acces denied after activating user in 389ds.
Jakub had some comments/ideas in ticket but I think it's better to discuss
about virtual attributes and timestamp cache on mailing list.

Yes, so the comment I have is that while this works, it might break some
strange LDAP servers.

We use modifyTimestamp as a 'positive' indicator that the entry has not
changed -- if the modifyTimestamp didn't change, we consider the cached
entry the same as what is on the server and only bump the timestamp
cache. If the timestamp is different, we do a deep-comparison of cached
attribute values with what is on the LDAP server and write the sysdb
cache entry only if the attributes differ.

I was wondering if we can use the modifyTimestamp at all, then, because
even if it's the same, we might want to check the attributes to see if
some of the values are different because some of the attributes might be
this operational/virtual attribute..

Sorry, sent too soon.

I think the questions are -- 1) can we enumerate the virtual attributes?

That might be a question for 389-ds developers.
But it's very likely it will be different on other LDAP servers.

2) Would different LDAP servers have different virtual attributes.

For 2) maybe a possible solution might be to set a non-existing
modifyTimestamp attribute value, but I would consider that only a
kludge, we shouldn't break existing setups..

I am not satisfied with this POC solution either.

So should we remove usage of modifyTimestamp for detecting changes?

I would prefer to ask the DS developers before removing it completely.

At least for large groups it might take a long time to compare all attribute
values and IIRC we don't depend on any virtual attributes for groups. Maybe
we could parametrize that part of the code and enable the fast way with
modifyTimestamps for 'known' server types, that is for setups with AD and
IPA providers.

For users, there is typically not as many attributes so we should be
fine deep-comparing all attributes.

I'm adding Thierry (so please reply-to-all to keep him in the thread).

Thierry, in the latest sssd version we tried to add a performance
improvement related to how we store SSSD entries in the cache. The short
version is that we store the modifyTimestamp attribute in the cache and
when we fetch an entry, we compare the entry modifyTimestamp with what
is on the server. When the two are the same, we say that the entry did
not change and don't update the cache.

This works fine for most attributes, but not for attributes like
nsAccountLock which do not change modifyTimestamp when they are
modified. So when an entry was already cached but then nsAccountLock
changed, we treated the entry as the same and never read the new
nsAccountLock value.

To fix this, I think we have several options:
      1) special-case the nsAccountLock. This seems a bit dangerous,
      because I'm not sure we can say that some other attribute we are
      interested in behaves the same as nsAccountLock.
      2) drop the modifyTimestamp optimization completely. Then we fall
      back to comparing the attribute values, which might work, but for
      huge objects like groups with thousands of members, this might be
      too expensive.
      3) only use the modifyTimestamp optimization for cases where we know
      we don't read any virtual attributes.

And my question is -- can we, in general, know if the modifyTimestamp
way of detecting changes is realiable for all LDAP servers? Or do you
think it should only be used for cases where we know we are not
interested in any virtual attributes (that would mostly be storing
groups from servers where we know exactly what is on the server side,
like IPA or AD).

Hello,

Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it,
except I think it is unchanged when updating some password policy
attributes.

Regarding virtual attribute, the only one I know in IPA is nsaccountlock.
nsaccountlock is an operational attribute (you need to request it to see it)
and is also a virtual attribute BUT only for 'staged' and 'deleted' users.
It is a stored attribute for regular users and we should update
modifytimestamp when it is set.

thanks
thierry

OK, in that case it seems like we can special-case it. But do you know
about any other attributes in any other LDAP servers?

Any LDAP server following standard should provide modifytimestamp that
reflect the last update of the entry. Now virtual attribute values may be
"attached" to the entry and its value change without modification of
modifytimestamp.
For 389-ds and IPA it is fine as virtual value of nsaccountlock is changed
only when the DN change.
For others LDAP servers I suppose it exists the same ability to define
service providers that return virtual attribute values. The difficulty is
that the schema may not give any hint if the retrieved attributes values
were stored or computed and consequently trust modifytimestamp to know if
the values changed or not.
For example in ODSEE, memberof is a virtual attribute.

Thank you, for the explanation Thierry.

Then to be on the safe side I propose:
    1) We add an (probably undocumented?) flag that says whether to use
       modifyTimestamp to detect entry changes or not
    2) for the generic LDAP provider we always really compare the
       attribute values, in other words the option would be set to
       false. If there is anyone with performance issues with a generic
       setup, we tell them to flip the option.
    3) For the IPA and AD providers, we set this option to true and use
       the modifyTimestamp value to detect changes
    4) We special case nsAccountLock

• Stage user, it is always 'nsaccountlock: True'
  When stage entry is created or updated, 'modifytimestamp' is also updated. So you can rely on modifytimestamp to detect a change in a stage user.
  There is no way to update nsaccountlock for a stage entry
• Deleted user. Idem Stage user
• Regular user. 'nsaccountlock' is *not* a virtual attribute, so if it is enable/disable you can rely on modifytimestamp to detect a change of 'nsaccountlock' for a regular user.
  Also any change on regular user will update 'modifytimestamp' so you can rely on it to detect a change.

So IMHO you can rely on modifytimestamp even for the value of nsaccountlock.

thanks
thierry

Lukas, do you agree?