[389-devel] Please review: [389 Project] #490: Slow role performance when using a lot of roles

[Noriko Hosoi]

https://fedorahosted.org/389/ticket/490

https://fedorahosted.org/389/attachment/ticket/490/0001-Ticket-490-Slow-role-performance-when-using-a-lot-of.patch

Bug description: Role uses the virtual attribute framework.
When the search with a filter including nsrole or a return
attribute list containing nsrole is being processed, the
virtual attribute code checks the entry if the vattr values
are valid or not by examining the watermark.  If it is valid,
the values are used as if they are static.  If it is not
valid, the entry is evaluated against the role definitions
and dynamically generated virtual attributes are set to the
list (e_virtual_attrs) with the proper watermark.

The current code additionally checks e_virtual_attrs to determine
the entry is already evaluated or not.  If it is NULL, it
considers the entry is not yet evaluated and it returns SLAPI_
ENTRY_VATTR_NOT_RESOLVED even if the watermark is valid.  That
is, all the entries which do not have virtual attributes are
unnecessarily evaluated every time search with nsrole is executed.

Fix description: This patch does not return SLAPI_ENTRY_VATTR_NOT_
RESOLVED but does SLAPI_ENTRY_VATTR_RESOLVED_ABSENT if e_virtual_
attrs is NULL AND the watermark is valid.  By skipping the not-
needed nsrole evaluation, it speeds up the virtual search once
virutual attribute values are placed in the entries in memory.

Comment:
  Using test data having 86568 entries in total; 98 nsRoleDefinition entries
  and 61542 nsRoleDn among them...

  Sample command line:
  ldapsearch -LLLx -h localhost -p 389 -D 'cn=directory manager' -w password
  -b "dc=example,dc=com" "(nsrole=cn=CN0,o=O0,dc=example,dc=com)" nsrole
  It returns 3291 entries with 8321 nsrole attribute values.

  With the patch:
  nsslapd-ndn-cache-enabled: on
  No entries in cache: 0m49.308s
  All entries in cache: 0m0.181s

  nsslapd-ndn-cache-enabled: off
  No entries in cache: 0m51.792s
  All entries in cache: 0m0.210s

  Without the patch:
  nsslapd-ndn-cache-enabled: on
  No entries in cache: 0m50.579s
  All entries in cache: 0m9.599s

  nsslapd-ndn-cache-enabled: off
  No entries in cache: 0m52.727s
  All entries in cache: 0m9.857s

  The patch has no impact on the elapsed time to generate virtual attributes
  (No entries in cache).  But once they are evaluated and placed in the
  entry cache, we could see the improvement (All entries in cache).  Please
  note that if all the entries in the database have virtual attributes, this
  patch would have no effect.

  In addition, I tested with nsslapd-ndn-cache-enabled: on and off.  It's
  not huge, but we could recognize steady improvement.  I recommend to
  enable the functionality, by default or at least, advertise it more (on
  1.3.0 or newer)...