Please review & comment: [389 Project] #412: memberof performance enhancement

[Noriko Hosoi]

https://fedorahosted.org/389/ticket/412

https://fedorahosted.org/389/attachment/ticket/412/0001-Trac-Ticket-412-m...

Fix description:
memberof.c: replaced DN strings with Slapi_DN and set the
normalized info to Slapi_Value flags. It reduces the number
of slaip_dn_normalize_ext call by ~25%.

attr.c, slapi-plugin.h: introduced a new API slapi_attr_
value_cmp_ext which takes Slapi_Value instead of struct
berval. By replacing with Slapi_Value, the value flag
(e.g., normalized info) can be passed to the syntax plugin.

value.c: changed slapi_value_compare to call slapi_attr_
value_cmp_ext instead of slapi_attr_value_cmp.
------------------------------------------------------------------------

Replying to nkinder <https://fedorahosted.org/389/ticket/412#comment:2>;:

    What sort of performance increase do these changes give you in your
    testing?

Unfortunately, there was no observable performance gain. Let me explain 
it with the test env next...
------------------------------------------------------------------------
As replied to the Nathan's question, I could not see any performance 
gain in the elapsed time, although it reduced the count of 
slapi_dn_normalize_ext by ~25%.

But the server's behaviour observed in the test was interesting. The 
dominant time of the elapsed time spent in the memberof operation is 
likely in libdb.  The time difference 14 sec vs. 27 sec between 
"Straight ldapmodify ...; sh memof_script.sh" and "ldapmodify ...; 
restart the server; [ldapsearch <all>; ]sh memof_script.sh" cannot be 
explained in the server level. Callgrind outputs almost the same graph 
except the libdb internal.  And once we put all memberof operations in 
one transaction, the response time is extremely short.  We have not 
enabled betxn by default yet.  Can we make a plan to do so for brushing 
up the code especially on Fedora?
------------------------------------------------------------------------
Result:

Note: With/Without? <https://fedorahosted.org/389/wiki/With/Without> the 
attached patch: 
0001-Trac-Ticket-412-memberof-performance-enhancement.patch​, there was 
no difference in the elapsed time.

This test adds 1000 user entries first (ldapmodify...).
Then memof_script.sh adds a group entry which contains the 1000 members; 
the script waits for all the user entries have memberof attribute value.
(The test files are attached to the trac #412 with the instructions.)

Straight ldapmodify ...; sh memof_script.sh: 14 seconds
ldapmodify ...; restart the server; sh memof_script.sh: 27 seconds
ldapmodify ...; restart the server; ldapsearch <all>; sh 
memof_script.sh: 27 seconds
(I.e, the difference is not due to the entries in the entry cache or not)

Enable betxn of the Memberof Plugin
dn: cn=MemberOf? <https://fedorahosted.org/389/wiki/MemberOf> 
Plugin,cn=plugins,cn=config
nsslapd-pluginType: betxnpostoperation
Straight ldapmodify ...; sh memof_script.sh: < 1 second
ldapmodify ...; restart the server; ldapsearch <all>; sh 
memof_script.sh: < 1 second
------------------------------------------------------------------------
Thanks,
--noriko