I've been running some more tests before setting up the ticket, but I think I have enough information now. The uniqueMember attribute has extra processing overhead, but the necessary optimization might apply across the board for all attributes. I found also that adding large sets of values for other attributes also increases modification times heavily, though not quite as much as uniqueMember. Luckily, the modification delay is based on the size of the modification rather than the size of the entry, so even if the modification is done to a 100K-value attribute, if the modification is only to remove a few members and add a few others, then the change is still relatively quick. The delay is noticed most when first setting up a group, for instance, adding 100K members to an empty group takes 2.5 hours on 389 as opposed to 1 minute on Sun DS.
Also during this testing I have noticed a memory leak when running large quantities of ldapmodify operations. When I set up a loop to delete and then re-add the eduPersonEntitlement attribute across 100K entries, I found that memory consumption continuously increased and the server crashed after the fifth iteration of this loop. (And this one really is with ldapmodify and is not related to my earlier issues with excessively creating tombstones by deleting and adding entire entries). Before digging into this too deeply and making another ticket, I wanted to ask if this had been noticed and fixed in the 1.2.10 release? I am using the default 188.8.131.52 release. I'm guessing it hasn't since I didn't see it in the release notes.
I am starting up the server with the valgrind command you recommended a few messages back to see if I can spot the leak, though of course with valgrind in the mix, the overhead and runtimes are, as might be expected, much increased.
On Apr 19, 2012, at 1:42 PM, Rich Megginson wrote:
OK. If you've ruled out the possibility that some plugin is interfering with the processing, then it must be something we will have to fix in the core server. Please file a ticket at https://fedorahosted.org/389