[389-users] Fwd: 389 v1.2.9.8 freeze/deadlock

Thu Sep 1 14:39:24 UTC 2011

On 09/01/2011 08:28 AM, Andrey Ivanov wrote:
> Hi Rich,
>
> The same test in 1.2.8.3 is ok, the important information being that
> it is also a paged search. here is the log for the same search for
> 1.2.8.3 (i'm in the process of rolling back to that version):
>
> [01/Sep/2011:16:19:39 +0200] conn=5 op=2 fd=128 closed - U1
> [01/Sep/2011:16:19:41 +0200] conn=6 fd=128 slot=128 connection from
> 129.104.31.63 to 129.104.69.49
> [01/Sep/2011:16:19:41 +0200] conn=6 op=0 BIND dn="" method=128 version=3
> [01/Sep/2011:16:19:41 +0200] conn=6 op=0 RESULT err=0 tag=97
> nentries=0 etime=0.017000 dn=""
> [01/Sep/2011:16:19:41 +0200] conn=6 op=1 SRCH
> base="ou=etudiants,ou=utilisateurs,dc=id,dc=polytechnique,dc=edu"
> scope=2 filter="(&(mail=*)(|(mail=le tallec*)(cn=le tallec*)(sn=le
> tallec*)(givenName=le tallec*)(displayName=le tallec*)))" attrs="cn cn
> mail roleOccupant display-name displayName sn sn co o o givenName
> legacyexchangedn objectClass uid mailnickname title company
> physicalDeliveryOfficeName telephoneNumber"
> [01/Sep/2011:16:19:41 +0200] conn=6 op=1 SORT cn (1)
> [01/Sep/2011:16:19:41 +0200] conn=6 op=1 RESULT err=0 tag=101
> nentries=0 etime=0.021000 notes=P
> [01/Sep/2011:16:19:41 +0200] conn=6 op=2 UNBIND
> [01/Sep/2011:16:19:41 +0200] conn=6 op=2 fd=128 closed - U1
>
>
>
> How do i compile the server with debug symbols? This would be
> sufficient or not:
> export CFLAGS="-g"
> export CXXFLAGS="-g"
Yes.  I usually do CFLAGS="-g" CXXFLAGS="-g" configure --enable-debug 
.... other configure args ....

You can install the debuginfo package:
debuginfo-install 389-ds-base

That will give you the full debugging symbols in gdb.

> ?
>
>
> @+
>
> 2011/9/1 Rich Megginson<rmeggins at redhat.com>:
>> On 09/01/2011 08:08 AM, Andrey Ivanov wrote:
>>> Hi,
>>>
>>> i've tried to install the 1.2.9.8 testing version in our production
>>> environment but there is a regular freeze/deadlock after a particular
>>> search.
>>>
>>> It is a search sent by outlook 2003 (you type the name of the person
>>> and then click "Check the name" button that generates an LDAP
>>> request). The person does not exist in the given subtree, here is the
>>> corresponding connection in the logs :
>>>
>>> [01/Sep/2011:13:42:34 +0200] conn=938 fd=129 slot=129 connection from
>>> x.x.x.x to y.y.y.y
>>> [01/Sep/2011:13:42:34 +0200] conn=938 op=0 BIND dn="" method=128 version=3
>>> [01/Sep/2011:13:42:34 +0200] conn=938 op=0 RESULT err=0 tag=97
>>> nentries=0 etime=0.000000 dn=""
>>> [01/Sep/2011:13:42:34 +0200] conn=938 op=1 SRCH
>>> base="ou=etudiants,ou=utilisateurs,dc=id,dc=polytechnique,dc=edu"
>>> scope=2 filter="(&(mail=*)(|(mail=le tallec*)(cn=le tallec*)(sn=le
>>> tallec*)(givenName=le tallec*)(displayName=le tallec*)))" attrs="cn cn
>>> mail roleOccupant display-name displayName sn sn co o o givenName
>>> legacyexchangedn objectClass uid mailnickname title company
>>> physicalDeliveryOfficeName telephoneNumber"
>>> [01/Sep/2011:13:42:34 +0200] conn=938 op=1 SORT cn (1)
>>> <end of access log, nothing in error log, server freezes>
>>>
>>>
>>> The problem is reproducible each time, here is the interesting part of
>>> the gdb trace :
>>>
>>> Thread 42 (Thread 0x42201940 (LWP 25005)):
>>> #0  0x00000038644cd722 in select () from /lib64/libc.so.6
>>> No symbol table info available.
>>> #1  0x00002b8ffb1bf959 in DS_Sleep () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #2  0x00002b900104e51e in deadlock_threadmain () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libback-ldbm.so
>>> No symbol table info available.
>>> #3  0x00000038670284ad in ?? () from /usr/lib64/libnspr4.so
>>> No symbol table info available.
>>> #4  0x000000386500673d in start_thread () from /lib64/libpthread.so.0
>>> No symbol table info available.
>>> #5  0x00000038644d44bd in clone () from /lib64/libc.so.6
>>> No symbol table info available.
>>> ...
>> This is the database housekeeping thread that checks for database deadlocks.
>>   This is normal.
>>> Thread 24 (Thread 0x4d613940 (LWP 25023)):
>>> #0  0x000000386500d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> No symbol table info available.
>>> #1  0x0000003865008e50 in _L_lock_1233 () from /lib64/libpthread.so.0
>>> No symbol table info available.
>>> #2  0x0000003865008dd3 in pthread_mutex_lock () from
>>> /lib64/libpthread.so.0
>>> No symbol table info available.
>>> #3  0x0000003867022ec9 in PR_Lock () from /usr/lib64/libnspr4.so
>>> No symbol table info available.
>>> #4  0x00002b8ffb18b308 in slapi_pblock_get () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #5  0x00002b8ffff8ac54 in DS_LASIpGetter () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libacl-plugin.so
>>> No symbol table info available.
>>> #6  0x00002b90001bfb08 in ACL_GetAttribute () from
>>> /Local/dirsrv/lib/dirsrv/libns-dshttpd.so.0
>>> No symbol table info available.
>>> #7  0x00002b90001be979 in LASIpEval () from
>>> /Local/dirsrv/lib/dirsrv/libns-dshttpd.so.0
>>> No symbol table info available.
>>> #8  0x00002b90001c0c30 in ACLEvalAce(NSErr_s*, ACLEvalHandle*,
>>> ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) ()
>>> from /Local/dirsrv/lib/dirsrv/libns-dshttpd.so.0
>>> No symbol table info available.
>>> #9  0x00002b90001c11ce in ACL_INTEvalTestRights(NSErr_s*,
>>> ACLEvalHandle*, char**, char**, char**, char**, char**, int*, unsigned
>>> long*) () from /Local/dirsrv/lib/dirsrv/libns-dshttpd.so.0
>>> No symbol table info available.
>>> #10 0x00002b90001c1956 in ACL_EvalTestRights () from
>>> /Local/dirsrv/lib/dirsrv/libns-dshttpd.so.0
>>> No symbol table info available.
>>> #11 0x00002b8ffff7e223 in acl__TestRights () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libacl-plugin.so
>>> No symbol table info available.
>>> #12 0x00002b8ffff813d6 in acl_access_allowed () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libacl-plugin.so
>>> No symbol table info available.
>>> #13 0x00002b8ffff959a4 in acl_access_allowed_main () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libacl-plugin.so
>>> No symbol table info available.
>>> #14 0x00002b8ffb192a61 in plugin_call_acl_plugin () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #15 0x00002b8ffb16247d in slapi_vattr_filter_test_ext_internal () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #16 0x00002b8ffb1622b8 in slapi_vattr_filter_test_ext_internal () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #17 0x00002b8ffb162544 in slapi_vattr_filter_test_ext () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #18 0x00002b900109010f in ldbm_back_next_search_entry_ext () from
>>> /Local/dirsrv/lib/dirsrv/plugins/libback-ldbm.so
>>> No symbol table info available.
>>> #19 0x00002b8ffb1868d1 in iterate () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #20 0x00002b8ffb187ce8 in op_shared_search () from
>>> /Local/dirsrv/lib/dirsrv/libslapd.so.0
>>> No symbol table info available.
>>> #21 0x0000000000429c78 in do_search ()
>>> No symbol table info available.
>>> #22 0x000000000041472f in connection_threadmain ()
>>> No symbol table info available.
>>> #23 0x00000038670284ad in ?? () from /usr/lib64/libnspr4.so
>>> No symbol table info available.
>>> #24 0x000000386500673d in start_thread () from /lib64/libpthread.so.0
>>> No symbol table info available.
>>> #25 0x00000038644d44bd in clone () from /lib64/libc.so.6
>>> No symbol table info available.
>> So it looks as though it is attempting to acquire the Connection mutex in
>> order to get the client IP address out of the Connection object, but it is
>> already locked.  You must have ip or dns based ACIs.
>>> I'll try to install the debug symbols and reproduce the stack trace
>>> with symbol table after i roll back to our previous production version
>>> (1.2.8.3)
>> Yes, please do.  I would like to know what ACIs you have too.
>>
>> This is very similar to a problem reported by Anthony Messina who also uses
>> dns/ip based ACIs.
>>> @+
>>> --
>>> 389 users mailing list
>>> 389-users at lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/389-users
>>