<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 08/29/2013 04:22 PM, Jeffrey Dunham
wrote:<br>
</div>
<blockquote
cite="mid:CAC_c4eoxFtdeqrB9Ho5Zr3ozQvG0pXU09W5UM6p9Gh9YtRW7eQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>So following your advice I was able to get some
stack traces while the server was hanging/slow to
respond. This is from one of our search hosts.<br>
</div>
I have shortened it here considerably because we do have
customer data that is present, I can do some more
scrubbing later if it will help.<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
I would like to have the full stack trace, all the way up to
connection_threadmain - if you need to elide/obscure customer
information, please do, but please include the full stack trace.<br>
<br>
<blockquote
cite="mid:CAC_c4eoxFtdeqrB9Ho5Zr3ozQvG0pXU09W5UM6p9Gh9YtRW7eQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
</div>
<div>Seems to me to be revolved around indexes, I know we
increased our allidslimit pretty high to 500000, I'm
wondering if that has anything to do with it.<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Looks like the unindexed searches are hogging all of the resources
and locking pages needed by updates.<br>
<br>
<blockquote
cite="mid:CAC_c4eoxFtdeqrB9Ho5Zr3ozQvG0pXU09W5UM6p9Gh9YtRW7eQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div><br>
</div>
<br>
</div>
Out of the 30 worker threads 28 of them are in a state like:
<br>
Thread 3 (Thread 0x2aef51f20940 (LWP 2569)):<br>
#0 0x000000328800b019 in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()
from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00002aeeae1ba4f6 in __db_pthread_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#2 0x00002aeeae242619 in __lock_get_internal () from
/lib64/<a moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#3 0x00002aeeae242b7f in __lock_vec () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#4 0x00002aeeae222d30 in __db_lget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#5 0x00002aeeae1cac72 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#6 0x00002aeeae1bd8d7 in ?? () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#7 0x00002aeeae1bea4f in ?? () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#8 0x00002aeeae218829 in __db_c_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#9 0x00002aeeadf289ed in idl_new_fetch (be=0x1dd03130,
db=<value optimized out>, inkey=0x2aef51f10760,
txn=<value optimized out>, a=0x1dd44940,
flag_err=0x2aef51f175bc, allidslimit=500000) at
ldap/servers/slapd/back-ldbm/idl_new.c:223<br>
<br>
</div>
There is a large unindex'd query running on one of the other
threads [ base: o=<a moz-do-not-send="true"
href="http://example.com">example.com</a>, filter:
(&(objectclass=posixaccount)(uid=*)) ] :<br>
Thread 8 (Thread 0x2aef4ed1b940 (LWP 2564)):<br>
#0 0x000000328800e5c8 in pread64 () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00002aeeae25c5dd in __os_io () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#2 0x00002aeeae25168b in __memp_pgread () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#3 0x00002aeeae2527dd in __memp_fget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#4 0x00002aeeae1ca938 in __bam_search () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#5 0x00002aeeae1bd8d7 in ?? () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#6 0x00002aeeae1bea4f in ?? () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#7 0x00002aeeae218829 in __db_c_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#8 0x00002aeeae220fe6 in __db_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#9 0x00002aeeae22115a in __db_get_pp () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#10 0x00002aeeadf24266 in id2entry (be=0x1dd03130, id=7630577,
txn=0x2aef4ed104e0, err=0x2aef4ed10544) at
ldap/servers/slapd/back-ldbm/id2entry.c:315<br>
inst = (ldbm_instance *) 0x1dc8d180<br>
db = (DB *) 0x1dd01080<br>
db_txn = (DB_TXN *) 0x0<br>
key = {data = 0x2aef4ed10450, size = 4, ulen = 0, dlen
= 0, doff = 0, flags = 0}<br>
data = {data = 0x0, size = 0, ulen = 0, dlen = 0, doff
= 0, flags = 4}<br>
e = (struct backentry *) 0x0<br>
ee = <value optimized out><br>
temp_id = "\000tnñ"<br>
<br>
</div>
<div>And another locked worker thread: <br>
#0 0x000000328800d654 in __lll_lock_wait () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x0000003288008f4a in _L_lock_1034 () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#2 0x0000003288008e0c in pthread_mutex_lock () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00002aeeae1ba54c in __db_pthread_mutex_lock () from
/lib64/<a moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#4 0x00002aeeae252a51 in __memp_fget () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#5 0x00002aeeae218d73 in __db_c_get () from /lib64/<a
moz-do-not-send="true" href="http://libdb-4.3.so">libdb-4.3.so</a><br>
No symbol table info available.<br>
#6 0x00002aeeadf28b63 in idl_new_fetch (be=0x1dd03130,
db=<value optimized out>, inkey=0x735755, txn=<value
optimized out>, a=0x1dd421f0, flag_err=0x2aef4e3115bc,
allidslimit=500000) at
ldap/servers/slapd/back-ldbm/idl_new.c:298<br>
<br>
</div>
And the replication thread appears to be locked as well:<br>
<br>
#0 0x000000328800d654 in __lll_lock_wait () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x0000003288008f80 in _L_lock_1233 () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#2 0x0000003288008f03 in pthread_mutex_lock () from
/lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x000000328ac23289 in PR_Lock () from /usr/lib64/libnspr4.so<br>
No symbol table info available.<br>
#4 0x000000328ac234cb in PR_EnterMonitor () from
/usr/lib64/libnspr4.so<br>
No symbol table info available.<br>
#5 0x00002aeeadf1496c in cache_lock_entry (cache=0x1dc8d208,
e=0x2af02d468c00) at ldap/servers/slapd/back-ldbm/cache.c:1455<br>
No locals.<br>
#6 0x00002aeeadf23b31 in find_entry_internal
(pb=0x2af022054ca0, be=0x1dd03130, addr=<value optimized
out>, lock=1, txn=0x2aef3ddf9cb0, flags=0) at
ldap/servers/slapd/back-ldbm/findentry.c:237<br>
No locals.<br>
#7 0x00002aeeadf4df1a in ldbm_back_modify (pb=0x2af022054ca0)
at ldap/servers/slapd/back-ldbm/ldbm_modify.c:269<br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Aug 21, 2013 at 9:14 AM, Rich
Megginson <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 08/21/2013 09:53 AM, David Boreham wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Another thing you might try :<br>
<br>
While the server is under stress, run the "pstack"
command a few times and save the output.<br>
</blockquote>
<br>
</div>
gdb will give much more detail<br>
<a moz-do-not-send="true"
href="http://port389.org/wiki/FAQ#Debugging_Hangs"
target="_blank">http://port389.org/wiki/FAQ#Debugging_Hangs</a>
<div class="HOEnZb">
<div class="h5"><br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
If you post the thread stacks here, someone familiar
with the code can say with more accuracy what's going
on. For example it will be obvious whether you have
starved out the thread pool, or you have threads
mostly waiting on page locks in the DB, etc.<br>
<br>
<br>
-- <br>
389 users mailing list<br>
<a moz-do-not-send="true"
href="mailto:389-users@lists.fedoraproject.org"
target="_blank">389-users@lists.fedoraproject.org</a><br>
<a moz-do-not-send="true"
href="https://admin.fedoraproject.org/mailman/listinfo/389-users"
target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a><br>
</blockquote>
<br>
--<br>
389 users mailing list<br>
<a moz-do-not-send="true"
href="mailto:389-users@lists.fedoraproject.org"
target="_blank">389-users@lists.fedoraproject.org</a><br>
<a moz-do-not-send="true"
href="https://admin.fedoraproject.org/mailman/listinfo/389-users"
target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">--
389 users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:389-users@lists.fedoraproject.org">389-users@lists.fedoraproject.org</a>
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/mailman/listinfo/389-users">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></pre>
</blockquote>
<br>
</body>
</html>