<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
p.msochpdefault, li.msochpdefault, div.msochpdefault
{mso-style-name:msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Calibri","sans-serif";
color:black;}
span.emailstyle17
{mso-style-name:emailstyle17;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hello Rich, thanks for the response. I was able to capture a core dump but I think, like you said, our version is too old for this method to work, here is the output when trying to load the dump file:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">======================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[root@ logs]# gdb /opt/fedora-ds/bin/slapd/server/ns-slapd /opt/fedora-ds/slapd-instance/logs/core.5916<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/bin/slapd/server/ns-slapd...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5972]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5978]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5977]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5976]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5975]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5974]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5973]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5971]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5970]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5969]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5968]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5967]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5966]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5965]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5964]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5963]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5962]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5961]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5960]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5959]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5958]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5957]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5956]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5955]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5954]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5953]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5952]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5951]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5950]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5949]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5948]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5947]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5946]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5945]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5944]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5943]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5942]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5941]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5940]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5939]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[New Thread 5916]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">./libslapd.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libssl3.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libnss3.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libsoftokn3.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libssldap60.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libldap60.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libprldap60.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libplc4.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libplds4.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libnspr4.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">./libdb-4.2.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: .dynamic section for "/usr/lib/libstdc++.so.6" is not at the expected address<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: difference appears to be caused by prelink, adjusting expectations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: .dynamic section for "/lib/libm.so.6" is not at the expected address<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: difference appears to be caused by prelink, adjusting expectations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: .dynamic section for "/lib/libresolv.so.2" is not at the expected address<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">warning: difference appears to be caused by prelink, adjusting expectations<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libicui18n.so.34: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libicuuc.so.34: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libicudata.so.34: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libns-dshttpd10.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Error while mapping shared library sections:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">../lib/libfreebl3.so: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ./libslapd.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libssl3.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libnss3.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libsoftokn3.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libssldap60.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libldap60.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libprldap60.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libplc4.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libplds4.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libnspr4.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libdl.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/libsasl2.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libcrypt.so.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[Thread debugging using libthread_db enabled]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ./libdb-4.2.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/libstdc++.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libm.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libgcc_s.so.1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/ld-linux.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libresolv.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /lib/libnss_files.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/sasl2/liblogin.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/sasl2/liblogin.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/sasl2/libsasldb.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/sasl2/libsasldb.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/sasl2/libplain.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/sasl2/libplain.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /usr/lib/sasl2/libanonymous.so.2...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /usr/lib/sasl2/libanonymous.so.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/syntax-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/syntax-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/liblcoll.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/liblcoll.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libicui18n.so.34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libicuuc.so.34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libicudata.so.34<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/pwdstorage-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/pwdstorage-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/des-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/des-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/attr-unique-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/attr-unique-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/acl-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/acl-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libns-dshttpd10.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/chainingdb-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/chainingdb-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/cos-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/cos-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/http-client-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/http-client-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/libback-ldbm.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/libback-ldbm.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/replication-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/replication-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/passthru-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/passthru-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/referint-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/referint-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/retrocl-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/retrocl-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/roles-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/roles-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/statechange-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/statechange-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/lib/views-plugin.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/lib/views-plugin.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Symbol file not found for ../lib/libfreebl3.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Reading symbols from /opt/fedora-ds/alias/libnssckbi.so...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Loaded symbols for /opt/fedora-ds/alias/libnssckbi.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Core was generated by `./ns-slapd -D /opt/fedora-ds/slapd-bhc -i /opt/fedora-ds/slapd-bhc/logs/pid -w'.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Program terminated with signal 11, Segmentation fault.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">#0 0x00f3b80c in ?? ()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">(gdb)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">========================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I did try running the rest of the gdb commands to create the stack trace but it never creates a file.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Nothing helpful (to me) in the error logs either:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">========================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - conn 4244 activity level = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - listener got signaled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - new connection on 70<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] NSMMReplicationPlugin - ruv_add_csn_inprogress: successfully inserted csn 4e402d7f000000010000 into pending list<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - activity on 70r<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - read activity on 70<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - conn 4245 activity level = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:39:59 -0500] - listener got signaled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">****crash*******<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:44 -0500] - Fedora-Directory/1.0.4 B2006.312.1539 starting up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:44 -0500] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:44 -0500] NSMMReplicationPlugin - agmtlist_config_init: found 0 replication agreements in DIT<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:44 -0500] NSMMReplicationPlugin - Purged state information from entry dc=aquila,dc=com up to CSN 4e36f2e4000000010000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:44 -0500] - slapd started. Listening on All Interfaces port 389 for LDAP requests<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[08/Aug/2011:13:40:48 -0500] - new connection on 64<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">========================================================================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Rich Megginson [mailto:rmeggins@redhat.com]
<br>
<b>Sent:</b> Monday, August 08, 2011 9:14 AM<br>
<b>To:</b> General discussion list for the 389 Directory server project.<br>
<b>Cc:</b> Wendt, Trevor<br>
<b>Subject:</b> Re: [389-users] Crashing<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On 08/05/2011 10:46 AM, Wendt, Trevor wrote: <o:p></o:p></p>
<div>
<p class="MsoNormal">Hello all, <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Need some help with tuning and crash debugging. We’re running Fedora-Directory/1.0.4 B2006.312.1539. The problem is on our “Dedicated Consumer” machine running on RHEL 5. We have over ~150,000 users authenticating against our FDS systems.
System resources are not a problem (~.39 load, free memory, 92k swap)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">For months, the system is solid without any issues then we seem to get a large spike in traffic and FDS crashes. I run Monit so the service is restarted automatically but I cannot figure out why the service keeps crashing.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">FDS was setup and tuned based off: <a href="http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux">
http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have reviewed <a href="http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes">
http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes</a> as well, but some of that is over my head.<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">Unfortunately these directions are for 1.1.x and later. Most of the paths/filenames have changed since 1.0.4 in the move to the FHS style layout, and there is no debuginfo
package. But we may still be able to get a core file and some stack information:<br>
<br>
sysctl -w fs.suid_dumpable=1<br>
<br>
edit /opt/fedora-ds/slapd-YOURINSTANCE/start-slapd<br>
somewhere near the top, add the line<br>
ulimit -c unlimited<br>
<br>
restart the directory server<br>
/opt/fedora-ds/slapd-YOURINSTANCE/restart-slapd<br>
<br>
If you get a crash, you should have a core file in /opt/fedora-ds/slapd-YOURINSTANCE/logs<br>
<br>
After that, install gdb<br>
<br>
follow the instructions at <a href="http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes">
http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes</a><br>
except:<br>
cd /opt/fedora-ds/slapd-YOURINSTANCE/logs<br>
gdb ../../bin/slapd/server/ns-slapd core.PID<br>
<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal">I have turned buffering off and increased the logging level in the LDAP config.
<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">What is the last operation in the access log before a crash? Any corresponding errors in the errors log?<br>
<br>
<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here is our “monitor” script output: <o:p></o:p></p>
<p class="MsoNormal">version: 1<o:p></o:p></p>
<p class="MsoNormal">dn: cn=monitor<o:p></o:p></p>
<p class="MsoNormal">objectClass: top<o:p></o:p></p>
<p class="MsoNormal">objectClass: extensibleObject<o:p></o:p></p>
<p class="MsoNormal">cn: monitor<o:p></o:p></p>
<p class="MsoNormal">version: Fedora-Directory/1.0.4 B2006.312.1539<o:p></o:p></p>
<p class="MsoNormal">threads: 30<o:p></o:p></p>
<p class="MsoNormal">currentconnections: 19<o:p></o:p></p>
<p class="MsoNormal">totalconnections: 11918<o:p></o:p></p>
<p class="MsoNormal">dtablesize: 8192<o:p></o:p></p>
<p class="MsoNormal">readwaiters: 0<o:p></o:p></p>
<p class="MsoNormal">opsinitiated: 43703<o:p></o:p></p>
<p class="MsoNormal">opscompleted: 43702<o:p></o:p></p>
<p class="MsoNormal">entriessent: 16086<o:p></o:p></p>
<p class="MsoNormal">bytessent: 2911011<o:p></o:p></p>
<p class="MsoNormal">currenttime: 20110805164243Z<o:p></o:p></p>
<p class="MsoNormal">starttime: 20110805114053Z<o:p></o:p></p>
<p class="MsoNormal">nbackends: 2<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">So about 8700 ops/hour. Not a heavy load.<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here is our “Access Log Analyzer” summary for a 24 hour period:
<o:p></o:p></p>
<p class="MsoNormal">---------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">Access Log Analyzer 6.0<o:p></o:p></p>
<p class="MsoNormal">Filename Total Lines Lines processed<o:p></o:p></p>
<p class="MsoNormal">---------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">/opt/fedora-ds/slapd/logs/access 298225 298231<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">----------- Access Log Output ------------<o:p></o:p></p>
<p class="MsoNormal">Restarts: 6<o:p></o:p></p>
<p class="MsoNormal">Total Connections: 39720<o:p></o:p></p>
<p class="MsoNormal">Peak Concurrent Connections: 84<o:p></o:p></p>
<p class="MsoNormal">Total Operations: 95471<o:p></o:p></p>
<p class="MsoNormal">Total Results: 95393<o:p></o:p></p>
<p class="MsoNormal">Overall Performance: 99.9%<o:p></o:p></p>
<p class="MsoNormal">Searches: 48215<o:p></o:p></p>
<p class="MsoNormal">Modifications: 167<o:p></o:p></p>
<p class="MsoNormal">Adds: 551<o:p></o:p></p>
<p class="MsoNormal">Deletes: 2<o:p></o:p></p>
<p class="MsoNormal">Mod RDNs: 0<o:p></o:p></p>
<p class="MsoNormal">6.x Stats<o:p></o:p></p>
<p class="MsoNormal">Persistent Searches: 0<o:p></o:p></p>
<p class="MsoNormal">Internal Operations: 0<o:p></o:p></p>
<p class="MsoNormal">Entry Operations: 0<o:p></o:p></p>
<p class="MsoNormal">Extended Operations: 845<o:p></o:p></p>
<p class="MsoNormal">Abandoned Requests: 0<o:p></o:p></p>
<p class="MsoNormal">Smart Referrals Received: 0<o:p></o:p></p>
<p class="MsoNormal">VLV Operations: 0<o:p></o:p></p>
<p class="MsoNormal">VLV Unindexed Searches: 0<o:p></o:p></p>
<p class="MsoNormal">SORT Operations: 0<o:p></o:p></p>
<p class="MsoNormal">SSL Connections: 0<o:p></o:p></p>
<p class="MsoNormal">Entire Search Base Queries: 0<o:p></o:p></p>
<p class="MsoNormal">Unindexed Searches: 6<o:p></o:p></p>
<p class="MsoNormal">FDs Taken: 39720<o:p></o:p></p>
<p class="MsoNormal">FDs Returned: 39657<o:p></o:p></p>
<p class="MsoNormal">Highest FD Taken: 93<o:p></o:p></p>
<p class="MsoNormal">Broken Pipes: 0<o:p></o:p></p>
<p class="MsoNormal">Connections Reset By Peer: 0<o:p></o:p></p>
<p class="MsoNormal">Resource Unavailable: 10872<o:p></o:p></p>
<p class="MsoNormal"> - 10872 (T1) Idle Timeout Exceeded<o:p></o:p></p>
<p class="MsoNormal">Binds: 45691<o:p></o:p></p>
<p class="MsoNormal">Unbinds: 27987<o:p></o:p></p>
<p class="MsoNormal">LDAP v2 Binds: 15694<o:p></o:p></p>
<p class="MsoNormal">LDAP v3 Binds: 29997<o:p></o:p></p>
<p class="MsoNormal">SSL Client Binds: 0<o:p></o:p></p>
<p class="MsoNormal">Failed SSL Client Binds: 0<o:p></o:p></p>
<p class="MsoNormal">SASL Binds: 0<o:p></o:p></p>
<p class="MsoNormal">Directory Manager Binds: 0<o:p></o:p></p>
<p class="MsoNormal">Anonymous Binds: 16346<o:p></o:p></p>
<p class="MsoNormal">Other Binds: 29345<o:p></o:p></p>
<p class="MsoNormal">---------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In FDS console: <o:p></o:p></p>
<p class="MsoNormal">-- Configuration > Performance tab: Size Limit: 2000, Time Limit: 3600, Idle Timeout: 60, Max file descriptors: 8192.
<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">The idle timeout is 1 minute - could be too low for some of your clients, which is why you're seeing a lot of (T1) Idle Timeout Exceeded connection closes.<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal">-- Configuration > Data > Database Link Settings > Connection Management: Max TCP Connections: 10, Bind timeout: 20, Max binds per connection: 20, Timeout before abandon: 10, Max LDAP Connections: 20, Max bind retries: 3, Max operations
per connection: 5, connection life: 60. <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">Are you using database links?<br>
<br>
I also suggest looking at your database cache tuning - see <a href="http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html-single/Administration_Guide/index.html#Monitoring_Server_and_Database_Activity-Monitoring_Database_Activity">
http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html-single/Administration_Guide/index.html#Monitoring_Server_and_Database_Activity-Monitoring_Database_Activity</a><br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We have talked about moving to the latest 389 Directory packages and I have a migration process tested out so it’s a matter of getting the OK and time but I doubt the upgrade will solve our crashing problem.<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">I can't say for sure, but 1.0.4 is very old, and since then we have fixed many issues which have caused crashes.<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal">It seems to me we are hitting some limits that just haven’t been accounted for yet and that is where I need help.
<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">Let's start with analyzing the crash data - if we can get a core file and a stack trace, then we can work from there to figure out why it's crashing.<br>
<br>
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Any suggestions on how to proceed with stopping these crashes is welcomed! Thanks for reading.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:7.5pt">Trevor</span></b><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:gray"><br>
This electronic message transmission contains information from Black Hills Corporation, its affiliate or subsidiary, which may be confidential or privileged. The information is intended to be for the use of the individual or entity named above. If you are not
the intended recipient, be aware the disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic transmission in error, please reply to sender immediately; then delete this message without copying
it or further reading.<br>
<br>
</span><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p></o:p></span></p>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>--<o:p></o:p></pre>
<pre>389 users mailing list<o:p></o:p></pre>
<pre><a href="mailto:389-users@lists.fedoraproject.org">389-users@lists.fedoraproject.org</a><o:p></o:p></pre>
<pre><a href="https://admin.fedoraproject.org/mailman/listinfo/389-users">https://admin.fedoraproject.org/mailman/listinfo/389-users</a><o:p></o:p></pre>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</body>
</html>