<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
On 08/05/2011 10:46 AM, Wendt, Trevor wrote:
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<style>
<!--
@font-face
{font-family:Calibri}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif"}
span.EmailStyle17
{font-family:"Calibri","sans-serif";
color:windowtext}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">
<p class="MsoNormal">Hello all, </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Need some help with tuning and crash
debugging. We’re running Fedora-Directory/1.0.4
B2006.312.1539. The problem is on our “Dedicated Consumer”
machine running on RHEL 5. We have over ~150,000 users
authenticating against our FDS systems. System resources are
not a problem (~.39 load, free memory, 92k swap)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">For months, the system is solid without any
issues then we seem to get a large spike in traffic and FDS
crashes. I run Monit so the service is restarted automatically
but I cannot figure out why the service keeps crashing.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">FDS was setup and tuned based off: <a
moz-do-not-send="true"
href="http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux">
http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux</a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I have reviewed <a moz-do-not-send="true"
href="http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes">
http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes</a> as
well, but some of that is over my head.</p>
</div>
</blockquote>
Unfortunately these directions are for 1.1.x and later. Most of the
paths/filenames have changed since 1.0.4 in the move to the FHS
style layout, and there is no debuginfo package. But we may still
be able to get a core file and some stack information:<br>
<br>
sysctl -w fs.suid_dumpable=1<br>
<br>
edit /opt/fedora-ds/slapd-YOURINSTANCE/start-slapd<br>
somewhere near the top, add the line<br>
ulimit -c unlimited<br>
<br>
restart the directory server<br>
/opt/fedora-ds/slapd-YOURINSTANCE/restart-slapd<br>
<br>
If you get a crash, you should have a core file in
/opt/fedora-ds/slapd-YOURINSTANCE/logs<br>
<br>
After that, install gdb<br>
<br>
follow the instructions at <a moz-do-not-send="true"
href="http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes">
http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes</a><br>
except:<br>
cd /opt/fedora-ds/slapd-YOURINSTANCE/logs<br>
gdb ../../bin/slapd/server/ns-slapd core.PID<br>
<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal">I have turned buffering off and increased
the logging level in the LDAP config.
</p>
</div>
</blockquote>
What is the last operation in the access log before a crash? Any
corresponding errors in the errors log?<br>
<br>
<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"> </p>
<p class="MsoNormal">Here is our “monitor” script output: </p>
<p class="MsoNormal">version: 1</p>
<p class="MsoNormal">dn: cn=monitor</p>
<p class="MsoNormal">objectClass: top</p>
<p class="MsoNormal">objectClass: extensibleObject</p>
<p class="MsoNormal">cn: monitor</p>
<p class="MsoNormal">version: Fedora-Directory/1.0.4
B2006.312.1539</p>
<p class="MsoNormal">threads: 30</p>
<p class="MsoNormal">currentconnections: 19</p>
<p class="MsoNormal">totalconnections: 11918</p>
<p class="MsoNormal">dtablesize: 8192</p>
<p class="MsoNormal">readwaiters: 0</p>
<p class="MsoNormal">opsinitiated: 43703</p>
<p class="MsoNormal">opscompleted: 43702</p>
<p class="MsoNormal">entriessent: 16086</p>
<p class="MsoNormal">bytessent: 2911011</p>
<p class="MsoNormal">currenttime: 20110805164243Z</p>
<p class="MsoNormal">starttime: 20110805114053Z</p>
<p class="MsoNormal">nbackends: 2</p>
</div>
</blockquote>
So about 8700 ops/hour. Not a heavy load.<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Here is our “Access Log Analyzer” summary
for a 24 hour period:
</p>
<p class="MsoNormal">---------------------------------------------------------------</p>
<p class="MsoNormal">Access Log Analyzer 6.0</p>
<p class="MsoNormal">Filename Total
Lines Lines processed</p>
<p class="MsoNormal">---------------------------------------------------------------</p>
<p class="MsoNormal">/opt/fedora-ds/slapd/logs/access
298225 298231</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">----------- Access Log Output ------------</p>
<p class="MsoNormal">Restarts: 6</p>
<p class="MsoNormal">Total Connections: 39720</p>
<p class="MsoNormal">Peak Concurrent Connections: 84</p>
<p class="MsoNormal">Total Operations: 95471</p>
<p class="MsoNormal">Total Results: 95393</p>
<p class="MsoNormal">Overall Performance: 99.9%</p>
<p class="MsoNormal">Searches: 48215</p>
<p class="MsoNormal">Modifications: 167</p>
<p class="MsoNormal">Adds: 551</p>
<p class="MsoNormal">Deletes: 2</p>
<p class="MsoNormal">Mod RDNs: 0</p>
<p class="MsoNormal">6.x Stats</p>
<p class="MsoNormal">Persistent Searches: 0</p>
<p class="MsoNormal">Internal Operations: 0</p>
<p class="MsoNormal">Entry Operations: 0</p>
<p class="MsoNormal">Extended Operations: 845</p>
<p class="MsoNormal">Abandoned Requests: 0</p>
<p class="MsoNormal">Smart Referrals Received: 0</p>
<p class="MsoNormal">VLV Operations: 0</p>
<p class="MsoNormal">VLV Unindexed Searches: 0</p>
<p class="MsoNormal">SORT Operations: 0</p>
<p class="MsoNormal">SSL Connections: 0</p>
<p class="MsoNormal">Entire Search Base Queries: 0</p>
<p class="MsoNormal">Unindexed Searches: 6</p>
<p class="MsoNormal">FDs Taken: 39720</p>
<p class="MsoNormal">FDs Returned: 39657</p>
<p class="MsoNormal">Highest FD Taken: 93</p>
<p class="MsoNormal">Broken Pipes: 0</p>
<p class="MsoNormal">Connections Reset By Peer: 0</p>
<p class="MsoNormal">Resource Unavailable: 10872</p>
<p class="MsoNormal"> - 10872 (T1) Idle Timeout Exceeded</p>
<p class="MsoNormal">Binds: 45691</p>
<p class="MsoNormal">Unbinds: 27987</p>
<p class="MsoNormal">LDAP v2 Binds: 15694</p>
<p class="MsoNormal">LDAP v3 Binds: 29997</p>
<p class="MsoNormal">SSL Client Binds: 0</p>
<p class="MsoNormal">Failed SSL Client Binds: 0</p>
<p class="MsoNormal">SASL Binds: 0</p>
<p class="MsoNormal">Directory Manager Binds: 0</p>
<p class="MsoNormal">Anonymous Binds: 16346</p>
<p class="MsoNormal">Other Binds: 29345</p>
<p class="MsoNormal">---------------------------------------------------------------</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">In FDS console: </p>
<p class="MsoNormal">-- Configuration > Performance tab: Size
Limit: 2000, Time Limit: 3600, Idle Timeout: 60, Max file
descriptors: 8192.
</p>
</div>
</blockquote>
The idle timeout is 1 minute - could be too low for some of your
clients, which is why you're seeing a lot of (T1) Idle Timeout
Exceeded connection closes.<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal">-- Configuration > Data > Database
Link Settings > Connection Management: Max TCP Connections:
10, Bind timeout: 20, Max binds per connection: 20, Timeout
before abandon: 10, Max LDAP Connections: 20, Max bind
retries: 3, Max operations per connection: 5, connection life:
60. </p>
</div>
</blockquote>
Are you using database links?<br>
<br>
I also suggest looking at your database cache tuning - see
<a class="moz-txt-link-freetext" href="http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html-single/Administration_Guide/index.html#Monitoring_Server_and_Database_Activity-Monitoring_Database_Activity">http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/8.2/html-single/Administration_Guide/index.html#Monitoring_Server_and_Database_Activity-Monitoring_Database_Activity</a><br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"> </p>
<p class="MsoNormal">We have talked about moving to the latest
389 Directory packages and I have a migration process tested
out so it’s a matter of getting the OK and time but I doubt
the upgrade will solve our crashing problem.</p>
</div>
</blockquote>
I can't say for sure, but 1.0.4 is very old, and since then we have
fixed many issues which have caused crashes.<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal">It seems to me we are hitting some limits
that just haven’t been accounted for yet and that is where I
need help.
</p>
</div>
</blockquote>
Let's start with analyzing the crash data - if we can get a core
file and a stack trace, then we can work from there to figure out
why it's crashing.<br>
<blockquote
cite="mid:24D9F209C12D1E469D6FF80CF72133F6023F746E@NEBVEXCHP01.bhcorp.ad"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"> </p>
<p class="MsoNormal">Any suggestions on how to proceed with
stopping these crashes is welcomed! Thanks for reading.
</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><b><span style="font-size: 7.5pt; color:
black;">Trevor</span></b><span style="font-size: 7.5pt;
color: black;"></span></p>
<p class="MsoNormal"> </p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
This electronic message transmission contains information from
Black Hills Corporation, its affiliate or subsidiary, which may
be confidential or privileged. The information is intended to be
for the use of the individual or entity named above. If you are
not the intended recipient, be aware the disclosure, copying,
distribution or use of the contents of this information is
prohibited. If you received this electronic transmission in
error, please reply to sender immediately; then delete this
message without copying it or further reading.<br>
</font>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
--
389 users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:389-users@lists.fedoraproject.org">389-users@lists.fedoraproject.org</a>
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/mailman/listinfo/389-users">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></pre>
</blockquote>
<br>
</body>
</html>