What do you think about change the OS overcommit_memory to 2 instead of 0???<div><br></div><div>Moses</div><div class="gmail_extra"><br><br><div class="gmail_quote">2012/11/26 Moisés Barba Pérez <span dir="ltr"><<a href="mailto:mbarperoi@gmail.com" target="_blank">mbarperoi@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi,</div><div><br></div><div> This is the end of graph I have get whit the command (while true; do ps -o 'vsz,rss' <PID>; sleep 60; done). Looks like there is not a big increase at any point. I had set swappiness to 10 and the ns-slapd lives for 2 days approx.</div>
<div><br></div><div>Any idea?</div><div><br></div><img src="cid:ii_13b3bbebfcecef3f" alt="Imágenes integradas 1" width="744" height="297"><br><div><br></div><div>Regards, Moses.</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">
2012/11/23 Moisés Barba Pérez <span dir="ltr"><<a href="mailto:mbarperoi@gmail.com" target="_blank">mbarperoi@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm monitoring it right now, sending it on monday.<div><br></div><div>Anyway, is there any tunning configuration for number of connections, memory, etc... that I can follow?? I have use <a href="https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html" target="_blank">https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system-tuning.html</a> and <a href="http://directory.fedoraproject.org/wiki/Performance_Tuning" target="_blank">http://directory.fedoraproject.org/wiki/Performance_Tuning</a> but I don't know if there is some specific for serveral databases, or multiple replication agreements, or very high number of searches (I have 27 database and 60 replication agreements and about 200 searches per second at rush hours)</div>
<div><br></div><div>Regards, Moses</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">2012/11/23 Ludwig Krispenz <span dir="ltr"><<a href="mailto:lkrispen@redhat.com" target="_blank">lkrispen@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi,<br>
<br>
from the data you show, the server process should never reach
11GB, so It could be that you run into a memory leak. Could you
monitor process size growth ?<br>
Start the server, prime the caches for all backends and monitor
process growth, eg running regular <br>
ps -o 'vsz,rss' <pid><br>
See how fast the process grows, if it is steadily or if there is a
pattern and you can relate it to some cliend load.<br>
<br>
Regards,<br>
Ludwig<div><div><br>
<br>
On 11/22/2012 02:33 PM, Moisés Barba Pérez wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<div><br>
</div>
Hi,
<div><br>
</div>
<div>I have been searching for memory usage in the server. This
are the results:</div>
<div><br>
</div>
<div><span style="font-family:arial,sans-serif;font-size:13px">389-ds
1.2.5 in a CentOS 5.5 64bits 4GB ram and 6GB swap</span><br>
</div>
<div><br>
</div>
<div>* The ns-slapd proccess reaches 11GB of virtual memory. pmap
shows multiple [anon] using the bigger part of that 11G virtual
memory. I think the [anon] are memory reservation from malloc
and mmap but I don't know what call this.</div>
<div><br>
</div>
<div>* Looking for cachememsize using this search for one of the
database</div>
<div><br>
</div>
<div>
<div><font color="#3d85c6">ldapsearch -H <a>ldaps://localhost</a> -x
-LLL -b "cn=monitor,cn=o_xxxx,cn=ldbm
database,cn=plugins,cn=config" -D "cn=Directory Manager" -W
"(objectclass=*)" | grep entrycache</font></div>
<div><font color="#3d85c6">Enter LDAP Password: </font></div>
<div><font color="#3d85c6">entrycachehitratio: 99<br>
</font></div>
<div><font color="#3d85c6">currententrycachesize: 49973691</font></div>
<div><font color="#3d85c6">maxentrycachesize: 125829120</font></div>
<div><font color="#3d85c6">currententrycachecount: 6521</font></div>
<div><font color="#3d85c6">maxentrycachecount: -1</font></div>
</div>
<div><br>
</div>
<div>I have prime that database searching all entries with -><font color="#3d85c6"> ldapsearch -H <a>ldaps://localhost</a> -x -LLL -b
"o=cabu,dc=sacyl,dc=es" -D "cn=directory manager" -W
"(objectclass=*)" 1.1 | grep dn: | wc -l </font></div>
<div>The result is 7610 entries in that database, so looking the
monitor again:</div>
<div><br>
</div>
<div>
<div><font color="#3d85c6">currententrycachesize: 59315175</font></div>
<div><font color="#3d85c6">maxentrycachesize: 125829120</font></div>
<div><font color="#3d85c6">currententrycachecount: 7611</font></div>
</div>
<div><br>
</div>
<div>The id2entry.db4 for that database is 59539456 so I guess I
can reduce the cachememsize from 125829120 to about 60000000
Correct me if I am wrong.</div>
<div>And the same for all the another database.</div>
<div><br>
</div>
<div>* Now dbcachesize:</div>
<div><br>
</div>
<div>
<div><font color="#3d85c6">ldapsearch -H <a>ldaps://localhost</a> -x
-LLL -b "cn=monitor, cn=ldbm database, cn=plugins,cn=config"
-D "cn=Directory Manager" -W "(objectclass=*)" | grep
dbcache</font></div>
<div><font color="#3d85c6">Enter LDAP Password: </font></div>
<div><font color="#3d85c6">dbcachehits: 1440910461</font></div>
<div><font color="#3d85c6">dbcachetries: 1440919648</font></div>
<div><font color="#3d85c6">dbcachehitratio: 99</font></div>
<div><font color="#3d85c6">dbcachepagein: 9187</font></div>
<div><font color="#3d85c6">dbcachepageout: 128041</font></div>
<div><font color="#3d85c6">dbcacheroevict: 9265</font></div>
<div><font color="#3d85c6">dbcacherwevict: 0</font></div>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">In some place I have read that
dbcacheroevict and dbcachepageout should be 0 or increase the
dbcachesize but if the ratio is 99 that should be ok, right?</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The thing is, if i search with db_stat
for cache statistics says ratio=99</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div class="gmail_extra">
<font color="#3d85c6">db_stat -h /var/lib/dirsrv/slapd-xxx/db/
-m</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Total cache size</font></div>
<div class="gmail_extra">
<font color="#3d85c6">1<span style="white-space:pre-wrap">
</span>Number of caches</font></div>
<div class="gmail_extra"><font color="#3d85c6">800MB<span style="white-space:pre-wrap"> </span>Pool individual
cache size</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Maximum memory-mapped
file size</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Maximum open file
descriptors</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Maximum sequential buffer
writes</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Sleep after writing
maximum sequential buffers</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Requested pages mapped
into the process' address space</font></div>
<div class="gmail_extra"><font color="#3d85c6">1448M<span style="white-space:pre-wrap"> </span>Requested pages
found in the cache (99%)</font></div>
<div class="gmail_extra"><font color="#3d85c6">9588<span style="white-space:pre-wrap"> </span>Requested pages
not found in the cache</font></div>
<div class="gmail_extra"><font color="#3d85c6">112<span style="white-space:pre-wrap"> </span>Pages created in the
cache</font></div>
<div class="gmail_extra"><font color="#3d85c6">9588<span style="white-space:pre-wrap"> </span>Pages read into
the cache</font></div>
<div class="gmail_extra"><font color="#3d85c6">129932<span style="white-space:pre-wrap"> </span>Pages written
from the cache to the backing file</font></div>
<div class="gmail_extra"><font color="#3d85c6">9668<span style="white-space:pre-wrap"> </span>Clean pages
forced from the cache</font></div>
<div class="gmail_extra"><font color="#3d85c6">1<span style="white-space:pre-wrap"> </span>Dirty pages forced from
the cache</font></div>
<div class="gmail_extra"><font color="#3d85c6">0<span style="white-space:pre-wrap"> </span>Dirty pages written by
trickle-sync thread</font></div>
<div class="gmail_extra"><font color="#3d85c6">98066<span style="white-space:pre-wrap"> </span>Current total
page count</font></div>
<div class="gmail_extra"><font color="#3d85c6">98005<span style="white-space:pre-wrap"> </span>Current clean
page count</font></div>
<div class="gmail_extra"><font color="#3d85c6">61<span style="white-space:pre-wrap"> </span>Current dirty page count</font></div>
<div class="gmail_extra"><font color="#3d85c6">131071<span style="white-space:pre-wrap"> </span>Number of hash
buckets used for page location</font></div>
<div class="gmail_extra"><font color="#3d85c6">1447M<span style="white-space:pre-wrap"> </span>Total number of
times hash chains searched for a page (1447895423)</font></div>
<div class="gmail_extra"><font color="#3d85c6">5<span style="white-space:pre-wrap"> </span>The longest hash chain
searched for a page</font></div>
<div class="gmail_extra"><font color="#3d85c6">2819M<span style="white-space:pre-wrap"> </span>Total number of
hash buckets examined for page location <a href="tel:%282819107178" value="+12819107178" target="_blank">(2819107178</a>)</font></div>
<div class="gmail_extra"><font color="#3d85c6">932<span style="white-space:pre-wrap"> </span>The number of hash bucket
locks that required waiting (0%)</font></div>
<div class="gmail_extra"><font color="#3d85c6">86<span style="white-space:pre-wrap"> </span>The maximum number of
times any hash bucket lock was waited for</font></div>
<div class="gmail_extra"><font color="#3d85c6">1<span style="white-space:pre-wrap"> </span>The number of region
locks that required waiting (0%)</font></div>
<div class="gmail_extra"><font color="#3d85c6">9728<span style="white-space:pre-wrap"> </span>The number of
page allocations</font></div>
<div class="gmail_extra"><font color="#3d85c6">60012<span style="white-space:pre-wrap"> </span>The number of
hash buckets examined during allocations</font></div>
<div class="gmail_extra"><font color="#3d85c6">1381<span style="white-space:pre-wrap"> </span>The maximum
number of hash buckets examined for an allocation</font></div>
<div class="gmail_extra"><font color="#3d85c6">9669<span style="white-space:pre-wrap"> </span>The number of
pages examined during allocations</font></div>
<div class="gmail_extra"><font color="#3d85c6">1<span style="white-space:pre-wrap"> </span>The max number of pages
examined for an allocation</font></div>
<div><br>
</div>
</div>
<div class="gmail_extra">If I look for an index like
inetuserstatus (pres and eq) I get "Requested pages found in the
cache" less than 99% so I search for "inetuserstatus=*" (pres)
and "inetuserstatus=active", "inetuserstatus=inactive" (eq) but
the "requested pages" don't reaches the 99 or 100% and there is
no more possibilities for that index.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The thing is, why ns-sldapd is growing to
consume all the swap and all the ram memory the SO lets it. </div>
<div class="gmail_extra">
Any idea or suggestion???</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2012/11/15 Ludwig Krispenz <span dir="ltr"><<a href="mailto:lkrispen@redhat.com" target="_blank">lkrispen@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>you could use <br>
ldapsearch ... -b "cn=ldbm
database,cn=plugins,cn=config" "cn=monitor"
currententrycachesize<br>
<br>
to monitor the usage of the entrycache. <br>
But be aware that the process uses more memory than just
the caches and the memory manager can also generate some
overhead.<br>
<br>
Regards,<br>
Ludwig
<div>
<div><br>
<br>
On 11/15/2012 02:55 PM, Moisés Barba Pérez wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">yes, thats correct, but
shouldn't use all that memory because don't need so
much memory
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2012/11/15 Ludwig
Krispenz <span dir="ltr"><<a href="mailto:lkrispen@redhat.com" target="_blank">lkrispen@redhat.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi,
<div>
<div><br>
On 11/15/2012 01:54 PM, Moisés Barba
Pérez wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">Hi,
<div><br>
</div>
<div>I have a memory issue with 389-ds
1.2.5 in a CentOS 5.5 64bits 4GB
ram. </div>
<div><br>
</div>
<div>The server swaps when the server
physical memory increase over 75%
approx. When the swap is full the
server reaches 100% of physical
memory and the SO kills the ns-ldapd
process.</div>
<div><br>
</div>
<div>
<div>Out of memory: Killed process
30383, UID 99, (ns-slapd).</div>
</div>
<div><br>
</div>
<div>The cache sizes are:</div>
<div><br>
</div>
<div>
<div>nsslapd-dbcachesize: 838860800</div>
<div>nsslapd-import-cachesize:
20000000</div>
<div>nsslapd-cachememsize: 125829120
(for each 26 db)</div>
</div>
</blockquote>
</div>
</div>
do you mean you have 26 db backends with
125MB entrycache each ? So you would reach
3.2GB for entrycache and 800MB dbcache.<br>
<br>
Regards,<br>
Ludwig
<div><br>
<blockquote type="cite">
<div><br>
</div>
<div>Which can be the problem?</div>
</blockquote>
<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<br>
<span><font color="#888888">
<fieldset></fieldset>
<br>
<pre>--
389 users mailing list
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></pre>
</font></span></blockquote>
<br>
</div>
<br>
--<br>
389 users mailing list<br>
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a><br>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>--
389 users mailing list
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
--<br>
389 users mailing list<br>
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a><br>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>--
389 users mailing list
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a></pre>
</blockquote>
<br>
</div></div></div>
<br>--<br>
389 users mailing list<br>
<a href="mailto:389-users@lists.fedoraproject.org" target="_blank">389-users@lists.fedoraproject.org</a><br>
<a href="https://admin.fedoraproject.org/mailman/listinfo/389-users" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/389-users</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>