[Fedora-directory-users] reducing memory footprint?

[Gerald Richter]

Hi,

I just made a test installation of FDS and saw that a ns-slapd without any 
user data takes about 120MB of (virtual) memory on my system.

I would like to run it on a system which limited memory resources, so I am 
looking for a way to use less memory.

I don't have high load on that system and never more than one or two quries in 
parallel, so it would be quite ok to reduce the number of threads and things 
like this, but beside reducing the cache size of the backend DB I didn't find 
any hints what can be done in this direction.

Any ideas?

Thanks

Gerald

 

 
** Virus checked by BB-5000 Mailfilter **

[George Holbert]

Hi Gerald,

HP has a tuning guide for their bundled Netscape DS, which may be 
somewhat useful to you for this:

http://docs.hp.com/en/7152/nds621_tuning_sizing_13.pdf

Of course, Fedora DS and HP's DS are not the same product, but they have 
common heritage.

Excerpt:
>  The Netscape Directory Server for HP-UX caches entry and indexing 
>  information in memory. HP-UX requires at
>  least 256 MB of memory for a small deployment. But for large directory 
>  servers, 512MB to 4GB RAM is needed for
>  best performance. To estimate how much RAM needed for Directory Server 
>  on a system, please use the following
>  formula:
>  Total_NDS_RAM = 1.2 * (base_RAM_need_for_slapd_process + caches)
>  Where
>  base_RAM_needed_for_slapd_process = 32MB + nsslapd-threadnumber * 1MB
>  caches = dbcache + SUM(all entry caches) + import_cache
>  Explanation:
>  · 1.2: 20% additional RAM needed for slapd process to handle incoming 
>  LDAP operations. 20% is an
>  estimated number, and it should be sufficient. However, testing is 
>  needed to ensure that it is enough before
>  going into production.
>  · 32MB: is the size of the slapd process.
>  · nsslapd-threadnumber *1MB: each thread needs about 1MB of memory.
>  · dbcache: specified as nsslapd-dbcachesize.
>  · All entry caches: specified as nsslapd-cachememsize. 

Gerald Richter wrote:
>  Hi,
>
>  I just made a test installation of FDS and saw that a ns-slapd without any 
>  user data takes about 120MB of (virtual) memory on my system.
>
>  I would like to run it on a system which limited memory resources, so I am 
>  looking for a way to use less memory.
>
>  I don't have high load on that system and never more than one or two quries in 
>  parallel, so it would be quite ok to reduce the number of threads and things 
>  like this, but beside reducing the cache size of the backend DB I didn't find 
>  any hints what can be done in this direction.
>
>  Any ideas?
>
>  Thanks
>
>  Gerald
>
>   
>
>   
>  ** Virus checked by BB-5000 Mailfilter ** 
>
>  --
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>

[Ulf Weltman]

Hello George and Gerald.  I'm afraid the tuning guide wont help much 
with reducing memory footprint, it focuses on increasing performance 
which involves using more memory among other things! :)  There is a 
document for the NSDS 7.0 which is not far from the FDS 1.0 codebase if 
you're still interested, but the measurements and tuning suggestions are 
meant for DS running on HP-UX.

It does answer one of Gerald's questions:  worker threads can be reduced 
with nsslapd-threadnumber, the default is 30.  I don't know that this 
will save you significant memory on Linux.

Ulf

George Holbert wrote:

>  Hi Gerald,
>
>  HP has a tuning guide for their bundled Netscape DS, which may be 
>  somewhat useful to you for this:
>
>  http://docs.hp.com/en/7152/nds621_tuning_sizing_13.pdf
>
>  Of course, Fedora DS and HP's DS are not the same product, but they 
>  have common heritage.
>
>  Excerpt:
>
> > The Netscape Directory Server for HP-UX caches entry and indexing 
> > information in memory. HP-UX requires at
> > least 256 MB of memory for a small deployment. But for large 
> > directory servers, 512MB to 4GB RAM is needed for
> > best performance. To estimate how much RAM needed for Directory 
> > Server on a system, please use the following
> > formula:
> > Total_NDS_RAM = 1.2 * (base_RAM_need_for_slapd_process + caches)
> > Where
> > base_RAM_needed_for_slapd_process = 32MB + nsslapd-threadnumber * 1MB
> > caches = dbcache + SUM(all entry caches) + import_cache
> > Explanation:
> > · 1.2: 20% additional RAM needed for slapd process to handle incoming 
> > LDAP operations. 20% is an
> > estimated number, and it should be sufficient. However, testing is 
> > needed to ensure that it is enough before
> > going into production.
> > · 32MB: is the size of the slapd process.
> > · nsslapd-threadnumber *1MB: each thread needs about 1MB of memory.
> > · dbcache: specified as nsslapd-dbcachesize.
> > · All entry caches: specified as nsslapd-cachememsize.
>
>
>
>  Gerald Richter wrote:
>
> > Hi,
> >
> > I just made a test installation of FDS and saw that a ns-slapd 
> > without any user data takes about 120MB of (virtual) memory on my 
> > system.
> >
> > I would like to run it on a system which limited memory resources, so 
> > I am looking for a way to use less memory.
> >
> > I don't have high load on that system and never more than one or two 
> > quries in parallel, so it would be quite ok to reduce the number of 
> > threads and things like this, but beside reducing the cache size of 
> > the backend DB I didn't find any hints what can be done in this 
> > direction.
> >
> > Any ideas?
> >
> > Thanks
> >
> > Gerald
> >
> >  
> >
> >  
> > ** Virus checked by BB-5000 Mailfilter **
> > -- 
> > Fedora-directory-users mailing list
> > Fedora-directory-users(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/fedora-directory-users
> >
> >
> >   
>
>
>
>
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[George Holbert]

Ulf, thanks for the clarification.

Gerald, I should have mentioned, the HP tuning guide is not explicitly a 
guide for reducing memory usage.  But, some of the text (such as the 
excerpt I pasted) may be of use to you.

Good luck,
-- George

Ulf Weltman wrote:
>  Hello George and Gerald.  I'm afraid the tuning guide wont help
> much 
>  with reducing memory footprint, it focuses on increasing performance 
>  which involves using more memory among other things! :)  There is a 
>  document for the NSDS 7.0 which is not far from the FDS 1.0 codebase 
>  if you're still interested, but the measurements and tuning 
>  suggestions are meant for DS running on HP-UX.
>
>  It does answer one of Gerald's questions:  worker threads can be 
>  reduced with nsslapd-threadnumber, the default is 30.  I don't know 
>  that this will save you significant memory on Linux.
>
>  Ulf
>
>  George Holbert wrote:
>
> > Hi Gerald,
> >
> > HP has a tuning guide for their bundled Netscape DS, which may be 
> > somewhat useful to you for this:
> >
> > http://docs.hp.com/en/7152/nds621_tuning_sizing_13.pdf
> >
> > Of course, Fedora DS and HP's DS are not the same product, but they 
> > have common heritage.
> >
> > Excerpt:
> >
> >> The Netscape Directory Server for HP-UX caches entry and indexing 
> >> information in memory. HP-UX requires at
> >> least 256 MB of memory for a small deployment. But for large 
> >> directory servers, 512MB to 4GB RAM is needed for
> >> best performance. To estimate how much RAM needed for Directory 
> >> Server on a system, please use the following
> >> formula:
> >> Total_NDS_RAM = 1.2 * (base_RAM_need_for_slapd_process + caches)
> >> Where
> >> base_RAM_needed_for_slapd_process = 32MB + nsslapd-threadnumber * 1MB
> >> caches = dbcache + SUM(all entry caches) + import_cache
> >> Explanation:
> >> · 1.2: 20% additional RAM needed for slapd process to handle 
> >> incoming LDAP operations. 20% is an
> >> estimated number, and it should be sufficient. However, testing is 
> >> needed to ensure that it is enough before
> >> going into production.
> >> · 32MB: is the size of the slapd process.
> >> · nsslapd-threadnumber *1MB: each thread needs about 1MB of memory.
> >> · dbcache: specified as nsslapd-dbcachesize.
> >> · All entry caches: specified as nsslapd-cachememsize.
> >
> >
> >
> > Gerald Richter wrote:
> >
> >> Hi,
> >>
> >> I just made a test installation of FDS and saw that a ns-slapd 
> >> without any user data takes about 120MB of (virtual) memory on my 
> >> system.
> >>
> >> I would like to run it on a system which limited memory resources, 
> >> so I am looking for a way to use less memory.
> >>
> >> I don't have high load on that system and never more than one or two 
> >> quries in parallel, so it would be quite ok to reduce the number of 
> >> threads and things like this, but beside reducing the cache size of 
> >> the backend DB I didn't find any hints what can be done in this 
> >> direction.
> >>
> >> Any ideas?
> >>
> >> Thanks
> >>
> >> Gerald
> >>
> >>  
> >>
> >>  
> >> ** Virus checked by BB-5000 Mailfilter **
> >> -- 
> >> Fedora-directory-users mailing list
> >> Fedora-directory-users(a)redhat.com
> >> https://www.redhat.com/mailman/listinfo/fedora-directory-users
> >>
> >>
> >>   
> >
> >
> >
> >
> >
> > -- 
> > Fedora-directory-users mailing list
> > Fedora-directory-users(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/fedora-directory-users
> >
>
>
>  -- 
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>

[Pete Rowley]

Gerald Richter wrote:

> Hi,
>
> I just made a test installation of FDS and saw that a ns-slapd without any 
> user data takes about 120MB of (virtual) memory on my system.
>
> I would like to run it on a system which limited memory resources, so I am 
> looking for a way to use less memory.
>   
>  
This has very much not been the typical deployment for this server.  
That is not to say that it cannot be made to run in a small space - just 
that not much effort has been put into finding out.

> I don't have high load on that system and never more than one or
> two quries in 
> parallel, so it would be quite ok to reduce the number of threads and things 
> like this, but beside reducing the cache size of the backend DB I didn't find 
> any hints what can be done in this direction.
>
> Any ideas?
>   
>  
In general turn things off that that you don't need /e.g./ disable 
plugins that provide features you do not need.  Reduce the size of or 
turn off the entry cache, in the default case it makes a reasonable 
guess at what useful a cache size would be for good performance.  If you 
are seriously hard up for ram you might consider removing schema files 
you do not need.

Let us know how lean you get it and what you did :)

-- 
Pete

[Fedora-directory-users] RE: reducing memory footprint?

[Gerald Richter]

Hi all,

Thanks for your suggestions.

I tried it out and here is the result (everything is after startup and
without any load or work. The numbers are the virtual memory that ps shows):

- default configuration ~ 122MB

- reduce db cache from 10MB to 1 MB ~ 113MB

- reduce number of threads from 30 to 3 ~ 57MB

- disable unneeded plugins ~ 55MB

BTW This is still a factor of 5 higher than OpenLdap, which takes about 10MB

> From what you wrote and from what I read and what my tests shows: It
> really 
seems that FDS is good for huge databases with high load, but not for the
small ones with limited computer resources.

Gerald
 

>  -----Original Message-----
>  From: Gerald Richter [mailto:[e-mail]] 
>  Sent: Thursday, December 22, 2005 6:23 AM
>  To: fedora-directory-users(a)redhat.com
>  Subject: reducing memory footprint?
>  
>  Hi,
>  
>  I just made a test installation of FDS and saw that a 
>  ns-slapd without any user data takes about 120MB of (virtual) 
>  memory on my system.
>  
>  I would like to run it on a system which limited memory 
>  resources, so I am looking for a way to use less memory.
>  
>  I don't have high load on that system and never more than one 
>  or two quries in parallel, so it would be quite ok to reduce 
>  the number of threads and things like this, but beside 
>  reducing the cache size of the backend DB I didn't find any 
>  hints what can be done in this direction.
>  
>  Any ideas?
>  
>  Thanks
>  
>  Gerald
>  
>   
>   

 
** Virus checked by BB-5000 Mailfilter **

[Fedora-directory-users] RE: reducing memory footprint?

[Mike Jackson]

Gerald Richter wrote:

>  I tried it out and here is the result (everything is after startup
> and
>  without any load or work. The numbers are the virtual memory that ps shows):
>  
>  - default configuration ~ 122MB
>  
>  - reduce db cache from 10MB to 1 MB ~ 113MB
>  
>  - reduce number of threads from 30 to 3 ~ 57MB
>  
>  - disable unneeded plugins ~ 55MB
>  
>  BTW This is still a factor of 5 higher than OpenLdap, which takes about 10MB 

And OpenLDAP is still ~100 times bigger than:

http://www.fefe.de/tinyldap/ - 11k


>  From what you wrote and from what I read and what my tests shows: It
> really
>  seems that FDS is good for huge databases with high load, but not for the
>  small ones with limited computer resources. 
That is not IMO a logical assumption or something which can be proven, 
as you have not defined "huge", "small ones", or "limited
computer 
resources".

BR,
--
mike

[Fedora-directory-users] RE: reducing memory footprint?

[David Boreham]

Gerald Richter wrote:

> Hi all,
>
> Thanks for your suggestions.
>
> I tried it out and here is the result (everything is after startup and
> without any load or work. The numbers are the virtual memory that ps shows):
>
> - default configuration ~ 122MB
>
> - reduce db cache from 10MB to 1 MB ~ 113MB
>
> - reduce number of threads from 30 to 3 ~ 57MB
>
> - disable unneeded plugins ~ 55MB
>
> BTW This is still a factor of 5 higher than OpenLdap, which takes about 10MB
>   
>  
Can you explain a little more about your goal here ?
Trying to reduce the VM footprint for an application seems (to me) to be
a fruitless exercise. It'd be the working set size that one would typically
want to reduce. VM size is an (almost) meaningless number. For example
when you reduced the number of threads, the VM size went down because
you removed N * (thread stack size) from the process address space.
However, in reality you didn't change the application's use of system
resources significantly at all because almost none of the pages would
have been comitted. OpenLDAP probably uses a smaller stack size.

I would be interested in hearing if the WSS is too large. I don't have
a running slapd to check to hand, but I don't remember it being
terribly large last time I checked (20 meg from memory).