Finger slow and optimizing performance

[Hartmann, Tim]

Hi!


I was spending some time today trying to make sure that I was getting 
the most bang for my buck today an my replica's and I notices two items 
of interest that I was wondering if anyone else had input on!

Firstly, after creating a number of indexs, my performance seems to be 
really good, the exception that I noticed was "finger" I noticed that 
finger takes a couple of seconds to return the data on RHDS whereas on 
OpenLDAP, it pops right now in real time! My first though was that I was 
doing an un-indexed search, but I can't for the life of me figure out 
what I might not be indexing that I should be!

The second thing I noticed was that on my servers, which are RHEL5, 
running 32bit OS's with the PAE Kernels, RHDS doesn't ever actually 
address more then 3 gig of ram! I was looking through the 
documentations, and it looks like by raising the "Maximum Cache Size" 
I'll be able to allow RHDS to use more of the available memory.. did I 
get that right?


Anyway, as always thanks in advance for all the help! This list has been 
a tremendous resource for an application that keeps on showing it's 
value in huge ways!


Best,

Tim

[Ivanov Andrey (M.)]

Hi,


There may be several attributes of interest to you as far as the memory
consumption is concerned (
http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
:
nsslapd-dbcachesize
nsslapd-cachememsize for every backend (by default, your data is in
cn=userRoot,cn=ldbm database,cn=plugins,cn=config)
nsslapd-import-cachesize (used only during ldif import)

You can adjust the corresponding values by monitoring the attributes like
currententrycachesize or entrycachehitratio of
cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config (
http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...
)



2009/6/26 Tim Hartmann <hartmann(a)fas.harvard.edu&gt;

>  Hi!
>
>
>  I was spending some time today trying to make sure that I was getting the
>  most bang for my buck today an my replica's and I notices two items of
>  interest that I was wondering if anyone else had input on!
>
>  Firstly, after creating a number of indexs, my performance seems to be
>  really good, the exception that I noticed was "finger" I noticed that finger
>  takes a couple of seconds to return the data on RHDS whereas on OpenLDAP, it
>  pops right now in real time! My first though was that I was doing an
>  un-indexed search, but I can't for the life of me figure out what I might
>  not be indexing that I should be!
>
>  The second thing I noticed was that on my servers, which are RHEL5, running
>  32bit OS's with the PAE Kernels, RHDS doesn't ever actually address more
>  then 3 gig of ram! I was looking through the documentations, and it looks
>  like by raising the "Maximum Cache Size" I'll be able to allow RHDS to use
>  more of the available memory.. did I get that right?
>
>
>  Anyway, as always thanks in advance for all the help! This list has been a
>  tremendous resource for an application that keeps on showing it's value in
>  huge ways!
>
>
>  Best,
>
>  Tim
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[Rich Megginson]

Andrey Ivanov wrote:
>  Hi,
>
>
>  There may be several attributes of interest to you as far as the 
>  memory consumption is concerned 
>
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
>
>  :
>  nsslapd-dbcachesize
>  nsslapd-cachememsize for every backend (by default, your data is in 
>  cn=userRoot,cn=ldbm database,cn=plugins,cn=config)
>  nsslapd-import-cachesize (used only during ldif import) 
Start with
nsslapd-cachememsize - make that as large as possible and 
minimize nsslapd-dbcachesize
>  You can adjust the corresponding values by monitoring the attributes 
>  like currententrycachesize or entrycachehitratio of  
>  cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config 
>
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
>  
You can also use the logconv.pl script to examine the access log to see 
what types of searches are being done and which are not indexed properly.
>  2009/6/26 Tim Hartmann <hartmann(a)fas.harvard.edu 
>  <mailto:[e-mail]>>
>
>      Hi!
>
>
>      I was spending some time today trying to make sure that I was
>      getting the most bang for my buck today an my replica's and I
>      notices two items of interest that I was wondering if anyone else
>      had input on!
>
>      Firstly, after creating a number of indexs, my performance seems
>      to be really good, the exception that I noticed was "finger" I
>      noticed that finger takes a couple of seconds to return the data
>      on RHDS whereas on OpenLDAP, it pops right now in real time! My
>      first though was that I was doing an un-indexed search, but I
>      can't for the life of me figure out what I might not be indexing
>      that I should be!
>
>      The second thing I noticed was that on my servers, which are
>      RHEL5, running 32bit OS's with the PAE Kernels, RHDS doesn't ever
>      actually address more then 3 gig of ram! I was looking through the
>      documentations, and it looks like by raising the "Maximum Cache
>      Size" I'll be able to allow RHDS to use more of the available
>      memory.. did I get that right?
>
>
>      Anyway, as always thanks in advance for all the help! This list
>      has been a tremendous resource for an application that keeps on
>      showing it's value in huge ways!
>
>
>      Best,
>
>      Tim
>
>      --
>      389 users mailing list
>      389-users(a)redhat.com <mailto:[e-mail]>
>      https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>  ------------------------------------------------------------------------
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[Hartmann, Tim]

Rich Megginson wrote:
>  Andrey Ivanov wrote:
> > Hi,
> >
> >
> > There may be several attributes of interest to you as far as the
> > memory consumption is concerned
> >
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
> > :
> > nsslapd-dbcachesize
> > nsslapd-cachememsize for every backend (by default, your data is in
> > cn=userRoot,cn=ldbm database,cn=plugins,cn=config)
> > nsslapd-import-cachesize (used only during ldif import)
>  Start with nsslapd-cachememsize - make that as large as possible and
>  minimize nsslapd-dbcachesize
> >
> > You can adjust the corresponding values by monitoring the attributes
> > like currententrycachesize or entrycachehitratio of 
> > cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
> >
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
> >
> >
>  You can also use the logconv.pl script to examine the access log to
>  see what types of searches are being done and which are not indexed
>  properly.
>  
So after playing with logconv a bit, it looked like finger was making
this call in the logs...

[26/Jun/2009:10:59:36 -0400] conn=283289 op=-1 fd=80 closed - B1
[26/Jun/2009:10:59:36 -0400] conn=283289 op=2 RESULT err=11 tag=101
nentries=0 etime=1 notes=U
[26/Jun/2009:10:59:35 -0400] conn=283289 op=2 SRCH
base="ou=really,ou=long,o=name,dc=school,dc=edu" scope=2
filter="(objectClass=posixAccount)" attrs="uid userPassword uidNumber
gidNumber cn homeDirectory loginShell gecos description objectClass"
[26/Jun/2009:10:59:35 -0400] conn=283289 op=1 RESULT err=0 tag=101
nentries=1 etime=0
[26/Jun/2009:10:59:35 -0400] conn=283289 op=1 SRCH
base="ou=systems,ou=services,o=hascs,dc=fas,dc=harvard,dc=edu" scope=2
filter="(&(objectClass=posixAccount)(uid=foo))" attrs="uid
userPassword
uidNumber gidNumber cn homeDirectory loginShell gecos description
objectClass"
[26/Jun/2009:10:59:35 -0400] conn=283289 op=0 RESULT err=0 tag=97
nentries=0 etime=0 dn=""
[26/Jun/2009:10:59:35 -0400] conn=283289 op=0 BIND dn="" method=128
version=3
[26/Jun/2009:10:59:35 -0400] conn=283289 fd=80 slot=80 connection from
1.2.3.4 to 4.3.2.1



But even after indexing "uid userPassword uidNumber gidNumber cn
homeDirectory loginShell gecos description objectClass"  finger still
responds slower then it does on comparison to the older openLDAP
servers... where we don't do indexing on all of these attributes, AND
still claims that I'm running an search that hasn't been indexed!   I'm
I missing something glaringly obvious? 

Thanks!

Tim

[Rich Megginson]

Tim Hartmann wrote:
>  Rich Megginson wrote:
>    
> > Andrey Ivanov wrote:
> >     
> >> Hi,
> >>
> >>
> >> There may be several attributes of interest to you as far as the
> >> memory consumption is concerned
> >>
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
> >> :
> >> nsslapd-dbcachesize
> >> nsslapd-cachememsize for every backend (by default, your data is in
> >> cn=userRoot,cn=ldbm database,cn=plugins,cn=config)
> >> nsslapd-import-cachesize (used only during ldif import)
> >>       
> > Start with nsslapd-cachememsize - make that as large as possible and
> > minimize nsslapd-dbcachesize
> >     
> >> You can adjust the corresponding values by monitoring the attributes
> >> like currententrycachesize or entrycachehitratio of 
> >> cn=monitor,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
> >>
> (http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...)
> >>
> >>
> >>       
> > You can also use the logconv.pl script to examine the access log to
> > see what types of searches are being done and which are not indexed
> > properly.
> >
> >     
>
>  So after playing with logconv a bit, it looked like finger was making
>  this call in the logs...
>
>  [26/Jun/2009:10:59:36 -0400] conn=283289 op=-1 fd=80 closed - B1
>  [26/Jun/2009:10:59:36 -0400] conn=283289 op=2 RESULT err=11 tag=101
>  nentries=0 etime=1 notes=U
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=2 SRCH
>  base="ou=really,ou=long,o=name,dc=school,dc=edu" scope=2
>  filter="(objectClass=posixAccount)" attrs="uid userPassword uidNumber
>  gidNumber cn homeDirectory loginShell gecos description objectClass"
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=1 RESULT err=0 tag=101
>  nentries=1 etime=0
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=1 SRCH
>  base="ou=systems,ou=services,o=hascs,dc=fas,dc=harvard,dc=edu" scope=2
>  filter="(&(objectClass=posixAccount)(uid=foo))" attrs="uid
> userPassword
>  uidNumber gidNumber cn homeDirectory loginShell gecos description
>  objectClass"
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=0 RESULT err=0 tag=97
>  nentries=0 etime=0 dn=""
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=0 BIND dn="" method=128
>  version=3
>  [26/Jun/2009:10:59:35 -0400] conn=283289 fd=80 slot=80 connection from
>  1.2.3.4 to 4.3.2.1
>
>
>
>  But even after indexing "uid userPassword uidNumber gidNumber cn
>  homeDirectory loginShell gecos description objectClass"  finger still
>  responds slower then it does on comparison to the older openLDAP
>  servers... where we don't do indexing on all of these attributes, AND
>  still claims that I'm running an search that hasn't been indexed!   I'm
>  I missing something glaringly obvious? 
>     
You only need to index the attributes used for searching:
(&(objectClass=posixAccount)(uid=foo))
You need an equality index on objectClass (which should already be 
there, it is one of the default indexes) and an equality index on uid 
(again, should already be there).

The problem is this:
[26/Jun/2009:10:59:36 -0400] conn=283289 op=2 RESULT err=11 tag=101 
nentries=0 etime=1 notes=U
[26/Jun/2009:10:59:35 -0400] conn=283289 op=2 SRCH
base="ou=really,ou=long,o=name,dc=school,dc=edu" scope=2
filter="(objectClass=posixAccount)" attrs="uid userPassword uidNumber
gidNumber cn homeDirectory loginShell gecos description objectClass"

The notes=U and err=11 mean that either the lookthrough limit has been 
exceeded, or you need to increase your nsslapd-idlistscanlimit:
http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...

This is not a good search anyway - the client is basically asking for 
all entries that match objectClass=posixAccount which could be thousands 
or more - what does the client intend to do with all of those entries?
>  Thanks!
>
>  Tim
>
>  --
>  389 users mailing list
>  389-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>

[Hartmann, Tim]

Rich Megginson wrote:
>  Tim Hartmann wrote:
> >
> >>     
> >
> > So after playing with logconv a bit, it looked like finger was making
> > this call in the logs...
> >
> > [26/Jun/2009:10:59:36 -0400] conn=283289 op=-1 fd=80 closed - B1
> > [26/Jun/2009:10:59:36 -0400] conn=283289 op=2 RESULT err=11 tag=101
> > nentries=0 etime=1 notes=U
> > [26/Jun/2009:10:59:35 -0400] conn=283289 op=2 SRCH
> > base="ou=really,ou=long,o=name,dc=school,dc=edu" scope=2
> > filter="(objectClass=posixAccount)" attrs="uid userPassword uidNumber
> > gidNumber cn homeDirectory loginShell gecos description objectClass"
> > [26/Jun/2009:10:59:35 -0400] conn=283289 op=1 RESULT err=0 tag=101
> > nentries=1 etime=0
> > [26/Jun/2009:10:59:35 -0400] conn=283289 op=1 SRCH
> > base="ou=systems,ou=services,o=hascs,dc=fas,dc=harvard,dc=edu" scope=2
> > filter="(&(objectClass=posixAccount)(uid=foo))" attrs="uid
> userPassword
> > uidNumber gidNumber cn homeDirectory loginShell gecos description
> > objectClass"
> > [26/Jun/2009:10:59:35 -0400] conn=283289 op=0 RESULT err=0 tag=97
> > nentries=0 etime=0 dn=""
> > [26/Jun/2009:10:59:35 -0400] conn=283289 op=0 BIND dn="" method=128
> > version=3
> > [26/Jun/2009:10:59:35 -0400] conn=283289 fd=80 slot=80 connection from
> > 1.2.3.4 to 4.3.2.1
> >
> >
> >
> > But even after indexing "uid userPassword uidNumber gidNumber cn
> > homeDirectory loginShell gecos description objectClass"  finger still
> > responds slower then it does on comparison to the older openLDAP
> > servers... where we don't do indexing on all of these attributes, AND
> > still claims that I'm running an search that hasn't been indexed!   I'm
> > I missing something glaringly obvious?   
>  You only need to index the attributes used for searching:
>  (&(objectClass=posixAccount)(uid=foo))
>  You need an equality index on objectClass (which should already be
>  there, it is one of the default indexes) and an equality index on uid
>  (again, should already be there).
>  
Ok, thats cool, those are both being indexed correctly....

>  The problem is this:
>  [26/Jun/2009:10:59:36 -0400] conn=283289 op=2 RESULT err=11 tag=101
>  nentries=0 etime=1 notes=U
>  [26/Jun/2009:10:59:35 -0400] conn=283289 op=2 SRCH
>  base="ou=really,ou=long,o=name,dc=school,dc=edu" scope=2
>  filter="(objectClass=posixAccount)" attrs="uid userPassword uidNumber
>  gidNumber cn homeDirectory loginShell gecos description objectClass"
>
>  The notes=U and err=11 mean that either the lookthrough limit has been
>  exceeded, or you need to increase your nsslapd-idlistscanlimit:
>
> http://www.redhat.com/docs/manuals/dir-server/8.1/cli/Configuration_Comma...
>
>
>  This is not a good search anyway - the client is basically asking for
>  all entries that match objectClass=posixAccount which could be
>  thousands or more - what does the client intend to do with all of
>  those entries?
>  
Thats apparently the search that "finger foo" on RHEL 5.2 generates! 
"finger foo" on Ubunutu 8.04 responds notibly faster, so I'm assuming
that it's generating a different search...  hmmm

[Nalin Dahyabhai]

On Fri, Jun 26, 2009 at 02:14:06PM -0600, Rich Megginson wrote:
>  This is not a good search anyway - the client is basically asking for
>  
>  all entries that match objectClass=posixAccount which could be thousands  
>  or more - what does the client intend to do with all of those entries? 
The finger command also tries to match your query against the first or
last name of a user, as stored in the GECOS field of user entries.  On
Fedora and derived systems (including RHEL), this part of its function
can be turned off with the -m switch.

HTH,

Nalin