Need help to tune 389 DS

[wodel youchi]

Hi,

We have a simple installation of 389DS, a master/slave installation.

Our directory server is mainly used with our mail and portal servers for
now.

We did create some custom attributes for our needs.

Here is a simple entry model :































*uid=lastname.firstname,ou=People,dc=domain;dc=tlddn:
uid=lastname.firstname,ou=People,dc=domain,dc=tldou: SUPPORT
TEAMrecruitmentDate: 20030510mailQuota: 2048576000
<%28204%29%20857-6000>teletexTerminalIdentifier: 3053maidenName:
MaindeNameemployeeNumber: 28813gender: FdepartmentNumber: 1121employeeType:
SmailAlternateAddress: f.lastname(a)domain.tlddisplayName: Lastname
Firstnamemail: lastname.firstname(a)domain.tldjobTitle: Supervisorsn:
Firstnamecn: LastnameobjectClass: topobjectClass: personobjectClass:
organizationalPersonobjectClass: inetorgpersonobjectClass:
shadowAccountobjectClass: mailrecipientobjectClass:
customprofilemailMessageStore: /var/vmail/lastname.firstnameuid:
lastname.firstnamemailHost: mail.domain.tldtitle: MuserPassword::
e1NIQX02TjBCaXpjcWRkNVJCUXlyVHV6TDlwT1NGK3c9*

*webtelxmail : f.lastname(a)telex.domain.tld*


We run logconv against our directory Server and we get a pretty long result
file, with some recommendations, but we don't know how proceed, especially
with indexes.

we have many entries like these












*  Unindexed Search #32105 (notes=A)  -  Date/Time:
13/Jan/2017:01:21:49  -  Connection Number:     434118  -  Operation
Number:      74292  -  Etime:                 0  -  Nentries:
0  -  IP Address:            172.16.16.9  -  Search Base:
ou=people,dc=domain,dc=tld  -  Search Scope:          2 (subtree)  -
Search Filter:
(&(null=uid=lastname.lastname,ou=people,dc=domain,dc=tld))  -  Bind
DN:               cn=directory manager*

*Unindexed Search #76240 (notes=A)  -  Date/Time:             *





*13/Jan/2017:00:16:55  -  Connection Number:     433231  -  Operation
Number:      205936  -  Etime:                 0  -  Nentries:
0  -  IP Address:            Unknown_Host  -  Search Base:
ou=people,*

*dc=domain,dc=tld    -  Search Scope:          2 (subtree)  -  Search
Filter:         (&(null=uid=lastname2.firstname2,ou=people,*


*dc=domain,dc=tld))Unindexed Component #1 (notes=U)  -
Date/Time:             *





*13/Jan/2017:00:01:19  -  Connection Number:     433951  -  Operation
Number:      1  -  Etime:                 0  -  Nentries:              0
-  IP Address:            172.16.16.1  -  Search Base:           ou=people,*


*dc=domain,dc=tld  -  Search Scope:          2 (subtree)  -  Search
Filter:         (&(objectclass=*)(uid=pat))  -  Bind DN:
uid=dovecot,**dc=domain,dc=tld*


In the end of the result file we got this :

In the end of the result file we got this :

FDs Taken:                    573
FDs Returned:                 569
Highest FD Taken:             81

Broken Pipes:                 0
Connections Reset By Peer:    0
Resource Unavailable:         235
     -  235  (T1) Idle Timeout Exceeded
Max BER Size Exceeded:        0

Binds:                        4627
Unbinds:                      45
 - LDAP v2 Binds:             24
 - LDAP v3 Binds:             4603
 - AUTOBINDs:                 0
 - SSL Client Binds:          0
 - Failed SSL Client Binds:   0
 - SASL Binds:                0
 - Directory Manager Binds:   4
 - Anonymous Binds:           24
 - Other Binds:               4599



----- Connection Latency Details -----

 (in seconds)        <=1    2    3    4-5    6-10    11-15    >15
 --------------------------------------------------------------------------
 (# of connections)    250    8    3    5    55    4    244

----- Current Open Connection IDs -----

Conn Number:  434505 (172.16.16.8)
Conn Number:  434171 (172.16.16.9)
Conn Number:  434506 (172.16.16.8)
Conn Number:  434301 (172.16.16.9)
Conn Number:  434150 (172.16.16.9)
Conn Number:  434118 (172.16.16.9)
Conn Number:  434507 (172.16.16.8)
Conn Number:  434508 (172.16.16.8)
Conn Number:  434504 (172.16.16.8)


----- Errors -----

err=0                295657    Successful Operations
err=32                  627    No Such Object
err=49                   44    Invalid Credentials (Bad Password)
err=4                    24    Size Limit Exceeded
err=11                    6    Administrative Limit Exceeded (Look Through
Limit)

----- Top 10 Failed Logins ------

12          uid=lastname.firstname,ou=people,dc=domain,dc=tld
12          uid=admin,ou=people,dc=domain,dc=tld
8           uid=lastname10.firstname10,ou=people,dc=domain,dc=tld
4           uid=lastname11.firstname11,ou=people,dc=domain,dc=tld
3           uid=lastname12.firstname12,ou=people,dc=domain,dc=tld
3           uid=lastname13.firstname13,ou=people,dc=domain,dc=tld
2           uid=dakar,ou=people,dc=domain,dc=tld

From the IP address(s) :

32          Unknown_Host
12          172.16.16.1


----- Total Connection Codes -----

B1      289   Bad Ber Tag Encountered
T1      235   Idle Timeout Exceeded
U1       45   Cleanly Closed Connections


----- Top 10 Clients -----

Number of Clients:  4

[1] Client: 172.16.16.8
       308 - Connections
       235 - T1 (Idle Timeout Exceeded)
        68 - B1 (Bad Ber Tag Encountered)

[2] Client: 172.16.16.1
       221 - Connections
       221 - B1 (Bad Ber Tag Encountered)

[3] Client: 172.16.16.14
        24 - Connections
        24 - U1 (Cleanly Closed Connections)

[4] Client: 172.16.16.9
        20 - Connections
        16 - U1 (Cleanly Closed Connections)



----- Top 10 Bind DN's -----

Number of Unique Bind DN's: 368

529             uid=dovecot,dc=domain,dc=tld
242             uid=helpdesk-ventes,ou=people,dc=domain,dc=tld
182             uid=helpdesk,ou=people,dc=domain,dc=tld
161             uid=lastname.firstname,ou=people,dc=domain,dc=tld
122             uid=lastname2.firstname2,ou=people,dc=domain,dc=tld
121             uid=lastname3.firstname3,ou=people,dc=domain,dc=tld
120             uid=lastname4.firstname4,ou=people,dc=domain,dc=tld
119             uid=lastname5.firstname5,ou=people,dc=domain,dc=tld
110             uid=lastname6.firstname6,ou=people,dc=domain,dc=tld
110             uid=lastname7.firstname7,ou=people,dc=domain,dc=tld


----- Top 10 Search Bases -----

Number of Unique Search Bases: 7628

192079          ou=people,dc=domain,dc=tld
352             uid=postmaster,ou=people,dc=domain,dc=tld
254             uid=helpdesk-ventes,ou=people,dc=domain,dc=tld
208             uid=helpdesk,ou=people,dc=domain,dc=tld
173             uid=lastname20.firstname20,ou=people,dc=domain,dc=tld
145             uid=lastname21.firstname21,ou=people,dc=domain,dc=tld
136             uid=lastname22.firstname22,ou=people,dc=domain,dc=tld
133             uid=lastname23.firstname23,ou=people,dc=domain,dc=tld
132             uid=lastname24.firstname24,ou=people,dc=domain,dc=tld
123             uid=lastname25.firstname25,ou=people,dc=domain,dc=tld


----- Top 10 Search Filters -----

Number of Unique Search Filters: 15887

95285           (objectclass=*)
456
(&(|(mail=domain.tld)(mailalternateaddress=domain.tld))(objectclass=inetorgperson))
352             (&amp;(|(mail=postmaster(a)domain.tld
)(mailalternateaddress=postmaster(a)domain.tld))(objectclass=inetorgperson))
242             (&amp;(|(mail=helpdesk-ventes(a)domain.tld
)(mailalternateaddress=helpdesk-ventes(a)domain.tld
))(objectclass=inetorgperson))
224             (&amp;(|(mail=helpdesk(a)domain.tld
)(mailalternateaddress=helpdesk(a)domain.tld))(objectclass=inetorgperson))
161             (&amp;(|(mail=lastname22.firstname22(a)domain.tld
)(mailalternateaddress=lastname22.firstname22(a)domain.tld
))(objectclass=inetorgperson))
161             (&amp;(|(mail=lastname10.firstname10(a)domain.tld
)(mailalternateaddress=lastname10.firstname10(a)domain.tld
))(objectclass=inetorgperson))
153             (&amp;(|(mail=lastname15.firstname15(a)domain.tld
)(mailalternateaddress=lastname15.firstname15(a)domain.tld
))(objectclass=inetorgperson))



----- Top 10 Most Frequent etimes -----

286125          etime=0
10227           etime=1
6               etime=2


----- Top 10 Longest etimes -----

etime=2         6
etime=1         10227
etime=0         286125


----- Top 10 Largest nentries -----

nentries=7622                  12
nentries=13                     1
nentries=10                     1
nentries=6                      1
nentries=1                 194973
nentries=0                  96743


----- Top 10 Most returned nentries -----

194973          nentries=1
96743           nentries=0
12              nentries=7622
1               nentries=13
1               nentries=6
1               nentries=10



----- Top 10 Most Requested Attributes -----

194890      mailQuota
96898       mail
95504       uid
95267       displayName
95267       sn
95261       createTimestamp
95261       creatorsName
95261       employeeType
95261       modifyTimestamp
95261       ou


----- Abandon Request Stats -----

 - UNKNOWN conn=433250 op=NOTFOUND msgid=102278 client=Unknown_Host
 - UNKNOWN conn=433250 op=NOTFOUND msgid=102667 client=Unknown_Host
 - UNKNOWN conn=433250 op=NOTFOUND msgid=89071 client=Unknown_Host
 - UNKNOWN conn=434150 op=NOTFOUND msgid=26399 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=50332 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=54880 client=172.16.16.9
 - UNKNOWN conn=433250 op=NOTFOUND msgid=88929 client=Unknown_Host
 - UNKNOWN conn=434150 op=NOTFOUND msgid=13986 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=21519 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=35708 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=43639 client=172.16.16.9
 - UNKNOWN conn=434150 op=NOTFOUND msgid=44806 client=172.16.16.9


----- Recommendations -----

 1.  You have unindexed searches, this can be caused from a search on an
unindexed attribute, or your returned results exceeded the
allidsthreshold.  Unindexed searches are not recommended. To refuse
unindexed searches, switch 'nsslapd-require-index' to 'on' under your
database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config).

 2.  You have unindexed components, this can be caused from a search on an
unindexed attribute, or your returned results exceeded the
allidsthreshold.  Unindexed components are not recommended. To refuse
unindexed searches, switch 'nsslapd-require-index' to 'on' under your
database entry (e.g. cn=UserRoot,cn=ldbm database,cn=plugins,cn=config).

 3.  You have some connections that are are being closed by the idletimeout
setting. You may want to increase the idletimeout if it is set low.

 4.  You have a significant difference between binds and unbinds.  You may
want to investigate this difference.

 5.  You have more abnormal connection codes than cleanly closed
connections.  You may want to investigate this difference.

Cleaning up temp files...
Done.

Any advice on how to interpret all this data?
Can these problems slow users connection?
Could someone point us to how improve our directory server.

If you need more info let me know.

Thanks in advance.

[Gordon Messmer]

On 02/22/2017 03:27 AM, wodel youchi wrote:
>  we have many entries like these
>
>  *  Unindexed Search #32105 (notes=A)
>  (&(null=uid=lastname.lastname,ou=people,dc=domain,dc=tld))
>    -  Bind DN:               cn=directory manager
>  * 
You've got two problems here.  One is that you appear to be using 
"cn=directory manager" for one of your LDAP clients, and that is a very 
serious security risk.  Don't do that!

The other is that this application appears to be misconfigured.  It 
looks like it's searching for an attribute whose value will be an LDAP 
DN, but it doesn't have the name of that attribute.  It might be looking 
for a group by "member" but because it doesn't know the name of the 
attribute, it's searching for an attribute named "null".

>  *Unindexed Component #1 (notes=U)
>  **  -  Search Base:           ou=people,***dc=domain,dc=tld*
>    -  Search Filter:         (&(objectclass=*)(uid=pat))
>    -  Bind DN:               uid=dovecot,***dc=domain,dc=tld** 
Take a look at either the 389 console or 
/etc/dirsrv/slapd-<instance>/dse.ldif (where you will look for default 
indexes,cn=config,cn=ldbm database,cn=plugins,cn=config). You should see 
equality indexes (nsIndexType: eq) for both of those attributes by 
default.  Is one of them not currently indexed?

[Noriko Hosoi]

On 02/22/2017 10:30 AM, Gordon Messmer wrote:
>  On 02/22/2017 03:27 AM, wodel youchi wrote:
> >
> > we have many entries like these
> >
> > *  Unindexed Search #32105 (notes=A)
> > (&(null=uid=lastname.lastname,ou=people,dc=domain,dc=tld))
> >   -  Bind DN:               cn=directory manager
> > *
>
>  You've got two problems here.  One is that you appear to be using 
>  "cn=directory manager" for one of your LDAP clients, and that is a 
>  very serious security risk.  Don't do that!
>
>  The other is that this application appears to be misconfigured. It 
>  looks like it's searching for an attribute whose value will be an LDAP 
>  DN, but it doesn't have the name of that attribute.  It might be 
>  looking for a group by "member" but because it doesn't know the name 
>  of the attribute, it's searching for an attribute named "null".
>
> > *Unindexed Component #1 (notes=U)
> > **  -  Search Base:           ou=people,***dc=domain,dc=tld*
> >   -  Search Filter:         (&(objectclass=*)(uid=pat))
> >   -  Bind DN:               uid=dovecot,***dc=domain,dc=tld**
>
>  Take a look at either the 389 console or 
>  /etc/dirsrv/slapd-<instance>/dse.ldif (where you will look for default 
>  indexes,cn=config,cn=ldbm database,cn=plugins,cn=config). You should 
>  see equality indexes (nsIndexType: eq) for both of those attributes by 
>  default.  Is one of them not currently indexed? 
A tiny correction...
presence (pres) for objectclass, and
equality (eq) for uid

And we recommend to tune nsslapd-idlistscanlimit with the appropriate 
value to avoid unnecessary unindexed search.
https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10...
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org

[Gordon Messmer]

On 02/22/2017 12:56 PM, Noriko Hosoi wrote:
> >
> > Take a look at either the 389 console or 
> > /etc/dirsrv/slapd-<instance>/dse.ldif (where you will look for 
> > default indexes,cn=config,cn=ldbm database,cn=plugins,cn=config). You 
> > should see equality indexes (nsIndexType: eq) for both of those 
> > attributes by default.  Is one of them not currently indexed?
>  A tiny correction...
>  presence (pres) for objectclass, and
>  equality (eq) for uid  

Has that changed in some recent release?  My installation says:

dn: cn=objectclass,cn=default indexes,cn=config,cn=ldbm 
database,cn=plugins,cn
  =config
objectClass: top
objectClass: nsIndex
cn: objectclass
nsSystemIndex: true
nsIndexType: eq

[William Brown]

On Wed, 2017-02-22 at 19:35 -0800, Gordon Messmer wrote:
>  On 02/22/2017 12:56 PM, Noriko Hosoi wrote:
>  >>
>  >> Take a look at either the 389 console or 
>  >> /etc/dirsrv/slapd-<instance>/dse.ldif (where you will look for 
>  >> default indexes,cn=config,cn=ldbm database,cn=plugins,cn=config). You 
>  >> should see equality indexes (nsIndexType: eq) for both of those 
>  >> attributes by default.  Is one of them not currently indexed?
>  > A tiny correction...
>  > presence (pres) for objectclass, and
>  > equality (eq) for uid 
>  
>  
>  Has that changed in some recent release?  My installation says:
>  
>  dn: cn=objectclass,cn=default indexes,cn=config,cn=ldbm 
>  database,cn=plugins,cn
>    =config
>  objectClass: top
>  objectClass: nsIndex
>  cn: objectclass
>  nsSystemIndex: true
>  nsIndexType: eq 
Default indexes only apply to new databases (It's a template iirc). You
need to edit the index on the cn=userRoot,cn=ldbm
database,cn=plugins,cn=config

Else it won't apply.


-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Brisbane

[Gordon Messmer]

On 02/22/2017 09:25 PM, William Brown wrote:
>  Default indexes only apply to new databases (It's a template
> iirc). You
>  need to edit the index on the cn=userRoot,cn=ldbm
>  database,cn=plugins,cn=config 

Thanks for clarification, but even when I look in the correct location, 
it's "eq" as I said originally:

dn: cn=objectclass,cn=index,cn=userRoot,cn=ldbm 
database,cn=plugins,cn=config
objectClass: top
objectClass: nsIndex
cn: objectclass
nsSystemIndex: true
nsIndexType: eq

[William Brown]

On Wed, 2017-02-22 at 22:20 -0800, Gordon Messmer wrote:
>  On 02/22/2017 09:25 PM, William Brown wrote:
>  > Default indexes only apply to new databases (It's a template iirc). You
>  > need to edit the index on the cn=userRoot,cn=ldbm
>  > database,cn=plugins,cn=config
>  
>  
>  Thanks for clarification, but even when I look in the correct location, 
>  it's "eq" as I said originally:
>  
>  dn: cn=objectclass,cn=index,cn=userRoot,cn=ldbm 
>  database,cn=plugins,cn=config
>  objectClass: top
>  objectClass: nsIndex
>  cn: objectclass
>  nsSystemIndex: true
>  nsIndexType: eq
>   
As Noriko pointed you, you are missing nsIndexType: pres on this :) 

>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org 
-- 
Sincerely,

William Brown
Software Engineer
Red Hat, Brisbane

[Rich Megginson]

On 02/23/2017 01:11 AM, William Brown wrote:
>  On Wed, 2017-02-22 at 22:20 -0800, Gordon Messmer wrote:
> > On 02/22/2017 09:25 PM, William Brown wrote:
> >> Default indexes only apply to new databases (It's a template iirc). You
> >> need to edit the index on the cn=userRoot,cn=ldbm
> >> database,cn=plugins,cn=config
> >
> > Thanks for clarification, but even when I look in the correct location,
> > it's "eq" as I said originally:
> >
> > dn: cn=objectclass,cn=index,cn=userRoot,cn=ldbm
> > database,cn=plugins,cn=config
> > objectClass: top
> > objectClass: nsIndex
> > cn: objectclass
> > nsSystemIndex: true
> > nsIndexType: eq
> >
>  As Noriko pointed you, you are missing nsIndexType: pres on this :) 
Do you really want a presence index on objectclass?  What I mean is - 
every single entry in the server has an objectclass.  A search which 
includes (objectclass=*) is redundant.

> > _______________________________________________
> > 389-users mailing list -- 389-users(a)lists.fedoraproject.org
> > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>
>
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org

[Gordon Messmer]

On 02/23/2017 12:11 AM, William Brown wrote:
>  As Noriko pointed you, you are missing nsIndexType: pres on this

I hate to repeat myself, but is that a thing that changed *recently*?  I 
just checked another, newer 389-ds server, and I don't see "pres" index 
on objectclass on any servers that I manage.

[Mark Reynolds]

On 02/23/2017 10:48 AM, Gordon Messmer wrote:
>  On 02/23/2017 12:11 AM, William Brown wrote:
> > As Noriko pointed you, you are missing nsIndexType: pres on this
>
>
>  I hate to repeat myself, but is that a thing that changed *recently*?   
No, it has
always only been indexed for "eq".

As Rich said, having a "pres" index on objectclass is redundant(and
wasteful).  Every entry has objectclass - so if this was indexed for
"presence" it could actually create overhead.  It's faster to read
directly from the DB, or candidate list, than trying to use an index
that contains every entry anyway.

>  I just checked another, newer 389-ds server, and I don't see
> "pres"
>  index on objectclass on any servers that I manage.
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org