[Fedora-directory-users] LDAP Accounts for large website

[Jared B. Griffith]

I was wondering if anyone here has ever used LDAP for a website, that will potentially
have millions of LDAP accounts. 
If so, are you experiencing slow query responses or other issues? 
If you were experiencing slow query responses, and were able to rectify the issue, how did
you do this? 
We are currently using FDS for our main website for customer accounts. We currently have
over 52,000 accounts in LDAP and have only been using this for 3 months. We are now
experiencing extreme slow down in query response when getting customer data into and out
of the LDAP servers. 
Any help would be greatly appreciated. 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542

[Chris G. Sellers]

Performance is often impacted greatly by

1) Memory on the LDAP server.  Make sure you can store as much of your  
directory data store in RAM for fast access
2) Indexing.  Make sure attributes that you search on freqently are  
indexed.   Also, limit what fields you search on to avoid having a  
heavy indexing tax.
3) Make sure your network connections are stable, and your not  
connecting on a 100MB half duplex connection while your network  
equiptment is expecting a full duplex connection.

Once you have auditing those situations, please check your performance  
again.

Sellers

50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core  
2 system with 1GB-2GB of RAM should perform okay.


On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote:

>  I was wondering if anyone here has ever used LDAP for a website,  
>  that will potentially have millions of LDAP accounts.
>  If so, are you experiencing slow query responses or other issues?
>  If you were experiencing slow query responses, and were able to  
>  rectify the issue, how did you do this?
>  We are currently using FDS for our main website for customer  
>  accounts.  We currently have over 52,000 accounts in LDAP and have  
>  only been using this for 3 months.  We are now experiencing extreme  
>  slow down in query response when getting customer data into and out  
>  of the LDAP servers.
>  Any help would be greatly appreciated.
>
>  -- 
>  - Thank you,
>  - Jared B. Griffith
>  - Farheap Solutions, Inc.
>  - Lead Systems Administrator
>  - California IT Department
>  - Email - jared.griffith(a)farheap.com
>  - Phone - 949.417.1500 ext. 266
>  - Cell Phone - 949.910.6542
>  --
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users 
______________________________________________
Chris G. Sellers			|	NITLE Technology
734.661.2318			|	chris.sellers(a)nitle.org
AIM: imthewherd			|	GTalk: cgseller(a)gmail.com

[Jared B. Griffith]

Here are the specs on the server: 
2 x 2.0Ghz Intel Dual Core Xeon 
4 x 1Gb Registered ECC RAM 
2 x 74Gb Western Digital Raptors 
Given that, the hardware issue should be more than sufficient. 
Network connections are Gigabit full duplex throughout our cage network. 

How would we go about indexing the attributes? 

Me and another sys admin have a distinct feeling that it is an issue with the query, but
they are going to point blame at us, so we want to make sure that we are golden before
saying it's the code. 







----- Original Message ----- 
From: "Chris G. Sellers" <chris.sellers(a)nitle.org&gt; 
To: "Jared B. Griffith" <jared.griffith(a)farheap.com&gt;, "General
discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt; 
Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/Los_Angeles 
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website 


Performance is often impacted greatly by 


1) Memory on the LDAP server. Make sure you can store as much of your directory data store
in RAM for fast access 
2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit
what fields you search on to avoid having a heavy indexing tax. 
3) Make sure your network connections are stable, and your not connecting on a 100MB half
duplex connection while your network equiptment is expecting a full duplex connection. 


Once you have auditing those situations, please check your performance again. 


Sellers 


50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with
1GB-2GB of RAM should perform okay. 






On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: 



I was wondering if anyone here has ever used LDAP for a website, that will potentially
have millions of LDAP accounts. 
If so, are you experiencing slow query responses or other issues? 
If you were experiencing slow query responses, and were able to rectify the issue, how did
you do this? 
We are currently using FDS for our main website for customer accounts. We currently have
over 52,000 accounts in LDAP and have only been using this for 3 months. We are now
experiencing extreme slow down in query response when getting customer data into and out
of the LDAP servers. 
Any help would be greatly appreciated. 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542 
-- 
Fedora-directory-users mailing list 
Fedora-directory-users(a)redhat.com 
https://www.redhat.com/mailman/listinfo/fedora-directory-users 




______________________________________________ 
Chris G. Sellers | NITLE Technology 
734.661.2318 | chris.sellers(a)nitle.org 
AIM: imthewherd | GTalk: cgseller(a)gmail.com 


-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542

[Chris G. Sellers]

if your queries are something like

ldapsearch '(&(uid=%)(cn=%*)(objectClass=account))' then you would  
want to make sure uid, cn are indexed.  maybe even objectClass.  It  
depends on what you are searching those are the attributes that you  
index.

Read up in Indexing.  It's very much like a database, you have to  
index the attributes you search often.
http://www.redhat.com/docs/manuals/dir-server/ag/7.1/index1.html#996824

Sellers

On Dec 13, 2007, at 3:04 PM, Jared B. Griffith wrote:

>  Here are the specs on the server:
>  2 x 2.0Ghz Intel Dual Core Xeon
>  4 x 1Gb Registered ECC RAM
>  2 x 74Gb Western Digital Raptors
>  Given that, the hardware issue should be more than sufficient.
>  Network connections are Gigabit full duplex throughout our cage  
>  network.
>
>  How would we go about indexing the attributes?
>
>  Me and another sys admin have a distinct feeling that it is an issue  
>  with the query, but they are going to point blame at us, so we want  
>  to make sure that we are golden before saying it's the code.
>
>
>
>
>
>
>
>  ----- Original Message -----
>  From: "Chris G. Sellers" <chris.sellers(a)nitle.org&gt;
>  To: "Jared B. Griffith" <jared.griffith(a)farheap.com&gt;, "General  
>  discussion list for the Fedora Directory server project."
> <fedora-directory-users(a)redhat.com 
>  >
>  Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/ 
>  Los_Angeles
>  Subject: Re: [Fedora-directory-users] LDAP Accounts for large website
>
>  Performance is often impacted greatly by
>
>  1) Memory on the LDAP server.  Make sure you can store as much of  
>  your directory data store in RAM for fast access
>  2) Indexing.  Make sure attributes that you search on freqently are  
>  indexed.   Also, limit what fields you search on to avoid having a  
>  heavy indexing tax.
>  3) Make sure your network connections are stable, and your not  
>  connecting on a 100MB half duplex connection while your network  
>  equiptment is expecting a full duplex connection.
>
>  Once you have auditing those situations, please check your  
>  performance again.
>
>  Sellers
>
>  50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz  
>  Core 2 system with 1GB-2GB of RAM should perform okay.
>
>
>  On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote:
>
>  I was wondering if anyone here has ever used LDAP for a website,  
>  that will potentially have millions of LDAP accounts.
>  If so, are you experiencing slow query responses or other issues?
>  If you were experiencing slow query responses, and were able to  
>  rectify the issue, how did you do this?
>  We are currently using FDS for our main website for customer  
>  accounts.  We currently have over 52,000 accounts in LDAP and have  
>  only been using this for 3 months.  We are now experiencing extreme  
>  slow down in query response when getting customer data into and out  
>  of the LDAP servers.
>  Any help would be greatly appreciated.
>
>  -- 
>  - Thank you,
>  - Jared B. Griffith
>  - Farheap Solutions, Inc.
>  - Lead Systems Administrator
>  - California IT Department
>  - Email - jared.griffith(a)farheap.com
>  - Phone - 949.417.1500 ext. 266
>  - Cell Phone - 949.910.6542
>  --
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>  ______________________________________________
>  Chris G. Sellers			|	NITLE Technology
>  734.661.2318			|	chris.sellers(a)nitle.org
>  AIM: imthewherd			|	GTalk: cgseller(a)gmail.com
>
>
>
>  -- 
>  - Thank you,
>  - Jared B. Griffith
>  - Farheap Solutions, Inc.
>  - Lead Systems Administrator
>  - California IT Department
>  - Email - jared.griffith(a)farheap.com
>  - Phone - 949.417.1500 ext. 266
>  - Cell Phone - 949.910.6542 
______________________________________________
Chris G. Sellers			|	NITLE Technology
734.661.2318			|	chris.sellers(a)nitle.org
AIM: imthewherd			|	GTalk: cgseller(a)gmail.com

[Steven Jones]

Operating system?

 

CPU and Ram look good....

 

Iostat?

 

Hi,

 

Where is your basic fault finding data? 

 

We run a database backend for spam control and find that cleaning up /
indexing the database has dramatic effects....like after 6 months our
Dell 6850 is screwed....clean it up and its disk i/o is 2% again....but
this shows up under iostat....using mrtg you can see the graph climbing
steadily over the months.....

 

A pair of SATA disks (I assume raid 1) is not very fast (on board
raid?...shudder...)....also these are SATA, SATA sucks for random i/o
and guess what you have a database doing random i/o........

 

In terms of code, yes it is often the code at fault. Somehow developers
who have written sucky code expect sys admins to spend serious time and
money on hardware compensating for their bad code....it does not work.

 

This could be the hard thing to prove....ie....is your disk i/o
inadequate or is their code so bad its causing the i/o!

 

If its disk i/o......

 

Generally, you are going to be spending most of your time reading from
disk....so you need to be optimising for reads....raid5 is ideal but
testing will prove this....LDAP can be distributed over disk sets...so
would 4 disks in two raid1s out perform a R5 3+1?  From my experience a
r5 3+1 for databases is 20% faster than raid 1s....

 

I suspect you are very budget conscious and have no-name white
boxes....so articles like this,

 

http://www.tomshardware.com/2007/11/30/more_serial_raid_controllers_from
_amcc/

 

Can give you good pointers as what to get....generally avoid SATA, look
at SAS....for databases the LSIMegaraid SAS 8888ELP in a raid 5 looks
worth buying....maybe a very small disk strip is in order so small raid5
sets....

 

You are ahead of me at present I'm still piloting FDS....

 

Regards

 

Steven Jones
Senior  Linux/Unix/San/Vmware System Administrator
APG -Technology Integration Team
Victoria University of Wellington
Phone: [phone]

________________________________

From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B.
Griffith
Sent: Friday, 14 December 2007 9:05 a.m.
To: Chris G. Sellers
Cc: General discussion list for the Fedora Directory server project.
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website

 


Here are the specs on the server:
2 x 2.0Ghz Intel Dual Core Xeon 
4 x 1Gb Registered ECC RAM
2 x 74Gb Western Digital Raptors
Given that, the hardware issue should be more than sufficient.
Network connections are Gigabit full duplex throughout our cage network.

How would we go about indexing the attributes?

Me and another sys admin have a distinct feeling that it is an issue
with the query, but they are going to point blame at us, so we want to
make sure that we are golden before saying it's the code.







----- Original Message -----
From: "Chris G. Sellers" <chris.sellers(a)nitle.org&gt;
To: "Jared B. Griffith" <jared.griffith(a)farheap.com&gt;, "General
discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt;
Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800)
America/Los_Angeles
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website

Performance is often impacted greatly by

 

1) Memory on the LDAP server.  Make sure you can store as much of your
directory data store in RAM for fast access

2) Indexing.  Make sure attributes that you search on freqently are
indexed.   Also, limit what fields you search on to avoid having a heavy
indexing tax.

3) Make sure your network connections are stable, and your not
connecting on a 100MB half duplex connection while your network
equiptment is expecting a full duplex connection.

 

Once you have auditing those situations, please check your performance
again.

 

Sellers

 

50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2
system with 1GB-2GB of RAM should perform okay.  

 

 

On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote:

	 

	I was wondering if anyone here has ever used LDAP for a website,
that will potentially have millions of LDAP accounts.
	If so, are you experiencing slow query responses or other
issues?
	If you were experiencing slow query responses, and were able to
rectify the issue, how did you do this?
	We are currently using FDS for our main website for customer
accounts.  We currently have over 52,000 accounts in LDAP and have only
been using this for 3 months.  We are now experiencing extreme slow down
in query response when getting customer data into and out of the LDAP
servers.
	Any help would be greatly appreciated.
	
	-- 
	- Thank you,
	- Jared B. Griffith
	- Farheap Solutions, Inc.
	- Lead Systems Administrator
	- California IT Department
	- Email - jared.griffith(a)farheap.com
	- Phone - 949.417.1500 ext. 266
	- Cell Phone - 949.910.6542
	--
	Fedora-directory-users mailing list
	Fedora-directory-users(a)redhat.com
	https://www.redhat.com/mailman/listinfo/fedora-directory-users

 

______________________________________________
Chris G. Sellers                                    |
NITLE Technology

734.661.2318                                       |
chris.sellers(a)nitle.org

AIM: imthewherd                                 |               GTalk:
cgseller(a)gmail.com

 



-- 
- Thank you,
- Jared B. Griffith
- Farheap Solutions, Inc.
- Lead Systems Administrator
- California IT Department
- Email - jared.griffith(a)farheap.com
- Phone - 949.417.1500 ext. 266
- Cell Phone - 949.910.6542

[Jared B. Griffith]

Centos 4 - would have been Gentoo 2007 if I could have gotten Directory Server working
correctly on it prior to having to have it up and running. 
Iostat - 2 % 
Raid 1 on the drives, SATA 3 drives. 


----- Original Message ----- 
From: "Steven Jones" <Steven.Jones(a)vuw.ac.nz&gt; 
To: "General discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt; 
Sent: Thursday, December 13, 2007 12:31:11 PM (GMT-0800) America/Los_Angeles 
Subject: RE: [Fedora-directory-users] LDAP Accounts for large website 





Operating system? 






CPU and Ram look good…. 



Iostat? 



Hi, 



Where is your basic fault finding data? 



We run a database backend for spam control and find that cleaning up / indexing the
database has dramatic effects….like after 6 months our Dell 6850 is screwed….clean it up
and its disk i/o is 2% again….but this shows up under iostat….using mrtg you can see the
graph climbing steadily over the months….. 



A pair of SATA disks (I assume raid 1) is not very fast (on board raid?...shudder…)….also
these are SATA, SATA sucks for random i/o and guess what you have a database doing random
i/o…….. 



In terms of code, yes it is often the code at fault. Somehow developers who have written
sucky code expect sys admins to spend serious time and money on hardware compensating for
their bad code….it does not work. 



This could be the hard thing to prove….ie….is your disk i/o inadequate or is their code so
bad its causing the i/o! 



If its disk i/o…… 



Generally, you are going to be spending most of your time reading from disk….so you need
to be optimising for reads….raid5 is ideal but testing will prove this….LDAP can be
distributed over disk sets…so would 4 disks in two raid1s out perform a R5 3+1? From my
experience a r5 3+1 for databases is 20% faster than raid 1s…. 



I suspect you are very budget conscious and have no-name white boxes….so articles like
this, 



http://www.tomshardware.com/2007/11/30/more_serial_raid_controllers_from_... 



Can give you good pointers as what to get….generally avoid SATA, look at SAS….for
databases the LSIMegaraid SAS 8888ELP in a raid 5 looks worth buying….maybe a very small
disk strip is in order so small raid5 sets…. 



You are ahead of me at present I’m still piloting FDS…. 



Regards 




Steven Jones 
Senior Linux/Unix/San/Vmware System Administrator 
APG -Technology Integration Team 
Victoria University of Wellington 
Phone: [phone] 




From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B. Griffith 
Sent: Friday, 14 December 2007 9:05 a.m. 
To: Chris G. Sellers 
Cc: General discussion list for the Fedora Directory server project. 
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website 




Here are the specs on the server: 
2 x 2.0Ghz Intel Dual Core Xeon 
4 x 1Gb Registered ECC RAM 
2 x 74Gb Western Digital Raptors 
Given that, the hardware issue should be more than sufficient. 
Network connections are Gigabit full duplex throughout our cage network. 

How would we go about indexing the attributes? 

Me and another sys admin have a distinct feeling that it is an issue with the query, but
they are going to point blame at us, so we want to make sure that we are golden before
saying it's the code. 







----- Original Message ----- 
From: "Chris G. Sellers" <chris.sellers(a)nitle.org&gt; 
To: "Jared B. Griffith" <jared.griffith(a)farheap.com&gt;, "General
discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt; 
Sent: Thursday, December 13, 2007 11:47:35 AM (GMT-0800) America/Los_Angeles 
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website 


Performance is often impacted greatly by 





1) Memory on the LDAP server. Make sure you can store as much of your directory data store
in RAM for fast access 


2) Indexing. Make sure attributes that you search on freqently are indexed. Also, limit
what fields you search on to avoid having a heavy indexing tax. 


3) Make sure your network connections are stable, and your not connecting on a 100MB half
duplex connection while your network equiptment is expecting a full duplex connection. 





Once you have auditing those situations, please check your performance again. 





Sellers 





50k accounts is not that much, and a 2GHz Pentium Class or 1.5GHz Core 2 system with
1GB-2GB of RAM should perform okay. 










On Dec 13, 2007, at 2:36 PM, Jared B. Griffith wrote: 






I was wondering if anyone here has ever used LDAP for a website, that will potentially
have millions of LDAP accounts. 
If so, are you experiencing slow query responses or other issues? 
If you were experiencing slow query responses, and were able to rectify the issue, how did
you do this? 
We are currently using FDS for our main website for customer accounts. We currently have
over 52,000 accounts in LDAP and have only been using this for 3 months. We are now
experiencing extreme slow down in query response when getting customer data into and out
of the LDAP servers. 
Any help would be greatly appreciated. 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542 
-- 
Fedora-directory-users mailing list 
Fedora-directory-users(a)redhat.com 
https://www.redhat.com/mailman/listinfo/fedora-directory-users 






______________________________________________ 
Chris G. Sellers | NITLE Technology 


734.661.2318 | chris.sellers(a)nitle.org 


AIM: imthewherd | GTalk: cgseller(a)gmail.com 





-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542

[Steven Jones]

Hi,

 

For applications there are lots of ways to improve performance....with
no information supplied there is no where to start...

 

Ie, look at your disk i/o...use iostat are the disks LDAP sits on at
100% utilisation? Even 80% is not good....if so make bigger raid sets
and/or distribute the database over differing raid controllers and disk
sets....get the utilisation down to 30%....

 

Is you memory full and you are into swapping? Make sure the issue is not
disk i/o bottlenecks....if not add more ram.....make sure your swap is
not exhausted....add more swap until you can get more ram...

 

Are your switches or routers too slow? Check their utilisation.....

 

CPU maxed out? Where is it going? On wait? Check disk i/o....consider
dual Quad core machines.....or even 4 way quad core machines....Dell
R900s are seriously grunty boxes, have 16 cores and hold 64gig of ram
cheaply, then attach them to a SAN....

 

regards

 

Steven Jones
Senior  Linux/Unix/San/Vmware System Administrator
APG -Technology Integration Team
Victoria University of Wellington
Phone: [phone]

________________________________

From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B.
Griffith
Sent: Friday, 14 December 2007 8:37 a.m.
To: fedora-directory-users
Subject: [Fedora-directory-users] LDAP Accounts for large website

 

I was wondering if anyone here has ever used LDAP for a website, that
will potentially have millions of LDAP accounts.
If so, are you experiencing slow query responses or other issues?
If you were experiencing slow query responses, and were able to rectify
the issue, how did you do this?
We are currently using FDS for our main website for customer accounts.
We currently have over 52,000 accounts in LDAP and have only been using
this for 3 months.  We are now experiencing extreme slow down in query
response when getting customer data into and out of the LDAP servers.
Any help would be greatly appreciated.

-- 
- Thank you,
- Jared B. Griffith
- Farheap Solutions, Inc.
- Lead Systems Administrator
- California IT Department
- Email - jared.griffith(a)farheap.com
- Phone - 949.417.1500 ext. 266
- Cell Phone - 949.910.6542

[Jared B. Griffith]

Resources are fine, and if we really need a 16 way machine this early in the game, then we
should just forgoe all of this and stick it in our regular db that sits on a 16way with
32g of mem and a fiber channel san attached to it. 
iostat is fine, 2% usage 
CPU is at 0, RAM is not even being used, so of course there is no swap going on. 

I think we need to do some of this indexing on particular attributes, then address the
possible query issue that we think is happening. 





----- Original Message ----- 
From: "Steven Jones" <Steven.Jones(a)vuw.ac.nz&gt; 
To: "General discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt; 
Sent: Thursday, December 13, 2007 12:04:36 PM (GMT-0800) America/Los_Angeles 
Subject: RE: [Fedora-directory-users] LDAP Accounts for large website 





Hi, 



For applications there are lots of ways to improve performance….with no information
supplied there is no where to start… 



Ie, look at your disk i/o…use iostat are the disks LDAP sits on at 100% utilisation? Even
80% is not good….if so make bigger raid sets and/or distribute the database over differing
raid controllers and disk sets….get the utilisation down to 30%.... 



Is you memory full and you are into swapping? Make sure the issue is not disk i/o
bottlenecks….if not add more ram…..make sure your swap is not exhausted….add more swap
until you can get more ram… 



Are your switches or routers too slow? Check their utilisation….. 



CPU maxed out? Where is it going? On wait? Check disk i/o….consider dual Quad core
machines…..or even 4 way quad core machines….Dell R900s are seriously grunty boxes, have
16 cores and hold 64gig of ram cheaply, then attach them to a SAN…. 



regards 




Steven Jones 
Senior Linux/Unix/San/Vmware System Administrator 
APG -Technology Integration Team 
Victoria University of Wellington 
Phone: [phone] 




From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B. Griffith 
Sent: Friday, 14 December 2007 8:37 a.m. 
To: fedora-directory-users 
Subject: [Fedora-directory-users] LDAP Accounts for large website 



I was wondering if anyone here has ever used LDAP for a website, that will potentially
have millions of LDAP accounts. 
If so, are you experiencing slow query responses or other issues? 
If you were experiencing slow query responses, and were able to rectify the issue, how did
you do this? 
We are currently using FDS for our main website for customer accounts. We currently have
over 52,000 accounts in LDAP and have only been using this for 3 months. We are now
experiencing extreme slow down in query response when getting customer data into and out
of the LDAP servers. 
Any help would be greatly appreciated. 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542 

-- 
- Thank you, 
- Jared B. Griffith 
- Farheap Solutions, Inc. 
- Lead Systems Administrator 
- California IT Department 
- Email - jared.griffith(a)farheap.com 
- Phone - 949.417.1500 ext. 266 
- Cell Phone - 949.910.6542

[Steven Jones]

Hi,

 

Sounds like the box is asleep....if the i/o is 2% this suggests nothing
is getting to the box.....not being a LDAP expert, but if that was a
"straight" database like Oracle or MYSQL I'd think that the problem was
external to the LDAP server....if the indexing was bad, I'd expect high
disk i/o as a result/indicator....

 

Be interesting to see if indexing does a thing....

 

regards

 

Steven Jones
Senior  Linux/Unix/San/Vmware System Administrator
APG -Technology Integration Team
Victoria University of Wellington
Phone: [phone]

________________________________

From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B.
Griffith
Sent: Friday, 14 December 2007 9:15 a.m.
To: General discussion list for the Fedora Directory server project.
Subject: Re: [Fedora-directory-users] LDAP Accounts for large website

 

Resources are fine, and if we really need a 16 way machine this early in
the game, then we should just forgoe all of this and stick it in our
regular db that sits on a 16way with 32g of mem and a fiber channel san
attached to it.
iostat is fine, 2% usage
CPU is at 0, RAM is not even being used, so of course there is no swap
going on.

I think we need to do some of this indexing on particular attributes,
then address the possible query issue that we think is happening.





----- Original Message -----
From: "Steven Jones" <Steven.Jones(a)vuw.ac.nz&gt;
To: "General discussion list for the Fedora Directory server project."
<fedora-directory-users(a)redhat.com&gt;
Sent: Thursday, December 13, 2007 12:04:36 PM (GMT-0800)
America/Los_Angeles
Subject: RE: [Fedora-directory-users] LDAP Accounts for large website

Hi,

 

For applications there are lots of ways to improve performance....with
no information supplied there is no where to start...

 

Ie, look at your disk i/o...use iostat are the disks LDAP sits on at
100% utilisation? Even 80% is not good....if so make bigger raid sets
and/or distribute the database over differing raid controllers and disk
sets....get the utilisation down to 30%....

 

Is you memory full and you are into swapping? Make sure the issue is not
disk i/o bottlenecks....if not add more ram.....make sure your swap is
not exhausted....add more swap until you can get more ram...

 

Are your switches or routers too slow? Check their utilisation.....

 

CPU maxed out? Where is it going? On wait? Check disk i/o....consider
dual Quad core machines.....or even 4 way quad core machines....Dell
R900s are seriously grunty boxes, have 16 cores and hold 64gig of ram
cheaply, then attach them to a SAN....

 

regards

 

Steven Jones
Senior  Linux/Unix/San/Vmware System Administrator
APG -Technology Integration Team
Victoria University of Wellington
Phone: [phone]

________________________________

From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Jared B.
Griffith
Sent: Friday, 14 December 2007 8:37 a.m.
To: fedora-directory-users
Subject: [Fedora-directory-users] LDAP Accounts for large website

 

I was wondering if anyone here has ever used LDAP for a website, that
will potentially have millions of LDAP accounts.
If so, are you experiencing slow query responses or other issues?
If you were experiencing slow query responses, and were able to rectify
the issue, how did you do this?
We are currently using FDS for our main website for customer accounts.
We currently have over 52,000 accounts in LDAP and have only been using
this for 3 months.  We are now experiencing extreme slow down in query
response when getting customer data into and out of the LDAP servers.
Any help would be greatly appreciated.

-- 
- Thank you,
- Jared B. Griffith
- Farheap Solutions, Inc.
- Lead Systems Administrator
- California IT Department
- Email - jared.griffith(a)farheap.com
- Phone - 949.417.1500 ext. 266
- Cell Phone - 949.910.6542



-- 
- Thank you,
- Jared B. Griffith
- Farheap Solutions, Inc.
- Lead Systems Administrator
- California IT Department
- Email - jared.griffith(a)farheap.com
- Phone - 949.417.1500 ext. 266
- Cell Phone - 949.910.6542

[Satish Chetty]

Jared,
	I would check to see if the problem if it is on the LDAP or  website. 
Ex. LDAP response may be good from the ldap host using ldapsearch. 
However the LDAP response by the ldap plugin/api from the 
webserver/appserver may not.

Some additional steps I would do...
* Run the ldap query using ldapsearch on the ldap host and then machine 
on same subnet and then from webserver/appserver machine. Both with and 
without website load. This will help me identify if the problem is on 
the network or LDAP server itself or load.

* Check on indexing, number of concurrent requests when server is fully 
loaded, FDs, swap space, TCP/IP settings etc. The load may be high 
enough that you need additional replicas or even masters. With 
additional masters you need to be watchful on how load balancing works 
when writing to different masters.

-Satish.

Jared B. Griffith wrote:
>  I was wondering if anyone here has ever used LDAP for a website, that
>
>  will potentially have millions of LDAP accounts.
>  If so, are you experiencing slow query responses or other issues?
>  If you were experiencing slow query responses, and were able to rectify 
>  the issue, how did you do this?
>  We are currently using FDS for our main website for customer accounts.  
>  We currently have over 52,000 accounts in LDAP and have only been using 
>  this for 3 months.  We are now experiencing extreme slow down in query 
>  response when getting customer data into and out of the LDAP servers.
>  Any help would be greatly appreciated.
>  
>  -- 
>  - Thank you,
>  - Jared B. Griffith
>  - Farheap Solutions, Inc.
>  - Lead Systems Administrator
>  - California IT Department
>  - Email - jared.griffith(a)farheap.com
>  - Phone - 949.417.1500 ext. 266
>  - Cell Phone - 949.910.6542
>  
>  
>  ------------------------------------------------------------------------
>  
>  --
>  Fedora-directory-users mailing list
>  Fedora-directory-users(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-directory-users