[Fedora-directory-users] Load Testing question

[Anderson, Cary]

I am still playing around with load/stress testing my FDS install.  I am using slamd to
run the tests, and I am running the basic load, and the basic search tests against my test
boxes,  The slamd client and the FDS server are on identical hardware running RHEL4 with
2G memory, and 2 Intel Xeon 3.6GHz processors.  The issue I am seeing is that I am getting
"cannot connect to the ldap server" errors once I push the "threads per
client" past 2400.  My question is am I hitting a limit on the OS (max tcp
connections)? Or a FDS limit?  My assumption was that given appropriate hardware, and
proper configuration, I could expect FDS to handle more than 2400 concurrent connections. 
Any insights as to what I might be missing would be greatly appreciated.

Thanks


FDS Config:

Performance settings are:
Time Limit: 3600
Max file descriptors: 65535

Database Link Connection Management Settings
Max TCP connections: 6
Bind Timeout: 15
Max binds per conn: 20
Timeout before Abandon: 1
Max LDAP Conn:  40
Max bind retries: 6
Max operations/conn: 2

Database Settings:
Max Cache Size:  66060288
Mem Avail for cache: 105622733

Note:  uidnumber is indexed


SLAMD test parameters:

Scheduled Start Time	  	 06/05/2007 13:10:46	 
Scheduled Stop Time	  	 (not specified)	 
Scheduled Duration	  	 300 seconds	 
Number of Clients	  	 1	 
Wait for Available Clients	  	 true	 
Monitor Clients if Available	  	 false	 
Threads per Client	  	 2400	 
Thread Startup Delay	  	 0 milliseconds	 
Statistics Collection Interval	  	 60 seconds	 
Job Dependencies	  	 (none specified)
	
Notify on Completion	  	 (none specified)	 
 	 
Parameter Information	 
Directory Server Host	  	 app2	 
Directory Server Port	  	 389	 
Bind DN	  	 (not specified)	 
Bind Password	  	 (not specified)	 
Search Base	  	 dc=calpers,dc=ca,dc=gov	 
Search Scope	  	 Whole Subtree	 
Search Filter	  	 uidnumber=50000	 
Filter File URL	  	 (not specified)	 
Attributes to Return	  	 uidnumber	 
Warm Up Time	  	 0	 
Cool Down Time	  	 0	 
Search Size Limit	  	 0	 
Search Time Limit	  	 0	 
Time Between Requests (ms)	  	 0	 
Use SSL	  	 false	 
Blindly Trust Any Certificate	  	 true	 
SSL Key Store	  	 (not specified)	 
SSL Key Store Password	  	 (not specified)	 
SSL Trust Store	  	 (not specified)	 
SSL Trust Store Password	  	 (not specified)	 
Number of Iterations	  	 -1	 
Always Disconnect	  	 false	 
Follow Referrals	  	 false	


 

Cary Anderson, Systems Software Specialist
UNIX/Linux Services
Information Technology Services Branch
Technology Services & Support Division / Data Center Section
System Software & Storage Infrastructure
fCalPERS 
Phone: [phone]  
Fax: [phone]

[Morris, Patrick]

On Wed, 06 Jun 2007, Anderson, Cary wrote:

>  I am still playing around with load/stress testing my FDS install.  I
> am using slamd to run the tests, and I am running the basic load, and the basic search
> tests against my test boxes,  The slamd client and the FDS server are on identical
> hardware running RHEL4 with 2G memory, and 2 Intel Xeon 3.6GHz processors.  The issue I am
> seeing is that I am getting "cannot connect to the ldap server" errors once I
> push the "threads per client" past 2400.  My question is am I hitting a limit on
> the OS (max tcp connections)? Or a FDS limit?  My assumption was that given appropriate
> hardware, and proper configuration, I could expect FDS to handle more than 2400 concurrent
> connections.  Any insights as to what I might be missing would be greatly appreciated.
Log entries from your server should give you some insight into why
connections are being refused.

There's a good chance you haven't allocated enough file handles, but
that's a guess.

[Anderson, Cary]

Thanks for the feedback, I have turned on more extensive logging on the
FDS server, Hopefully that will turn something up, but based on the
settings in my sysctl.conf, limits.conf, and ulimit, I should be able to
handle more than 2500 connections, unless I am missing a configuration
parameter.

Ulimit 805696

Sysctl.conf
##############################
# Set Kernel Parameters
# semaphores, SEMMSL, SEMMNS, SEMOPM, SEMMNI:
kernel.sem = 256 32000 100 142
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.msgmax = 8192
kernel.msgmnb = 65535
kernel.msgmni = 2878
fs.file-max = 131072
net.ipv4.ip_local_port_range = 32000 65000

Limits.conf

# Custom setup for Global System Limits:
*               hard    nproc           32767
*               soft    nproc           32767
*               hard    nofile          65536
*               soft    nofile          65536
*               hard    fsize         5000000
*               soft    fsize         1000000



Cary Anderson, Systems Software Specialist
UNIX/Linux Services
Information Technology Services Branch
Technology Services & Support Division / Data Center Section
System Software & Storage Infrastructure
fCalPERS 
Phone: [phone]  
Fax: [phone] 



-----Original Message-----
From: fedora-directory-users-bounces(a)redhat.com
[mailto:[e-mail]] On Behalf Of Patrick
Morris
Sent: Wednesday, June 06, 2007 12:33 PM
To: General discussion list for the Fedora Directory server project.
Subject: Re: [Fedora-directory-users] Load Testing question


On Wed, 06 Jun 2007, Anderson, Cary wrote:

>  I am still playing around with load/stress testing my FDS install.  I
>
>  am using slamd to run the tests, and I am running the basic load, and 
>  the basic search tests against my test boxes,  The slamd client and 
>  the FDS server are on identical hardware running RHEL4 with 2G memory, 
>  and 2 Intel Xeon 3.6GHz processors.  The issue I am seeing is that I
>
>  am getting "cannot connect to the ldap server" errors once I push the 
>  "threads per client" past 2400.  My question is am I hitting a limit 
>  on the OS (max tcp connections)? Or a FDS limit?  My assumption was 
>  that given appropriate hardware, and proper configuration, I could 
>  expect FDS to handle more than 2400 concurrent connections.  Any 
>  insights as to what I might be missing would be greatly appreciated. 
Log entries from your server should give you some insight into why
connections are being refused.

There's a good chance you haven't allocated enough file handles, but
that's a guess.

--
Fedora-directory-users mailing list Fedora-directory-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-directory-users

[Morris, Patrick]

On Wed, 06 Jun 2007, Anderson, Cary wrote:

>  Thanks for the feedback, I have turned on more extensive logging on
> the
>  FDS server, Hopefully that will turn something up, but based on the
>  settings in my sysctl.conf, limits.conf, and ulimit, I should be able to
>  handle more than 2500 connections, unless I am missing a configuration
>  parameter.
>  
>  Ulimit 805696
>  
>  Sysctl.conf
>  ##############################
>  # Set Kernel Parameters
>  # semaphores, SEMMSL, SEMMNS, SEMOPM, SEMMNI:
>  kernel.sem = 256 32000 100 142
>  kernel.shmall = 2097152
>  kernel.shmmax = 2147483648
>  kernel.shmmni = 4096
>  kernel.msgmax = 8192
>  kernel.msgmnb = 65535
>  kernel.msgmni = 2878
>  fs.file-max = 131072
>  net.ipv4.ip_local_port_range = 32000 65000
>  
>  Limits.conf
>  
>  # Custom setup for Global System Limits:
>  *               hard    nproc           32767
>  *               soft    nproc           32767
>  *               hard    nofile          65536
>  *               soft    nofile          65536
>  *               hard    fsize         5000000
>  *               soft    fsize         1000000 
You may be hitting a limit in the FDS configuration. Again, a quick
glance at your LDAP logs (the error log in particular) should tell you
if that's the case.