[389-users] Master caught in infinite loop

[Rich Megginson]

On 11/18/2011 05:08 AM, Daniel Fenert wrote:
> Hi,
>
> I'm using 389ds 1.2.5 with replication, my current setup:
>
> Master
> |     \
> L1     L2
> | \    |  \
> S1 S2 S3  S4
>
> L* - acting as slave to "master" and master to "S*"
> S* - slaves to L*
>
>
>  From time to time (usually few months between problems) we encounter
> "master" going to some infinite loop.
> After analyzing access log, it looks like it stops doing queries, and
> accepts new connections until it runs out of fd's.
> After that, it won't stop peacefully, only SIGKILL saves the day.
>
> Workload:
> Master is used only for updates, maybe 20 connections/s.
> L* are used only for replication.
> All bind's and search queries are targeted to S* which are read only.
>
> With previous setup (less complicated), we've also seen this problem:
> Master
> |  |  |  \
> S1 S2 S3  S4...
>
> Is there a chance that upgrading to latest version will fix the problem?
> Were there any fixes nearby? Upgrade will be complex as hell ;)
>
> Error log from last problem:
>   - Not listening for new connections - too many fds open
Have you tried increasing the number of fds to 8192?
>   - slapd shutting down - signaling operation threads
>   - slapd shutting down - waiting for 120 threads to terminate
Does the server shutdown on its own, or did you shut it down normally 
(i.e. service dirsrv stop)?
> ... SIGKILL ...
>   - 389-Directory/1.2.5 B2010.012.2034 starting up
>   - Detected Disorderly Shutdown last time Directory Server was running,
> recovering database.
>   - slapd started.  Listening on All Interfaces port 389 for LDAP requests
>
> Number of fds: 4096.
Since 1.2.5 we have fixed a number of bugs around connection handling.  
You might find that 1.2.9.9 (current stable version) works much better 
for you.