[389-users] CMP operations against pwdPolicySubentry hanging

[Iain Morgan]

On Wed, Feb 15, 2012 at 18:19:10 -0600, Rich Megginson wrote:
> On 02/15/2012 03:51 PM, Iain Morgan wrote:
> > On Wed, Feb 15, 2012 at 15:04:52 -0600, Rich Megginson wrote:
> >> On 02/15/2012 01:56 PM, Iain Morgan wrote:
> >>> On Tue, Feb 14, 2012 at 19:54:39 -0600, Rich Megginson wrote:
> >>>> On 02/14/2012 06:37 PM, Iain Morgan wrote:
> >>>>> Hello,
> >>>>>
> >>>>> On a fairly frequent basis, one of my 389 DS servers hangs after certain
> >>>>> CMP operations. Once this happens, the server cannot be shutdown
> >>>>> gracefully. This has been going on for several weeks, and I have not yet
> >>>>> found a solution.
> >>>>>
> >>>>> My setup consists of two systems running RHEL 6.2 with 389 DS 1.2.9.16.
> >>>>> Multimaster replication is enabled between the two servers, but the
> >>>>> client systems (currently just two test systems) preferrentially use the
> >>>>> same server, ServerA. The second server, ServerB, is the one which is
> >>>>> experiencing the problem.
> >>>>>
> >>>>> We are using class-of-service entries to to set the values for the
> >>>>> shadowMax, shadowMin, and shadowWarning attributes. And we are
> >>>>> conditionally setting a pwdPolicySubentry attribute for some entries in
> >>>>> the same manner.
> >>>>>
> >>>>> If I execute an ldapcompare command, such as the following:
> >>>>>
> >>>>> # ldapcompare uid=imorgan,ou=People,dc=example,dc=com \
> >>>>> 	pwdpolicysubentry:"cn=Special Policy,ou=Policies,dc=example,dc=com"
> >>>>>
> >>>>> the command will occassionally hang. Most of the time, the command
> >>>>> succeeds and indicates that the attribute is not defined for that entry.
> >>>>> However, once or twice a day it will simply hang.
> >>>>>
> >>>>> The access log shows that the CMP request was received, but no result is
> >>>>> logged. After this occurs, the server will not shut down gracefully. The
> >>>>> init script fails to shut down the server and I end up having to send a
> >>>>> SIGKILL to ns-slapd.
> >>>> When you get the hang, can you attach to the process with gdb?
> >>>> ps -ef|grep ns-slapd
> >>>> gdb /usr/sbin/ns-slapd pid-of-ns-slapd
> >>>>> The error log does not report any issues.
> >>>>>
> >>>>> CMP operations against other attributes, such as loginShell, do not seem
> >>>>> to exhibit this problem. Also, the problem does not occur on ServerA;
> >>>>> only on ServerB. Once the CMP operation has hung, comparisons against
> >>>>> other attributes, even shadowMax, continue to work.
> >>>>>
> >>>>> As noted above, most of the time the CMP operation returns normally.
> >>>>> However, if I reinitialize ServerB from ServerA, the problem occurs with
> >>>>> the first CMP operation against ServerB.
> >>>>>
> >>>>> Both servers have the same set of RPMs and the dse.ldif on both systems
> >>>>> do not have any significant differences.
> >>>>>
> >>>>> Has anyone seen a similar issue? Any suggestions on how to debug of fix
> >>>>> this?
> >>>>>
> >>>>> A somewhat simplified and redacted version of the class-of-service
> >>>>> configuration is listed below.
> >>>>>
> >>>>> Thanks
> >>> A gzip'd copy of the 'thread apply all bt full' output is attached.
> >>>
> >> Thanks.  Can you do this again after installing the
> >> 389-ds-base-debuginfo package?
> >> debuginfo-install 389-ds-base
> > Ah, sorry about that. Here's the updated output.
> >
> >> Are you using Views?
> >> http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-views.html
> > No.
> >
> Thanks!  This looks like a symptom of 
> https://fedorahosted.org/389/ticket/247 fixed in 1.2.10

Hello Rich,

Thanks, I upgraded both of the servers to 1.2.10.1. Unfortunately, it
did not resolve the issue. I also noticed that if I run the same
ldapcompare command after the first try fails, the server crashes. I
can't say whether that is a change in the behaviour, but it is a new
observation.

I've attached gdb output for the case where the first ldapcompare is
hanging. And, I've also attached the gdb analysis of the core dump.

-- 
Iain Morgan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 389-hang.txt.gz
Type: application/x-gunzip
Size: 8857 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/389-users/attachments/20120223/34637f30/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 389-coredump.txt.gz
Type: application/x-gunzip
Size: 9432 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/389-users/attachments/20120223/34637f30/attachment-0001.bin>