[389-devel] Please review: [389 Project] #47737: Under heavy stress, failure of turning a tombstone into glue makes the server hung

[Noriko Hosoi]

https://fedorahosted.org/389/ticket/47737

https://fedorahosted.org/389/attachment/ticket/47737/0001-Ticket-47737-Under-heavy-stress-failure-of-turning-a.patch
>   Turning a tombstone entry to a glue entry is done in a while loop
>   (create_glue_entry:urp_glue.c)  Unless the transformation is successful
>   (or LDAP_NO_SUCH_OBJECT), it cannot exit from the loop.  But under a
>   stress, there could be a tombstone and a conflict entry coexist, and
>   do_create_glue_entry keeps returning LDAP_ALREADY_EXISTS.  In such a case,
>   we need to give up greating a glue.
>   {{{
>   [..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000:
>   Can't created glue entry
>   ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com
>   uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
>   [..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000:
>   Can't created glue entry
>   ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com
>   uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
>   [..] NSMMReplicationPlugin - conn=7 op=1939 csn=531a1443000700030000:
>   Can't created glue entry
>   ou=test,ou=projects,o=bees,ou=organizations,dc=cvsdude,dc=com
>   uniqueid=ee68e001-a62811e3-bc8ab407-12c832a2, error 68
>   [..]
>   }}}
>
>
>   {{{
>   Thread 32 (Thread 0x7f6ac77fe700 (LWP 24906)):
>   #0  0x00007f6ae4e3e74d in fsync () at ../sysdeps/unix/syscall-
>   template.S:81
>   #1  0x00007f6ae5492e8b in pt_Fsync (fd=0x7f6ae81b15c0) at
>   ../../../nspr/pr/src/pthreads/ptio.c:1530
>   #2  0x00007f6ae6e8afe7 in vslapd_log_error (fp=0x7f6ae81b15c0,
>   subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
>       fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error
>   %d\n", ap=0x7f6ac77f7438, locked=1)
>       at ldap/servers/slapd/log.c:1953
>   #3  0x00007f6ae6e8aa52 in slapd_log_error_proc_internal
>   (subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
>       fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error
>   %d\n", ap_err=0x7f6ac77f7420,
>       ap_file=0x7f6ac77f7438) at ldap/servers/slapd/log.c:1809
>   #4  0x00007f6ae6e8b1d5 in slapi_log_error (severity=0,
>   subsystem=0x7f6adb19ad90 "NSMMReplicationPlugin",
>       fmt=0x7f6adb1a6fa0 "%s: Can't created glue entry %s uniqueid=%s, error
>   %d\n") at ldap/servers/slapd/log.c:1994
>   #5  0x00007f6adb17a6b3 in create_glue_entry (pb=0x7f6aa40b7f90,
>       sessionid=0x7f6ac77f7690 "conn=7 op=1939 csn=531a1443000700030000",
>   dn=0x7f6aa40c6370,
>       uniqueid=0x7f6aa40bee60 "ee68e001-a62811e3-bc8ab407-12c832a2",
>   opcsn=0x7f6aa40bee40)
>       at ldap/servers/plugins/replication/urp_glue.c:257
>   #6  0x00007f6adb1791eb in urp_add_resolve_parententry (pb=0x7f6aa40b7f90,
>       sessionid=0x7f6ac77f7690 "conn=7 op=1939 csn=531a1443000700030000",
>   entry=0x7f6aa40badf0, parententry=0x0,
>       opcsn=0x7f6aa40bee40) at ldap/servers/plugins/replication/urp.c:908
>   #7  0x00007f6adb177e29 in urp_add_operation (pb=0x7f6aa40b7f90) at
>   ldap/servers/plugins/replication/urp.c:165
>   #8  0x00007f6adb15ae22 in multimaster_bepreop_add (pb=0x7f6aa40b7f90) at
>   ldap/servers/plugins/replication/repl5_plugins.c:711
>   #9  0x00007f6ae6eade99 in plugin_call_func (list=0x7f6ae830fd90,
>   operation=450, pb=0x7f6aa40b7f90, call_one=0)
>       at ldap/servers/slapd/plugin.c:1453
>   #10 0x00007f6ae6eadd59 in plugin_call_list (list=0x7f6ae830fd90,
>   operation=450, pb=0x7f6aa40b7f90)
>       at ldap/servers/slapd/plugin.c:1415
>   #11 0x00007f6ae6eabfe1 in plugin_call_plugins (pb=0x7f6aa40b7f90,
>   whichfunction=450) at ldap/servers/slapd/plugin.c:398
>   #12 0x00007f6adc085696 in ldbm_back_add (pb=0x7f6aa40b7f90) at
>   ldap/servers/slapd/back-ldbm/ldbm_add.c:257
>   #13 0x00007f6ae6e478aa in op_shared_add (pb=0x7f6aa40b7f90) at
>   ldap/servers/slapd/add.c:681
>   #14 0x00007f6ae6e468b4 in do_add (pb=0x7f6aa40b7f90) at
>   ldap/servers/slapd/add.c:258
>   #15 0x00007f6ae7379935 in connection_dispatch_operation
>   (conn=0x7f6ae71e3f48, op=0x7f6aa40b6330, pb=0x7f6aa40b7f90)
>       at ldap/servers/slapd/connection.c:579
>   #16 0x00007f6ae737b32c in connection_threadmain () at
>   ldap/servers/slapd/connection.c:2339
>   #17 0x00007f6ae5494c86 in _pt_root (arg=0x7f6ae84eb130) at
>   ../../../nspr/pr/src/pthreads/ptthread.c:204
>   #18 0x00007f6ae4e37d15 in start_thread (arg=0x7f6ac77fe700) at
>   pthread_create.c:308
>   #19 0x00007f6ae495453d in clone () at
>   ../sysdeps/unix/sysv/linux/x86_64/clone.S:114
>   }}}
>