<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Mahadevan,<br>
<br>
<blockquote>I think you hit a new bug that I was able to
reproduce.<br>
<br>
The problem is an incorrect handling of operation return code
when there is an error in SLAPI_PLUGIN_BE_TXN_POST_ADD_FN.<br>
When the operation succeeds in the DB, DS (as a master) updates
the CL and RUV in SLAPI_PLUGIN_BE_TXN_POST_ADD_FN phase.<br>
At this moment the operation is successful. However the update
of CL/RUV may fail (your case). It correctly aborts the update
of the CL AND the txn related to the operation in the DB (like
if the operation never happened)... but it looks like there is a
bug where it keeps the operation returned code.<br>
The operation return code should be a failure, but it is
successful.<br>
<br>
I reproduced the problem with a debug version where I simulate a
DB_DEADLOCK failure<br>
<blockquote><tt>[17/Jun/2013:11:07:21 +0200] - slapd started.
Listening on All Interfaces port 24502 for LDAP requests</tt><br>
<tt>[17/Jun/2013:11:08:23 +0200] NSMMReplicationPlugin -
changelog program - _cl5WriteOperationTxn: failed to write
entry with csn (51bed207000000010000); db error - -30993
BDB0068 DB_LOCK_DEADLOCK: Locker killed to resolve a
deadlock</tt><br>
<tt>[17/Jun/2013:11:08:23 +0200] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=td,dc=com (uniqid: 74589a01-d72d11e2-b4c2bdbf-b0a22af5,
optype: 16) to changelog csn 51bed207000000010000</tt><br>
<br>
<br>
<tt>[tbordaz@pctbordaz userRoot]$ tail -20
/var/log/dirsrv/slapd-master_cl/access</tt><br>
<tt>...</tt><br>
<tt>[17/Jun/2013:11:08:23 +0200] conn=2 op=1 ADD
dn="uid=td,dc=com"</tt><br>
<tt>[17/Jun/2013:11:08:23 +0200] conn=2 op=1 RESULT err=0
tag=105 nentries=0 etime=0 csn=51bed207000000010000</tt><br>
</blockquote>
<br>
I do not know why your environment is prone to trigger
db_deadlock (lot of replica agreements, VM, slow disks...).<br>
I think the best way to progress is that you fill a ticket/bug
so that we may track the issue. Note this bug is possibly
affecting all operations (ADD/MOD/MODRDN/DEL)<br>
<br>
</blockquote>
best regards<br>
thierry<br>
On 06/15/2013 01:03 AM, Mahadevan, Venkat wrote:<br>
</div>
<blockquote
cite="mid:57ABCAA446A34248A21072281010E3305B2CA00C@S-ITSV-MBX01P.ead.ubc.ca"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;
mso-fareast-language:EN-US;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;
mso-fareast-language:EN-US;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;
mso-fareast-language:EN-US;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">>
If the operation fails to write into the changelog, the
operation fails. In your case, it means that the ldapclient
should receive an error.
<br>
> So it is like if the operation never happened and is
not replicated.<br>
<br>
Hi Thierry,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">That’s
what I would expect to, but that does not seem to be the
case. In my access log on the master server below,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">the
client successfully adds the entry to the master server and
receives an error code return of 0:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:36:59
-0700] conn=19048 op=0 BIND dn="cn=Directory Manager"
method=128 version=3<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:36:59
-0700] conn=19048 op=0 RESULT err=0 tag=97 nentries=0
etime=0 dn="cn=directory manager"<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:36:59
-0700] conn=19048 op=1 ADD
dn="uid=jmeter325,dc=tst,dc=id,dc=ubc,dc=ca"<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:37:04
-0700] conn=19048 op=1 RESULT err=0 tag=105 nentries=0
etime=5 csn=51b8dbec002b02bd0000<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">In
the error log at the same time the above happens:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:37:02
-0700] NSMMReplicationPlugin -
replica_replace_ruv_tombstone: failed to update replication
update vector for replica dc=tst,dc=id,dc=ubc,dc=ca: LDAP
error - 51<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:37:04
-0700] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: retry (49) the transaction
(csn=51b8dbec002b02bd0000) failed (rc=-30994
(DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock))<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:37:04
-0700] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: failed to write entry with csn
(51b8dbec002b02bd0000); db error - -30994 DB_LOCK_DEADLOCK:
Locker killed to resolve a deadlock<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">[12/Jun/2013:13:37:04
-0700] NSMMReplicationPlugin - write_changelog_and_ruv:
can't add a change for
uid=jmeter325,dc=tst,dc=id,dc=ubc,dc=ca (uniqid:
c0a7827c-d39f11e2-879add5e-44f44922, optype: 16) to
changelog csn 51b8dbec002b02bd0000<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">The
net result is that the entry is successfully added to the
master server but will never replication to any of the
consumers.
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-CA">VM<o:p></o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>