Please review: ticket #48766 Replication changelog can incorrectly skip over updates

[Ludwig Krispenz]

This is the latest version of the "changelog buffer processing" fixes.


https://fedorahosted.org/389/ticket/48766

https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...

The background for the fix is here, I would like to get feedback on this 
as well to clarify what is unclear
http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

[thierry bordaz]

On 05/23/2016 03:06 PM, Ludwig Krispenz wrote:
>  This is the latest version of the "changelog buffer
> processing" fixes.
>
>
>  https://fedorahosted.org/389/ticket/48766
>
>
> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...
>
>
>
>  The background for the fix is here, I would like to get feedback on 
>  this as well to clarify what is unclear
>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...
>
>
>  
Hello Ludwig,

I have not yet reviewed the patch. I was looking at the design.

Regarding your note: 
http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta....

It is said that the anchorCSN will not be the from the consumerRID. What 
is the mechanism that guaranty that the consumer will receive all the 
updates it was the originator ?

thanks
thierry

[Ludwig Krispenz]

Hi,

On 05/23/2016 06:29 PM, thierry bordaz wrote:
>  On 05/23/2016 03:06 PM, Ludwig Krispenz wrote:
> > This is the latest version of the "changelog buffer processing" fixes.
> >
> >
> > https://fedorahosted.org/389/ticket/48766
> >
> >
> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...
>
> >
> >
> > The background for the fix is here, I would like to get feedback on 
> > this as well to clarify what is unclear
> >
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...
>
> >
> >
>
>  Hello Ludwig,
>
>  I have not yet reviewed the patch. I was looking at the design.
>
>  Regarding your note: 
>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta....
If you refer to this part:


      Special case: RID of the consumer in the current replication session

If the consumer in the replication session is also a master its RID will 
be contained at least in the consumerRUV. If it is also in the supplier 
RUV the question is if it should be considered in the decision if 
updates should be sent. Normally a master has the latest changes applied 
to itself, so there would be no need to check and send updates for its 
RID. But there can be scenarios where this is not the case: if the 
consumer has been restored from an older backup the latest csn for its 
own RID might be older than changes available on other servers.

|NOTE: The current implementation ignores anchorCSNs based on the consumer RID. If, by
chance, the anchor csn used is older than this csn, the changes will be sent, but they
also ca nbe lost.
|


this referres to the "current" implementation before the fix, the doc 
started as a post-design doc, and it shoul dbe correctedd
with the fix the if the supplier has newer changes for the consumerRID 
than the consumer it will be reflected in the anchor csn calculation.


>  It is said that the anchorCSN will not be the from the consumerRID. 
>  What is the mechanism that guaranty that the consumer will receive all 
>  the updates it was the originator ?
>
>  thanks
>  thierry
>  -- 
>  389-devel mailing list
>  389-devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail] 
>  
-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

[thierry bordaz]

Hi Ludwig,

Thanks for your explanation. The design looks very good. I think it 
would be good to put into the code (especially clcache_adjust_anchorcsn) 
the reference to the related design paragraph.

There is something I do not understand in clcache_skip_change.
My understanding is that this is the only place where the 
consumer_maxcsn is updated.
But there are several conditions, if we decide to skip the update that 
the consumer_maxcsn is not updated.
One of them is 'rid == buf->buf_consumer_rid'.
Does that mean that the consumer_maxcsn remains unchanged for the RID of 
the consumer ?

thanks
thierry




On 05/24/2016 09:22 AM, Ludwig Krispenz wrote:
>  Hi,
>
>  On 05/23/2016 06:29 PM, thierry bordaz wrote:
> >
> >
> > On 05/23/2016 03:06 PM, Ludwig Krispenz wrote:
> >> This is the latest version of the "changelog buffer processing" fixes.
> >>
> >>
> >> https://fedorahosted.org/389/ticket/48766
> >>
> >>
> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...
>
> >>
> >>
> >> The background for the fix is here, I would like to get feedback on 
> >> this as well to clarify what is unclear
> >>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...
>
> >>
> >>
> >
> > Hello Ludwig,
> >
> > I have not yet reviewed the patch. I was looking at the design.
> >
> > Regarding your note: 
> >
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta....
>  If you refer to this part:
>
>
>        Special case: RID of the consumer in the current replication session
>
>  If the consumer in the replication session is also a master its RID 
>  will be contained at least in the consumerRUV. If it is also in the 
>  supplier RUV the question is if it should be considered in the 
>  decision if updates should be sent. Normally a master has the latest 
>  changes applied to itself, so there would be no need to check and send 
>  updates for its RID. But there can be scenarios where this is not the 
>  case: if the consumer has been restored from an older backup the 
>  latest csn for its own RID might be older than changes available on 
>  other servers.
>
>  |NOTE: The current implementation ignores anchorCSNs based on the 
>  consumer RID. If, by chance, the anchor csn used is older than this 
>  csn, the changes will be sent, but they also ca nbe lost. |
>
>  this referres to the "current" implementation before the fix, the doc 
>  started as a post-design doc, and it shoul dbe correctedd
>  with the fix the if the supplier has newer changes for the consumerRID 
>  than the consumer it will be reflected in the anchor csn calculation.
>
>
> >
> > It is said that the anchorCSN will not be the from the consumerRID. 
> > What is the mechanism that guaranty that the consumer will receive 
> > all the updates it was the originator ?
> >
> > thanks
> > thierry
> > -- 
> > 389-devel mailing list
> > 389-devel(a)lists.fedoraproject.org
> > http://lists.fedoraproject.org/admin/lists/[e-mail] 
> >
>
>  -- 
>  Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn,
>  Commercial register: Amtsgericht Muenchen, HRB 153243,
>  Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael
> O'Neill
>
>
>  --
>  389-devel mailing list
>  389-devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail]

[Ludwig Krispenz]

On 05/24/2016 04:20 PM, thierry bordaz wrote:
>  Hi Ludwig,
>
>  Thanks for your explanation. The design looks very good. I think it 
>  would be good to put into the code (especially 
>  clcache_adjust_anchorcsn) the reference to the related design paragraph.
>
>  There is something I do not understand in clcache_skip_change.
>  My understanding is that this is the only place where the 
>  consumer_maxcsn is updated.
>  But there are several conditions, if we decide to skip the update that 
>  the consumer_maxcsn is not updated.
>  One of them is 'rid == buf->buf_consumer_rid'.
>  Does that mean that the consumer_maxcsn remains unchanged for the RID 
>  of the consumer ? 
the condition is:
if( rid == buf->buf_consumer_rid && buf->buf_ignoreConsumerRID)

so it will only be skipped if we determined that we don't need to send 
anything for the consumers own rid

>  thanks
>  thierry
>
>
>
>
>  On 05/24/2016 09:22 AM, Ludwig Krispenz wrote:
> > Hi,
> >
> > On 05/23/2016 06:29 PM, thierry bordaz wrote:
> >>
> >>
> >> On 05/23/2016 03:06 PM, Ludwig Krispenz wrote:
> >>> This is the latest version of the "changelog buffer processing"
> fixes.
> >>>
> >>>
> >>> https://fedorahosted.org/389/ticket/48766
> >>>
> >>>
> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...
>
> >>>
> >>>
> >>> The background for the fix is here, I would like to get feedback on 
> >>> this as well to clarify what is unclear
> >>>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...
>
> >>>
> >>>
> >>
> >> Hello Ludwig,
> >>
> >> I have not yet reviewed the patch. I was looking at the design.
> >>
> >> Regarding your note: 
> >>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta....
> > If you refer to this part:
> >
> >
> >       Special case: RID of the consumer in the current
> >       replication session
> >
> > If the consumer in the replication session is also a master its RID 
> > will be contained at least in the consumerRUV. If it is also in the 
> > supplier RUV the question is if it should be considered in the 
> > decision if updates should be sent. Normally a master has the latest 
> > changes applied to itself, so there would be no need to check and 
> > send updates for its RID. But there can be scenarios where this is 
> > not the case: if the consumer has been restored from an older backup 
> > the latest csn for its own RID might be older than changes available 
> > on other servers.
> >
> > |NOTE: The current implementation ignores anchorCSNs based on the consumer RID. If,
> by chance, the anchor csn used is older than this csn, the changes will be sent, but they
> also ca nbe lost.
> > |
> >
> > this referres to the "current" implementation before the fix, the doc 
> > started as a post-design doc, and it shoul dbe correctedd
> > with the fix the if the supplier has newer changes for the 
> > consumerRID than the consumer it will be reflected in the anchor csn 
> > calculation.
> >
> >
> >>
> >> It is said that the anchorCSN will not be the from the consumerRID. 
> >> What is the mechanism that guaranty that the consumer will receive 
> >> all the updates it was the originator ?
> >>
> >> thanks
> >> thierry
> >> -- 
> >> 389-devel mailing list
> >> 389-devel(a)lists.fedoraproject.org
> >> http://lists.fedoraproject.org/admin/lists/[e-mail] 
> >>
> >
> > -- 
> > Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn,
> > Commercial register: Amtsgericht Muenchen, HRB 153243,
> > Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael
> O'Neill
> >
> >
> > --
> > 389-devel mailing list
> > 389-devel(a)lists.fedoraproject.org
> > http://lists.fedoraproject.org/admin/lists/[e-mail]
>
>
>
>  --
>  389-devel mailing list
>  389-devel(a)lists.fedoraproject.org
>  http://lists.fedoraproject.org/admin/lists/[e-mail] 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

[Ludwig Krispenz]

Hi,

here is the latest correction to the changelog fix, in fact it is 
Thierry's version of teh fix which simplified the logic a bit
https://fedorahosted.org/389/attachment/ticket/48766/0001-PATCH-use-a-con...

Ludwig

On 05/23/2016 03:06 PM, Ludwig Krispenz wrote:
>  This is the latest version of the "changelog buffer
> processing" fixes.
>
>
>  https://fedorahosted.org/389/ticket/48766
>
>
> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach...
>
>
>
>  The background for the fix is here, I would like to get feedback on 
>  this as well to clarify what is unclear
>
> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-sta...
>
>
>  
-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
Shander