[389-devel] fractional replication monitoring proposal

[Mark Reynolds]

On 10/16/2013 11:05 AM, Ludwig Krispenz wrote:
>
> On 10/15/2013 10:41 PM, Mark Reynolds wrote:
>> https://fedorahosted.org/389/ticket/47368
>>
>> So we run into issues when trying to figure out if replicas are in 
>> synch(if those replicas use fractional replication and "strip 
>> mods").  What happens is that an update is made on master A, but due 
>> to fractional replication there is no update made to any replicas. So 
>> if you look at the ruv in the tombstone entry on each server, it 
>> would appear they are out of synch.  So using the ruv in the db 
>> tombstone is no longer accurate when using fractional replication.
>>
>> I'm proposing a new ruv to be stored in the backend replica entry: 
>> e.g. cn=replica,cn="dc=example,dc=com",cn=mapping tree,cn=config. I'm 
>> calling this the "replicated ruv".  So whenever we actually send an 
>> update to a replica, this ruv will get updated.
> I don't see how this will help, you have an additional info on waht 
> has been replicated (which is available on the consumer as well) and 
> you have a max csn, but you don't know if there are outstanding 
> fractional changes to be sent.
Well you will know on master A what operations get replicated(this 
updates the new ruv before sending any changes), and you can use this 
ruv to compare against the other master B's ruv(in its replication 
agreement).   Maybe I am missing your point?  Do you mean changes that 
have not been read from the changelog yet?  My plan was to update the 
new ruv in perform_operation() - right after all the "stripping" has 
been done and there is something to replicate.  We need to have a ruv 
for replicated operations.

I guess there are other scenarios I didn't think of, like if replication 
is in a backoff state, and valid changes are coming in. Maybe, we could 
do test "stripping" earlier in the replication process(when writing to 
the changelog?), and then update the new ruv there instead of waiting 
until we try and send the changes.
>> Since we can not compare this "replicated ruv" to the replicas 
>> tombstone ruv, we can instead compare the "replicated ruv" to the ruv 
>> in the replica's repl agreement(unless it is a dedicated consumer - 
>> here we might be able to still look at the db tombstone ruv to 
>> determine the status).
>>
>> Problems with this approach:
>>
>> -  All the servers need to have the same replication 
>> configuration(the same fractional replication policy and attribute 
>> stripping) to give accurate results.
>>
>> -  If one replica has an agreement that does NOT filter the updates, 
>> but has agreements that do filter updates, then we can not correctly 
>> determine its synchronization state with the fractional replicas.
>>
>> -  Performance hit from updating another ruv(in cn=config)?
>>
>>
>> Fractional replication simply breaks our monitoring process. I'm not 
>> sure, not without updating the repl protocol, that we can cover all 
>> deployment scenarios(mixed fractional repl agmts, etc). However, I 
>> "think" this approach would work for most deployments(compared to 
>> none at the moment).  For IPA, since they don't use consumers, this 
>> approach would work for them.  And finally, all of this would have to 
>> be handled by a updated version of repl-monitor.pl.
>>
>> This is just my preliminary idea on how to handle this. Feedback is 
>> welcome!!
>>
>> Thanks in advance,
>> Mark
>>
>> -- 
>> Mark Reynolds
>> 389 Development Team
>> Red Hat, Inc
>> mreynolds at redhat.com
>>
>>
>> --
>> 389-devel mailing list
>> 389-devel at lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/389-devel
>
>
>
> --
> 389-devel mailing list
> 389-devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/389-devel

-- 
Mark Reynolds
389 Development Team
Red Hat, Inc
mreynolds at redhat.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/389-devel/attachments/20131016/f451362d/attachment.html>