[389-users] replication failure - clock skew

[Greg Kuchyt]

Yesterday afternoon, one of my consumers randomly crashed/rebooted. Upon 
rebooting, its replication agreement with its master failed with the 
following error:

Unable to acquire replica: Excessive clock skew between the supplier and 
the consumer. Replication is aborting

I did a little bit of Google searching and found some list traffic from 
a few years ago. From that I derived that this replica was hosed and I 
would need to re-initialize it. No problem. A re-initialization didn't 
do anything, same error. Starting from scratch from a completely 
new/fresh replica produces the same result. That's when I noticed the 
following errors in the logs on the master.

csngen_new_csn - Warning: too much time skew (-115319 secs). Current 
seqnum=1

I downloaded the readNsState.py script attached to the following ticket 
(https://bugzilla.redhat.com/show_bug.cgi?id=233642). Running this on 
the master produced the following output

For replica cn=replica,cn=o\3Dpotsdam.edu,cn=mapping tree,cn=config
   len of nsstate is 40
   CSN generator state:
     Replica ID    : 6560
     Sampled Time  : 1328928777
     Time in hex   : 0x4f35d809
     Time as str   : Fri Feb 10 21:52:57 2012
     Local Offset  : 0
     Remote Offset : 261
     Seq. num      : 1
     System time   : Thu Feb  9 14:00:01 2012
     Diff in sec.  : -114776

This leads me to believe that the clock skew problem is on the master.

I am not really sure how the clock skew happened. All of these systems 
synchronize their clocks via a centralized time server and all the times 
on their clocks are correct. There are 3 or 4 other replicas that are 
still receiving incremental updates fine, but any attempt to add a new 
replica results in a failed replication agreement due to excessive clock 
skew.

I am writing to get a better understanding of the situation and see if 
there is anything to be done to resolve this. At the moment it seems as 
if I am caught in an unfortunate situation that will require 
re-initialization of my master from a back-up.

Thanks for any help that can be provided.