After upgrade, only one direction replication while should be bi-directions replication

[Kathy Zhu]

Hi Team,

We upgraded our Centos 7 IPA masters to the latest:

CentOS Linux release 7.9.2009 (Core)

*ipa*-server.x86_64                      4.6.8-5.el7.centos.10

*389-ds*-base.x86_64                     1.3.10.2-15.el7_9

*389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9

*389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9

*slapi*-nis.x86_64                       0.56.5-3.el7_9


After that, 8 of 10 masters had replication issues. After reinitializing, 2
of them are still having issues. They can accept replication from other
masters but their own changes can not be replicated to others.


Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:


[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389): Data required to update replica has been purged from the
changelog. If the error persists the replica must be reinitialized.

[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=
dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389) -
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
changelog (DB rc=-30988). If replication stops, the consumer may need to be
reinitialized.

[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
changelog program - repl_plugin_name_cl - agmt="cn=
dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN
627e26a50005001d0000 not found, we aren't as up to date, or we purged

[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com"
(dc3-ipa1:389): Data required to update replica has been purged from the
changelog. If the error persists the replica must be reinitialized.

[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=
dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer
- Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If
replication stops, the consumer may need to be reinitialized.

[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
changelog program - repl_plugin_name_cl - agmt="cn=
dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN
627e26a50005001d0000 not found, we aren't as up to date, or we purged

[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com"
(ipa0:389): Data required to update replica has been purged from the
changelog. If the error persists the replica must be reinitialized.

[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=
dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389) -
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
changelog (DB rc=-30988). If replication stops, the consumer may need to be
reinitialized.

[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
changelog program - repl_plugin_name_cl - agmt="cn=
dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN
627e26a50005001d0000 not found, we aren't as up to date, or we purged

[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389): Data required to update replica has been purged from the
changelog. If the error persists the replica must be reinitialized.


Note, those messages are after being reinitialized.


Any idea what's wrong here?


Thanks.


Kathy.

[Rob Crittenden]

Kathy Zhu via FreeIPA-users wrote:
>  Hi Team, 
>  
>  We upgraded our Centos 7 IPA masters to the latest: 
>  
>  CentOS Linux release 7.9.2009 (Core)
>  
>  *ipa*-server.x86_64                      4.6.8-5.el7.centos.10 
>  
>  *389-ds*-base.x86_64                     1.3.10.2-15.el7_9    
>  
>  *389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9   
>  
>  *389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9  
>  
>  *slapi*-nis.x86_64                       0.56.5-3.el7_9 
>  
>  
>  After that, 8 of 10 masters had replication issues. After
>  reinitializing, 2 of them are still having issues. They can accept
>  replication from other masters but their own changes can not be
>  replicated to others. 
>  
>  
>  Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
>  
>  
>  [01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
>  send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
>  Data required to update replica has been purged from the changelog. If
>  the error persists the replica must be reinitialized.
>  
>  [01/Jun/2022:21:53:03.396330801 -0700] - ERR -
>  agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389) -
>  clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  changelog (DB rc=-30988). If replication stops, the consumer may need to
>  be reinitialized.
>  
>  [01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
>  changelog program - repl_plugin_name_cl -
>  agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389):
>  CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
>  
>  [01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
>  send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389):
>  Data required to update replica has been purged from the changelog. If
>  the error persists the replica must be reinitialized.
>  
>  [01/Jun/2022:21:53:04.411599251 -0700] - ERR -
>  agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389) -
>  clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  changelog (DB rc=-30988). If replication stops, the consumer may need to
>  be reinitialized.
>  
>  [01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
>  changelog program - repl_plugin_name_cl -
>  agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): CSN
>  627e26a50005001d0000 not found, we aren't as up to date, or we purged
>  
>  [01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
>  send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): Data
>  required to update replica has been purged from the changelog. If the
>  error persists the replica must be reinitialized.
>  
>  [01/Jun/2022:21:53:05.482898290 -0700] - ERR -
>  agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389) -
>  clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  changelog (DB rc=-30988). If replication stops, the consumer may need to
>  be reinitialized.
>  
>  [01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
>  changelog program - repl_plugin_name_cl -
>  agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
>  CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
>  
>  [01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
>  send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
>  Data required to update replica has been purged from the changelog. If
>  the error persists the replica must be reinitialized.
>  
>  
>  Note, those messages are after being reinitialized. 
>  
>  
>  Any idea what's wrong here?  
I'm not sure, cc'ing one of the 389ds
developers.

Mark, any ideas?

rob

[Mark Reynolds]

On 6/2/22 1:38 PM, Rob Crittenden wrote:
>  Kathy Zhu via FreeIPA-users wrote:
> > Hi Team,
> >
> > We upgraded our Centos 7 IPA masters to the latest:
> >
> > CentOS Linux release 7.9.2009 (Core)
> >
> > *ipa*-server.x86_64                      4.6.8-5.el7.centos.10
> >
> > *389-ds*-base.x86_64                     1.3.10.2-15.el7_9
> >
> > *389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9
> >
> > *389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9
> >
> > *slapi*-nis.x86_64                       0.56.5-3.el7_9
> >
> >
> > After that, 8 of 10 masters had replication issues. After
> > reinitializing, 2 of them are still having issues. They can accept
> > replication from other masters but their own changes can not be
> > replicated to others.
> >
> >
> > Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
> >
> >
> > [01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
> > send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
> > Data required to update replica has been purged from the changelog. If
> > the error persists the replica must be reinitialized.
> >
> > [01/Jun/2022:21:53:03.396330801 -0700] - ERR -
> > agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389) -
> > clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > changelog (DB rc=-30988). If replication stops, the consumer may need to
> > be reinitialized.
> >
> > [01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
> > changelog program - repl_plugin_name_cl -
> > agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389):
> > CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
> >
> > [01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
> > send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389):
> > Data required to update replica has been purged from the changelog. If
> > the error persists the replica must be reinitialized.
> >
> > [01/Jun/2022:21:53:04.411599251 -0700] - ERR -
> > agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389) -
> > clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > changelog (DB rc=-30988). If replication stops, the consumer may need to
> > be reinitialized.
> >
> > [01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
> > changelog program - repl_plugin_name_cl -
> > agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): CSN
> > 627e26a50005001d0000 not found, we aren't as up to date, or we purged
> >
> > [01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
> > send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): Data
> > required to update replica has been purged from the changelog. If the
> > error persists the replica must be reinitialized.
> >
> > [01/Jun/2022:21:53:05.482898290 -0700] - ERR -
> > agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389) -
> > clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > changelog (DB rc=-30988). If replication stops, the consumer may need to
> > be reinitialized.
> >
> > [01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
> > changelog program - repl_plugin_name_cl -
> > agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
> > CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
> >
> > [01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
> > send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389):
> > Data required to update replica has been purged from the changelog. If
> > the error persists the replica must be reinitialized.
> >
> >
> > Note, those messages are after being reinitialized.
> >
> >
> > Any idea what's wrong here?
>  I'm not sure, cc'ing one of the 389ds developers.
>
>  Mark, any ideas? 
Looks like https://github.com/389ds/389-ds-base/issues/5098

This was fixed fairly recently, and it has not been added to any RHEL 
7.9 builds yet.

Pierre worked on this, but I think the issue happens when the LDIF used 
to init replication has the RUV entry as the first entry in the LDIF 
file.  If it is moved to the end of the file I think it will fix the 
init issue.  So don't do an online init.  Export the database with 
replication data (-r) from a good supplier, edit the ldif file and make 
sure the RUV tombstone entry is the last entry in the file, then import 
it on the consumer.

HTH,
Mark

>  rob
>  
-- 
Directory Server Development Team

[Kathy Zhu]

Thanks, Mark, to identify and explain the issue.

Could someone pass a document or article how to apply the fix/work around?
Many thanks!

Kathy.

On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds <mareynol(a)redhat.com&gt; wrote:

>  On 6/2/22 1:38 PM, Rob Crittenden wrote:
>  > Kathy Zhu via FreeIPA-users wrote:
>  >> Hi Team,
>  >>
>  >> We upgraded our Centos 7 IPA masters to the latest:
>  >>
>  >> CentOS Linux release 7.9.2009 (Core)
>  >>
>  >> *ipa*-server.x86_64                      4.6.8-5.el7.centos.10
>  >>
>  >> *389-ds*-base.x86_64                     1.3.10.2-15.el7_9
>  >>
>  >> *389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9
>  >>
>  >> *389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9
>  >>
>  >> *slapi*-nis.x86_64                       0.56.5-3.el7_9
>  >>
>  >>
>  >> After that, 8 of 10 masters had replication issues. After
>  >> reinitializing, 2 of them are still having issues. They can accept
>  >> replication from other masters but their own changes can not be
>  >> replicated to others.
>  >>
>  >>
>  >> Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
>  >>
>  >>
>  >> [01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
>  >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
>  >> Data required to update replica has been purged from the changelog. If
>  >> the error persists the replica must be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:03.396330801 -0700] - ERR -
>  >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>" (dc3-ipa1:389)
> -
>  >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  >> changelog (DB rc=-30988). If replication stops, the consumer may need to
>  >> be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
>  >> changelog program - repl_plugin_name_cl -
>  >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> (dc3-ipa1:389):
>  >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
>  purged
>  >>
>  >> [01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
>  >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> (dc3-ipa1:389):
>  >> Data required to update replica has been purged from the changelog. If
>  >> the error persists the replica must be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:04.411599251 -0700] - ERR -
>  >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389) -
>  >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  >> changelog (DB rc=-30988). If replication stops, the consumer may need to
>  >> be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
>  >> changelog program - repl_plugin_name_cl -
>  >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): CSN
>  >> 627e26a50005001d0000 not found, we aren't as up to date, or we purged
>  >>
>  >> [01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
>  >> send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
>  >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389): Data
>  >> required to update replica has been purged from the changelog. If the
>  >> error persists the replica must be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:05.482898290 -0700] - ERR -
>  >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>" (dc2-ipa1:389)
> -
>  >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
>  >> changelog (DB rc=-30988). If replication stops, the consumer may need to
>  >> be reinitialized.
>  >>
>  >> [01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
>  >> changelog program - repl_plugin_name_cl -
>  >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
>  >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
>  purged
>  >>
>  >> [01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
>  >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
>  >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
>  >> Data required to update replica has been purged from the changelog. If
>  >> the error persists the replica must be reinitialized.
>  >>
>  >>
>  >> Note, those messages are after being reinitialized.
>  >>
>  >>
>  >> Any idea what's wrong here?
>  > I'm not sure, cc'ing one of the 389ds developers.
>  >
>  > Mark, any ideas?
>
>  Looks like https://github.com/389ds/389-ds-base/issues/5098
>
>  This was fixed fairly recently, and it has not been added to any RHEL
>  7.9 builds yet.
>
>  Pierre worked on this, but I think the issue happens when the LDIF used
>  to init replication has the RUV entry as the first entry in the LDIF
>  file.  If it is moved to the end of the file I think it will fix the
>  init issue.  So don't do an online init.  Export the database with
>  replication data (-r) from a good supplier, edit the ldif file and make
>  sure the RUV tombstone entry is the last entry in the file, then import
>  it on the consumer.
>
>  HTH,
>  Mark
>
>  >
>  > rob
>  >
>  --
>  Directory Server Development Team
>
>

[Kathy Zhu]

Do you mean /etc/dirsrv/slapd-CORP-NURO-TEAM/dse.ldif file? There is no
entry with RUV.

Thanks.

Kathy.

On Thu, Jun 2, 2022 at 5:57 PM Kathy Zhu wrote:

>  Thanks, Mark, to identify and explain the issue.
>
>  Could someone pass a document or article how to apply the fix/work around?
>  Many thanks!
>
>  Kathy.
>
>  On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds <mareynol(a)redhat.com&gt; wrote:
>
> >
> > On 6/2/22 1:38 PM, Rob Crittenden wrote:
> > > Kathy Zhu via FreeIPA-users wrote:
> > >> Hi Team,
> > >>
> > >> We upgraded our Centos 7 IPA masters to the latest:
> > >>
> > >> CentOS Linux release 7.9.2009 (Core)
> > >>
> > >> *ipa*-server.x86_64                      4.6.8-5.el7.centos.10
> > >>
> > >> *389-ds*-base.x86_64                     1.3.10.2-15.el7_9
> > >>
> > >> *389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9
> > >>
> > >> *389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9
> > >>
> > >> *slapi*-nis.x86_64                       0.56.5-3.el7_9
> > >>
> > >>
> > >> After that, 8 of 10 masters had replication issues. After
> > >> reinitializing, 2 of them are still having issues. They can accept
> > >> replication from other masters but their own changes can not be
> > >> replicated to others.
> > >>
> > >>
> > >> Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
> > >>
> > >>
> > >> [01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
> > >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
> > >> Data required to update replica has been purged from the changelog. If
> > >> the error persists the replica must be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:03.396330801 -0700] - ERR -
> > >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> (dc3-ipa1:389)
> > -
> > >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > >> changelog (DB rc=-30988). If replication stops, the consumer may need
> > to
> > >> be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
> > >> changelog program - repl_plugin_name_cl -
> > >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> (dc3-ipa1:389):
> > >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
> > purged
> > >>
> > >> [01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
> > >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> (dc3-ipa1:389):
> > >> Data required to update replica has been purged from the changelog. If
> > >> the error persists the replica must be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:04.411599251 -0700] - ERR -
> > >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389) -
> > >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > >> changelog (DB rc=-30988). If replication stops, the consumer may need
> > to
> > >> be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
> > >> changelog program - repl_plugin_name_cl -
> > >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389):
> CSN
> > >> 627e26a50005001d0000 not found, we aren't as up to date, or we purged
> > >>
> > >> [01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
> > >> send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> > >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389):
> Data
> > >> required to update replica has been purged from the changelog. If the
> > >> error persists the replica must be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:05.482898290 -0700] - ERR -
> > >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389)
> > -
> > >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> > >> changelog (DB rc=-30988). If replication stops, the consumer may need
> > to
> > >> be reinitialized.
> > >>
> > >> [01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
> > >> changelog program - repl_plugin_name_cl -
> > >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
> > >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
> > purged
> > >>
> > >> [01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
> > >> send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> > >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> (dc2-ipa1:389):
> > >> Data required to update replica has been purged from the changelog. If
> > >> the error persists the replica must be reinitialized.
> > >>
> > >>
> > >> Note, those messages are after being reinitialized.
> > >>
> > >>
> > >> Any idea what's wrong here?
> > > I'm not sure, cc'ing one of the 389ds developers.
> > >
> > > Mark, any ideas?
> >
> > Looks like https://github.com/389ds/389-ds-base/issues/5098
> >
> > This was fixed fairly recently, and it has not been added to any RHEL
> > 7.9 builds yet.
> >
> > Pierre worked on this, but I think the issue happens when the LDIF used
> > to init replication has the RUV entry as the first entry in the LDIF
> > file.  If it is moved to the end of the file I think it will fix the
> > init issue.  So don't do an online init.  Export the database with
> > replication data (-r) from a good supplier, edit the ldif file and make
> > sure the RUV tombstone entry is the last entry in the file, then import
> > it on the consumer.
> >
> > HTH,
> > Mark
> >
> > >
> > > rob
> > >
> > --
> > Directory Server Development Team
> >
> >

[Kathy Zhu]

I found the error, will error number 18 help to fix the problem? Thanks.

Kathy.

[root@g-ipa1 ~]# ldapsearch -D "cn=directory manager" -W -b "cn=
g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config"

Enter LDAP Password:

# extended LDIF

#

# LDAPv3

# base
<cn=g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config> with scope subtree

# filter: (objectclass=*)

# requesting: ALL

#


# g-ipa1.example.com-to-h-ipa1.example.com, replica, dc\3Dexample\2Cdc\3

 Dcom, mapping tree, config

dn: cn=g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dc

 orp\2Cdc\3Dnuro\2Cdc\3Dteam,cn=mapping tree,cn=config

objectClass: nsds5replicationagreement

objectClass: ipaReplTopoManagedAgreement

objectClass: top

cn: g-ipa1.example.com-to-h-ipa1.example.com

nsDS5ReplicaHost: h-ipa1.example.com

nsDS5ReplicaPort: 389

nsds5replicaTimeout: 300

nsDS5ReplicaRoot: dc=example,dc=com

description: g-ipa1.example.com to h-ipa1.example.com

ipaReplTopoManagedAgreementState: managed agreement - generated by topology
pl

 ugin

nsDS5ReplicaTransportInfo: LDAP

nsDS5ReplicaBindMethod: SASL/GSSAPI

nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof
idnssoaserial

  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount

nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName
in

 ternalModifyTimestamp

nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn
krblasts

 uccessfulauth krblastfailedauth krbloginfailedcount

nsds5replicareapactive: 0

nsds5replicaLastUpdateStart: 19700101000000Z

nsds5replicaLastUpdateEnd: 19700101000000Z

nsds5replicaChangesSentSinceStartup:

nsds5replicaLastUpdateStatus: Error (18) Replication error acquiring
replica:

 Incremental update transient warning.  Backing off, will retry update
later.

 (transient warning)

nsds5replicaUpdateInProgress: FALSE

nsds5replicaLastInitStart: 19700101000000Z

nsds5replicaLastInitEnd: 19700101000000Z


# search result

search: 2

result: 0 Success


# numResponses: 2

# numEntries: 1

[root@g-ipa1 ~]#

On Thu, Jun 2, 2022 at 8:01 PM Kathy Zhu wrote:

>  Do you mean /etc/dirsrv/slapd-CORP-NURO-TEAM/dse.ldif file? There is
> no
>  entry with RUV.
>
>  Thanks.
>
>  Kathy.
>
>  On Thu, Jun 2, 2022 at 5:57 PM Kathy Zhu wrote:
>
> > Thanks, Mark, to identify and explain the issue.
> >
> > Could someone pass a document or article how to apply the fix/work
> > around? Many thanks!
> >
> > Kathy.
> >
> > On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds <mareynol(a)redhat.com&gt;
> > wrote:
> >
> >>
> >> On 6/2/22 1:38 PM, Rob Crittenden wrote:
> >> > Kathy Zhu via FreeIPA-users wrote:
> >> >> Hi Team,
> >> >>
> >> >> We upgraded our Centos 7 IPA masters to the latest:
> >> >>
> >> >> CentOS Linux release 7.9.2009 (Core)
> >> >>
> >> >> *ipa*-server.x86_64                      4.6.8-5.el7.centos.10
> >> >>
> >> >> *389-ds*-base.x86_64                     1.3.10.2-15.el7_9
> >> >>
> >> >> *389-ds*-base-libs.x86_64                1.3.10.2-15.el7_9
> >> >>
> >> >> *389-ds*-base-snmp.x86_64                1.3.10.2-15.el7_9
> >> >>
> >> >> *slapi*-nis.x86_64                       0.56.5-3.el7_9
> >> >>
> >> >>
> >> >> After that, 8 of 10 masters had replication issues. After
> >> >> reinitializing, 2 of them are still having issues. They can accept
> >> >> replication from other masters but their own changes can not be
> >> >> replicated to others.
> >> >>
> >> >>
> >> >> Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
> >> >>
> >> >>
> >> >> [01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin -
> >> >> send_updates -
> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> >> (dc2-ipa1:389):
> >> >> Data required to update replica has been purged from the changelog. If
> >> >> the error persists the replica must be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:03.396330801 -0700] - ERR -
> >> >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> >> (dc3-ipa1:389) -
> >> >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> >> >> changelog (DB rc=-30988). If replication stops, the consumer may need
> >> to
> >> >> be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin -
> >> >> changelog program - repl_plugin_name_cl -
> >> >> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> >> (dc3-ipa1:389):
> >> >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
> >> purged
> >> >>
> >> >> [01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin -
> >> >> send_updates -
> agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc3-ipa1.example.com>"
> >> (dc3-ipa1:389):
> >> >> Data required to update replica has been purged from the changelog. If
> >> >> the error persists the replica must be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:04.411599251 -0700] - ERR -
> >> >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> >> >> <http://dc1-ipa1.example.com-to-ipa0.example.com>" (ipa0:389)
> -
> >> >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> >> >> changelog (DB rc=-30988). If replication stops, the consumer may need
> >> to
> >> >> be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin -
> >> >> changelog program - repl_plugin_name_cl -
> >> >> agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> >> >> <http://dc1-ipa1.example.com-to-ipa0.example.com>"
> (ipa0:389): CSN
> >> >> 627e26a50005001d0000 not found, we aren't as up to date, or we
> purged
> >> >>
> >> >> [01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin -
> >> >> send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com
> >> >> <http://dc1-ipa1.example.com-to-ipa0.example.com>"
> (ipa0:389): Data
> >> >> required to update replica has been purged from the changelog. If the
> >> >> error persists the replica must be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:05.482898290 -0700] - ERR -
> >> >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> >> (dc2-ipa1:389) -
> >> >> clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the
> >> >> changelog (DB rc=-30988). If replication stops, the consumer may need
> >> to
> >> >> be reinitialized.
> >> >>
> >> >> [01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin -
> >> >> changelog program - repl_plugin_name_cl -
> >> >> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> >> (dc2-ipa1:389):
> >> >> CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
> >> purged
> >> >>
> >> >> [01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin -
> >> >> send_updates -
> agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com
> >> >> <http://dc1-ipa1.example.com-to-dc2-ipa1.example.com>"
> >> (dc2-ipa1:389):
> >> >> Data required to update replica has been purged from the changelog. If
> >> >> the error persists the replica must be reinitialized.
> >> >>
> >> >>
> >> >> Note, those messages are after being reinitialized.
> >> >>
> >> >>
> >> >> Any idea what's wrong here?
> >> > I'm not sure, cc'ing one of the 389ds developers.
> >> >
> >> > Mark, any ideas?
> >>
> >> Looks like https://github.com/389ds/389-ds-base/issues/5098
> >>
> >> This was fixed fairly recently, and it has not been added to any RHEL
> >> 7.9 builds yet.
> >>
> >> Pierre worked on this, but I think the issue happens when the LDIF used
> >> to init replication has the RUV entry as the first entry in the LDIF
> >> file.  If it is moved to the end of the file I think it will fix the
> >> init issue.  So don't do an online init.  Export the database with
> >> replication data (-r) from a good supplier, edit the ldif file and make
> >> sure the RUV tombstone entry is the last entry in the file, then import
> >> it on the consumer.
> >>
> >> HTH,
> >> Mark
> >>
> >> >
> >> > rob
> >> >
> >> --
> >> Directory Server Development Team
> >>
> >>