Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn= dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn= dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn= dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn= dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn= dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn= dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
Thanks.
Kathy.
Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
I'm not sure, cc'ing one of the 389ds developers.
Mark, any ideas?
rob
On 6/2/22 1:38 PM, Rob Crittenden wrote:
Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
I'm not sure, cc'ing one of the 389ds developers.
Mark, any ideas?
Looks like https://github.com/389ds/389-ds-base/issues/5098
This was fixed fairly recently, and it has not been added to any RHEL 7.9 builds yet.
Pierre worked on this, but I think the issue happens when the LDIF used to init replication has the RUV entry as the first entry in the LDIF file. If it is moved to the end of the file I think it will fix the init issue. So don't do an online init. Export the database with replication data (-r) from a good supplier, edit the ldif file and make sure the RUV tombstone entry is the last entry in the file, then import it on the consumer.
HTH, Mark
rob
Thanks, Mark, to identify and explain the issue.
Could someone pass a document or article how to apply the fix/work around? Many thanks!
Kathy.
On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds mareynol@redhat.com wrote:
On 6/2/22 1:38 PM, Rob Crittenden wrote:
Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
I'm not sure, cc'ing one of the 389ds developers.
Mark, any ideas?
Looks like https://github.com/389ds/389-ds-base/issues/5098
This was fixed fairly recently, and it has not been added to any RHEL 7.9 builds yet.
Pierre worked on this, but I think the issue happens when the LDIF used to init replication has the RUV entry as the first entry in the LDIF file. If it is moved to the end of the file I think it will fix the init issue. So don't do an online init. Export the database with replication data (-r) from a good supplier, edit the ldif file and make sure the RUV tombstone entry is the last entry in the file, then import it on the consumer.
HTH, Mark
rob
-- Directory Server Development Team
Do you mean /etc/dirsrv/slapd-CORP-NURO-TEAM/dse.ldif file? There is no entry with RUV.
Thanks.
Kathy.
On Thu, Jun 2, 2022 at 5:57 PM Kathy Zhu wrote:
Thanks, Mark, to identify and explain the issue.
Could someone pass a document or article how to apply the fix/work around? Many thanks!
Kathy.
On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds mareynol@redhat.com wrote:
On 6/2/22 1:38 PM, Rob Crittenden wrote:
Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389)
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com" (dc3-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389)
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com" (dc2-ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
I'm not sure, cc'ing one of the 389ds developers.
Mark, any ideas?
Looks like https://github.com/389ds/389-ds-base/issues/5098
This was fixed fairly recently, and it has not been added to any RHEL 7.9 builds yet.
Pierre worked on this, but I think the issue happens when the LDIF used to init replication has the RUV entry as the first entry in the LDIF file. If it is moved to the end of the file I think it will fix the init issue. So don't do an online init. Export the database with replication data (-r) from a good supplier, edit the ldif file and make sure the RUV tombstone entry is the last entry in the file, then import it on the consumer.
HTH, Mark
rob
-- Directory Server Development Team
I found the error, will error number 18 help to fix the problem? Thanks.
Kathy.
[root@g-ipa1 ~]# ldapsearch -D "cn=directory manager" -W -b "cn= g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config"
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# g-ipa1.example.com-to-h-ipa1.example.com, replica, dc\3Dexample\2Cdc\3
Dcom, mapping tree, config
dn: cn=g-ipa1.example.com-to-h-ipa1.example.com,cn=replica,cn=dc\3Dc
orp\2Cdc\3Dnuro\2Cdc\3Dteam,cn=mapping tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: g-ipa1.example.com-to-h-ipa1.example.com
nsDS5ReplicaHost: h-ipa1.example.com
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=example,dc=com
description: g-ipa1.example.com to h-ipa1.example.com
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
uccessfulauth krblastfailedauth krbloginfailedcount
nsds5replicareapactive: 0
nsds5replicaLastUpdateStart: 19700101000000Z
nsds5replicaLastUpdateEnd: 19700101000000Z
nsds5replicaChangesSentSinceStartup:
nsds5replicaLastUpdateStatus: Error (18) Replication error acquiring replica:
Incremental update transient warning. Backing off, will retry update later.
(transient warning)
nsds5replicaUpdateInProgress: FALSE
nsds5replicaLastInitStart: 19700101000000Z
nsds5replicaLastInitEnd: 19700101000000Z
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
[root@g-ipa1 ~]#
On Thu, Jun 2, 2022 at 8:01 PM Kathy Zhu wrote:
Do you mean /etc/dirsrv/slapd-CORP-NURO-TEAM/dse.ldif file? There is no entry with RUV.
Thanks.
Kathy.
On Thu, Jun 2, 2022 at 5:57 PM Kathy Zhu wrote:
Thanks, Mark, to identify and explain the issue.
Could someone pass a document or article how to apply the fix/work around? Many thanks!
Kathy.
On Thu, Jun 2, 2022 at 12:38 PM Mark Reynolds mareynol@redhat.com wrote:
On 6/2/22 1:38 PM, Rob Crittenden wrote:
Kathy Zhu via FreeIPA-users wrote:
Hi Team,
We upgraded our Centos 7 IPA masters to the latest:
CentOS Linux release 7.9.2009 (Core)
*ipa*-server.x86_64 4.6.8-5.el7.centos.10
*389-ds*-base.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-libs.x86_64 1.3.10.2-15.el7_9
*389-ds*-base-snmp.x86_64 1.3.10.2-15.el7_9
*slapi*-nis.x86_64 0.56.5-3.el7_9
After that, 8 of 10 masters had replication issues. After reinitializing, 2 of them are still having issues. They can accept replication from other masters but their own changes can not be replicated to others.
Here are the logs in /var/log/dirsrv/slapd-EXAMPLE-COM/errors:
[01/Jun/2022:21:53:02.324756398 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389):
Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:03.396330801 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com"
(dc3-ipa1:389) -
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:03.396502102 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com"
(dc3-ipa1:389):
CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:03.396694568 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc3-ipa1.example.com http://dc1-ipa1.example.com-to-dc3-ipa1.example.com"
(dc3-ipa1:389):
Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:04.411599251 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389) - clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:04.411753186 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): CSN 627e26a50005001d0000 not found, we aren't as up to date, or we purged
[01/Jun/2022:21:53:04.411893312 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-ipa0.example.com http://dc1-ipa1.example.com-to-ipa0.example.com" (ipa0:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
[01/Jun/2022:21:53:05.482898290 -0700] - ERR - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389) -
clcache_load_buffer - Can't locate CSN 627e26a50005001d0000 in the changelog (DB rc=-30988). If replication stops, the consumer may need
to
be reinitialized.
[01/Jun/2022:21:53:05.483231727 -0700] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389):
CSN 627e26a50005001d0000 not found, we aren't as up to date, or we
purged
[01/Jun/2022:21:53:05.483483005 -0700] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=dc1-ipa1.example.com-to-dc2-ipa1.example.com http://dc1-ipa1.example.com-to-dc2-ipa1.example.com"
(dc2-ipa1:389):
Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Note, those messages are after being reinitialized.
Any idea what's wrong here?
I'm not sure, cc'ing one of the 389ds developers.
Mark, any ideas?
Looks like https://github.com/389ds/389-ds-base/issues/5098
This was fixed fairly recently, and it has not been added to any RHEL 7.9 builds yet.
Pierre worked on this, but I think the issue happens when the LDIF used to init replication has the RUV entry as the first entry in the LDIF file. If it is moved to the end of the file I think it will fix the init issue. So don't do an online init. Export the database with replication data (-r) from a good supplier, edit the ldif file and make sure the RUV tombstone entry is the last entry in the file, then import it on the consumer.
HTH, Mark
rob
-- Directory Server Development Team
freeipa-users@lists.fedorahosted.org
-
Kathy Zhu -
Mark Reynolds -
Rob Crittenden