I have tried (providing password and our actual domain name scrubbed below):
clean dangling ruvs using ``` root@ipa01.mgmt ~ # ipa-replica-manage clean-dangling-ruv unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 No dangling RUVs found ``` cleaning that ruv specifically: ``` root@ipa01 ~ # ipa-replica-manage --force --cleanup clean-ruv 27 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 Replica ID 27 not found ``` ldapmodify - which runs without error and no change: ``` root@ipa01 ~ # cat clean-27.ldif dn: cn=clean 27,cn=cleanallruv,cn=tasks,cn=config changetype: add objectclass: top objectclass: extensibleObject replica-base-dn: dc=ipa,dc=example,dc=net replica-id: 27 cn: clean 27
root@ipa01 ~ # ldapmodify -D "cn=directory manager" -f clean-27.ldif adding new entry "cn=clean 27,cn=cleanallruv,cn=tasks,cn=config" ```
Here's the topology (you can see ipa and idm nodes - working to eventually replace the CentOS 7 ipa nodes with RHEL8 idm nodes - but this replica 27 has been around ... forever as far as I'm concerned): ``` root@ipa01 ~ # ipa topologysegment-find domain ------------------ 9 segments matched ------------------ Segment name: idm01.example.net-to-idm02.example.net Left node: idm01.example.net Right node: idm02.example.net Connectivity: both
Segment name: idm01.example.net-to-ipa01.example.net Left node: idm01.example.net Right node: ipa01.example.net Connectivity: both
Segment name: idm02.example.net-to-idm04.example.net Left node: idm02.example.net Right node: idm04.example.net Connectivity: both
Segment name: idm03.example.net-to-idm01.example.net Left node: idm03.example.net Right node: idm01.example.net Connectivity: both
Segment name: idm04.example.net-to-idm03.example.net Left node: idm04.example.net Right node: idm03.example.net Connectivity: both
Segment name: ipa01.example.net-to-ipa02.example.net Left node: ipa01.example.net Right node: ipa02.example.net Connectivity: both
Segment name: ipa01.example.net-to-ipa03.example.net Left node: ipa01.example.net Right node: ipa03.example.net Connectivity: both
Segment name: ipa02.example.net-to-ipa04.example.net Left node: ipa02.example.net Right node: ipa04.example.net Connectivity: both
Segment name: ipa03.example.net-to-ipa04.example.net Left node: ipa03.example.net Right node: ipa04.example.net Connectivity: both ---------------------------- Number of entries returned 9 ---------------------------- root@ipa01 ~ # ipa topologysegment-find ca ------------------ 9 segments matched ------------------ Segment name: idm01.example.net-to-idm02.example.net Left node: idm01.example.net Right node: idm02.example.net Connectivity: both
Segment name: idm01.example.net-to-ipa01.example.net Left node: idm01.example.net Right node: ipa01.example.net Connectivity: both
Segment name: idm02.example.net-to-idm04.example.net Left node: idm02.example.net Right node: idm04.example.net Connectivity: both
Segment name: idm03.example.net-to-idm01.example.net Left node: idm03.example.net Right node: idm01.example.net Connectivity: both
Segment name: idm04.example.net-to-idm03.example.net Left node: idm04.example.net Right node: idm03.example.net Connectivity: both
Segment name: ipa01.example.net-to-ipa02.example.net Left node: ipa01.example.net Right node: ipa02.example.net Connectivity: both
Segment name: ipa01.example.net-to-ipa03.example.net Left node: ipa01.example.net Right node: ipa03.example.net Connectivity: both
Segment name: ipa02.example.net-to-ipa04.example.net Left node: ipa02.example.net Right node: ipa04.example.net Connectivity: both
Segment name: ipa03.example.net-to-ipa04.example.net Left node: ipa03.example.net Right node: ipa04.example.net Connectivity: both ---------------------------- Number of entries returned 9 ---------------------------- ``` I've even gone off the rails and manually deleted the nsds50ruv and nsruvReplicaLastModified entries for these on sub entries of cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config (where I found *something* referencing it).
I've pored over the google results for this - including some from this mailing list - the unsuccessful threads seem to simply die from lack of response, while the successful ones found a segment they could delete.
Any help would be appreciated.
Thanks, - chris
Hi,
On Wed, Feb 12, 2025 at 1:06 AM Chris Jacobs via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
I have tried (providing password and our actual domain name scrubbed below):
Which ipa / 389ds version is installed?
clean dangling ruvs using
root@ipa01.mgmt ~ # ipa-replica-manage clean-dangling-ruv unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 No dangling RUVs foundcleaning that ruv specifically:
root@ipa01 ~ # ipa-replica-manage --force --cleanup clean-ruv 27 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 Replica ID 27 not foundldapmodify - which runs without error and no change:
root@ipa01 ~ # cat clean-27.ldif dn: cn=clean 27,cn=cleanallruv,cn=tasks,cn=config changetype: add objectclass: top objectclass: extensibleObject replica-base-dn: dc=ipa,dc=example,dc=net replica-id: 27 cn: clean 27 root@ipa01 ~ # ldapmodify -D "cn=directory manager" -f clean-27.ldif adding new entry "cn=clean 27,cn=cleanallruv,cn=tasks,cn=config"This ldapmodify creates a task that runs in the background. You would need
to check if it completed, either by looking at the logs in /var/log/dirsrv/slapd-XXX/errors or by performing a ldapsearch on this clean allruv task. Look for the attributes nstaskstatus and nsTaskExitCode. You can find more information here https://www.port389.org/docs/389ds/howto/howto-cleanruv.html#cleanallruv.
HTH, flo
Here's the topology (you can see ipa and idm nodes - working to eventually
replace the CentOS 7 ipa nodes with RHEL8 idm nodes - but this replica 27 has been around ... forever as far as I'm concerned):
root@ipa01 ~ # ipa topologysegment-find domain ------------------ 9 segments matched ------------------ Segment name: idm01.example.net-to-idm02.example.net Left node: idm01.example.net Right node: idm02.example.net Connectivity: both Segment name: idm01.example.net-to-ipa01.example.net Left node: idm01.example.net Right node: ipa01.example.net Connectivity: both Segment name: idm02.example.net-to-idm04.example.net Left node: idm02.example.net Right node: idm04.example.net Connectivity: both Segment name: idm03.example.net-to-idm01.example.net Left node: idm03.example.net Right node: idm01.example.net Connectivity: both Segment name: idm04.example.net-to-idm03.example.net Left node: idm04.example.net Right node: idm03.example.net Connectivity: both Segment name: ipa01.example.net-to-ipa02.example.net Left node: ipa01.example.net Right node: ipa02.example.net Connectivity: both Segment name: ipa01.example.net-to-ipa03.example.net Left node: ipa01.example.net Right node: ipa03.example.net Connectivity: both Segment name: ipa02.example.net-to-ipa04.example.net Left node: ipa02.example.net Right node: ipa04.example.net Connectivity: both Segment name: ipa03.example.net-to-ipa04.example.net Left node: ipa03.example.net Right node: ipa04.example.net Connectivity: both ---------------------------- Number of entries returned 9 ---------------------------- root@ipa01 ~ # ipa topologysegment-find ca ------------------ 9 segments matched ------------------ Segment name: idm01.example.net-to-idm02.example.net Left node: idm01.example.net Right node: idm02.example.net Connectivity: both Segment name: idm01.example.net-to-ipa01.example.net Left node: idm01.example.net Right node: ipa01.example.net Connectivity: both Segment name: idm02.example.net-to-idm04.example.net Left node: idm02.example.net Right node: idm04.example.net Connectivity: both Segment name: idm03.example.net-to-idm01.example.net Left node: idm03.example.net Right node: idm01.example.net Connectivity: both Segment name: idm04.example.net-to-idm03.example.net Left node: idm04.example.net Right node: idm03.example.net Connectivity: both Segment name: ipa01.example.net-to-ipa02.example.net Left node: ipa01.example.net Right node: ipa02.example.net Connectivity: both Segment name: ipa01.example.net-to-ipa03.example.net Left node: ipa01.example.net Right node: ipa03.example.net Connectivity: both Segment name: ipa02.example.net-to-ipa04.example.net Left node: ipa02.example.net Right node: ipa04.example.net Connectivity: both Segment name: ipa03.example.net-to-ipa04.example.net Left node: ipa03.example.net Right node: ipa04.example.net Connectivity: both ---------------------------- Number of entries returned 9 ----------------------------I've even gone off the rails and manually deleted the nsds50ruv and nsruvReplicaLastModified entries for these on sub entries of cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config (where I found *something* referencing it).
I've pored over the google results for this - including some from this mailing list - the unsuccessful threads seem to simply die from lack of response, while the successful ones found a segment they could delete.
Any help would be appreciated.
Thanks,
- chris
-- _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
I use ldapmodify again to load that site's version of the cleanallruv task (it's missing changetype: add - not very encouraging), and the status shows it completed successfully, and yet it remains. Ditto for the cleanruv task (ran on all the machines):
Applied ldifs (former on just one, latter on all): ``` dn: cn=clean 27, cn=cleanallruv, cn=tasks, cn=config changetype: add objectclass: extensibleObject replica-base-dn: dc=ipa,dc=example,dc=net replica-id: 27 replica-force-cleaning: no cn: clean 27
dn: cn=dc\3Dipa\2Cdc\example\2Cdc\3Dnet,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV27 ``` Logs: ``` [12/Feb/2025:19:01:00.535438472 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Initiating CleanAllRUV Task... [12/Feb/2025:19:01:00.537275387 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Retrieving maxcsn... [12/Feb/2025:19:01:00.780146144 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Found maxcsn (00000000000000000000) [12/Feb/2025:19:01:00.797466366 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaning rid (27)... [12/Feb/2025:19:01:00.799644572 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Waiting to process all the updates from the deleted replica... [12/Feb/2025:19:01:00.802433994 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Waiting for all the replicas to be online... [12/Feb/2025:19:01:01.141760326 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Waiting for all the replicas to receive all the deleted replica updates... [12/Feb/2025:19:01:01.474804904 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Sending cleanAllRUV task to all the replicas... [12/Feb/2025:19:01:01.809894586 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaning local ruv's... [12/Feb/2025:19:01:02.816896586 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Waiting for all the replicas to be cleaned... [12/Feb/2025:19:01:03.385360109 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Waiting for all the replicas to finish cleaning... [12/Feb/2025:19:01:03.866052570 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Original task deletes Keep alive entry (27). [12/Feb/2025:19:01:03.884521429 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): No Keep-Alive entry to remove (cn=repl keep alive 27,dc=ipa,dc=nweacolo,dc=pvt) [12/Feb/2025:19:01:03.887912359 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaning agmt... [12/Feb/2025:19:01:05.904619389 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaning agmt... [12/Feb/2025:19:01:06.157516660 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaning agmt... [12/Feb/2025:19:01:06.466607314 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Cleaned replication agreements. [12/Feb/2025:19:01:06.469788291 +0000] - INFO - NSMMReplicationPlugin - CleanAllRUV Task (rid 27): Successfully cleaned rid(27) ``` clean-dangling-run output: ``` unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 unable to decode: {replica 27} 60771c010007001b0000 60771c010007001b0000 No dangling RUVs found ```
Problem remains.
Thanks, - chris
Anyone? I'd really like it if this thread didn't die out the same way all the previous ones have where the usual tooling fails to do the job and then the thread goes crickets. There's something, somewhere causing these to exist even when deleted manually - restart the servers and the nsds50ruv and nsruvReplicaLastModified entries return.
Thanks, - chris
On 2/14/25 6:44 PM, Chris Jacobs via FreeIPA-users wrote:
Anyone? I'd really like it if this thread didn't die out the same way all the previous ones have where the usual tooling fails to do the job and then the thread goes crickets. There's something, somewhere causing these to exist even when deleted manually - restart the servers and the nsds50ruv and nsruvReplicaLastModified entries return.
As previously asked, what rpm version of 389-ds-base are you running? There have been enhancements and fixes to the cleaning task over the years.
After running the cleanallruv task can you please run this ldapsearch on the Directory Server:
$ ldapsearch -xLLL -H ldap://localhost:389 -D "cn=directory manager" -W -b "YOUR SUFFIX HERE" '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
If you still see that RUV element from the search above you can try using the force cleaning option. If you are on a newer of DS you can try running the cleaning task using the force option via DS CLI, or modify your ldapmodify LDIF to set force cleaning to "yes"
# dsconf slapd-YOUR_DS_INSTANCE_NAME repl-tasks cleanallruv --replica-id 27 --suffix YOUR_SUFFIX --force-cleaning
Now part of the cleaning process happens in a separate thread where it cleans the replication changelog of any updates from that old replica. If your changelog is large it could take some time to finish. If you restart the server before it ends it "could" pollute the RUV with that old replica id. Usually the changelog purging runs pretty quickly (less than a minute). Regardless a DS restart is /not/ required for any of this to work.
It's also important to make sure you don't have any lingering replication agreements to that old/deleted replica, and that old replica should have been completely removed from the deployment. The RUV could also get re-polluted if you have an old IPA server in the mix that does not support cleanallruv (like an old RHEL 6 server).
HTH,
Mark
Thanks,
- chris
389-ds version info: ``` 389-ds-base-libs-1.4.3.39-9.module+el8.10.0+22552+36a96aa1.x86_64 389-ds-base-1.4.3.39-9.module+el8.10.0+22552+36a96aa1.x86_64 ``` ldapsearch results: ``` dn: cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dnet,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 573209f4000000040000 nsds50ruv: {replica 151 ldap://idm01.mgmt.example.net:389} 67917209000100970000 67b5042e000000970000 nsds50ruv: {replica 60 ldap://ipa01.mgmt.example.net:389} 649cea6f37d6003c0000 67b5042d0021003c0000 nsds50ruv: {replica 124 ldap://ipa02.mgmt.example.net:389} 67732ee90000007c0000 67b5042b0014007c0000 nsds50ruv: {replica 121 ldap://ipa04.mgmt.example.net:389} 67630af8000000790000 67b50415002000790000 nsds50ruv: {replica 70 ldap://ipa03.mgmt.example.net:389} 65a9f083000000460000 67b5041d002400460000 nsds50ruv: {replica 153 ldap://idm03.mgmt.example.net:389} 67a1528c000100990000 67b5028c000000990000 nsds50ruv: {replica 156 ldap://idm04.mgmt.example.net:389} 67a15d6d0001009c0000 67b4fea40000009c0000 nsds50ruv: {replica 160 ldap://idm02.mgmt.example.net:389} 67a26d03000000a00000 67b50272000000a00000 ``` however clean-dangling-ruv still sees that replica 27 entry.
I then ran those dsconf commands for both suffixes and... it's gone!
That's the magic sauce I needed, much appreciated Mark! - chris
On 2/18/25 5:07 PM, Chris Jacobs via FreeIPA-users wrote:
389-ds version info:
389-ds-base-libs-1.4.3.39-9.module+el8.10.0+22552+36a96aa1.x86_64 389-ds-base-1.4.3.39-9.module+el8.10.0+22552+36a96aa1.x86_64ldapsearch results:
dn: cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dnet,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 573209f4000000040000 nsds50ruv: {replica 151 ldap://idm01.mgmt.example.net:389} 67917209000100970000 67b5042e000000970000 nsds50ruv: {replica 60 ldap://ipa01.mgmt.example.net:389} 649cea6f37d6003c0000 67b5042d0021003c0000 nsds50ruv: {replica 124 ldap://ipa02.mgmt.example.net:389} 67732ee90000007c0000 67b5042b0014007c0000 nsds50ruv: {replica 121 ldap://ipa04.mgmt.example.net:389} 67630af8000000790000 67b50415002000790000 nsds50ruv: {replica 70 ldap://ipa03.mgmt.example.net:389} 65a9f083000000460000 67b5041d002400460000 nsds50ruv: {replica 153 ldap://idm03.mgmt.example.net:389} 67a1528c000100990000 67b5028c000000990000 nsds50ruv: {replica 156 ldap://idm04.mgmt.example.net:389} 67a15d6d0001009c0000 67b4fea40000009c0000 nsds50ruv: {replica 160 ldap://idm02.mgmt.example.net:389} 67a26d03000000a00000 67b50272000000a00000however clean-dangling-ruv still sees that replica 27 entry.
I then ran those dsconf commands for both suffixes and... it's gone!
That's the magic sauce I needed, much appreciated Mark!
Glad it worked!
Mark
- chris
freeipa-users@lists.fedorahosted.org
-
Chris Jacobs -
Florence Blanc-Renaud -
Mark Reynolds