Help with Ghost Replica removal please.

[Nicholas Cross]

I am on a long journey upgrading from a mixture of different versions of
freeipa to the lastest and i am almost there, currently on 4.10.
After removing some older replicas yesterday, a new replica had a fault and
we had to restart dirsrv.
When it came back up we saw we have ghost replicas across the whole estate
6x replicas in two locations.
Issues are reported via `cipa`

Digging into the tombstone records, i find there are not the usual
tombstone replica records but more like the below.

My question is, how do i remove these redundant records?  As I believe this
is what `cipa` is counting to report the errors.
(marked with    <<<<<<<<<<<<<<)

The usual things do not work:

# ipa-replica-manage -p $pass clean-ruv 15
Replica ID 15 not found


$ cat remove_ruv.ldif
dn: cn=clean 15,cn=cleanallruv,cn=tasks,cn=config
changetype: add
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=ad,dc=companyx,dc=fm
replica-id: 15
cn: clean 15

ldapmodify -Y GSSAPI -f remove_ruv.ldif



Thanks in advance.
Nick


$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
"dc=ad,dc=companyx,dc=fm"
'(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
nsDS5ReplicaBindDnGroupCheckInterval: 60
nsDS5ReplicaId: 56
nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
nsDS5ReplicaType: 3
nsState:: OAAAAAAAAABSPEJkAAAAAAAAAAAAAAAA7AAAAAAAAAAGAAAAAAAAAA==
nsds5ReplicaBackoffMax: 300
nsds5ReplicaLegacyConsumer: off
nsds5ReplicaReleaseTimeout: 60
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 5d9e2076000000040000
nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389}
63ece66f000000380000 64423d4e000000380000
nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389}
63dbcc200001002e0000 64423cf6000f002e0000
nsds50ruv: {replica 21 ldap://etcd0dc.ad.companyx.fm:389}
5f2d4bc1000000150000 64319ec3276200150000
nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389}
63ea4e54000100300000 6442397e000a00300000
nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389}
643d03280001003a0000 644232c00001003a0000
nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389}
643d19680001003c0000 644232660004003c0000
nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389}
643d491e0001003e0000 644233340000003e0000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm
;389;62;644238c9000000380000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm
;389;48;644238c9000000380000
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-etcd2dc.ad.companyx.fm;etcd2dc.ad.companyx.fm;389;unavailable;64423cf6000f002e0000
     <<<<<<<<<<<<<<
nsds5agmtmaxcsn: dc=ad,dc=companyx,dc=fm;
ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm
;389;46;644238c9000000380000
nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389}
64423c62
nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389}
64423c0c
nsruvReplicaLastModified: {replica 21 ldap://etcd0dc.ad.companyx.fm:389}
00000000
nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389}
64423894
nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389}
644231da
nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389}
6442317a
nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389}
64423249
nsruvReplicaLastModified: {replica 12} 644180f7  
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 15} 644180f7  
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 25} 644180f7  
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 23} 644180f7  
<<<<<<<<<<<<<<
nsruvReplicaLastModified: {replica 40} 644180f7  
<<<<<<<<<<<<<<
nsds5ReplicaChangeCount: 734077
nsds5replicareapactive: 0

[Nicholas Cross]

I think i have a handle on this now.

There are a number of issues that i am now aware of.

1. old replication agreement to oldbox1 on newbox6

2. corrupt RUVs, giving the impression of Ghost Replicas.

For #1 i normally can delete these fine with a ldap command.  BUT!  running this crashes
the dirsrv service.

ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x
"cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
tree,cn=config"


Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server
AD-DICE-FM....
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Failed with
result 'signal'.
Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Main process
exited, code=killed, status=6/ABRT
Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd:
ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion
`prim_op' failed.


For #2 , this issue of not being able to remove the replication agreement, stops the
removal of the corrupt RUVs.

As you can see i have tried to kick off some RUV removals but they are failing as not all
replicas are online. (but they dont exist as #1)

$ ipa-replica-manage list-clean-ruv -p $pass
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
CLEANALLRUV tasks
RID 12: Not all replicas online, retrying in 40 seconds...
RID restarted-2658134: Not all replicas online, retrying in 320 seconds...
RID restarted-2658136: Not all replicas online, retrying in 320 seconds...

No abort CLEANALLRUV tasks running



So, more questions really, 
	Why is the ldapdelete crashing the service?
	How do i fix it?
	
thanks,
Nick

[Rob Crittenden]

Nicholas Cross via FreeIPA-users wrote:
>  I think i have a handle on this now.
>  
>  There are a number of issues that i am now aware of.
>  
>  1. old replication agreement to oldbox1 on newbox6
>  
>  2. corrupt RUVs, giving the impression of Ghost Replicas.
>  
>  For #1 i normally can delete these fine with a ldap command.  BUT!  running this crashes
> the dirsrv service.
>  
>  ldapdelete -D "cn=Directory Manager" -w $pwd -p 389 -h localhost -x
> "cn=newbox6.ad.dice.fm-to-oldbox1.ad.dice.fm,cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping
> tree,cn=config"
>  
>  
>  Apr 21 11:35:48 newbox6.ad.dice.fm systemd[1]: Starting 389 Directory Server
> AD-DICE-FM....
>  Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Failed with
> result 'signal'.
>  Apr 21 11:35:11 newbox6.ad.dice.fm systemd[1]: dirsrv(a)AD-DICE-FM.service: Main process
> exited, code=killed, status=6/ABRT
>  Apr 21 11:35:11 newbox6.ad.dice.fm ns-slapd[2934110]: ns-slapd:
> ldap/servers/plugins/sync/sync_persist.c:234: sync_update_persist_op: Assertion
> `prim_op' failed.
>  
>  
>  For #2 , this issue of not being able to remove the replication agreement, stops the
> removal of the corrupt RUVs.
>  
>  As you can see i have tried to kick off some RUV removals but they are failing as not all
> replicas are online. (but they dont exist as #1)
>  
>  $ ipa-replica-manage list-clean-ruv -p $pass
>  ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
> denied: '/var/log/ipa/cli.log'
>  CLEANALLRUV tasks
>  RID 12: Not all replicas online, retrying in 40 seconds...
>  RID restarted-2658134: Not all replicas online, retrying in 320 seconds...
>  RID restarted-2658136: Not all replicas online, retrying in 320 seconds...
>  
>  No abort CLEANALLRUV tasks running
>  
>  
>  
>  So, more questions really, 
>  	Why is the ldapdelete crashing the service? 
Can you install the debuginfo packages and get a stack trace? I'm sure
the 389-ds developers would like to take a look at it. It may already
have been addressed. You haven't provided any version or distro information.

>  	How do i fix it? 
If you shut down dirsrv then you can edit
/etc/dirsrv/slapd-REALM/dse.ldif and manually remove the replication
agreement.

But you have to shut down dirsrv first.

rob

[Nicholas Cross]

Thanks for the tips.  I'll run the trace on Monday.

Also i'll edit the ldif Monday too.

Distro: alma9

# rpm -qa | grep ipa | sort
almalinux-logos-ipa-90.5.1-1.1.el9.noarch
ipa-client-4.10.0-8.el9_1.x86_64
ipa-client-common-4.10.0-8.el9_1.noarch
ipa-common-4.10.0-8.el9_1.noarch
ipa-healthcheck-0.9-9.el9.noarch
ipa-healthcheck-core-0.9-9.el9.noarch
ipa-selinux-4.9.8-7.el9_0.noarch
ipa-server-4.10.0-8.el9_1.x86_64
ipa-server-common-4.10.0-8.el9_1.noarch
ipa-server-dns-4.10.0-8.el9_1.noarch
libipa_hbac-2.7.3-4.el9_1.3.x86_64
python3-ipaclient-4.10.0-8.el9_1.noarch
python3-ipalib-4.10.0-8.el9_1.noarch
python3-ipaserver-4.10.0-8.el9_1.noarch
python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
sssd-ipa-2.7.3-4.el9_1.3.x86_64

[Nicholas Cross]

Shutdown dirsrv
backed up dse.ldif
edited the ldif
restarted dirsrv

replication agreement came back!

Is it being sync'ed from some where else? another file?

[Rob Crittenden]

Nicholas Cross via FreeIPA-users wrote:
>  Shutdown dirsrv
>  backed up dse.ldif
>  edited the ldif
>  restarted dirsrv
>  
>  replication agreement came back!
>  
>  Is it being sync'ed from some where else? another file? 
It sounds like dirsrv was still running if values in cn=config were
restored. It writes the file on exit. This subtree is not replicated.

rob

[Nicholas Cross]

Tested this again making sure that dirsrv is not running and the replica record is back.

I am obviously doing something wrong.  My steps are below.  I appreciate your time on
this.



#
# check dirsrv is currently running
#
[root@ipa006 ~]# ps aux | grep dirsrv
dirsrv   3221639 31.4  5.4 2418488 883856 ?      Ssl  Apr24 322:04 /usr/sbin/ns-slapd -D
/etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid
root     3281205  0.0  0.0   6412  2204 pts/2    S+   09:11   0:00 grep --color=auto
dirsrv

#
# shutdown dirsrv
#
[root@ipa006 ~]# time systemctl stop dirsrv(a)AD-companyx-FM.service

real	10m0.130s
user	0m0.009s
sys	0m0.012s

#
# check dirsrv is not running 1
#
[root@ipa006 ~]# ps aux | grep dirsrv
root     3282962  0.0  0.0   6412  2244 pts/2    S+   09:47   0:00 grep --color=auto
dirsrv

#
# check dirsrv is not running 2
#
[root@ipa006 slapd-AD-companyx-FM]# ipactl status
Directory Service: STOPPED
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
1 service(s) are not running

#
# go to right folder
#
[root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/


#
# make a backup just incase
#
[root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23

#
# edit ldif
#
[root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif


#
# remove this record. Hoping its the right thing to do.
#
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
 ce\2Cdc\3Dfm,cn=mapping tree,cn=config
objectClass: nsds5replicationagreement
objectClass: ipaReplTopoManagedAgreement
objectClass: top
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
nsDS5ReplicaPort: 389
nsds5replicaTimeout: 300
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
 ugin
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
  entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
 ternalModifyTimestamp
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
 uccessfulauth krblastfailedauth krbloginfailedcount
creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config
modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config
createTimestamp: 20230425095140Z
modifyTimestamp: 20230425095140Z


#
# check no records exist in dse.ldif
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
[root@ipa006 slapd-AD-companyx-FM]#

[root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv(a)AD-companyx-FM.service

real	0m12.343s
user	0m0.006s
sys	0m0.007s

#
# Look in logs
#
Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325
+0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp -
agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm"
(bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't
contact LDAP server) ()

#
# check dse.ldif again - find entry is back !
#
[root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
dn:
cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm


#
# scratch head and ponder life, the universe and everything
#

[Rob Crittenden]

Nicholas Cross via FreeIPA-users wrote:
>  Tested this again making sure that dirsrv is not running and the
> replica record is back.
>  
>  I am obviously doing something wrong.  My steps are below.  I appreciate your time on
> this.
>  
>  
>  
>  #
>  # check dirsrv is currently running
>  #
>  [root@ipa006 ~]# ps aux | grep dirsrv
>  dirsrv   3221639 31.4  5.4 2418488 883856 ?      Ssl  Apr24 322:04 /usr/sbin/ns-slapd -D
> /etc/dirsrv/slapd-AD-companyx-FM -i /run/dirsrv/slapd-AD-companyx-FM.pid
>  root     3281205  0.0  0.0   6412  2204 pts/2    S+   09:11   0:00 grep --color=auto
> dirsrv
>  
>  #
>  # shutdown dirsrv
>  #
>  [root@ipa006 ~]# time systemctl stop dirsrv(a)AD-companyx-FM.service
>  
>  real	10m0.130s
>  user	0m0.009s
>  sys	0m0.012s
>  
>  #
>  # check dirsrv is not running 1
>  #
>  [root@ipa006 ~]# ps aux | grep dirsrv
>  root     3282962  0.0  0.0   6412  2244 pts/2    S+   09:47   0:00 grep --color=auto
> dirsrv
>  
>  #
>  # check dirsrv is not running 2
>  #
>  [root@ipa006 slapd-AD-companyx-FM]# ipactl status
>  Directory Service: STOPPED
>  krb5kdc Service: RUNNING
>  kadmin Service: RUNNING
>  named Service: RUNNING
>  httpd Service: RUNNING
>  ipa-custodia Service: RUNNING
>  pki-tomcatd Service: RUNNING
>  ipa-otpd Service: RUNNING
>  ipa-dnskeysyncd Service: RUNNING
>  1 service(s) are not running
>  
>  #
>  # go to right folder
>  #
>  [root@ipa006 ~]# cd /etc/dirsrv/slapd-AD-companyx-FM/
>  
>  
>  #
>  # make a backup just incase
>  #
>  [root@ipa006 slapd-AD-companyx-FM]# cp dse.ldif dse.ldif.nickx-25apr23
>  
>  #
>  # edit ldif
>  #
>  [root@ipa006 slapd-AD-companyx-FM]# vi dse.ldif
>  
>  
>  #
>  # remove this record. Hoping its the right thing to do.
>  #
>  dn:
> cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
>   ce\2Cdc\3Dfm,cn=mapping tree,cn=config
>  objectClass: nsds5replicationagreement
>  objectClass: ipaReplTopoManagedAgreement
>  objectClass: top
>  cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
>  nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
>  nsDS5ReplicaPort: 389
>  nsds5replicaTimeout: 300
>  nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
>  description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
>  ipaReplTopoManagedAgreementState: managed agreement - generated by topology pl
>   ugin
>  nsDS5ReplicaTransportInfo: LDAP
>  nsDS5ReplicaBindMethod: SASL/GSSAPI
>  nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial
>    entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
>  nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName in
>   ternalModifyTimestamp
>  nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblasts
>   uccessfulauth krblastfailedauth krbloginfailedcount
>  creatorsName: cn=IPA Topology Configuration,cn=plugins,cn=config
>  modifiersName: cn=IPA Topology Configuration,cn=plugins,cn=config
>  createTimestamp: 20230425095140Z
>  modifyTimestamp: 20230425095140Z
>  
>  
>  #
>  # check no records exist in dse.ldif
>  #
>  [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
>  [root@ipa006 slapd-AD-companyx-FM]#
>  
>  [root@ipa006 slapd-AD-companyx-FM]# time systemctl start dirsrv(a)AD-companyx-FM.service
>  
>  real	0m12.343s
>  user	0m0.006s
>  sys	0m0.007s
>  
>  #
>  # Look in logs
>  #
>  Apr 25 09:51:51 ipa006.ad.companyx.fm ns-slapd[3283119]: [25/Apr/2023:09:51:51.484197325
> +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp -
> agmt="cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm"
> (bad_serverdc:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't
> contact LDAP server) ()
>  
>  #
>  # check dse.ldif again - find entry is back !
>  #
>  [root@ipa006 slapd-AD-companyx-FM]# grep bad_server dse.ldif
>  dn:
> cn=ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm,cn=replica,cn=dc\3Dad\2Cdc\3Ddi
>  cn: ipa006.ad.companyx.fm-to-bad_serverdc.ad.companyx.fm
>  nsDS5ReplicaHost: bad_serverdc.ad.companyx.fm
>  description: ipa006.ad.companyx.fm to bad_serverdc.ad.companyx.fm
>  
>  
>  #
>  # scratch head and ponder life, the universe and everything
>  # 
I have a guess what is happening. The topology is also stored within IPA
itself. I wonder if this is self-healing.

I assume you've ensure that the bad server is not anywhere else inside
of IPA? ipa server-del bad_server? Does its host entry exist?

ipa-replica-manage list -v `hostname` on the server the agreement exists?

rob

[Nicholas Cross]

Ah got it!  Wonderful.

The trick as to run the topologysegement-del on the same server it was on.

It seems i am moving forward with this now - thanks.


#
# To remove the topology segment, which removed the replica agreement
#

#
# Show the bad replication agreement
#

# ipa-replica-manage list -v `hostname`
Directory Manager password:

bad_server.ad.companyx.fm: replica
  last update status: Error (-1) Problem connecting to replica - LDAP error: Can't
contact LDAP server (connection error)
  last update ended: 1970-01-01 00:00:00+00:00
ipa003dc.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
  last update ended: 2023-04-26 06:43:07+00:00
ipa005.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
  last update ended: 2023-04-26 06:43:14+00:00
ipa007.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
  last update ended: 2023-04-26 06:43:02+00:00
  

#
# find the segment (domain or ca)
#   
$ ipa topologysegment-find domain | grep etcd
  Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
  Right node: bad_server.ad.companyx.fm

#
# delete that segment
#
$ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
---------------------------------------------------------
Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm"
---------------------------------------------------------

#
# check it has gone - tada!
#
$ ipa-replica-manage list -v `hostname`
ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
denied: '/var/log/ipa/cli.log'
ipa003dc.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update started
  last update ended: 1970-01-01 00:00:00+00:00
ipa005.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update started
  last update ended: 1970-01-01 00:00:00+00:00
ipa007.ad.companyx.fm: replica
  last update status: Error (0) Replica acquired successfully: Incremental update
succeeded
  last update ended: 2023-04-26 06:46:08+00:00
  
  

#
# Next up, removing the "LDAP Conflicts" but - "Removal of Segment
disconnects topology.Deletion not allowed."
#

$ ldapdelete
cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm
SASL/GSSAPI authentication started
SASL username: nicholas.cross(a)AD.companyx.FM
SASL SSF: 256
SASL data security layer installed.
ldap_delete: Server is unwilling to perform (53)
	additional info: Removal of Segment disconnects topology.Deletion not allowed.
	
#
# I think this is the solution: https://access.redhat.com/solutions/5507711
#
# Question1: during running the above RedHat solution, does this only disable the topology
replication? and leaves all other dirsrv components running?
#


#
# After that - finally remove the Ghost Replicas - which was the original question. 
#

$ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
"dc=ad,dc=companyx,dc=fm"
'(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=replication manager,cn=config
nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
nsDS5ReplicaBindDnGroupCheckInterval: 60
nsDS5ReplicaId: 56
nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
nsDS5ReplicaType: 3
nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA==
nsds5ReplicaBackoffMax: 300
nsds5ReplicaLegacyConsumer: off
nsds5ReplicaReleaseTimeout: 60
objectClass: top
objectClass: nsds5replica
objectClass: extensibleobject
nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm
nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm
nsds50ruv: {replicageneration} 5d9e2076000000040000
nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000
6448d15d000400380000
nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000
6448d115000e002e0000
nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000
6448d115000700300000
nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000
6448ca410000003a0000
nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000
6448c9e40009003c0000
nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000
6448cab40000003e0000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000
nsds5agmtmaxcsn:
dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000
nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071
nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b
nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956
nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb
nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9
nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000
nsds5ReplicaChangeCount: 790081
nsds5replicareapactive: 0

#
#  Question2: How to remove these? from the above
#

nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 12} 00000000
nsruvReplicaLastModified: {replica 21} 00000000



# this sort of thing doesn't seem to work.

dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config
changetype: add
objectclass: top
objectclass: extensibleObject
replica-base-dn: dc=ad,dc=companyx,dc=fm
replica-id: 12
cn: clean 12



Many thanks.

[Rob Crittenden]

Nicholas Cross via FreeIPA-users wrote:
>  Ah got it!  Wonderful.
>  
>  The trick as to run the topologysegement-del on the same server it was on.
>  
>  It seems i am moving forward with this now - thanks.
>  
>  
>  #
>  # To remove the topology segment, which removed the replica agreement
>  #
>  
>  #
>  # Show the bad replication agreement
>  #
>  
>  # ipa-replica-manage list -v `hostname`
>  Directory Manager password:
>  
>  bad_server.ad.companyx.fm: replica
>    last update status: Error (-1) Problem connecting to replica - LDAP error: Can't
> contact LDAP server (connection error)
>    last update ended: 1970-01-01 00:00:00+00:00
>  ipa003dc.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> succeeded
>    last update ended: 2023-04-26 06:43:07+00:00
>  ipa005.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> succeeded
>    last update ended: 2023-04-26 06:43:14+00:00
>  ipa007.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> succeeded
>    last update ended: 2023-04-26 06:43:02+00:00
>    
>  
>  #
>  # find the segment (domain or ca)
>  #   
>  $ ipa topologysegment-find domain | grep etcd
>    Segment name: ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
>    Right node: bad_server.ad.companyx.fm
>  
>  #
>  # delete that segment
>  #
>  $ ipa topologysegment-del domain ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm
>  ---------------------------------------------------------
>  Deleted segment "ipa006.ad.companyx.fm-to-bad_server.ad.companyx.fm"
>  ---------------------------------------------------------
>  
>  #
>  # check it has gone - tada!
>  #
>  $ ipa-replica-manage list -v `hostname`
>  ipa: ERROR: Cannot open log file '/var/log/ipa/cli.log': [Errno 13] Permission
> denied: '/var/log/ipa/cli.log'
>  ipa003dc.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> started
>    last update ended: 1970-01-01 00:00:00+00:00
>  ipa005.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> started
>    last update ended: 1970-01-01 00:00:00+00:00
>  ipa007.ad.companyx.fm: replica
>    last update status: Error (0) Replica acquired successfully: Incremental update
> succeeded
>    last update ended: 2023-04-26 06:46:08+00:00
>    
>    
>  
>  #
>  # Next up, removing the "LDAP Conflicts" but - "Removal of Segment
> disconnects topology.Deletion not allowed."
>  #
>  
>  $ ldapdelete
> cn=bad_server.ad.companyx.fm-to-ipa006.ad.companyx.fm+nsuniqueid=34b26c01-ceee11ed-9d1d82de-03f3a8a3,cn=ca,cn=topology,cn=ipa,cn=etc,dc=ad,dc=companyx,dc=fm
>  SASL/GSSAPI authentication started
>  SASL username: nicholas.cross(a)AD.companyx.FM
>  SASL SSF: 256
>  SASL data security layer installed.
>  ldap_delete: Server is unwilling to perform (53)
>  	additional info: Removal of Segment disconnects topology.Deletion not allowed.
>  	
>  #
>  # I think this is the solution: https://access.redhat.com/solutions/5507711
>  #
>  # Question1: during running the above RedHat solution, does this only disable the
> topology replication? and leaves all other dirsrv components running?
>  #
>  
>  
>  #
>  # After that - finally remove the Ghost Replicas - which was the original question. 
>  #
>  
>  $ ldapsearch -D "cn=Directory Manager" -w $pass -Q -o ldif-wrap=no -LLL -b
> "dc=ad,dc=companyx,dc=fm"
> '(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))'
>  dn: cn=replica,cn=dc\3Dad\2Cdc\3Dcompanyx\2Cdc\3Dfm,cn=mapping tree,cn=config
>  cn: replica
>  nsDS5Flags: 1
>  nsDS5ReplicaBindDN: cn=replication manager,cn=config
>  nsDS5ReplicaBindDNGroup: cn=replication
> managers,cn=sysaccounts,cn=etc,dc=ad,dc=companyx,dc=fm
>  nsDS5ReplicaBindDnGroupCheckInterval: 60
>  nsDS5ReplicaId: 56
>  nsDS5ReplicaName: a6b5640c-ad3911ed-a50980fb-6203228c
>  nsDS5ReplicaRoot: dc=ad,dc=companyx,dc=fm
>  nsDS5ReplicaType: 3
>  nsState:: OAAAAAAAAABf0EhkAAAAAAAAAAAAAAAA7AAAAAAAAAAFAAAAAAAAAA==
>  nsds5ReplicaBackoffMax: 300
>  nsds5ReplicaLegacyConsumer: off
>  nsds5ReplicaReleaseTimeout: 60
>  objectClass: top
>  objectClass: nsds5replica
>  objectClass: extensibleobject
>  nsds5ReplicaCleanRUV: 15:no:0:dc=ad,dc=companyx,dc=fm
>  nsds5ReplicaCleanRUV: 24:no:0:dc=ad,dc=companyx,dc=fm
>  nsds50ruv: {replicageneration} 5d9e2076000000040000
>  nsds50ruv: {replica 56 ldap://ipa006.ad.companyx.fm:389} 63ece66f000000380000
> 6448d15d000400380000
>  nsds50ruv: {replica 46 ldap://ipa005.ad.companyx.fm:389} 63dbcc200001002e0000
> 6448d115000e002e0000
>  nsds50ruv: {replica 48 ldap://ipa007.ad.companyx.fm:389} 63ea4e54000100300000
> 6448d115000700300000
>  nsds50ruv: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 643d03280001003a0000
> 6448ca410000003a0000
>  nsds50ruv: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 643d19680001003c0000
> 6448c9e40009003c0000
>  nsds50ruv: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 643d491e0001003e0000
> 6448cab40000003e0000
>  nsds5agmtmaxcsn:
> dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa003dc.ad.companyx.fm;ipa003dc.ad.companyx.fm;389;62;6448cf8e000800380000
>  nsds5agmtmaxcsn:
> dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa005.ad.companyx.fm;ipa005.ad.companyx.fm;389;46;6448cf8e000800380000
>  nsds5agmtmaxcsn:
> dc=ad,dc=companyx,dc=fm;ipa006.ad.companyx.fm-to-ipa007.ad.companyx.fm;ipa007.ad.companyx.fm;389;48;6448cf8e000800380000
>  nsruvReplicaLastModified: {replica 56 ldap://ipa006.ad.companyx.fm:389} 6448d071
>  nsruvReplicaLastModified: {replica 46 ldap://ipa005.ad.companyx.fm:389} 6448d02b
>  nsruvReplicaLastModified: {replica 48 ldap://ipa007.ad.companyx.fm:389} 6448d02b
>  nsruvReplicaLastModified: {replica 58 ldap://ipa001dc.ad.companyx.fm:389} 6448c956
>  nsruvReplicaLastModified: {replica 60 ldap://ipa002dc.ad.companyx.fm:389} 6448c8fb
>  nsruvReplicaLastModified: {replica 62 ldap://ipa003dc.ad.companyx.fm:389} 6448c9c9
>  nsruvReplicaLastModified: {replica 25} 00000000
>  nsruvReplicaLastModified: {replica 23} 00000000
>  nsruvReplicaLastModified: {replica 40} 00000000
>  nsruvReplicaLastModified: {replica 12} 00000000
>  nsruvReplicaLastModified: {replica 21} 00000000
>  nsds5ReplicaChangeCount: 790081
>  nsds5replicareapactive: 0
>  
>  #
>  #  Question2: How to remove these? from the above
>  #
>  
>  nsruvReplicaLastModified: {replica 25} 00000000
>  nsruvReplicaLastModified: {replica 23} 00000000
>  nsruvReplicaLastModified: {replica 40} 00000000
>  nsruvReplicaLastModified: {replica 12} 00000000
>  nsruvReplicaLastModified: {replica 21} 00000000
>  
>  
>  
>  # this sort of thing doesn't seem to work.
>  
>  dn: cn=clean 12,cn=cleanallruv,cn=tasks,cn=config
>  changetype: add
>  objectclass: top
>  objectclass: extensibleObject
>  replica-base-dn: dc=ad,dc=companyx,dc=fm
>  replica-id: 12
>  cn: clean 12 
You can try ipa-replica-manage clean-ruv <value> to try to remove
specific values.

rob

[Nicholas Cross]

Just an update on this.

Came back from the long weekend and 50% of our servers (3) were not responding, the dirsrv
was crashing everytime it had an update from the CA master (we could not figure out why). 
If we closed the firewall between replica and CA master the servers stayed up.

After a few days of trying various things to resurrect the down servers we rebuilt the
whole cluster based off the master CA server.  None of the original servers are now
present.

After another long weekend we seem (so far) to have a stable cluster. Ignoring the usual
replication conflicts we get with heavy server creation/deletion due to AWS spot
instances.

The only out standing item now is the records that make "cipa" think we have 
"ghost replicas"

nsruvReplicaLastModified: {replica 25} 00000000
nsruvReplicaLastModified: {replica 23} 00000000
nsruvReplicaLastModified: {replica 40} 00000000
nsruvReplicaLastModified: {replica 21} 00000000

There are no RUVs to match these replicas (21,23, 25, 40).
So it looks like these key/value pairs are the only things left.

Any ideas on how to remove them?

Many thanks.