ipa replication issues

[Sandor Juhasz]

Hello,

we are using freeipa in a 4way multi master replication setup.
Servers ipa14,ipa15 and ipa34,ipa35 on
CentOS Linux release 7.3.1611 (Core) with version
ipa-server-common-4.4.0-14.el7.centos.7.noarch.

We have an issue where one of the servers log a missing CSN. It happens
even after
ipa replication reinitialized.
We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we
see it in those files listed on ipa15 and the error is reported there.
Please see attached file with logs.

How can we fix this?

--
*Sándor Juhász*
System Administrator
*ChemAxon* *Ltd*.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: [phone]

[Ludwig Krispenz]

On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
>  Hello,
>
>  we are using freeipa in a 4way multi master replication setup.
>  Servers ipa14,ipa15 and ipa34,ipa35 on
>  CentOS Linux release 7.3.1611 (Core) with version
>  ipa-server-common-4.4.0-14.el7.centos.7.noarch.
>
>  We have an issue where one of the servers log a missing CSN. It 
>  happens even after
>  ipa replication reinitialized.
>  We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, 
>  but we see it in those files listed on ipa15 and the error is reported 
>  there.
>  Please see attached file with logs. 
the missing csn is from Nov,13,2017 - so it is
not unlikely it was 
trimmed. But in some RUV there seems to be a reference to it, and 
replication uses to position it in the changelog.


>  How can we fix this? 
we first should get a full picture of the replicaids and RUVs
on all 
servers, could you do on all servers the following search:
ldapsearch  .... -o ldif-wrap=no  -D "cn=directory manager" .... -b 
cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv

That should help in deciding what to do.

There is also on option to kick an agreement to ingnore a missing change:

do the following change on the failing replication agreement, but it 
would be better to have the data first:

ldapmodify ....
dn: <agmt>
replace: nsds5ReplicaIgnoreMissingChange
nsds5ReplicaIgnoreMissingChange: once

>  --
>  *Sándor Juhász*
>  System Administrator
>  *ChemAxon**Ltd*.
>  Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
>  Cell: [phone]
>
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
Shander

[Sandor Juhasz]

here are the results:

~~~~~~~~

[root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no  -D
"cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
nsds5replicaid nsds50ruv
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 4
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
5ad07160000000040000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
5ad06adb000900070000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
5ad0711c003a000b0000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 6
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
5ad07153000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
5a0da16d000200100000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa14 ~]#

~~~~~~~~

[root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D "cn=directory
manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid
nsds50ruv
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 15
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
5ad071c20000000f0000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
5ad06adb000900070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
5ad071af002d00040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
5ad071d20021000b0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 16
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
5a0da16d000200100000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
5ad07153000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa15 ~]#

~~~~~~~~

[root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D "cn=directory
manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid
nsds50ruv
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 11
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
5ad072120003000b0000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
5ad06adb000900070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
5ad071af002d00040000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 12
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
59d74c4e0004000c0000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
5a0a27d9000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
589adeca000000080000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa34 ~]#

~~~~~~~~

[root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D "cn=directory
manager" -x -W -b cn=config "objectclass=nsds5replica" nsds5replicaid
nsds50ruv
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=config> with scope subtree
# filter: objectclass=nsds5replica
# requesting: nsds5replicaid nsds50ruv
#

# replica, dc\3Dcxn, mapping tree, config
dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
nsds5replicaid: 7
nsds50ruv: {replicageneration} 58987d9e000000040000
nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
5ad07248001800070000
nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
5ad071af002d00040000
nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
5ad072490010000b0000
nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
5ad06e1a0004000f0000

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 8
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
589adeca000000080000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
5a0a27d9000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
59d74c4e0004000c0000

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2
[root@ipa35 ~]#

~~~~~~~~

--
*Sándor Juhász*
System Administrator
*ChemAxon* *Ltd*.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: [phone]

On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
>
>  Hello,
>
>  we are using freeipa in a 4way multi master replication setup.
>  Servers ipa14,ipa15 and ipa34,ipa35 on
>  CentOS Linux release 7.3.1611 (Core) with version
>  ipa-server-common-4.4.0-14.el7.centos.7.noarch.
>
>  We have an issue where one of the servers log a missing CSN. It happens
>  even after
>  ipa replication reinitialized.
>  We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but we
>  see it in those files listed on ipa15 and the error is reported there.
>  Please see attached file with logs.
>
>  the missing csn is from Nov,13,2017 - so it is not unlikely it was
>  trimmed. But in some RUV there seems to be a reference to it, and
>  replication uses to position it in the changelog.
>
>
>
>  How can we fix this?
>
>  we first should get a full picture of the replicaids and RUVs on all
>  servers, could you do on all servers the following search:
>  ldapsearch  .... -o ldif-wrap=no  -D "cn=directory manager" .... -b
>  cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
>
>  That should help in deciding what to do.
>
>  There is also on option to kick an agreement to ingnore a missing change:
>
>  do the following change on the failing replication agreement, but it would
>  be better to have the data first:
>
>  ldapmodify ....
>  dn: <agmt>
>  replace: nsds5ReplicaIgnoreMissingChange
>  nsds5ReplicaIgnoreMissingChange: once
>
>
>  --
>  *Sándor Juhász*
>  System Administrator
>  *ChemAxon* *Ltd*.
>  Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
>  Cell: [phone]
>
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
>
>  --
>  Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>  Commercial register: Amtsgericht Muenchen, HRB 153243,
>  Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
> Shander
>
>
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>
>

[Ludwig Krispenz]

Hi,
when extracting the relevant data, we see:

[root@ipa14 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 6
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
5ad07153000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 
5a0da16d000200100000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
589adeca000000080000
~~~~~~~~
[root@ipa15 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 16
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 
5a0da16d000200100000
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
5ad07153000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
59d74c4e0004000c0000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
589adeca000000080000
~~~~~~~~
[root@ipa34 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 12
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
59d74c4e0004000c0000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
5a0a27d9000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
589adeca000000080000
~~~~~~~~
[root@ipa35 ~]
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nsds5replicaid: 8
nsds50ruv: {replicageneration} 58987e19000000060000
nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
589adeca000000080000
nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
5a0a27d9000000060000
nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
59d74c4e0004000c0000


this indicates that all replicas are in sync for replicaid 8 and 12, but 
for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older 
data.
I cannot say what has happened, but I think you need reinit 34 and 35 
from either 14 or 15

On 04/13/2018 11:13 AM, Sandor Juhasz wrote:
>  here are the results:
>
>  ~~~~~~~~
>
>  [root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no  -D 
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 4
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 
>  5ad07160000000040000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 
>  5ad06adb000900070000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 
>  5ad0711c003a000b0000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 6
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
>  5ad07153000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 
>  5a0da16d000200100000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa14 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D 
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 15
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 
>  5ad071c20000000f0000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 
>  5ad06adb000900070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 
>  5ad071af002d00040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 
>  5ad071d20021000b0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 16
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000 
>  5a0da16d000200100000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
>  5ad07153000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa15 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D 
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 11
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 
>  5ad072120003000b0000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 
>  5ad06adb000900070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 
>  5ad071af002d00040000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 12
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
>  5a0a27d9000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa34 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D 
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 7
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000 
>  5ad07248001800070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000 
>  5ad071af002d00040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000 
>  5ad072490010000b0000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000 
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 8
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000 
>  589adeca000000080000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000 
>  5a0a27d9000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000 
>  59d74c4e0004000c0000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa35 ~]#
>
>  ~~~~~~~~
>
>  --
>  *Sándor Juhász*
>  System Administrator
>  *ChemAxon**Ltd*.
>  Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
>  Cell: [phone]
>
>  On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users 
>  <freeipa-users(a)lists.fedorahosted.org 
>  <mailto:[e-mail]>> wrote:
>
>
>      On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
> >     Hello,
> >
> >     we are using freeipa in a 4way multi master replication setup.
> >     Servers ipa14,ipa15 and ipa34,ipa35 on
> >     CentOS Linux release 7.3.1611 (Core) with version
> >     ipa-server-common-4.4.0-14.el7.centos.7.noarch.
> >
> >     We have an issue where one of the servers log a missing CSN. It
> >     happens even after
> >     ipa replication reinitialized.
> >     We are guessing that CSN 5a0a27d9000000060000 only exists on
> >     ipa35, but we see it in those files listed on ipa15 and the error
> >     is reported there.
> >     Please see attached file with logs.
>      the missing csn is from Nov,13,2017 - so it is not unlikely it was
>      trimmed. But in some RUV there seems to be a reference to it, and
>      replication uses to position it in the changelog.
>
>
> >
> >     How can we fix this?
>      we first should get a full picture of the replicaids and RUVs on
>      all servers, could you do on all servers the following search:
>      ldapsearch  .... -o ldif-wrap=no  -D "cn=directory manager" ....
>      -b cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
>
>      That should help in deciding what to do.
>
>      There is also on option to kick an agreement to ingnore a missing
>      change:
>
>      do the following change on the failing replication agreement, but
>      it would be better to have the data first:
>
>      ldapmodify ....
>      dn: <agmt>
>      replace: nsds5ReplicaIgnoreMissingChange
>      nsds5ReplicaIgnoreMissingChange: once
>
> >
> >     --
> >     *Sándor Juhász*
> >     System Administrator
> >     *ChemAxon**Ltd*.
> >     Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary,
> >     H-1031
> >     Cell: [phone]
> >
> >
> >     _______________________________________________
> >     FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org 
> <mailto:[e-mail]>
> >     To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org 
> <mailto:[e-mail]>
>
>      -- 
>      Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn,
>      Commercial register: Amtsgericht Muenchen, HRB 153243,
>      Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
> Shander
>
>
>      _______________________________________________
>      FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>      To unsubscribe send an email to
>      freeipa-users-leave(a)lists.fedorahosted.org
>      <mailto:[e-mail]>
>
>  
-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
Shander

[Sandor Juhasz]

re-initialize was not sufficient.
We needed to trash ipa34 and ipa35 completely and install new replica.
We had some hassle with hosts file and permission denied on the
ipa-replica-install step,
in the end it worked out.
--
*Sándor Juhász*
System Administrator
*ChemAxon* *Ltd*.
Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
Cell: [phone]


On Fri, Apr 13, 2018 at 11:56 AM Ludwig Krispenz <lkrispen(a)redhat.com&gt;
wrote:

>  Hi,
>  when extracting the relevant data, we see:
>
>  [root@ipa14 ~]
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 6
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5ad07153000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
>  5a0da16d000200100000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>  ~~~~~~~~
>  [root@ipa15 ~]
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 16
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
>  5a0da16d000200100000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5ad07153000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>  ~~~~~~~~
>  [root@ipa34 ~]
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 12
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5a0a27d9000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>  ~~~~~~~~
>  [root@ipa35 ~]
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 8
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5a0a27d9000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>
>
>  this indicates that all replicas are in sync for replicaid 8 and 12, but
>  for rid 16, ipa 34 and ipa 35 have no data and for rid 6 they have older
>  data.
>  I cannot say what has happened, but I think you need reinit 34 and 35 from
>  either 14 or 15
>
>  On 04/13/2018 11:13 AM, Sandor Juhasz wrote:
>
>  here are the results:
>
>  ~~~~~~~~
>
>  [root@ipa14 ~]# ldapsearch -H ldap://ipa14.bpo.cxn -o ldif-wrap=no  -D
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 4
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
>  5ad07160000000040000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
>  5ad06adb000900070000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
>  5ad0711c003a000b0000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 6
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5ad07153000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
>  5a0da16d000200100000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa14 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa15 ~]# ldapsearch -H ldap://ipa15 -o ldif-wrap=no -D
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 15
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
>  5ad071c20000000f0000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
>  5ad06adb000900070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
>  5ad071af002d00040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
>  5ad071d20021000b0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 16
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389} 5a0c4f48000000100000
>  5a0da16d000200100000
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5ad07153000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa15 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa34 ~]# ldapsearch -H ldap://ipa34 -o ldif-wrap=no -D
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 11
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
>  5ad072120003000b0000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
>  5ad06adb000900070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
>  5ad071af002d00040000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 12
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5a0a27d9000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa34 ~]#
>
>  ~~~~~~~~
>
>  [root@ipa35 ~]# ldapsearch -H ldap://ipa35 -o ldif-wrap=no -D
>  "cn=directory manager" -x -W -b cn=config "objectclass=nsds5replica"
>  nsds5replicaid nsds50ruv
>  Enter LDAP Password:
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=config> with scope subtree
>  # filter: objectclass=nsds5replica
>  # requesting: nsds5replicaid nsds50ruv
>  #
>
>  # replica, dc\3Dcxn, mapping tree, config
>  dn: cn=replica,cn=dc\3Dcxn,cn=mapping tree,cn=config
>  nsds5replicaid: 7
>  nsds50ruv: {replicageneration} 58987d9e000000040000
>  nsds50ruv: {replica 7 ldap://ipa35.bph.cxn:389} 5898a473000000070000
>  5ad07248001800070000
>  nsds50ruv: {replica 4 ldap://ipa14.bpo.cxn:389} 58987d9e000100040000
>  5ad071af002d00040000
>  nsds50ruv: {replica 11 ldap://ipa34.bph.cxn:389} 59d74b730000000b0000
>  5ad072490010000b0000
>  nsds50ruv: {replica 15 ldap://ipa15.bpo.cxn:389} 5a0c4ed00000000f0000
>  5ad06e1a0004000f0000
>
>  # replica, o\3Dipaca, mapping tree, config
>  dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>  nsds5replicaid: 8
>  nsds50ruv: {replicageneration} 58987e19000000060000
>  nsds50ruv: {replica 8 ldap://ipa35.bph.cxn:389} 5898a4e0000000080000
>  589adeca000000080000
>  nsds50ruv: {replica 16 ldap://ipa15.bpo.cxn:389}
>  nsds50ruv: {replica 6 ldap://ipa14.bpo.cxn:389} 58987e1c000000060000
>  5a0a27d9000000060000
>  nsds50ruv: {replica 12 ldap://ipa34.bph.cxn:389} 59d74be60000000c0000
>  59d74c4e0004000c0000
>
>  # search result
>  search: 2
>  result: 0 Success
>
>  # numResponses: 3
>  # numEntries: 2
>  [root@ipa35 ~]#
>
>  ~~~~~~~~
>
>  --
>  *Sándor Juhász*
>  System Administrator
>  *ChemAxon* *Ltd*.
>  Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
>  Cell: [phone]
>
>  On Fri, Apr 13, 2018 at 10:51 AM, Ludwig Krispenz via FreeIPA-users <
>  freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> >
> > On 04/13/2018 08:25 AM, Sandor Juhasz via FreeIPA-users wrote:
> >
> > Hello,
> >
> > we are using freeipa in a 4way multi master replication setup.
> > Servers ipa14,ipa15 and ipa34,ipa35 on
> > CentOS Linux release 7.3.1611 (Core) with version
> > ipa-server-common-4.4.0-14.el7.centos.7.noarch.
> >
> > We have an issue where one of the servers log a missing CSN. It happens
> > even after
> > ipa replication reinitialized.
> > We are guessing that CSN 5a0a27d9000000060000 only exists on ipa35, but
> > we see it in those files listed on ipa15 and the error is reported there.
> > Please see attached file with logs.
> >
> > the missing csn is from Nov,13,2017 - so it is not unlikely it was
> > trimmed. But in some RUV there seems to be a reference to it, and
> > replication uses to position it in the changelog.
> >
> >
> >
> > How can we fix this?
> >
> > we first should get a full picture of the replicaids and RUVs on all
> > servers, could you do on all servers the following search:
> > ldapsearch  .... -o ldif-wrap=no  -D "cn=directory manager" .... -b
> > cn=config "objectclass=nsds5replica" nsds5replicaid nsds50ruv
> >
> > That should help in deciding what to do.
> >
> > There is also on option to kick an agreement to ingnore a missing change:
> >
> > do the following change on the failing replication agreement, but it
> > would be better to have the data first:
> >
> > ldapmodify ....
> > dn: <agmt>
> > replace: nsds5ReplicaIgnoreMissingChange
> > nsds5ReplicaIgnoreMissingChange: once
> >
> >
> > --
> > *Sándor Juhász*
> > System Administrator
> > *ChemAxon* *Ltd*.
> > Building Hx, GraphiSoft Park, Záhony utca 7, Budapest, Hungary, H-1031
> > Cell: [phone]
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> >
> >
> > --
> > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> > Commercial register: Amtsgericht Muenchen, HRB 153243,
> > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
> Shander
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> >
> >
>
>  --
>  Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>  Commercial register: Amtsgericht Muenchen, HRB 153243,
>  Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric
> Shander
>
>