ipa-replica-manage: unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000

[Harald Dunkel]

Hi folks,

somehow my ipa servers became out of sync. ipa4 has an
additional host entry, not known on the others. On
examining I stumbled over this:

[root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv

unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
These RUVs are dangling and will be removed:
Host: ipabak.ac.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Host: ipa1.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Host: ipa0.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Host: ipa3.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Host: ipa4.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Host: ipa2.example.de
         RUVs:
                 id: 11, hostname: ipabak.ac.example.de
         CS-RUVs:
Proceed with cleaning? [no]: yes
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
Clean the Replication Update Vector for ipabak.ac.example.de:389
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
Cleanup task created

[root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv

unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
No dangling RUVs found

[root@ipa0 ~]# ipa-replica-manage list-ruv

unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
Replica Update Vectors:
         ipa0.example.de:389: 12
         ipa2.example.de:389: 5
         ipa1.example.de:389: 4
         ipa4.example.de:389: 8
         ipa3.example.de:389: 6
         ipabak.ac.example.de:389: 13
Certificate Server Replica Update Vectors:
         ipa0.example.de:389: 1095
         ipa2.example.de:389: 97
         ipa1.example.de:389: 96
         ipabak.ac.example.de:389: 1090

The ruvs are the same on all 6 hosts (AFAICS), so I wonder how I could
fix this?

Every helpful comment is highly appreciated.
Harri

[Ludwig Krispenz]

Hi,

to get rid of this ruv entry with replicaid 7 you could try to run the 
cleanallruv task directly. On any server (and onöy on one) run

ldapmodify ..... -D "cn=directory manager"

|dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
changetype: add
objectclass: extensibleObject
replica-base-dn: <your suffix >
replica-id: 7
replica-force-cleaning: yes

|

|But I would like to understand how you did get in|to this state, we 
have seen this occasionly, but have no reproducer. Unfortunately the csn 
for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you will probably 
not remember
||

On 03/12/2018 03:55 PM, Harald Dunkel via FreeIPA-users wrote:
> Hi folks,
>
> somehow my ipa servers became out of sync. ipa4 has an
> additional host entry, not known on the others. On
> examining I stumbled over this:
>
> [root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv
>
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> These RUVs are dangling and will be removed:
> Host: ipabak.ac.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Host: ipa1.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Host: ipa0.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Host: ipa3.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Host: ipa4.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Host: ipa2.example.de
>         RUVs:
>                 id: 11, hostname: ipabak.ac.example.de
>         CS-RUVs:
> Proceed with cleaning? [no]: yes
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> Clean the Replication Update Vector for ipabak.ac.example.de:389
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> Cleanup task created
>
> [root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv
>
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> No dangling RUVs found
>
> [root@ipa0 ~]# ipa-replica-manage list-ruv
>
> unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> Replica Update Vectors:
>         ipa0.example.de:389: 12
>         ipa2.example.de:389: 5
>         ipa1.example.de:389: 4
>         ipa4.example.de:389: 8
>         ipa3.example.de:389: 6
>         ipabak.ac.example.de:389: 13
> Certificate Server Replica Update Vectors:
>         ipa0.example.de:389: 1095
>         ipa2.example.de:389: 97
>         ipa1.example.de:389: 96
>         ipabak.ac.example.de:389: 1090
>
> The ruvs are the same on all 6 hosts (AFAICS), so I wonder how I could
> fix this?
>
> Every helpful comment is highly appreciated.
> Harri
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]

[thierry bordaz]

Hi Harald,

What version of DS are you running ?
We have a reproducer (not systematic) for versions before 
https://bugzilla.redhat.com/show_bug.cgi?id=1516309 but we have not 
reproduced it since then, you may need to upgrade.

best regards
thierry

On 03/12/2018 05:10 PM, Ludwig Krispenz wrote:
> Hi,
>
> to get rid of this ruv entry with replicaid 7 you could try to run the 
> cleanallruv task directly. On any server (and onöy on one) run
>
> ldapmodify ..... -D "cn=directory manager"
> |dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config changetype: add 
> objectclass: extensibleObject replica-base-dn: <your suffix > 
> replica-id: 7 replica-force-cleaning: yes |
> |But I would like to understand how you did get in|to this state, we 
> have seen this occasionly, but have no reproducer. Unfortunately the 
> csn for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you will 
> probably not remember
> ||
>
> On 03/12/2018 03:55 PM, Harald Dunkel via FreeIPA-users wrote:
> > Hi folks,
> >
> > somehow my ipa servers became out of sync. ipa4 has an
> > additional host entry, not known on the others. On
> > examining I stumbled over this:
> >
> > [root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv
> >
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > These RUVs are dangling and will be removed:
> > Host: ipabak.ac.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Host: ipa1.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Host: ipa0.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Host: ipa3.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Host: ipa4.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Host: ipa2.example.de
> >         RUVs:
> >                 id: 11, hostname: ipabak.ac.example.de
> >         CS-RUVs:
> > Proceed with cleaning? [no]: yes
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > Clean the Replication Update Vector for ipabak.ac.example.de:389
> > Background task created to clean replication data. This may take a 
> > while.
> > This may be safely interrupted with Ctrl+C
> > Cleanup task created
> >
> > [root@ipa0 ~]# ipa-replica-manage clean-dangling-ruv
> >
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > No dangling RUVs found
> >
> > [root@ipa0 ~]# ipa-replica-manage list-ruv
> >
> > unable to decode: {replica 7} 58809c7c000300070000 58809c7c000300070000
> > Replica Update Vectors:
> >         ipa0.example.de:389: 12
> >         ipa2.example.de:389: 5
> >         ipa1.example.de:389: 4
> >         ipa4.example.de:389: 8
> >         ipa3.example.de:389: 6
> >         ipabak.ac.example.de:389: 13
> > Certificate Server Replica Update Vectors:
> >         ipa0.example.de:389: 1095
> >         ipa2.example.de:389: 97
> >         ipa1.example.de:389: 96
> >         ipabak.ac.example.de:389: 1090
> >
> > The ruvs are the same on all 6 hosts (AFAICS), so I wonder how I could
> > fix this?
> >
> > Every helpful comment is highly appreciated.
> > Harri
> > _______________________________________________
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]
>
> -- 
> Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander

[Harald Dunkel]

Hi Thierry,

On 03/12/18 17:52, thierry bordaz via FreeIPA-users wrote:
> Hi Harald,
>
> What version of DS are you running ?
> We have a reproducer (not systematic) for versions before https://bugzilla.redhat.com/show_bug.cgi?id=1516309 but we have not reproduced it since then, you may need to upgrade.

I am highly alarmed by this one. Sounds like a fork bomb.
Would you suggest to not run cleanallruv?

Platform is Centos 7.4, directory service was 389-ds-base-\
1.3.6.1-26.el7_4, but this problem might have been introduced
using older version. After the upgrade this morning it is
version 1.3.6.1-28.el7_4.

Regards
Harri

[Harald Dunkel]

Hi Ludwig,

On 03/12/18 17:10, Ludwig Krispenz via FreeIPA-users wrote:
> Hi,
>
> to get rid of this ruv entry with replicaid 7 you could try to run the cleanallruv task directly. On any server (and onöy on one) run
>
> ldapmodify ..... -D "cn=directory manager"
>
> |dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config changetype: add objectclass: extensibleObject replica-base-dn: <your suffix > replica-id: 7 replica-force-cleaning: yes |
>
> |But I would like to understand how you did get in|to this state, we have seen this occasionly, but have no reproducer. Unfortunately the csn for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you will probably not remember
> ||

Not sure if its related, but I wrote an EMail to freeipa-users on that day,
reporting an internal error. See

https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html

The conflict is still not resolved completely. There are 2 entries I
could not cleanup, see below. What would you suggest?

[root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" -w secret -b "dc=example,dc=de" "nsds5ReplConflict=*" * nsds5ReplConflict
# extended LDIF
#
# LDAPv3
# base <dc=example,dc=de> with scope subtree
# filter: nsds5ReplConflict=*
# requesting: * nsds5ReplConflict
#

# ipaservers + 109be302-ccd911e6-a5b3d0c8-d8da17db, hostgroups, accounts, example.de
dn: cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
memberOf: cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
objectClass: top
objectClass: ipahostgroup
objectClass: ipaobject
objectClass: groupOfNames
objectClass: nestedGroup
objectClass: mepOriginEntry
description: IPA server hosts
cn: ipaservers
ipaUniqueID: 14a4041e-ccd9-11e6-b194-fe4936c476ff
nsds5ReplConflict: namingConflict cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de

# ipaservers + 109be304-ccd911e6-a5b3d0c8-d8da17db, ng, alt, example.de
dn: cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
memberHost: cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
objectClass: ipanisnetgroup
objectClass: ipaobject
objectClass: mepManagedEntry
objectClass: ipaAssociation
objectClass: top
nisDomainName: example.de
cn: ipaservers
description: ipaNetgroup ipaservers
mepManagedBy: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
ipaUniqueID: 15699da0-ccd9-11e6-b194-fe4936c476ff
nsds5ReplConflict: namingConflict cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de

# search result
search: 2
result: 0 Success

# numResponses: 3
# numEntries: 2

[root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" -w secret -b "cn=hostgroups,cn=accounts,dc=example,dc=de"
# extended LDIF
#
# LDAPv3
# base <cn=hostgroups,cn=accounts,dc=example,dc=de> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# hostgroups, accounts, example.de
dn: cn=hostgroups,cn=accounts,dc=example,dc=de
objectClass: top
objectClass: nsContainer
cn: hostgroups

# admin_hosts, hostgroups, accounts, example.de
dn: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
memberOf: ipaUniqueID=18c7ac56-c9a3-11e5-a675-00165cee60d7,cn=hbac,dc=example,dc=de
memberOf: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
member: fqdn=srvl023.ac.example.de,cn=computers,cn=accounts,dc=example,dc=de
mepManagedEntry: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
objectClass: ipahostgroup
objectClass: ipaobject
objectClass: nestedGroup
objectClass: groupOfNames
objectClass: top
objectClass: mepOriginEntry
cn: admin_hosts
description: hosts with restricted access
ipaUniqueID: ecc0a97c-c9a3-11e5-bcd9-00165cee60d7

# ipaservers, hostgroups, accounts, example.de
dn: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
memberOf: cn=Replication Administrators,cn=privileges,cn=pbac,dc=example,dc=de
memberOf: cn=Add Replication Agreements,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Modify Replication Agreements,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Remove Replication Agreements,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Modify DNA Range,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Read PassSync Managers Configuration,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Modify PassSync Managers Configuration,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Read LDBM Database Configuration,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Add Configuration Sub-Entries,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Read DNA Range,cn=permissions,cn=pbac,dc=example,dc=de
memberOf: cn=Read Replication Agreements,cn=permissions,cn=pbac,dc=example,dc=de
member: fqdn=ipa3.example.de,cn=computers,cn=accounts,dc=example,dc=de
member: fqdn=ipa2.example.de,cn=computers,cn=accounts,dc=example,dc=de
member: fqdn=ipa1.example.de,cn=computers,cn=accounts,dc=example,dc=de
member: fqdn=ipa4.example.de,cn=computers,cn=accounts,dc=example,dc=de
member: fqdn=ipa0.example.de,cn=computers,cn=accounts,dc=example,dc=de
member: fqdn=ipabak.ac.example.de,cn=computers,cn=accounts,dc=example,dc=de
mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
objectClass: top
objectClass: ipahostgroup
objectClass: ipaobject
objectClass: groupOfNames
objectClass: nestedGroup
objectClass: mepOriginEntry
description: IPA server hosts
cn: ipaservers
ipaUniqueID: 115a2f2c-ccd9-11e6-93fa-fe49d2c33fca

# ipaservers + 109be302-ccd911e6-a5b3d0c8-d8da17db, hostgroups, accounts, example.de
dn: cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
memberOf: cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
objectClass: top
objectClass: ipahostgroup
objectClass: ipaobject
objectClass: groupOfNames
objectClass: nestedGroup
objectClass: mepOriginEntry
description: IPA server hosts
cn: ipaservers
ipaUniqueID: 14a4041e-ccd9-11e6-b194-fe4936c476ff

# search result
search: 2
result: 0 Success

# numResponses: 5
# numEntries: 4

[root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" -w secret -b "cn=ng,cn=alt,dc=example,dc=de"
# extended LDIF
#
# LDAPv3
# base <cn=ng,cn=alt,dc=example,dc=de> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# ng, alt, example.de
dn: cn=ng,cn=alt,dc=example,dc=de
objectClass: nsContainer
objectClass: top
cn: ng

# admin_hosts, ng, alt, example.de
dn: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
objectClass: ipanisnetgroup
objectClass: ipaobject
objectClass: mepManagedEntry
objectClass: ipaAssociation
objectClass: top
nisDomainName: example.de
cn: admin_hosts
memberHost: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
description: ipaNetgroup admin_hosts
mepManagedBy: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
ipaUniqueID: ecc27612-c9a3-11e5-bcd9-00165cee60d7

# ipaservers + 109be304-ccd911e6-a5b3d0c8-d8da17db, ng, alt, example.de
dn: cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
memberHost: cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
objectClass: ipanisnetgroup
objectClass: ipaobject
objectClass: mepManagedEntry
objectClass: ipaAssociation
objectClass: top
nisDomainName: example.de
cn: ipaservers
description: ipaNetgroup ipaservers
mepManagedBy: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
ipaUniqueID: 15699da0-ccd9-11e6-b194-fe4936c476ff

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3

Please note that there is no "cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de"

Every helpful comment is highly appreciated.
Harri

[Ludwig Krispenz]

On 03/13/2018 09:07 AM, Harald Dunkel via FreeIPA-users wrote:
> Hi Ludwig,
>
> On 03/12/18 17:10, Ludwig Krispenz via FreeIPA-users wrote:
> > Hi,
> >
> > to get rid of this ruv entry with replicaid 7 you could try to run 
> > the cleanallruv task directly. On any server (and onöy on one) run
> >
> > ldapmodify ..... -D "cn=directory manager"
> >
> > |dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config changetype: add 
> > objectclass: extensibleObject replica-base-dn: <your suffix > 
> > replica-id: 7 replica-force-cleaning: yes |
> >
> > |But I would like to understand how you did get in|to this state, we 
> > have seen this occasionly, but have no reproducer. Unfortunately the 
> > csn for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you will 
> > probably not remember
> > ||
>
> Not sure if its related, but I wrote an EMail to freeipa-users on that 
> day,
> reporting an internal error. See
>
> https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html

the csn of the replicaid 7 is from Jan,19th, two days later. maybe you 
did ad a new replica, or removed one or ...

Looks like you have three problems and I don't think they are related.

1] corrupt RUV element, as Thierry said, we believe to have an an 
explanation how it could occur before a specific fix, and since it is 
there for one year you didn't have that fix. To get rid of this you 
could apply cleanallruv as I suggested in a previous mail, if you do not 
want to do it, I think it was there for quite some time and doesn't do 
really any harm except raising errors in decoding it.

2] conflicts, looks like you have conflitcts remaining which are relate 
to managed entries, these are espcially bad and difficult to handle. if 
you delete the conflict of an original entry it will delete the real 
managed entry and not the conflict of the managed entry. And the managed 
entry plugin is trying to prevent you to modify these entries directly.
So what you can do is: determine which entries should be there, then 
delete them or rename them, if managed enty plugin gets in the way, 
either disable the MEP plugin temporarily or modify teh entries by 
deleting the mep objectclasses and attributes, cleanup the conflicts and 
fix them again

3] Retro Changelog errors:   An error occured while adding change number 
14065299 .....
This looks like the retro changelog did mess up the counter to generate 
the next entry. a restart should hopefully reset it and let the RCL continue
> The conflict is still not resolved completely. There are 2 entries I
> could not cleanup, see below. What would you suggest?
>
> [root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" 
> -w secret -b "dc=example,dc=de" "nsds5ReplConflict=*" * 
> nsds5ReplConflict
> # extended LDIF
> #
> # LDAPv3
> # base <dc=example,dc=de> with scope subtree
> # filter: nsds5ReplConflict=*
> # requesting: * nsds5ReplConflict
> #
>
> # ipaservers + 109be302-ccd911e6-a5b3d0c8-d8da17db, hostgroups, 
> accounts, example.de
> dn: 
> cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
> memberOf: 
> cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
> mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
> objectClass: top
> objectClass: ipahostgroup
> objectClass: ipaobject
> objectClass: groupOfNames
> objectClass: nestedGroup
> objectClass: mepOriginEntry
> description: IPA server hosts
> cn: ipaservers
> ipaUniqueID: 14a4041e-ccd9-11e6-b194-fe4936c476ff
> nsds5ReplConflict: namingConflict 
> cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
>
> # ipaservers + 109be304-ccd911e6-a5b3d0c8-d8da17db, ng, alt, example.de
> dn: 
> cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
> memberHost: 
> cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
> objectClass: ipanisnetgroup
> objectClass: ipaobject
> objectClass: mepManagedEntry
> objectClass: ipaAssociation
> objectClass: top
> nisDomainName: example.de
> cn: ipaservers
> description: ipaNetgroup ipaservers
> mepManagedBy: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
> ipaUniqueID: 15699da0-ccd9-11e6-b194-fe4936c476ff
> nsds5ReplConflict: namingConflict 
> cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 3
> # numEntries: 2
>
> [root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" 
> -w secret -b "cn=hostgroups,cn=accounts,dc=example,dc=de"
> # extended LDIF
> #
> # LDAPv3
> # base <cn=hostgroups,cn=accounts,dc=example,dc=de> with scope subtree
> # filter: (objectclass=*)
> # requesting: ALL
> #
>
> # hostgroups, accounts, example.de
> dn: cn=hostgroups,cn=accounts,dc=example,dc=de
> objectClass: top
> objectClass: nsContainer
> cn: hostgroups
>
> # admin_hosts, hostgroups, accounts, example.de
> dn: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
> memberOf: 
> ipaUniqueID=18c7ac56-c9a3-11e5-a675-00165cee60d7,cn=hbac,dc=example,dc=de
> memberOf: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
> member: 
> fqdn=srvl023.ac.example.de,cn=computers,cn=accounts,dc=example,dc=de
> mepManagedEntry: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
> objectClass: ipahostgroup
> objectClass: ipaobject
> objectClass: nestedGroup
> objectClass: groupOfNames
> objectClass: top
> objectClass: mepOriginEntry
> cn: admin_hosts
> description: hosts with restricted access
> ipaUniqueID: ecc0a97c-c9a3-11e5-bcd9-00165cee60d7
>
> # ipaservers, hostgroups, accounts, example.de
> dn: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
> memberOf: cn=Replication 
> Administrators,cn=privileges,cn=pbac,dc=example,dc=de
> memberOf: cn=Add Replication 
> Agreements,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Modify Replication 
> Agreements,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Remove Replication 
> Agreements,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Modify DNA Range,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Read PassSync Managers 
> Configuration,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Modify PassSync Managers 
> Configuration,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Read LDBM Database 
> Configuration,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Add Configuration 
> Sub-Entries,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Read DNA Range,cn=permissions,cn=pbac,dc=example,dc=de
> memberOf: cn=Read Replication 
> Agreements,cn=permissions,cn=pbac,dc=example,dc=de
> member: fqdn=ipa3.example.de,cn=computers,cn=accounts,dc=example,dc=de
> member: fqdn=ipa2.example.de,cn=computers,cn=accounts,dc=example,dc=de
> member: fqdn=ipa1.example.de,cn=computers,cn=accounts,dc=example,dc=de
> member: fqdn=ipa4.example.de,cn=computers,cn=accounts,dc=example,dc=de
> member: fqdn=ipa0.example.de,cn=computers,cn=accounts,dc=example,dc=de
> member: 
> fqdn=ipabak.ac.example.de,cn=computers,cn=accounts,dc=example,dc=de
> mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
> objectClass: top
> objectClass: ipahostgroup
> objectClass: ipaobject
> objectClass: groupOfNames
> objectClass: nestedGroup
> objectClass: mepOriginEntry
> description: IPA server hosts
> cn: ipaservers
> ipaUniqueID: 115a2f2c-ccd9-11e6-93fa-fe49d2c33fca
>
> # ipaservers + 109be302-ccd911e6-a5b3d0c8-d8da17db, hostgroups, 
> accounts, example.de
> dn: 
> cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
> memberOf: 
> cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
> mepManagedEntry: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
> objectClass: top
> objectClass: ipahostgroup
> objectClass: ipaobject
> objectClass: groupOfNames
> objectClass: nestedGroup
> objectClass: mepOriginEntry
> description: IPA server hosts
> cn: ipaservers
> ipaUniqueID: 14a4041e-ccd9-11e6-b194-fe4936c476ff
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 5
> # numEntries: 4
>
> [root@ipa1 ~]# ldapsearch -o ldif-wrap=no -x -D "cn=directory manager" 
> -w secret -b "cn=ng,cn=alt,dc=example,dc=de"
> # extended LDIF
> #
> # LDAPv3
> # base <cn=ng,cn=alt,dc=example,dc=de> with scope subtree
> # filter: (objectclass=*)
> # requesting: ALL
> #
>
> # ng, alt, example.de
> dn: cn=ng,cn=alt,dc=example,dc=de
> objectClass: nsContainer
> objectClass: top
> cn: ng
>
> # admin_hosts, ng, alt, example.de
> dn: cn=admin_hosts,cn=ng,cn=alt,dc=example,dc=de
> objectClass: ipanisnetgroup
> objectClass: ipaobject
> objectClass: mepManagedEntry
> objectClass: ipaAssociation
> objectClass: top
> nisDomainName: example.de
> cn: admin_hosts
> memberHost: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
> description: ipaNetgroup admin_hosts
> mepManagedBy: cn=admin_hosts,cn=hostgroups,cn=accounts,dc=example,dc=de
> ipaUniqueID: ecc27612-c9a3-11e5-bcd9-00165cee60d7
>
> # ipaservers + 109be304-ccd911e6-a5b3d0c8-d8da17db, ng, alt, example.de
> dn: 
> cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
> memberHost: 
> cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
> objectClass: ipanisnetgroup
> objectClass: ipaobject
> objectClass: mepManagedEntry
> objectClass: ipaAssociation
> objectClass: top
> nisDomainName: example.de
> cn: ipaservers
> description: ipaNetgroup ipaservers
> mepManagedBy: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
> ipaUniqueID: 15699da0-ccd9-11e6-b194-fe4936c476ff
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 4
> # numEntries: 3
>
> Please note that there is no 
> "cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de"
>
> Every helpful comment is highly appreciated.
> Harri
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]

[Harald Dunkel]

Hi Ludwig,

On 03/13/18 14:47, Ludwig Krispenz via FreeIPA-users wrote:
> On 03/13/2018 09:07 AM, Harald Dunkel via FreeIPA-users wrote:
> > Hi Ludwig,
> >
> > On 03/12/18 17:10, Ludwig Krispenz via FreeIPA-users wrote:
> > > Hi,
> > >
> > > to get rid of this ruv entry with replicaid 7 you could try to run the cleanallruv task directly. On any server (and onöy on one) run
> > >
> > > ldapmodify ..... -D "cn=directory manager"
> > >
> > > |dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config changetype: add objectclass: extensibleObject replica-base-dn: <your suffix > replica-id: 7 replica-force-cleaning: yes |
> > >
> > > |But I would like to understand how you did get in|to this state, we have seen this occasionly, but have no reproducer. Unfortunately the csn for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you will probably not remember
> > > ||
> >
> > Not sure if its related, but I wrote an EMail to freeipa-users on that day,
> > reporting an internal error. See
> >
> > https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
>
> the csn of the replicaid 7 is from Jan,19th, two days later. maybe you did ad a new replica, or removed one or ...

Got confused by the "17". But maybe the problem was introduced when I
tried to examine or fix the conflict. Just a wild guess.
> Looks like you have three problems and I don't think they are related.
>
> 1] corrupt RUV element, as Thierry said, we believe to have an an explanation how it could occur before a specific fix, and since it is there for one year you didn't have that fix. To get rid of this you could apply cleanallruv as I suggested in a previous mail, if you do not want to do it, I think it was there for quite some time and doesn't do really any harm except raising errors in decoding it.

I will try the cleanallruv at the weekend.
> 2] conflicts, looks like you have conflitcts remaining which are relate to managed entries, these are espcially bad and difficult to handle. if you delete the conflict of an original entry it will delete the real managed entry and not the conflict of the managed entry. And the managed entry plugin is trying to prevent you to modify these entries directly.
> So what you can do is: determine which entries should be there, then delete them or rename them, if managed enty plugin gets in the way, either disable the MEP plugin temporarily or modify teh entries by deleting the mep objectclasses and attributes, cleanup the conflicts and fix them again

I have the impression that the entry "cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de"
has been lost. Is it?

If I got you correctly I have to rename

cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de
to
    cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de

Next I have to add

memberOf: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
to
    dn: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de

The entry

dn: cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de

can be deleted then. If the mep complains, then I could try to delete
the mepManagedBy entry and add it later again.

Is this correct?
> 3] Retro Changelog errors:   An error occured while adding change number 14065299 .....
> This looks like the retro changelog did mess up the counter to generate the next entry. a restart should hopefully reset it and let the RCL continue

The restart did not help, but I could re-initialize ipa1 and ipa4 from
ipa2. They appear to be in sync again, but I would like to make sure
that they *stay* in sync.

Thanx for your support
Harri

[Ludwig Krispenz]

On 03/14/2018 09:10 AM, Harald Dunkel via FreeIPA-users wrote:
> Hi Ludwig,
>
> On 03/13/18 14:47, Ludwig Krispenz via FreeIPA-users wrote:
> > On 03/13/2018 09:07 AM, Harald Dunkel via FreeIPA-users wrote:
> > > Hi Ludwig,
> > >
> > > On 03/12/18 17:10, Ludwig Krispenz via FreeIPA-users wrote:
> > > > Hi,
> > > >
> > > > to get rid of this ruv entry with replicaid 7 you could try to run 
> > > > the cleanallruv task directly. On any server (and onöy on one) run
> > > >
> > > > ldapmodify ..... -D "cn=directory manager"
> > > >
> > > > |dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config changetype: 
> > > > add objectclass: extensibleObject replica-base-dn: <your suffix > 
> > > > replica-id: 7 replica-force-cleaning: yes |
> > > >
> > > > |But I would like to understand how you did get in|to this state, 
> > > > we have seen this occasionly, but have no reproducer. Unfortunately 
> > > > the csn for replicaid 7 is from Jan, 19th 2017 11:01:16 - so you 
> > > > will probably not remember
> > > > ||
> > >
> > > Not sure if its related, but I wrote an EMail to freeipa-users on 
> > > that day,
> > > reporting an internal error. See
> > >
> > > https://www.redhat.com/archives/freeipa-users/2017-January/msg00286.html
> >
> > the csn of the replicaid 7 is from Jan,19th, two days later. maybe 
> > you did ad a new replica, or removed one or ...
>
> Got confused by the "17". But maybe the problem was introduced when I
> tried to examine or fix the conflict. Just a wild guess.

yes, after that time it is difficult to make any sound statements
> > Looks like you have three problems and I don't think they are related.
> >
> > 1] corrupt RUV element, as Thierry said, we believe to have an an 
> > explanation how it could occur before a specific fix, and since it is 
> > there for one year you didn't have that fix. To get rid of this you 
> > could apply cleanallruv as I suggested in a previous mail, if you do 
> > not want to do it, I think it was there for quite some time and 
> > doesn't do really any harm except raising errors in decoding it.
>
> I will try the cleanallruv at the weekend.

good
> > 2] conflicts, looks like you have conflitcts remaining which are 
> > relate to managed entries, these are espcially bad and difficult to 
> > handle. if you delete the conflict of an original entry it will 
> > delete the real managed entry and not the conflict of the managed 
> > entry. And the managed entry plugin is trying to prevent you to 
> > modify these entries directly.
> > So what you can do is: determine which entries should be there, then 
> > delete them or rename them, if managed enty plugin gets in the way, 
> > either disable the MEP plugin temporarily or modify teh entries by 
> > deleting the mep objectclasses and attributes, cleanup the conflicts 
> > and fix them again
>
> I have the impression that the entry 
> "cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de"
> has been lost. Is it?
>
> If I got you correctly I have to rename
> cn=ipaservers+nsuniqueid=109be304-ccd911e6-a5b3d0c8-d8da17db,cn=ng,cn=alt,dc=example,dc=de 
>
> to
>     cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
>
> Next I have to add
> memberOf: cn=ipaservers,cn=ng,cn=alt,dc=example,dc=de
>
> to
>     dn: cn=ipaservers,cn=hostgroups,cn=accounts,dc=example,dc=de
>
> The entry
> dn: 
>
> cn=ipaservers+nsuniqueid=109be302-ccd911e6-a5b3d0c8-d8da17db,cn=hostgroups,cn=accounts,dc=example,dc=de
>
> can be deleted then. If the mep complains, then I could try to delete
> the mepManagedBy entry and add it later again.
>
> Is this correct?

yes, that should work
> > 3] Retro Changelog errors:   An error occured while adding change 
> > number 14065299 .....
> > This looks like the retro changelog did mess up the counter to 
> > generate the next entry. a restart should hopefully reset it and let 
> > the RCL continue
>
> The restart did not help, but I could re-initialize ipa1 and ipa4 from
> ipa2. They appear to be in sync again, but I would like to make sure
> that they *stay* in sync.

ok, if it happens again we should investigate more
> Thanx for your support
> Harri
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]

[Harald Dunkel]

PS: I see tons of error messages like

:
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.819967301 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.824391203 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.827605284 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.831834938 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.834895574 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.839071959 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.842392544 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.846665902 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.850143845 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.854698721 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.858155502 +0100] - ERR - DSRetroclPlugin - retrocl_postob - Operation failure [68]
Mar 12 22:38:42 ipa1 ns-slapd: [12/Mar/2018:22:38:42.862632097 +0100] - ERR - DSRetroclPlugin - write_replog_db - An error occured while adding change number 14065299, dn = changenumber=14065299,cn=changelog: Already exists.
:

on ipa1

Regards
Harri