On 12/06/17 18:29, Mark Reynolds wrote:
On 06/12/2017 07:32 AM, Nick Campion via FreeIPA-users wrote:
Thanks Mark,
So this example is a user password change using kinit, the password has been changed on freeipa02 but not then replicated to the others. This happens for other records, but I don't have examples of these at the moment.
As far as I'm aware, there is no fractal replication set up.
IPA uses fractional replication, and it's possible these attributes are ignored/skipped. To confirm you can run this search on freeipa02:
ldapsearch -D "cn=directory manager" -W -b cn=config -xLLL objectclass=nsds5ReplicationAgreement
Freeipa02:
dn: cn=meTofreeipa01.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
cn: meTofreeipa01.mgmt.example.com
description: me to freeipa01.mgmt.example.com
nsDS5ReplicaBindMethod: SASL/GSSAPI
nsDS5ReplicaHost: freeipa01.mgmt.example.com
nsDS5ReplicaPort: 389
nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com
nsDS5ReplicaTransportInfo: LDAP
nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
nsds50ruv: {replicageneration} 57867ff5000000040000
nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389} 57867ffe000000040000 593693b7001100040000
nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389} 57867ffa000000030000 5930e345000200030000 nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389} 59355988000000050000 59369317000300050000 nsds5ReplicaEnabled: on nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp nsds5replicaTimeout: 120 nsruvReplicaLastModified: {replica 4 ldap://freeipa01.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 3 ldap://freeipa02.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 5 ldap://freeipa03.mgmt.example.com:389} 00000000 objectClass: nsds5replicationagreement objectClass: top objectClass: ipaReplTopoManagedAgreement ipaReplTopoManagedAgreementState: managed agreement - controlled by topology plugin nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090432Z nsds5replicaLastUpdateEnd: 20170613090432Z nsds5replicaChangesSentSinceStartup:: MzoxNTkvMjM5ODI0NyA1OjI0LzAg nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
dn: cn=meTofreeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample2Cdc\3Dcom,cn=mapping tree,cn=config cn: meTofreeipa03.mgmt.example.com objectClass: nsds5replicationagreement objectClass: top objectClass: ipaReplTopoManagedAgreement nsDS5ReplicaTransportInfo: LDAP nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com nsDS5ReplicaHost: freeipa03.mgmt.example.com nsds5replicaTimeout: 120 nsDS5ReplicaPort: 389 nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount description: me to freeipa03.mgmt.example.com nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount ipaReplTopoManagedAgreementState: managed agreement - controlled by topology plugin nsds5ReplicaEnabled: on nsds50ruv: {replicageneration} 57867ff5000000040000 nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389} 59355988000000050000 5936937b000200050000 nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389} 57867ffa000000030000 5930e345000200030000 nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389} 57867ffe000000040000 593693b7000c00040000 nsruvReplicaLastModified: {replica 5 ldap://freeipa03.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 3 ldap://freeipa02.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 4 ldap://freeipa01.mgmt.example.com:389} 00000000 nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090432Z nsds5replicaLastUpdateEnd: 20170613090432Z nsds5replicaChangesSentSinceStartup:: MzoxMzkvMTkxMDI0NSA0OjkwNS8wIA== nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
dn: cn=cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat description: cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindMethod: Simple nsDS5ReplicaCredentials: Redacted nsDS5ReplicaHost: freeipa01.mgmt.example.com nsDS5ReplicaPort: 389 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaTransportInfo: TLS nsds50ruv: {replicageneration} 57868040000000600000 nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389} 57868041000000600000 593692b2000000600000 nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389} 57868050000000610000 59355a39000400610000 nsruvReplicaLastModified: {replica 96 ldap://freeipa01.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 97 ldap://freeipa02.mgmt.example.com:389} 00000000 objectClass: top objectClass: nsds5replicationagreement objectClass: ipaReplTopoManagedAgreement ipaReplTopoManagedAgreementState: managed agreement - controlled by topology plugin nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090226Z nsds5replicaLastUpdateEnd: 20170613090226Z nsds5replicaChangesSentSinceStartup: nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
Then please share these entries so we can see how they are configured. Perhaps do this on freeipa01 as well for comparison.
Freeipa01:
dn: cn=freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexamle\2Cdc\3Dcom,cn=mapping tree,cn=config objectClass: nsds5replicationagreement objectClass: ipaReplTopoManagedAgreement objectClass: top cn: freeipa01.mgmt.example.com-to-freeipa03.mgmt.example.com nsDS5ReplicaHost: freeipa03.mgmt.example.com nsDS5ReplicaPort: 389 nsds5replicaTimeout: 300 nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com description: freeipa01.mgmt.example.com to freeipa03.mgmt.example.com ipaReplTopoManagedAgreementState: managed agreement - generated by topology plugin nsDS5ReplicaTransportInfo: LDAP nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsds50ruv: {replicageneration} 57867ff5000000040000 nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389} 59355988000000050000 593b4a2f000300050000 nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389} 57867ffa000000030000 5937cccd000300030000 nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389} 57867ffe000000040000 593b4b2f000700040000 nsruvReplicaLastModified: {replica 5 ldap://freeipa03.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 3 ldap://freeipa02.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 4 ldap://freeipa01.mgmt.example.com:389} 00000000 nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090421Z nsds5replicaLastUpdateEnd: 20170613090421Z nsds5replicaChangesSentSinceStartup:: NDoxMTM0MS8yNTkyNzgxMyA= nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
dn: cn=meTofreeipa02.mgmt.example.com,cn=replica,cn=dc\3Dipa\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config cn: meTofreeipa02.mgmt.example.com description: me to freeipa02.mgmt.example.com nsDS5ReplicaBindMethod: SASL/GSSAPI nsDS5ReplicaHost: freeipa02.mgmt.example.com nsDS5ReplicaPort: 389 nsDS5ReplicaRoot: dc=ipa,dc=example,dc=com nsDS5ReplicaTransportInfo: LDAP nsDS5ReplicatedAttributeList: (objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsDS5ReplicatedAttributeListTotal: (objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount nsds50ruv: {replicageneration} 57867ff5000000040000 nsds50ruv: {replica 3 ldap://freeipa02.mgmt.example.com:389} 57867ffa000000030000 5937ccd3000a00030000 nsds50ruv: {replica 4 ldap://freeipa01.mgmt.example.com:389} 57867ffe000000040000 593b4b2f000700040000 nsds50ruv: {replica 5 ldap://freeipa03.mgmt.example.com:389} 59355988000000050000 593b49d8000400050000 nsds5ReplicaEnabled: on nsds5ReplicaStripAttrs: modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp nsds5replicaTimeout: 120 nsruvReplicaLastModified: {replica 3 ldap://freeipa02.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 4 ldap://freeipa01.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 5 ldap://freeipa03.mgmt.example.com:389} 00000000 objectClass: nsds5replicationagreement objectClass: top objectClass: ipaReplTopoManagedAgreement ipaReplTopoManagedAgreementState: managed agreement - controlled by topology plugin nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090421Z nsds5replicaLastUpdateEnd: 20170613090421Z nsds5replicaChangesSentSinceStartup:: NDoxMDkyNy8yNTYzNjIwNiA1OjM3OC8wIDA6MTQvMCA= nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
dn: cn=masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat description: masterAgreement1-freeipa02.mgmt.example.com-pki-tomcat nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa02.mgmt.example.com-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindMethod: Simple nsDS5ReplicaCredentials: Redacted nsDS5ReplicaHost: freeipa02.mgmt.example.com nsDS5ReplicaPort: 389 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaTransportInfo: TLS nsds50ruv: {replicageneration} 57868040000000600000 nsds50ruv: {replica 97 ldap://freeipa02.mgmt.example.com:389} 57868050000000610000 59355a39000400610000 nsds50ruv: {replica 96 ldap://freeipa01.mgmt.example.com:389} 57868047000000600000 593b488f000000600000 nsruvReplicaLastModified: {replica 97 ldap://freeipa02.mgmt.example.com:389} 00000000 nsruvReplicaLastModified: {replica 96 ldap://freeipa01.mgmt.example.com:389} 00000000 objectClass: top objectClass: nsds5replicationagreement objectClass: ipaReplTopoManagedAgreement ipaReplTopoManagedAgreementState: managed agreement - controlled by topology plugin nsds5replicareapactive: 0 nsds5replicaLastUpdateStart: 20170613090225Z nsds5replicaLastUpdateEnd: 20170613090226Z nsds5replicaChangesSentSinceStartup:: OTY6MzM3LzAg nsds5replicaLastUpdateStatus: Error (0) Replica acquired successfully: Incremental update succeeded nsds5replicaUpdateInProgress: FALSE nsds5replicaLastInitStart: 19700101000000Z nsds5replicaLastInitEnd: 19700101000000Z
Freeipa01:
# dynamic-kepler, users, accounts, ipa.example.com dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com uid: dynamic-kepler krbLastPwdChange: 20170608170011Z krbPasswordExpiration: 20170608170011Z
Freeipa02:
# dynamic-kepler, users, accounts, ipa.example.com dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com uid: dynamic-kepler krbLastPwdChange: 20170608170021Z krbPasswordExpiration: 20170906170021Z
Freeipa03:
# dynamic-kepler, users, accounts, ipa.example.com dn: uid=dynamic-kepler,cn=users,cn=accounts,dc=ipa,dc=example,dc=com uid: dynamic-kepler krbLastPwdChange: 20170608170011Z krbPasswordExpiration: 20170608170011Z
Errors on Freeipa02:
[08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn: opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted opcsn=5938ac8b000600030000 [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn: opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted opcsn=5939402f000900030000 [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn: opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted opcsn=5939e009001000030000
The other nodes have no errors from this data.
Access logs:
Freeipa01:
[08/Jun/2017:01:46:50.635529447 +0000] replica_generate_next_csn: opcsn=5938ac8b000500030000 <= basecsn=5938ac8b000500040000, adjusted opcsn=5938ac8b000600030000 [08/Jun/2017:12:16:46.497249649 +0000] replica_generate_next_csn: opcsn=5939402f000500030000 <= basecsn=5939402f000800040000, adjusted opcsn=5939402f000900030000 [08/Jun/2017:23:38:48.197750001 +0000] replica_generate_next_csn: opcsn=5939e009000100030000 <= basecsn=5939e009000f00040000, adjusted opcsn=5939e009001000030000
This is from an error log :-)
Freeipa02:
Shows no logs "to" the other 2 nodes.
Well it would only show incoming connections, not outgoing.
Freeipa03:
[08/Jun/2017:17:10:06.343697044 +0000] conn=9237 fd=70 slot=70 connection from 192.168.0.12 to 192.168.0.13 [08/Jun/2017:19:54:05.025713675 +0000] conn=9665 fd=70 slot=70 connection from 192.168.0.12 to 192.168.0.13
Freeipa02 replication logging:
[09/Jun/2017:11:24:58.827281135 +0000] NSMMReplicationPlugin - csnplCommitALL: processing data csn 593964af000900030000
Repeats 800 - 900 time per second with a different csn.
It looks like its replicating to other replicas, but some updates are skipped. This again could be fractional replication "working".
If you look through freeipa01's access log what operation is this csn from: 5937cccd000f00030000 ? Could this be one of the password updates that is not replicated? This update is not sent to the other replicas that's why I'm asking.
I cant find that csn anywhere but the error log on freeipa02. Both servers are logging csn's around the same time, just not this one.
Cheers Nick
Thanks, Mark
On 08/06/17 15:45, Mark Reynolds wrote:
On 06/07/2017 10:58 AM, Nick Campion via FreeIPA-users wrote:
Hi all,
We have a 3 master setup that is failing to replicate changes from a particular node to the other IPA instances. The replication status says it's all fine, however the record hasn't been changed on the other servers. We've seen this on user password changes, adding hosts and services. The only thing we've found that seems to fix this temporarily is to re-initialize from the master with the changed record. A force-sync doesn't pick up the changed record.
What is the change you making, what attribute are you updating? Could it be possible that its being excluded by fractional replication? Or is it all changes?
Any errors in the logs on the nodes(good and bad): /var/log/dirsrv/slapd-INSTANCE/errors
Do you see replication sessions starting between the bad node and good ones? Are they talking? Check the access log ( /var/log/dirsrv/slapd-INSTANCE/access) on a good node and look for "connection from <BAD NODE IP address>"
Next would be to enable replication logging on the bad node and reproduce the problem (then disable repl logging right away), then send us the logs to look at. See https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/ht...
Regards, Mark
Not sure what logs would be helpful to diagnose what is happening in this setup.
# ipa-replica-manage -v list `hostname` freeipa03.mgmt.example.com: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2017-06-07 14:43:53+00:00 freeipa02.mgmt.example.com: replica last init status: None last init ended: 1970-01-01 00:00:00+00:00 last update status: Error (0) Replica acquired successfully: Incremental update succeeded last update ended: 2017-06-07 14:43:53+00:00
# ldapsearch -W -x -D "cn=directory manager" -b "cn=users,cn=accounts,dc=ipa,dc=example,dc=com" "nsds5ReplConflict=*" * nsds5ReplConflict Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=users,cn=accounts,dc=ipa,dc=example,dc=com> with scope subtree # filter: nsds5ReplConflict=* # requesting: * nsds5ReplConflict #
# search result search: 2 result: 0 Success
# numResponses: 1
Any help in what else can be checked or what logs would be helpful would be appreciated.
Thanks
Nick
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org