How to analyze large Multi Master Replication (test)-network?

[Eugen Lamers]

We are testing scenarios with Multi Master Replication (MMR) with many replicas. We need
to gather experience in this area.
The current scenario is a star topology with one central server with replication
agreements with all others, and the others only with the central server. It is scaled up
to 63 servers replicating with the central server. 
To analyze the behaviour in certain situations we would like to have a reliable indicator
for the status of the replications.

What we found was the attribute nsds50ruv of the replication agreement. RedHat
documentation says:

Information about the replication topology — all of the suppliers which are supplying
updates to each other and other replicas within the same replication group — is contained
in a set of metadata called the replica update vector (RUV). The RUV contains information
about the supplier like its ID and URL, its latest change state number for changes made on
the local server, and the CSN of the first change. Both suppliers and consumers store RUV
information, and they use it to control replication updates.

1. How is the attribute managed and used, and are there limitations for highly scaled MMR
scenarios?
2. Is it necessary that all replicas of all servers in the MMR are listed in this
attribute? Are the entries automatically distributed, and who controls this?
3. How do those entries have to look like for the certain server "types": certal
servier with replication agreements to all others (1) resp. the "others" (2)?
  (1) {replicageneration} <ID> + n * {replica <replica ID>
ldap://<FQDN>:389}
  (2) {replicageneration} <ID> + {replica <replica ID of central server>
ldap://<FQDN of central server>:389}
4. Some entries end with some kind of IDs, possibly replication generation IDs, some
don't. Why is that so?
5. What is the meaning of the identifier under the attribute nsruvreplicalastmodified;
i.e: replica 1 ldap://<FQDN>:389} 00000000
6. What is a replication group and is that a way to help handling a highly scaled MMR
scenario while limiting replication traffic and conflicts, or does "replication
group" mean the complete network of the servers in the topology?
7. What must be the order to set up the MMR network? First set up all other MMR server and
then setup the central server?
8. We have noticed that the entries are made only after a reboot of the central MMR
server. Is this correct or are there other steps that are required to configure the MMR?

Documentation read so far:
[RHC 10, Ch. 3.1.16.12] :
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
[RHA 10, Ch. 15]:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
https://www.port389.org/docs/389ds/design/csn-pending-lists-and-ruv-updat...
https://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html

Best regards,
Eugen

[Gordon Messmer]

On 2/26/21 5:22 AM, Eugen Lamers wrote:
>  Documentation read so far: 

It sounds like you're building a new replication monitor, so I'd think 
that a good place to start would be with a review of the existing one:

https://directory.fedoraproject.org/docs/389ds/howto/howto-monitor-replic...

https://directory.fedoraproject.org/docs/389ds/design/replication-monitor...

[Ivanov Andrey (M.)]

Hi,

Use the RHDS 11 documentation instead of 10, it's more up-to-date
(https://access.redhat.com/documentation/en-us/red_hat_directory_server/11).

Concerning the rpelication, you can check the whole chapter
https://access.redhat.com/documentation/en-us/red_hat_directory_server/11...

What you are trying to do (checking the consistency of LDAP replicas) is probably
completely or partially implemented by thefollowing two utilities :
* "ds-replcheck" that compares two replicas:
https://access.redhat.com/documentation/en-us/red_hat_directory_server/11...
* and "dsconf replication monitor" comparing just the time skew and the
coherence of RUVs
(https://access.redhat.com/documentation/en-us/red_hat_directory_server/11...)


In our production environment we check the state of replication from time to time by
ds-replcheck to be sure the replicas contain identical data.


As for the order of configuration, you can create replication agreements in any order,
then initialize them. The best practice is to initialize all the servers in MMR topology
from the same initial server. Something like this for 3 servers MMR with ldap1 as central
hub:

# Activate replicas and changelogs, create replication managers
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication create-manager --name
'cn=repman,cn=config' --passwd 'repman_secret_password'
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication enable --suffix='dc=example,dc=com'
--role='master' --replica-id=1 --bind-dn='cn=repman,cn=config'

/usr/sbin/dsconf ldaps://ldap2.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication create-manager --name
'cn=repman,cn=config' --passwd 'repman_secret_password'
/usr/sbin/dsconf ldaps://ldap2.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication enable --suffix='dc=example,dc=com'
--role='master' --replica-id=2 --bind-dn='cn=repman,cn=config'

/usr/sbin/dsconf ldaps://ldap3.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication create-manager --name
'cn=repman,cn=config' --passwd 'repman_secret_password'
/usr/sbin/dsconf ldaps://ldap3.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' replication enable --suffix='dc=example,dc=com'
--role='master' --replica-id=3 --bind-dn='cn=repman,cn=config'


# Create all MMR replication agreements
/usr/sbin/dsconf ldaps://ldap2.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt create --suffix='dc=example,dc=com'
--host='ldap1.example.com' --port=636 --conn-protocol=LDAPS
--bind-dn='cn=repman,cn=config' --bind-passwd='repman_secret_password'
--bind-method=SIMPLE 'Replication from ldap2.example.com to ldap1.example.com'
/usr/sbin/dsconf ldaps://ldap3.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt create --suffix='dc=example,dc=com'
--host='ldap1.example.com' --port=636 --conn-protocol=LDAPS
--bind-dn='cn=repman,cn=config' --bind-passwd='repman_secret_password'
--bind-method=SIMPLE 'Replication from ldap3.example.com to ldap1.example.com'
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt create --suffix='dc=example,dc=com'
--host='ldap2.example.com' --port=636 --conn-protocol=LDAPS
--bind-dn='cn=repman,cn=config' --bind-passwd='repman_secret_password'
--bind-method=SIMPLE 'Replication from ldap1.example.com to ldap2.example.com'
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt create --suffix='dc=example,dc=com'
--host='ldap3.example.com' --port=636 --conn-protocol=LDAPS
--bind-dn='cn=repman,cn=config' --bind-passwd='repman_secret_password'
--bind-method=SIMPLE 'Replication from ldap1.example.com to ldap3.example.com'

# Start initialization of replica ldap2 from ldap1
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt init --suffix='dc=example,dc=com'
'Replication from ldap1.example.com to ldap2.example.com'
# and wait for its end showing progression every 5 seconds
INITSTATE=`/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory
Manager' -w 'dir_man_secret_password' repl-agmt init-status
--suffix='dc=example,dc=com' 'Replication from ldap1.example.com to
ldap2.example.com'`; while [[ $INITSTATE == 'Agreement initialization in
progress.' ]]; do sleep 5; echo -n '.';INITSTATE=`/usr/sbin/dsconf
ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt init-status
--suffix='dc=id,dc=polytechnique,dc=edu' 'Replication from ldap1.example.com
to ldap2.example.com'`; done; echo; echo; echo '   
>>'$INITSTATE'<<'; echo


# Start initialization of replica ldap3 from ldap1
/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt init --suffix='dc=example,dc=com'
'Replication from ldap1.example.com to ldap3.example.com'
# and wait for its end showing progression every 5 seconds
INITSTATE=`/usr/sbin/dsconf ldaps://ldap1.example.com:636 -D 'cn=Directory
Manager' -w 'dir_man_secret_password' repl-agmt init-status
--suffix='dc=example,dc=com' 'Replication from ldap1.example.com to
ldap3.example.com'`; while [[ $INITSTATE == 'Agreement initialization in
progress.' ]]; do sleep 5; echo -n '.';INITSTATE=`/usr/sbin/dsconf
ldaps://ldap1.example.com:636 -D 'cn=Directory Manager' -w
'dir_man_secret_password' repl-agmt init-status
--suffix='dc=id,dc=polytechnique,dc=edu' 'Replication from ldap1.example.com
to ldap3.example.com'`; done; echo; echo; echo '   
>>'$INITSTATE'<<'; echo

Regards,
Andrey


----- Mail original -----
>  De: "Eugen Lamers" <eugen.lamers(a)br-automation.com&gt;
>  À: "General discussion list for the 389 Directory server, project."
> <389-users(a)lists.fedoraproject.org&gt;
>  Envoyé: Vendredi 26 Février 2021 14:22:03
>  Objet: [389-users] How to analyze large Multi Master Replication (test)-network? 
> We are testing scenarios with Multi Master Replication (MMR) with many replicas.
> We need to gather experience in this area.
> The current scenario is a star topology with one central server with replication
> agreements with all others, and the others only with the central server. It is
> scaled up to 63 servers replicating with the central server.
> To analyze the behaviour in certain situations we would like to have a reliable
> indicator for the status of the replications.
> 
> What we found was the attribute nsds50ruv of the replication agreement. RedHat
> documentation says:
> 
> Information about the replication topology — all of the suppliers which are
> supplying updates to each other and other replicas within the same replication
> group — is contained in a set of metadata called the replica update vector
> (RUV). The RUV contains information about the supplier like its ID and URL, its
> latest change state number for changes made on the local server, and the CSN of
> the first change. Both suppliers and consumers store RUV information, and they
> use it to control replication updates.
> 
> 1. How is the attribute managed and used, and are there limitations for highly
> scaled MMR scenarios?
> 2. Is it necessary that all replicas of all servers in the MMR are listed in
> this attribute? Are the entries automatically distributed, and who controls
> this?
> 3. How do those entries have to look like for the certain server "types":
certal
> servier with replication agreements to all others (1) resp. the "others"
(2)?
>  (1) {replicageneration} <ID> + n * {replica <replica ID>
ldap://<FQDN>:389}
>  (2) {replicageneration} <ID> + {replica <replica ID of central server>
>  ldap://<FQDN of central server>:389}
> 4. Some entries end with some kind of IDs, possibly replication generation IDs,
> some don't. Why is that so?
> 5. What is the meaning of the identifier under the attribute
> nsruvreplicalastmodified; i.e: replica 1 ldap://<FQDN>:389} 00000000
> 6. What is a replication group and is that a way to help handling a highly
> scaled MMR scenario while limiting replication traffic and conflicts, or does
> "replication group" mean the complete network of the servers in the
topology?
> 7. What must be the order to set up the MMR network? First set up all other MMR
> server and then setup the central server?
> 8. We have noticed that the entries are made only after a reboot of the central
> MMR server. Is this correct or are there other steps that are required to
> configure the MMR?
> 
> Documentation read so far:
> [RHC 10, Ch. 3.1.16.12] :
>
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
> [RHA 10, Ch. 15]:
>
https://access.redhat.com/documentation/en-us/red_hat_directory_server/10...
> https://www.port389.org/docs/389ds/design/csn-pending-lists-and-ruv-updat...
> https://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html
> 
> Best regards,
> Eugen
> _______________________________________________
> 389-users mailing list -- 389-users(a)lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure