new/fresh server can't be setup as replica due to "The remote replica has a different database generation ID ..."

[Chris Dagdigian]

Hi folks,

Related to my posts from earlier in the week. I'm stuck in catch-22 land 
with no seemingly viable way forward ...

I am stuck with 2x IPA masters in different AWS regions that refuse to 
replicate because the topology is disconnected, I can't seem to force 
the re-connect so I'm trying to expand my topology options by building 
new fresh masters from scratch.  CentOS 7.3 with fully updated IPA 
software.

The fresh replica install fails with a "Local LDAP" error, these seem to 
be the corresponding errors in the /var/log/dirserv logs:

[02/Jun/2017:14:29:31.965022647 +0000] 389-Directory/1.3.5.10 
B2017.145.2037 starting up
[02/Jun/2017:14:29:31.976521839 +0000] default_mr_indexer_create: 
warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match
[02/Jun/2017:14:29:32.102416271 +0000] slapd started.  Listening on All 
Interfaces port 389 for LDAP requests
[02/Jun/2017:14:29:32.104077504 +0000] Listening on All Interfaces port 
636 for LDAPS requests
[02/Jun/2017:14:29:32.105380691 +0000] Listening on 
/var/run/slapd-companyIDM-ORG.socket for LDAPI requests
[02/Jun/2017:14:29:35.776066609 +0000] NSMMReplicationPlugin - 
agmt="cn=meTodeawilidmp001.companyidm.org" (deawilidmp001:389): The 
remote replica has a different database generation ID than the local 
database.  You may have to reinitialize the remote replica, or the local 
replica.

And here is the output from trying to perform the replica setup:

[root@usaeilidmp003 centos]# ipa-replica-install --setup-ca --principal 
admin --admin-password SEKRIT

Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Discovery was successful!
Client hostname: usaeilidmp003.companyidm.org
Realm: companyIDM.ORG
DNS Domain: companyidm.org
IPA Server: deawilidmp001.companyidm.org
BaseDN: dc=companyidm,dc=org

Skipping synchronizing time with NTP server.
Enrolled in IPA realm companyIDM.ORG
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm companyIDM.ORG
trying https://deawilidmp001.companyidm.org/ipa/json
Forwarding 'schema' to json server 
'https://deawilidmp001.companyidm.org/ipa/json'
trying https://deawilidmp001.companyidm.org/ipa/session/json
Forwarding 'ping' to json server 
'https://deawilidmp001.companyidm.org/ipa/session/json'
Forwarding 'ca_is_enabled' to json server 
'https://deawilidmp001.companyidm.org/ipa/session/json'
Systemwide CA database updated.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 
'https://deawilidmp001.companyidm.org/ipa/session/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring companyidm.org as NIS domain.
Client configuration complete.

Run connection check to master
Connection check OK
Configuring NTP daemon (ntpd)
   [1/4]: stopping ntpd
   [2/4]: writing configuration
   [3/4]: configuring ntpd to start on boot
   [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 1 minute
   [1/44]: creating directory server user
   [2/44]: creating directory server instance
   [3/44]: updating configuration in dse.ldif
   [4/44]: restarting directory server
   [5/44]: adding default schema
   [6/44]: enabling memberof plugin
   [7/44]: enabling winsync plugin
   [8/44]: configuring replication version plugin
   [9/44]: enabling IPA enrollment plugin
   [10/44]: enabling ldapi
   [11/44]: configuring uniqueness plugin
   [12/44]: configuring uuid plugin
   [13/44]: configuring modrdn plugin
   [14/44]: configuring DNS plugin
   [15/44]: enabling entryUSN plugin
   [16/44]: configuring lockout plugin
   [17/44]: configuring topology plugin
   [18/44]: creating indices
   [19/44]: enabling referential integrity plugin
   [20/44]: configuring certmap.conf
   [21/44]: configure autobind for root
   [22/44]: configure new location for managed entries
   [23/44]: configure dirsrv ccache
   [24/44]: enabling SASL mapping fallback
   [25/44]: restarting directory server
   [26/44]: creating DS keytab
   [27/44]: retrieving DS Certificate
   [28/44]: restarting directory server
   [29/44]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 15 seconds elapsed
[deawilidmp001.companyidm.org] reports: Update failed! Status: [-2  - 
LDAP error: Local error]

[error] RuntimeError: Failed to start replication
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(Replica): ERROR    Failed to 
start replication
ipa.ipapython.install.cli.install_tool(Replica): ERROR    The 
ipa-replica-install command failed. See /var/log/ipareplica-install.log 
for more information
[root@usaeilidmp003 centos]#

\