Need a clue re: broken topology and broken replication in a simple 2-server setup

[Chris Dagdigian]

Situation: Two servers housed in 2 different AWS regions are completely 
disconnected and totally out of sync. I can't fix replication at all so 
I'm looking for clues or tips ..

Backstory:
  - Complex Active Directory needs including transitive trusts across 
multiple child domains of the AD Forrest
  - This means that we've been constantly upgrading IPA and subsystems 
like sssd* given the speed at which AD integration is being improved/fixed
  - We've been doing "yum update" and "ipa-server-upgrade"
commands all 
the way from ipa-3.x to current v4.4.0
  - Due to incremental upgrades over time we've been at "domain level 0" 
until very recently

Issues
  - Two servers work but they are islands to their own - no replication 
seems to be occurring
  - IPA connection-check scripts seem to all pass
  - IPA replication-manage "list" commands seem to work fine
  - forcing replication or forcing a complete reinit has zero effect
  - IPA topologysegment-find domain commands seem to show the proper 
segments
  - BUT -- the topology-verify command clearly shows broken topology and 
disconnected state

It was only recently that I discovered the broken topology status - had 
spent too much time in the weeds looking at debug output trying to 
figure out why replication was not working .

I'm wondering what the best next-step is to regaining a unified IPA 
view. From reading the admin guide I'm thinking that I need to bring up 
new IPA servers so that I have more "nodes" to play with when 
potentially connecting and fixing the topology segments -- seems easier 
to fix segments when you have more nodes to play with.

I'm not sure what to fix first -- is the broken topology segment the 
cause for broken replication or is something wrong in the replication 
internals that results in a disconnected topology?

Guidance appreciated. I'm appending some redacted command output below.

Regards,
Chris

###


# ipa-replica-manage list
us-idmp001.COMPANYidm.org: master
eu-idmp001.COMPANYidm.org: master


# ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
   Segment name: us-idmp001.COMPANYidm.org-to-eu-idmp001.COMPANYidm.org
   Left node: us-idmp001.COMPANYidm.org
   Right node: eu-idmp001.COMPANYidm.org
   Connectivity: left-right
----------------------------
Number of entries returned 1
----------------------------


#ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
   Segment name: eu-idmp001.COMPANYidm.org-to-us-idmp001.COMPANYidm.org
   Left node: eu-idmp001.COMPANYidm.org
   Right node: us-idmp001.COMPANYidm.org
   Connectivity: left-right
----------------------------
Number of entries returned 1
----------------------------
[root@eu-idmp001 centos]#


# ipa topologysuffix-verify domain
========================================================
Replication topology of suffix "domain" contains errors.
========================================================
------------------------
Topology is disconnected
------------------------
   Server eu-idmp001.COMPANYidm.org can't contact servers: 
us-idmp001.COMPANYidm.org
[root@us-idmp001 centos]#


# ipa topologysuffix-verify domain
========================================================
Replication topology of suffix "domain" contains errors.
========================================================
------------------------
Topology is disconnected
------------------------
   Server us-idmp001.COMPANYidm.org can't contact servers: 
eu-idmp001.COMPANYidm.org
[root@eu-idmp001 centos]#
[root@eu-idmp001 centos]#


#  /usr/sbin/ipa-replica-conncheck --replica eu-idmp001.COMPANYidm.org
Check connection from master to remote replica 'eu-idmp001.COMPANYidm.org':
    Directory Service: Unsecure port (389): OK
    Directory Service: Secure port (636): OK
    Kerberos KDC: TCP (88): OK
    Kerberos KDC: UDP (88): WARNING
    Kerberos Kpasswd: TCP (464): OK
    Kerberos Kpasswd: UDP (464): WARNING
    HTTP Server: Unsecure port (80): OK
    HTTP Server: Secure port (443): OK
The following UDP ports could not be verified as open: 88, 464
This can happen if they are already bound to an application
and ipa-replica-conncheck cannot attach own UDP responder.

Connection from master to replica is OK.


ipa-replica-conncheck --master us-idmp001.COMPANYidm.org
Check connection from replica to remote master 'us-idmp001.COMPANYidm.org':
    Directory Service: Unsecure port (389): OK
    Directory Service: Secure port (636): OK
    Kerberos KDC: TCP (88): OK
    Kerberos Kpasswd: TCP (464): OK
    HTTP Server: Unsecure port (80): OK
    HTTP Server: Secure port (443): OK

The following list of ports use UDP protocol and would need to be
checked manually:
    Kerberos KDC: UDP (88): SKIPPED
    Kerberos Kpasswd: UDP (464): SKIPPED

Connection from replica to master is OK.
Start listening on required ports for remote master check
Listeners are started. Use CTRL+C to terminate the listening part after 
the test.

Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica eu-idmp001.COMPANYidm.org