Thanks for your swift response Rob,
My apologies, it looks like my superficial replication check was insufficient.
ipa-replica-manage -v list ipa2.mydom
ipa3.mydom: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (0) Replica acquired successfully: Incremental update succeeded
last update ended: 2018-02-01 11:47:10+00:00
ipa1.mydom: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (18) Replication error acquiring replica: Incremental update transient error. Backing off, will retry update later. (transient error)
last update ended: 1970-01-01 00:00:00+00:00
Which led me to check on the snowflake where I'm seeing
Feb 1 11:48:49 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:49.866140639 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.916537089 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer - Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.919314318 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged
Feb 1 11:48:52 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:52.922208937 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.956362678 +0000] - ERR - agmt="cn=meToipa1.mydom" (ipa1:389) - clcache_load_buffer - Can't locate CSN 5a687250000500100000 in the changelog (DB rc=-30988). If replication stops, the consumer may need to be reinitialized.
Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.959110311 +0000] - ERR - NSMMReplicationPlugin - changelog program - repl_plugin_name_cl - agmt="cn=meToipa1.mydom" (ipa1:389): CSN 5a687250000500100000 not found, we aren't as up to date, or we purged
Feb 1 11:48:55 ipa2 ns-slapd[9471]: [01/Feb/2018:11:48:55.961578933 +0000] - ERR - NSMMReplicationPlugin - send_updates - agmt="cn=meToipa1.mydom" (ipa1:389): Data required to update replica has been purged from the changelog. If the error persists the replica must be reinitialized.
The only obvious error (which I suspect is unrelated) I could spot in http land was:
[Thu Feb 01 10:40:39.686959 2018] [wsgi:error] [pid 7302:tid 140268792428288] [remote
10.70.64.26:57792] ipa: ERROR: plugin index generation failed: Supplied plugin directory path is not a directory
I'll aim to reinitialise the problem box based on this. Without wanting to make excuses for my ineptitude, are there any plans to increase visibility for replication issues to surface them more obviously?
Thank you so much for your guidance, hugely appreciated.
David