I'm running 5 ipa servers with (the latest on CentOS 8) 4.9.2.

Synchronization had stopped yesterday and also 3 days ago.  It actually stopped yesterday after I stopped / modified / started "ipa1" to configure rotating logs longer so I could track down what happened 3 days ago.

2021-07-27 17:22:46 ipactl stop

2021-07-27 17:22:59 emacs dse.ldif    # Modify to access and error log rotation values

Below seems to be what kicked off the bad behavior.  I've seen a few posts about removing the keys out of dse.ldif when this happens.  I'm a bit leery of doing this, as I don't fully understand what is going on.  (is it comparable to clearing out known_host entries when using ssh?)

[27/Jul/2021:17:23:49.818525015 -0600] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
[27/Jul/2021:17:23:49.820422259 -0600] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the
 key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.040967207 -0600] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
[27/Jul/2021:17:23:50.043074553 -0600] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the
 key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.044268421 -0600] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
[27/Jul/2021:17:23:50.263786473 -0600] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
[27/Jul/2021:17:23:50.266090934 -0600] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.470918523 -0600] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
[27/Jul/2021:17:23:50.472915669 -0600] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[27/Jul/2021:17:23:50.474282471 -0600] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
[27/Jul/2021:17:23:50.891048127 -0600] - ERR - schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!

Then ipa1 can't talk to the replicas (ipa2,ipa3,ipa5,ipa6) shown below:

[27/Jul/2021:17:23:51.081696109 -0600] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.hpc.example.com@HPC.EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[27/Jul/2021:17:23:51.086755379 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa4.hpc.example.com" (ipa4:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:23:51.091748474 -0600] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.hpc.example.com@HPC.EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[27/Jul/2021:17:23:51.093430455 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=ipa1.hpc.example.com-to-ipa6.hpc.example.com" (ipa6:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:23:51.094725291 -0600] - ERR - schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds!
[27/Jul/2021:17:23:51.096059194 -0600] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.hpc.example.com@HPC.EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[27/Jul/2021:17:23:51.097152619 -0600] - INFO - slapd_daemon - slapd started.  Listening on All Interfaces port 389 for LDAP requests
[27/Jul/2021:17:23:51.098356748 -0600] - INFO - slapd_daemon - Listening on All Interfaces port 636 for LDAPS requests
[27/Jul/2021:17:23:51.099577958 -0600] - INFO - slapd_daemon - Listening on /var/run/slapd-HPC-EXAMPLE-COM.socket for LDAPI requests
[27/Jul/2021:17:23:51.100701349 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa3.hpc.example.com" (ipa3:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:23:51.101782194 -0600] - ERR - set_krb5_creds - Could not get initial credentials for principal [ldap/ipa1.hpc.example.com@HPC.EXAMPLE.COM] in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
[27/Jul/2021:17:23:51.103848248 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa5.hpc.example.com" (ipa5:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:23:58.152621025 -0600] - ERR - schema-compat-plugin - Finished plugin initialization.
[27/Jul/2021:17:24:21.201225830 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa2.hpc.example.com" (ipa2:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:24:21.203158794 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=ipa1.hpc.example.com-to-ipa6.hpc.example.com" (ipa6:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:24:21.204833314 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa3.hpc.example.com" (ipa3:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:24:21.206099975 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa5.hpc.example.com" (ipa5:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[27/Jul/2021:17:54:03.675297221 -0600] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa2.hpc.example.com" (ipa2:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()

After realizing I had a problem this morning, I rebooted ipa1 but it did not help syncing.  I re-initialized ipa1 from ipa3, this got them all authenticating to each other and in sync.

[28/Jul/2021:08:09:10.347094254 -0600] - INFO - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=caToipa3.hpc.inl.gov" (ipa3:389): Replication bind with GSSAPI auth resumed
[28/Jul/2021:08:09:10.449170075 -0600] - INFO - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa3.hpc.inl.gov" (ipa3:389): Replication bind with GSSAPI auth resumed
[....]

I changed the Data Manager password with "dsconf" -- but that was between the first failure and the second.  Could that be causing problems?  What direction to go from here?  Thank you!

Scott