Here’s my workaround:

It appears that this happens only when using commercial certs. It's trying to fetch the Directory Manager password (encrypted) from the primary to put it in the new sysstem. I commented out custodiainstance.py:211,

    def import_dm_password(self):
        cli = self._get_custodia_client()
#        cli.fetch_key('dm/DMHash')                                                                                    <

and copied it manually.

On the primary, open /etc/dirsrv/slapd-CS-RUTGERS-EDU/dse.ldif. Look for

nsslapd-rootpw: {SSHA}

It should be under cn=config. Now shutdown ipa on the new server (ipactl stop), edit /etc/dirsrv/slapd-CS-RUTGERS-EDU/dse.ldif, and replace that line with the one you copied from the original server. Restart ipa.

On Jan 24, 2020, at 7:52 PM, Charles Hedrick <hedrick@rutgers.edu> wrote:

We are moving from Centos 7 to 8. I did a test on copies and it worked with 8.0. i made the mistake of doing it on the production servers under 8.1. It fails.

I removed one server and recreated it as a replica. It worked fine. However the second one failed near the end of the process:

Restart of krb5kdc.service complete
Waiting up to 300 seconds to see our keys appear on host ldap://krb1.cs.rutgers.edu
Starting new HTTPS connection (1): krb1.cs.rutgers.edu:443
https://krb1.cs.rutgers.edu:443 "GET /ipa/keys/dm/DMHash?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.KgdU3jtIIC3bRIoqToXzmZIl3QFUKqbrBbT0sBerqmR2YWNWQTEp8ABbTSHINOUhtgPubXhwaAsqPzXTee3urtrK6lmf9wJ6OkecdVPY1PS9sWhMNUz4gEJkR-vVM8bN6gfk4g2Lc8jq2o2LMFloNMgCqUQyeRuiec09NsjIvR8X18xYQfXJXvlhuz-d2OJW1CsKO6_T1z8O_vsxlZ-vAeB8j3dbZiXJOlzdcxYYqjMHY-IM4LroUzCVNXtHloiq28e6R-uVTX9O7ActEbiSy6UePgE76K0cWVl1kJyHFozEZChH1_rzCgP6zdhAf8QqPOdde_860nxIUmroRuECjA.gnnrHcTs9ucgqLntquJltw.GAWBOG_aMTgwzwxQqSIFrThgTTiqg3fM3POZWccCqqs3PiwJq5vW2S-tF9VsV1topXcRdlKb6fUOyjE6wrffJ5hYRyE1c3ocAlG3QTVC8QWRn7Ol_IfoVfW-hTe-cAhELcdIOIEand_BYjSTEO6rDXv83iXRFxwno9ZYYppF8bQY7EC1r_wW5xTdXftILCDmkJbhXmGPnlCQ2Ah9cG3qZAKNBRsvk400_kRQec-4LBKWGYYd0y56zd6-PpcVO6p72AldDF_YoeettzaaxbYyH0bRFt7y9aHH3GaD5BOkVp_ZgSHZWbWf8-2zB76f1OKrz6TktCfcb4_ChUZ6BZZ41MX6T06Xjp3ft6p5KzPfY_gUq0fKWWESHMLOEZg8fAl15l9ZwMiRmpd1PZW3oLVxF3rO94OM4H7_8WVehrcO3dAuAVA7_ykmIKv-WBWvjNHbsXXTyb76a2ka2WYuVxeKGMklEyQgOaMPJa7BqSOCiPljt7juTXAMGRupuDG62bP9PdFQkervv4p_9wvwpEZkuWPLlHqgzrdspgBbQoXkbcyiv9qf7oyB_xHQaoMxlwfvGwlNu8Go9t8oHJkalVdjxCPL-qG0GxKHuh0uFNYR0Z3uP545HkzVECv8uUkm08Jc.SCBVE0utvtniR8-8qAe02swg5GzDZxfN0O6JkKsWN2Y HTTP/1.1" 502 415
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 179, in execute
   return_value = self.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 340, in run
   return cfgr.run()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run
   return self.execute()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute
   for rval in self._executor():
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
   exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
   self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
   six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
   raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
   step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
   step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
   six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
   raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
   value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure
   next(executor)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
   exc_handler(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
   self._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception
   self.__parent._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
   six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
   raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception
   super(ComponentBase, self)._handle_exception(exc_info)
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
   six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
   raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
   step()
File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
   step = lambda: next(self.__gen)
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
   six.reraise(*exc_info)
File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
   raise value
File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
   value = gen.send(prev_value)
File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65, in _install
   for unused in self._installer(self.parent):
File "/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py", line 590, in main
   replica_install(self)
File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 402, in decorated
   func(installer)
File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 1298, in install
   custodia.import_dm_password()
File "/usr/lib/python3.6/site-packages/ipaserver/install/custodiainstance.py", line 211, in import_dm_password
   cli.fetch_key('dm/DMHash')
File "/usr/lib/python3.6/site-packages/ipaserver/secrets/client.py", line 120, in fetch_key
   r.raise_for_status()
File "/usr/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
   raise HTTPError(http_error_msg, response=self)

The ipa-replica-install command failed, exception: HTTPError: 502 Server Error: Proxy Error for url: https://krb1.cs.rutgers.edu/ipa/keys/dm/DMHash?xxxx
502 Server Error: Proxy Error for url: https://krb1.cs.rutgers.edu/ipa/keys/dm/DMHash?ccc

At this point I’m pretty much stuck.