One freeipa replica install fails, while other is going through

[D S]

Hello, 
I am trying to install 3 replicas agains the same master. Two out of 3 installs succeed, while the other one fails with

On replica:
Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Check RPC connection to remote master
Execute check on remote master
ERROR: Remote master check failed with following error message(s):
an internal error has occurred

2024-03-28T09:09:28Z DEBUG Starting external process
2024-03-28T09:09:28Z DEBUG args=['/usr/sbin/ipa-client-install', '--unattended', '--uninstall']
2024-03-28T09:09:31Z DEBUG Process finished, return code=0
2024-03-28T09:09:31Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in run
    return cfgr.run()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 358, in run
    self.validate()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 368, in validate
    for _nothing in self._validator():
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 633, in _configure
    next(validator)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/__init__.py", line 597, in main
    replica_promote_check(self)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 401, in decorated
    func(installer)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 423, in decorated
    func(installer)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/replicainstall.py", line 1196, in promote_check
    ca_cert_file=cafile)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/replication.py", line 129, in replica_conn_check
    "Connection check failed!"

2024-03-28T09:09:31Z DEBUG The ipa-replica-install command failed, exception: ScriptError: Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
2024-03-28T09:09:31Z ERROR Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
2024-03-28T09:09:31Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

On master:
[Thu Mar 28 09:09:27.891561 2024] [:error] [pid 22098] ipa: ERROR: non-public: DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[Thu Mar 28 09:09:27.891666 2024] [:error] [pid 22098] Traceback (most recent call last):
[Thu Mar 28 09:09:27.891683 2024] [:error] [pid 22098]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 369, in wsgi_execute
[Thu Mar 28 09:09:27.891694 2024] [:error] [pid 22098]     result = command(*args, **options)
[Thu Mar 28 09:09:27.891705 2024] [:error] [pid 22098]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 450, in __call__
[Thu Mar 28 09:09:27.891717 2024] [:error] [pid 22098]     return self.__do_call(*args, **options)
[Thu Mar 28 09:09:27.891727 2024] [:error] [pid 22098]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 478, in __do_call
[Thu Mar 28 09:09:27.891737 2024] [:error] [pid 22098]     ret = self.run(*args, **options)
[Thu Mar 28 09:09:27.891748 2024] [:error] [pid 22098]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 800, in run
[Thu Mar 28 09:09:27.891928 2024] [:error] [pid 22098]     return self.execute(*args, **options)
[Thu Mar 28 09:09:27.891951 2024] [:error] [pid 22098]   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 933, in execute
[Thu Mar 28 09:09:27.891962 2024] [:error] [pid 22098]     ret, stdout, _stderr = server.conncheck(keys[-1])
[Thu Mar 28 09:09:27.891973 2024] [:error] [pid 22098]   File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 70, in __call__
[Thu Mar 28 09:09:27.891983 2024] [:error] [pid 22098]     return self._proxy_method(*args, **keywords)
[Thu Mar 28 09:09:27.891994 2024] [:error] [pid 22098]   File "/usr/lib64/python2.7/site-packages/dbus/proxies.py", line 145, in __call__
[Thu Mar 28 09:09:27.892005 2024] [:error] [pid 22098]     **keywords)
[Thu Mar 28 09:09:27.892016 2024] [:error] [pid 22098]   File "/usr/lib64/python2.7/site-packages/dbus/connection.py", line 651, in call_blocking
[Thu Mar 28 09:09:27.892026 2024] [:error] [pid 22098]     message, timeout)
[Thu Mar 28 09:09:27.892037 2024] [:error] [pid 22098] DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
[Thu Mar 28 09:09:27.892955 2024] [:error] [pid 22098] ipa: INFO: [jsonserver_kerb] [e-mail]: server_conncheck(u'ipamaster01.example.com', u'ipa-replica03.example.com', version=u'2.162'): InternalError
[Thu Mar 28 09:09:30.121019 2024] [:error] [pid 20997] ipa: INFO: [jsonserver_kerb] host/[e-mail]: host_disable(u'ipa-replica03.example.com'): SUCCESS

[D S]

Any ideas on where to look next?

[Mauricio Tavares]

On Fri, Mar 29, 2024 at 4:58 AM D S via FreeIPA-users
 wrote:
> Any ideas on where to look next?

Replica-to-be log says connection check failed. Master seems to
say its replies are being ignored. Could the replica-to-be's firewall
be blocking things?

[Rob Crittenden]

D S via FreeIPA-users wrote:
> Any ideas on where to look next?

I believe you posed the same question on the freeipa-container package.

It might have helped if you'd posted here that you were using
containers, what underlying OS's were being used and the version of IPA.

Did you try pointing the 3rd replica to one of the other two?

Did you try with --skip-conncheck? The connection checking isn't always
perfect which is why the option exists.

rob