ipa-client-install randomly fails with slapi_access_allowed does not allow WRITE to ipaProtectedOperation; write_keys!
Hi there, We have IPA (VERSION: 4.9.10, API_VERSION: 2.248) running on Alma Linux on 8.7 with total of 4 replicas. We're running in a cloud, so have an automated process in place where new instances automatically enrol to IPA when launching (they all use the same IPA user and fetch the password from a secrets manager). For a while now we have been seeing instances fail to enrol to IPA on random occasions, which is more pronounced when multiple instances are starting at the same time.
Each instance runs ipa-client-install, like below, when it starts: ipa-client-install --mkhomedir --ssh-trust-dns --domain=example.com -w${PASSW} -phost-enrollment --unattended --force-join --no-dns-sshfp
This sometimes fails with the following: Starting external process args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpdqzuq_ts', '-A', '-n', 'CA certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmpdqzuq_ts/pwdfile.txt'] Process finished, return code=0 stdout= stderr= failed to find session_cookie in persistent storage for principal 'host/ip-172-26-1-238.xxx@EXAMPLE.COM' trying https://ipa2.example.com/ipa/json New HTTP connection (ipa2.example.com) HTTP connection destroyed (ipa2.example.com) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 120, in get_package plugins = api._remote_plugins AttributeError: 'API' object has no attribute '_remote_plugins' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in get_auth_info response = self._sec_context.step() File "<decorator-gen-15>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in check_last_err return func(self, *args, **kwargs) File "<decorator-gen-5>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in catch_and_return_token return func(self, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in step return self._initiator_step(token=token) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in _initiator_step token) File "gssapi/raw/sec_contexts.pyx", line 244, in gssapi.raw.sec_contexts.init_sec_context gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM' File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in run return cfgr.run() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 3961, in main install(self) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2655, in install _install(options) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2972, in _install api.finalize() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in __do_if_not_done getattr(self, name)() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in load_plugins for package in self.packages: File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 128, in get_package plugins = schema.get_package(server_info, client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 546, in get_package schema = Schema(client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 395, in __init__ fingerprint, ttl = self._fetch(client, ignore_cache=read_failed) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 407, in _fetch client.connect(verbose=False) File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in connect conn = self.create_connection(*args, **kw) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in create_connection command([], {} ) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call return self.__request(name, args) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in __request verbose=self.__verbose >= 3, File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) The ipa-client-install command failed, exception: KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM' Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM' The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information This program will set up IPA client. Version 4.9.10
On IPA server the following pops up in logs: ERR - is_allowed_to_access_attr - [file ipa_pwd_extop.c, line 825]: slapi_access_allowed does not allow WRITE to ipaProtectedOperation;write_keys! ERR - ipapwd_getkeytab - [file ipa_pwd_extop.c, line 1714]: Not allowed to set keytab on [host/ip-172-26-1-238.xxx@EXAMPLE.COM]!
This doesn't happen every time - even when multiple instances are launched from the same image, some will fail and some will enrol successfully. It's worse when instances are in different cloud region that IPA (even when they are very close, network-wise, so latency shouldn't be an issue), but can still happen within the same region. For some reason, this has also become worse when we switched from forcing a specific IPA server (--server to ipa-client-install) to DNS auto-discovery. We commonly have situations where 5 instances try to launch at mostly the same time and try to enrol using 2 replicas - and all 5 will fail, both IPAs showing the same errors (as above).
We've run out of ideas of what to debug and how, so any clues would be appreciated.
In some cases the error message from ipa-client-install is different (but still thrown at certuril):
Starting external process args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpiat7ggvf', '-A', '-n', 'CA certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmpiat7ggvf/pwdfile.txt'] Process finished, return code=0 stdout= stderr= failed to find session_cookie in persistent storage for principal 'host/ip-172-22-1-106.xxx@EXAMPLE.COM' trying https://ipa2.example.com/ipa/json New HTTP connection (ipa2.example.com) HTTP connection destroyed (ipa2.example.com) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 120, in get_package plugins = api._remote_plugins AttributeError: 'API' object has no attribute '_remote_plugins' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in get_auth_info response = self._sec_context.step() File "<decorator-gen-15>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in check_last_err return func(self, *args, **kwargs) File "<decorator-gen-5>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in catch_and_return_token return func(self, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in step return self._initiator_step(token=token) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in _initiator_step token) File "gssapi/raw/sec_contexts.pyx", line 244, in gssapi.raw.sec_contexts.init_sec_context gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in run return cfgr.run() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 3961, in main install(self) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2655, in install _install(options) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2972, in _install api.finalize() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in __do_if_not_done getattr(self, name)() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in load_plugins for package in self.packages: File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 128, in get_package plugins = schema.get_package(server_info, client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 546, in get_package schema = Schema(client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 395, in __init__ fingerprint, ttl = self._fetch(client, ignore_cache=read_failed) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 407, in _fetch client.connect(verbose=False) File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in connect conn = self.create_connection(*args, **kw) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in create_connection command([], {} ) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call return self.__request(name, args) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in __request verbose=self.__verbose >= 3, File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) The ipa-client-install command failed, exception: KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information This program will set up IPA client. Version 4.9.10
However the error on IPA server side is the same.
On ma, 21 marras 2022, Paulina Budzon via FreeIPA-users wrote:
In some cases the error message from ipa-client-install is different (but still thrown at certuril):
Starting external process args=['/usr/bin/certutil', '-d', 'sql:/tmp/tmpiat7ggvf', '-A', '-n', 'CA certificate 1', '-t', 'C,,', '-a', '-f', '/tmp/tmpiat7ggvf/pwdfile.txt'] Process finished, return code=0 stdout= stderr=
certutil returned 0, so it is just fine. The output below is unrelated to certutil use; you can ignore certutil part. Please see more below.
failed to find session_cookie in persistent storage for principal 'host/ip-172-22-1-106.xxx@EXAMPLE.COM' trying https://ipa2.example.com/ipa/json New HTTP connection (ipa2.example.com) HTTP connection destroyed (ipa2.example.com) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 120, in get_package plugins = api._remote_plugins AttributeError: 'API' object has no attribute '_remote_plugins' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 644, in get_auth_info response = self._sec_context.step() File "<decorator-gen-15>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 167, in check_last_err return func(self, *args, **kwargs) File "<decorator-gen-5>", line 2, in step File "/usr/lib64/python3.6/site-packages/gssapi/_utils.py", line 127, in catch_and_return_token return func(self, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 521, in step return self._initiator_step(token=token) File "/usr/lib64/python3.6/site-packages/gssapi/sec_contexts.py", line 542, in _initiator_step token) File "gssapi/raw/sec_contexts.pyx", line 244, in gssapi.raw.sec_contexts.init_sec_context gssapi.raw.misc.GSSError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked
This is coming from an attempt to get a Kerberos service ticket using credentials for the user you are using to enroll this machine. Since you are passing '-w$password' and not any specific principal, this means it is the machine itself, hence we see
failed to find session_cookie in persistent storage for principal 'host/ip-172-22-1-106.xxx@EXAMPLE.COM'
this is fine at that point because we need a session cookie to talk to IPA server's API endpoint and we don't have any yet. So we attempt to kinit with that password you passed and fail.
'TGT has been revoked' error comes from your KDC on IPA master. Please check /var/log/krb5kdc.log on IPA server you connected to for this deployment. There should be one of explaining messages prior to rejection. It might be prefixed with 'PAC issue:' string
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) ipalib.errors.KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute return_value = self.run() File "/usr/lib/python3.6/site-packages/ipapython/install/cli.py", line 344, in run return cfgr.run() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.6/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.6/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.6/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 3961, in main install(self) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2655, in install _install(options) File "/usr/lib/python3.6/site-packages/ipaclient/install/client.py", line 2972, in _install api.finalize() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 753, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 432, in __do_if_not_done getattr(self, name)() File "/usr/lib/python3.6/site-packages/ipalib/plugable.py", line 632, in load_plugins for package in self.packages: File "/usr/lib/python3.6/site-packages/ipalib/__init__.py", line 952, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/__init__.py", line 128, in get_package plugins = schema.get_package(server_info, client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 546, in get_package schema = Schema(client) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 395, in __init__ fingerprint, ttl = self._fetch(client, ignore_cache=read_failed) File "/usr/lib/python3.6/site-packages/ipaclient/remote_plugins/schema.py", line 407, in _fetch client.connect(verbose=False) File "/usr/lib/python3.6/site-packages/ipalib/backend.py", line 69, in connect conn = self.create_connection(*args, **kw) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1064, in create_connection command([], {} ) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1276, in _call return self.__request(name, args) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 1243, in __request verbose=self.__verbose >= 3, File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 697, in single_request self.get_auth_info() File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 646, in get_auth_info self._handle_exception(e, service=service) File "/usr/lib/python3.6/site-packages/ipalib/rpc.py", line 605, in _handle_exception raise errors.KerberosError(message=unicode(e)) The ipa-client-install command failed, exception: KerberosError: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638932): TGT has been revoked The ipa-client-install command failed. See /var/log/ipaclient-install.log for more information This program will set up IPA client. Version 4.9.10
However the error on IPA server side is the same. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Thanks for your help!
This is coming from an attempt to get a Kerberos service ticket using credentials for the user you are using to enroll this machine. Since you are passing '-w$password' and not any specific principal, this means it is the machine itself, hence we see
I'm passing -phost-enrollment (host-enrollment is the user for the password in -w), should I be adding something more?
'TGT has been revoked' error comes from your KDC on IPA master. Please check /var/log/krb5kdc.log on IPA server you connected to for this deployment. There should be one of explaining messages prior to rejection. It might be prefixed with 'PAC issue:' string
There's nothing around this exact time, the only bits regarding a specific failed host that I could find are: krb5kdc[4526](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), a es128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: NEEDED_PREAUTH: host/ip-172-22-2-123.xxx@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required krb5kdc[4526](info): closing down fd 4 krb5kdc[4525](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: ISSUE: authtime 1669105826, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, host/ip-172-22-2-123.xxx@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
(Different host in the logs above than before, but we had this issue again this morning and it was easier to look up that older logs).
In terms of errors, the only one I could find in the logs was: krb5kdc[27486](Error): PAC issue: ipadb_get_principal failed.
This was logged right before the whole set of instances got their errors.
I did also notice that the some of the same hostnames exist in older Kerberos logs (hostnames will get repeated in our cloud env every now and then), could this be the cause? A host with a previously used hostname trying to enrol again? We have an automated process in place that calls host-del to IPA when an instance is terminated to delete it and its data from IPA, but maybe we should be clearing something from Kerberos directly too?
On ti, 22 marras 2022, Paulina Budzon via FreeIPA-users wrote:
Thanks for your help!
This is coming from an attempt to get a Kerberos service ticket using credentials for the user you are using to enroll this machine. Since you are passing '-w$password' and not any specific principal, this means it is the machine itself, hence we see
I'm passing -phost-enrollment (host-enrollment is the user for the password in -w), should I be adding something more?
'TGT has been revoked' error comes from your KDC on IPA master. Please check /var/log/krb5kdc.log on IPA server you connected to for this deployment. There should be one of explaining messages prior to rejection. It might be prefixed with 'PAC issue:' string
There's nothing around this exact time, the only bits regarding a specific failed host that I could find are: krb5kdc[4526](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), a es128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: NEEDED_PREAUTH: host/ip-172-22-2-123.xxx@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM, Additional pre-authentication required krb5kdc[4526](info): closing down fd 4 krb5kdc[4525](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25)}) 172.22.2.123: ISSUE: authtime 1669105826, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, host/ip-172-22-2-123.xxx@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
(Different host in the logs above than before, but we had this issue again this morning and it was easier to look up that older logs).
In terms of errors, the only one I could find in the logs was: krb5kdc[27486](Error): PAC issue: ipadb_get_principal failed.
Can you please share with me the log lines around this one?
Also,
ipa user-show --all --raw host-enrollment ipa trustconfig-show --all --raw
You can send them privately, if needed.
This was logged right before the whole set of instances got their errors.
I did also notice that the some of the same hostnames exist in older Kerberos logs (hostnames will get repeated in our cloud env every now and then), could this be the cause? A host with a previously used hostname trying to enrol again? We have an automated process in place that calls host-del to IPA when an instance is terminated to delete it and its data from IPA, but maybe we should be clearing something from Kerberos directly too?
If host is already enrolled, this would cause a problem to enroll unless you pass --force to ipa-client-install. This is unrelated to the issues you are seeing as you are getting the error in a different stage of the installer.
For reference to @freeipa-users, since I very much don’t like open threads that moved to private and were left unanswered.
Big thanks to Alexander for helping with debugging. It seems we are affected by https://pagure.io/freeipa/issue/9228 https://pagure.io/freeipa/issue/9228. To confirm this: we don’t have much in terms of Kerberos logs on the IPA server that the host initially enrolled to, but we can see "PAC issue: ipadb_get_principal failed” and "TGT has been revoked” errors for this host in Kerberos logs on the second IPA in this region, which I understand is a typical sign of this issue.
@Alexander - do you know if forcing —server to ipa-client-install would help as a temporary work-around to force the installation to only use a specific server?
Paulina Budzoń via FreeIPA-users wrote:
For reference to @freeipa-users, since I very much don’t like open threads that moved to private and were left unanswered.
Big thanks to Alexander for helping with debugging. It seems we are affected by https://pagure.io/freeipa/issue/9228. To confirm this: we don’t have much in terms of Kerberos logs on the IPA server that the host initially enrolled to, but we can see "PAC issue: ipadb_get_principal failed” and "TGT has been revoked” errors for this host in Kerberos logs on the second IPA in this region, which I understand is a typical sign of this issue.
@Alexander - do you know if forcing —server to ipa-client-install would help as a temporary work-around to force the installation to only use a specific server?
I think it should help. The downside is that the resulting configuration will be pinned to that one server. You'd need to go in afterward and manually tweak the configuration on each client to use DNS discovery again (at least krb5.conf and sssd.conf).
rob
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Paulina Budzon -
Paulina Budzoń
-
Rob Crittenden