Client part of server install failing - KRB5CCNAME not defined in HTTP request environment

[Simon Williams]

I am having an issue attempting to install IPA Server.  The server
component install processes correctly, but when it comes to set up the
client components it fails:

2020-04-28T22:41:42Z DEBUG failed to find session_cookie in persistent
storage for principal 'host/[e-mail]'
2020-04-28T22:41:42Z INFO trying https://ipa.mydomain.com/ipa/json
2020-04-28T22:41:42Z DEBUG Created connection context.rpcclient_1954644240
2020-04-28T22:41:42Z INFO [try 1]: Forwarding 'schema' to json server
'https://ipa.mydomain.com/ipa/json'
2020-04-28T22:41:42Z DEBUG New HTTP connection (ipa.mydomain.com)
2020-04-28T22:41:53Z DEBUG HTTP connection destroyed (ipa.mydomain.com)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 732, in
single_request
    response.msg)
ProtocolError: <ProtocolError for ipa.mydomain.com/ipa/json: 500 Internal
Server Error>
2020-04-28T22:41:53Z DEBUG Destroyed connection context.rpcclient_1954644240
2020-04-28T22:41:53Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line
319, in run
    return cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
360, in run
    return self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
386, in execute
    for rval in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
655, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
421, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line
418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line
59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line
65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
3671, in main
    install(self)
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
2392, in install
    _install(options)
  File "/usr/lib/python2.7/site-packages/ipaclient/install/client.py", line
2734, in _install
    api.finalize()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 739, in
finalize
    self.__do_if_not_done('load_plugins')
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in
__do_if_not_done
    getattr(self, name)()
  File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 619, in
load_plugins
    for package in self.packages:
  File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 949, in
packages
    ipaclient.remote_plugins.get_package(self),
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py",
line 134, in get_package
    plugins = schema.get_package(server_info, client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line
553, in get_package
    schema = Schema(client)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line
401, in __init__
    fingerprint, ttl = self._fetch(client, ignore_cache=read_failed)
  File
"/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line
426, in _fetch
    schema = client.forward(u'schema', **kwargs)['result']
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 1190, in
forward
    raise NetworkError(uri=server, error=e.errmsg)

2020-04-28T22:41:53Z DEBUG The ipa-client-install command failed,
exception: NetworkError: cannot connect to 'https://ipa.mydomain.com/ipa/json':
Internal Server Error

The relevant services appear to be running

certmonger.service         loaded active running   Certificate monitoring
and
  [e-mail] loaded active running   389 Directory S
  gssproxy.service           loaded active running   GSSAPI Proxy Daemon
  httpd.service              loaded active running   The Apache HTTP Server
  ipa-custodia.service       loaded active running   IPA Custodia Service
  ipa-dnskeysyncd.service    loaded active running   IPA key daemon
  ipa.service                loaded active exited    Identity, Policy, Audit
  kadmin.service             loaded active running   Kerberos 5
Password-changin
  krb5kdc.service            loaded active running   Kerberos 5 KDC
  named-pkcs11.service       loaded active running   Berkeley Internet Name
Doma
  ntpd.service               loaded active running   Network Time Service
  oddjobd.service            loaded active running   privileged operations
for u
  [e-mail] loaded active running   PKI Tomcat Server
pki-t

I can use kinit to obtain a ticket for admin, but any ipa command that I
attempt to run gives an error along the following lines

ipa: DEBUG: failed to find session_cookie in persistent storage for
principal '[e-mail]'
ipa: INFO: trying https://ipa.mydomain.com/ipa/json
ipa: DEBUG: Created connection context.rpcclient_1964217648
ipa: INFO: [try 1]: Forwarding 'schema' to json server 'https://ipa.mydomain
.com/ipa/json'
ipa: DEBUG: New HTTP connection (ipa.mydomain.com)
ipa: DEBUG: HTTP connection destroyed (ipa.mydomain.com)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 732, in
single_request
    response.msg)
ProtocolError: <ProtocolError for ipa.mydomain.com/ipa/json: 500 Internal
Server Error>
ipa: DEBUG: Destroyed connection context.rpcclient_1964217648
ipa: ERROR: cannot connect to 'https://ipa.mydomain.com/ipa/json': Internal
Server Error

In the httpd error log, I see the same error for every ipa command issued

[Wed Apr 29 14:51:19.119357 2020] [:error] [pid 8505] ipa: ERROR: 500
Internal Server Error: KerberosWSGIExecutioner.__call__: KRB5CCNAME not
defined in HTTP request environment
[Wed Apr 29 14:51:19.120223 2020] [:error] [pid 8505] [remote
192.168.0.2:16498] mod_wsgi (pid=8505): Exception occurred processing WSGI
script '/usr/share/ipa/wsgi.py'.
[Wed Apr 29 14:51:19.120335 2020] [:error] [pid 8505] [remote
192.168.0.2:16498] RuntimeError: response has not been started

The same error is present at the time the install failed.

The Kerberos ticket is valid as ldapsearch works using it

[root@ipa1 ~]# ldapsearch -h ipa.mydomain.com -b ou=people,o=ipaca -Y
GSSAPI -s sub "(uid=admin)" dn uid
SASL/GSSAPI authentication started
SASL username: [e-mail]
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <ou=people,o=ipaca> with scope subtree
# filter: (uid=admin)
# requesting: dn uid
#

# admin, people, ipaca
dn: uid=admin,ou=people,o=ipaca
uid: admin

# search result
search: 4
result: 0 Success

# numResponses: 2
# numEntries: 1

but doesn't without it

[root@ipa1 ~]# ldapsearch -h ipa.mydomain.com -b ou=people,o=ipaca -s sub
"(uid=admin)" dn uid
SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
additional info: SASL(-4): no mechanism available:

Does anyone have any ideas? I'm tearing my hair out here!