Weub UI fails with "Login failed due to an unknown reason."

[Thomas Boroske]

Dear FreeIPA experts, 

our FreeIPA installation has not allowed anyone to login using username/password into the
web UI for some days. 

At first only the secondary ipa server failed in that way, now its both. 

I have found several similar threads in the mailing list but the underlying problem always
seemed to be something else. 

I tried to debug the problem. Looking into /var/log/httpd/error_log I see that each login
attempt generates the following error:

CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_9920 -X
X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X
X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit
status 1

Trying the kinit command on console fails because it asks me for a password for
wellknown/anonymous that I do not know, I guess that is because the armor_9920 file does
not exist. I do not know if that file exists when the webgui runs the kinit, but I guess
it does and something else fails later.

We are running FreeIPA 4.6.6.11-el7.centos on centos 7.8.

 kinit -k -t /var/lib/ipa/gssproxy/http.keytab HTTP/ipa1.ida.ing.tu-bs.de(a)NET.IDA

does not give an error. 

We are running a CA-less FreeIPA install as far as I understand, the httpd on ipa1 shows a
valid certificate signed by an external (to us) cert authority. 

Any help is much appreciated, 

Thomas Boroske