Hi everyone!
We've been experiencing some issues with our FreeIPA setup for the past few months. First of all:
Our package versions are:
ipa-client-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-client-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-client-epn-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-server-dns-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch
We are running a peculiar containerized environment based on the CentOS 8 image.
Specifically, we've been having trouble accessing the FreeIPA API and performing web UI logins, which we suspect is due to the /run/ipa/ccaches directory becoming littered with too many files. For example, on one of the troubled servers, we ran the command:
[root@ipa-server /]# ls -l /run/ipa/ccaches/ | wc -l 174314
We've already tried deleting files in the directory, but the problem persists. The errors we're seeing are something like this:
ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (69206018): gss_display_status call returned failure (major 327680, minor 100007). Decoding code: 69206018
Or this:
ipa: ERROR: No valid Negotiate header in server response
I tried looking up the code of mod_auth_gssapi to find the probable cause, but to no effect. I need help with this. First of all, shouldn't the STs in GssapiDelegCcacheDir be deleted by the module? For now the only solution is the container restart which is equivalent to the "ipactl restart" I guess.
I'm interested in learning more about how mod_auth_gssapi is handling ST deletion and what might be causing it to fail in general. If anyone has any insights or suggestions, we would greatly appreciate it.
On 27-03-2023 07:52, terrible person via FreeIPA-users wrote:
Hi everyone!
We've been experiencing some issues with our FreeIPA setup for the past few months. First of all:
Our package versions are:
ipa-client-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-client-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-client-epn-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-server-dns-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch
We are running a peculiar containerized environment based on the CentOS 8 image.
Specifically, we've been having trouble accessing the FreeIPA API and performing web UI logins, which we suspect is due to the /run/ipa/ccaches directory becoming littered with too many files. For example, on one of the troubled servers, we ran the command:
[root@ipa-server /]# ls -l /run/ipa/ccaches/ | wc -l 174314
We've already tried deleting files in the directory, but the problem persists. The errors we're seeing are something like this:
ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (69206018): gss_display_status call returned failure (major 327680, minor 100007). Decoding code: 69206018
Or this:
ipa: ERROR: No valid Negotiate header in server response
I tried looking up the code of mod_auth_gssapi to find the probable cause, but to no effect. I need help with this. First of all, shouldn't the STs in GssapiDelegCcacheDir be deleted by the module? For now the only solution is the container restart which is equivalent to the "ipactl restart" I guess.
I'm interested in learning more about how mod_auth_gssapi is handling ST deletion and what might be causing it to fail in general. If anyone has any insights or suggestions, we would greatly appreciate it.
Not only that. Why aren't old files cleaned up from that directory? There are two tmpfiles rules in /usr/lib/tmpfiles.d/ipa.conf
d /run/ipa/ccaches 06770 ipaapi ipaapi a+ /run/ipa/ccaches - - - - g:apache:rwx
But these don't cleanup anything.
Kees Bakker via FreeIPA-users wrote:
On 27-03-2023 07:52, terrible person via FreeIPA-users wrote:
Hi everyone!
We've been experiencing some issues with our FreeIPA setup for the past few months. First of all:
Our package versions are:
ipa-client-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-client-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-client-epn-4.9.8-7.module_el8.6.0+1103+a004f6a8.x86_64 ipa-server-common-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch ipa-server-dns-4.9.8-7.module_el8.6.0+1103+a004f6a8.noarch
We are running a peculiar containerized environment based on the CentOS 8 image.
Specifically, we've been having trouble accessing the FreeIPA API and performing web UI logins, which we suspect is due to the /run/ipa/ccaches directory becoming littered with too many files. For example, on one of the troubled servers, we ran the command:
[root@ipa-server /]# ls -l /run/ipa/ccaches/ | wc -l 174314
We've already tried deleting files in the directory, but the problem persists. The errors we're seeing are something like this:
ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (69206018): gss_display_status call returned failure (major 327680, minor 100007). Decoding code: 69206018
Or this:
ipa: ERROR: No valid Negotiate header in server response
I tried looking up the code of mod_auth_gssapi to find the probable cause, but to no effect. I need help with this. First of all, shouldn't the STs in GssapiDelegCcacheDir be deleted by the module? For now the only solution is the container restart which is equivalent to the "ipactl restart" I guess.
I'm interested in learning more about how mod_auth_gssapi is handling ST deletion and what might be causing it to fail in general. If anyone has any insights or suggestions, we would greatly appreciate it.
Not only that. Why aren't old files cleaned up from that directory? There are two tmpfiles rules in /usr/lib/tmpfiles.d/ipa.conf
d /run/ipa/ccaches 06770 ipaapi ipaapi a+ /run/ipa/ccaches - - - - g:apache:rwxBut these don't cleanup anything.
This was fixed in 4.9.11 upstream, https://pagure.io/freeipa/issue/9231
rob
Thank you, Rob. Would you happen to know why the IPA API errors I provided earlier are related to the issue of the /run/ directory being consumed? Although there is still plenty of space, freeing it does not seem to resolve the problem. I apologize for asking, but I noticed that there are maintainers of mod_auth_gssapi present who might be able to shed some light on the matter.
terrible person via FreeIPA-users wrote:
Thank you, Rob. Would you happen to know why the IPA API errors I provided earlier are related to the issue of the /run/ directory being consumed? Although there is still plenty of space, freeing it does not seem to resolve the problem. I apologize for asking, but I noticed that there are maintainers of mod_auth_gssapi present who might be able to shed some light on the matter.
mod_auth_gssapi is probably not the root of the problem.
What else have you tried doing?
rob
Hi Rob,
Apologies for the delay in responding to your message. I have been trying to investigate the issue by looking at the logs for KDC and httpd, but so far, I have not been able to find any useful information. I also attempted to enable debug mode in KDC, but it appears that there is no debug output available for the server.
The problem seems to occur when someone encounters it, such as when my colleagues are actively using certmonger and need to add hosts and certificates for their services. At times, the API suddenly throws error messages, and I have limited room to investigate the problem properly without resorting to restarting the service, which is not an ideal solution.
I was hoping that someone could provide me with a general dataflow model from the time a basic command, such as 'ipa ca-show ipa', is issued until the result is returned. While I have a basic understanding of the Kerberos exchange, I tend to get lost when Apache mod_auth_gssapi and gssproxy are involved in the exchange, and I am unsure of who is requesting what.
Maybe after that realization of where to look for the problem will come.
On la, 15 huhti 2023, terrible person via FreeIPA-users wrote:
Hi Rob,
Apologies for the delay in responding to your message. I have been trying to investigate the issue by looking at the logs for KDC and httpd, but so far, I have not been able to find any useful information. I also attempted to enable debug mode in KDC, but it appears that there is no debug output available for the server.
The problem seems to occur when someone encounters it, such as when my colleagues are actively using certmonger and need to add hosts and certificates for their services. At times, the API suddenly throws error messages, and I have limited room to investigate the problem properly without resorting to restarting the service, which is not an ideal solution.
I was hoping that someone could provide me with a general dataflow model from the time a basic command, such as 'ipa ca-show ipa', is issued until the result is returned. While I have a basic understanding of the Kerberos exchange, I tend to get lost when Apache mod_auth_gssapi and gssproxy are involved in the exchange, and I am unsure of who is requesting what.
Maybe after that realization of where to look for the problem will come.
Use search for this list. About a year ago I did a bit of explanation of the setup for these problems:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
(I registered internal account, can't login via openid for some reason)
Ty for thread link, this will definitely help I have been investigating a problem I'm facing further and it seems to stem from the gssproxy.service. However, it appears in two forms.
The first is a clear error form. When attempting to log in, it shows a message "Login failed due to an unknown reason". When doing a request via ipa cli, it shows the following error message:
[root@sandbox-dev-01 ~]# ipa ping ipa: ERROR: Could not create log_dir '/root/.ipa/log' ipa: ERROR: No valid Negotiate header in server response
In the httpd error_log, I can see a clear error:
[Mon Apr 17 19:31:36.132659 2023] [auth_gssapi:error] [pid 2539:tid 140105910048512] [client 172.28.46.83:38428] GSS ERROR In Negotiate Auth: gss_accept_sec_context() failed: [Unspecified GSS failure. Minor code may provide more information ( No such file or directory (filename: /var/lib/gssproxy/rcache/krb5_0.rcache2))], referer: https://sandbox-dev-01.test.ipa.gtp/ipa/xml
There is actually no file /var/lib/gssproxy/rcache/krb5_0.rcache2, or even an 'rcache' directory for some reason I'm not sure of right now. I believe the problem will be resolved if the missing file is in place.
The second form is obscure because there is no log of the problem in the httpd error_log. When attempting to log in, it shows a message "Your session has expired. Please log in again". When using the ipa cli tool, the first request fails with a GSS type of error, but the second one succeeds. If I do "kdestroy -A" and try again, there will be an error on the first attempt again:
[root@ipa-test-server /]# ipa ping ipa: ERROR: Could not create log_dir '/root/.ipa/log' ipa: ERROR: Ticket expired
[root@ipa-test-server /]# kinit admin Password for admin@TEST.IPA.GTP:
[root@ipa-test-server /]# ipa ping ipa: ERROR: Could not create log_dir '/root/.ipa/log' ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (69206018): gss_display_status call returned failure (major 327680, minor 100007). Decoding code: 69206018
[root@ipa-test-server /]# ipa ping ipa: ERROR: Could not create log_dir '/root/.ipa/log' ipa: WARNING: Failed to write schema: [Errno 2] No such file or directory: '/root/.cache/ipa' ipa: WARNING: Failed to write server info: [Errno 2] No such file or directory: '/root/.cache/ipa'
IPA server version 4.9.10. API version 2.248 [root@ipa-test-server /]# kdestroy -A
[root@ipa-test-server /]# kinit admin Password for admin@TEST.IPA.GTP:
[root@ipa-test-server /]# ipa ping ipa: ERROR: Could not create log_dir '/root/.ipa/log' ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (69206018): gss_display_status call returned failure (major 327680, minor
The file /var/lib/gssproxy/rcache/krb5_0.rcache2 is in place and in this case problem actually resolves with restart of gssproxy.service
Can you shed some light on what the rcache2 file is?
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
dweller dweller -
Kees Bakker -
Rob Crittenden -
terrible person