On Sun, 2022-02-06 at 12:43 +0200, Alexander Bokovoy wrote:
# Protect /ipa and everything below it in webspace with Apache Kerberos
auth
<Location "/ipa">
AuthType GSSAPI
AuthName "Kerberos Login"
GssapiUseSessions On
Session On
SessionCookieName ipa_session path=/ipa;httponly;secure;
SessionHeader IPASESSION
# Uncomment the following to have shorter sessions, but beware this
may break
# old IPA client tols that incorrectly parse cookies.
# SessionMaxAge 1800
GssapiSessionKey file:$GSSAPI_SESSION_KEY
On both of my servers, this is actually:
GssapiSessionKey file:/etc/httpd/alias/ipasession.key
GssapiImpersonate On
GssapiDelegCcacheDir $IPA_CCACHES
On my servers:
GssapiDelegCcacheDir /run/ipa/ccaches
GssapiDelegCcachePerms mode:0660
GssapiDelegCcacheUnique On
GssapiUseS4U2Proxy on
GssapiAllowedMech krb5
Require valid-user
ErrorDocument 401 /ipa/errors/unauthorized.html
Header always append X-Frame-Options DENY
Header always append Content-Security-Policy "frame-ancestors
'none'"
# mod_session always sets two copies of the cookie, and this
confuses our
# legacy clients, the unset here works because it ends up unsetting
only one
# of the 2 header tables set by mod_session, leaving the other
intact
Header unset Set-Cookie
# Disable etag http header. Doesn't work well with mod_deflate
#
https://issues.apache.org/bugzilla/show_bug.cgi?id=45023
# Usage of last-modified header and modified-since validator is
sufficient.
Header unset ETag
FileETag None
</Location>
The stale ccaches then removed by
a script that is run periodically as a systemd timer
(ipa-ccache-sweep.timer which runs ipa-ccache-sweep.service).
On both of my servers, this timer is disabled:
# systemctl status ipa-ccache-sweep.timer
● ipa-ccache-sweep.timer - Remove Expired Kerberos Credential Caches
Loaded: loaded (/usr/lib/systemd/system/ipa-ccache-sweep.timer; disabled; vendor
preset: disabled)
Active: inactive (dead)
Trigger: n/a
Some kind of installation bug? Should these be enabled?
Basically, this means we have following setup:
- httpd with mod_auth_gssapi runs under 'apache' user
- IPA API in httpd with mod_wsgi runs under 'ipaapi' user
- GSSProxy manages content of the credential caches accessed from
both
processes.
Yup, I have all of that:
# ps axufw|egrep '(httpd|ipaapi)'
root 88164 0.0 0.0 221928 1156 pts/9 S+ 10:23 0:00 \_ grep -E
--color=auto (httpd|ipaapi)
root 85393 0.0 0.1 328104 14180 ? Ss 09:56 0:00 /usr/sbin/httpd
-DFOREGROUND
apache 85407 0.0 0.1 341988 8488 ? S 09:57 0:00 \_ /usr/sbin/httpd
-DFOREGROUND
ipaapi 85410 0.1 1.2 1025540 94084 ? Sl 09:57 0:02 \_ (wsgi:ipa)
-DFOREGROUND
ipaapi 85411 0.1 1.1 1025540 92496 ? Sl 09:57 0:02 \_ (wsgi:ipa)
-DFOREGROUND
ipaapi 85412 0.1 1.2 1028100 96480 ? Sl 09:57 0:02 \_ (wsgi:ipa)
-DFOREGROUND
ipaapi 85413 0.1 1.1 1025540 93152 ? Sl 09:57 0:02 \_ (wsgi:ipa)
-DFOREGROUND
apache 85414 0.0 0.3 2081292 29020 ? Sl 09:57 0:00 \_ /usr/sbin/httpd
-DFOREGROUND
apache 85415 0.0 0.2 2077184 22264 ? Sl 09:57 0:00 \_ /usr/sbin/httpd
-DFOREGROUND
apache 85416 0.0 0.3 2212508 28068 ? Sl 09:57 0:00 \_ /usr/sbin/httpd
-DFOREGROUND
apache 85693 0.0 0.3 2081292 29696 ? Sl 09:57 0:00 \_ /usr/sbin/httpd
-DFOREGROUND
To do so, we rely on POSIX ACLs set on /run/ipa/ccaches that allow
apache group to read/write to the files in /run/ipa/ccaches and force
ownership to ipaapi:ipaapi user and group.
Here is an example:
---------------------
[root@dc ~]# ls -la /run/ipa/ccaches/
total 0
drwsrws---+ 2 ipaapi ipaapi 40 Feb 4 11:11 .
drwx--x--x. 3 root root 100 Feb 4 11:12 ..
[root@dc ~]# getfacl /run/ipa/ccaches/
getfacl: Removing leading '/' from absolute path names
# file: run/ipa/ccaches/
# owner: ipaapi
# group: ipaapi
# flags: ss-
user::rwx
group::rwx
group:apache:rwx
mask::rwx
other::---
Perhaps this is where the problem lies then. On my working replica, I
see the same as above but on the non-webUI-working one I have:
# getfacl /run/ipa/ccaches/
getfacl: Removing leading '/' from absolute path names
# file: run/ipa/ccaches/
# owner: ipaapi
# group: ipaapi
user::rwx
group::rwx
other::---
It's missing the group:apache:rwx, but the tmpfile that should be
creating it does seem to be present:
# grep -r ccaches /usr/lib/tmpfiles.d/
/usr/lib/tmpfiles.d/ipa.conf:d /run/ipa/ccaches 6770 ipaapi ipaapi
/usr/lib/tmpfiles.d/ipa.conf:a+ /run/ipa/ccaches - - - - g:apache:rwx
So, to correct it:
# setfacl -m g:apache:rwx /run/ipa/ccaches/
# getfacl /run/ipa/ccaches/
getfacl: Removing leading '/' from absolute path names
# file: run/ipa/ccaches/
# owner: ipaapi
# group: ipaapi
user::rwx
group::rwx
group:apache:rwx
mask::rwx
other::---
But that doesn't seem to have helped:
# kinit admin
Password for admin(a)EXAMPLE.COM
# ipa ping
ipa: ERROR: Major (851968): Unspecified GSS failure. Minor code may provide more
information, Minor (2598845122): Credentials cache permissions incorrect (filename:
/run/ipa/ccaches/admin(a)EXAMPLE.COM-wZC7kh)
# ls -la /run/ipa/ccaches/
total 24
drwxrwx---+ 2 ipaapi ipaapi 100 Feb 6 09:47 .
drwx--x--x. 3 root root 100 Feb 5 10:08 ..
-rw-rw----. 1 apache apache 5624 Feb 6 09:47 admin(a)EXAMPLE.COM-wZC7kh
/run/ipa/ccaches/ seems to also be missing the user and group suid
bits. Fixing that seems to make ipa ping at least work now:
# ipa ping
-------------------------------------------
IPA server version 4.9.6. API version 2.245
-------------------------------------------
# ls -la /run/ipa/ccaches/
total 24
drwsrws---+ 2 ipaapi ipaapi 80 Feb 6 09:54 .
drwx--x--x. 3 root root 100 Feb 5 10:08 ..
-rw-------. 1 ipaapi ipaapi 9808 Feb 6 09:54 admin(a)EXAMPLE.COM-vxMhbf
-rw-------. 1 ipaapi ipaapi 9808 Feb 6 09:54 admin(a)EXAMPLE.COM-ycq8Ye
If you look with klist into that ccache, you'll only see
encrypted
gssproxy object data:
------------------------
[root@dc ~]# klist -c /run/ipa/ccaches/admin(a)IPA.TEST-GEIcn3
Ticket cache: FILE:/run/ipa/ccaches/admin@IPA.TEST-GEIcn3
Default principal: admin(a)IPA.TEST
Valid starting Expires Service principal
010170 00:00:00 010170 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
I have two caches and they are:
# klist -c /run/ipa/ccaches/admin(a)EXAMPLE.COM-RJLFGU
Ticket cache: FILE:/run/ipa/ccaches/admin@EXAMPLE.COM-RJLFGU
Default principal: admin(a)EXAMPLE.COM
Valid starting Expires Service principal
1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
# klist -c /run/ipa/ccaches/admin(a)EXAMPLE.COM-V05fsT
Ticket cache: FILE:/run/ipa/ccaches/admin@EXAMPLE.COM-V05fsT
Default principal: admin(a)EXAMPLE.COM
Valid starting Expires Service principal
1969-12-31 19:00:00 1969-12-31 19:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
The valid and expiration dates on mine seem to be different, but they
still seem to be working.
WebUI is still not working though. When I try to use the WebUI from a
client which I have obtained a tgt on, the client does not even get an
HTTP principle ticket, nor does the /run/ipa/ccaches/ dir on the server
I am trying to use the webUI on have anything in /run/ipa/ccaches for
that user.
The httpd error log does log:
[Sun Feb 06 10:09:08.523430 2022] [wsgi:error] [pid 85411:tid 139661198296832] [remote
fd31:aeb1:48df:0:2374:a2d1:e1b7:89a3:36228] ipa: INFO: [jsonserver_i18n_messages] UNKNOWN:
i18n_messages(version='2.245'): SUCCESS
[Sun Feb 06 10:09:08.710420 2022] [auth_gssapi:error] [pid 85414:tid 139661028603648]
[client fd31:aeb1:48df:0:2374:a2d1:e1b7:89a3:36228] Failed to unseal session data!,
referer:
https://server.example.com/ipa/ui/
[Sun Feb 06 10:09:08.730708 2022] [auth_gssapi:error] [pid 85414:tid 139660995032832]
[client fd31:aeb1:48df:0:2374:a2d1:e1b7:89a3:36228] Failed to unseal session data!,
referer:
https://server.example.com/ipa/ui/
when that webUI login attempt fails.
- /run/ipa/ccaches has wrong setup, not corresponding to what is
shown above
It was clearly wrong, as I detailed above, (which seems to be a
tmpfiles problem perhaps? I will reboot and see what it is set up as
on reboot) but those are fixed now.
- /run/ipa/ccaches/user@REALM-suffix file is not accesible to
httpd
processes, or damaged, making its content not decryptable by
GSSProxy
But I am not even getting that file when I try to log into the webUI.
So things are not even making it that far.
- ipa_session cookie generated by mod_auth_gssapi earlier and
received
by httpd process now is encrypted by a different mod_auth_gssapi
session key, rendering its content damaged with regards to what
Kerberos 5 library expects for a ccache
I'm not sure how to assess if this is the problem or not.
- GSSProxy session key is different to what was used to encrypt
ccaches
in /run/ipa/ccaches. This happens on reboot, for example.
What is the way to determine if this is the case, or at least rectify
it so that things are back in sync?
Cheers,
b.