Hello!
I'm trying to migrate away from our old IPA server running on Centos 7. After solving multiple problems I've reached the following setup: ipa.domain - old centos 7 VM ipa1.domain - new replica, rocky 8 freeipa docker container running on Ubuntu 22.04 (because AFAIK it's not supported to go straight from 7 to 9) ipa2.domain - new replica, rocky 9 freeipa docker container running on Ubuntu 22.04
Mostly everything seems to work fine, I can authenticate against any of 3 servers but I've hit annoying problem that I've haven't yet solved despite extensive google-fu.
So scenario is as follows: 1) kinit on my laptop - works ok (ubuntu 22.04) 2) use firefox to access https://ipa2.domain - SSO works perfectly without asking any credentials. 3) restart ipa2.domain container 4) hit f5 in firefox - login screen, asks for username and password - not working quite as expected.
No amount of kdestroy or kinit against any of 3 servers will solve it (I don't have IPA DNS setup and set servers manually in krb5.conf). I can login with entering username and password manually but SSO won't work (but SSO works perfectly against ipa.domain and ipa1.domain). Same scenario is happening with ipa1.domain when I restart it. I'm not really eager to restart ipa.domain since our users are using it but I'm 99% sure that it wont have the same problem since it's not running in container and previous restarts have not produced such issue. What helps - deleting cookies in browser - then SSO starts to work again - until next restart of container.
httpd/error_log: [Tue Jul 09 14:51:00.648879 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.649030 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI jsonserver_i18n_messages.__call__: [Tue Jul 09 14:51:00.649309 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI jsonserver.__call__: [Tue Jul 09 14:51:00.649343 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI WSGIExecutioner.__call__: [Tue Jul 09 14:51:00.655929 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: raw: i18n_messages(version='2.253') [Tue Jul 09 14:51:00.656035 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: i18n_messages(version='2.253') [Tue Jul 09 14:51:00.660245 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: INFO: [jsonserver_i18n_messages] UNKNOWN: i18n_messages(version='2.253'): SUCCESS [Tue Jul 09 14:51:00.660291 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: [jsonserver_i18n_messages] UNKNOWN: i18n_messages(version='2.253'): SUCCESS etime=10778664 [Tue Jul 09 14:51:00.801704 2024] [:warn] [pid 326:tid 408] [client myIP:38908] KRB5CCNAME file (/run/ipa/ccaches/myuser@DOMAIN-8gyvGu) lookup failed!, referer: https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.802729 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.802792 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: WSGI jsonserver_session.__call__: [Tue Jul 09 14:51:00.802861 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: Valid Referer https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.806518 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: ccache expired or invalid, deleting session, need login [Tue Jul 09 14:51:00.806570 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: 401 Unauthorized need login [Tue Jul 09 14:51:00.818339 2024] [:warn] [pid 466:tid 486] [client myIP:38914] KRB5CCNAME file (/run/ipa/ccaches/myuser@DOMAIN-8gyvGu) lookup failed!, referer: https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.819251 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.819313 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: WSGI KerberosLogin.__call__: [Tue Jul 09 14:51:00.819378 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: Valid Referer https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.822213 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: ccache expired or invalid, deleting session, need login [Tue Jul 09 14:51:00.822279 2024] [wsgi:error] [pid 324:tid 556] [remote :38914] ipa: DEBUG: 401 Unauthorized need login
/run/ipa/ccaches/ folder is indeed empty. When I delete cookies and are able to log on then I see files created in that folder but they disappear after container restart.
docker command that I used to setup up replica is slight modification to one described in: https://hub.docker.com/r/freeipa/freeipa-server/ docker run -ti --dns=IP1 --dns=IP2 --name freeipa-server -h ipa2.domain --read-only -v /var/lib/ipa-data:/data:Z -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 -p 88:88/udp -p 464:464/udp -p 123:123/udp freeipa/freeipa-server:rocky-9 ipa-replica-install --password 'random-password' --server=ipa1.domain --domain=domain
Any ideas how to overcome this problem?
BR, Paavo
On Аўт, 09 ліп 2024, Paavo Nugin via FreeIPA-users wrote:
Hello!
I'm trying to migrate away from our old IPA server running on Centos 7. After solving multiple problems I've reached the following setup: ipa.domain - old centos 7 VM ipa1.domain - new replica, rocky 8 freeipa docker container running on Ubuntu 22.04 (because AFAIK it's not supported to go straight from 7 to 9) ipa2.domain - new replica, rocky 9 freeipa docker container running on Ubuntu 22.04
Mostly everything seems to work fine, I can authenticate against any of 3 servers but I've hit annoying problem that I've haven't yet solved despite extensive google-fu.
So scenario is as follows:
- kinit on my laptop - works ok (ubuntu 22.04)
- use firefox to access https://ipa2.domain - SSO works perfectly without asking any credentials.
- restart ipa2.domain container
- hit f5 in firefox - login screen, asks for username and password - not working quite as expected.
No amount of kdestroy or kinit against any of 3 servers will solve it (I don't have IPA DNS setup and set servers manually in krb5.conf). I can login with entering username and password manually but SSO won't work (but SSO works perfectly against ipa.domain and ipa1.domain). Same scenario is happening with ipa1.domain when I restart it. I'm not really eager to restart ipa.domain since our users are using it but I'm 99% sure that it wont have the same problem since it's not running in container and previous restarts have not produced such issue. What helps - deleting cookies in browser - then SSO starts to work again - until next restart of container.
httpd/error_log: [Tue Jul 09 14:51:00.648879 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.649030 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI jsonserver_i18n_messages.__call__: [Tue Jul 09 14:51:00.649309 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI jsonserver.__call__: [Tue Jul 09 14:51:00.649343 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: WSGI WSGIExecutioner.__call__: [Tue Jul 09 14:51:00.655929 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: raw: i18n_messages(version='2.253') [Tue Jul 09 14:51:00.656035 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: i18n_messages(version='2.253') [Tue Jul 09 14:51:00.660245 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: INFO: [jsonserver_i18n_messages] UNKNOWN: i18n_messages(version='2.253'): SUCCESS [Tue Jul 09 14:51:00.660291 2024] [wsgi:error] [pid 322:tid 550] [remote myIP:38914] ipa: DEBUG: [jsonserver_i18n_messages] UNKNOWN: i18n_messages(version='2.253'): SUCCESS etime=10778664 [Tue Jul 09 14:51:00.801704 2024] [:warn] [pid 326:tid 408] [client myIP:38908] KRB5CCNAME file (/run/ipa/ccaches/myuser@DOMAIN-8gyvGu) lookup failed!, referer: https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.802729 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.802792 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: WSGI jsonserver_session.__call__: [Tue Jul 09 14:51:00.802861 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: Valid Referer https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.806518 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: ccache expired or invalid, deleting session, need login [Tue Jul 09 14:51:00.806570 2024] [wsgi:error] [pid 321:tid 553] [remote myIP:38908] ipa: DEBUG: 401 Unauthorized need login [Tue Jul 09 14:51:00.818339 2024] [:warn] [pid 466:tid 486] [client myIP:38914] KRB5CCNAME file (/run/ipa/ccaches/myuser@DOMAIN-8gyvGu) lookup failed!, referer: https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.819251 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: WSGI wsgi_dispatch.__call__: [Tue Jul 09 14:51:00.819313 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: WSGI KerberosLogin.__call__: [Tue Jul 09 14:51:00.819378 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: Valid Referer https://ipa2.domain/ipa/ui/ [Tue Jul 09 14:51:00.822213 2024] [wsgi:error] [pid 324:tid 556] [remote myIP:38914] ipa: DEBUG: ccache expired or invalid, deleting session, need login [Tue Jul 09 14:51:00.822279 2024] [wsgi:error] [pid 324:tid 556] [remote :38914] ipa: DEBUG: 401 Unauthorized need login
/run/ipa/ccaches/ folder is indeed empty. When I delete cookies and are able to log on then I see files created in that folder but they disappear after container restart.
# mount |grep /run tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=800996k,nr_inodes=819200,mode=755,inode64) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400496k,nr_inodes=100124,mode=700,inode64)
This is on VM and in container it will be the same, so /run is gone when container instance is gone.
The lookup for ccache referenced from the cookie is done by mod_auth_gssapi. If it is missing, the file path will still be set (so KRB5CCNAME will be defined for IPA framework to see) but there will be no use for it as we just delete the session cookie and redirect for a login again. This is what you see in the log above.
I think what might be happening here as well is that mod_auth_gssapi session key gets wiped too so old session cookie cannot be decrypted by new container instance.
We use the following location for the session key:
GssapiSessionKey file:/etc/httpd/alias/ipasession.key
freeipa-container moves /etc/httpd/alias to the /data volume, so theoretically ipasession.key should persist.
Can you check that after restart this volume contains ipasession.key and it is the same as before?
docker command that I used to setup up replica is slight modification to one described in: https://hub.docker.com/r/freeipa/freeipa-server/ docker run -ti --dns=IP1 --dns=IP2 --name freeipa-server -h ipa2.domain --read-only -v /var/lib/ipa-data:/data:Z -v /etc/localtime:/etc/localtime:ro -v /etc/timezone:/etc/timezone:ro -p 80:80 -p 443:443 -p 389:389 -p 636:636 -p 88:88 -p 464:464 -p 88:88/udp -p 464:464/udp -p 123:123/udp freeipa/freeipa-server:rocky-9 ipa-replica-install --password 'random-password' --server=ipa1.domain --domain=domain
Any ideas how to overcome this problem?
BR, Paavo -- _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 09.07.2024 20:53, Alexander Bokovoy wrote:
# mount |grep /run tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=800996k,nr_inodes=819200,mode=755,inode64) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400496k,nr_inodes=100124,mode=700,inode64)
This is on VM and in container it will be the same, so /run is gone when container instance is gone.
The lookup for ccache referenced from the cookie is done by mod_auth_gssapi. If it is missing, the file path will still be set (so KRB5CCNAME will be defined for IPA framework to see) but there will be no use for it as we just delete the session cookie and redirect for a login again. This is what you see in the log above.
I think what might be happening here as well is that mod_auth_gssapi session key gets wiped too so old session cookie cannot be decrypted by new container instance.
We use the following location for the session key:
GssapiSessionKey file:/etc/httpd/alias/ipasession.key
freeipa-container moves /etc/httpd/alias to the /data volume, so theoretically ipasession.key should persist.
Can you check that after restart this volume contains ipasession.key and it is the same as before?
Thank you for quick reply! Date and sha256 for that file is same before and after container restart, so it persists correctly. What else to check?
BR, Paavo
Paavo Nugin via FreeIPA-users wrote:
On 09.07.2024 20:53, Alexander Bokovoy wrote:
# mount |grep /run tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=800996k,nr_inodes=819200,mode=755,inode64) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400496k,nr_inodes=100124,mode=700,inode64)
This is on VM and in container it will be the same, so /run is gone when container instance is gone.
The lookup for ccache referenced from the cookie is done by mod_auth_gssapi. If it is missing, the file path will still be set (so KRB5CCNAME will be defined for IPA framework to see) but there will be no use for it as we just delete the session cookie and redirect for a login again. This is what you see in the log above.
I think what might be happening here as well is that mod_auth_gssapi session key gets wiped too so old session cookie cannot be decrypted by new container instance.
We use the following location for the session key:
GssapiSessionKey file:/etc/httpd/alias/ipasession.key
freeipa-container moves /etc/httpd/alias to the /data volume, so theoretically ipasession.key should persist.
Can you check that after restart this volume contains ipasession.key and it is the same as before?
Thank you for quick reply! Date and sha256 for that file is same before and after container restart, so it persists correctly. What else to check?
FWIW this is reproducible outside of a container by rm -rf /run/ipa/ccaches/*
I also duplicated that removing the cookie resolves it.
So the fix is we need to invalidate the ipa_session cookie in at least the KerberosSession class needs_login() method. There is a logout_cookie context variable. Perhaps we can re-use that to indicate that a login requires a logout (session reset) first.
I filed a ticket on it https://pagure.io/freeipa/issue/9624
rob
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Paavo Nugin -
Rob Crittenden