"No credentials cache found"; Fedora 23; gssproxy not searching /var/lib/gssproxy/clients
Hello everyone,
I'm having difficulty configuring GSS Proxy with NFS4.2 on Fedora 23. My set up is fairly minimal (details below): all machines run Fedora 23. I have also tested using a CentOS 7 NFS4 client, which was previously connected to a different FreeIPA domain and a FreeNAS NFS server (GSS Proxy worked in this case). When joined to the new domain and Fedora NFS server however, it displayed the same behaviour as the Fedora 23 clients.
Using the current Fedora 23 configuration, NFS seems to work correctly. I'm able to mount all shares, and kinit to access them as expected. Groups and users are mapped correctly on both client and server.
Unfortunately, gssproxy and rpc.gssd are generating errors as soon as I mount these shares on the NFS client: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
On the NFS client, strace of gssproxy during mount shows:
strace -p $(pidof rpc.gssd) -s4096 -f
[...] [pid 599] open("/var/lib/gssproxy/clients/krb5cc_0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) [pid 599] open("/var/lib/gssproxy/clients/0.keytab", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 599] writev(2, [{"gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found\n", 141}], 1) = 141 [pid 599] sendto(3, "<83>Feb 7 00:09:44 gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found\n", 161, MSG_NOSIGNAL, NULL, 0) = 161 Note: this also occurs during file access.
On the NFS client, the journal shows this on mount (RPCGSSDARGS="-vvv"): CHILD forked pid 19084 rpc.gssd[607]: Full hostname for 'nfs-server.example.com' is 'nfs-server.example.com' rpc.gssd[607]: Full hostname for 'nfs-client.example.com' is 'nfs-client.example.com' rpc.gssd[607]: No key table entry found for nfs-client$@EXAMPLE.COM while getting keytab entry for 'nfs-client$@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for NFS-CLIENT$@EXAMPLE.COM while getting keytab entry for 'NFS-CLIENT$@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for root/nfs-client.example.com@EXAMPLE.COM while getting keytab entry for 'root/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for nfs/nfs-client.example.com@EXAMPLE.COM while getting keytab entry for 'nfs/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: Success getting keytab entry for 'host/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1454848445 rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1454848445 rpc.gssd[607]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine creds rpc.gssd[607]: using environment variable to select krb5 ccache FILE:/tmp/krb5ccmachine_EXAMPLE.COM rpc.gssd[607]: creating tcp client for server nfs-server.example.com rpc.gssd[607]: creating context with server nfs@nfs-server.example.com rpc.gssd[607]: doing downcall: lifetime_rec=83766 acceptor=nfs@nfs-server.example.com
And this when UID 40058920 accesses its share: rpc.gssd[607]: handle_gssd_upcall: 'mech=krb5 uid=40058920 enctypes=18,17,16,23,3,1,2 ' (nfs/clntb) rpc.gssd[3914]: CHILD forked pid 3914 rpc.gssd[3914]: krb5_not_machine_creds: uid 40058920 tgtname (null) gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found rpc.gssd[3914]: ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - No Kerberos credentials available (default cache: KEYRING:persistent:40058920) rpc.gssd[3914]: looking for client creds with uid 40058920 for server nfs-server.example.com in /tmp rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' being considered, with preferred realm 'EXAMPLE.COM' rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' owned by 0, not 40058920 rpc.gssd[3914]: looking for client creds with uid 40058920 for server nfs-server.example.com in /run/user/%U rpc.gssd[3914]: doing error downcall gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
I can't figure out why GSS Proxy isn't looking for the daemon's keytabs in /var/lib/gssproxy/clients/. I note though that the error message occurs even before accessing files on the mount. I would really appreciate any advice on checking my configuration, and troubleshooting the issue.
Thank you very much, James
----------------------- CONFIGURATION ----------------------- FREEIPA SERVER: 10.0.40.2 ipaserver.example.com (Fedora 23)
uname -a
Linux ipaserver.example.com 4.3.4-300.fc23.x86_64
ipa-server-install --domain=example.com --hostname=example.example.com --realm=EXAMPLE.COM --ip-address=10.0.40.2 --http-cert-file ipa-server.example.com.pfx --dirsrv-cert-file ipa-server.example.com.pfx
Note: external DNS server
Default /etc/gssproxy/gssproxy.conf ----------------------- NFS SERVER 10.0.40.10 nfs-server.example.com (Fedora 23)
uname -a
Linux nfs-server.example.com 4.3.4-300.fc23.x86_64
ipa-client-install kinit admin@EXAMPLE.COM ipa-getkeytab --server ipa-server.example.com --principal nfs/nfs-server.example.com@EXAMPLE.COM --keytab /etc/krb5.keytab
cat /etc/exports
/srv/raid *(fsid=0,sec=krb5p:krb5i,sync) /srv/raid/download *(sec=krb5i,sync) /srv/raid/media *(sec=krb5i,sync) /srv/raid/user *(sec=krb5p,sync) Note: setting the security to "sys" only does not prevent the gssproxy errors, nor does sharing directories owned by unmapped UID/GID.
firewall-cmd --list-all-zones
[...] user (active) [...] services: mdns nfs [...]
dnf list installed
freeipa-client.x86_64 4.2.3-2.fc23 gssproxy.x86_64 0.4.1-4.fc23 nfs-utils.x86_64 1:1.3.3-6.rc3.fc23
Default /etc/gssproxy/gssproxy.conf ----------------------- FEDORA NFS CLIENT 10.0.40.5 nfs-client.example.com (Fedora 23)
uname -a
Linux nfs-client.example.com 4.3.4-300.fc23.x86_64
ipa-client-install
kinit admin@EXAMPLE.COM ipa-getkeytab --server ipa-server.example.com --principal emby@EXAMPLE.COM --keytab /var/lib/gssproxy/clients/40058920.keytab
Note: 40058920 is the correct UID of emby@EXAMPLE.COM
dnf list installed
freeipa-client.x86_64 4.2.3-2.fc23 gssproxy.x86_64 0.4.1-4.fc23 nfs-utils.x86_64 1:1.3.3-6.rc3.fc23
Default /etc/gssproxy/gssproxy.conf ----------------------- CENTOS NFS CLIENT (known working client configuration, but not working with current IPA domain and NFS server) 10.0.40.7 nfs-client2.example.com (CentOS 7)
uname -a
Linux nfs-client2.example.com 3.10.0-327.el7.x86_64
ipa-client-install
dnf list installed
ipa-client.x86_64 4.2.0-15.el7.centos.3 gssproxy.x86_64 0.4.1-7.el7 nfs-utils.x86_64 1:1.3.0-0.21.el7
Default /etc/gssproxy/gssproxy.conf -----------------------
James Denton jdenton432@yahoo.com writes:
Hello everyone,
I'm having difficulty configuring GSS Proxy with NFS4.2 on Fedora 23. My set up is fairly minimal (details below): all machines run Fedora 23. I have also tested using a CentOS 7 NFS4 client, which was previously connected to a different FreeIPA domain and a FreeNAS NFS server (GSS Proxy worked in this case). When joined to the new domain and Fedora NFS server however, it displayed the same behaviour as the Fedora 23 clients.
Using the current Fedora 23 configuration, NFS seems to work correctly. I'm able to mount all shares, and kinit to access them as expected. Groups and users are mapped correctly on both client and server.
Unfortunately, gssproxy and rpc.gssd are generating errors as soon as I mount these shares on the NFS client: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
On the NFS client, strace of gssproxy during mount shows:
strace -p $(pidof rpc.gssd) -s4096 -f
[...] [pid 599] open("/var/lib/gssproxy/clients/krb5cc_0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) [pid 599] open("/var/lib/gssproxy/clients/0.keytab", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 599] writev(2, [{"gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found\n", 141}], 1) = 141 [pid 599] sendto(3, "<83>Feb 7 00:09:44 gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found\n", 161, MSG_NOSIGNAL, NULL, 0) = 161 Note: this also occurs during file access.
On the NFS client, the journal shows this on mount (RPCGSSDARGS="-vvv"): CHILD forked pid 19084 rpc.gssd[607]: Full hostname for 'nfs-server.example.com' is 'nfs-server.example.com' rpc.gssd[607]: Full hostname for 'nfs-client.example.com' is 'nfs-client.example.com' rpc.gssd[607]: No key table entry found for nfs-client$@EXAMPLE.COM while getting keytab entry for 'nfs-client$@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for NFS-CLIENT$@EXAMPLE.COM while getting keytab entry for 'NFS-CLIENT$@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for root/nfs-client.example.com@EXAMPLE.COM while getting keytab entry for 'root/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: No key table entry found for nfs/nfs-client.example.com@EXAMPLE.COM while getting keytab entry for 'nfs/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: Success getting keytab entry for 'host/nfs-client.example.com@EXAMPLE.COM' rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1454848445 rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until 1454848445 rpc.gssd[607]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine creds rpc.gssd[607]: using environment variable to select krb5 ccache FILE:/tmp/krb5ccmachine_EXAMPLE.COM rpc.gssd[607]: creating tcp client for server nfs-server.example.com rpc.gssd[607]: creating context with server nfs@nfs-server.example.com rpc.gssd[607]: doing downcall: lifetime_rec=83766 acceptor=nfs@nfs-server.example.com
And this when UID 40058920 accesses its share: rpc.gssd[607]: handle_gssd_upcall: 'mech=krb5 uid=40058920 enctypes=18,17,16,23,3,1,2 ' (nfs/clntb) rpc.gssd[3914]: CHILD forked pid 3914 rpc.gssd[3914]: krb5_not_machine_creds: uid 40058920 tgtname (null) gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found rpc.gssd[3914]: ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure. Minor code may provide more information) - No Kerberos credentials available (default cache: KEYRING:persistent:40058920) rpc.gssd[3914]: looking for client creds with uid 40058920 for server nfs-server.example.com in /tmp rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' being considered, with preferred realm 'EXAMPLE.COM' rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' owned by 0, not 40058920 rpc.gssd[3914]: looking for client creds with uid 40058920 for server nfs-server.example.com in /run/user/%U rpc.gssd[3914]: doing error downcall gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
I can't figure out why GSS Proxy isn't looking for the daemon's keytabs in /var/lib/gssproxy/clients/. I note though that the error message occurs even before accessing files on the mount. I would really appreciate any advice on checking my configuration, and troubleshooting the issue.
According to the logs you posted, gssproxy is in fact looking for the keytab. Can you check the permissions on it?
--Robbie
Hi Robbie,
Thanks very much for your reply! I apologise but I was indeed mistaken with that statement. I should have said that gssproxy was only looking for 0.keytab rather than the actual user's keytab. I'm sorry for the misunderstanding.
In case it would still be informative, here are the permissions for the keytabs and credential cache:
-rw-------. 1 root root unconfined_u:object_r:gssproxy_var_lib_t:s0 538 Feb 23 19:06 40058920.keytab -rw-------. 1 root root unconfined_u:object_r:gssproxy_var_lib_t:s0 302 Feb 23 21:52 40059091.keytab -rw-------. 1 root root system_u:object_r:gssproxy_var_lib_t:s0 1575 Feb 23 19:10 krb5cc_40058920 -rw-------. 1 root root system_u:object_r:gssproxy_var_lib_t:s0 1619 Feb 23 21:53 krb5cc_40059091
I've since made a little bit of progress on the issue. When mounting the NFS share the journal is still being spammed with: gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
However, restarting the gssproxy service via systemd stops the errors, and allows the credential cache for the required users to be created in /var/lib/gssproxy/clients/krb5cc_%U as expected. These errors recur on reboot.
The biggest problem is I am unable to access the NFS shares using this credentials cache. When I try to do so as the user emby@EXAMPLE.COM (UID 40058920), I get this:
strace -p $(pidof gssproxy) -s4096 -f [...] [pid 2913] read(4, "\0", 1) = 1 [pid 2913] epoll_ctl(5, EPOLL_CTL_ADD, 12, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=2618044368, u64=94552028098512}}) = 0 [pid 2913] epoll_wait(5, [{EPOLLOUT, {u32=2618044368, u64=94552028098512}}], 1, 30000) = 1 [pid 2913] writev(12, [{"\200\0\4\320", 4}, {"[REDACTED]emby@EXAMPLE.COM[REDACTED](nfs/nfs-server.example.com@EXAMPLE.COM[REDACTED]/etc/krb5.conf[REDACTED]emby@EXAMPLE.COM[REDACTED]emby@EXAMPLE.COM[REDACTED]emby@EXAMPLE.COM[REDACTED](nfs/nfs-server.example.com@EXAMPLE.COM[REDACTED](nfs/nfs-server.example.com@EXAMPLE.COM(nfs/nfs-server.example.com@EXAMPLE.COM[REDACTED]", 1232}], 2) = 1236 [pid 2913] epoll_ctl(5, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=2618044608, u64=94552028098752}}) = -1 EEXIST (File exists) [pid 2913] epoll_ctl(5, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=2618044368, u64=94552028098512}}) = 0 [pid 2913] epoll_ctl(5, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=2618044608, u64=94552028098752}}) = 0 [pid 2913] epoll_wait(5, [{EPOLLIN|EPOLLHUP, {u32=2618044608, u64=94552028098752}}], 1, 30000) = 1 [pid 2913] read(12, "", 4) = 0 [pid 2913] close(12) = 0 [pid 2913] epoll_ctl(5, EPOLL_CTL_DEL, 12, 0x7fff055fa5e0) = -1 EBADF (Bad file descriptor) [pid 2913] epoll_wait(5, [], 1, 30000) = 0
I've redacted line 4 for brevity. Let me know if there is any more information I can provide.
Thanks again for your consideration James
Hello again,
I've made more progress on the issue.
In FreeIPA, setting the NFS client host to be managed by the FreeIPA server itself is allowing gssproxy to work.
E.g. On the FreeIPA server (freeipaserver.example.com): kinit admin ipa host-add-managedby nfs-client.example.com --hosts=freeipa-server-example.com
I didn't run into this issue in the past because the above step was required in order to get the keytab on the FreeIPA server (to be transferred to the client with scp). With my current setup, I am running ipa-getkeytab on the FreeIPA clients directly so delegation is not required at that stage. It is apparently required for gssproxy, however.
So now the only issue remaining is that if gssproxy is running when NFS share is mounted, the logs are getting this repeatedly:
gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
Additionally, clients are not able to access their shares using the cached credentials. This behaviour does not occur before the client is joined to the FreeIPA domain.
But when the gssproxy is restarted at this stage using "systemctl restart gssproxy", everything works perfectly.
I would appreciate any of your thoughts on the issue.
Thanks so much, James
On 02/23/2016 10:14 PM, James Denton wrote:
Hello again,
I've made more progress on the issue.
In FreeIPA, setting the NFS client host to be managed by the FreeIPA server itself is allowing gssproxy to work.
E.g. On the FreeIPA server (freeipaserver.example.com): kinit admin ipa host-add-managedby nfs-client.example.com --hosts=freeipa-server-example.com
I didn't run into this issue in the past because the above step was required in order to get the keytab on the FreeIPA server (to be transferred to the client with scp). With my current setup, I am running ipa-getkeytab on the FreeIPA clients directly so delegation is not required at that stage. It is apparently required for gssproxy, however.
So now the only issue remaining is that if gssproxy is running when NFS share is mounted, the logs are getting this repeatedly:
gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may provide more information, No credentials cache found
Additionally, clients are not able to access their shares using the cached credentials. This behaviour does not occur before the client is joined to the FreeIPA domain.
But when the gssproxy is restarted at this stage using "systemctl restart gssproxy", everything works perfectly.
I would appreciate any of your thoughts on the issue.
Do I get it right that the scenario is:
a) A client that is not enrolled b) You add a host on the server c) Use ipa-getkeytab on the client
You observe failures unless you restart gssproxy.
I think gssproxy needs to be restarted to read the keytab after you configured it and provisioned the keytab. To the best of my knowledge gssproxy reads keytab only at the beginning.
Thanks so much, James _______________________________________________ gss-proxy mailing list gss-proxy@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/gss-proxy@lists.fedorahosted.org
On Wed, 2016-02-24 at 07:19 -0500, Dmitri Pal wrote:
On 02/23/2016 10:14 PM, James Denton wrote:
Hello again,
I've made more progress on the issue.
In FreeIPA, setting the NFS client host to be managed by the FreeIPA
server itself is allowing gssproxy to work.
E.g. On the FreeIPA server (freeipaserver.example.com): kinit admin ipa host-add-managedby nfs-client.example.com
--hosts=freeipa-server-example.com
I didn't run into this issue in the past because the above step was
required in order to get the keytab on the FreeIPA server (to be transferred to the client with scp). With my current setup, I am running ipa-getkeytab on the FreeIPA clients directly so delegation is not required at that stage. It is apparently required for gssproxy, however.
So now the only issue remaining is that if gssproxy is running when
NFS share is mounted, the logs are getting this repeatedly:
gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS
failure. Minor code may provide more information, No credentials cache found
Additionally, clients are not able to access their shares using the
cached credentials. This behaviour does not occur before the client is joined to the FreeIPA domain.
But when the gssproxy is restarted at this stage using "systemctl
restart gssproxy", everything works perfectly.
I would appreciate any of your thoughts on the issue.
Do I get it right that the scenario is:
a) A client that is not enrolled b) You add a host on the server c) Use ipa-getkeytab on the client
You observe failures unless you restart gssproxy.
I think gssproxy needs to be restarted to read the keytab after you configured it and provisioned the keytab. To the best of my knowledge gssproxy reads keytab only at the beginning.
A restart shouldn't actually be necessary unless the gssproxy configuration was changed.
A no credentials cache error hints at a user ccache missing not a keytab missing. Do users actually have a ccache in this case ? And where is their ccache located ? How are they logging in the system ?
Simo.
gss-proxy@lists.fedorahosted.org
-
Dmitri Pal -
James Denton -
Robbie Harwood -
Simo Sorce