"No credentials cache found"; Fedora 23; gssproxy not searching /var/lib/gssproxy/clients

[James Denton]

Hello everyone,

I'm having difficulty configuring GSS Proxy with NFS4.2 on Fedora 23. My set up is
fairly minimal (details below): all machines run Fedora 23. I have also tested using a
CentOS 7 NFS4 client, which was previously connected to a different FreeIPA domain and a
FreeNAS NFS server (GSS Proxy worked in this case). When joined to the new domain and
Fedora NFS server however, it displayed the same behaviour as the Fedora 23 clients.

Using the current Fedora 23 configuration, NFS seems to work correctly. I'm able to
mount all shares, and kinit to access them as expected. Groups and users are mapped
correctly on both client and server.

Unfortunately, gssproxy and rpc.gssd are generating errors as soon as I mount these shares
on the NFS client:
gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
provide more information, No credentials cache found
gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. 
Minor code may provide more information, No credentials cache found

On the NFS client, strace of gssproxy during mount shows:
> strace -p $(pidof rpc.gssd) -s4096 -f 
[...]
[pid   599] open("/var/lib/gssproxy/clients/krb5cc_0", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)
[pid   599] open("/var/lib/gssproxy/clients/0.keytab", O_RDONLY) = -1 ENOENT (No
such file or directory)
[pid   599] writev(2, [{"gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified
GSS failure.  Minor code may provide more information, No credentials cache found\n",
141}], 1) = 141
[pid   599] sendto(3, "<83>Feb  7 00:09:44 gssproxy[594]: (OID: { 1 2 840
113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, No
credentials cache found\n", 161, MSG_NOSIGNAL, NULL, 0) = 161
Note: this also occurs during file access.

On the NFS client, the journal shows this on mount (RPCGSSDARGS="-vvv"):
CHILD forked pid 19084
rpc.gssd[607]: Full hostname for 'nfs-server.example.com' is
'nfs-server.example.com'
rpc.gssd[607]: Full hostname for 'nfs-client.example.com' is
'nfs-client.example.com'
rpc.gssd[607]: No key table entry found for nfs-client$(a)EXAMPLE.COM while getting keytab
entry for &#39;nfs-client$(a)EXAMPLE.COM&#39;
rpc.gssd[607]: No key table entry found for NFS-CLIENT$(a)EXAMPLE.COM while getting keytab
entry for &#39;NFS-CLIENT$(a)EXAMPLE.COM&#39;
rpc.gssd[607]: No key table entry found for root/nfs-client.example.com(a)EXAMPLE.COM while
getting keytab entry for &#39;root/nfs-client.example.com(a)EXAMPLE.COM&#39;
rpc.gssd[607]: No key table entry found for nfs/nfs-client.example.com(a)EXAMPLE.COM while
getting keytab entry for &#39;nfs/nfs-client.example.com(a)EXAMPLE.COM&#39;
rpc.gssd[607]: Success getting keytab entry for
&#39;host/nfs-client.example.com(a)EXAMPLE.COM&#39;
rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are
good until 1454848445
rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are
good until 1454848445
rpc.gssd[607]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine
creds
rpc.gssd[607]: using environment variable to select krb5 ccache
FILE:/tmp/krb5ccmachine_EXAMPLE.COM
rpc.gssd[607]: creating tcp client for server nfs-server.example.com
rpc.gssd[607]: creating context with server nfs(a)nfs-server.example.com
rpc.gssd[607]: doing downcall: lifetime_rec=83766 acceptor=nfs(a)nfs-server.example.com

And this when UID 40058920 accesses its share:
rpc.gssd[607]: handle_gssd_upcall: 'mech=krb5 uid=40058920 enctypes=18,17,16,23,3,1,2
' (nfs/clntb)
rpc.gssd[3914]: CHILD forked pid 3914
rpc.gssd[3914]: krb5_not_machine_creds: uid 40058920 tgtname (null)
gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
provide more information, No credentials cache found
rpc.gssd[3914]: ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE (Unspecified
GSS failure.  Minor code may provide more information) - No Kerberos credentials available
(default cache: KEYRING:persistent:40058920)
rpc.gssd[3914]: looking for client creds with uid 40058920 for server
nfs-server.example.com in /tmp
rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' being considered, with
preferred realm 'EXAMPLE.COM'
rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' owned by 0, not 40058920
rpc.gssd[3914]: looking for client creds with uid 40058920 for server
nfs-server.example.com in /run/user/%U
rpc.gssd[3914]: doing error downcall
gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
provide more information, No credentials cache found
gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. 
Minor code may provide more information, No credentials cache found

I can't figure out why GSS Proxy isn't looking for the daemon's keytabs in
/var/lib/gssproxy/clients/. I note though that the error message occurs even before
accessing files on the mount. I would really appreciate any advice on checking my
configuration, and troubleshooting the issue.

Thank you very much,
James

-----------------------
CONFIGURATION
-----------------------
FREEIPA SERVER:
10.0.40.2
ipaserver.example.com (Fedora 23)
> uname -a 
Linux ipaserver.example.com 4.3.4-300.fc23.x86_64

> ipa-server-install --domain=example.com --hostname=example.example.com
> --realm=EXAMPLE.COM --ip-address=10.0.40.2 --http-cert-file ipa-server.example.com.pfx
> --dirsrv-cert-file ipa-server.example.com.pfx 
Note: external DNS server

Default /etc/gssproxy/gssproxy.conf
-----------------------
NFS SERVER
10.0.40.10
nfs-server.example.com (Fedora 23)
> uname -a 
Linux nfs-server.example.com 4.3.4-300.fc23.x86_64

> ipa-client-install
> kinit admin(a)EXAMPLE.COM
> ipa-getkeytab --server ipa-server.example.com --principal
> nfs/nfs-server.example.com(a)EXAMPLE.COM --keytab /etc/krb5.keytab 
> cat /etc/exports 
/srv/raid             
*(fsid=0,sec=krb5p:krb5i,sync)
/srv/raid/download     *(sec=krb5i,sync)
/srv/raid/media        *(sec=krb5i,sync)
/srv/raid/user         *(sec=krb5p,sync)
Note: setting the security to "sys" only does not prevent the gssproxy errors,
nor does sharing directories owned by unmapped UID/GID.

> firewall-cmd --list-all-zones 
[...]
user (active)
[...]
services: mdns nfs
[...]

> dnf list installed 
freeipa-client.x86_64    4.2.3-2.fc23
gssproxy.x86_64          0.4.1-4.fc23
nfs-utils.x86_64         1:1.3.3-6.rc3.fc23

Default /etc/gssproxy/gssproxy.conf
-----------------------
FEDORA NFS CLIENT
10.0.40.5
nfs-client.example.com (Fedora 23)
> uname -a 
Linux nfs-client.example.com 4.3.4-300.fc23.x86_64

> ipa-client-install 
> kinit admin(a)EXAMPLE.COM
> ipa-getkeytab --server ipa-server.example.com --principal emby(a)EXAMPLE.COM --keytab
> /var/lib/gssproxy/clients/40058920.keytab 
Note: 40058920 is the correct UID of
emby(a)EXAMPLE.COM

> dnf list installed 
freeipa-client.x86_64    4.2.3-2.fc23
gssproxy.x86_64          0.4.1-4.fc23
nfs-utils.x86_64         1:1.3.3-6.rc3.fc23

Default /etc/gssproxy/gssproxy.conf
-----------------------
CENTOS NFS CLIENT (known working client configuration, but not working with current IPA
domain and NFS server)
10.0.40.7
nfs-client2.example.com (CentOS 7)
> uname -a 
Linux nfs-client2.example.com 3.10.0-327.el7.x86_64

> ipa-client-install 
> dnf list installed 
ipa-client.x86_64       
4.2.0-15.el7.centos.3
gssproxy.x86_64          0.4.1-7.el7
nfs-utils.x86_64         1:1.3.0-0.21.el7

Default /etc/gssproxy/gssproxy.conf
-----------------------

[Robbie Harwood]

James Denton <jdenton432(a)yahoo.com&gt; writes:

>  Hello everyone,
>
>  I'm having difficulty configuring GSS Proxy with NFS4.2 on Fedora 23. My set up is
> fairly minimal (details below): all machines run Fedora 23. I have also tested using a
> CentOS 7 NFS4 client, which was previously connected to a different FreeIPA domain and a
> FreeNAS NFS server (GSS Proxy worked in this case). When joined to the new domain and
> Fedora NFS server however, it displayed the same behaviour as the Fedora 23 clients.
>
>  Using the current Fedora 23 configuration, NFS seems to work correctly. I'm able to
> mount all shares, and kinit to access them as expected. Groups and users are mapped
> correctly on both client and server.
>
>  Unfortunately, gssproxy and rpc.gssd are generating errors as soon as I mount these
> shares on the NFS client:
>  gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
> provide more information, No credentials cache found
>  gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. 
> Minor code may provide more information, No credentials cache found
>
>  On the NFS client, strace of gssproxy during mount shows:
> >strace -p $(pidof rpc.gssd) -s4096 -f
>  [...]
>  [pid   599] open("/var/lib/gssproxy/clients/krb5cc_0", O_RDONLY|O_CLOEXEC) = -1
> ENOENT (No such file or directory)
>  [pid   599] open("/var/lib/gssproxy/clients/0.keytab", O_RDONLY) = -1 ENOENT
> (No such file or directory)
>  [pid   599] writev(2, [{"gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified
> GSS failure.  Minor code may provide more information, No credentials cache found\n",
> 141}], 1) = 141
>  [pid   599] sendto(3, "<83>Feb  7 00:09:44 gssproxy[594]: (OID: { 1 2 840
> 113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, No
> credentials cache found\n", 161, MSG_NOSIGNAL, NULL, 0) = 161
>  Note: this also occurs during file access.
>
>  On the NFS client, the journal shows this on mount (RPCGSSDARGS="-vvv"):
>  CHILD forked pid 19084
>  rpc.gssd[607]: Full hostname for 'nfs-server.example.com' is
> 'nfs-server.example.com'
>  rpc.gssd[607]: Full hostname for 'nfs-client.example.com' is
> 'nfs-client.example.com'
>  rpc.gssd[607]: No key table entry found for nfs-client$(a)EXAMPLE.COM while getting keytab
> entry for &#39;nfs-client$(a)EXAMPLE.COM&#39;
>  rpc.gssd[607]: No key table entry found for NFS-CLIENT$(a)EXAMPLE.COM while getting keytab
> entry for &#39;NFS-CLIENT$(a)EXAMPLE.COM&#39;
>  rpc.gssd[607]: No key table entry found for root/nfs-client.example.com(a)EXAMPLE.COM while
> getting keytab entry for &#39;root/nfs-client.example.com(a)EXAMPLE.COM&#39;
>  rpc.gssd[607]: No key table entry found for nfs/nfs-client.example.com(a)EXAMPLE.COM while
> getting keytab entry for &#39;nfs/nfs-client.example.com(a)EXAMPLE.COM&#39;
>  rpc.gssd[607]: Success getting keytab entry for
> &#39;host/nfs-client.example.com(a)EXAMPLE.COM&#39;
>  rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are
> good until 1454848445
>  rpc.gssd[607]: INFO: Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are
> good until 1454848445
>  rpc.gssd[607]: using FILE:/tmp/krb5ccmachine_EXAMPLE.COM as credentials cache for machine
> creds
>  rpc.gssd[607]: using environment variable to select krb5 ccache
> FILE:/tmp/krb5ccmachine_EXAMPLE.COM
>  rpc.gssd[607]: creating tcp client for server nfs-server.example.com
>  rpc.gssd[607]: creating context with server nfs(a)nfs-server.example.com
>  rpc.gssd[607]: doing downcall: lifetime_rec=83766 acceptor=nfs(a)nfs-server.example.com
>
>  And this when UID 40058920 accesses its share:
>  rpc.gssd[607]: handle_gssd_upcall: 'mech=krb5 uid=40058920 enctypes=18,17,16,23,3,1,2
> ' (nfs/clntb)
>  rpc.gssd[3914]: CHILD forked pid 3914
>  rpc.gssd[3914]: krb5_not_machine_creds: uid 40058920 tgtname (null)
>  gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
> provide more information, No credentials cache found
>  rpc.gssd[3914]: ERROR: GSS-API: error in gss_acquire_cred(): GSS_S_FAILURE (Unspecified
> GSS failure.  Minor code may provide more information) - No Kerberos credentials available
> (default cache: KEYRING:persistent:40058920)
>  rpc.gssd[3914]: looking for client creds with uid 40058920 for server
> nfs-server.example.com in /tmp
>  rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' being considered, with
> preferred realm 'EXAMPLE.COM'
>  rpc.gssd[3914]: CC '/tmp/krb5ccmachine_EXAMPLE.COM' owned by 0, not 40058920
>  rpc.gssd[3914]: looking for client creds with uid 40058920 for server
> nfs-server.example.com in /run/user/%U
>  rpc.gssd[3914]: doing error downcall
>  gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
> provide more information, No credentials cache found
>  gssproxy[587]: gssproxy[594]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. 
> Minor code may provide more information, No credentials cache found
>
>  I can't figure out why GSS Proxy isn't looking for the daemon's
>  keytabs in /var/lib/gssproxy/clients/. I note though that the error
>  message occurs even before accessing files on the mount. I would
>  really appreciate any advice on checking my configuration, and
>  troubleshooting the issue. 
According to the logs you posted, gssproxy is in fact looking for the
keytab.  Can you check the permissions on it?

--Robbie

[James Denton]

Hi Robbie,

Thanks very much for your reply! I apologise but I was indeed mistaken with that
statement. I should have said that gssproxy was only looking for 0.keytab rather than the
actual user's keytab. I'm sorry for the misunderstanding.

In case it would still be informative, here are the permissions for the keytabs and
credential cache:

-rw-------. 1 root root unconfined_u:object_r:gssproxy_var_lib_t:s0  538 Feb 23 19:06
40058920.keytab
-rw-------. 1 root root unconfined_u:object_r:gssproxy_var_lib_t:s0  302 Feb 23 21:52
40059091.keytab
-rw-------. 1 root root system_u:object_r:gssproxy_var_lib_t:s0     1575 Feb 23 19:10
krb5cc_40058920
-rw-------. 1 root root system_u:object_r:gssproxy_var_lib_t:s0     1619 Feb 23 21:53
krb5cc_40059091

I've since made a little bit of progress on the issue. When mounting the NFS share the
journal is still being spammed with:
gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may
provide more information, No credentials cache found

However, restarting the gssproxy service via systemd stops the errors, and allows the
credential cache for the required users to be created in
/var/lib/gssproxy/clients/krb5cc_%U as expected. These errors recur on reboot.

The biggest problem is I am unable to access the NFS shares using this credentials cache.
When I try to do so as the user emby(a)EXAMPLE.COM (UID 40058920), I get this:

strace -p $(pidof gssproxy) -s4096 -f
[...]
[pid  2913] read(4, "\0", 1)            = 1
[pid  2913] epoll_ctl(5, EPOLL_CTL_ADD, 12, {EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=2618044368,
u64=94552028098512}}) = 0
[pid  2913] epoll_wait(5, [{EPOLLOUT, {u32=2618044368, u64=94552028098512}}], 1, 30000) =
1
[pid  2913] writev(12, [{"\200\0\4\320", 4},
{"[REDACTED][e-mail][REDACTED](nfs/[e-mail][REDACTED]/etc/krb5.conf[REDACTED][e-mail][REDACTED][e-mail][REDACTED][e-mail][REDACTED](nfs/[e-mail][REDACTED](nfs/[e-mail](nfs/nfs-server.example.com(a)EXAMPLE.COM[REDACTED]",
1232}], 2) = 1236
[pid  2913] epoll_ctl(5, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=2618044608,
u64=94552028098752}}) = -1 EEXIST (File exists)
[pid  2913] epoll_ctl(5, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP,
{u32=2618044368, u64=94552028098512}}) = 0
[pid  2913] epoll_ctl(5, EPOLL_CTL_MOD, 12, {EPOLLIN|EPOLLERR|EPOLLHUP, {u32=2618044608,
u64=94552028098752}}) = 0
[pid  2913] epoll_wait(5, [{EPOLLIN|EPOLLHUP, {u32=2618044608, u64=94552028098752}}], 1,
30000) = 1
[pid  2913] read(12, "", 4)             = 0
[pid  2913] close(12)                   = 0
[pid  2913] epoll_ctl(5, EPOLL_CTL_DEL, 12, 0x7fff055fa5e0) = -1 EBADF (Bad file
descriptor)
[pid  2913] epoll_wait(5, [], 1, 30000) = 0

I've redacted line 4 for brevity. Let me know if there is any more information I can
provide.

Thanks again for your consideration
James

[James Denton]

Hello again,

I've made more progress on the issue.

In FreeIPA, setting the NFS client host to be managed by the FreeIPA server itself is
allowing gssproxy to work.

E.g. On the FreeIPA server (freeipaserver.example.com):
kinit admin
ipa host-add-managedby nfs-client.example.com --hosts=freeipa-server-example.com

I didn't run into this issue in the past because the above step was required in order
to get the keytab on the FreeIPA server (to be transferred to the client with scp). With
my current setup, I am running ipa-getkeytab on the FreeIPA clients directly so delegation
is not required at that stage. It is apparently required for gssproxy, however.

So now the only issue remaining is that if gssproxy is running when NFS share is mounted,
the logs are getting this repeatedly:

gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may
provide more information, No credentials cache found

Additionally, clients are not able to access their shares using the cached credentials.
This behaviour does not occur before the client is joined to the FreeIPA domain.

But when the gssproxy is restarted at this stage using "systemctl restart
gssproxy", everything works perfectly.

I would appreciate any of your thoughts on the issue.

Thanks so much,
James

[Dmitri Pal]

On 02/23/2016 10:14 PM, James Denton wrote:
>  Hello again,
>
>  I've made more progress on the issue.
>
>  In FreeIPA, setting the NFS client host to be managed by the FreeIPA server itself is
> allowing gssproxy to work.
>
>  E.g. On the FreeIPA server (freeipaserver.example.com):
>  kinit admin
>  ipa host-add-managedby nfs-client.example.com --hosts=freeipa-server-example.com
>
>  I didn't run into this issue in the past because the above step was required in order
> to get the keytab on the FreeIPA server (to be transferred to the client with scp). With
> my current setup, I am running ipa-getkeytab on the FreeIPA clients directly so delegation
> is not required at that stage. It is apparently required for gssproxy, however.
>
>  So now the only issue remaining is that if gssproxy is running when NFS share is mounted,
> the logs are getting this repeatedly:
>
>  gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure. Minor code may
> provide more information, No credentials cache found
>
>  Additionally, clients are not able to access their shares using the cached credentials.
> This behaviour does not occur before the client is joined to the FreeIPA domain.
>
>  But when the gssproxy is restarted at this stage using "systemctl restart
> gssproxy", everything works perfectly.
>
>  I would appreciate any of your thoughts on the issue. 

Do I get it right that the scenario is:

a) A client that is not enrolled
b) You add a host on the server
c) Use ipa-getkeytab on the client

You observe failures unless you restart gssproxy.

I think gssproxy needs to be restarted to read the keytab after you
configured it and provisioned the keytab. To the best of my knowledge
gssproxy reads keytab only at the beginning.

>  Thanks so much,
>  James
>  _______________________________________________
>  gss-proxy mailing list
>  gss-proxy(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/admin/lists/[e-mail] 

-- 
Thank you,
Dmitri Pal

Engineering Director, Identity Management and Platform Security
Red Hat, Inc.

[Simo Sorce]

On Wed, 2016-02-24 at 07:19 -0500, Dmitri Pal wrote:
>  On 02/23/2016 10:14 PM, James Denton wrote:
>  > Hello again,
>  >
>  > I've made more progress on the issue.
>  >
>  > In FreeIPA, setting the NFS client host to be managed by the FreeIPA
>  server itself is allowing gssproxy to work.
>  >
>  > E.g. On the FreeIPA server (freeipaserver.example.com):
>  > kinit admin
>  > ipa host-add-managedby nfs-client.example.com
>  --hosts=freeipa-server-example.com
>  >
>  > I didn't run into this issue in the past because the above step was
>  required in order to get the keytab on the FreeIPA server (to be
>  transferred to the client with scp). With my current setup, I am
>  running ipa-getkeytab on the FreeIPA clients directly so delegation is
>  not required at that stage. It is apparently required for gssproxy,
>  however.
>  >
>  > So now the only issue remaining is that if gssproxy is running when
>  NFS share is mounted, the logs are getting this repeatedly:
>  >
>  > gssproxy[642]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS
>  failure. Minor code may provide more information, No credentials cache
>  found
>  >
>  > Additionally, clients are not able to access their shares using the
>  cached credentials. This behaviour does not occur before the client is
>  joined to the FreeIPA domain.
>  >
>  > But when the gssproxy is restarted at this stage using "systemctl
>  restart gssproxy", everything works perfectly.
>  >
>  > I would appreciate any of your thoughts on the issue.
>  
>  
>  Do I get it right that the scenario is:
>  
>  a) A client that is not enrolled
>  b) You add a host on the server
>  c) Use ipa-getkeytab on the client
>  
>  You observe failures unless you restart gssproxy.
>  
>  I think gssproxy needs to be restarted to read the keytab after you
>  configured it and provisioned the keytab. To the best of my knowledge
>  gssproxy reads keytab only at the beginning. 
A restart shouldn't actually be necessary unless the gssproxy
configuration was changed.

A no credentials cache error hints at a user ccache missing not a keytab
missing. Do users actually have a ccache in this case ?
And where is their ccache located ?
How are they logging in the system ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York