[gssproxy] CentOS 7.2 + kerberized NFS not working. Only errors I see are from gssproxy

[Ray]

Hi there,

I'm trying (failing, actually) to set up kerberized NFS bewteen two 
CentOS 7.2 (1511) machines. When I try to mount my test export I see 
errors flying by that originate from gssproxy. Server runs IPA.

Setup:

Server:
   exports entry:
   /export  
192.168.10.0/24(rw,sec=krb5:krb5i:krb5p,no_subtree_check,no_root_squash)

Mount on client works just fine when I change security level to sec=sys

Client /etc/krb5.keytab looks like this:

Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- 
--------------------------------------------------------------------------
    5 host/client.example.org(a)EXAMPLE.ORG
    5 host/client.example.org(a)EXAMPLE.ORG
    5 host/client.example.org(a)EXAMPLE.ORG
    5 host/client.example.org(a)EXAMPLE.ORG
    3 nfs/client.example.org(a)EXAMPLE.ORG
    3 nfs/client.example.org(a)EXAMPLE.ORG
    3 nfs/client.example.org(a)EXAMPLE.ORG
    3 nfs/client.example.org(a)EXAMPLE.ORG

GSSProxy is version 0.4.1:

root@client:~# rpm -qa |grep gssproxy
gssproxy-0.4.1-7.el7.x86_64


gss-proxy.conf:

[gssproxy]
   debug = true

[service/HTTP]
   mechs = krb5
   cred_store = keytab:/etc/gssproxy/http.keytab
   cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
   euid = 48

[service/nfs-server]
   mechs = krb5
   socket = /run/gssproxy.sock
   cred_store = keytab:/etc/krb5.keytab
   trusted = yes
   kernel_nfsd = yes
   euid = 0

[service/nfs-client]
   mechs = krb5
   cred_store = keytab:/etc/krb5.keytab
   cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U
   cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab
   cred_usage = initiate
   allow_any_uid = yes
   trusted = yes
   euid = 0


Mounting the above export with sec=krb* results in this:

mount -t nfs4 -o sec=krb5p -v server.example.org:/export /mnt
mount.nfs4: timeout set for Wed Jun 22 10:24:35 2016
mount.nfs4: trying text-based options 
'sec=krb5p,addr=192.168.10.236,clientaddr=192.168.10.182'
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service 
"nfs-client", euid: 0, socket: (null)
gssproxy[10360]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS 
failure.  Minor code may provide more information, No credentials cache 
found
gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service 
"nfs-client", euid: 0, socket: (null)
gssproxy[10360]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS 
failure.  Minor code may provide more information, No credentials cache 
found
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 
server.example.org:/export

gssproxy.conf file is autogenerated, not hand-written (except for the 
debug = true line).

strace on 'open' indicates that /etc/krb5.keytab is not even read:

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpopt.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libverto.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libini_config.so.3", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libbasicobjects.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libref_array.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libcollection.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libgssrpc.so.4", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libgssapi_krb5.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpath_utils.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
Debug Enabled
open("/etc/gssproxy/gssproxy.conf", O_RDONLY) = 3
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 4
Debug Enabled
open("/etc/krb5.conf", O_RDONLY)        = 3
open("/var/lib/sss/pubconf/krb5.include.d//localauth_plugin", O_RDONLY) 
= 5
open("/var/lib/sss/pubconf/krb5.include.d//domain_realm_example_org", 
O_RDONLY) = 5
open("/dev/urandom", O_RDONLY)          = 3
open("/dev/urandom", O_RDONLY)          = 3
open("/dev/urandom", O_RDONLY)          = 3
open("/dev/urandom", O_RDONLY)          = 3
+++ exited with 0 +++



Searching Google for "(OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS 
failure.  Minor code may provide more information, No credentials cache 
found" (and parts of it) did not result in anything helpful.

I spent three solid days now examinig this and would appreciate a 
pointer about what's going wrong here.


Best,
Ray