Hello all,
I cannot get Kerberos security working on an NFSv4 server I'm setting up on RHEL7, using sssd with Microsoft Active Directory. The problem seems to lie with gssproxy. But I'm having a very difficult time with debugging.
gssproxy has a "-d" flag that enables debugging. But it is pretty much useless. :-( The only additional information it syslogs is:
Nov 13 15:00:08 server gssproxy: gp_rpc_execute: executing 9 (GSSX_ACCEPT_SEC_CONTEXT) for service "nfs-server", euid: 0, socket: /run/gssproxy.sock Nov 13 15:00:08 server rpc.gssd[4921]: destroying client /var/lib/nfs/rpc_pipefs/nfsd4_cb/clnt2
If I strace the gssproxy daemon, I see error messages like this:
2464 open("/var/lib/nfs/rpc_pipefs/nfsd4_cb/clnta/krb5", O_RDWR) = -1 ENOENT (No such file or directory) 2464 open("/var/lib/nfs/rpc_pipefs/nfsd4_cb/clnta/gssd", O_RDWR) = -1 ENOENT (No such file or directory)
But I'm not sure if these errors are important.
If I tcpdump the connection between the client and the server, I see that the server responds to the client's AP-REQ attempts by simply closing the TCP connection.
We already have RHEL7 NFSv4 clients that use sec=krb5 with a commercial NAS NFSv4 server, so I know the basics of configuring NFSv4 with krb5 security. The same client configuration works perfectly with the NAS server using krb5/krb5i/krb5p security. And I can mount the Linux NFS server as long as I use sec=sys; it only fails when I use sec=krb5 as a mount option. So it's very clearly the Kerberos piece that's failing on the server.
But I have no idea where to go from here, because the debugging information that gssproxy emits gives me no clues as to what's wrong on the server.
There has to be some way to get more debugging information than this. Help?
On 13/11/15 18:31, James Ralston wrote:
Hello all,
I cannot get Kerberos security working on an NFSv4 server I'm setting up on RHEL7, using sssd with Microsoft Active Directory.
It is possible you are hitting this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1213852
The problem seems to lie with gssproxy. But I'm having a very difficult time with debugging.
gssproxy has a "-d" flag that enables debugging. But it is pretty much useless. :-( The only additional information it syslogs is:
Nov 13 15:00:08 server gssproxy: gp_rpc_execute: executing 9 (GSSX_ACCEPT_SEC_CONTEXT) for service "nfs-server", euid: 0, socket: /run/gssproxy.sock Nov 13 15:00:08 server rpc.gssd[4921]: destroying client /var/lib/nfs/rpc_pipefs/nfsd4_cb/clnt2
If I strace the gssproxy daemon, I see error messages like this:
2464 open("/var/lib/nfs/rpc_pipefs/nfsd4_cb/clnta/krb5", O_RDWR) = -1 ENOENT (No such file or directory) 2464 open("/var/lib/nfs/rpc_pipefs/nfsd4_cb/clnta/gssd", O_RDWR) = -1 ENOENT (No such file or directory)
But I'm not sure if these errors are important.
If I tcpdump the connection between the client and the server, I see that the server responds to the client's AP-REQ attempts by simply closing the TCP connection.
We already have RHEL7 NFSv4 clients that use sec=krb5 with a commercial NAS NFSv4 server, so I know the basics of configuring NFSv4 with krb5 security. The same client configuration works perfectly with the NAS server using krb5/krb5i/krb5p security. And I can mount the Linux NFS server as long as I use sec=sys; it only fails when I use sec=krb5 as a mount option. So it's very clearly the Kerberos piece that's failing on the server.
But I have no idea where to go from here, because the debugging information that gssproxy emits gives me no clues as to what's wrong on the server.
There has to be some way to get more debugging information than this. Help?
We are improving debugging options for future versions. One trick you can use is to start gssproxy with the KRB5_TRACE environment variable set. The value is a file path (for example /tmp/gssproxy_trace, to see if there are krb5 errors underneath the gssapi interface).
Simo.
On Sat, Nov 14, 2015 at 7:05 PM, Simo Sorce simo@redhat.com wrote:
On 13/11/15 18:31, James Ralston wrote:
Hello all,
I cannot get Kerberos security working on an NFSv4 server I'm setting up on RHEL7, using sssd with Microsoft Active Directory.
It is possible you are hitting this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1213852
My Bugzilla account isn't authorized to access that bug. Is there a public summary of it?
We are improving debugging options for future versions.
Good to know.
With debugging output, patience, and (if necessary) a tcpdump capture, I can usually figure out most issues. But since the tcpdump output isn't helpful (the server doesn't return an error; it just closes the TCP connection), debugging output is all I have to go on.
One trick you can use is to start gssproxy with the KRB5_TRACE environment variable set. The value is a file path (for example /tmp/gssproxy_trace, to see if there are krb5 errors underneath the gssapi interface).
Thanks; I wasn't aware of KRB5_TRACE. I'll give that a try on Monday.
James
On 15/11/15 16:12, James Ralston wrote:
On Sat, Nov 14, 2015 at 7:05 PM, Simo Sorce simo@redhat.com wrote:
On 13/11/15 18:31, James Ralston wrote:
Hello all,
I cannot get Kerberos security working on an NFSv4 server I'm setting up on RHEL7, using sssd with Microsoft Active Directory.
It is possible you are hitting this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1213852
My Bugzilla account isn't authorized to access that bug. Is there a public summary of it?
Ah sorry it is private as it began as a customer case. Long story short there is a kernel bug that triggers this, we have patches for both the kernel and a wrokaround for gssproxy that should deal with this.
We are improving debugging options for future versions.
Good to know.
With debugging output, patience, and (if necessary) a tcpdump capture, I can usually figure out most issues. But since the tcpdump output isn't helpful (the server doesn't return an error; it just closes the TCP connection), debugging output is all I have to go on.
One trick you can use is to start gssproxy with the KRB5_TRACE environment variable set. The value is a file path (for example /tmp/gssproxy_trace, to see if there are krb5 errors underneath the gssapi interface).
Thanks; I wasn't aware of KRB5_TRACE. I'll give that a try on Monday.
Can you tell me also what kernel and gssproxy version do you have ?
Simo.
On Sun, Nov 15, 2015 at 4:22 PM, Simo Sorce simo@redhat.com wrote:
Long story short there is a kernel bug that triggers this, we have patches for both the kernel and a wrokaround for gssproxy that should deal with this.
If you can point me to the workaround patches for gssproxy, I'll roll a local gssproxy RPM with the patches to see if that addresses our problem.
Can you tell me also what kernel and gssproxy version do you have ?
$ repoquery --envra --installed gssproxy kernel 0:gssproxy-0.3.0-10.el7.x86_64 0:kernel-3.10.0-123.el7.x86_64 0:kernel-3.10.0-229.14.1.el7.x86_64 0:kernel-3.10.0-229.20.1.el7.x86_64
$ uname -r 3.10.0-229.20.1.el7.x86_64
Thanks, James
On 15/11/15 16:38, James Ralston wrote:
On Sun, Nov 15, 2015 at 4:22 PM, Simo Sorce simo@redhat.com wrote:
Long story short there is a kernel bug that triggers this, we have patches for both the kernel and a wrokaround for gssproxy that should deal with this.
If you can point me to the workaround patches for gssproxy, I'll roll a local gssproxy RPM with the patches to see if that addresses our problem.
Patches were release with 0.4.0.
Can you tell me also what kernel and gssproxy version do you have ?
$ repoquery --envra --installed gssproxy kernel 0:gssproxy-0.3.0-10.el7.x86_64 0:kernel-3.10.0-123.el7.x86_64 0:kernel-3.10.0-229.14.1.el7.x86_64 0:kernel-3.10.0-229.20.1.el7.x86_64
$ uname -r 3.10.0-229.20.1.el7.x86_64
Btw if you are a RH customer open a case and we can help you with an hotfix until the packages is released for general availability (real soon now anyway I think).
Simo.
On Mon, Nov 16, 2015 at 9:22 AM, Simo Sorce simo@redhat.com wrote:
Patches were release with 0.4.0.
I rebuilt gssproxy-0.4.1-2.fc23.src.rpm for RHEL7 and installed it on the NFS server, and indeed, I can perform NFS v4.0 mounts against the server now. So it would seem that we are indeed being hit by BZ#1213852.
Thanks much for pointing us in this direction; it's very unlikely we would've figured this out on our own!
Btw if you are a RH customer open a case and we can help you with an hotfix until the packages is released for general availability (real soon now anyway I think).
Yes, we are a RH customer, and we will pile on. :-)
Two other quick questions, if you have anything to add:
1. Although mounting with nfsvers=4.0 works fine, when I attempt to mount with nfsvers=4.1 or nfsvers=4.2 (if I explicitly enable it), the server returns NFS4ERR_WRONG_CRED in response to the CREATE_SESSION request. (gssproxy doesn't log anything different.)
Red Hat claims to support NFSv4.1 clients and servers on RHEL7. Do you know if NFS 4.1/4.2 support is also a known issue with sec=krb5 with Microsoft AD, or is this an issue you haven't heard about?
We really want to use NFS 4.1 instead of 4.0, because otherwise we have to change many firewall rules to permit callbacks from the server to the clients. (NFS 4.2 would be even better, because that would get us SELinux file context support.)
2. On the NFS client, is there a way to tell gssproxy to use the $KRB5CCNAME credentials if I sudo to root, instead of using the client's host credentials from /etc/krb5.keytab? Because otherwise, users who sudo to root will lose all access to their NFS-mounted home directories (unless they temporarily give the client's host credentials access to their home directories before they sudo).
Thanks, James
On 17/11/15 15:28, James Ralston wrote:
On Mon, Nov 16, 2015 at 9:22 AM, Simo Sorce simo@redhat.com wrote:
Patches were release with 0.4.0.
I rebuilt gssproxy-0.4.1-2.fc23.src.rpm for RHEL7 and installed it on the NFS server, and indeed, I can perform NFS v4.0 mounts against the server now. So it would seem that we are indeed being hit by BZ#1213852.
Thanks much for pointing us in this direction; it's very unlikely we would've figured this out on our own!
Btw if you are a RH customer open a case and we can help you with an hotfix until the packages is released for general availability (real soon now anyway I think).
Yes, we are a RH customer, and we will pile on. :-)
Two other quick questions, if you have anything to add:
- Although mounting with nfsvers=4.0 works fine, when I attempt to
mount with nfsvers=4.1 or nfsvers=4.2 (if I explicitly enable it), the server returns NFS4ERR_WRONG_CRED in response to the CREATE_SESSION request. (gssproxy doesn't log anything different.)
Red Hat claims to support NFSv4.1 clients and servers on RHEL7. Do you know if NFS 4.1/4.2 support is also a known issue with sec=krb5 with Microsoft AD, or is this an issue you haven't heard about?
I am not aware of any difference in the kernel or userspace code when it comes to rpcgss handling, please open a case/bugzilla, if you are seeing something beyond BZ#1213852.
We really want to use NFS 4.1 instead of 4.0, because otherwise we have to change many firewall rules to permit callbacks from the server to the clients. (NFS 4.2 would be even better, because that would get us SELinux file context support.)
Yeah I share your preference :)
- On the NFS client, is there a way to tell gssproxy to use the
$KRB5CCNAME credentials if I sudo to root, instead of using the client's host credentials from /etc/krb5.keytab? Because otherwise, users who sudo to root will lose all access to their NFS-mounted home directories (unless they temporarily give the client's host credentials access to their home directories before they sudo).
I think what you want is achieved with the rpc.gssd -n argument (see rpc.gssd manpage), you can set arguments in /etc/sysconfig/nfs and then restart nfs-config.service and nfs-secure.service
Simo.
On Tue, Nov 17, 2015 at 6:31 PM, Simo Sorce simo@redhat.com wrote:
On 17/11/15 15:28, James Ralston wrote:
Red Hat claims to support NFSv4.1 clients and servers on RHEL7. Do you know if NFS 4.1/4.2 support is also a known issue with sec=krb5 with Microsoft AD, or is this an issue you haven't heard about?
I am not aware of any difference in the kernel or userspace code when it comes to rpcgss handling, please open a case/bugzilla, if you are seeing something beyond BZ#1213852.
It looks like your assistance will be needed with this one:
https://bugzilla.redhat.com/show_bug.cgi?id=1283341
On the NFS client, is there a way to tell gssproxy to use the $KRB5CCNAME credentials if I sudo to root, instead of using the client's host credentials from /etc/krb5.keytab?
I think what you want is achieved with the rpc.gssd -n argument
Yes, but then NFS4.0 mounts with sec=krb5 break. :-(
I'll file a separate bug for that.
Thanks, James
On Fri, 2015-11-20 at 14:09 -0500, James Ralston wrote:
On Tue, Nov 17, 2015 at 6:31 PM, Simo Sorce simo@redhat.com wrote:
On 17/11/15 15:28, James Ralston wrote:
Red Hat claims to support NFSv4.1 clients and servers on RHEL7. Do you know if NFS 4.1/4.2 support is also a known issue with sec=krb5 with Microsoft AD, or is this an issue you haven't heard about?
I am not aware of any difference in the kernel or userspace code when it comes to rpcgss handling, please open a case/bugzilla, if you are seeing something beyond BZ#1213852.
It looks like your assistance will be needed with this one:
From what I can read the problem is entirely in kernel, where it refuses to use a valid principal name.
On the NFS client, is there a way to tell gssproxy to use the $KRB5CCNAME credentials if I sudo to root, instead of using the client's host credentials from /etc/krb5.keytab?
I think what you want is achieved with the rpc.gssd -n argument
Yes, but then NFS4.0 mounts with sec=krb5 break. :-(
:-/
I'll file a separate bug for that.
Thanks.
Simo.
gss-proxy@lists.fedorahosted.org
-
James Ralston -
Simo Sorce