Kerberized NFS on two identical VMs. But mounting works only from one.

[Ray]

Hi,

I run FreeIPA across a few sites with five replicted servers. The IPA 
version is the current CentOS one: 4.5.0-21

At two of those sites a kerberized NFS service is offered to the client 
machines. All clients and servers involved in the are CentOS 7.4 boxes.

For both NFS servers I configured NFS service pricipals and when I click 
my way in the GUI Identity -> Services -> nfs.server1 resp. nfs.server2 
I get to see "Kerberos Key Present, Service Provisioned" for both. So 
far things seem ok.

However, mounting works only from server1, for clients at both sites 
(site1 to site2 mounting and vice versa is allowed). Mounting anything 
from server2 keeps failing:

Site 2: local mount attempt:
[e-mail]2:~# mount -vv -t nfs4 -osec=krb5p 
server.at.site2:/local/test /mnt
mount.nfs4: timeout set for Sat Dec  9 17:03:02 2017
mount.nfs4: trying text-based options 
'sec=krb5p,vers=4.1,addr=xx.xx.xx.xx,clientaddr=yy.yy.yy.yy'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting 
server.at.site2:/local/test
[e-mail]2:~#

Site 2: remote mount attempt:
[e-mail]2:~# mount -vv -t nfs4 -osec=krb5p 
server.at.site1:/local/test /mnt
mount.nfs4: timeout set for Sat Dec  9 17:03:10 2017
mount.nfs4: trying text-based options 
'sec=krb5p,vers=4.1,addr=zz.zz.zz.zz,clientaddr=yy.yy.yy.yy'
[e-mail]2:~#

At site2's server I disabled:
   - the firewall
   - selinux

Exports are identical at both sites

I don't see what might be the problem here. How can I debug this? Tried 
to enable all sorts of debug flags in /etc/sysconfig/nfs on site2's 
server:

RPCNFSDARGS="-d"
RPCMOUNTDOPTS="-dall"
STATDARG=""
SMNOTIFYARGS=""
RPCIDMAPDARGS=""
RPCGSSDARGS="-vvv -r"
RPCSVCGSSDARGS="-vvv"
GSS_USE_PROXY="yes"
BLKMAPDARGS=""
SECURE_NFS="yes"

I did restart nfs with systemctl restart nfs-server, but neither there's 
not much happening in tail -f /var/log/messages not journalctl -f show 
anything new on failing mount attemppts as shown above.

I did read http://wiki.linux-nfs.org/wiki/index.php/NFS_and_FreeIPA, but 
this seems outdated in several spots: Is SECURE_NFS still required in 
/etv/sysconfig/nfs, for instance?

The fact that I can mount anything at all on the client indicates that 
the client is ok. In desparation, I reinstalled the NFS server at site2 
last weekend from scratch. But now I run into the same issue as before. 
Might there be something wrong with the service principals after all?

I would sincerely appreciate suggestions that help me solve this.

Best,
Ray

[Robbie Harwood]

Ray via FreeIPA-users  writes:
> I run FreeIPA across a few sites with five replicted servers. The IPA
> version is the current CentOS one: 4.5.0-21
>
> At two of those sites a kerberized NFS service is offered to the
> client machines. All clients and servers involved in the are CentOS
> 7.4 boxes.

Unfortunately a lot of this code changes in 7.5, but let me check if
anything obvious is wrong.
> For both NFS servers I configured NFS service pricipals and when I
> click my way in the GUI Identity -> Services -> nfs.server1
> resp. nfs.server2 I get to see "Kerberos Key Present, Service
> Provisioned" for both. So far things seem ok.
>
> However, mounting works only from server1, for clients at both sites
> (site1 to site2 mounting and vice versa is allowed). Mounting anything
> from server2 keeps failing:
>
> Site 2: local mount attempt:
> [e-mail]2:~# mount -vv -t nfs4 -osec=krb5p 
> server.at.site2:/local/test /mnt
> mount.nfs4: timeout set for Sat Dec  9 17:03:02 2017
> mount.nfs4: trying text-based options 
> 'sec=krb5p,vers=4.1,addr=xx.xx.xx.xx,clientaddr=yy.yy.yy.yy'
> mount.nfs4: mount(2): Permission denied
> mount.nfs4: access denied by server while mounting 
> server.at.site2:/local/test
> [e-mail]2:~#

How long does this failure take?  Is it immediate, or does it take more
than a minute or so?
> Site 2: remote mount attempt:
> [e-mail]2:~# mount -vv -t nfs4 -osec=krb5p 
> server.at.site1:/local/test /mnt
> mount.nfs4: timeout set for Sat Dec  9 17:03:10 2017
> mount.nfs4: trying text-based options 
> 'sec=krb5p,vers=4.1,addr=zz.zz.zz.zz,clientaddr=yy.yy.yy.yy'
> [e-mail]2:~#

Can you check rpc-gssd logs on the machine you're mounting from?
> At site2's server I disabled:
> the firewall
> selinux

If you turn on selinux, do things change?
> I did restart nfs with systemctl restart nfs-server, but neither
> there's not much happening in tail -f /var/log/messages not journalctl
> -f show anything new on failing mount attemppts as shown above.

Can you post gssproxy logs during the failed mount attempt from site2?
> The fact that I can mount anything at all on the client indicates that
> the client is ok. In desparation, I reinstalled the NFS server at
> site2 last weekend from scratch. But now I run into the same issue as
> before.  Might there be something wrong with the service principals
> after all?

`klist -ek` the keytab on both sites.  Also check kvno for all
principals involved.

Thanks,
--Robbie

[Ray]

I have it working in the meantime, but I'm not sure why: I had a 
suspicion this was perhaps related to 
https://bugzilla.redhat.com/show_bug.cgi?id=1488629 and decided to 
downgrade gssproxy to 0.4.1, hoping this would resolve the issue. But it 
didn't. So I upgraded both ends to gssproxy-0.7.0-4 again, rebooted, and 
now it works.

Thanks,
Ray

Am 2017-12-12 23:00, schrieb Robbie Harwood via FreeIPA-users:
> Ray via FreeIPA-users  writes:
> > I run FreeIPA across a few sites with five replicted servers. The IPA
> > version is the current CentOS one: 4.5.0-21
> >
> > At two of those sites a kerberized NFS service is offered to the
> > client machines. All clients and servers involved in the are CentOS
> > 7.4 boxes.
>
> Unfortunately a lot of this code changes in 7.5, but let me check if
> anything obvious is wrong.
> > For both NFS servers I configured NFS service pricipals and when I
> > click my way in the GUI Identity -> Services -> nfs.server1
> > resp. nfs.server2 I get to see "Kerberos Key Present, Service
> > Provisioned" for both. So far things seem ok.
> >
> > However, mounting works only from server1, for clients at both sites
> > (site1 to site2 mounting and vice versa is allowed). Mounting anything
> > from server2 keeps failing:
> >
> > Site 2: local mount attempt:
> > [e-mail]2:~# mount -vv -t nfs4 -osec=krb5p
> > server.at.site2:/local/test /mnt
> > mount.nfs4: timeout set for Sat Dec  9 17:03:02 2017
> > mount.nfs4: trying text-based options
> > 'sec=krb5p,vers=4.1,addr=xx.xx.xx.xx,clientaddr=yy.yy.yy.yy'
> > mount.nfs4: mount(2): Permission denied
> > mount.nfs4: access denied by server while mounting
> > server.at.site2:/local/test
> > [e-mail]2:~#
>
> How long does this failure take?  Is it immediate, or does it take more
> than a minute or so?
> > Site 2: remote mount attempt:
> > [e-mail]2:~# mount -vv -t nfs4 -osec=krb5p
> > server.at.site1:/local/test /mnt
> > mount.nfs4: timeout set for Sat Dec  9 17:03:10 2017
> > mount.nfs4: trying text-based options
> > 'sec=krb5p,vers=4.1,addr=zz.zz.zz.zz,clientaddr=yy.yy.yy.yy'
> > [e-mail]2:~#
>
> Can you check rpc-gssd logs on the machine you're mounting from?
> > At site2's server I disabled:
> > the firewall
> > selinux
>
> If you turn on selinux, do things change?
> > I did restart nfs with systemctl restart nfs-server, but neither
> > there's not much happening in tail -f /var/log/messages not journalctl
> > -f show anything new on failing mount attemppts as shown above.
>
> Can you post gssproxy logs during the failed mount attempt from site2?
> > The fact that I can mount anything at all on the client indicates that
> > the client is ok. In desparation, I reinstalled the NFS server at
> > site2 last weekend from scratch. But now I run into the same issue as
> > before.  Might there be something wrong with the service principals
> > after all?
>
> `klist -ek` the keytab on both sites.  Also check kvno for all
> principals involved.
>
> Thanks,
> --Robbie
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]