[gssproxy] Re: gssproxy broken on fedora

[Rob Verduijn]

I did another default install with centos7 applied all the patches, and
made it a domain member, configured the automount (as above)

configured it the same way I described above with the minor difference that
I used only 1 configuration file for gssproxy:
cat /etc/gssproxy/gssproxy.conf
[gssproxy]
debug = true
debug_level = 9

[service/HTTP]
  mechs = krb5
  cred_store = client_keytab:/etc/gssproxy/http.keytab
  cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
  euid = 48

[service/nfs-server]
  mechs = krb5
  socket = /run/gssproxy.sock
  cred_store = keytab:/etc/krb5.keytab
  trusted = yes
  kernel_nfsd = yes
  euid = 0

[service/nfs-client]
  mechs = krb5
  cred_store = keytab:/etc/krb5.keytab
  cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U
  cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab
  cred_usage = initiate
  allow_any_uid = yes
  trusted = yes
  euid = 0

In the service/HTTP instance I changed keytab to client_keytab.
and tested it and it worked.

so I did an strace on rpc.gssd on centos 7.3
And got this :
epoll_wait(4,
{{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
read(9, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45
clone(child_stack=0x7fd125bfdeb0,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7fd125bfe9d0, tls=0x7fd125bfe700,
child_tidptr=0x7fd125bfe9d0) = 2531
epoll_wait(4,

on the fedora25-kde the strace looks like this :
epoll_wait(4,
[{EPOLLIN, {u32=12, u64=12}}], 32, -1) = 1
read(12, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45
clone(child_stack=0x7f75c4aa3ef0,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tidptr=0x7f75c4aa49d0, tls=0x7f75c4aa4700,
child_tidptr=0x7f75c4aa49d0) = 16294
epoll_wait(4,

This is al more or less giberish to me, but I think the important bit is
'mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45'
and that is the same on both systems.

Also the environment of the rpc.gssd proc entry

the output from /proc/<pid>/environ on centos7.3:
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binRPCNFSDARGS=
8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=GSSDARGS=SVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes

The output from /proc/<pid>/environ on fedora25-kde:
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binJOURNAL_STREAM=8:22877RPCNFSDARGS=
8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=RPCGSSDARGS=RPCSVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes

Rob Verduijn

2017-01-29 15:33 GMT+01:00 Rob Verduijn :
> Hi,
>
> I guess I'm missing something very basic here.
>
> Just to make sure I'm not missing something very elementary here, could
> you review the following ?
>
> So all my exports work fine when using ordinary user accounts on fedora
> clients (kde-spin)
> So I doubt that there is something wrong on the nfs server side or the ipa
> bit.
> I installed the default fedora25-kde spin with additional groups :
> @domain-client
> no other packages where added.
>
> A host and dns entry were created in freeipa
> An otp password is set
>
> The command 'ipa-client install --enable-dns-updates -U -w ********'  has
> been applied
> The command 'ipa-client-automount' has been issued.
>
> a http principal has been created for this host via the freeipa gui
> imported it to the client with
> ipa-getkeytab -p HTTP/[e-mail] -k
> /var/lib/gssproxy/clients/httpd.keytab
>
> created the 00-apache.conf
> cat > /etc/gssproxy/00-apache.conf << EOF
> [service/apache]
>   mechs = krb5
>   cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache
>   cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab
>   cred_usage = initiate
>   euid = 48
> EOF
>
> added debug settings to gssproxy.conf
> echo 'debug = true' >> /etc/gssproxy/gssproxy.conf
> echo 'debug_level = 9' >> /etc/gssproxy/gssproxy.conf
>
> nfsidmap resolving works better for me when the method is set to sss.
> the command 'sed -i 's/#Method.*/Method=sss/' /etc/idmapd.conf'  is applied
>
> updated everything :
> dnf update -y
>
> added the latest gssproxy from testing
> dnf update -y gssproxy --enablerepo=updates-testing
>
> add httpd
> dnf install -y httpd
>
> reboot
>
> the automount map from ipa that I am using :
> ipa automountkey-show
> Location: default
> Map: auto.direct
> Key: /network/music
>   Key: /network/music
>   Mount information: -rw,noatime,timeo=14,soft,sec=krb5p,acl
> nfs.example.com:/music
>
> login to the system as root
> 'ls /network/music' gives a directory listing
>
> issue 'su - apache -s /bin/bash'
> now 'ls /network/music' gives 'ls: cannot open directory '/network/music':
> Permission denied'
>
> I logout
> ctrl-d , ctrl-d
>
> I login as an ipa domain user. (not admin)
> 'ls /network/music' gives a directory listing.
>
> the root user does not exist in the domain
> the apache user does not exits in the domain
>
> the apache user has local id 48 on the nfs server and local id 48 on the
> nfs client.
> the apache user has group apache with id 48 on the nfs client and nfs
> server.
>
> 2017-01-29 14:10 GMT+01:00 Simo Sorce :
> > So it turned out I was not reproducing the issue, I had an older version
> > of gssproxy than I thought, and incorrect logging level in the config file.
> >
> > Once I had time to sit down and fix my client by installing the latest
> > updates-testing package and making sure the debug level was properly set
> > all worked as expected for me on Fedora 25.
> >
> > I was suspecting some issues with a relatively recent patchset that went
> > into nfs-utils which switched from using fork() to use threads, but it
> > turned out this situation is already properly handled in the gss plugin
> > used to talk to gssproxy. (CCing Jeff who reviewed those patches so he
> > knows it is all good for now :-)
> >
> > Rob,
> > in order to find out what is going on you may want to try to strace
> > rpc.gssd on the client system and try to figure out why gssproxy is not
> > being used, also check if /proc/<pid>/env of rpc.gssd does in fact include
> > the proper environment variable.
> >
> > Simo.
> >
> > ----- Original Message -----
> > > From: "Simo Sorce" 
> > > To: "The GSS-Proxy developers and users mailing list" <
> >
> > [e-mail]>
> > > Sent: Saturday, January 28, 2017 7:05:41 AM
> > > Subject: Re: [gssproxy] Re: gssproxy broken on fedora
> > >
> > > Hi Rob,
> > > I can reproduce your issue now and I am looking into this issue and I
> > > suspect nfs-utils may have broken gss-proxy support recently.
> > >
> > > I am not sure yet, but some critical areas have been changed, so I need
> > > to better investigate them. It will take me a little time as I am at
> > > Devconf.cz now, but hopefully I will have an answer for you in a few
> > > days.
> > >
> > > Simo.
> > >
> > > On Sat, 2017-01-21 at 13:10 +0100, Rob Verduijn wrote:
> > > > Another fresh install of a fedora25 client, now with the new gssproxy
> > > > 0.6.0 package, but that one does not work either.
> > > >
> > > > Created the file /etc/gssproxy/00-apache.conf
> > > > [service/apache]
> > > >   mechs = krb5
> > > >   cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache
> > > >   cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab
> > > >   cred_usage = initiate
> > > >   euid = 48
> > > >
> > > > put the keytab in the specified place
> > > > added debug settings to /etc/gssproxy/gssproxy.conf
> > > > [gssproxy]
> > > > debug = true
> > > > debug_level = 9
> > > >
> > > > I did  not touch the file /etc/sysconfig/nfs
> > > > Since the default setting of GSS_USE_PROXY="yes" is what I want
> > > >
> > > > Checked the logs, nothing in there besides what you get from
> > > > systemctl status gssproxy.service
> > > >
> > > > su - apache -s /bin/bash
> > > > and no access to the shares is allowed
> > > >
> > > > Weird thing I noticed, when browsing as root the nfs mounts are
> > > > readable ( root is squashed ) but not as the apache user.
> > > > Root gets to read it with nobody:nobody privileges but apache with
> > > > apache:apache is refused.
> > > >
> > > > I'm really at a loss as to what to do next.
> > > >
> > > > Do I need to set an environment variable to make this work ?
> > > >
> > > > Rob Verduijn
> > > >
> > > > 2017-01-04 23:14 GMT+01:00 Rob Verduijn :
> > > > > 2017-01-04 20:56 GMT+01:00 Simo Sorce :
> > > > > > On Wed, 2017-01-04 at 19:41 +0100, Rob Verduijn wrote:
> > > > > > > 2017-01-04 19:27 GMT+01:00 Dmitri Pal :
> > > > > > >
> > > > > > > > On 01/04/2017 01:13 PM, Rob Verduijn wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-01-04 14:59 GMT+01:00 Simo Sorce :
> > > > > > > >
> > > > > > > >> On Wed, 2017-01-04 at 10:16 +0100, Rob Verduijn wrote:
> > > > > > > >> > ---------- Forwarded message ----------
> > > > > > > >> > From: Simo Sorce 
> > > > > > > >> > Date: 2017-01-03 17:32 GMT+01:00
> > > > > > > >> > Subject: [gssproxy] Re: gssproxy broken on fedora
> > > > > > > >> > To: The GSS-Proxy developers and users mailing list <
> > > > > > > >> > [e-mail]>
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, 2017-01-02 at 19:22 +0100, Rob Verduijn wrote:
> > > > > > > >> > >
> > > > > > > >> > > Nope that does not work on either fc24 or fc25.
> > > > > > > >> > > I did not try centos73 since it already worked on that
> > > > > > one.
> > > > > > > >> >
> > > > > > > >> > Given you tried manually, make sure you delete the ccache
> > > > > > before trying
> > > > > > > >> > with the client_keytab setting.
> > > > > > > >> >
> > > > > > > >> > If that doesn't work can you set debug = True in the
> > > > > > global section and
> > > > > > > >> > tell me if you get any useful output/error ?
> > > > > > > >> >
> > > > > > > >> > Simo.
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Simo Sorce * Red Hat, Inc * New York
> > > > > > > >> > _______________________________________________
> > > > > > > >> > gss-proxy mailing list -- [e-mail]
> > > > > > > >> > To unsubscribe send an email to [e-mail]
> > > > > > rahosted.org
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Hi,
> > > > > > > >> >
> > > > > > > >> > I checked for the cache, but there were no cache files
> > > > > > present in
> > > > > > > >> > /var/lib/gssproxy/clients.
> > > > > > > >> > I cleaned the sssd cache.
> > > > > > > >> > I set the debug entry, did a reboot, but also no log
> > > > > > entries appeared
> > > > > > > >> >
> > > > > > > >> > current /etc/gssproxy/gssproxy.conf
> > > > > > > >> >
> > > > > > > >> > [gssproxy]
> > > > > > > >> > debug=True
> > > > > > > >> >
> > > > > > > >> > [service/HTTP]
> > > > > > > >> >   mechs = krb5
> > > > > > > >> >   cred_store = keytab:/etc/gssproxy/http.keytab
> > > > > > > >> >   cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
> > > > > > > >> >   cred_store = client_keytab:/etc/gssproxy/http.keytab
> > > > > > > >> >   euid = 48
> > > > > > > >> >
> > > > > > > >> > and tested it with
> > > > > > > >> > su - apache -s /bin/bash
> > > > > > > >> >
> > > > > > > >> > The mount works fine for a regular ipa user on fedora
> > > > > > 24/25
> > > > > > > >> > according to systemctl status gssproxy the service is up
> > > > > > and running,
> > > > > > > >> >
> > > > > > > >> > [root@fedora-24 ~]# systemctl status gssproxy
> > > > > > > >> > ● gssproxy.service - GSSAPI Proxy Daemon
> > > > > > > >> >    Loaded: loaded
> > > > > > (/usr/lib/systemd/system/gssproxy.service; disabled;
> > > > > > > >> > vendor preset: disabled)
> > > > > > > >> >    Active: active (running) since Wed 2017-01-04 10:05:55
> > > > > > CET; 8min ago
> > > > > > > >> >  Main PID: 987 (gssproxy)
> > > > > > > >> >    CGroup: /system.slice/gssproxy.service
> > > > > > > >> >            └─987 /usr/sbin/gssproxy -D
> > > > > > > >> >
> > > > > > > >> > systemd[1]: Starting GSSAPI Proxy Daemon...
> > > > > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Debug Enabled
> > > > > > (level: 1)
> > > > > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Client connected (fd
> > > > > > =
> > > > > > > >> 10)[2017/01/04
> > > > > > > >> > 09:05:55]:  (pid = 987) (uid = 0) (gid = 0)[2017/01/04
> > > > > > 09:05:55]:
> > > > > > > >> (context
> > > > > > > >> > = system_u:system_r:kernel_t:s0)[2017/01/04 09:05:55]:
> > > > > > > >> > Started GSSAPI Proxy Daemon.
> > > > > > > >>
> > > > > > > >> If you turn on rpc.gssd debugging and kernel rpc debugging
> > > > > > do you see
> > > > > > > >> anything relevant ?
> > > > > > > >>
> > > > > > > >> Simo.
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Simo Sorce * Red Hat, Inc * New York
> > > > > > > >> _______________________________________________
> > > > > > > >> gss-proxy mailing list -- [e-mail]
> > > > > > > >> To unsubscribe send an email to [e-mail]
> > > > > > hosted.org
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > It does not seem to look for the credits specified in the
> > > > > > gssproxy.conf
> > > > > > > > file.
> > > > > > > > How can I verify the running configuration of gssproxy ?
> > > > > > > >
> > > > > > > > Rob Verduijn
> > > > > > > >
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]:
> > > > > > #012handle_gssd_upcall:
> > > > > > > > 'mech=krb5 uid=48 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt0)
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]:
> > > > > > krb5_not_machine_creds: uid 48
> > > > > > > > tgtname (null)
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: ERROR: GSS-API:
> > > > > > error in
> > > > > > > > gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure.
> > > > > > Minor code may
> > > > > > > > provide more information) - No Kerberos credentials available
> > > > > > (default
> > > > > > > > cache: KEYRING:persistent:48)
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client
> > > > > > creds with
> > > > > > > > uid 48 for server nfs.example.com in /tmp
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: CC
> > > > > > '/tmp/krb5ccmachine_EXAMPLE.COM'
> > > > > > > > being considered, with preferred realm 'EXAMPLE.COM'
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: CC
> > > > > > '/tmp/krb5ccmachine_EXAMPLE.COM'
> > > > > > > > owned by 0, not 48
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client
> > > > > > creds with
> > > > > > > > uid 48 for server nfs.example.com in /run/user/%U
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: Error doing scandir
> > > > > > on directory
> > > > > > > > '/run/user/48': No such file or directory
> > > > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: doing error
> > > > > > downcall
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > gss-proxy mailing list -- [e-mail]
> > > > > > > > To unsubscribe send an email to [e-mail]
> > > > > > osted.org
> > > > > > > >
> > > > > > > >
> > > > > > > > Why are you preferring credential cache in a file over a
> > > > > > keyring which is
> > > > > > > > default?
> > > > > > > > Have you tried without cred_store =
> > > > > > ccache:/var/lib/gssproxy/clients/krb5cc_%U
> > > > > > > > ?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Thank you,
> > > > > > > > Dmitri Pal
> > > > > > > >
> > > > > > > > Engineering Director, Identity Management and Platform
> > > > > > Security
> > > > > > > > Red Hat, Inc.
> > > > > > > >
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > gss-proxy mailing list -- [e-mail]
> > > > > > > > To unsubscribe send an email to [e-mail]
> > > > > > osted.org
> > > > > > > >
> > > > > > > >
> > > > > > > Because it said so in the example here :
> > > > > > > https://fedorahosted.org/gss-proxy/wiki/Apache
> > > > > > >
> > > > > > > But  have tried it without and it still fails.
> > > > > > >
> > > > > > > Rob Verduijn
> > > > > > > _______________________________________________
> > > > > > > gss-proxy mailing list -- [e-mail]
> > > > > > > To unsubscribe send an email to [e-mail]
> > > > > > ted.org
> > > > > >
> > > > > > Rob is rpc.gssd running with the USE_GSS_PROXY=Yes environment
> > > > > > variable
> > > > > > on ?
> > > > > > If not then gssproxy is simply not involved here
> > > > > >
> > > > > > Simo.
> > > > > >
> > > > > > --
> > > > > > Simo Sorce * Red Hat, Inc * New York
> > > > > > _______________________________________________
> > > > > > gss-proxy mailing list -- [e-mail]
> > > > > > To unsubscribe send an email to [e-mail]
> > > > > > d.org
> > > > >
> > > > > There is a
> > > > > GSS_USE_PROXY=yes in /etc/sysconfig/nfs
> > > > > so I added USE_GSS_PROXY=yes to it and also to the script
> > > > > /usr/libexec/nfs-utils/nfs-utils_env.sh
> > > > > so that it gets applied to the the file
> > > > > /run/sysconfig/nfs-utils at boot.
> > > > > I double checked after a reboot
> > > > > verified the share was working for an ordinary user
> > > > > but not for the apache user when using 'su - apache -s /bin/bash'
> > > > > also checked the kvno of the http.keytab just to make sure I wasn't
> > > > > missing the obvious.
> > > > > still no go.
> > > > >
> > > > > Rob Verduijn
> > > >
> > > > gss-proxy mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > g
> >
> > --
> > Simo Sorce * Red Hat, Inc. * New York
> > _______________________________________________
> > gss-proxy mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]