Hi again,
I've tried quite a lot of things (including the very latest version 0.7.0 of gssproxy) but I keep failing to get it to work on fedora25.
So in a last ditch attempt I created a reproducer in the form of a vagrant script that creates a test environment that has the problem.
I've tested the vagrant script on fedora25 and using kvm/libvirt with all the latest patches applied. vagrant versions : vagrant-libvirt-doc-0.0.35-3.fc25.noarch vagrant-1.8.5-2.fc25.noarch vagrant-libvirt-0.0.35-3.fc25.noarch <- this should be 0.0.35-3 or later
vagrant-libvirt must be up2date with the latest build or else it will fail to configure the network properly. ( https://bugzilla.redhat.com/show_bug.cgi?id=1426565)
all passwords are 'centos73'
dns/ip info ipa.example.test 192.168.122.2/24 nfs.example.test 192.168.122.3/24 fedoraclient.example.test 192.168.122.4/24
I hope you are willing to give this a try cause I'm kinda out of options here.
there should be a plain text attachment ;Vagrantfile' to this email, if not pm me and I will send it to you.
Rob Verduijn
I doubt that you are new to vagrant, butl some instructions anyway:
if you don't have the kvm/libvirt setup yet dnf install -y @virtualization && systemctl enable libvirtd.service && systemctl start libvirtd.service
install vagrant : dnf install -y vagrant
create a working dir and change to it mkdir $HOME/vagrant cd $HOME/vagrant
put the file 'Vagrantfile' in the directory
start the vagrant script vagrant up
this command will take a (very) long time to finish.
Here is what it does (in parallel where possible)
vagrant will download the vagrant box images for fedora25 and centos73
create a virtual nat network 'vagrant_libvirt' 192.168.121.0/24 in which the dns and gateway are 192.168.121.1
all systems are dual-homed because vagrant requires a control/management interface that resides in the vagrant_libvirt network
the environment will use the virtual nat network 'default' the default subnet for that network is 192.168.122.0/24
install and configure ipa on centos73 add hosts, users and automount maps and finally configure the firewall
configure a nfs server on centos73 and when it notices the ipa server is available: join the ipa domain create a principal for the kerberised nfs import the principal to the local keytab
configure a fedora25 ipa client join the ipa domain configure automounts install apache create a principal for apache configure gssproxy restart services where needed
during the installation it is very likely the ipa server will starve for entrophy, usually a lot of messages saying 'waiting for ipa configuration to finish' are a good hint more entrophy is needed.
scroll up to see if there is a entrophy message somewhere.
and if you see such a message you can either wait a very long time for the entrophy to appear (really, it will appear)
or log in to the console of the ipa server via the virt-manager ( as not to interrupt the script unless you like to start over, see below how) and generate some entrophy in the usual way (typing/drive access/etc/etc)
When the scipt is finished:
(The 'vagrant ssh' commands won't work untill the 'vagrant up' command is finished)
use the following command to log in to the fedoraclient: vagrant ssh fedoraclient then use sudo to become root (no password on sudo)
the nfs sercer vagrant ssh nfsserver
the ipa server vagrant ssh ipaserver
in case you interupted the 'vagrant up' reset everything by using the destroy command.
Or if you simply want to get rid of the vagrant machines. vagrant destroy
the destroy command will not delete the box images, you will need to remove those manually if you want those gone. they can be found in /var/lib/libvirt/images (which also contains any other vm images you might have created)
2017-01-29 17:31 GMT+01:00 Rob Verduijn rob.verduijn@gmail.com:
Hello,
Since you mentioned something about nfs-utils and changes I thought I try the latest nfs-utils from testing as well. (whoa....major version upgrade coming up....lets see if I can find a changelog)
Anyway , still no go. Normal users work fine though.
Rob Verduijn
2017-01-29 17:03 GMT+01:00 Rob Verduijn rob.verduijn@gmail.com:
I did another default install with centos7 applied all the patches, and made it a domain member, configured the automount (as above)
configured it the same way I described above with the minor difference that I used only 1 configuration file for gssproxy: cat /etc/gssproxy/gssproxy.conf [gssproxy] debug = true debug_level = 9
[service/HTTP] mechs = krb5 cred_store = client_keytab:/etc/gssproxy/http.keytab cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U euid = 48
[service/nfs-server] mechs = krb5 socket = /run/gssproxy.sock cred_store = keytab:/etc/krb5.keytab trusted = yes kernel_nfsd = yes euid = 0
[service/nfs-client] mechs = krb5 cred_store = keytab:/etc/krb5.keytab cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab cred_usage = initiate allow_any_uid = yes trusted = yes euid = 0
In the service/HTTP instance I changed keytab to client_keytab. and tested it and it worked.
so I did an strace on rpc.gssd on centos 7.3 And got this : epoll_wait(4, {{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1 read(9, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45 clone(child_stack=0x7fd125bfdeb0, flags=CLONE_VM|CLONE_FS|CLONE_ FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS| CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fd125bfe9d0, tls=0x7fd125bfe700, child_tidptr=0x7fd125bfe9d0) = 2531 epoll_wait(4,
on the fedora25-kde the strace looks like this : epoll_wait(4, [{EPOLLIN, {u32=12, u64=12}}], 32, -1) = 1 read(12, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45 clone(child_stack=0x7f75c4aa3ef0, flags=CLONE_VM|CLONE_FS|CLONE_ FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS| CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f75c4aa49d0, tls=0x7f75c4aa4700, child_tidptr=0x7f75c4aa49d0) = 16294 epoll_wait(4,
This is al more or less giberish to me, but I think the important bit is 'mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45' and that is the same on both systems.
Also the environment of the rpc.gssd proc entry
the output from /proc/<pid>/environ on centos7.3: LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binRPCNFSDARGS= 8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=GSSDARGS =SVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes
The output from /proc/<pid>/environ on fedora25-kde: LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/ sbin:/usr/binJOURNAL_STREAM=8:22877RPCNFSDARGS= 8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=RPCGSSDA RGS=RPCSVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes
Rob Verduijn
2017-01-29 15:33 GMT+01:00 Rob Verduijn rob.verduijn@gmail.com:
Hi,
I guess I'm missing something very basic here.
Just to make sure I'm not missing something very elementary here, could you review the following ?
So all my exports work fine when using ordinary user accounts on fedora clients (kde-spin) So I doubt that there is something wrong on the nfs server side or the ipa bit. I installed the default fedora25-kde spin with additional groups : @domain-client no other packages where added.
A host and dns entry were created in freeipa An otp password is set
The command 'ipa-client install --enable-dns-updates -U -w ********' has been applied The command 'ipa-client-automount' has been issued.
a http principal has been created for this host via the freeipa gui imported it to the client with ipa-getkeytab -p HTTP/fedora25-kde.example.com@EXAMPLE.COM -k /var/lib/gssproxy/clients/httpd.keytab
created the 00-apache.conf cat > /etc/gssproxy/00-apache.conf << EOF [service/apache] mechs = krb5 cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab cred_usage = initiate euid = 48 EOF
added debug settings to gssproxy.conf echo 'debug = true' >> /etc/gssproxy/gssproxy.conf echo 'debug_level = 9' >> /etc/gssproxy/gssproxy.conf
nfsidmap resolving works better for me when the method is set to sss. the command 'sed -i 's/#Method.*/Method=sss/' /etc/idmapd.conf' is applied
updated everything : dnf update -y
added the latest gssproxy from testing dnf update -y gssproxy --enablerepo=updates-testing
add httpd dnf install -y httpd
reboot
the automount map from ipa that I am using : ipa automountkey-show Location: default Map: auto.direct Key: /network/music Key: /network/music Mount information: -rw,noatime,timeo=14,soft,sec=krb5p,acl nfs.example.com:/music
login to the system as root 'ls /network/music' gives a directory listing
issue 'su - apache -s /bin/bash' now 'ls /network/music' gives 'ls: cannot open directory '/network/music': Permission denied'
I logout ctrl-d , ctrl-d
I login as an ipa domain user. (not admin) 'ls /network/music' gives a directory listing.
the root user does not exist in the domain the apache user does not exits in the domain
the apache user has local id 48 on the nfs server and local id 48 on the nfs client. the apache user has group apache with id 48 on the nfs client and nfs server.
2017-01-29 14:10 GMT+01:00 Simo Sorce ssorce@redhat.com:
So it turned out I was not reproducing the issue, I had an older version of gssproxy than I thought, and incorrect logging level in the config file.
Once I had time to sit down and fix my client by installing the latest updates-testing package and making sure the debug level was properly set all worked as expected for me on Fedora 25.
I was suspecting some issues with a relatively recent patchset that went into nfs-utils which switched from using fork() to use threads, but it turned out this situation is already properly handled in the gss plugin used to talk to gssproxy. (CCing Jeff who reviewed those patches so he knows it is all good for now :-)
Rob, in order to find out what is going on you may want to try to strace rpc.gssd on the client system and try to figure out why gssproxy is not being used, also check if /proc/<pid>/env of rpc.gssd does in fact include the proper environment variable.
Simo.
----- Original Message -----
From: "Simo Sorce" simo@redhat.com To: "The GSS-Proxy developers and users mailing list" <
gss-proxy@lists.fedorahosted.org>
Sent: Saturday, January 28, 2017 7:05:41 AM Subject: Re: [gssproxy] Re: gssproxy broken on fedora
Hi Rob, I can reproduce your issue now and I am looking into this issue and I suspect nfs-utils may have broken gss-proxy support recently.
I am not sure yet, but some critical areas have been changed, so I
need
to better investigate them. It will take me a little time as I am at Devconf.cz now, but hopefully I will have an answer for you in a few days.
Simo.
On Sat, 2017-01-21 at 13:10 +0100, Rob Verduijn wrote:
Another fresh install of a fedora25 client, now with the new
gssproxy
0.6.0 package, but that one does not work either.
Created the file /etc/gssproxy/00-apache.conf [service/apache] mechs = krb5 cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab cred_usage = initiate euid = 48
put the keytab in the specified place added debug settings to /etc/gssproxy/gssproxy.conf [gssproxy] debug = true debug_level = 9
I did not touch the file /etc/sysconfig/nfs Since the default setting of GSS_USE_PROXY="yes" is what I want
Checked the logs, nothing in there besides what you get from systemctl status gssproxy.service
su - apache -s /bin/bash and no access to the shares is allowed
Weird thing I noticed, when browsing as root the nfs mounts are readable ( root is squashed ) but not as the apache user. Root gets to read it with nobody:nobody privileges but apache with apache:apache is refused.
I'm really at a loss as to what to do next.
Do I need to set an environment variable to make this work ?
Rob Verduijn
2017-01-04 23:14 GMT+01:00 Rob Verduijn rob.verduijn@gmail.com: > > 2017-01-04 20:56 GMT+01:00 Simo Sorce simo@redhat.com: > > On Wed, 2017-01-04 at 19:41 +0100, Rob Verduijn wrote: > > > 2017-01-04 19:27 GMT+01:00 Dmitri Pal dpal@redhat.com: > > > > > > > On 01/04/2017 01:13 PM, Rob Verduijn wrote: > > > > > > > > > > > > > > > > 2017-01-04 14:59 GMT+01:00 Simo Sorce simo@redhat.com: > > > > > > > >> On Wed, 2017-01-04 at 10:16 +0100, Rob Verduijn wrote: > > > >> > ---------- Forwarded message ---------- > > > >> > From: Simo Sorce simo@redhat.com > > > >> > Date: 2017-01-03 17:32 GMT+01:00 > > > >> > Subject: [gssproxy] Re: gssproxy broken on fedora > > > >> > To: The GSS-Proxy developers and users mailing list < > > > >> > gss-proxy@lists.fedorahosted.org> > > > >> > > > > >> > > > > >> > On Mon, 2017-01-02 at 19:22 +0100, Rob Verduijn wrote: > > > >> > > > > > >> > > Nope that does not work on either fc24 or fc25. > > > >> > > I did not try centos73 since it already worked on that > > one. > > > >> > > > > >> > Given you tried manually, make sure you delete the ccache > > before trying > > > >> > with the client_keytab setting. > > > >> > > > > >> > If that doesn't work can you set debug = True in the > > global section and > > > >> > tell me if you get any useful output/error ? > > > >> > > > > >> > Simo. > > > >> > > > > >> > -- > > > >> > Simo Sorce * Red Hat, Inc * New York > > > >> > _______________________________________________ > > > >> > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.o
rg
> > > >> > To unsubscribe send an email to
gss-proxy-leave@lists.fedo
> > rahosted.org > > > >> > > > > >> > > > > >> > Hi, > > > >> > > > > >> > I checked for the cache, but there were no cache files > > present in > > > >> > /var/lib/gssproxy/clients. > > > >> > I cleaned the sssd cache. > > > >> > I set the debug entry, did a reboot, but also no log > > entries appeared > > > >> > > > > >> > current /etc/gssproxy/gssproxy.conf > > > >> > > > > >> > [gssproxy] > > > >> > debug=True > > > >> > > > > >> > [service/HTTP] > > > >> > mechs = krb5 > > > >> > cred_store = keytab:/etc/gssproxy/http.keytab > > > >> > cred_store = ccache:/var/lib/gssproxy/clien
ts/krb5cc_%U
> > > >> > cred_store = client_keytab:/etc/gssproxy/http.keytab > > > >> > euid = 48 > > > >> > > > > >> > and tested it with > > > >> > su - apache -s /bin/bash > > > >> > > > > >> > The mount works fine for a regular ipa user on fedora > > 24/25 > > > >> > according to systemctl status gssproxy the service is up > > and running, > > > >> > > > > >> > [root@fedora-24 ~]# systemctl status gssproxy > > > >> > ● gssproxy.service - GSSAPI Proxy Daemon > > > >> > Loaded: loaded > > (/usr/lib/systemd/system/gssproxy.service; disabled; > > > >> > vendor preset: disabled) > > > >> > Active: active (running) since Wed 2017-01-04 10:05:55 > > CET; 8min ago > > > >> > Main PID: 987 (gssproxy) > > > >> > CGroup: /system.slice/gssproxy.service > > > >> > └─987 /usr/sbin/gssproxy -D > > > >> > > > > >> > systemd[1]: Starting GSSAPI Proxy Daemon... > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Debug Enabled > > (level: 1) > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Client connected
(fd
> > = > > > >> 10)[2017/01/04 > > > >> > 09:05:55]: (pid = 987) (uid = 0) (gid = 0)[2017/01/04 > > 09:05:55]: > > > >> (context > > > >> > = system_u:system_r:kernel_t:s0)[2017/01/04 09:05:55]: > > > >> > Started GSSAPI Proxy Daemon. > > > >> > > > >> If you turn on rpc.gssd debugging and kernel rpc debugging > > do you see > > > >> anything relevant ? > > > >> > > > >> Simo. > > > >> > > > >> -- > > > >> Simo Sorce * Red Hat, Inc * New York > > > >> _______________________________________________ > > > >> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > >> To unsubscribe send an email to
gss-proxy-leave@lists.fedora
> > hosted.org > > > >> > > > > > > > > > > > > It does not seem to look for the credits specified in the > > gssproxy.conf > > > > file. > > > > How can I verify the running configuration of gssproxy ? > > > > > > > > Rob Verduijn > > > > > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: > > #012handle_gssd_upcall: > > > > 'mech=krb5 uid=48 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt0) > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: > > krb5_not_machine_creds: uid 48 > > > > tgtname (null) > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: ERROR: GSS-API: > > error in > > > > gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure. > > Minor code may > > > > provide more information) - No Kerberos credentials
available
> > (default > > > > cache: KEYRING:persistent:48) > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client > > creds with > > > > uid 48 for server nfs.example.com in /tmp > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: CC > > '/tmp/krb5ccmachine_EXAMPLE.COM' > > > > being considered, with preferred realm 'EXAMPLE.COM' > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: CC > > '/tmp/krb5ccmachine_EXAMPLE.COM' > > > > owned by 0, not 48 > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client > > creds with > > > > uid 48 for server nfs.example.com in /run/user/%U > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: Error doing
scandir
> > on directory > > > > '/run/user/48': No such file or directory > > > > Jan 4 18:52:50 fedora-24 rpc.gssd[1034]: doing error > > downcall > > > > > > > > > > > > > > > > _______________________________________________ > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > To unsubscribe send an email to
gss-proxy-leave@lists.fedorah
> > osted.org > > > > > > > > > > > > Why are you preferring credential cache in a file over a > > keyring which is > > > > default? > > > > Have you tried without cred_store = > > ccache:/var/lib/gssproxy/clients/krb5cc_%U > > > > ? > > > > > > > > -- > > > > Thank you, > > > > Dmitri Pal > > > > > > > > Engineering Director, Identity Management and Platform > > Security > > > > Red Hat, Inc. > > > > > > > > > > > > _______________________________________________ > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > > To unsubscribe send an email to
gss-proxy-leave@lists.fedorah
> > osted.org > > > > > > > > > > > Because it said so in the example here : > > > https://fedorahosted.org/gss-proxy/wiki/Apache > > > > > > But have tried it without and it still fails. > > > > > > Rob Verduijn > > > _______________________________________________ > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > > To unsubscribe send an email to gss-proxy-leave@lists.fedoraho
s
> > ted.org > > > > Rob is rpc.gssd running with the USE_GSS_PROXY=Yes environment > > variable > > on ? > > If not then gssproxy is simply not involved here > > > > Simo. > > > > -- > > Simo Sorce * Red Hat, Inc * New York > > _______________________________________________ > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org > > To unsubscribe send an email to gss-proxy-leave@lists.fedoraho
ste
> > d.org > > > > > There is a > GSS_USE_PROXY=yes in /etc/sysconfig/nfs > so I added USE_GSS_PROXY=yes to it and also to the script > /usr/libexec/nfs-utils/nfs-utils_env.sh > so that it gets applied to the the file > /run/sysconfig/nfs-utils at boot. > I double checked after a reboot > verified the share was working for an ordinary user > but not for the apache user when using 'su - apache -s /bin/bash' > also checked the kvno of the http.keytab just to make sure I
wasn't
> missing the obvious. > still no go. > > Rob Verduijn > >
gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedoraho
sted.or
g
-- Simo Sorce * Red Hat, Inc. * New York _______________________________________________ gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org