NFS4 kerberos auth for local services

[Djerk Geurts]

Hi all,

Judging by my online searches, I’m far from the first to ask the question, but I’m keft with holes in my understanding of Kerberos and how services can authenticate via Kerberos (keytab).

I’m switching from sec=sys to sec=krb5p and either way struggle with local services which must place files on an NFS share for backup purposes. Using sec=sys things just work but the uid/gid numbers get matched locally and this often worked fine (when local services used the same aid/gid. But this doesn’t scale well, so I’m looking for ways to deal with this.

One way is to create a user in FreeIPA with the name of the service (for example bhsvc for Nakivo backup), and then adjust the uid on the local server to the IPA issued one, which is quick. But requires finding any file with the old id and changing it to the new one, which can be time consuming.

As the nfs client is a 3CX server, which don’t do well when manually configured as 3CX treat them as appliances. (God forbid someone might want to centrally manage these beast…); I would prefer not to change the uid of the local system account (phonesystem) to an IPA assigned one.

What are my options?

Despite finding how to configure gssproxy, I don’t yet understand how a daemon running as a certain user is mapped to an SPN with related keytab. Creating an SPN in IPA is easy, but how does the nfs-client know that a local system account should use/fetch a keytab for a certain SPN?
I could just manually set the uid of the local user on the nfs server, but while this worked with sec=sys, I don’t think this works with sec=krb5. So an option is to revert to sec=system, but I’d prefer not to.

The gssproxy config I created for the 3cxpbx daemon(s):

user@3cx04:~$ cat /etc/gssproxy/00-3cxpbx.conf
[service/3CXPBX]
mechs = krb5
cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_3cxpbx
cred_store = client_keytab:/var/lib/gssproxy/clients/3cxpbx.keytab
cred_usage = initiate
euid = 998

[Christian Heimes]

On 21/05/2024 12.15, Djerk Geurts via FreeIPA-users wrote:
> Hi all,
>
> Judging by my online searches, I’m far from the first to ask the 
> question, but I’m keft with holes in my understanding of Kerberos and 
> how services can authenticate via Kerberos (keytab).
>
> I’m switching from sec=sys to sec=krb5p and either way struggle with 
> local services which must place files on an NFS share for backup 
> purposes. Using sec=sys things just work but the uid/gid numbers get 
> matched locally and this often worked fine (when local services used 
> the same aid/gid. But this doesn’t scale well, so I’m looking for ways 
> to deal with this.
>
> One way is to create a user in FreeIPA with the name of the service 
> (for example bhsvc for Nakivo backup), and then adjust the uid on the 
> local server to the IPA issued one, which is quick. But requires 
> finding any file with the old id and changing it to the new one, which 
> can be time consuming.
>
> As the nfs client is a 3CX server, which don’t do well when manually 
> configured as 3CX treat them as appliances. (God forbid someone might 
> want to centrally manage these beast…); I would prefer not to change 
> the uid of the local system account (phonesystem) to an IPA assigned one.
>
> What are my options?
> Despite finding how to configure gssproxy, I don’t yet understand
> how a daemon running as a certain user is mapped to an SPN with
> related keytab. Creating an SPN in IPA is easy, but how does the
> nfs-client know that a local system account should use/fetch a
> keytab for a certain SPN?
> I could just manually set the uid of the local user on the nfs
> server, but while this worked with sec=sys, I don’t think this
> works with sec=krb5. So an option is to revert to sec=system, but
> I’d prefer not to.
>
> The gssproxy config I created for the 3cxpbx daemon(s):
>
> user@3cx04:~$ cat /etc/gssproxy/00-3cxpbx.conf
> [service/3CXPBX]
> mechs = krb5
> cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_3cxpbx
> cred_store = client_keytab:/var/lib/gssproxy/clients/3cxpbx.keytab
> cred_usage = initiate
> euid = 998

gss-proxy maps an execution context to a configuration and keytab. In 
your case, it maps the context by effective uid. gss-proxy then uses the 
SPN of the first entry of the keytab to acquire a TGT with that keytab.

By the way, did you set the environment variable "GSS_USE_PROXY=yes" for 
your daemons, too? The easiest way is to use a systemd override with 
e.g. "systemctl edit yourdaemon.service", then add:

[Service]
Environment=GSS_USE_PROXY=yes

You can test the gssproxy configuration by switching to the user (sudo, 
su), then running "GSS_USE_PROXY=yes ipa ping". If everything is working 
correctly, then ipa CLI will automatically acquire a TGT for you. Don't 
try this with kinit, it doesn't use GSS-API.

Christian

[Djerk Geurts]

Thank you, that’s really helpful, especially how to test.

For the 3CX service I do indeed need to add the GSS_USE_PROXY=yes, but as a side note, I’ll need to work out which service needs it as there are many daemons that make up 3CX. Anyway, this is on my todo list.

What I need to do first is create a Service Principal for this service to use. The GSS proxy config references the local uid, but how does it match the SPN? I guess I’m not clear on the service name as the host and REALM parts are straight forward. Is the username used for this? If so the SPN should be phonesystem/FQDN@REALM, as the uid is a number, so can I assume that the local machine uses the local account name belonging to the uid for this?

[Christian Heimes]

On 21/05/2024 13.11, Djerk Geurts wrote:
> Thank you, that’s really helpful, especially how to test.
>
> For the 3CX service I do indeed need to add the GSS_USE_PROXY=yes, but 
> as a side note, I’ll need to work out which service needs it as there 
> are many daemons that make up 3CX. Anyway, this is on my todo list.
>
> What I need to do first is create a Service Principal for this service 
> to use. The GSS proxy config references the local uid, but how does it 
> match the SPN? I guess I’m not clear on the service name as the host 
> and REALM parts are straight forward. Is the username used for this? 
> If so the SPN should be phonesystem/FQDN@REALM, as the uid is a 
> number, so can I assume that the local machine uses the local account 
> name belonging to the uid for this?

GSS-Proxy uses the power of Unix sockets. "GSS_USE_PROXY=yes" enables 
the interposer library in the client. When the interposer intercepts 
some GSS-API calls and forwards them to GSS-Proxy's Unix domain socket. 
The proxy daemon uses getsockopt() with SO_PEERCRED to get the euid and 
egid of the client from the Linux Kernel. In your case, it maps euid 998 
to "service/3CXPBX" and "3cxpbx.keytab". In client keytab mode, 
GSS-Proxy then uses the SPN of the first keytab slot. The keytab 
contains the SPN:

# ktutil
ktutil:  rkt /var/lib/ipa/gssproxy/http.keytab
ktutil:  l
slot KVNO Principal
---- ---- 
---------------------------------------------------------------------
    1    1 HTTP/[e-mail]
    2    1 HTTP/[e-mail]
    3    1 HTTP/[e-mail]
    4    1 HTTP/[e-mail]

[Djerk Geurts]

Great in depth detail, I'm learning loads from you.

So, an I right in deducting that would mean the keytab is manually populated, not generated by gssproxy? Sorry, feeling like a real noob here ...

Thanks, 
Djerk Geurts

On 21 May 2024, 13:25, at 13:25, Christian Heimes  wrote:
> On 21/05/2024 13.11, Djerk Geurts wrote:
> > Thank you, that’s really helpful, especially how to test.
> >
> > For the 3CX service I do indeed need to add the GSS_USE_PROXY=yes,
>
> but
> > as a side note, I’ll need to work out which service needs it as there
> > are many daemons that make up 3CX. Anyway, this is on my todo list.
> >
> > What I need to do first is create a Service Principal for this
>
> service
> > to use. The GSS proxy config references the local uid, but how does
>
> it
> > match the SPN? I guess I’m not clear on the service name as the host 
> > and REALM parts are straight forward. Is the username used for this? 
> > If so the SPN should be phonesystem/FQDN@REALM, as the uid is a 
> > number, so can I assume that the local machine uses the local account
> > name belonging to the uid for this?
>
> GSS-Proxy uses the power of Unix sockets. "GSS_USE_PROXY=yes" enables 
> the interposer library in the client. When the interposer intercepts 
> some GSS-API calls and forwards them to GSS-Proxy's Unix domain socket.
>
> The proxy daemon uses getsockopt() with SO_PEERCRED to get the euid and
>
> egid of the client from the Linux Kernel. In your case, it maps euid
> 998 
> to "service/3CXPBX" and "3cxpbx.keytab". In client keytab mode, 
> GSS-Proxy then uses the SPN of the first keytab slot. The keytab 
> contains the SPN:
>
> # ktutil
> ktutil:  rkt /var/lib/ipa/gssproxy/http.keytab
> ktutil:  l
> slot KVNO Principal
>
> 1    1 HTTP/[e-mail]
>    2    1 HTTP/[e-mail]
>    3    1 HTTP/[e-mail]
>    4    1 HTTP/[e-mail]
>
> -- 
> Christian Heimes
> Principal Software Engineer, Identity Management and Platform Security
>
> Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
> O'Neill

[Djerk Geurts]

Thank you, after creating the keytab with `ipa-getkeytab -p 3CXPBX/3cx04.domain.com -k 3cxpbx.keytab` and moving the keytab to the machine (the Debian host didn’t have ipa-getkeytab) to the right location.

The ipa ping now works:

phonesystem@3cx04:/home/user$ GSS_USE_PROXY=yes ipa ping
--------------------------------------------
IPA server version 4.11.1. API version 2.253
--------------------------------------------

Thank you for your advice!

[Djerk Geurts]

Now that I have gssproxy configured with the keytab for the service and can do an ipa ping, the next thing I need to wrap my head around is how to grant a local service access to NFS shared files. A keytab for a user doesn’t equate to file permissions, so how does the NFS server match the SPN to file permissions?

What’s the right way forward from here?

auth_to_local (presumably on the NFS server, to map the SPN to a ‘local’ account.
However, my thinking then is that if I create a user in IPA to assign file and folder ownership to, then I may as well use the SPN of this user for the service on the client, avoiding the need to map things on the NFS server.
idmapd

Mapping a local daemon (systemd service) to an IPA user SPN just seems to make more sense to me.

— 
Djerk Geurts
> On 21 May 2024, at 17:22, Djerk Geurts  wrote:
>
> Thank you, after creating the keytab with `ipa-getkeytab -p 3CXPBX/3cx04.domain.com -k 3cxpbx.keytab` and moving the keytab to the machine (the Debian host didn’t have ipa-getkeytab) to the right location.
>
> The ipa ping now works:
> phonesystem@3cx04:/home/user$ GSS_USE_PROXY=yes ipa ping
> IPA server version 4.11.1. API version 2.253
>
> Thank you for your advice!
>
> -- 
> Djerk Geurts
> > On 21 May 2024, at 15:17, Djerk Geurts  wrote:
> >
> > Great in depth detail, I'm learning loads from you.
> >
> > So, an I right in deducting that would mean the keytab is manually populated, not generated by gssproxy? Sorry, feeling like a real noob here ...
> >
> > Thanks, 
> > Djerk Geurts 
> > On 21 May 2024, at 13:25, Christian Heimes  wrote: On 21/05/2024 13.11, Djerk Geurts wrote:
> > Thank you, that’s really helpful, especially how to test.
> >
> > For the 3CX service I do indeed need to add the GSS_USE_PROXY=yes, but 
> > as a side note, I’ll need to work out which service needs it as there 
> > are many daemons that make up 3CX. Anyway, this is on my todo list.
> >
> > What I need to do first is create a Service Principal for this service 
> > to use. The GSS proxy config references the local uid, but how does it 
> > match the SPN? I guess I’m not clear on the service name as the host 
> > and REALM parts are straight forward. Is the username used for this? 
> > If so the SPN should be phonesystem/FQDN@REALM, as the uid is a 
> > number, so can I assume that the local machine uses the local account 
> > name belonging to the uid for this?
> > GSS-Proxy uses the power of Unix sockets. "GSS_USE_PROXY=yes" enables 
> > the interposer library in the client. When the interposer intercepts 
> > some GSS-API calls and forwards them to GSS-Proxy's Unix domain socket. 
> > The proxy daemon uses getsockopt() with SO_PEERCRED to get the euid and 
> > egid of the client from the Linux Kernel. In your case, it maps euid 998 
> > to "service/3CXPBX" and "3cxpbx.keytab". In client keytab mode, 
> > GSS-Proxy then uses the SPN of the first keytab slot. The keytab 
> > contains the SPN:
> >
> > # ktutil
> > ktutil:  rkt /var/lib/ipa/gssproxy/http.keytab
> > ktutil:  l
> > slot KVNO Principal
> >
> > 1    1 HTTP/[e-mail]
> >    2    1 HTTP/[e-mail]
> >    3    1 HTTP/[e-mail]
> >    4    1 HTTP/[e-mail]

[Djerk Geurts]

Answering my own final queries for posterity.

Auth_to_local was the missing piece. It maps the SPN to a ‘local’ user, and I found that an NFS server will regard an IPA user as a local user for these mappings. I added the following two auth_to_local lines to the realms section in /etc/krb5.conf (only on the NFS server):

[realms]
  DOMAIN.COM = {
    pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
    pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem

auth_to_local = RULE:[2:$1;$2](^3CXPBX;3cx.*.domain.com$)s/^.*$/backup_3cxpbx/
    auth_to_local = DEFAULT
  }

This maps the SPN “3CXPBX/3cx*[e-mail] ” to (IPA) user backup_3cxpbx, which enables the file permissions to be checked successfully on the server, as without it there’s no way to map the service SPN to a uid or gid.

In summary, the client systemd service username is mapped to a uid and gid on the server as follows:

NFS-client:
Systemd process runs with a local system account (phonesystem/<uid below 1000>)
Gssproxy maps the local uid to an SPN and requests the ticket using a keytab file (created using ipa-getkeytab)
NFS-server
Auth_to_local maps the SPN to a local (or IPA) user and checks file and folder permissions using the uid and gid of the ‘local’ account.

Both client and server report the file and folder permissions as per the values set on the server. So no special idmapping is done or required.

In my testing I did use ‘kinit -kt <keytab> <principal>’ to be able to browse and test the NFS share access from the client.

I guess no auth_to_local mapping is needed when using the SPN of an IPA user for the systemd service, instead of a service SPN, incidentally this user does not need to have a password set when only keytab based auth is used. As always, YMMV, but this worked for me.

Huge thanks to Christian Heimes for his pointers and insight.
> On 22 May 2024, at 10:41, Djerk Geurts  wrote:
>
> Now that I have gssproxy configured with the keytab for the service and can do an ipa ping, the next thing I need to wrap my head around is how to grant a local service access to NFS shared files. A keytab for a user doesn’t equate to file permissions, so how does the NFS server match the SPN to file permissions?
>
> What’s the right way forward from here?
>
> auth_to_local (presumably on the NFS server, to map the SPN to a ‘local’ account.
> However, my thinking then is that if I create a user in IPA to assign file and folder ownership to, then I may as well use the SPN of this user for the service on the client, avoiding the need to map things on the NFS server.
> idmapd
>
> Mapping a local daemon (systemd service) to an IPA user SPN just seems to make more sense to me.
>
> — 
> Djerk Geurts
> > On 21 May 2024, at 17:22, Djerk Geurts  wrote:
> >
> > Thank you, after creating the keytab with `ipa-getkeytab -p 3CXPBX/3cx04.domain.com -k 3cxpbx.keytab` and moving the keytab to the machine (the Debian host didn’t have ipa-getkeytab) to the right location.
> >
> > The ipa ping now works:
> > phonesystem@3cx04:/home/user$ GSS_USE_PROXY=yes ipa ping
> > IPA server version 4.11.1. API version 2.253
> >
> > Thank you for your advice!
> >
> > -- 
> > Djerk Geurts
> > > On 21 May 2024, at 15:17, Djerk Geurts  wrote:
> > >
> > > Great in depth detail, I'm learning loads from you.
> > >
> > > So, an I right in deducting that would mean the keytab is manually populated, not generated by gssproxy? Sorry, feeling like a real noob here ...
> > >
> > > Thanks, 
> > > Djerk Geurts 
> > > On 21 May 2024, at 13:25, Christian Heimes  wrote: On 21/05/2024 13.11, Djerk Geurts wrote:
> > > Thank you, that’s really helpful, especially how to test.
> > >
> > > For the 3CX service I do indeed need to add the GSS_USE_PROXY=yes, but 
> > > as a side note, I’ll need to work out which service needs it as there 
> > > are many daemons that make up 3CX. Anyway, this is on my todo list.
> > >
> > > What I need to do first is create a Service Principal for this service 
> > > to use. The GSS proxy config references the local uid, but how does it 
> > > match the SPN? I guess I’m not clear on the service name as the host 
> > > and REALM parts are straight forward. Is the username used for this? 
> > > If so the SPN should be phonesystem/FQDN@REALM, as the uid is a 
> > > number, so can I assume that the local machine uses the local account 
> > > name belonging to the uid for this?
> > > GSS-Proxy uses the power of Unix sockets. "GSS_USE_PROXY=yes" enables 
> > > the interposer library in the client. When the interposer intercepts 
> > > some GSS-API calls and forwards them to GSS-Proxy's Unix domain socket. 
> > > The proxy daemon uses getsockopt() with SO_PEERCRED to get the euid and 
> > > egid of the client from the Linux Kernel. In your case, it maps euid 998 
> > > to "service/3CXPBX" and "3cxpbx.keytab". In client keytab mode, 
> > > GSS-Proxy then uses the SPN of the first keytab slot. The keytab 
> > > contains the SPN:
> > >
> > > # ktutil
> > > ktutil:  rkt /var/lib/ipa/gssproxy/http.keytab
> > > ktutil:  l
> > > slot KVNO Principal
> > >
> > > 1    1 HTTP/[e-mail]
> > >    2    1 HTTP/[e-mail]
> > >    3    1 HTTP/[e-mail]
> > >    4    1 HTTP/[e-mail]