pam_krb5 & wrong principal in request

[Ondrej Valousek]

Hello,

I am running sssd on RHEL-6 with AD and recently found out, that I can no longer authenticate via ssh.
The message is:
pam_krb5[27045]: TGT failed verification using keytab and key for 'nfs/[e-mail]': Wrong principal in 
request

My krb5.keytab file has been created using Samba with "net ads join".
I can still run 'kinit <username>' to obtain a TGT.

Does anyone know what is going on here? Why is sshd verifying the TGT via just the nfs/ service principal?

Sorry for a bit off-topic question, but I hoped someone in this list might point me in the right direction.

Ondrej

[Ondrej Valousek]

On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> Hello,
>
> I am running sssd on RHEL-6 with AD and recently found out, that I can no longer authenticate via ssh.
> The message is:
> pam_krb5[27045]: TGT failed verification using keytab and key for 'nfs/[e-mail]': Wrong principal in 
> request
>
> My krb5.keytab file has been created using Samba with "net ads join".
> I can still run 'kinit <username>' to obtain a TGT.
>
> Does anyone know what is going on here? Why is sshd verifying the TGT via just the nfs/ service principal?
>
> Sorry for a bit off-topic question, but I hoped someone in this list might point me in the right direction.
>
> Ondrej

Note that with:
kinit -S nfs/[e-mail] ondrejv

I am able to successfully obtain the nfs/ principal - so it looks like a bug in pam_krb5, right?

Many thanks,
Ondrej

[Jan Zelený]

> On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> > Hello,
> >
> > I am running sssd on RHEL-6 with AD and recently found out, that I can no
> > longer authenticate via ssh. The message is:
> > pam_krb5[27045]: TGT failed verification using keytab and key for
> > 'nfs/[e-mail]': Wrong principal in
> > request
> >
> > My krb5.keytab file has been created using Samba with "net ads join".
> > I can still run 'kinit <username>' to obtain a TGT.
> >
> > Does anyone know what is going on here? Why is sshd verifying the TGT via
> > just the nfs/ service principal?
> >
> > Sorry for a bit off-topic question, but I hoped someone in this list
> > might point me in the right direction.
> >
> > Ondrej
>
> Note that with:
> kinit -S nfs/[e-mail] ondrejv
>
> I am able to successfully obtain the nfs/ principal - so it looks like a
> bug in pam_krb5, right?

You should also try kinit -k to see if kinit with the keytab is ok.

Thanks
Jan

[Ondrej Valousek]

On 03/01/2012 02:04 PM, Jan Zelený wrote:
> > On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> > > Hello,
> > >
> > > I am running sssd on RHEL-6 with AD and recently found out, that I can no
> > > longer authenticate via ssh. The message is:
> > > pam_krb5[27045]: TGT failed verification using keytab and key for
> > > 'nfs/[e-mail]': Wrong principal in
> > > request
> > >
> > > My krb5.keytab file has been created using Samba with "net ads join".
> > > I can still run 'kinit<username>' to obtain a TGT.
> > >
> > > Does anyone know what is going on here? Why is sshd verifying the TGT via
> > > just the nfs/ service principal?
> > >
> > > Sorry for a bit off-topic question, but I hoped someone in this list
> > > might point me in the right direction.
> > >
> > > Ondrej
> >
> > Note that with:
> > kinit -S nfs/[e-mail] ondrejv
> >
> > I am able to successfully obtain the nfs/ principal - so it looks like a
> > bug in pam_krb5, right?
>
> You should also try kinit -k to see if kinit with the keytab is ok.
>
> Thanks
> Jan

kinit -k -S nfs/[e-mail] draco$
works fine, too.

So I am probably going to file a bugreport, right?

Ondrej

[Jan Zelený]

> On 03/01/2012 02:04 PM, Jan Zelený wrote:
> > > On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> > > > Hello,
> > > >
> > > > I am running sssd on RHEL-6 with AD and recently found out, that I can
> > > > no longer authenticate via ssh. The message is:
> > > > pam_krb5[27045]: TGT failed verification using keytab and key for
> > > > 'nfs/[e-mail]': Wrong principal
> > > > in request
> > > >
> > > > My krb5.keytab file has been created using Samba with "net ads join".
> > > > I can still run 'kinit<username>' to obtain a TGT.
> > > >
> > > > Does anyone know what is going on here? Why is sshd verifying the TGT
> > > > via just the nfs/ service principal?
> > > >
> > > > Sorry for a bit off-topic question, but I hoped someone in this list
> > > > might point me in the right direction.
> > > >
> > > > Ondrej
> > >
> > > Note that with:
> > > kinit -S nfs/[e-mail] ondrejv
> > >
> > > I am able to successfully obtain the nfs/ principal - so it looks like a
> > > bug in pam_krb5, right?
> >
> > You should also try kinit -k to see if kinit with the keytab is ok.
> >
> > Thanks
> > Jan
>
> kinit -k -S nfs/[e-mail] draco$
> works fine, too.
>
> So I am probably going to file a bugreport, right?

Just to be sure, try klist -k and send me the result.

Thanks
Jan

[Ondrej Valousek]

On 03/01/2012 02:24 PM, Jan Zelený wrote:
> > On 03/01/2012 02:04 PM, Jan Zelený wrote:
> > > > On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> > > > > Hello,
> > > > >
> > > > > I am running sssd on RHEL-6 with AD and recently found out, that I can
> > > > > no longer authenticate via ssh. The message is:
> > > > > pam_krb5[27045]: TGT failed verification using keytab and key for
> > > > > 'nfs/[e-mail]': Wrong principal
> > > > > in request
> > > > >
> > > > > My krb5.keytab file has been created using Samba with "net ads join".
> > > > > I can still run 'kinit<username>' to obtain a TGT.
> > > > >
> > > > > Does anyone know what is going on here? Why is sshd verifying the TGT
> > > > > via just the nfs/ service principal?
> > > > >
> > > > > Sorry for a bit off-topic question, but I hoped someone in this list
> > > > > might point me in the right direction.
> > > > >
> > > > > Ondrej
> > > >
> > > > Note that with:
> > > > kinit -S nfs/[e-mail] ondrejv
> > > >
> > > > I am able to successfully obtain the nfs/ principal - so it looks like a
> > > > bug in pam_krb5, right?
> > >
> > > You should also try kinit -k to see if kinit with the keytab is ok.
> > >
> > > Thanks
> > > Jan
> >
> > kinit -k -S nfs/[e-mail] draco$
> > works fine, too.
> >
> > So I am probably going to file a bugreport, right?
>
> Just to be sure, try klist -k and send me the result.
>
> Thanks
> Jan

Just a plain kinit -k does not work here:
[root@draco ~]# kinit -k
kinit: Client 'host/[e-mail]' not found in Kerberos database while getting initial credentials

... but this is pretty much normal as I am using AD based KDC.
This works fine though:

[root@draco ~]# kinit -k draco$

Another strange thing is that sshd complains when I attempt to log in via GSSAPI & existing Kerberos TGT. It says:

debug1: Unspecified GSS failure.  Minor code may provide more information
No key table entry found for host/[e-mail]

But again,

kinit -S host/[e-mail] ondrejv

Works just fine (just need to type my password).

Ondrej

[Ondrej Valousek]

Got it finally working by re-creating /etc/krb5.keytab file from the scratch.
It turned out, that I did not use Samba to join the machine to AD, but I used Centrify.

There is nothing wrong with that, the only thing is, that unlike Samba & SSSD, Centrify does refresh principals in /etc/krb5.keytab creating 
multiple principals in keytab that differ only by the KVNO number.

This is nice, but both sshd & pam_krb5.so seem to totally ignore KVNOs, picking the first principal they find in keytab.

So I am going to file RH bugzilla report

Ondrej

[Stephen Gallagher]

On Thu, 2012-03-01 at 15:23 +0100, Ondrej Valousek wrote:
> There is nothing wrong with that, the only thing is, that unlike Samba
> & SSSD, Centrify does refresh principals in /etc/krb5.keytab creating
> multiple principals in keytab that differ only by the KVNO number.

FYI: https://fedorahosted.org/sssd/ticket/1041 is tracking this
(scheduled for SSSD 1.11 in 9 months).

[John Hodrien]

On Thu, 1 Mar 2012, Ondrej Valousek wrote:
> Got it finally working by re-creating /etc/krb5.keytab file from the scratch.
> It turned out, that I did not use Samba to join the machine to AD, but I used Centrify.
>
> There is nothing wrong with that, the only thing is, that unlike Samba &
> SSSD, Centrify does refresh principals in /etc/krb5.keytab creating multiple
> principals in keytab that differ only by the KVNO number.
>
> This is nice, but both sshd & pam_krb5.so seem to totally ignore KVNOs,
> picking the first principal they find in keytab.

Why would you want to collect the old credentials?  Aren't the credentials
with the lower KVNOs obsolete?
> So I am going to file RH bugzilla report

Mmm, sounds like it might be worthwhile.

jh

[Ondrej Valousek]

On 03/01/2012 03:38 PM, John Hodrien wrote:
> On Thu, 1 Mar 2012, Ondrej Valousek wrote:
> > Got it finally working by re-creating /etc/krb5.keytab file from the scratch.
> > It turned out, that I did not use Samba to join the machine to AD, but I used Centrify.
> >
> > There is nothing wrong with that, the only thing is, that unlike Samba &
> > SSSD, Centrify does refresh principals in /etc/krb5.keytab creating multiple
> > principals in keytab that differ only by the KVNO number.
> >
> > This is nice, but both sshd & pam_krb5.so seem to totally ignore KVNOs,
> > picking the first principal they find in keytab.
>
> Why would you want to collect the old credentials?  Aren't the credentials
> with the lower KVNOs obsolete?

I am asking this question myself, too :-) . But there must be some reason behind it, otherwise we would not need KVNOs at all.
> FYI:https://fedorahosted.org/sssd/ticket/1041  is tracking this
> (scheduled for SSSD 1.11 in 9 months).

Nice, hopefully pam_krb5 would be fixed before then!

Ondrej

[Simo Sorce]

On Thu, 2012-03-01 at 14:38 +0000, John Hodrien wrote:
> On Thu, 1 Mar 2012, Ondrej Valousek wrote:
> > Got it finally working by re-creating /etc/krb5.keytab file from the scratch.
> > It turned out, that I did not use Samba to join the machine to AD, but I used Centrify.
> >
> > There is nothing wrong with that, the only thing is, that unlike Samba &
> > SSSD, Centrify does refresh principals in /etc/krb5.keytab creating multiple
> > principals in keytab that differ only by the KVNO number.
> >
> > This is nice, but both sshd & pam_krb5.so seem to totally ignore KVNOs,
> > picking the first principal they find in keytab.
>
> Why would you want to collect the old credentials?  Aren't the credentials
> with the lower KVNOs obsolete?

No, they are not until all clients have obtained new tickets.

If you immediately remove a key upon refresh, all clients that have
acquire a ticket against the old key and have it still valid (validity
depends on local KDC configuration but may be 24h or even days) they
will just fail to connect to a service if said service does not have
access to the previous key.

The problem is compounded by the fact you have no way to tell the
clients that you got new keys, so client simply error out. The user is
required to do destroy his credentials and obtain new ones and then go
fetch new tickets for the service.

Old kvnos can be rotated away once the Realm specific limits on
credential expiration are reached so that it is technically impossible
to have around valid tickets obtained against the old key.
> > So I am going to file RH bugzilla report
>
> Mmm, sounds like it might be worthwhile.

Yes those tools should support cases where a keytab has keys with
different kvnos and use the highest one they can find. So a bugzilla is
appropriate here.

simo.

[Ondrej Valousek]

> No, they are not until all clients have obtained new tickets.
>
> If you immediately remove a key upon refresh, all clients that have
> acquire a ticket against the old key and have it still valid (validity
> depends on local KDC configuration but may be 24h or even days) they
> will just fail to connect to a service if said service does not have
> access to the previous key.
>
> The problem is compounded by the fact you have no way to tell the
> clients that you got new keys, so client simply error out. The user is
> required to do destroy his credentials and obtain new ones and then go
> fetch new tickets for the service.
>
> Old kvnos can be rotated away once the Realm specific limits on
> credential expiration are reached so that it is technically impossible
> to have around valid tickets obtained against the old key.

Thanks for the explanation, Simo.
Filed RH bugzilla report:
https://bugzilla.redhat.com/show_bug.cgi?id=799007

Ondrej

[Jan Zelený]

> On 03/01/2012 02:24 PM, Jan Zelený wrote:
> > > On 03/01/2012 02:04 PM, Jan Zelený wrote:
> > > > > On 03/01/2012 12:21 PM, Ondrej Valousek wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I am running sssd on RHEL-6 with AD and recently found out, that I
> > > > > > can no longer authenticate via ssh. The message is:
> > > > > > pam_krb5[27045]: TGT failed verification using keytab and key for
> > > > > > 'nfs/[e-mail]': Wrong principal
> > > > > > in request
> > > > > >
> > > > > > My krb5.keytab file has been created using Samba with "net ads join".
> > > > > > I can still run 'kinit<username>' to obtain a TGT.
> > > > > >
> > > > > > Does anyone know what is going on here? Why is sshd verifying the TGT
> > > > > > via just the nfs/ service principal?
> > > > > >
> > > > > > Sorry for a bit off-topic question, but I hoped someone in this list
> > > > > > might point me in the right direction.
> > > > > >
> > > > > > Ondrej
> > > > >
> > > > > Note that with:
> > > > > kinit -S nfs/[e-mail] ondrejv
> > > > >
> > > > > I am able to successfully obtain the nfs/ principal - so it looks like
> > > > > a bug in pam_krb5, right?
> > > >
> > > > You should also try kinit -k to see if kinit with the keytab is ok.
> > > >
> > > > Thanks
> > > > Jan
> > >
> > > kinit -k -S nfs/[e-mail] draco$
> > > works fine, too.
> > >
> > > So I am probably going to file a bugreport, right?
> >
> > Just to be sure, try klist -k and send me the result.
> >
> > Thanks
> > Jan
>
> Just a plain kinit -k does not work here:
> [root@draco ~]# kinit -k
> kinit: Client 'host/[e-mail]' not
> found in Kerberos database while getting initial credentials
>
> ... but this is pretty much normal as I am using AD based KDC.
> This works fine though:
>
> [root@draco ~]# kinit -k draco$
>
> Another strange thing is that sshd complains when I attempt to log in via
> GSSAPI & existing Kerberos TGT. It says:
>
> debug1: Unspecified GSS failure.  Minor code may provide more information
> No key table entry found for
> host/[e-mail]
>
> But again,
>
> kinit -S host/[e-mail] ondrejv
>
> Works just fine (just need to type my password).

Since you already worked out the issue, just for the record: I originally 
wanted results of klist -k to see what principals you had in your keytab. I 
anticipated issue similar to what you eventually found.

Jan