The dreaded: 4 (System error) with SSH

[Omen Wild]

A small group of us have been trying to get our Ubuntu hosts fully
integrated into AD using sssd. We have slowly chipped away at the
issues. We believe we are left with one major issue, when we try to
login with SSH we get 4: (System error).

The host is Ubuntu 16.04.1, up to date as of today, so sssd
1.13.4-1ubuntu1. All PAM files are the defaults.

We used the `realm` command to join AD:
realm -v join tou.t3.ucdavis.edu -U [e-mail]

Our AD is set up with TOU.T3.UCDAVIS.EDU as a child domain in the same
forest as the parent domain, T3.UCDAVIS.EDU, with users in
T3.UCDAVIS.EDU and computers and groups in TOU.T3.UCDAVIS.EDU.

All sssd logs (debug_level = 9) and config files are here:

https://descolada.ucdavis.edu/415bfd2c-b0fa-11e6-97b8-3417ebb1df52/

The timing that generated those log files:

13:02: Clear logs, restart sssd

13:03: id omen

13:04: ssh omen@ (correct password, 4 (System error))

In /var/log/auth.log:
 Nov 22 13:04:41 phys-adtest sshd[29803]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193  user=omen
 Nov 22 13:04:42 phys-adtest sshd[29803]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193 user=omen
 Nov 22 13:04:42 phys-adtest sshd[29803]: pam_sss(sshd:auth): received for user omen: 4 (System error)
 Nov 22 13:04:43 phys-adtest sshd[29803]: Failed password for omen from 169.237.42.193 port 42414 ssh2: RSA SHA256:FJYFiUaVTKvx6cL9QG07WURCN/hqRLMZ1WvZCSJXN/g

13:05: ssh omen@ (incorrect password)

In /var/log/auth.log:
 Nov 22 13:05:34 phys-adtest sshd[29823]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193  user=omen
 Nov 22 13:05:34 phys-adtest sshd[29823]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193 user=omen
 Nov 22 13:05:34 phys-adtest sshd[29823]: pam_sss(sshd:auth): received for user omen: 17 (Failure setting user credentials)
 Nov 22 13:05:37 phys-adtest sshd[29823]: Failed password for omen from 169.237.42.193 port 42434 ssh2: RSA SHA256:FJYFiUaVTKvx6cL9QG07WURCN/hqRLMZ1WvZCSJXN/g
 Nov 22 13:05:38 phys-adtest sshd[29823]: Connection closed by 169.237.42.193 port 42434 [preauth]

13:06: systemctl stop sssd

Thanks!
   Omen

[Sumit Bose]

On Fri, Dec 09, 2016 at 11:25:29AM -0800, Omen Wild wrote:
> A small group of us have been trying to get our Ubuntu hosts fully
> integrated into AD using sssd. We have slowly chipped away at the
> issues. We believe we are left with one major issue, when we try to
> login with SSH we get 4: (System error).
>
> The host is Ubuntu 16.04.1, up to date as of today, so sssd
> 1.13.4-1ubuntu1. All PAM files are the defaults.
>
> We used the `realm` command to join AD:
> realm -v join tou.t3.ucdavis.edu -U [e-mail]
>
> Our AD is set up with TOU.T3.UCDAVIS.EDU as a child domain in the same
> forest as the parent domain, T3.UCDAVIS.EDU, with users in
> T3.UCDAVIS.EDU and computers and groups in TOU.T3.UCDAVIS.EDU.
>
> All sssd logs (debug_level = 9) and config files are here:
>
> https://descolada.ucdavis.edu/415bfd2c-b0fa-11e6-97b8-3417ebb1df52/

It it (again) about Kerberos ticket validation. If you set
'krb5_validate = False' in the [domain/...] section of sssd.conf
authentication should work (with the correct password :-).

It looks like realmd's backend (adcli or net ads) has put the
RestrictedKrbHost/[e-mail] at the end of
/etc/krb5.keytab but the DC of TOU.T3.UCDAVIS.EDU does not know this
principal. Can you check if the host entry for your host in AD has
servicePrincipalName entries for all entries in /etc/krb5.keytab (klist
-k) except for the one with the '$' at the end of the hostname?

bye,
Sumit
> The timing that generated those log files:
>
> 13:02: Clear logs, restart sssd
>
> 13:03: id omen
>
> 13:04: ssh omen@ (correct password, 4 (System error))
>
> In /var/log/auth.log:
>  Nov 22 13:04:41 phys-adtest sshd[29803]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193  user=omen
>  Nov 22 13:04:42 phys-adtest sshd[29803]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193 user=omen
>  Nov 22 13:04:42 phys-adtest sshd[29803]: pam_sss(sshd:auth): received for user omen: 4 (System error)
>  Nov 22 13:04:43 phys-adtest sshd[29803]: Failed password for omen from 169.237.42.193 port 42414 ssh2: RSA SHA256:FJYFiUaVTKvx6cL9QG07WURCN/hqRLMZ1WvZCSJXN/g
>
> 13:05: ssh omen@ (incorrect password)
>
> In /var/log/auth.log:
>  Nov 22 13:05:34 phys-adtest sshd[29823]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193  user=omen
>  Nov 22 13:05:34 phys-adtest sshd[29823]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=169.237.42.193 user=omen
>  Nov 22 13:05:34 phys-adtest sshd[29823]: pam_sss(sshd:auth): received for user omen: 17 (Failure setting user credentials)
>  Nov 22 13:05:37 phys-adtest sshd[29823]: Failed password for omen from 169.237.42.193 port 42434 ssh2: RSA SHA256:FJYFiUaVTKvx6cL9QG07WURCN/hqRLMZ1WvZCSJXN/g
>  Nov 22 13:05:38 phys-adtest sshd[29823]: Connection closed by 169.237.42.193 port 42434 [preauth]
>
> 13:06: systemctl stop sssd
>
> Thanks!
>    Omen
>
> -- 
> Omen Wild
> Systems Administrator
> Metro Cluster
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Omen Wild]

Quoting Sumit Bose  on Sat, Dec 10 17:56:
> It it (again) about Kerberos ticket validation. If you set
> 'krb5_validate = False' in the [domain/...] section of sssd.conf
> authentication should work (with the correct password :-).

Awesome, this worked!
> It looks like realmd's backend (adcli or net ads) has put the
> RestrictedKrbHost/[e-mail] at the end of
> /etc/krb5.keytab but the DC of TOU.T3.UCDAVIS.EDU does not know this
> principal. Can you check if the host entry for your host in AD has
> servicePrincipalName entries for all entries in /etc/krb5.keytab (klist
> -k) except for the one with the '$' at the end of the hostname?

Hmmm, the host entry in AD actually has nothing in servicePrincipalName: <not set>.

I noticed all the entries in `klist -k` have ALL CAPS, i.e.:
RestrictedKrbHost/[e-mail]

However, `realm list` shows a mixed case:

----- Begin quote realm list -----
root@phys-adtest:/var/log/sssd# realm list
tou.T3.UCDAVIS.EDU
  type: kerberos
  realm-name: TOU.T3.UCDAVIS.EDU
  domain-name: tou.t3.ucdavis.edu
  configured: kerberos-member
  server-software: active-directory
  client-software: sssd
  required-package: sssd-tools
  required-package: sssd
  required-package: libnss-sss
  required-package: libpam-sss
  required-package: adcli
  required-package: samba-common-bin
  login-formats: [e-mail]
  login-policy: allow-realm-logins
----- End quote realm list -----

The distinguishedName attribute has the same mixed case in the DC:

CN=phys-adtest,[snip],DC=tou,DC=T3,DC=UCDAVIS,DC=EDU

Could tou.T3 vs TOU.T3 cause this kind of issue? If so, is there a
workaround on the sssd side?

[Sumit Bose]

On Mon, Dec 12, 2016 at 11:21:31AM -0800, Omen Wild wrote:
> Quoting Sumit Bose  on Sat, Dec 10 17:56:
> > It it (again) about Kerberos ticket validation. If you set
> > 'krb5_validate = False' in the [domain/...] section of sssd.conf
> > authentication should work (with the correct password :-).
>
> Awesome, this worked!
> > It looks like realmd's backend (adcli or net ads) has put the
> > RestrictedKrbHost/[e-mail] at the end of
> > /etc/krb5.keytab but the DC of TOU.T3.UCDAVIS.EDU does not know this
> > principal. Can you check if the host entry for your host in AD has
> > servicePrincipalName entries for all entries in /etc/krb5.keytab (klist
> > -k) except for the one with the '$' at the end of the hostname?
>
> Hmmm, the host entry in AD actually has nothing in servicePrincipalName: <not set>.

hmm, in theory the attribute should be filled during the join with
values matching the keytab entry. You can try to add the entries
manually. Here at least 'RestrictedKrbHost/phys-adtest' and
'RestrictedKrbHost/phys-adtest.tou.t3.ucdavis.edu' should be added.

You can check if it is working or not without SSSD by calling:

kinit [e-mail]
    kvno RestrictedKrbHost/[e-mail]

or 
    kvno 'RestrictedKrbHost/[e-mail]'
> I noticed all the entries in `klist -k` have ALL CAPS, i.e.:
> RestrictedKrbHost/[e-mail]
>
> However, `realm list` shows a mixed case:
>
> ----- Begin quote realm list -----
> root@phys-adtest:/var/log/sssd# realm list
> tou.T3.UCDAVIS.EDU
>   type: kerberos
>   realm-name: TOU.T3.UCDAVIS.EDU
>   domain-name: tou.t3.ucdavis.edu
>   configured: kerberos-member
>   server-software: active-directory
>   client-software: sssd
>   required-package: sssd-tools
>   required-package: sssd
>   required-package: libnss-sss
>   required-package: libpam-sss
>   required-package: adcli
>   required-package: samba-common-bin
>   login-formats: [e-mail]
>   login-policy: allow-realm-logins
> ----- End quote realm list -----
>
> The distinguishedName attribute has the same mixed case in the DC:
>
> CN=phys-adtest,[snip],DC=tou,DC=T3,DC=UCDAVIS,DC=EDU
>
> Could tou.T3 vs TOU.T3 cause this kind of issue? If so, is there a
> workaround on the sssd side?

According to the related RFCs Kerberos is case-sensitive. But AD
typically treats the Kerberos principal case-insensitive which sometimes
might cause issues on a client. But here I think this is not the reason
for the issues you see (it the missing servicePrincipalName, see above).

In my experience when creating the domain AD will always upper-case all
characters in the domain name to create the Kerberos realm, so the all
upper-case version is the canonical form and is always safe to use.
Since SSSD picks an entry from the keytab here which should be created
by the membership software (adcli or net ads) together with
servicePrincipalName value the case should in general always match.

HTH

bye,
Sumit
> -- 
> Omen Wild
> Systems Administrator
> Metro Cluster
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Lukas Slebodnik]

On (13/12/16 09:51), Sumit Bose wrote:
> On Mon, Dec 12, 2016 at 11:21:31AM -0800, Omen Wild wrote:
> > Quoting Sumit Bose  on Sat, Dec 10 17:56:
> > > It it (again) about Kerberos ticket validation. If you set
> > > 'krb5_validate = False' in the [domain/...] section of sssd.conf
> > > authentication should work (with the correct password :-).
> >
> > Awesome, this worked!
> > > It looks like realmd's backend (adcli or net ads) has put the
> > > RestrictedKrbHost/[e-mail] at the end of
> > > /etc/krb5.keytab but the DC of TOU.T3.UCDAVIS.EDU does not know this
> > > principal. Can you check if the host entry for your host in AD has
> > > servicePrincipalName entries for all entries in /etc/krb5.keytab (klist
> > > -k) except for the one with the '$' at the end of the hostname?
> >
> > Hmmm, the host entry in AD actually has nothing in servicePrincipalName: <not set>.
>
> hmm, in theory the attribute should be filled during the join with
> values matching the keytab entry. You can try to add the entries
> manually. Here at least 'RestrictedKrbHost/phys-adtest' and
> 'RestrictedKrbHost/phys-adtest.tou.t3.ucdavis.edu' should be added.
>
> You can check if it is working or not without SSSD by calling:
>
> kinit [e-mail]
>    kvno RestrictedKrbHost/[e-mail]
>
> or 
>    kvno 'RestrictedKrbHost/[e-mail]'

Maybe it might be better to provide content of keytab "klist -kt"
IIRC older version of sssd did not pick up right principal "SOMETHING$"
And it was fixed in newver version of sssd.

LS

[Omen Wild]

Quoting Lukas Slebodnik  on Tue, Dec 13 10:36:
> Maybe it might be better to provide content of keytab "klist -kt"
> IIRC older version of sssd did not pick up right principal "SOMETHING$"
> And it was fixed in newver version of sssd.

root@phys-adtest:~# klist -kt
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 host/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   3 10/24/2016 10:00:40 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   4 11/23/2016 12:54:18 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   4 11/23/2016 12:54:18 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   4 11/23/2016 12:54:18 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   4 11/23/2016 12:54:18 PHYS-ADTEST$@TOU.T3.UCDAVIS.EDU
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 host/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]
   4 11/23/2016 12:54:18 RestrictedKrbHost/[e-mail]

[Omen Wild]

Quoting Sumit Bose  on Tue, Dec 13 09:51:
> hmm, in theory the attribute should be filled during the join with
> values matching the keytab entry. You can try to add the entries
> manually. Here at least 'RestrictedKrbHost/phys-adtest' and
> 'RestrictedKrbHost/phys-adtest.tou.t3.ucdavis.edu' should be added.
>
> You can check if it is working or not without SSSD by calling:
> kinit [e-mail]

This works before and after the change to servicePrincipalName in AD.
klist shows a good service principal.
> kvno RestrictedKrbHost/[e-mail]

Before I updated AD to add the entries to servicePrincipalName this
failed with:

root@phys-adtest:~# kvno RestrictedKrbHost/[e-mail]
kvno: Server not found in Kerberos database while getting credentials for RestrictedKrbHost/[e-mail]

After I updated servicePrincipalName this succeeds(!) with:
root@phys-adtest:~# kvno RestrictedKrbHost/[e-mail] RestrictedKrbHost/[e-mail]: kvno = 3

So, looking over the verbose output to how we joined the domain (realm
-v join tou.t3.ucdavis.edu -U [e-mail]) I
see an error when it tried to set servicePrincipalName (full output attached):

! Couldn't set service principals on computer account CN=phys-adtest,OU=METRO-OU-AdminPCS,OU=METRO-OU-Computers,OU=METRO,OU=DEPARTMENTS,DC=tou,DC=T3,DC=UCDAVIS,DC=EDU: 00002083: AtrErr: DSID-03151785, #1:
 0: 00002083: DSID-03151785, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90303 (servicePrincipalName)

It seems likely this is because we have to pre-stage the computer object
in AD in order to place it in a folder/location where we have write
privileges. Does anyone know of a workaround that doesn't involve us
manually editing servicePrincipalName after the fact?

Thanks everyone for the help!

[Omen Wild]

Quoting Omen Wild  on Tue, Dec 13 12:28:
> So, looking over the verbose output to how we joined the domain (realm
> -v join tou.t3.ucdavis.edu -U [e-mail]) I
> see an error when it tried to set servicePrincipalName (full output attached):
>
> ! Couldn't set service principals on computer account CN=phys-adtest,OU=METRO-OU-AdminPCS,OU=METRO-OU-Computers,OU=METRO,OU=DEPARTMENTS,DC=tou,DC=T3,DC=UCDAVIS,DC=EDU: 00002083: AtrErr: DSID-03151785, #1:
>  0: 00002083: DSID-03151785, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90303 (servicePrincipalName)
>
> It seems likely this is because we have to pre-stage the computer object
> in AD in order to place it in a folder/location where we have write
> privileges. Does anyone know of a workaround that doesn't involve us
> manually editing servicePrincipalName after the fact?

I found a user with a similar problem
https://bugs.freedesktop.org/show_bug.cgi?id=86107 and the workaround
described there (adcli delete-computer , adcli join ... --host-fqdn=$(facter fqdn)) works!

So it looks like the issue is not sssd at this point, but how adcli does
the join. If anyone has any more tips I would love to hear them.

Thank you everyone for your help!

[Sumit Bose]

On Tue, Dec 13, 2016 at 05:28:42PM -0800, Omen Wild wrote:
> Quoting Omen Wild  on Tue, Dec 13 12:28:
> > So, looking over the verbose output to how we joined the domain (realm
> > -v join tou.t3.ucdavis.edu -U [e-mail]) I
> > see an error when it tried to set servicePrincipalName (full output attached):
> >
> > ! Couldn't set service principals on computer account CN=phys-adtest,OU=METRO-OU-AdminPCS,OU=METRO-OU-Computers,OU=METRO,OU=DEPARTMENTS,DC=tou,DC=T3,DC=UCDAVIS,DC=EDU: 00002083: AtrErr: DSID-03151785, #1:
> >  0: 00002083: DSID-03151785, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90303 (servicePrincipalName)
> >
> > It seems likely this is because we have to pre-stage the computer object
> > in AD in order to place it in a folder/location where we have write
> > privileges. Does anyone know of a workaround that doesn't involve us
> > manually editing servicePrincipalName after the fact?
>
> I found a user with a similar problem
> https://bugs.freedesktop.org/show_bug.cgi?id=86107 and the workaround
> described there (adcli delete-computer , adcli join ... --host-fqdn=$(facter fqdn)) works!
>
> So it looks like the issue is not sssd at this point, but how adcli does
> the join. If anyone has any more tips I would love to hear them.

I guess making sure that the hostname command returns the
fully-qualified domain name of the host would help as well. If there are
other application running on the host expecting the short name maybe the
fqdn can be set just during the join?

bye,
Sumit
> Thank you everyone for your help!
>
> -- 
> Omen Wild
> Systems Administrator
> Metro Cluster
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]