sssd performance on large domains
by zfnoctis＠gmail.com
Hi,
I'm wondering if there are any plans to improve sssd performance on large active directory domains (100k+ users, 40k+ groups), or if there are settings I am not aware of that can greatly improve performance, specifically for workstation use cases.
Currently if I do not set "ignore_group_members = True" in sssd.conf, logins can take upwards of 6 minutes and "sssd_be" will max the CPU for up to 20 minutes after logon, which makes it a non-starter. The reason I want to allow group members to be seen is that I want certain domain groups to be able to perform elevated actions using polkit. If I ignore group members, polkit reports that the group is empty and so no one can elevate in the graphical environment.
Ultimately this means that Linux workstations are at a severe disadvantage since they cannot be bound to the domain and have the normal set of access features users and IT expect from macOS or Windows.
Distributions used: Ubuntu 16.04 (sssd 1.13.4-1ubuntu1.1), Ubuntu 16.10 (sssd 1.13.4-3) and Fedora 24 (sssd-1.13.4-3.fc24). All exhibit the same problems.
I've also tried "ldap_group_nesting_level = 1" without seeing any noticeable improvement with respect to performance. Putting the database on /tmp isn't viable as these are workstations that will reboot semi-frequently, and I don't believe this is an I/O bound performance issue anyways.
Thanks for your time.
1 year, 4 months
kcm, gssproxy and klist
by Winberg Adam
With KCM and gssproxy we often see a long list of credentials when doing a 'klist':
[user.u@lxserv2114 ~]$ klist
Ticket cache: KCM:17098:66803
Default principal: user.u@AD
Valid starting Expires Service principal
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
01/01/1970 00:00:00 01/01/1970 00:00:00 Encrypted/Credentials/v1@X-GSSPROXY:
and so on...
The actual gssproxy credentials at /var/lib/gssproxy/clients/ does not correspond with this output, it only contains what could be expected - a TGT and maybe some service tickets.
The ever growing 'klist' list of credentials is a problem, after a while the user can no longer get any new credentials and therefore has no access to its NFS homedir (sec=krb5). I'm guessing it's the 'max_uid_ccaches' option in sssd-kcm that prevents this.
What is going on here - have we configured gssproxy/kcm wrong or is this a bug?
Regards
Adam
1 year, 5 months
Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
by Spike White
Sssd experts,
*Short summary: * How can we troubleshoot sssd’s ‘Automatic Kerberos Host
Keytab Renewal’ process? We have ~0.4% of our Linux servers dropping
off the AD domain monthly.
*Longer explanation:*
Over the past two years, we have on-boarded sssd as our Linux AD
integration component. Largely displacing a former commercial product that
did the same.
We have about ~20K Linux servers that are sssd-enabled. A mix of RHEL6,
RHEL7, RHEL8, Oracle Linux 6, 7 and 8. We have ~7K Linux servers still on
the old commercial product. (For certain edge-case scenarios, such as
DMZs, the commercial product works better.)
Our AD forest is a single AD forest, with 4 regional child domains. All
with transitive trust. Sssd auto-discovers parent domain and all 4 child
domains, no problem – whenever it’s adcli joined to its regional local
domain.
Why are I writing this?
Because we are researching an ongoing problem reported by L1 server ops.
About 70 – 80 sssd-enabled Linux servers / month drop off the domain. Out
of our current sssd-enabled population of ~20K server, that’s not
horrible. But still it should be better. (Our former commercial product
did better.)
It’s not limited to one particular OS, OS version, build location or
region. We have surveyed; it seems to occur randomly among all OS
versions, regions and locations.
To be clear, it’s extremely likely that this behavior arising from some
subtle misconfiguration on our part – not from any sssd or adcli or
Kerberos bug. We have a couple of configuration improvements we’re
pursuing. (Kerberos max ticket lifetime mismatch between AD and
/etc/krb5.conf file for instance.)
We are taking sssd’s default settings for
ad_maximum_machine_account_password_age and
ad_machine_account_password_renewal_opts. So after 30 days, sssd will
attempt daily to renew the host Kerberos keytab file. It should re-attempt
daily if not renewed. By company policy, our AD disables any machine
accounts that have not renewed their credentials in 40 days. So when we
find servers that have dropped off the domain, it’s because they have not
renewed their AD machine accounts in 40 days.
We have SR’s open with our OS vendors (Redhat and Oracle respectively) for
months now. To no great help. (They gave a few suggestions, but none
panned out.)
We thought we were hitting this bug:
https://github.com/SSSD/sssd/issues/4762
But packet captures proved that adcli update is using TCP on RHEL7/8.
Thus, this might be a potential problem, but only on RHEL6. (BTW
‘udp_preference_limit = 0’ doesn’t force use of TCP for the kpasswd
invocation in RHEL6 – it still uses UDP. Thus, the recommended work-around
for this bug doesn’t work.)
So that isn’t our underlying problem.
We’re at a loss now – as you can see, we’re grasping at straws.
How can we troubleshoot sssd’s ‘automatic Kerberos Host keytab renewal’
process? Whenever we inspect a particular server it works. We can’t run
all sssd clients at debug level 9; it fills up /var/log filesystem after a
few days of that. We’re interested in troubleshooting that one particular
sssd process on all clients; not all parts of sssd.
Other than a steep learning curve (on our part), obscure situations (like
DMZ auto-discovery of AD controllers) and exotic scenarios (like above),
we’re quite happy with our 2 yr journey of direct AD integration with
sssd. Obviously, the troubleshooting tools on RHEL6 are very minimal.
But certainly, overall the quality of sssd on RHEL7/8 is excellent. AD
integration has innumerable devils in the details; I’m amazed that sssd
performs as well as it does against our multi-domain forest.
Spike
PS the problem with sssd auto-discovery of AD controllers in DMZs has been
fixed in a recent sssd release. The better discovery algorithm was
implemented – same one used by Windows clients and commercial products.
It’s just that recent sssd version is not on RHEL7 or 8.
1 year, 7 months
Any way to auto create groups while still mapping UIDs to uidNumber?
by Kurt Stine
I was told this would be a better place than github issues.
We're moving from an ldap environment to an AD environment. This means we have a large amount of users who are still linked with their original ldap UIDs. Unfortunately, changing the UIDs to the automatically assigned AD UIDs is not an option. Is there a way for SSSD to both inherit a UID if it exists or create its own if it doesn't? Also, is it possible to sync AD groups as well?
Right now we have this:
#ldap_id_mapping = True
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
Unfortunately, even removing the last line doesn't sync AD groups. Are there any options to sync them?
Thanks,
Kurt
1 year, 8 months
SSSD and Windows ACLs
by Patrick Goetz
The Redhat systems administration documentation includes this comment:
----------------------------
Important
Red Hat only supports running Samba as a server with the winbindd
service to provide domain users and groups to the local system. Due to
certain limitations, such as missing Windows access control list (ACL)
support and NT LAN Manager (NTLM) fallback, SSSD is not supported.
----------------------------
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/...
I'm struggling to understand what this even means. ACLs are filesystem
metadata, and AFAIK SSSD doesn't do filesystems, it does authentication.
It makes sense to talk about Samba support for Windows ACLs because
Samba serves up files, so can keep track of associated metadata. What
are they talking about here?
I have no problem using extended POSIX ACLs on my sssd-based linux
systems because the underlying filesystems (local or NFS remote) support
extended POSIX ACLs and I don't recall doing anything at all to alert
sssd to the fact that I'm going to use extended POSIX ACLs.
Elaborating, with proposed explanation:
Here are the files of a former user who was removed from AD:
root@helios:/home/hgj258/Crystal/JanssenPreF# ls -l
total 8372
-rw-r--r--+ 1 196740954 domain users 1775277 Aug 28 2018 4mms.pdb
-rw-r--r--+ 1 196740954 domain users 6178772 Aug 28 2018 4mms_phases.mtz
drwxr-xr-x+ 3 196740954 domain users 4096 Aug 29 2018 CCP4i
drwxr-xr-x+ 4 196740954 domain users 4096 Jan 18 2019 Coot
...
Normally sssd fills in the AD UserName (?) attribute, but since that's
no longer available, it's showing the mapped UID, 196740954. That means
this UID is stored in the inode of the file, along with any ACLs.
Does the Redhat statement above "sssd is missing Windows ACL support"
mean that in addition to doing authentication, sssd is somehow
involved in the authorization process and can't read Windows ACLs out of
the file metadata? The only case I can think of where this would be
applicable (since no linux filesystems I'm aware of support Windows
ACLs) is if you are mounting a Samba or Windows filesystem using the
cifs mount type -- is that what they're talking about?
1 year, 8 months
auto_private_groups -- How does it work?
by Patrick Goetz
There was a discussion on another list involving how to use sssd for
authentication on an HPC cluster, and the issue of auto_private_groups
came up.
I realized I have no idea how this works. I know sssd keeps the GID
(obviously known immediately from the UID) on the local host, but what
is stored as the primary group for such files on the fileserver? Let's
say my UID is 1562224688. How does the file server distinguish between
files that are supposed to have the GID 1562224688 vs the ones set to,
say, 1007000513 ?
Also, does this mean that sssd short circuits group authorization requests?
1 year, 8 months
D-Bus / SSSD / LDAP authentication from a java application
by Daniil Kirilyuk
We're developing a java application, which should authenticate users against both LDAP and custom formatted files containing user information. Both username/password and certificate authentication are planned to be supported. Our application should run mainly on RHEL. We were estimating the possibility to use SSSD for this purpose. After some investigation it seems, that SSSD can be called from java code only via D-Bus. It also seems, that it can be used mainly for fetching user information. but not for authentication.
E.g. for fetching user by uid:
dbus-send --print-reply --system --dest=org.freedesktop.sssd.infopipe /org/freedesktop/sssd/infopipe/Users org.freedesktop.sssd.infopipe.Users.FindByName string:<UID>
For retrieving user groups:
dbus-send --print-reply --system --dest=org.freedesktop.sssd.infopipe /org/freedesktop/sssd/infopipe/Users/<DOMAIN>/<UID> orgfreedesktop.DBus.Properties.Get string:org.freedesktop.sssd.infopipe.Users.User string:groups
For retrieving some extra attributes (after adding them to sssd.conf);
dbus-send --print-reply --system --dest=org.freedesktop.sssd.infopipe /org/freedesktop/sssd/infopipe/Users/<DOMAIN>/<UID> orgfreedesktop.DBus.Properties.Get string:org.freedesktop.sssd.infopipe.Users.User string:"extraAttributes"
Somewhat promising looks method FindByNameAndCertificate:
dbus-send --print-reply --system --dest=org.freedesktop.sssd.infopipe /org/freedesktop/sssd/infopipe/Users org.freedesktop.sssd.infopipe.Users.FindByNameAndCertificate string:<UID> string:<PEM_CERT>
But as far as I understand, FindByNameAndCertificate just compares string representation of a pem certificate and is far from client certificate authentication.
Do I understand correctly, that at the moment there is no possibility to perform user authentication via D-Bus API through SSSD in LDAP? Or am I missing something?
1 year, 8 months
Re: Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
by James Ralston
On 2021-09-08 at 14:18-0400 Todd Mote <moter(a)austin.utexas.edu> wrote:
> The $ at the end of the host name is for AD. <short hostname>$ is
> the actual name of the account in AD. The Kerberos utilities are
> just asking the KDC to renew tickets for accounts. Computer
> accounts in AD happen to have a $ appended to them under the covers.
> They are obfuscated from most human views. Msktutil may be
> appending the $ under its covers, you'd have to examine the source
> to know
From the msktutil man page:
--computer-name <name>
Specifies that the new account should use <name> for the
computer account name and the SAM Account Name. Note that a
'$' will be automatically appended to the SAM Account Name.
Defaults to the machine's hostname, excluding the realm, with
dots replaced with dashes.
That is: if the realm is EXAMPLE.COM, and the hostname is
FOO.EXAMPLE.COM, the default computer name is FOO. If the
hostname is FOO.BAR.EXAMPLE.COM, the default computer name is
FOO-BAR.
--account-name <name>
An alias for --computer-name that can be used when operating
on service accounts. Note that a '$' will not be
automatically appended to the SAM Account Name when using
service accounts.
We love msktutil (including its magnificent man page); it is our
preferred mechanism for joining hosts to AD.
1 year, 8 months
Re: Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
by Patrick Goetz
So i think you just want:
kinit -R -k $HOSTNAME$
On 8/30/21 11:28 AM, Mote, Todd wrote:
> I wrote that from memory. What's needed is the shortname and a $, but thinking more about it now, that's needed at the end of the shortname not the beginning.
>
> -----Original Message-----
> From: Patrick Goetz <pgoetz(a)math.utexas.edu>
> Sent: Monday, August 30, 2021 11:25 AM
> To: sssd-users(a)lists.fedorahosted.org
> Subject: [SSSD-users]Re: Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
>
> Todd,
>
> On 8/27/21 9:41 AM, Mote, Todd wrote:
>> We ultimately decided to deploy a cron job with the install that ran periodically (less than the renewal period) to keep the keytab fresh (kinit -R -k $($hostname -s)). We haven't had computers falling off the domain since we implemented that.
>>
>
> Are you sure about this syntax? Adding a -s flag in $( ) containing a bash variable doesn't do anything.
>
> _______________________________________________
> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahoste...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>>> This message is from an external sender. Learn more about why this <<
>>> matters at https://links.utexas.edu/rtyclf. <<
> _______________________________________________
> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahoste...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>>> This message is from an external sender. Learn more about why this <<
>>> matters at https://links.utexas.edu/rtyclf. <<
1 year, 8 months
