Design discussion: Performance enhancements for the 1.14 release
by Jakub Hrozek
Hi,
I wrote up the proposed design of the performance enhancement I'm
working on for 1.14 here:
https://fedorahosted.org/sssd/wiki/DesignDocs/OneFourteenPerformanceImpro...
For your convenience, the text of the page is also copied below. Please
note that there is one open question..
= Feature Name =
SSSD Performance enhancements for the 1.14 release
Related ticket(s):
* https://fedorahosted.org/sssd/ticket/2602
* https://fedorahosted.org/sssd/ticket/2062
=== Problem statement ===
At the moment SSSD doesn't perform well in large environments. Most of
the use-cases we've had reported revolved around logins of users who are
members of large groups or a large amount of groups. Another reported
use-case was the time it takes to resolve a large group.
While workarounds are available for some of the issues (such as using
`ignore_group_members` for resolution of large groups), our goal is to be
able to perform well without these workarounds.
=== Use cases ===
* User who is a member of a large amount of AD groups logs in to a Linux server that is a member of the AD domain.
* User who is a member of a large amount of AD or IPA groups logs in to a Linux server that is a member of an IPA domain with a trust relationship to an AD domain
* Administrator of a Linux server runs "ls -l" in a directory where files are owned by a large group. An example would be group called "students" in an university setup
=== Overview of the solution ===
During performance analysis with systemtap, we found out that the biggest
delay happens when SSSD writes an entry to the cache, especially for large group entries.
This is also confirmed by empirical evidence from our users, where most
deployments were OK with SSSD performance once the cache was moved to tmpfs or
even when `ignore_group_members` option was enabled.
We can't skip cache writes completely, even if no attributes changed,
because we store also the expiration timestamps in the cache. Also, even if
a single attribute (like the timestamp) changes, ldb would need to unpack the
whole entry, change the record, pack it back and then write the whole blob.
In order to mitigate the costly cache writes, we should avoid writing the
whole cache entry on every cache update, but only write the entries if something
actually changed.
To avoid this, we will split the monolithic ldb cache representing the
sysdb cache into two ldb files. One would contain the entry itself and would
be fully synchronous. The other (new one) would only contain the timestamps
and would be open using the `LDB_FLG_NOSYNC` to avoid synchronous cache writes.
This would have two advantages:
1. If we detect that the entry hasn't changed on the LDAP server at all, we could avoid writing into the main ldb cache which would still be costly. We would use the value of the `modifyTimestamp` attribute of the LDAP entry to see if the entry had changed or not.
1. The writes to the new async ldb cache would be much faster, because the entry is smaller and because the writes wouldn't call `fsync()` due to using the async flag, but rather rely on the underlying filesystem to sync the data to the disk.
On SSSD shutdown, we would write a canary to both the timestamp cache
and the main sysdb cache, denoting graceful shutdown. On SSSD startup,
if the canary wasn't found or if the values differ, we would just ditch
the timestamp cache, which would result in refresh and write of the entry
on the next lookup.
The basic idea is to use a combination of the operational `modifyTimestamp`
attribute and checking the entry itself to see if the entry changed at
all and if not, avoid writing to the cache.
Checking the value of `modifyTimestamp` would be enough for group entries,
which should be the first iteration of this work. For checking if other
entries (mostly users) have changed, we need to compare the value of
the attributes in the cache with what we are about to store in the cache.
Therefore, these enhancements are proposed for the 1.14 versions, sorted by the importance as observed with systemtap testing:
* only write the cache entry if the `modifyTimestamp` of the original entry had changed. If it hasn't changed, only the timestamps would be written to the timestamp cache
* if the `modifyTimestamp` had changed, compare the attributes of the cache entry with the attributes we are about to write. If there are no differences, only write to the timestamp cache
* refactor the nested group processing to make sure expensive lookups (such as lookups of all members of the group, there can potentially be thousands of these) are only performed once and intermediate results are stored in-memory.
* attempt to shortcut parsing the attributes of the entry returned from LDAP sooner. The idea behind this is that if the `modifyTimestamp` did not change, we can reuse the entry we already cached.
Minor enhancements in later versions might include:
* using syncrepl in the server mode for HBAC rules and external groups in refreshAndPersistMode. This would provide performance benefit for legacy clients that rely on server's HBAC rules for access control.
* using syncrepl in the server mode for external groups in refreshAndPersistMode. This would mainly simplify the external groups handling, rather than improve performance
* A lot of time is spent looking up attributes in the `sysdb_attrs` array. This is something we might want to optimize after we're done with the cache writes.
* We might even consider offering syncrepl in refreshOnly mode as an client-side option for enumeration. However, this would have to be an opt-in because every refresh causes the server to walk the changelog since the last refresh operation. Enabling this option on all clients would trash the server performance.
=== Implementation details ===
The `sysdb_ctx` already contains a handle of the main sysdb cache. We
would add another ldb file that only contains the timestamp and the DN of
an entry. This ldb file would be opened in the nosync mode. Attributes used
for lookups, like `dataExpireTimestamp` must be indexed in this database
as well.
When storing a user or a group to sysdb using functions like
`sysdb_store_user`, we first check the difference between `modifyTimestamp`
attributes. If there are no differences, only the timestamp attributes,
such as `lastUpdate` or `dataExpireTimestamp` would be updated in the
timestamp cache. We need to do this check in the lower-level sysdb calls
to make sure this enhancement also works for users and groups retrieved
through the extop plugin.
If the value of `modifyTimestamp` differs, we proceed to checking the diff
between values in the cache and the values read from LDAP.
Details about shortcut of attribute parsing will be added to this design
page later.
=== Open questions ===
When SSSD switches to another server (a replica), this replica might be
out-of-sync. To be on the safe side, we should erase the timestamp cache
to make sure we update the entries from the server. However, if the servers
are really out-of-sync, then the modifyTimestamp would differ and we would
write the entries anyway. Therefore, I think we can always rely on the
modifyTimestamp value.
=== Configuration changes ===
Currently no configuration changes are expected. We might add some if we decide to implement on-demand syncrepl.
=== How To Test ===
If the entries on the server did not change (except timestamps), then actions like user and group lookups and logins should be considerably faster.
The SSSD should also correctly detect when the entries in fact did change on the server. In this case, a full cache write will be performed.
=== Authors ===
* Jakub Hrozek <jhrozek(a)redhat.com> with the kind help of
* Sumit Bose <sbose(a)redhat.com>
* Ludwig Krispenz <lkrispen(a)redhat.com>
* Simo Sorce <simo(a)redhat.com>
8 years
Announcing SSSD 1.13.4
by Jakub Hrozek
== SSSD 1.13.4 ===
The SSSD team is proud to announce the release of version 1.13.4 of
the System Security Services Daemon.
As always, the source is available from https://fedorahosted.org/sssd
RPM packages will be made available for Fedora shortly.
== Feedback ==
Please provide comments, bugs and other feedback via the sssd-devel
or sssd-users mailing lists:
https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
https://lists.fedorahosted.org/mailman/listinfo/sssd-users
== Highlights ==
* The IPA sudo provider was reimplemented. The new version reads the
data from IPA's LDAP tree (as opposed to the compat tree populated by
the slapi-nis plugin that was used previously). The benefit is that
deployments which don't require the compat tree for other purposes,
such as support for non-SSSD clients can disable those autogenerated
LDAP trees to conserve resources that slapi-nis otherwise requires. There
should be no visible changes to the end user.
* SSSD now has the ability to renew the machine credentials (keytabs)
when the ad provider is used. Please note that a recent version of
the adcli (0.8 or newer) package is required for this feature to work.
* The automatic ID mapping feature was improved so that the administrator
is no longer required to manually set the range size in case a RID in
the AD domain is larger than the default range size
* A potential infinite loop in the NFS ID mapping plugin that was
resulting in an excessive memory usage was fixed
* Clients that are pinned to a particular AD site using the ad_site
option no longer communicate with DCs outside that site during service
discovery.
* The IPA identity provider is now able to resolve external
(typically coming from a trusted AD forest) group members during
get-group-information requests. Please note that resolving external
group memberships for AD users during the initgroup requests used to
work even prior to this update. This feature is mostly useful for cases
where an IPA client is using the compat tree to resolve AD trust users.
* The IPA ID views feature now works correctly even for deployments
without a trust relationship. Previously, the subdomains IPA provider
failed to read the views data if no master domain record was created
on the IPA server during trust establishment.
* A race condition in the client libraries between the SSSD closing
the socket as idle and the client application using the socket was
fixed. This bug manifested with a Broken Pipe error message on the
client.
* SSSD is now able to resolve users with the same usernames in different
OUs of an AD domain
* The smartcard authentication now works properly with gnome-screensaver
== Packaging Changes ==
* The krb5.include.d directory is now owned by the sssd user and
packaged in the krb5-common subpackage
== Documentation Changes ==
* A new option ldap_idmap_helper_table_size was added. This option can
help tune allocation of new ID mapping slices for AD domains with a high
RID values. Most deployments can use the default value of this option.
* Several PAM services were added to the lists that are used to map
Windows logon services to Linux PAM services. The newly added PAM
services include login managers (lightdm, lxdm, sddm and xdm) as well
as the cockpit service.
* The AD machine credentials renewal task can be fine-tuned using
the ad_machine_account_password_renewal_opts to change the initial
delay and period of the credentials renewal task. In addition, the new
ad_maximum_machine_account_password_age option allows the administrator
to select how old the machine credential must be before trying to
renew it.
* The administrator can use the new option pam_account_locked_message to
set a custom informational message when the account logging in is locked.
== Tickets Fixed ==
https://fedorahosted.org/sssd/ticket/1041
[RFE] Support Automatic Renewing of Kerberos Host Keytabs
https://fedorahosted.org/sssd/ticket/1108
[RFE] SUDO: Support the IPA schema
https://fedorahosted.org/sssd/ticket/2188
automatically assign new slices for any AD domain
https://fedorahosted.org/sssd/ticket/2522
[RFE] IPA: resolve external group memberships of IPA groups during
getgrnam and getgrgid
https://fedorahosted.org/sssd/ticket/2626
Retry EPIPE from clients
https://fedorahosted.org/sssd/ticket/2764
the colondb intreface has no unit tests
https://fedorahosted.org/sssd/ticket/2765
ad_site parameter does not work
https://fedorahosted.org/sssd/ticket/2785
incompatibility between sparkleshare and sss_ssh_knownhostsproxy due
to setlocale()
https://fedorahosted.org/sssd/ticket/2791
sssd dereference processing failed : Input/output error
https://fedorahosted.org/sssd/ticket/2829
collapse_srv_lookups frees fo_server structure that is returned by
fail over API
https://fedorahosted.org/sssd/ticket/2839
Allow SSSD to notify user of denial due to AD account lockout
https://fedorahosted.org/sssd/ticket/2849
cache_req: don't search override values in LDAP when using LOCAL view
https://fedorahosted.org/sssd/ticket/2865
sssd_nss memory usage keeps growing on sssd-1.12.4-47.el6.x86_64
(RHEL6.7) when trying to retrieve non-existing netgroups
https://fedorahosted.org/sssd/ticket/2881
MAN: Clarify that subdomains always use service discovery
https://fedorahosted.org/sssd/ticket/2888
SRV lookups with id_provider=proxy and auth_provider=krb5
https://fedorahosted.org/sssd/ticket/2899
[sssd] Trusted (AD) user's info stays in sssd cache for much more
than expected.
https://fedorahosted.org/sssd/ticket/2902
Review and update wiki pages for 1.13.4
https://fedorahosted.org/sssd/ticket/2904
sssd_be AD segfaults on missing A record
https://fedorahosted.org/sssd/ticket/2906
Cannot retrieve users after upgrade from 1.12 to 1.13
https://fedorahosted.org/sssd/ticket/2909
extreme memory usage in libnfsidmap sss.so plug-in when resolving
groups with many members
https://fedorahosted.org/sssd/ticket/2910
sssd mixup nested group from AD trusted domains
https://fedorahosted.org/sssd/ticket/2912
refresh_expired_interval stops sss_cache from working
https://fedorahosted.org/sssd/ticket/2917
Properly remove OriginalMemberOf attribute in SSSD cache if user has
no secondary groups anymore
https://fedorahosted.org/sssd/ticket/2922
ID mapping - bug in computing max id for slice range
https://fedorahosted.org/sssd/ticket/2925
Add gnome-screensaver to the list of PAM services considered for
Smartcard authentication
https://fedorahosted.org/sssd/ticket/2931
Warn if user cannot read krb5.conf
https://fedorahosted.org/sssd/ticket/2934
After removing certificate from user in IPA and even after sss_cache,
FindByCertificate still finds the user
https://fedorahosted.org/sssd/ticket/2937
sss_obfuscate: SyntaxError: Missing parentheses in call to 'print'
https://fedorahosted.org/sssd/ticket/2938
Cannot start sssd after switching to non-root
https://fedorahosted.org/sssd/ticket/2959
The delete operation of the memberof plugin allocates memory on
NULL context
https://fedorahosted.org/sssd/ticket/2960
IPA view: view name not stored properly with default FreeIPA installation
https://fedorahosted.org/sssd/ticket/2961
Initgroups in AD provider might fail if user is stored in a non-default ou
https://fedorahosted.org/sssd/ticket/2962
GPO: Access denied in non-root mode
https://fedorahosted.org/sssd/ticket/2964
GPO: Access denied after blocking connection to AD.
https://fedorahosted.org/sssd/ticket/2969
sudorule not working with ipa sudo_provider on older freeipa
https://fedorahosted.org/sssd/ticket/2970
sudo smart refresh does not work correctly on openldap
https://fedorahosted.org/sssd/ticket/2971
SSSD PAM module does not support multiple password prompts (e.g. Password
+ Token) with sudo
https://fedorahosted.org/sssd/ticket/2972
IPA sudo: support the externalUser attribute
https://fedorahosted.org/sssd/ticket/2980
sssd_be[11010]: segfault at 0 ip 00007ff889ff61bb sp 00007ffc7d66a3b0
error 4 in libsss_ipa.so[7ff889fcf000+5d000]
== Detailed Changelog ==
Dan Lavu (1):
* PAM: Fix man for pam_account_{expired,locked}_message
David Disseldorp (1):
* build: detect endianness at configure time
Jakub Hrozek (17):
* Upgrading the version for the 1.13.4 release
* SDAP: Make it possible to silence errors from dereference
* Add a new option ldap_group_external_member
* IPA: Add interface to call into IPA provider from LDAP provider
* LDAP: Use the IPA provider interface to resolve external group members
* FO: Don't free rc-allocated structure
* tests: Reduce failover code duplication
* FO: Use refcount to keep track of servers returned to callers
* FO: Use tevent_req_defer_callback() when notifying callers
* memberof: Don't allocate on a NULL context
* tests: Add a unit test for the external groups resolution
* MAN: Remove duplicate description of the pam_account_locked_message option
* AD: Recognize Windows Server 2016
* memberof: Fix a memory leak when removing ghost users
* memberof: Don't allocate on NULL when deleting memberUids
* tests: Check NULL context in sysdb-tests when removing group members
* Updating translations for the 1.13.4 release
Lukas Slebodnik (33):
* SPEC: Change package ownership of %{pubconfpath}/krb5.include.d
* CONFIGURE: Replace obsoleted macro AC_PROG_LIBTOOL
* TESTS: Fix race condition in python test
* PYTHON: sss_obfuscate should work with python3
* PYTHON: Fix pep8 errors in sss_obfuscate
* UTIL: Backport error code ERR_ACCOUNT_LOCKED
* sss_idmap-tests: Fix segmentation fault
* krb5_child: Warn if user cannot read krb5.conf
* Fix typos reported by lintian
* UTIL: Use prefix for debug function
* UTIL: Provide varargs version of debug_fn
* UTIL: Use sss_vdebug_fn for callbacks
* Revert "DEBUG: Preventing chown_debug_file if journald on"
* DEBUG: Ignore ENOENT for change owner of log files
* TOOLS: Fix minor memory leak in sss_colondb_writeline
* CI: Use yum-deprecated instead of dnf
* FAIL_OVER: Fix warning value computed is not used
* UTIL: Fix indentation in dlinklist.h
* UTIL: Fix warning misleading-indentation
* CLIENT: Reduce code duplication
* CLIENT: Retry request after EPIPE
* UTIL: Move debug part from util.h -> new debug.h
* UTIL: Allow to append new line in sss_vdebug_fn
* AUTOMAKE: Force usage of parallel test harness
* CI: Use make check instead of make-check-wrap
* test_ipa_subdom_server: Workaround for slow krb5 + SELinux
* SPEC: Run extra unit tests with epel
* GPO: Soften umask in gpo_child
* GPO_CHILD: Create directories in gpo_cache with right permissions
* GPO: Process GPOS in offline mode if ldap search failed
* IPA: Check RDN in ipa_add_ad_memberships_get_next
* dp_ptask: Fix memory leak in synchronous ptask
* test_be_ptask: Check leaks in tests
Michal Židek (6):
* NSS: do not skip cache check for netgoups
* util: Continue if setlocale fails
* server_setup: Log failed attempt to set locale
* tests: Run intgcheck without libsemanage
* tests: Regression test with wrong LC_ALL
* GPO: log specific ini parse error messages
Pavel Březina (37):
* AD SRV: prefer site-local DCs in LDAP ping
* SDAP: do not fail if refs are found but not processed
* SDAP: Add request that iterates over all search bases
* SDAP: rename sdap_get_id_specific_filter
* SDAP: support empty filters in sdap_combine_filters()
* SUDO: use sdap_search_bases instead custom sb iterator
* SUDO: make sudo sysdb interface more reusable
* SUDO: move code shared between ldap and ipa to separate module
* SUDO: allow to disable ptask
* SUDO: fail on failed request that cannot be retry
* IPA: add ipa_get_rdn and ipa_check_rdn
* SDAP: use ipa_get_rdn() in nested groups
* IPA SUDO: choose between IPA and LDAP schema
* IPA SUDO: Add ipasudorule mapping
* IPA SUDO: Add ipasudocmdgrp mapping
* IPA SUDO: Add ipasudocmd mapping
* IPA SUDO: Implement sudo handler
* IPA SUDO: Implement full refresh
* IPA SUDO: Implement rules refresh
* IPA SUDO: Remember USN
* SDAP: Add sdap_or_filters
* IPA SUDO: Implement smart refresh
* SUDO: sdap_sudo_set_usn() do not steal usn
* SUDO: remove full_refresh_in_progress
* SUDO: assume zero if usn is unknown
* SUDO: allow disabling full refresh
* SUDO: remember usn as number instead of string
* SUDO: simplify usn filter
* IPA SUDO: Add support for ipaSudoRunAsExt* attributes
* sdap_connect_send: fail if uri or sockaddr is NULL
* cache_req: simplify cache_req_cache_check()
* cache_req: do not lookup views if possible
* remove user certificate if not found on the server
* IPA SUDO: download externalUser attribute
* IPA SUDO: fix typo
* IPA SUDO: support old ipasudocmd rdn
* SUDO: be able to parse modifyTimestamp correctly
Pavel Reichl (11):
* sudo: remove unused param name in sdap_sudo_get_usn()
* sudo: remove unused param. in ldap_get_sudo_options
* IDMAP: Fix computing max id for slice range
* IDMAP: New structure for domain range params
* IDMAP: Add support for automatic adding of ranges
* IDMAP: Fix minor memory leak
* IDMAP: Man change for ldap_idmap_range_size option
* NSS: Fix memory leak netgroup
* IDMAP: Add test to validate off by one bug
* SDAP: Add return code ERR_ACCOUNT_LOCKED
* PAM: Pass account lockout status and display message
Petr Cech (6):
* KRB5: Adding DNS SRV lookup for krb5 provider
* TOOLS: Fix memory leak after getline() failed
* TOOLS: Add comments on functions in colondb
* TEST_TOOLS_COLONDB: Add tests for sss_colondb_*
* REFACTOR: umask(077) --> umask(SSS_DFL_X_UMASK)
* REFACTOR: umask(0177) --> umask(SSS_DFL_UMASK)
Stephen Gallagher (2):
* GPO: Add Cockpit to the Remote Interactive defaults
* GPO: Add other display managers to interactive logon
Sumit Bose (20):
* nfs idmap: fix infinite loop
* Use right domain for user lookups
* sdap_save_grpmem: determine domain by SID if possible
* ipa_s2n_save_objects(): use configured user and group timeout
* ldap: remove originalMeberOf if there is no memberOf
* UTIL: allow to skip default options for child processes
* DP_TASK: add be_ptask_get_timeout()
* AD: add task to renew the machine account password if needed
* FO: add fo_get_active_server()
* FO: add be_fo_get_active_server_name()
* AD: try to use current server in the renewal task
* p11: add gnome-screensaver to list of allowed services
* IPA: lookup idview name even if there is no master domain record
* IPA: invalidate override data if original view is missing
* sdap: improve filtering of multiple results in GC lookups
* pam_sss: reorder pam_message array
* sss_override: do not generate DN, search object
* tools: read additional data of the master domain
* sss_override: only add domain if name is not fully qualified
* intg: local override for user with mixed case name
8 years
