sssd performance on large domains
by zfnoctis＠gmail.com
Hi,
I'm wondering if there are any plans to improve sssd performance on large active directory domains (100k+ users, 40k+ groups), or if there are settings I am not aware of that can greatly improve performance, specifically for workstation use cases.
Currently if I do not set "ignore_group_members = True" in sssd.conf, logins can take upwards of 6 minutes and "sssd_be" will max the CPU for up to 20 minutes after logon, which makes it a non-starter. The reason I want to allow group members to be seen is that I want certain domain groups to be able to perform elevated actions using polkit. If I ignore group members, polkit reports that the group is empty and so no one can elevate in the graphical environment.
Ultimately this means that Linux workstations are at a severe disadvantage since they cannot be bound to the domain and have the normal set of access features users and IT expect from macOS or Windows.
Distributions used: Ubuntu 16.04 (sssd 1.13.4-1ubuntu1.1), Ubuntu 16.10 (sssd 1.13.4-3) and Fedora 24 (sssd-1.13.4-3.fc24). All exhibit the same problems.
I've also tried "ldap_group_nesting_level = 1" without seeing any noticeable improvement with respect to performance. Putting the database on /tmp isn't viable as these are workstations that will reboot semi-frequently, and I don't believe this is an I/O bound performance issue anyways.
Thanks for your time.
1 year, 7 months
id_overide issue | 03111823
by Abhijit Roy
Cus is using id override for user(bjsadmin(a)munimadrid.es). The user has its
uid overrided to 16792947
After restart of sssd id bjsadmin(a)munimadrid.es shows 'no such user'.
Workaround: Performing a gid-override for bjsadmin(a)munimadrid.es resolves
the issue
The versions details:
Machine RHEL 8.0 where it works -> sssd-2.2.0-19.el8_1.1.x86_64
Machine RHEL 8.0 where it does not work -->
sssd-2.3.0-9.el8.x86_64/sssd-2.2.3-20.el8.x86_64/sssd-2.5.2-2.el8_5.1.x86_64
RHEL 8.5 and sssd-2.5.2-2.el8_5.1.x86_64, it does not work. We need to add
gid-override for bjsadmin(a)munimadrid.es
It works on all RHEL 7 machines.
Did we change anything from in terms of override from
sssd-2.2.0-19.el8_1.1.x86_64 to
sssd-2.3.0-9.el8.x86_64/sssd-2.2.3-20.el8.x86_64
Thank you,
*Abhijit Roy*
He/Him/His
Technical Support Engineer
Red Hat Insights - Predict Risk. Get Guidance. Improve Security.
<https://www.redhat.com/en/technologies/management/insights>
Red Hat Enterprise Linux 8. Any Cloud. Any Workload. One OS.
<https://www.redhat.com/en/enterprise-linux-8>
<https://red.ht/sig>
1 year, 7 months
PAM troubleshooting
by Rob Crittenden
I'm trying to get a multi-threaded PAM app working to log in users using
the 'login' service to generate client load:
https://github.com/freeipa/freeipa-perftest/blob/master/src/pamtest.c
The load is generated by running multiple client VMs and on each client
running the pam tester with multiple threads.
It works about 99% of the time and I'm struggling to find out why it
occasionally fails with: Authentication failure (7)
The environment is IPA with pre-created users all set with the same
known LDAP and Kerberos password. The password is not marked as expired.
The journal is reporting a different PAM error:
Jan 24 16:11:57 client003.ipa.test pamtest[4285]: pam_sss(login:auth):
authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
user=user19client003.ipa.test
Jan 24 16:11:57 client003.ipa.test pamtest[4285]: pam_sss(login:auth):
received for user user19client003.ipa.test: 6 (Permission denied)
Jan 24 16:11:59 client003.ipa.test pamtest[4285]:
pam_unix(login:session): session opened for user
user19client003.ipa.test(uid=1751200171) by (uid=0)
Jan 24 16:14:01 client003.ipa.test pamtest[4285]:
pam_unix(login:session): session closed for user user19client003.ipa.test
The sssd pam log reports:
* (2022-01-24 16:11:50): [pam] [pam_cmd_authenticate] (0x0100):
entering pam_cmd_authenticate
* (2022-01-24 16:11:50): [pam] [sss_domain_get_state] (0x1000):
Domain ipa.test is Active
[ short snip of parse_name ]
* (2022-01-24 16:11:50): [pam] [pd_set_primary_name] (0x0400):
User's primary name is user19client003.ipa.test(a)ipa.test
* (2022-01-24 16:11:50): [pam] [pam_initgr_check_timeout] (0x2000):
User [user19client003.ipa.test] found in PAM cache.
* (2022-01-24 16:11:50): [pam] [pam_dp_send_req] (0x0100): Sending
request [CID #2] with the following data:
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
command: SSS_PAM_AUTHENTICATE
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
domain: ipa.test
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
user: user19client003.ipa.test(a)ipa.test
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
service: login
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
tty: not set
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
ruser: not set
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
rhost: not set
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
authtok type: 1 (Password)
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
newauthtok type: 0 (No authentication token available)
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
priv: 1
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
cli_pid: 4285
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
logon name: user19client003.ipa.test
* (2022-01-24 16:11:50): [pam] [pam_print_data] (0x0100): [CID #2]
flags: 2
* (2022-01-24 16:11:50): [pam] [pam_dom_forwarder] (0x0100):
pam_dp_send_req returned 0
* (2022-01-24 16:11:53): [pam] [pam_initgr_cache_remove] (0x2000):
[user20client003.ipa.test] removed from PAM initgroup cache
* (2022-01-24 16:11:55): [pam] [pam_initgr_cache_remove] (0x2000):
[user19client003.ipa.test] removed from PAM initgroup cache
* (2022-01-24 16:11:57): [pam] [sbus_dispatch] (0x4000): Dispatching.
* (2022-01-24 16:11:57): [pam] [pam_dp_send_req_done] (0x0200):
received: [6 (Permission denied)][ipa.test][CID #2]
* (2022-01-24 16:11:57): [pam] [pam_reply] (0x4000): pam_reply
initially called with result [6]: Permission denied. this result might
be changed during processing
I just can't tell what the permission denied means. Permission to do
what? I can provide fuller logs if needed.
I suspect it may be a timeout as there is a several second delay in some
of the processing.
There is nothing wrong with this particular user. I can manually re-run
the tool and it will pass 100% but the server is under significantly
less stress.
Hoping for suggestions on where to start looking.
thanks
rob
1 year, 7 months
[SSSD] Announcing SSSD 2.6.3
by Pavel Březina
# SSSD 2.6.3
The SSSD team is proud to announce the release of version 2.6.3 of the
System Security Services Daemon. The tarball can be downloaded from:
https://github.com/SSSD/sssd/releases/tag/2.6.3
See the full release notes at:
https://sssd.io/release-notes/sssd-2.6.3.html
RPM packages will be made available for Fedora shortly.
## Feedback
Please provide comments, bugs and other feedback via the sssd-devel
or sssd-users mailing lists:
https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
https://lists.fedorahosted.org/mailman/listinfo/sssd-users
## Highlights
### Important fixes
* A regression introduced in sssd-2.6.2 in the IPA provider that
prevented users from login was fixed. Access control always denied
access because the selinux_child returned an unexpected reply.
* A critical regression that prevented authentication of users via AD
and IPA providers was fixed. LDAP port was reused for Kerberos
communication and this provider would send incomprehensible information
to this port.
* When authenticating AD users, backtrace was triggered even though
everything was working correctly. This was caused by a search in the
global catalog. Servers from the global catalog are filtered out of the
list before writing the KDC info file. With this fix, SSSD does not
attempt to write to the KDC info file when performing a GC lookup.
1 year, 8 months
kvon in keytab is getting out of sync
by Sebastian Grebe
Hello,
we are getting report from users where they suddenly can‘t authenticate to their Linux computers anymore. These computers are joint to ore MS Domain using adcli und sssd. Checking the log reveals that the kerberos tickets stored in /etc/krb5.keytab do not have the expected KVON. At the moment we can’t tell what’s causing the issue. It happens only sporadically. I’m under the impression only computer without permanent network connection (Laptops) are affected.
The log shows:
Jan 11 09:30:52 lc015564 systemd[1]: Starting System Security Services Daemon...
Jan 11 09:30:52 lc015564 sssd[1376]: Starting up
Jan 11 09:30:52 lc015564 sssd_be[1609]: Starting up
Jan 11 09:30:52 lc015564 sssd_ifp[1633]: Starting up
Jan 11 09:30:52 lc015564 systemd[1]: Started System Security Services Daemon.
Jan 11 09:30:55 lc015564 sssd_be[1609]: Backend is offline
Jan 11 09:49:32 lc015564 sssd_be[1609]: Backend is online
Jan 11 09:49:41 lc015564 krb5_child[6111]: Cannot find key for LC015564$(a)WAGO.LOCAL kvno 11 in keytab
Jan 11 09:49:41 lc015564 krb5_child[6111]: Cannot find key for LC015564$(a)WAGO.LOCAL kvno 11 in keytab
Jan 11 09:49:49 lc015564 adcli[6102]: GSSAPI client step 1
Jan 11 09:49:49 lc015564 adcli[6102]: GSSAPI client step 1
Jan 11 09:49:50 lc015564 adcli[6102]: GSSAPI client step 1
Jan 11 10:00:57 lc015564 krb5_child[6838]: Cannot find key for LC015564$(a)WAGO.LOCAL kvno 11 in keytab
Jan 11 10:00:57 lc015564 krb5_child[6838]: Cannot find key for LC015564$(a)WAGO.LOCAL kvno 11 in keytab
And klist -k shows:
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
10 LC015564$(a)WAGO.LOCAL
10 LC015564$(a)WAGO.LOCAL
10 LC015564$(a)WAGO.LOCAL
10 host/LC015564(a)WAGO.LOCAL
10 host/LC015564(a)WAGO.LOCAL
10 host/LC015564(a)WAGO.LOCAL
10 host/lc015564.wago.local(a)WAGO.LOCAL
10 host/lc015564.wago.local(a)WAGO.LOCAL
10 host/lc015564.wago.local(a)WAGO.LOCAL
10 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
10 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
10 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
10 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
10 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
10 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
9 LC015564$(a)WAGO.LOCAL
9 LC015564$(a)WAGO.LOCAL
9 LC015564$(a)WAGO.LOCAL
9 host/LC015564(a)WAGO.LOCAL
9 host/LC015564(a)WAGO.LOCAL
9 host/LC015564(a)WAGO.LOCAL
9 host/lc015564.wago.local(a)WAGO.LOCAL
9 host/lc015564.wago.local(a)WAGO.LOCAL
9 host/lc015564.wago.local(a)WAGO.LOCAL
9 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
9 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
9 RestrictedKrbHost/LC015564(a)WAGO.LOCAL
9 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
9 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
9 RestrictedKrbHost/lc015564.wago.local(a)WAGO.LOCAL
This is a our sssd.conf (it's from o different computer):
[sssd]
domains = wago.local
config_file_version = 2
services = ifp
[domain/wago.local]
default_shell = /bin/bash
fallback_homedir = /home/%d/%u
cache_credentials = true
krb5_store_password_if_offline = true
krb5_realm = WAGO.LOCAL
krb5_ccname_template = /tmp/krb5cc_%U
realmd_tags = manages-system joined-with-adcli
id_provider = ad
access_provider = ad
ad_domain = wago.local
ad_enabled_domains = wago.local
ad_hostname = lc017547.wago.local
use_fully_qualified_names = false
ldap_id_mapping = true
ldap_user_gecos = displayName
ldap_use_tokengroups = false
ldap_search_base = dc=wago,dc=local?subtree?
ldap_user_search_base = ou=User,ou=Minden,ou=Germany,dc=wago,dc=local?subtree??ou=User,ou=Administration,dc=wago,dc=local?onelevel?(&(objectClass=user)(cn=a2*))?ou=Service,dc=wago,dc=local?subtree?
ldap_group_search_base = cn=Users,dc=wago,dc=local?onelevel?(&(objectClass=group)(cn=Domain Users))?ou=Groups,ou=Minden,ou=Germany,dc=wago,dc=local?onelevel?(&(objectClass=group)(cn=&01-PC-Support))
ldap_netgroup_search_base = cn=Users,dc=wago,dc=local?onelevel?
ignore_group_members = true
enumerate = false
dyndns_update = true
dyndns_refresh_interval = 7200
dyndns_update_ptr = true
dyndns_server = 10.1.100.2
case_sensitive = Preserving
[nss]
filter_users = root
filter_groups = root
[pam]
offline_credentials_expiration = 0
offline_failed_login_attempts = 3
offline_failed_login_delay = 5
And the krb5.conf:
[libdefaults]
ticket_lifetime = 240:00:00
renew_lifetime = 240:00:00
clock_skew = 300
renewable = true
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
default_realm = WAGO.LOCAL
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
udp_preference_limit = 1
noaddresses = true
fcc-mit-ticketflags = true
[realms]
WAGO.LOCAL = {
admin_server = 10.1.101.200
admin_server = 10.1.100.1
admin_server = 10.1.100.253
admin_server = 10.1.100.2
}
[domain_realm]
.wago.local = WAGO.LOCAL
wago.local = WAGO.LOCAL
[login]
krb4_convert = true
krb4_get_tickets = false
To solve the issue we delete the computer from the domain, delete the krb5.keytab and rejoin them.
1 year, 8 months
sssd-1.16.5-10.0.1.el7_9.11.x86_64 is under-discovering AD domains
by Spike White
sssd experts,
This sssd version (released Tue 23 Nov 2021) is under-discovering AD
domains.
A similar sssd bug occurred last July, where sssd over-discovered AD
domains (AD domains for which there was not a legal trust relationship with
this AD domain.) Now, it appears that sssd is under-discovering AD domains
(not discovering AD domains which have a valid trust relationship with this
AD domain).
Ultimately, an sssd developer on this sssd mailing list (Sumit Rose)
resolved the July bug of AD domain over-discovery. Hopefully, someone
will recognize this new AD domain under-discovery bug.
We have opened RedHat support case #03124778 for this new AD domain
under-discovery bug. But to be frank, such complicated sssd bugs don’t get
resolved by RHEL L1 customer support. (RHEL customer support is great, but
not for complicated sssd bugs.)
For now, we have pulled sssd-*-1.16.5-10.0.1.el7_9.11.x86_64 RPMs out of
our Jan OS patching cycle.
Spike
1 year, 8 months
Trouble resolving a AD group on one machine
by Orion Poplawski
We have a particular machine that is having trouble resolving an AD group -
"domain admins". The relevant log entries seem to be:
(2021-12-29 13:40:17): [nss] [cache_req_search_cache] (0x0400): CR #152:
Looking up [domain admins(a)ad.nwra.com] in cache
(2021-12-29 13:40:17): [nss] [sysdb_search_override_by_name] (0x0400): No user
override found for name [domain admins(a)ad.nwra.com].
(2021-12-29 13:40:17): [nss] [sysdb_getgrnam_with_views] (0x4000): Group
object [name=domain admins(a)ad.nwra.com,cn=groups,cn=ad.nwra.com,cn=sysdb],
contains ghost entries which must be resolved before overrides can be applied.
(2021-12-29 13:40:17): [nss] [sysdb_getgrnam_with_views] (0x4000): Returning
empty result.
(2021-12-29 13:40:17): [nss] [cache_req_search_cache] (0x0400): CR #152:
Object [domain admins(a)ad.nwra.com] was not found in cache
(2021-12-29 13:40:17): [nss] [cache_req_search_ncache_add_to_domain] (0x0400):
CR #152: Adding [domain admins(a)ad.nwra.com] to negative cache
(2021-12-29 13:40:17): [nss] [sss_ncache_set_str] (0x0400): Adding
[NCE/GROUP/ad.nwra.com/domain admins(a)ad.nwra.com] to negative cache
(2021-12-29 13:40:17): [nss] [cache_req_process_result] (0x0400): CR #152:
Finished: Not found
(2021-12-29 13:40:17): [nss] [sss_domain_get_state] (0x1000): Domain
ad.nwra.com is Active
(2021-12-29 13:40:17): [nss] [nss_protocol_done] (0x4000): Sending reply: not
found
on working systems we don't have the sysdb_getgrnam_with_views message. I'd
rather not clear the sssd database. Is there anything else that can be done?
'sss_cache -g "domain admins"' does not help.
We're using an IPA <-> AD trust.
Thanks
--
Orion Poplawski
IT Systems Manager 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
1 year, 8 months
