URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: opened
PR body: """ If the files managed by the files provider contain many users or groups processing them might take a considerable amount of time. To keep the backend responsive this patch splits the update into multiple steps running one after the other but returning to the main loop in between.
This avoids issues during startup because the watchdog timer state is reset properly. Additionally SBUS messages are process and as a result the domain can be marked inconsistent in the frontends properly. """
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
I didn't include changes with respect to use talloc-pools in this PR because currently I cannot reproduce the original performance improvements I have seen. I will investigate this further.
I think https://github.com/SSSD/sssd/issues/5091 is fixed as a side effect of this patch but so far I haven't found a proper ticket for this PR. @alexey-tikhonov, do you have a suggestion or shall I create a new ticket for the batch processing?
As you can see there are some delays and counts hardcoded which can easily made configurable. But so far I think there is no need for configuration options.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-806570686
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ So far when we talked about "batches" I thought we meant "accumulating few consequent update events and processing in a single 'batch' (once)" - to avoid an issue in a case user populates very large /etc/passwd with a script.
But this PR aims to fix another valid issue - single operation during single update taking > 3 * WD `timeout`s. I think we don't have a ticket for this issue. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-806632204
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
So far when we talked about "batches" I thought we meant "accumulating few consequent update events and processing in a single 'batch' (once)" - to avoid an issue in a case user populates very large /etc/passwd with a script.
Hi,
I'm sorry for not being more specific here. I meant `batch` in the sense we use it in the refresh code.
In the patch there is an initiial delay, which can be used to wait some time for more changes to the files before doing the real update.
But this PR aims to fix another valid issue - single operation during single update taking > 3 * WD `timeout`s. I think we don't have a ticket for this issue.
Would you mind to open a suitable ticket and link it to related downstream tickets?
Thanks.
bye, Sumit
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-806726567
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
But this PR aims to fix another valid issue - single operation during single update taking > 3 * WD `timeout`s. I think we don't have a ticket for this issue.
Would you mind to open a suitable ticket and link it to related downstream tickets?
Hi,
#5557 """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-808552359
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
the latest version only updates the commit message.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-810403814
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Waiting for review
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: -Waiting for review
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Changes requested
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
pbrezina commented: """ What is the reason for this patches? To be able to deliver domain-inconsistent message? If that's the case you can send the message via monitor connection. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-811817675
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
What is the reason for this patches?
IIUC, to avoid single large blocking operation. see #5557
To be able to deliver domain-inconsistent message? If that's the case you can send the message via monitor connection.
What do you mean?
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-811824464
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
pbrezina commented: """
What is the reason for this patches?
IIUC, to avoid single large blocking operation. see #5557
IMHO in this case it would be better to do a diff update to improve performance first. It would also solve https://github.com/SSSD/sssd/issues/4614. It wouldn't of course solve the initial cache population so some batching is needed, but it will affect the design a lot therefore it might be better to start with it.
To be able to deliver domain-inconsistent message? If that's the case you can send the message via monitor connection.
What do you mean?
The reason why the message is not delivered is because we send it over `provider->sbus_conn` which requires accessing the backend's tevent loop because the backend runs this message bus. If `be_ctx->mon_conn` is used instead the message is sent through the monitor's message bus immediately.
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-811832977
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ ``` ============================================================================================================== [VANILIA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd] POWTORZYC BO SIE KOMPILOWALO W MIEDZYCZASIE!!!
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent.
Test environment is sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. With microseconds enabled in SSSD logs I was trying to estimate time needed to mark domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash for loop.
Test environment is sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
|Version|100 users creation|100 users deletion|250 users creation|250 users deletion|500 users creation|500 users deletion| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. With microseconds enabled in SSSD logs I was trying to estimate time needed to mark domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash for loop.
Test environment is sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
|Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. With microseconds enabled in SSSD logs I was trying to estimate time needed to mark domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop.
Test environment is sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
|Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. With microseconds enabled in SSSD logs I was trying to estimate time needed to mark domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
|Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens time needed to mark domain as inconsistent it is still affected by the size of `/etc/passwd` file. Another observation is that total time of adding / deleting N users has been increased by this PR.
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. With microseconds enabled in SSSD logs I was trying to estimate time needed to mark domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
|Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` to cache - Introduce 1st semi step which will send DBUS commands only even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling `useradd` / `userdel` in `for` loop `N` times: |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` to cache - Introduce 1st semi step which will send DBUS commands only even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` to cache - Introduce 1st semi step which will send DBUS commands only even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands only even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps to read file part by part
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps to read file part by part
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: `time for N in `seq X`; do useradd test-user-$N; done` `time for N in `seq X`; do userdel -rf test-user-$N; done` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps to read file part by part
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I did some testing comparing this PR with vanilla SSSD. I added extra log in `sf_passwd_cb()` which is `/etc/passwd` file modification callback and in `sss_resp_domain_inconsistent()` which is receiver of DBUS command `SetInconsisten` marking domain as inconsistent. Another test was creation and deletion of 100, 250 and 500 users in bash `for` loop. I have not run dedicated test for `/etc/groups` based routines as I thought result of `/etc/passwd` modifications will be already meaningful.
Test environment was sssd-test-suite VM (1 core CPU, 2G RAM) running Fedora 31. As a base I used upstream commit: 66960c76952baf6b636d4f1949822710a3dc7da8 On top of it I added my commit with extra logs and then this PR. Full log from test is on the bottom.
TL; DR;
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
Conclusion: While this PR shortens the time needed to mark a domain as inconsistent it is still affected by the size of `/etc/passwd` file. The issues reported for files provider with 1k+ users may be solved now but they will be still valid for 10k+ etc. Another observation is that total time of adding / deleting N users via bash script has been increased by this PR. I guess playing with `state->initial_delay` and `state->delay` could help fix this. Positive observation is drop in CPU usage. Load on CPU for 500 users adding / deleting was still heavy but not that heavy as in vanilla version and better balanced.
To speed up domain marking as inconsistent there are three options: - As @pbrezina suggested use monitor DBUS connector to send `SetInconsistent` - Introduce 1st semi step which will send DBUS commands even before `/etc/passwd` / `/etc/groups` processing to release backend tevent loop faster and push DBUS commands - Split 1st step which is file read into even smaller substeps to read file part by part
``` ============================================================================================================== [VANILLA SSSD] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1715:(2021-04-03 14:15:23:911881): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:215:(2021-04-03 14:15:24:007537): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 911881 + 007537 = 95656
./sssd_files.log:2430:(2021-04-03 14:16:42:521671): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:243:(2021-04-03 14:16:42:610820): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 610820 - 521671 = 89149
./sssd_files.log:3145:(2021-04-03 14:17:00:105384): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:271:(2021-04-03 14:17:00:193427): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 193427 - 105384 = 88043
Average = (95656 + 89149 + 88043) / 3 = 90949us = 90,949ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m43.464s user 0m7.187s sys 0m3.635s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5125:(2021-04-03 14:27:10:556898): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:260:(2021-04-03 14:27:10:750840): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 750840 - 556898 = 193942
./sssd_files.log:7540:(2021-04-03 14:27:41:406064): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:288:(2021-04-03 14:27:41:600166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 600166 - 406064 = 194102
./sssd_files.log:9955:(2021-04-03 14:27:58:037802): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:316:(2021-04-03 14:27:58:221111): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221111 - 037802 = 183309
Average = (193942 + 194102 + 183309) / 3 = 190451us = 190,451ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m41.084s user 0m6.838s sys 0m3.729s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 2m46.462s user 0m34.869s sys 0m18.381s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10205:(2021-04-03 18:04:14:997171): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:161:(2021-04-03 18:04:15:422140): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 997171 + 422140 = 424969
./sssd_files.log:15170:(2021-04-03 18:04:38:941796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:189:(2021-04-03 18:04:39:387461): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 941796 + 387461 = 445665
./sssd_files.log:20135:(2021-04-03 18:04:56:860883): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:217:(2021-04-03 18:04:57:322166): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 860883 + 322166 = 461283
Average = (424969 + 445665 + 461283) / 3 = 443972us = 443,972ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 2m42.784s user 0m34.563s sys 0m17.203s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 9m55.823s user 2m15.133s sys 1m7.711s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passw" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:18705:(2021-04-03 19:08:04:485795): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:157:(2021-04-03 19:08:05:358181): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 485795 + 358181 = 872386
./sssd_files.log:27920:(2021-04-03 19:08:23:710157): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:185:(2021-04-03 19:08:24:608777): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 710157 + 608777 = 898620
./sssd_files.log:37135:(2021-04-03 19:08:50:350387): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:213:(2021-04-03 19:08:51:253827): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 350387 + 253827 = 903440
Average = (872386 + 898620 + 903440) / 3 = 891482us = 891,482ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 9m15.531s user 2m9.902s sys 1m2.880s
============================================================================================================== [SSSD FILES BATCHING] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1755:(2021-04-03 20:38:17:173733): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:263:(2021-04-03 20:38:17:208845): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 208845 - 173733 = 35112
./sssd_files.log:2485:(2021-04-03 20:38:39:735251): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:291:(2021-04-03 20:38:39:765298): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 765298 - 735251 = 30047
./sssd_files.log:3215:(2021-04-03 20:38:58:006912): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:319:(2021-04-03 20:38:58:042773): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 042773 - 006912 = 35861
Average = (35112 + 30047 + 35861) / 3 = 33673us = 33,673ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m53.304s user 0m6.711s sys 0m3.675s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5215:(2021-04-03 20:46:34:054796): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:265:(2021-04-03 20:46:34:113920): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 113920 - 054796 = 59124
./sssd_files.log:7675:(2021-04-03 20:47:06:303398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:293:(2021-04-03 20:47:06:369933): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 369933 - 303398 = 66535
./sssd_files.log:10135:(2021-04-03 20:47:21:847043): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:321:(2021-04-03 20:47:21:910637): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 910637 - 847043 = 63594
Average = (59124 + 66535 + 63594) / 3 = 63084us = 63,084ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m48.572s user 0m6.181s sys 0m3.636s -------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m42.061s user 0m29.853s sys 0m15.093s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10385:(2021-04-03 20:57:13:771709): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:168:(2021-04-03 20:57:13:886122): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 886122 - 771709 = 114413
./sssd_files.log:15440:(2021-04-03 20:57:32:787196): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:196:(2021-04-03 20:57:32:888939): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 888939 - 787196 = 101743
./sssd_files.log:20495:(2021-04-03 20:57:49:476424): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:224:(2021-04-03 20:57:49:576838): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 576838 - 476424 = 100414
Average = (114413 + 101743 + 100414) / 3 = 105523us = 105,523ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 4m16.405s user 0m28.943s sys 0m14.883s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 20m15.965s user 2m20.204s sys 1m9.866s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19035:(2021-04-03 21:43:11:230076): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-03 21:43:11:438640): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 438640 - 230076 = 208564
./sssd_files.log:28415:(2021-04-03 21:43:25:944105): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-03 21:43:26:106063): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 1000000 - 944105 + 106063 = 161958
./sssd_files.log:37795:(2021-04-03 21:44:08:229897): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-03 21:44:08:424929): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 424929 - 229897 = 195032
Average = (208564 + 161958 + 195032) / 3 = 188518us = 188,518ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 18m1.768s user 1m53.758s sys 0m59.567s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-812930403
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
Version 100 useradd 100 userdel 250 useradd 250 userdel 500 useradd 500 userdel Vanilla 0m43.464s 0m41.084s 2m46.462s 2m46.462s 9m55.823s 9m15.531s Batching 0m53.304s 0m48.572s 3m42.061s 4m16.405s 20m15.965s 18m1.768s
Why this change isn't linear? 100 useradd: +23%, 500 useradd: +102%
9m -> 20m is worrying. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814023198
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
Version 100 useradd 100 userdel 250 useradd 250 userdel 500 useradd 500 userdel Vanilla 0m43.464s 0m41.084s 2m46.462s 2m46.462s 9m55.823s 9m15.531s Batching 0m53.304s 0m48.572s 3m42.061s 4m16.405s 20m15.965s 18m1.768s
Why this change isn't linear? 100 useradd: +23%, 500 useradd: +102%
9m -> 20m is worrying.
To be precise it was 30 + 100 users and 30 + 500 users. I was using default 30 /etc/passwd entries as sort of zero - reference.
9m -> 20m is worrying.
Please take a look at this comment: https://github.com/SSSD/sssd/pull/5552#discussion_r605794536 I think when I was spamming SSSD with useradd / userdell commands end result was multiple batching tasks running in parallel, where each one was doing some database operations. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814033099
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ ``` ============================================================================================================== [BATCH WITH ZERO STEP] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1743:(2021-04-06 10:27:35:422398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 10:27:35:423142): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 423142 - 422398 = 744
./sssd_files.log:2477:(2021-04-06 10:28:02:358816): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 10:28:02:360463): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 360463 - 358816 = 1647
./sssd_files.log:3211:(2021-04-06 10:28:21:213798): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 10:28:21:214862): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 214862 - 213798 = 1064
Average = (744 + 1647 + 1064) / 3 = 1152us = 1,152ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m56.830s user 0m6.624s sys 0m3.577s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5203:(2021-04-06 10:31:42:194803): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:160:(2021-04-06 10:31:42:195815): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 195815 - 194803 = 1012
./sssd_files.log:7667:(2021-04-06 10:32:03:338062): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:188:(2021-04-06 10:32:03:339720): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 339720 - 338062 = 1658
./sssd_files.log:10131:(2021-04-06 10:32:15:219538): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:216:(2021-04-06 10:32:15:221051): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221051 - 219538 = 1513
Average = (1012 + 1658 + 1513) / 3 = 1394us = 1,394ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m53.676s user 0m6.299s sys 0m3.504s
-------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m48.812s user 0m28.150s sys 0m14.616s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10393:(2021-04-06 10:40:16:284891): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:164:(2021-04-06 10:40:16:285288): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 285288 - 284891 = 397
./sssd_files.log:15452:(2021-04-06 10:41:54:614785): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:192:(2021-04-06 10:41:54:616448): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 616448 - 614785 = 1663
./sssd_files.log:20511:(2021-04-06 10:42:14:447177): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:220:(2021-04-06 10:42:14:447551): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 447551 - 447177 = 374
Average = (397 + 1663 + 374) / 3 = 811us = 0,811ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 3m46.155s user 0m27.973s sys 0m14.632s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 15m27.024s user 1m48.542s sys 0m55.511s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19043:(2021-04-06 11:07:34:234339): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 11:07:34:234736): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 234736 - 234339 = 397
./sssd_files.log:28427:(2021-04-06 11:07:49:242399): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 11:07:49:244265): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 244265 - 242399 = 1866
./sssd_files.log:37811:(2021-04-06 11:08:04:169746): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 11:08:04:170919): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 170919 - 169746 = 1173
Average = (397 + 1866 + 1173) / 3 = 1145us = 1,145ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 14m56.387s user 1m47.032s sys 0m55.798s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814043592
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I modified this PR by adding initial zero step which only starts database transaction and returns to tevent loop: https://github.com/elkoniu/sssd/commit/8bb65dc777d333f146b30383759aea9b09485... End results is that DBUS commands can be send before "heavy" calculations starts.
Please take a look at updated test results (previous test result included):
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms| |Zero step|1,1ms|1,4ms|0,8ms|1,1ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s| |Zero step|0m56.830s|0m53.676s|3m48.812s|3m46.155s|15m27.024s|14m56.387s|
Conclusion: With dummy zero steep this PR will be able to solve https://github.com/SSSD/sssd/issues/5091 by marking domain as inconsistent fast enough.
``` ============================================================================================================== [BATCH WITH ZERO STEP] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1743:(2021-04-06 10:27:35:422398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 10:27:35:423142): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 423142 - 422398 = 744
./sssd_files.log:2477:(2021-04-06 10:28:02:358816): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 10:28:02:360463): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 360463 - 358816 = 1647
./sssd_files.log:3211:(2021-04-06 10:28:21:213798): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 10:28:21:214862): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 214862 - 213798 = 1064
Average = (744 + 1647 + 1064) / 3 = 1152us = 1,152ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m56.830s user 0m6.624s sys 0m3.577s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5203:(2021-04-06 10:31:42:194803): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:160:(2021-04-06 10:31:42:195815): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 195815 - 194803 = 1012
./sssd_files.log:7667:(2021-04-06 10:32:03:338062): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:188:(2021-04-06 10:32:03:339720): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 339720 - 338062 = 1658
./sssd_files.log:10131:(2021-04-06 10:32:15:219538): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:216:(2021-04-06 10:32:15:221051): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221051 - 219538 = 1513
Average = (1012 + 1658 + 1513) / 3 = 1394us = 1,394ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m53.676s user 0m6.299s sys 0m3.504s
-------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m48.812s user 0m28.150s sys 0m14.616s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10393:(2021-04-06 10:40:16:284891): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:164:(2021-04-06 10:40:16:285288): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 285288 - 284891 = 397
./sssd_files.log:15452:(2021-04-06 10:41:54:614785): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:192:(2021-04-06 10:41:54:616448): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 616448 - 614785 = 1663
./sssd_files.log:20511:(2021-04-06 10:42:14:447177): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:220:(2021-04-06 10:42:14:447551): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 447551 - 447177 = 374
Average = (397 + 1663 + 374) / 3 = 811us = 0,811ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 3m46.155s user 0m27.973s sys 0m14.632s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 15m27.024s user 1m48.542s sys 0m55.511s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19043:(2021-04-06 11:07:34:234339): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 11:07:34:234736): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 234736 - 234339 = 397
./sssd_files.log:28427:(2021-04-06 11:07:49:242399): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 11:07:49:244265): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 244265 - 242399 = 1866
./sssd_files.log:37811:(2021-04-06 11:08:04:169746): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 11:08:04:170919): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 170919 - 169746 = 1173
Average = (397 + 1866 + 1173) / 3 = 1145us = 1,145ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 14m56.387s user 1m47.032s sys 0m55.798s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814043592
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
Version 100 useradd 100 userdel 250 useradd 250 userdel 500 useradd 500 userdel Vanilla 0m43.464s 0m41.084s 2m46.462s 2m46.462s 9m55.823s 9m15.531s Batching 0m53.304s 0m48.572s 3m42.061s 4m16.405s 20m15.965s 18m1.768s
Why this change isn't constant? 100 useradd: +23%, 500 useradd: +102%
9m -> 20m is worrying. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814023198
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
The reason why the message is not delivered is because we send it over `provider->sbus_conn` which requires accessing the backend's tevent loop because the backend runs this message bus. If `be_ctx->mon_conn` is used instead the message is sent through the monitor's message bus immediately.
Hi @pbrezina,
this sounds like the proper solution for #5091. There were discussions about dropping the monitor and relying on systemd to start the SSSD components. I do not remember the current state of those discussions but if we ever want to move that way this would be another task of the monitor which must be replaced.
About @elkoniu's suggestion to wait with the processing of the files until the messages are send. This can be easily combined with @alexey-tikhonov 's original understanding of batches where we should not immediately start processing if we are notified of a change but wait for e.g. 1s of inactivity on the files and start processing. With this we should be long enough in the event loop to properly finish the communication with the NSS responder, but I guess you would still recommend to send the message via the monitor?
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814828582
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
What is the reason for this patches?
IIUC, to avoid single large blocking operation. see #5557
IMHO in this case it would be better to do a diff update to improve performance first. It would also solve #4614. It wouldn't of course solve the initial cache population so some batching is needed, but it will affect the design a lot therefore it might be better to start with it.
Hi @pbrezina,
where do you see the difficulties to add a diff approach? The step the remove the cache has to be replaced by the diff step and then the entries which changed will be processed. Even with the diff approach we might need process the changes in batches if there are many to not block the backend for too long.
bye, Sumit
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814832590
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I modified this PR by adding initial zero step which only starts database transaction and returns to tevent loop: https://github.com/elkoniu/sssd/tree/file-batching-zero-step End results is that DBUS commands can be send before "heavy" calculations starts.
Please take a look at updated test results (previous test result included):
Estimated time needed to mark domain as inconsistent: |Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|91ms|190ms|444ms|891ms| |Batching|34ms|63ms|105ms|188ms| |Zero step|1,1ms|1,4ms|0,8ms|1,1ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s| |Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s| |Zero step|0m56.830s|0m53.676s|3m48.812s|3m46.155s|15m27.024s|14m56.387s|
Conclusion: With dummy zero steep this PR will be able to solve https://github.com/SSSD/sssd/issues/5091 by marking domain as inconsistent fast enough.
``` ============================================================================================================== [BATCH WITH ZERO STEP] ============================================================================================================== [30 entries in /etc/passwd]
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 30
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:1743:(2021-04-06 10:27:35:422398): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 10:27:35:423142): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 423142 - 422398 = 744
./sssd_files.log:2477:(2021-04-06 10:28:02:358816): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 10:28:02:360463): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 360463 - 358816 = 1647
./sssd_files.log:3211:(2021-04-06 10:28:21:213798): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 10:28:21:214862): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 214862 - 213798 = 1064
Average = (744 + 1647 + 1064) / 3 = 1152us = 1,152ms -------------------------------------------------------------------------------------------------------------- [30 + 100 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N; done
real 0m56.830s user 0m6.624s sys 0m3.577s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 130
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:5203:(2021-04-06 10:31:42:194803): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:160:(2021-04-06 10:31:42:195815): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 195815 - 194803 = 1012
./sssd_files.log:7667:(2021-04-06 10:32:03:338062): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:188:(2021-04-06 10:32:03:339720): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 339720 - 338062 = 1658
./sssd_files.log:10131:(2021-04-06 10:32:15:219538): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:216:(2021-04-06 10:32:15:221051): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 221051 - 219538 = 1513
Average = (1012 + 1658 + 1513) / 3 = 1394us = 1,394ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf test-user-$N; done
real 0m53.676s user 0m6.299s sys 0m3.504s
-------------------------------------------------------------------------------------------------------------- [30 + 250 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N; done
real 3m48.812s user 0m28.150s sys 0m14.616s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 280
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:10393:(2021-04-06 10:40:16:284891): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:164:(2021-04-06 10:40:16:285288): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 285288 - 284891 = 397
./sssd_files.log:15452:(2021-04-06 10:41:54:614785): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:192:(2021-04-06 10:41:54:616448): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 616448 - 614785 = 1663
./sssd_files.log:20511:(2021-04-06 10:42:14:447177): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:220:(2021-04-06 10:42:14:447551): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 447551 - 447177 = 374
Average = (397 + 1663 + 374) / 3 = 811us = 0,811ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf test-user-$N; done
real 3m46.155s user 0m27.973s sys 0m14.632s -------------------------------------------------------------------------------------------------------------- [30 + 500 entries in /etc/passwd]
# Adding users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N; done
real 15m27.024s user 1m48.542s sys 0m55.511s
# Counting users in passwd [root@master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l 530
# Cache cleaning and daemon restart [root@master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf /var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions 3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation ./sssd_files.log:19043:(2021-04-06 11:07:34:234339): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:156:(2021-04-06 11:07:34:234736): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 234736 - 234339 = 397
./sssd_files.log:28427:(2021-04-06 11:07:49:242399): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:184:(2021-04-06 11:07:49:244265): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 244265 - 242399 = 1866
./sssd_files.log:37811:(2021-04-06 11:08:04:169746): [be[files]] [sf_passwd_cb] (0x0200): >>>PASSWD_MODIFIED_CB ./sssd_nss.log:212:(2021-04-06 11:08:04:170919): [nss] [sss_resp_domain_inconsistent] (0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED 170919 - 169746 = 1173
Average = (397 + 1866 + 1173) / 3 = 1145us = 1,145ms
# Deleting users [root@master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf test-user-$N; done
real 14m56.387s user 1m47.032s sys 0m55.798s ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814043592
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
pbrezina commented: """
The reason why the message is not delivered is because we send it over `provider->sbus_conn` which requires accessing the backend's tevent loop because the backend runs this message bus. If `be_ctx->mon_conn` is used instead the message is sent through the monitor's message bus immediately.
Hi @pbrezina,
this sounds like the proper solution for #5091. There were discussions about dropping the monitor and relying on systemd to start the SSSD components. I do not remember the current state of those discussions but if we ever want to move that way this would be another task of the monitor which must be replaced.
Ideally, we want to spawn private session of dbus-broker and use only one private message bus instead of running multiple. But so far we can use the monitor for situations like this.
About transition to systemd, I'm all for it. We already rely on systemd at some places anyway so I think it is time to finally drop the monitor. But lets not discuss it here.
About @elkoniu's suggestion to wait with the processing of the files until the messages are send. This can be easily combined with @alexey-tikhonov 's original understanding of batches where we should not immediately start processing if we are notified of a change but wait for e.g. 1s of inactivity on the files and start processing. With this we should be long enough in the event loop to properly finish the communication with the NSS responder, but I guess you would still recommend to send the message via the monitor?
If the processing is delayed then the provider gets to the loop and sends the message so monitor is not needed in this case.
where do you see the difficulties to add a diff approach? The step the remove the cache has to be replaced by the diff step and then the entries which changed will be processed. Even with the diff approach we might need process the changes in batches if there are many to not block the backend for too long.
Let me ask a question first - if the domain is correctly marked as inconsistent, why do we need to have the files backend responsive?
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814906355
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
About transition to systemd, I'm all for it. We already rely on systemd at some places anyway so I think it is time to finally drop the monitor. But lets not discuss it here.
If the processing is delayed then the provider gets to the loop and sends the message so monitor is not needed in this case.
Can we agree to fix https://github.com/SSSD/sssd/issues/5091 as a part of this PR with this zero-step approach rather than using monitor message bus? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-814934160
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
where do you see the difficulties to add a diff approach? The step the remove the cache has to be replaced by the diff step and then the entries which changed will be processed. Even with the diff approach we might need process the changes in batches if there are many to not block the backend for too long.
Let me ask a question first - if the domain is correctly marked as inconsistent, why do we need to have the files backend responsive?
Hi,
I meant "block" or "responsive" with respect to the watchdog. To avoid that the backend is killed by the watchdog we should make sure to return to the main loop regularly when know that processing the data needs more time.
bye, Sumit
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-815471181
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
pbrezina commented: """ Ok, feel free to continue with this PR. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-818665556
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ @sumit-bose Maybe temporary we should convert this PR to [WIP] state? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-843631952
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
in the latest version I added a patch which implements a fallback to the next nss module in case the files provider backend is busy with updating the data and a patch which should make sure that only one refresh is run at a time.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-852841159
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: -Changes requested
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Waiting for review
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ Debian: ``` FAIL test_files_provider.py::test_getgrnam_members_groups_first FAIL test_files_provider.py::test_getgrnam_user_member_and_ghost FAIL test_files_provider.py::test_realloc_groups_exact FAIL test_files_provider.py::test_multiple_files_created_after_startup ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-862326611
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ Debian: ``` FAIL test_files_provider.py::test_getgrnam_members_groups_first FAIL test_files_provider.py::test_getgrnam_user_member_and_ghost FAIL test_files_provider.py::test_realloc_groups_exact FAIL test_files_provider.py::test_multiple_files_created_after_startup ```
F33, Rawhide, RHEL8: ``` FAILED test_files_provider.py::test_realloc_groups_exact - assert 0 == 1 FAILED test_files_provider.py::test_realloc_groups - assert 0 == 1 ```
F32: ``` FAILED test_files_provider.py::test_getgrnam_members_users_first - assert 0 == 1 FAILED test_files_provider.py::test_getgrnam_members_groups_first - assert 0 ... FAILED test_files_provider.py::test_realloc_groups_exact - assert 0 == 1 ```
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-862326611
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ Perhaps tests expectations are wrong now and has to be adjusted?
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-862329878
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
in the latest version a copy-and-paste error which caused one test failure is fixed and the new `fallback_to_nss = False` option is added to more tests which do not have a fallback.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-862931453
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
Estimated time needed to mark domain as inconsistent: |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:|:---:| |Vanilla|103ms|161ms|124ms| |Batching|0,4ms|0,9ms|0,4ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|1m41.923s|2m4.702s|6m36.025s|7m8.865s|19m17.027s|18m52.852s| |Batching|2m57.306s|2m54.875s|9m13.329s|8m52.207s|24m13.363s|24m13.268s|
Result: - marking domain as inconsistent is instant which solves https://github.com/SSSD/sssd/issues/5091 - time needed for add / delete users in bash loop has increased by minimum 25%
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-863841885
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
Estimated time needed to mark domain as inconsistent: |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |Vanilla|103ms|161ms|124ms| |Batching|0,4ms|0,9ms|0,4ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|1m41.923s|2m4.702s|6m36.025s|7m8.865s|19m17.027s|18m52.852s| |Batching|2m57.306s|2m54.875s|9m13.329s|8m52.207s|24m13.363s|24m13.268s|
Result: - marking domain as inconsistent is instant which solves https://github.com/SSSD/sssd/issues/5091 - time needed for add / delete users in bash loop has increased by minimum 25%
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-863841885
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
Estimated time needed to mark domain as inconsistent: |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |Vanilla|103ms|161ms|124ms| |Batching|0,4ms|0,9ms|0,4ms|
Time consumed by calling operations on `X` users: ``` time for N in `seq X`; do useradd test-user-$N; done time for N in `seq X`; do userdel -rf test-user-$N; done ``` |Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Vanilla|1m41.923s|2m4.702s|6m36.025s|7m8.865s|19m17.027s|18m52.852s| |Batching|2m57.306s|2m54.875s|9m13.329s|8m52.207s|24m13.363s|24m13.268s|
Result: - marking domain as inconsistent is instant which solves https://github.com/SSSD/sssd/issues/5091 - time needed for add / delete users in bash loop has increased by minimum 25%
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-863841885
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
* time needed for add / delete users in bash loop has increased by minimum 25%
This is quite unexpected taking into account "files: delay refresh and not run in parallel" patch. The purpose of this patch to have a significant performance gain exactly in this scenario...
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-863857769
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Right now I am running 2nd set of tests to include raw files provider and CPU usage chart. Should have result in about 1h. Side note: if some of you recently used debug_microseconds option? I had impression that it stopped working.
W dniu pt., 18.06.2021 o 10:24 Alexey A Tikhonov ***@***.***> napisał(a):
- time needed for add / delete users in bash loop has increased by minimum 25%
This is quite unexpected taking into account "files: delay refresh and not run in parallel" patch. The purpose of this patch to have a significant performance gain exactly in this scenario...
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/SSSD/sssd/pull/5552#issuecomment-863857769, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARUPLCFY46VMIRNAJRVUB3TTL7DVANCNFSM4ZZES23A .
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Right now I am running 2nd set of tests to include raw files provider and CPU usage chart. Should have result in about 1h. Side note: if some of you recently used debug_microseconds option? I had impression that it stopped working.
W dniu pt., 18.06.2021 o 10:24 Alexey A Tikhonov ***@***.***> napisał(a):
- time needed for add / delete users in bash loop has increased by minimum 25%
This is quite unexpected taking into account "files: delay refresh and not run in parallel" patch. The purpose of this patch to have a significant performance gain exactly in this scenario...
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/SSSD/sssd/pull/5552#issuecomment-863857769, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARUPLCFY46VMIRNAJRVUB3TTL7DVANCNFSM4ZZES23A .
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
I run 3 tests with add/delete 100, 250 and 500 users in for loop: ``` for X in 100 250 500; do time for N in `seq $X`; do useradd test-user-$N; done time for N in `seq $X`; do userdel -rf test-user-$N; done done ```
Chart represents the result on common timeline.
[black] raw file provider (SSSD disabled) [green] latest SSSD master [orange] SSSD with batching from this PR
[Y scale] CPU usage (0 - 100%) [X scale] time (one measurement every 2s)
Result: - raw file provider is fastest and consumes up to 15% of CPU - without batching SSSD is heavy on CPU and hits up to 60% usage (orange chart) - batching version is the slowest one but CPU consumption is spreaded in time and in result is around 20% lower than vanilla SSSD (orange chart is less dense than green one) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864251686
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
I run 3 tests with add/delete 100, 250 and 500 users in for loop: ``` for X in 100 250 500; do time for N in `seq $X`; do useradd test-user-$N; done time for N in `seq $X`; do userdel -rf test-user-$N; done done ```
Chart represents the result on common timeline.
[black] raw file provider (SSSD disabled) [green] latest SSSD master [orange] SSSD with batching from this PR
[Y scale] CPU usage (0 - 100%) [X scale] time (one measurement every 2s)
If you count chart volume: black - 5901 green - 80714 orange - 63912 (-20% comparing to green)
Result: It is possible to see "waves" representing each user batch: 100, 250 and 500. 1) Raw file provider is fastest and consumes max 15% CPU in each run. 2) Vanilla SSSD is heaviest on CPU and hits 50% even for 100 users add / delete run 3) Batching version is the slowest time wise but about 20% less heavy on CPU than vanilla SSSD (based on chart volume) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864251686
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ What is "raw file provider"? libnss_files.so? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864258003
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
What is "raw file provider"? libnss_files.so?
This means `systemctl stop sssd` and set `nsswitch.conf` to: ``` ... passwd: files group: files ... ``` """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864258801
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
What is "raw file provider"? libnss_files.so?
This means `systemctl stop sssd` and set `nsswitch.conf` to: ``` ... passwd: files group: files ... ``` This way SSSD is not taking any part in processing `/etc/passwd` when useradd / userdel is used. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864258801
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
What is "raw file provider"? libnss_files.so?
This means `systemctl stop sssd` and set `nsswitch.conf` to: ``` ... passwd: files group: files ... ``` This way SSSD should not be involved in processing `/etc/passwd` when useradd / userdel is used. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864258801
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
Test setup: Fedora 32 virtual machine, 1 CPU, 2G RAM
I run 3 tests with add/delete 100, 250 and 500 users in for loop: ``` for X in 100 250 500; do time for N in `seq $X`; do useradd test-user-$N; done time for N in `seq $X`; do userdel -rf test-user-$N; done done ```
Chart represents the result on common timeline collected for each run by calling: ``` vmstat 2 ```
[black] raw file provider (SSSD disabled) [green] latest SSSD master [orange] SSSD with batching from this PR
[Y scale] CPU usage (0 - 100%) [X scale] time (one measurement every 2s)
If you count chart volume: black - 5901 green - 80714 orange - 63912 (-20% comparing to green)
Result: It is possible to see "waves" representing each user batch: 100, 250 and 500. 1) Raw file provider is fastest and consumes max 15% CPU in each run. 2) Vanilla SSSD is heaviest on CPU and hits 50% even for 100 users add / delete run 3) Batching version is the slowest time wise but about 20% less heavy on CPU than vanilla SSSD (based on chart volume) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864251686
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Based on @alexey-tikhonov comments I can run tests again with changes on `state->batch_size` and `state->delay`. @sumit-bose can you advise what else we should experiment with? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864266888
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
Based on @alexey-tikhonov comments I can run tests again with changes on `state->batch_size` and `state->delay`. @sumit-bose can you advise what else we should experiment with?
If `delay` is in microseconds, then I wouldn't touch it. This is already "as small as possible" (and in reality I guess actual delay is a couple of orders of magnitude higher).
But I would try batch size around 500+-. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864375190
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
thanks for testing and reviewing, yes `batch_size` should be larger, I think `1000` would be ok.
@elkoniu, do you, by chance, have a second domain in `sssd.conf` in your test environment?
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-864735696
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ``` 
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ``` |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
|Time needed to mark domain as inconsistent| |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent: |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent: |Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms| Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop: |Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files| |SSSD master| |SSSD batch_size = 10| |SSSD batch_size = 1000|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop: |Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m58.315s|1m2.276s|4m6.709s|4m2.857s|14m2.478s|14m54.913s| |SSSD master| |SSSD batch_size = 10| |SSSD batch_size = 1000|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop: |Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10| |SSSD batch_size = 1000|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop: |Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m37.513s|2m38.450s|8m19.219s|8m15.790s|23m49.023s|21m52.085s| |SSSD batch_size = 1000|2m33.065s|2m33.831s|7m43.810s|7m39.830s|20m13.238s|19m9.398s|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop:
|Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m37.513s|2m38.450s|8m19.219s|8m15.790s|23m49.023s|21m52.085s| |SSSD batch_size = 1000|2m33.065s|2m33.831s|7m43.810s|7m39.830s|20m13.238s|19m9.398s|
Result: * Bigger batch not improved total processing time
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop:
|Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m37.513s|2m38.450s|8m19.219s|8m15.790s|23m49.023s|21m52.085s| |SSSD batch_size = 1000|2m33.065s|2m33.831s|7m43.810s|7m39.830s|20m13.238s|19m9.398s|
Result: * Bigger batch improved processing time just a little
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop:
|Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m37.513s|2m38.450s|8m19.219s|8m15.790s|23m49.023s|21m52.085s| |SSSD batch_size = 1000|2m33.065s|2m33.831s|7m43.810s|7m39.830s|20m13.238s|19m9.398s|
Result: * Bigger batch improved processing time just a little
CPU usage during test for all versions:
|Version|Raw files|SSSD master|SSSD batch_size = 10|SSSD batch_size = 1000| |:---:|:---:|:---:|:---:|:---:| ||2840|59699|33885|23768|
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop:
|Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m37.513s|2m38.450s|8m19.219s|8m15.790s|23m49.023s|21m52.085s| |SSSD batch_size = 1000|2m33.065s|2m33.831s|7m43.810s|7m39.830s|20m13.238s|19m9.398s|
Result: * Bigger batch improved processing time just a little
CPU usage during test for all versions:
[blue] Short blue chart on the left shows CPU consumed by raw file provider (SSSD is OFF) [yellow] SSSD build from latest master, without batching implemented [red] SSSD with batching and `batch_size = 10` [green] SSSD with batching and `batch_size = 1000` It is hard to show it on chart well but please compare "density" of yellow, red and green charts (green < red << yellow)
|Version|Raw files|SSSD master|SSSD batch_size = 10|SSSD batch_size = 1000| |:---:|:---:|:---:|:---:|:---:| |Chart volume|2840|59699|33885|23768|
Result: * Batching with batch_`size = 10` changes SSSD CPU usage by -40% * Batching with batch_`size = 1000` changes SSSD CPU usage by -60%
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run another test run to include `batch_size = 1000`. @sumit-bose I have only one domain in `sssd.conf`: ``` [sssd] config_file_version = 2 services = nss, pam, ifp, sudo debug_level = 0x3ff0 domains = files user = root
[nss] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true
[pam] timeout = 30000 debug_level = 0x3ff0
[sudo] timeout = 30000 debug_level = 0x3ff0
[ifp] timeout = 30000 debug_level = 0x3ff0
[domain/files] timeout = 30000 debug_level = 0x3ff0 debug_microseconds = true id_provider = files ```
Time needed to mark domain as inconsistent:
|Version|30 + 100 users|30 + 250 users|30 + 500 users| |:---:|:---:|:---:|:---:| |SSSD master|204ms|558ms|1168ms| |SSSD batch_size = 10|0,5ms|0,5ms|70ms| |SSSD batch_size = 1000|0,4ms|0,5ms|0,6ms|
Result: * Each variant of batching speeds up marking domain as inconsistent and solves https://github.com/SSSD/sssd/issues/5091
Time needed to add / remove 100, 250 and 500 users in bash for loop:
|Version|Add 100|Del 100|Add 250|Del 250|Add 500|Del 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 7.463s|0m 7.618s|0m 19.567s|0m 18.589s|0m 40.271s|0m 39.313s| |SSSD master|0m 58.315s|1m 2.276s|4m 6.709s|4m 2.857s|14m 2.478s|14m 54.913s| |SSSD batch_size = 10|2m 37.513s|2m 38.450s|8m 19.219s|8m 15.790s|23m 49.023s|21m 52.085s| |SSSD batch_size = 1000|2m 33.065s|2m 33.831s|7m 43.810s|7m 39.830s|20m 13.238s|19m 9.398s|
Result: * Bigger batch improved processing time just a little
CPU usage during test for all versions:
[blue] Short blue chart on the left shows CPU consumed by raw file provider (SSSD is OFF) [yellow] SSSD build from latest master, without batching implemented [red] SSSD with batching and `batch_size = 10` [green] SSSD with batching and `batch_size = 1000` It is hard to show it on chart well but please compare "density" of yellow, red and green charts (green < red << yellow)
|Version|Raw files|SSSD master|SSSD batch_size = 10|SSSD batch_size = 1000| |:---:|:---:|:---:|:---:|:---:| |Chart volume|2840|59699|33885|23768|
Result: * Batching with batch_`size = 10` changes SSSD CPU usage by -40% * Batching with batch_`size = 1000` changes SSSD CPU usage by -60%
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867257247
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi @elkoniu,
thank you for the latest tests, what still makes me wonder is that if I run one of those tests I see:
``` # time for N in `seq 500`; do useradd test-user-$N; done
real 1m29.317s user 0m9.067s sys 0m6.530s ```
Can you send me your SSSD logs for the run where 100 users are added?
bye, Sumit
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867496767
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
Hi @elkoniu,
thank you for the latest tests, what still makes me wonder is that if I run one of those tests I see:
# time for N in `seq 500`; do useradd test-user-$N; done real 1m29.317s user 0m9.067s sys 0m6.530sCan you send me your SSSD logs for the run where 100 users are added?
bye, Sumit
Hi @sumit-bose , just sent you complete logs and configs via email. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-867866873
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
thanks for the logs, they made me realize that my test environment is so old that `useradd` does not call `sss_cache` and this is blocked because `ldb_transaction_start()`, which blocks access for other processes, was called too early and effectively `sss_cache` had to wait until the files provider refresh was done.
I moved `ldb_transaction_start()` and it now should not block `sss_cache` anymore. Please run your tests again.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-868313131
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
[green] Raw file provider (SSSD disabled) [yellow] SSSD from latest master [blue] SSSD witch `batch_size = 1000` [red] SSSD witch `batch_size = 2000`
|Volume|Raw files|SSSD master|SSSD batch 1000|SSSD batch 2000| ||3169|67567|3935|4223|
|Version|useradd 100|userdel 100|useradd 250|userdel 250|useradd500|userdel 500| |Raw files|0m 8,943s|0m 8,668s|0m 22,577s|0m 21,934s|0m 47,950s|0m 47,141s| |Master|1m 11,141s|1m 9,929s|4m 52,753s|4m 35,610s|15m 51,789s|15m 23,659s| |Batch 1000|0m 26,761s|0m 26,532s|1m 7,819s|1m 5,323s|2m 13,062s|2m 12,565s| |Batch 2000|0m 27,328s|0m 27,011s|1m 8,881s|1m 6,857s|2m 15,424s|2m 11,804s|
Results: * Batching version is more than 2 times faster than SSSD master * Batching version is more than 15 times lighter on CPU than SSSD master * Batch size bigger than 1000 makes almost no difference it time or CPU usage
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870505352
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
[green] Raw file provider (SSSD disabled) [yellow] SSSD from latest master [blue] SSSD witch `batch_size = 1000` [red] SSSD witch `batch_size = 2000`
|Volume|Raw files|SSSD master|SSSD batch 1000|SSSD batch 2000| |:---:|:---:|:---:|:---:|:---:| ||3169|67567|3935|4223|
|Version|useradd 100|userdel 100|useradd 250|userdel 250|useradd500|userdel 500| |Raw files|0m 8,943s|0m 8,668s|0m 22,577s|0m 21,934s|0m 47,950s|0m 47,141s| |Master|1m 11,141s|1m 9,929s|4m 52,753s|4m 35,610s|15m 51,789s|15m 23,659s| |Batch 1000|0m 26,761s|0m 26,532s|1m 7,819s|1m 5,323s|2m 13,062s|2m 12,565s| |Batch 2000|0m 27,328s|0m 27,011s|1m 8,881s|1m 6,857s|2m 15,424s|2m 11,804s|
Results: * Batching version is more than 2 times faster than SSSD master * Batching version is more than 15 times lighter on CPU than SSSD master * Batch size bigger than 1000 makes almost no difference it time or CPU usage
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870505352
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ 
[green] Raw file provider (SSSD disabled) [yellow] SSSD from latest master [blue] SSSD witch `batch_size = 1000` [red] SSSD witch `batch_size = 2000`
|Version|Raw files|SSSD master|SSSD batch 1000|SSSD batch 2000| |:---:|:---:|:---:|:---:|:---:| |Chart volume|3169|67567|3935|4223|
|Version|useradd 100|userdel 100|useradd 250|userdel 250|useradd500|userdel 500| |:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Raw files|0m 8,943s|0m 8,668s|0m 22,577s|0m 21,934s|0m 47,950s|0m 47,141s| |Master|1m 11,141s|1m 9,929s|4m 52,753s|4m 35,610s|15m 51,789s|15m 23,659s| |Batch 1000|0m 26,761s|0m 26,532s|1m 7,819s|1m 5,323s|2m 13,062s|2m 12,565s| |Batch 2000|0m 27,328s|0m 27,011s|1m 8,881s|1m 6,857s|2m 15,424s|2m 11,804s|
Results: * Batching version is more than 2 times faster than SSSD master * Batching version is more than 15 times lighter on CPU than SSSD master * Batch size bigger than 1000 makes almost no difference it time or CPU usage
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870505352
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
Batching version is more than 2 times faster than SSSD master
It's actually better according to your results: 73 vs 951 seconds (i.e. ~ x13) for 500 users. And "only" ~1.5 times slower than libnss_files.so """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870617812
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ @elkoniu , did you check if this PR fixes #5091? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870661479
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Yes, it does. I am running separated test against 5000 users to confirm that. In parallel I am doing normal code review for this PR. Hope to finish both later today. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870664628
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ `files_id_ctx`: ``` const char **passwd_files; const char **group_files; ``` -- do we really need support of multiple passwd/group local files..? This complicates code...
(I know it's not introduced in this PR, just wondering...) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870688111
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ There are several minor covscan issues: https://cov01.lab.eng.brq.redhat.com/covscanhub/task/223840/log/added.err (sorry for the private link) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870758099
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ I have one strange observation but I am not sure if it is related to batching process.
Steeps: 1) Add 100 users calling useradd in loop 2) Dell 100 users calling userdell in loop 3) Add 250 users calling useradd in loop
If there is no delay between steep (2) and (3) useradd returns "User already exists" for 100 users and then it adds another 150 (250 in total). Those 100 "existing" users are not presented in `/etc/passwd`. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-870987006
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
There are several minor covscan issues: https://cov01.lab.eng.brq.redhat.com/covscanhub/task/223840/log/added.err (sorry for the private link)
Thanks, fixed. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-871283222
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
the latest version contains fixes for @alexey-tikhonov's review comments, I'll try to understand what is happening in @elkoniu latest test now.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-871286314
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """
I have one strange observation but I am not sure if it is related to batching process.
Steeps:
1. Add 100 users calling useradd in loop 2. Dell 100 users calling userdell in loop 3. Add 250 users calling useradd in loopIf there is no delay between steep (2) and (3) useradd returns "User already exists" for 100 users and then it adds another 150 (250 in total). Those 100 "existing" users are not presented in `/etc/passwd`.
Hi,
thanks for running this kind of test. The fallback to nss was not complete because the cache_req tried to find the related object in the cache. I added a new patch to cache_req to check if the domain is inconsistent or not.
@pbrezina, do you agree with https://github.com/SSSD/sssd/pull/5552/commits/715b4074171030950661875d39a74...
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872056928
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run test against 5000 users checking how fast we mark domain as inconsistent. In some cases this took up to 3 seconds. It is still much faster than latest master version but I think it is not fast enough to solve https://github.com/SSSD/sssd/issues/5091 """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872104407
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Just run test against 5000 users to check marking domain as inconsistent. This PR can be considered as solution for https://github.com/SSSD/sssd/issues/5091 """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872104407
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """
I have one strange observation but I am not sure if it is related to batching process. Steeps:
1. Add 100 users calling useradd in loop 2. Dell 100 users calling userdell in loop 3. Add 250 users calling useradd in loopIf there is no delay between steep (2) and (3) useradd returns "User already exists" for 100 users and then it adds another 150 (250 in total). Those 100 "existing" users are not presented in `/etc/passwd`.
Hi,
thanks for running this kind of test. The fallback to nss was not complete because the cache_req tried to find the related object in the cache. I added a new patch to cache_req to check if the domain is inconsistent or not.
@pbrezina, do you agree with [715b407](https://github.com/SSSD/sssd/commit/715b4074171030950661875d39a747a2415d0032...
bye, Sumit
It looks like in the latest version this issue is solved. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872212180
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Another observation I have is that with 5000 users `systemctl restart sssd` takes about 40 seconds. Normally it would hit watchdog, with this PR it starts normally. The question is: if this long start time should be addressed as a part of this PR or separated PR related to cache optimization itself? I am voting for 2nd option. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872213598
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] On top of steps from [ISSUE 1] 6) Stop SSSD: `systemctl stop sssd` 7) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 8) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped) """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] On top of steps from [ISSUE 1] 6) Stop SSSD: `systemctl stop sssd` 7) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 8) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped)
It looks like those two issues are related to each other. The open question is if they are directly related to this PR as in [ISSUE 2] SSSD is not running and it is on top of "raw" files provider. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] - On top of steps from [ISSUE 1] 6) Stop SSSD: `systemctl stop sssd` 7) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 8) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped)
It looks like those two issues are related to each other. The open question is if they are directly related to this PR as in [ISSUE 2] SSSD is not running and it is on top of "raw" files provider. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] - On top of steps from [ISSUE 1] 1) Stop SSSD: `systemctl stop sssd` 2) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 3) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped)
It looks like those two issues are related to each other. The open question is if they are directly related to this PR as in [ISSUE 2] SSSD is not running and it is on top of "raw" files provider. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] - On top of steps from [ISSUE 1] 1) Stop SSSD: `systemctl stop sssd` 2) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 3) Call `time for N in `seq 5000`; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped)
It looks like those two issues are related to each other. The open question is if they are directly related to this PR as in [ISSUE 2] SSSD is not running and it is on top of "raw" files provider. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Two more issues exists so far:
[ISSUE 1] 1) Add 5000 users without SSSD (raw NSS only, systemctl stop sssd, /etc/nsswitch.conf -> files only) 2) Enable sss in /etc/nsswitch.conf (passwd -> sss files, group -> sss files) 3) Start SSSD 4) Remove few users manually from /etc/passwd 5) Call `time for N in 'seq 5000'; do userdel -r test-user-$N; done`
Process will remove 1 user and will hang with `sssd_be` consuming `99%` of CPU, then switching to `sss_cahce` consuming `99%` of CPU. After about 1min it will continue to remove next user. At the end it takes around 1min to remove single user.
[ISSUE 2] - On top of steps from [ISSUE 1] 1) Stop SSSD: `systemctl stop sssd` 2) /etc/nsswitch.conf -> files only (sss removed from passwd and group to have only raw NSS provider) 3) Call `time for N in 'seq 5000'; do userdel -r test-user-$N; done`
userdel running on top of nss (SSSD stoped) still blocks and takes about 30s to remove single user. Also `sss_cache` is visible in `top` command consuming up to 60% of CPU (even with SSSD stopped)
It looks like those two issues are related to each other. The open question is if they are directly related to this PR as in [ISSUE 2] SSSD is not running and it is on top of "raw" files provider. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872230342
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ The [ISSUE 1] and [ISSUE 2] I described above are presented on master branch build too. This way I do not think we are introducing any regression with this PR. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872348867
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
I think the latest issues you've found are not strictly related to the patches. The reason is that if there are many users and groups in the cache the `sss_cache -UG` needs more time to invalidate all users and groups. With 5000 users and the related primary groups it looks like `sss_cache` runs for more then 1s even if all entries are already invalidated. But this means that the next `userdel` command is called more than 1s after the last `userdel` which means that it is called while a refresh is running. This is no issue for the `userdel` command itself, but `userdel` will call `sss_cache` which will try to modify the cache which is locked by the transaction covering the refresh. So `sss_cache` has to wait until the refresh is finished which might take some time. For the next `userdel` call the same happens again.
It might be possible to mitigate this by increasing the initial delay before the refresh starts (`struct timeval delay` in `check_state()`) but this will delay the refresh for all other cases as well.
Another way might be to speed up the refresh by not deleting and adding everything from scratch but with a diff like approach. But as discussed earlier this should be part of a PR.
It might also be possible to modify `sss_cache` so that it e.g. only sends a command to SSSD and not tries to modify the caches on its own.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872351523
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: synchronized
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
sumit-bose commented: """ Hi,
the latest version just adds the `:config:` tag for the new option.
bye, Sumit """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-872393948
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Accepted
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: -Waiting for review
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """
I think the latest issues you've found are not strictly related to the patches. The reason is that if there are many users and groups in the cache the `sss_cache -UG` needs more time to invalidate all users and groups. With 5000 users and the related primary groups it looks like `sss_cache` runs for more then 1s even if all entries are already invalidated. But this means that the next `userdel` command is called more than 1s after the last `userdel` which means that it is called while a refresh is running. This is no issue for the `userdel` command itself, but `userdel` will call `sss_cache` which will try to modify the cache which is locked by the transaction covering the refresh. So `sss_cache` has to wait until the refresh is finished which might take some time. For the next `userdel` call the same happens again.
...
It might also be possible to modify `sss_cache` so that it e.g. only sends a command to SSSD and not tries to modify the caches on its own.
The thing is, IIRC, the real use of sss_cache here is to drop mem-cache. Invalidation of sysdb cache (for all domains!) is undesirable side effect (that does only harm - for "files" provider expiration time doesn't have any affect anyway). Details are here: https://bugzilla.redhat.com/show_bug.cgi?id=1693379#c6 """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-876569077
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ My questions were addressed. Thank you. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-876569430
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ My questions were addressed. Thank you.
Covscan is clean. """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-876569430
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Ready to push
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
pbrezina commented: """ Pushed PR: https://github.com/SSSD/sssd/pull/5552
* `master` * b4ee698ac078e74df51197b5f92432b4ed712d99 - cache_req: do not return cached data if domain is inconsistent * 19b850636399fdf5f1018671ba5e2ff7c9deaa2f - files: queue certmap requests if a refresh is running * dd1aa57950294e0b821a0be2893a159c5e5488a6 - files: delay refresh and not run in parallel * 0fbd67404a4b48b76e8750f0cdc727ed0f8de424 - files: add new option fallback_to_nss * 5288ddaa283bb5e710a2864ff3866bf87f56d03f - files: split update into batches
"""
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-877055082
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: +Pushed
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: -Accepted
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
Label: -Ready to push
URL: https://github.com/SSSD/sssd/pull/5552 Author: sumit-bose Title: #5552: files: split update into batches Action: closed
To pull the PR as Git branch: git remote add ghsssd https://github.com/SSSD/sssd git fetch ghsssd pull/5552/head:pr5552 git checkout pr5552
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
alexey-tikhonov commented: """ Hi @elkoniu,
is #5086 fixed by this PR? """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-877065218
URL: https://github.com/SSSD/sssd/pull/5552 Title: #5552: files: split update into batches
elkoniu commented: """ Hi @alexey-tikhonov , I think it is addressed by part of this PR which is commit https://github.com/SSSD/sssd/pull/5552/commits/eae03fbe65488d2379131cf32274e... """
See the full comment at https://github.com/SSSD/sssd/pull/5552#issuecomment-877067191
sssd-devel@lists.fedorahosted.org
-
alexey-tikhonov -
elkoniu
-
pbrezina
-
sumit-bose