[sssd PR#332][opened] sydb: index improvements

[sumit-bose]

   URL: https://github.com/SSSD/sssd/pull/332
Author: sumit-bose
 Title: #332: sydb: index improvements
Action: opened

PR body:
"""
This patch first adds some missing attributes to the index. The most
important one here is 'ghost' which is used in the backed during group
lookups.

Additionally the index for one-level searches @IDXONE is removed. One
level searches were only used in a few places and are replace by this
patch with sub-tree searches. The main reason for the removal is that
maintaining the index is quite costly because it is basically a single
huge blob in the underlying tdb database.

Finally this patch removes the index on the objectClass attribute and
adds a new index on an new attribute called objectCategory which is used
instead of objectClass for all objects expect user and group. Typically
user and group searches are done by name or ID attributes which are more
specific then objectClass. And since most of the objects in the cache
will be users and groups a search for all users or groups will be near
to a full database search so that the index won't help much in this case
either. The reason for removing it are the costs to manage it when there
are many users or groups.

Due to the index changes some search results are returned in different
order. I updated the related tests so that the checks do not depend on a
specific order anymore.

If 'LDB_WARN_UNINDEXED=1' is set in /etc/sysconfig/sssd full database
searches are indicated with a 'ldb FULL SEARCH: ...' debug message.
Since there are no extra costs we might want to enable this by default
with a certain debug level.

Currently there are two types of un-indexed searches. Searches with
'(distinguishedName=*)' in the filter are related to sub-tree deletes
and '(dataExpireTimestamp<=...)' are related to refresh and cleanup
tasks. Please note that '<=' and sub-string searches cannot be indexed.
"""

To pull the PR as Git branch:
git remote add ghsssd https://github.com/SSSD/sssd
git fetch ghsssd pull/332/head:pr332
git checkout pr332

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
I think the patches are great and we've seen already reports by users who run the
patches and reported that performance improved substantially.

There are some objects that are not user or group but still use OBJECTCLASS and not
CATEGORY. At least `sysdb_save_autofsmap` still uses `SYSDB_OBJECTCLASS`, then
`sysdb_svc_add` (or rather the sysdb_services module), the sysdb_ssh module,
sysdb_netgroups module and the sudo module.
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-323966202

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
Setting changes requested as per the review.

(note that this is NOT a nudge for anyone to work faster, I'm just reconciling the PR
statuses)
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-325455185

[sssd PR#332][+Changes requested] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

Label: +Changes requested

[sssd PR#332][synchronized] sydb: index improvements

[sumit-bose]

   URL: https://github.com/SSSD/sssd/pull/332
Author: sumit-bose
 Title: #332: sydb: index improvements
Action: synchronized

To pull the PR as Git branch:
git remote add ghsssd https://github.com/SSSD/sssd
git fetch ghsssd pull/332/head:pr332
git checkout pr332

[sssd PR#332][comment] sydb: index improvements

[sumit-bose]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

sumit-bose commented:
"""
I push a new, simplified version of the patch which only adds the missing indices. 
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-327199577

[sssd PR#332][-Changes requested] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

Label: -Changes requested

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
btw now I would also prefer to have 2 tickets - one for indexing fixes and one for
objectcategory. The latter first in 1.16.0 with a possible backport later. What do you
think?
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-327401033

[sssd PR#332][comment] sydb: index improvements

[sumit-bose]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

sumit-bose commented:
"""
sure, would you mind  to open them? 
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-327402970

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
The ticket for the objectcategory addition is https://pagure.io/SSSD/sssd/issue/3503
The ticket for the indexing changes is already in 1.15.4
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-327594151

[sssd PR#332][synchronized] sydb: index improvements

[sumit-bose]

   URL: https://github.com/SSSD/sssd/pull/332
Author: sumit-bose
 Title: #332: sydb: index improvements
Action: synchronized

To pull the PR as Git branch:
git remote add ghsssd https://github.com/SSSD/sssd
git fetch ghsssd pull/332/head:pr332
git checkout pr332

[sssd PR#332][comment] sydb: index improvements

[sumit-bose]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

sumit-bose commented:
"""
Just added a reference to the pagure ticket in the commit messages.
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-327698402

[sssd PR#332][synchronized] sydb: index improvements

[sumit-bose]

   URL: https://github.com/SSSD/sssd/pull/332
Author: sumit-bose
 Title: #332: sydb: index improvements
Action: synchronized

To pull the PR as Git branch:
git remote add ghsssd https://github.com/SSSD/sssd
git fetch ghsssd pull/332/head:pr332
git checkout pr332

[sssd PR#332][comment] sydb: index improvements

[sumit-bose]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

sumit-bose commented:
"""
Thank you for the rigid review, it got lost while splitting the patch. New version
attached.
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-329411115

[sssd PR#332][-Changes requested] sydb: index improvements

[sumit-bose]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

Label: -Changes requested

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
ACK, this version upgrades the cache fine and using this command:
```
ldbsearch -s base -b "@INDEXLIST" -H /var/lib/sss/db/cache_ipa.test.ldb
```

I verified the indices were indeed aded.
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-329423088

[sssd PR#332][+Accepted] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

Label: +Accepted

[sssd PR#332][comment] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

jhrozek commented:
"""
* master: 9acdf51bf32d7b4389f3faea0fc6b73c56b6da71
"""

See the full comment at https://github.com/SSSD/sssd/pull/332#issuecomment-329511895

[sssd PR#332][closed] sydb: index improvements

[jhrozek]

   URL: https://github.com/SSSD/sssd/pull/332
Author: sumit-bose
 Title: #332: sydb: index improvements
Action: closed

To pull the PR as Git branch:
git remote add ghsssd https://github.com/SSSD/sssd
git fetch ghsssd pull/332/head:pr332
git checkout pr332

[sssd PR#332][+Pushed] sydb: index improvements

[jhrozek]

  URL: https://github.com/SSSD/sssd/pull/332
Title: #332: sydb: index improvements

Label: +Pushed