LMDB vs BDB where locks are exhausted

[Emmanuel Kasprzyk]

Hi
I am working on large Directory Server topology, which is reaching very
fast the amount of available locks in BDB ( cf
https://bugzilla.redhat.com/show_bug.cgi?id=1831812 )

- Can the planned switch in 389-ds-base-1.4.next to LMDB help for such
cases ?
( Especially after reading "The database structure is multi-versioned so
readers run with no locks" on http://www.lmdb.tech/doc/index.html )
- Is the switch to lmdb planned in an amount of quartals or years? I know
nobody likes to put dates on roadmaps, it's just that having a rough
estimate would help me to know how much effort I should put into tuning BDB
right now.

Thank you


-- 

Emmanuel Kasper

Technical Account Manager

Red Hat EMEA <https://www.redhat.com>

<https://www.redhat.com>

[David Boreham]

On 6/23/2020 9:34 AM, Emmanuel Kasprzyk wrote:

>  I am working on large Directory Server topology, which is reaching 
>  very fast the amount of available locks in BDB ( cf 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1831812 )
>
>  - Can the planned switch in 389-ds-base-1.4.next to LMDB help for such 
>  cases ?
>  ( Especially after reading "The database structure is multi-versioned 
>  so readers run with no locks" on http://www.lmdb.tech/doc/index.html )
>  
Probably better to fix the bug in DS that cases it to run a long running 
transaction with repeatable reads isolation?

[Mark Reynolds]

On 6/23/20 11:42 AM, David Boreham wrote:
>  On 6/23/2020 9:34 AM, Emmanuel Kasprzyk wrote:
>
> > I am working on large Directory Server topology, which is reaching 
> > very fast the amount of available locks in BDB ( cf 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1831812 )
> >
> > - Can the planned switch in 389-ds-base-1.4.next to LMDB help for 
> > such cases ? 
Yes, we should not see this particular issue with LMDB
> > ( Especially after reading "The database structure is
> multi-versioned 
> > so readers run with no locks" on http://www.lmdb.tech/doc/index.html )
> >
>  Probably better to fix the bug in DS that cases it to run a long 
>  running transaction with repeatable reads isolation? 
In 389 what we are seeing is that our backend txn plugins are doing 
unindexed searches, but I would not call it a bug.  It's really a 
configuration/indexing issue.  But yes, there are long running 
operations/txns in regards to many plugins doing a lot of things while 
the database is being updated in the same nested operation.  Now when 
these internal searches are properly indexed the db lock issue 
completely goes away.  The bug that Emmanuel is referring to is still 
under investigation.  We have yet to identify the internal unindexed 
search in this customer's case, but a patch/fix was created that would 
log the required information we need to track it down and fix the config.

>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 

389 Directory Server Development Team

[David Boreham]

On 6/23/2020 10:07 AM, Mark Reynolds wrote:
>  In 389 what we are seeing is that our backend txn plugins are doing 
>  unindexed searches, but I would not call it a bug. 
The unindexed search is fine per se (although probably not a great idea 
if you want the op the plugin hooked to complete quickly).

What's not fine is that all the DB reads under that search should be 
done in the same transaction with strong isolation.

>  It's really a configuration/indexing issue.  But yes, there are
> long 
>  running operations/txns in regards to many plugins doing a lot of 
>  things while the database is being updated in the same nested 
>  operation.  Now when these internal searches are properly indexed the 
>  db lock issue completely goes away. 
If missing an index were to result in poor performance, agreed -- it's a 
configuration issue. The server process exiting seems quite an extreme 
consequence.

Wondering if this is the result of an old fix for a deadlock problem 
(bringing the internal op under the main transaction to cure the deadlock)?

How is a regular (non-internal) unindexed search run? Surely that 
doesn't burn through one lock per page touched?

[Mark Reynolds]

On 6/23/20 12:22 PM, David Boreham wrote:
>  On 6/23/2020 10:07 AM, Mark Reynolds wrote:
> >
> > In 389 what we are seeing is that our backend txn plugins are doing 
> > unindexed searches, but I would not call it a bug.
>
>  The unindexed search is fine per se (although probably not a great 
>  idea if you want the op the plugin hooked to complete quickly).
>
>  What's not fine is that all the DB reads under that search should be 
>  done in the same transaction with strong isolation. 
First I'm not that intimately familiar with this issue, Thierry and 
Ludwig did most of that investigation.  But this happens during a modify 
operation that triggers some BE txn plugins that do searches and updates 
under the same parent transaction.  So under these conditions is when it 
just starts consuming a ton of db locks.

Unindexed searches by themselves do not cause this issue, it's when we 
are updating the database under the same txn.  So the mod takes a lock 
on a db page, then we call the be postop plugins, which in turn starts 
doing these expensive searches and updates - that is when the db lock 
issue pops up.  I seem to recall from previous similar cases that this 
"mod update" involved a very large static group, and the RI or memberOf 
plugin doing its work. Maybe Thierry recalls some of the past cases?

> > It's really a configuration/indexing issue.  But yes, there are long 
> > running operations/txns in regards to many plugins doing a lot of 
> > things while the database is being updated in the same nested 
> > operation.  Now when these internal searches are properly indexed the 
> > db lock issue completely goes away.
>
>  If missing an index were to result in poor performance, agreed -- it's 
>  a configuration issue. The server process exiting seems quite an 
>  extreme consequence. 
It's not exactly crashing, but the db can get corrupted
and it needs to 
be reinitialized.  That sounds like a libdb bug to me :-) Running out of 
db locks should not corrupt the database.
>  Wondering if this is the result of an old fix for a deadlock problem 
>  (bringing the internal op under the main transaction to cure the 
>  deadlock)? 
Maybe :-)  Haven't looked at that code in quite a few years...
>  How is a regular (non-internal) unindexed search run? Surely that 
>  doesn't burn through one lock per page touched? 
No it doesn't.  See my comment above, standalone unindexed searches do 
not trigger this issue.

Mark

>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 

389 Directory Server Development Team

[Ludwig Krispenz]

On 23.06.20 19:19, Mark Reynolds wrote:
>  On 6/23/20 12:22 PM, David Boreham wrote:
> >
> > On 6/23/2020 10:07 AM, Mark Reynolds wrote:
> >>
> >> In 389 what we are seeing is that our backend txn plugins are doing 
> >> unindexed searches, but I would not call it a bug.
> >
> > The unindexed search is fine per se (although probably not a great 
> > idea if you want the op the plugin hooked to complete quickly).
> >
> > What's not fine is that all the DB reads under that search should be 
> > done in the same transaction with strong isolation.
>
>  First I'm not that intimately familiar with this issue, Thierry and 
>  Ludwig did most of that investigation.  But this happens during a 
>  modify operation that triggers some BE txn plugins that do searches 
>  and updates under the same parent transaction.  So under these 
>  conditions is when it just starts consuming a ton of db locks. 
If a transactional
operation (eg modify) triggers a search by a plugin 
it already holds a coupke of page locks as write locks. If the search 
would try to access this pages without using the txn of the parent it 
would have to wait - and the whole operation would self deadlock. So all 
db accesses inside a txn need to use this txn directly or as a paent txn.
>
> Unindexed searches by themselves do not cause this issue, it's when we 
> are updating the database under the same txn.  So the mod takes a lock 
> on a db page, then we call the be postop plugins, which in turn starts 
> doing these expensive searches and updates - that is when the db lock 
> issue pops up.  I seem to recall from previous similar cases that this 
> "mod update" involved a very large static group, and the RI or 
> memberOf plugin doing its work. Maybe Thierry recalls some of the past 
> cases?
>
>>
>>> It's really a configuration/indexing issue.  But yes, there are long 
>>> running operations/txns in regards to many plugins doing a lot of 
>>> things while the database is being updated in the same nested 
>>> operation.  Now when these internal searches are properly indexed 
>>> the db lock issue completely goes away.
>>
>> If missing an index were to result in poor performance, agreed -- 
>> it's a configuration issue. The server process exiting seems quite an 
>> extreme consequence.
> It's not exactly crashing, but the db can get corrupted and it needs 
> to be reinitialized.  That sounds like a libdb bug to me :-) Running 
> out of db locks should not corrupt the database.
>>
>>
>> Wondering if this is the result of an old fix for a deadlock problem 
>> (bringing the internal op under the main transaction to cure the 
>> deadlock)?
> Maybe :-)  Haven't looked at that code in quite a few years...
>>
>> How is a regular (non-internal) unindexed search run? Surely that 
>> doesn't burn through one lock per page touched?
>
> No it doesn't.  See my comment above, standalone unindexed searches do 
> not trigger this issue.
>
> Mark
>
>>
>> _______________________________________________
>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org
>> To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct: 
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: 
>> https://lists.fedoraproject.org/archives/list/[e-mail]...
>