[discuss] Entry cache and backend txn plugin problems

[Mark Reynolds]

I want to start a brief discussion about a major problem we have backend 
transaction plugins and the entry caches.  I'm finding that when we get 
into a nested state of be txn plugins and one of the later plugins that 
is called fails then while we don't commit the disk changes (they are 
aborted/rolled back) we DO keep the entry cache changes!

For example, a modrdn operation triggers the referential integrity 
plugin which renames the member attribute in some group and changes that 
group's entry cache entry, but then later on the memberOf plugin fails 
for some reason.  The database transaction is aborted, but the entry 
cache changes that RI plugin did are still present :-(  I have also 
found other entry cache issues with modrdn and BE TXN plugins, and we 
know of other currently non-reproducible entry cache crashes as well 
related to mishandling of cache entries after failed operations.

It's time to rework how we use the entry cache.  We basically need a 
transaction style caching mechanism - we should not commit any entry 
cache changes until the original operation is fully successful.  
Unfortunately the way the entry cache is currently designed and used it 
will be a major change to try to change it.

William wrote up this doc: 
http://www.port389.org/docs/389ds/design/cache_redesign.html

But this also does not currently cover the nested plugin scenario either 
(not yet).  I do know how how difficult it would be to implement 
William's proposal, or how difficult it would be to incorporate the txn 
style caching into his design.  What kind of time frame could this even 
be implemented in?  William what are your thoughts?

If William's design is too huge of a change that will take too long to 
safely implement then perhaps we need to look into revising the existing 
cache design where we use "cache_add_tentative" style functions and only 
apply them at the end of the op.  This is also not a trivial change.

And what impact would changing the entry cache have on Ludwig's plugable 
backend work?

Anyway we need to start thinking about redesigning the entry cache - no 
matter what approach we want to take.  If anyone has any ideas or 
comments please share them, but I think due to the severity of this flaw 
redesigning the entry cache should be one of our next major goals in DS 
(1.4.1?).

Thanks,

Mark

[William Brown]

> On 23 Feb 2019, at 02:46, Mark Reynolds  wrote:
>
> I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
>
> For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
>
> It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful.  Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
>
> William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
>
> But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?

I like coffee? How cool are planes? My thoughts are simple :)

I think there is a pretty simple mental simplification we can make here though. Nested transactions “don’t really exist”. We just have *recursive* operations inside of one transaction.

Once reframed like that, the entire situation becomes simpler. We have one thread in a write transaction that can have recursive/batched operations as required, which means that either “all operations succeed” or “none do”. Really, this is the behaviour we want anyway, and it’s the transaction model of LMDB and other kv stores that we could consider (wired tiger, sled in the future).
> If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.

It’s pretty massive as a change - if we want to do it right. I’d say we need:

* development and testing of a MVCC/COW cache implementation (proof that it really really works transactionally)
* allow “disable/disconnect” of the entry cache, but with the higher level txn’s so that we can prove the txn semantics are correct
* re-architect our transaction calls so that they are “higher” up. An example is that internal_modify shouldn’t start a txn, it should be given the current txn state as an arg. Combined with the above, we can prove we haven’t corrupted our server transaction guarantees.
* integrate the transactional cache.

I don’t know if I would still write a transactional cache the same way as I proposed in that design, but I think the ideas are on the right path.
> And what impact would changing the entry cache have on Ludwig's plugable backend work?

Should be none, it’s seperate layers. If anything this change is going to make Ludwig’s work better because our current model won’t really take good advantage of the MVCC nature of modern kv stores.
> Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
>
> Thanks,
>
> Mark
> _______________________________________________
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

—
Sincerely,

William Brown
Software Engineer, 389 Directory Server
SUSE Labs

[Ludwig Krispenz]

Hi, I need a bit of time to read the docs and clear my thoughts, but one 
comment below
On 02/25/2019 01:49 AM, William Brown wrote:
> > On 23 Feb 2019, at 02:46, Mark Reynolds  wrote:
> >
> > I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
> >
> > For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
> >
> > It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful.  Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
> >
> > William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
> >
> > But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?
>
> I like coffee? How cool are planes? My thoughts are simple :)
>
> I think there is a pretty simple mental simplification we can make here though. Nested transactions “don’t really exist”. We just have *recursive* operations inside of one transaction.
>
> Once reframed like that, the entire situation becomes simpler. We have one thread in a write transaction that can have recursive/batched operations as required, which means that either “all operations succeed” or “none do”. Really, this is the behaviour we want anyway, and it’s the transaction model of LMDB and other kv stores that we could consider (wired tiger, sled in the future).

I think the recursive/nested transaction on the database level are not 
the problem, we do this correctly already, either all or no change 
becomes persistent.
What we do not manage is modifications we do in parallel on the in 
memory structure like the entry cache, changes to the EC are not managed 
by any txn and I do not see how any of the database txn models would 
help, they do not know about ec and can abort changes.
We would need to incorporate the EC into a generic txn model, or have a 
way to flag ec entries as garbage for if a txn is aborted
> > If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.
>
> It’s pretty massive as a change - if we want to do it right. I’d say we need:
> development and testing of a MVCC/COW cache implementation (proof that it really really works transactionally)
> allow “disable/disconnect” of the entry cache, but with the higher level txn’s so that we can prove the txn semantics are correct
> re-architect our transaction calls so that they are “higher” up. An example is that internal_modify shouldn’t start a txn, it should be given the current txn state as an arg. Combined with the above, we can prove we haven’t corrupted our server transaction guarantees.
> integrate the transactional cache.
>
> I don’t know if I would still write a transactional cache the same way as I proposed in that design, but I think the ideas are on the right path.
> > And what impact would changing the entry cache have on Ludwig's plugable backend work?
>
> Should be none, it’s seperate layers. If anything this change is going to make Ludwig’s work better because our current model won’t really take good advantage of the MVCC nature of modern kv stores.
> > Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
> >
> > Thanks,
> >
> > Mark
> > _______________________________________________
> > 389-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
>
> —
> Sincerely,
>
> William Brown
> Software Engineer, 389 Directory Server
> SUSE Labs
> _______________________________________________
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

[William Brown]

> On 26 Feb 2019, at 18:32, Ludwig Krispenz  wrote:
>
> Hi, I need a bit of time to read the docs and clear my thoughts, but one comment below
> On 02/25/2019 01:49 AM, William Brown wrote:
> > > On 23 Feb 2019, at 02:46, Mark Reynolds  wrote:
> > >
> > > I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
> > >
> > > For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
> > >
> > > It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful. Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
> > >
> > > William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
> > >
> > > But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?
> >
> > I like coffee? How cool are planes? My thoughts are simple :)
> >
> > I think there is a pretty simple mental simplification we can make here though. Nested transactions “don’t really exist”. We just have *recursive* operations inside of one transaction.
> >
> > Once reframed like that, the entire situation becomes simpler. We have one thread in a write transaction that can have recursive/batched operations as required, which means that either “all operations succeed” or “none do”. Really, this is the behaviour we want anyway, and it’s the transaction model of LMDB and other kv stores that we could consider (wired tiger, sled in the future).
>
> I think the recursive/nested transaction on the database level are not the problem, we do this correctly already, either all or no change becomes persistent.
> What we do not manage is modifications we do in parallel on the in memory structure like the entry cache, changes to the EC are not managed by any txn and I do not see how any of the database txn models would help, they do not know about ec and can abort changes.
> We would need to incorporate the EC into a generic txn model, or have a way to flag ec entries as garbage for if a txn is aborted

The issue is we allow parallel writes, which breaks the consistency guarantees of the EC anyway. LMDB won’t allow parallel writes (it’s single write - concurrent parallel readers), and most other modern kv stores take this approach too, so we should be remodelling our transactions to match this IMO. It will make the process of how we reason about the EC much much simpler I think.
> > > If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.
> >
> > It’s pretty massive as a change - if we want to do it right. I’d say we need:
> > development and testing of a MVCC/COW cache implementation (proof that it really really works transactionally)
> > allow “disable/disconnect” of the entry cache, but with the higher level txn’s so that we can prove the txn semantics are correct
> > re-architect our transaction calls so that they are “higher” up. An example is that internal_modify shouldn’t start a txn, it should be given the current txn state as an arg. Combined with the above, we can prove we haven’t corrupted our server transaction guarantees.
> > integrate the transactional cache.
> >
> > I don’t know if I would still write a transactional cache the same way as I proposed in that design, but I think the ideas are on the right path.
> > > And what impact would changing the entry cache have on Ludwig's plugable backend work?
> >
> > Should be none, it’s seperate layers. If anything this change is going to make Ludwig’s work better because our current model won’t really take good advantage of the MVCC nature of modern kv stores.
> > > Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
> > >
> > > Thanks,
> > >
> > > Mark
> > > _______________________________________________
> > > 389-devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> >
> > —
> > Sincerely,
> >
> > William Brown
> > Software Engineer, 389 Directory Server
> > SUSE Labs
> > _______________________________________________
> > 389-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
>
> -- 
> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
> _______________________________________________
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

—
Sincerely,

William Brown
Software Engineer, 389 Directory Server
SUSE Labs

[Rich Megginson]

On 2/26/19 4:26 PM, William Brown wrote:
> > On 26 Feb 2019, at 18:32, Ludwig Krispenz  wrote:
> >
> > Hi, I need a bit of time to read the docs and clear my thoughts, but one comment below
> > On 02/25/2019 01:49 AM, William Brown wrote:
> > > > On 23 Feb 2019, at 02:46, Mark Reynolds  wrote:
> > > >
> > > > I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
> > > >
> > > > For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
> > > >
> > > > It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful. Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
> > > >
> > > > William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
> > > >
> > > > But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?
> > >
> > > I like coffee? How cool are planes? My thoughts are simple :)
> > >
> > > I think there is a pretty simple mental simplification we can make here though. Nested transactions “don’t really exist”. We just have *recursive* operations inside of one transaction.
> > >
> > > Once reframed like that, the entire situation becomes simpler. We have one thread in a write transaction that can have recursive/batched operations as required, which means that either “all operations succeed” or “none do”. Really, this is the behaviour we want anyway, and it’s the transaction model of LMDB and other kv stores that we could consider (wired tiger, sled in the future).
> >
> > I think the recursive/nested transaction on the database level are not the problem, we do this correctly already, either all or no change becomes persistent.
> > What we do not manage is modifications we do in parallel on the in memory structure like the entry cache, changes to the EC are not managed by any txn and I do not see how any of the database txn models would help, they do not know about ec and can abort changes.
> > We would need to incorporate the EC into a generic txn model, or have a way to flag ec entries as garbage for if a txn is aborted
>
> The issue is we allow parallel writes, which breaks the consistency guarantees of the EC anyway. LMDB won’t allow parallel writes (it’s single write - concurrent parallel readers), and most other modern kv stores take this approach too, so we should be remodelling our transactions to match this IMO. It will make the process of how we reason about the EC much much simpler I think.

Some sort of in-memory data structure with fast lookup and transactional semantics (modify operations are stored as mvcc/cow so each read of the database with a given txn handle sees its own 
view of the ec, a txn commit updates the parent txn ec view, or the global ec view if no parent, from the copy, a txn abort deletes the txn's copy of the ec) is needed.  A quick google search 
turns up several hits.  I'm not sure if the B+Tree proposed at http://www.port389.org/docs/389ds/design/cache_redesign.html has transactional semantics, or if such code could be added to its 
implementation.

With LMDB, if we could make the on-disk entry representation the same as the in-memory entry representation, then we could use LMDB as the entry cache too - the database would be the entry 
cache as well.
> > > > If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.
> > >
> > > It’s pretty massive as a change - if we want to do it right. I’d say we need:
> > > development and testing of a MVCC/COW cache implementation (proof that it really really works transactionally)
> > > allow “disable/disconnect” of the entry cache, but with the higher level txn’s so that we can prove the txn semantics are correct
> > > re-architect our transaction calls so that they are “higher” up. An example is that internal_modify shouldn’t start a txn, it should be given the current txn state as an arg. Combined with the above, we can prove we haven’t corrupted our server transaction guarantees.
> > > integrate the transactional cache.
> > >
> > > I don’t know if I would still write a transactional cache the same way as I proposed in that design, but I think the ideas are on the right path.
> > > > And what impact would changing the entry cache have on Ludwig's plugable backend work?
> > >
> > > Should be none, it’s seperate layers. If anything this change is going to make Ludwig’s work better because our current model won’t really take good advantage of the MVCC nature of modern kv stores.
> > > > Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
> > > >
> > > > Thanks,
> > > >
> > > > Mark
> > > > _______________________________________________
> > > > 389-devel mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> > >
> > > —
> > > Sincerely,
> > >
> > > William Brown
> > > Software Engineer, 389 Directory Server
> > > SUSE Labs
> > > _______________________________________________
> > > 389-devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> >
> > -- 
> > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> > Commercial register: Amtsgericht Muenchen, HRB 153243,
> > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
> > _______________________________________________
> > 389-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
>
> —
> Sincerely,
>
> William Brown
> Software Engineer, 389 Directory Server
> SUSE Labs
> _______________________________________________
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

[William Brown]

> On 27 Feb 2019, at 10:21, Rich Megginson  wrote:
>
> On 2/26/19 4:26 PM, William Brown wrote:
> > > On 26 Feb 2019, at 18:32, Ludwig Krispenz  wrote:
> > >
> > > Hi, I need a bit of time to read the docs and clear my thoughts, but one comment below
> > > On 02/25/2019 01:49 AM, William Brown wrote:
> > > > > On 23 Feb 2019, at 02:46, Mark Reynolds  wrote:
> > > > >
> > > > > I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
> > > > >
> > > > > For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
> > > > >
> > > > > It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful. Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
> > > > >
> > > > > William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
> > > > >
> > > > > But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?
> > > >
> > > > I like coffee? How cool are planes? My thoughts are simple :)
> > > >
> > > > I think there is a pretty simple mental simplification we can make here though. Nested transactions “don’t really exist”. We just have *recursive* operations inside of one transaction.
> > > >
> > > > Once reframed like that, the entire situation becomes simpler. We have one thread in a write transaction that can have recursive/batched operations as required, which means that either “all operations succeed” or “none do”. Really, this is the behaviour we want anyway, and it’s the transaction model of LMDB and other kv stores that we could consider (wired tiger, sled in the future).
> > >
> > > I think the recursive/nested transaction on the database level are not the problem, we do this correctly already, either all or no change becomes persistent.
> > > What we do not manage is modifications we do in parallel on the in memory structure like the entry cache, changes to the EC are not managed by any txn and I do not see how any of the database txn models would help, they do not know about ec and can abort changes.
> > > We would need to incorporate the EC into a generic txn model, or have a way to flag ec entries as garbage for if a txn is aborted
> >
> > The issue is we allow parallel writes, which breaks the consistency guarantees of the EC anyway. LMDB won’t allow parallel writes (it’s single write - concurrent parallel readers), and most other modern kv stores take this approach too, so we should be remodelling our transactions to match this IMO. It will make the process of how we reason about the EC much much simpler I think.
>
> Some sort of in-memory data structure with fast lookup and transactional semantics (modify operations are stored as mvcc/cow so each read of the database with a given txn handle sees its own view of the ec, a txn commit updates the parent txn ec view, or the global ec view if no parent, from the copy, a txn abort deletes the txn's copy of the ec) is needed.  A quick google search turns up several hits.  I'm not sure if the B+Tree proposed at http://www.port389.org/docs/389ds/design/cache_redesign.html has transactional semantics, or if such code could be added to its implementation.

It does, this is a MVCC B+Tree implementation.
> With LMDB, if we could make the on-disk entry representation the same as the in-memory entry representation, then we could use LMDB as the entry cache too - the database would be the entry cache as well.

Yes, Ludwig has suggested this because it would remove the need for an Entry Cache at all.
> > > > > If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.
> > > >
> > > > It’s pretty massive as a change - if we want to do it right. I’d say we need:
> > > > development and testing of a MVCC/COW cache implementation (proof that it really really works transactionally)
> > > > allow “disable/disconnect” of the entry cache, but with the higher level txn’s so that we can prove the txn semantics are correct
> > > > re-architect our transaction calls so that they are “higher” up. An example is that internal_modify shouldn’t start a txn, it should be given the current txn state as an arg. Combined with the above, we can prove we haven’t corrupted our server transaction guarantees.
> > > > integrate the transactional cache.
> > > >
> > > > I don’t know if I would still write a transactional cache the same way as I proposed in that design, but I think the ideas are on the right path.
> > > > > And what impact would changing the entry cache have on Ludwig's plugable backend work?
> > > >
> > > > Should be none, it’s seperate layers. If anything this change is going to make Ludwig’s work better because our current model won’t really take good advantage of the MVCC nature of modern kv stores.
> > > > > Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mark
> > > > > _______________________________________________
> > > > > 389-devel mailing list -- [e-mail]
> > > > > To unsubscribe send an email to [e-mail]
> > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> > > >
> > > > —
> > > > Sincerely,
> > > >
> > > > William Brown
> > > > Software Engineer, 389 Directory Server
> > > > SUSE Labs
> > > > _______________________________________________
> > > > 389-devel mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> > >
> > > -- 
> > > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> > > Commercial register: Amtsgericht Muenchen, HRB 153243,
> > > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
> > > _______________________________________________
> > > 389-devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
> >
> > —
> > Sincerely,
> >
> > William Brown
> > Software Engineer, 389 Directory Server
> > SUSE Labs
> > _______________________________________________
> > 389-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
>
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

—
Sincerely,

William Brown
Software Engineer, 389 Directory Server
SUSE Labs

[Mark Reynolds]

On 2/22/19 11:46 AM, Mark Reynolds wrote:
> I want to start a brief discussion about a major problem we have 
> backend transaction plugins and the entry caches.  I'm finding that 
> when we get into a nested state of be txn plugins and one of the later 
> plugins that is called fails then while we don't commit the disk 
> changes (they are aborted/rolled back) we DO keep the entry cache 
> changes!
>
> For example, a modrdn operation triggers the referential integrity 
> plugin which renames the member attribute in some group and changes 
> that group's entry cache entry, but then later on the memberOf plugin 
> fails for some reason.  The database transaction is aborted, but the 
> entry cache changes that RI plugin did are still present :-(  I have 
> also found other entry cache issues with modrdn and BE TXN plugins, 
> and we know of other currently non-reproducible entry cache crashes as 
> well related to mishandling of cache entries after failed operations.
>
> It's time to rework how we use the entry cache.  We basically need a 
> transaction style caching mechanism - we should not commit any entry 
> cache changes until the original operation is fully successful.  
> Unfortunately the way the entry cache is currently designed and used 
> it will be a major change to try to change it.
>
> William wrote up this doc: 
> http://www.port389.org/docs/389ds/design/cache_redesign.html
>
> But this also does not currently cover the nested plugin scenario 
> either (not yet).  I do know how how difficult it would be to 
> implement William's proposal, or how difficult it would be to 
> incorporate the txn style caching into his design.  What kind of time 
> frame could this even be implemented in?  William what are your thoughts?
>
> If William's design is too huge of a change that will take too long to 
> safely implement then perhaps we need to look into revising the 
> existing cache design where we use "cache_add_tentative" style 
> functions and only apply them at the end of the op.  This is also not 
> a trivial change.
>
> And what impact would changing the entry cache have on Ludwig's 
> plugable backend work?
>
> Anyway we need to start thinking about redesigning the entry cache - 
> no matter what approach we want to take.  If anyone has any ideas or 
> comments please share them, but I think due to the severity of this 
> flaw redesigning the entry cache should be one of our next major goals 
> in DS (1.4.1?).

We are actually seeing more of these cases popping up now, so we need to 
do something soon.  I had proposed we could always just flush the entire 
cache when a backend txn op fails, but Ludwig had a much better idea 
that we could implement a type of csn in the entry cache.  So when a 
backend txn plugin fails, we flush the entry cache entries with a csn >= 
start of the parent operation.

So until LMDB or a new caching mechanism is implemented this could be a 
viable/realistic option.

Mark
> Thanks,
>
> Mark
> _______________________________________________
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/[e-mail]....

[William Brown]

> On 5 Mar 2019, at 08:30, Mark Reynolds  wrote:
>
> On 2/22/19 11:46 AM, Mark Reynolds wrote:
> > I want to start a brief discussion about a major problem we have backend transaction plugins and the entry caches.  I'm finding that when we get into a nested state of be txn plugins and one of the later plugins that is called fails then while we don't commit the disk changes (they are aborted/rolled back) we DO keep the entry cache changes!
> >
> > For example, a modrdn operation triggers the referential integrity plugin which renames the member attribute in some group and changes that group's entry cache entry, but then later on the memberOf plugin fails for some reason.  The database transaction is aborted, but the entry cache changes that RI plugin did are still present :-(  I have also found other entry cache issues with modrdn and BE TXN plugins, and we know of other currently non-reproducible entry cache crashes as well related to mishandling of cache entries after failed operations.
> >
> > It's time to rework how we use the entry cache.  We basically need a transaction style caching mechanism - we should not commit any entry cache changes until the original operation is fully successful.  Unfortunately the way the entry cache is currently designed and used it will be a major change to try to change it.
> >
> > William wrote up this doc: http://www.port389.org/docs/389ds/design/cache_redesign.html
> >
> > But this also does not currently cover the nested plugin scenario either (not yet).  I do know how how difficult it would be to implement William's proposal, or how difficult it would be to incorporate the txn style caching into his design.  What kind of time frame could this even be implemented in?  William what are your thoughts?
> >
> > If William's design is too huge of a change that will take too long to safely implement then perhaps we need to look into revising the existing cache design where we use "cache_add_tentative" style functions and only apply them at the end of the op.  This is also not a trivial change.
> >
> > And what impact would changing the entry cache have on Ludwig's plugable backend work?
> >
> > Anyway we need to start thinking about redesigning the entry cache - no matter what approach we want to take.  If anyone has any ideas or comments please share them, but I think due to the severity of this flaw redesigning the entry cache should be one of our next major goals in DS (1.4.1?).
>
> We are actually seeing more of these cases popping up now, so we need to do something soon.  I had proposed we could always just flush the entire cache when a backend txn op fails, but Ludwig had a much better idea that we could implement a type of csn in the entry cache.  So when a backend txn plugin fails, we flush the entry cache entries with a csn >= start of the parent operation.
>
> So until LMDB or a new caching mechanism is implemented this could be a viable/realistic option.

Well, the cache currently works by having a version number (I think watermark?) in the cache. That’s how old content is removed (entries < watermark). So perhaps we could get the watermark id at the start of an operation and anything where watermark >= current op, we flush on rc != 0.

(disclaimer, this is from my memory and may not represent real cache behaviour, so could be wildly wrong).
> Mark
> > Thanks,
> >
> > Mark
> > _______________________________________________
> > 389-devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....
>
> 389-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]....

—
Sincerely,

William Brown
Software Engineer, 389 Directory Server
SUSE Labs