[PATCH] prevent slapd from hanging under unlikely circumstances

[Jay Fenlason]

I have a small FreeIPA deployment of ~6-8 servers running on Centos
7.7.  Do to the addition and removal of some of the servers, some
cruft (tombstones, replication conflicts, etc) have crept in to the
directory.  I noticed that when I attempted to delete some of the
cruft entries, ns-slapd would hang, failirg to process requests, or
even shut down.  In my investigation I found that pre-write callback
of the plugins (including the nis plugin) were being called, but no
corresponding post-write call was happening.  The nis plugin takes a
write lock in the pre-write callback and releases it in the post-write
callback.  Never releasing the lock causes the next attempt to take it
to hang.  I wrote the attached patch, which simply ensures the
post-write callback is always called if the pre-write callback was
called.

This is such a simple patch that it scarcely needs a signoff.
Nonetheless:

Signed-off-by: Jay Fenlason <ds389(a)nerd-marrow.com&gt;

--- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig	2020-01-31 07:28:04.085861174 -0500
+++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c	2020-01-31 07:30:33.932947489 -0500
@@ -81,6 +81,7 @@
     Connection *pb_conn;
     int32_t parent_op = 0;
     struct timespec parent_time;
+    int pre_delete_called = 0;
 
     if (slapi_pblock_get(pb, SLAPI_CONN_ID, &conn_id) < 0) {
         conn_id = 0; /* connection is NULL */
@@ -371,6 +372,7 @@
                 }
                 if (retval == 0) {
                     retval = plugin_call_plugins(pb, SLAPI_PLUGIN_BE_PRE_DELETE_FN);
+		    pre_delete_called = 1;
                 }
                 if (retval)
                 {
@@ -1491,7 +1493,7 @@
      * The bepostop is called even if the operation fails,
      * but not if the operation is purging tombstones.
      */
-    if (!delete_tombstone_entry) {
+    if (pre_delete_called) {
         plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
     }
     /* Need to return to cache after post op plugins are called */

[Mark Reynolds]

On 1/31/20 9:10 PM, Jay Fenlason wrote:
>  I have a small FreeIPA deployment of ~6-8 servers running on Centos
>  7.7.  Do to the addition and removal of some of the servers, some
>  cruft (tombstones, replication conflicts, etc) have crept in to the
>  directory.  I noticed that when I attempted to delete some of the
>  cruft entries, ns-slapd would hang, failirg to process requests, or
>  even shut down.  In my investigation I found that pre-write callback
>  of the plugins (including the nis plugin) were being called, but no
>  corresponding post-write call was happening.  The nis plugin takes a
>  write lock in the pre-write callback and releases it in the post-write
>  callback.  Never releasing the lock causes the next attempt to take it
>  to hang.  I wrote the attached patch, which simply ensures the
>  post-write callback is always called if the pre-write callback was
>  called. 
I need to look at this closer, but I would suspect you should not ignore 
the check for delete_tombstone_entry:

+    if (pre_delete_called) {

To

+    if (pre_delete_called || !delete_tombstone_entry) {


Does it still resolve your problem?  Probably, but best to check...

Like I said this needs a closer investigation/testing to make sure it does not cause any
regressions.  Do you have a stack trace from the hang you encountered?  If not, could you
get one for us?

Thanks,
Mark

>  This is such a simple patch that it scarcely needs a signoff.
>  Nonetheless:
>
>  Signed-off-by: Jay Fenlason <ds389(a)nerd-marrow.com&gt;
>
>  --- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig	2020-01-31 07:28:04.085861174
> -0500
>  +++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c	2020-01-31 07:30:33.932947489 -0500
>  @@ -81,6 +81,7 @@
>        Connection *pb_conn;
>        int32_t parent_op = 0;
>        struct timespec parent_time;
>  +    int pre_delete_called = 0;
>    
>        if (slapi_pblock_get(pb, SLAPI_CONN_ID, &conn_id) < 0) {
>            conn_id = 0; /* connection is NULL */
>  @@ -371,6 +372,7 @@
>                    }
>                    if (retval == 0) {
>                        retval = plugin_call_plugins(pb, SLAPI_PLUGIN_BE_PRE_DELETE_FN);
>  +		    pre_delete_called = 1;
>                    }
>                    if (retval)
>                    {
>  @@ -1491,7 +1493,7 @@
>         * The bepostop is called even if the operation fails,
>         * but not if the operation is purging tombstones.
>         */
>  -    if (!delete_tombstone_entry) {
>  +    if (pre_delete_called) {
>            plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
>        }
>        /* Need to return to cache after post op plugins are called */
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 

389 Directory Server Development Team

[Jay Fenlason]

Here's a backtrace of the hung server:
All of the other threads are in pthread_cond_wait(), select() or poll()

#0  0x00007fce1454f35e in pthread_rwlock_wrlock ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:85
#1  0x00007fce16e1acca in slapi_rwlock_wrlock (rwlock=<optimized out>)
    at ldap/servers/slapd/slapi2nspr.c:237
#2  0x00007fce06a74748 in wrap_rwlock_wrlock (rwlock=<optimized out>)
    at wrap.c:328
#3  0x00007fce06a7344c in plugin_wrlock () at map.c:1242
#4  0x00007fce06a5d639 in backend_be_pre_write_cb (pb=<optimized out>)
    at back-sch.c:2381
#5  0x00007fce16df8028 in plugin_call_func (list=0x55ddade18840,
operation=operation@entry=453, pb=pb@entry=0x55ddaf08afc0, call_one=call_one@entry=0)
    at ldap/servers/slapd/plugin.c:2028
#6  0x00007fce16df82e3 in plugin_call_list (pb=0x55ddaf08afc0, operation=453,
list=<optimized out>) at ldap/servers/slapd/plugin.c:1972
#7  0x00007fce16df82e3 in plugin_call_plugins (pb=pb@entry=0x55ddaf08afc0,
whichfunction=whichfunction@entry=453) at ldap/servers/slapd/plugin.c:442
#8  0x00007fce08a31c88 in ldbm_back_delete (pb=0x55ddaf08afc0)
    at ldap/servers/slapd/back-ldbm/ldbm_delete.c:373
#9  0x00007fce16da8feb in op_shared_delete (pb=pb@entry=0x55ddaf08afc0)
    at ldap/servers/slapd/delete.c:324
#10 0x00007fce16da9383 in do_delete (pb=pb@entry=0x55ddaf08afc0)
    at ldap/servers/slapd/delete.c:97
#11 0x000055ddac64e62c in connection_dispatch_operation (pb=0x55ddaf08afc0,
op=0x55ddadd17340, conn=0x55ddaf12ed00) at ldap/servers/slapd/connection.c:615
#12 0x000055ddac64e62c in connection_threadmain ()
    at ldap/servers/slapd/connection.c:1790
#13 0x00007fce14babc5b in _pt_root (arg=0x55ddaf0186c0)
    at ../../../nspr/pr/src/pthreads/ptthread.c:201
#14 0x00007fce1454be65 in start_thread (arg=0x7fcdf405d700)
    at pthread_create.c:307
#15 0x00007fce13bf788d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111



My thought when writing the patch was that calls to the pre delete callback
and the post delete callback should be balanced: either both of them should be
called or neither.  But you know the plugin abi better than I do, and if
you think it's ok to have the post callback called without a corresponding pre
who am I to argue?  I've just confirmed that this patch also prevents
the hangs I'm seeing:

--- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig	2020-01-31 07:28:04.085861174 -0500
+++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c	2020-01-31 07:30:33.932947489 -0500
@@ -81,6 +81,7 @@
     Connection *pb_conn;
     int32_t parent_op = 0;
     struct timespec parent_time;
+    int pre_delete_called = 0;
 
     if (slapi_pblock_get(pb, SLAPI_CONN_ID, &conn_id) < 0) {
         conn_id = 0; /* connection is NULL */
@@ -371,6 +372,7 @@
                 }
                 if (retval == 0) {
                     retval = plugin_call_plugins(pb, SLAPI_PLUGIN_BE_PRE_DELETE_FN);
+		    pre_delete_called = 1;
                 }
                 if (retval)
                 {
@@ -1491,7 +1493,7 @@
      * The bepostop is called even if the operation fails,
      * but not if the operation is purging tombstones.
      */
-    if (!delete_tombstone_entry) {
+    if (!delete_tombstone_entry || pre_delete_called) {
         plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
     }
     /* Need to return to cache after post op plugins are called */


Signed-off-by: Jay Fenlason <ds389(a)nerd-marrow.com&gt;

[Ludwig Krispenz]

Hi,

I agree with you that calls to pre and post should be balance, but not 
sure if your approach is the correct one. There is a condition for post 
"!delete_tombstone_entries" which prevented the call for postop plugins 
in case of the deletion of a tombstone entry. Your patch now ensures 
that it will be called in this cases. I think the correct way would be 
to also prevent the calls to preop in case of delete_tombstone_entry, 
this is an operation to remove replication artifacts and should not be 
known by plugins

Regards,
Ludwig

On 02/03/2020 04:57 PM, Jay Fenlason wrote:
>  Here's a backtrace of the hung server:
>  All of the other threads are in pthread_cond_wait(), select() or poll()
>
>  #0  0x00007fce1454f35e in pthread_rwlock_wrlock ()
>       at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:85
>  #1  0x00007fce16e1acca in slapi_rwlock_wrlock (rwlock=<optimized out>)
>       at ldap/servers/slapd/slapi2nspr.c:237
>  #2  0x00007fce06a74748 in wrap_rwlock_wrlock (rwlock=<optimized out>)
>       at wrap.c:328
>  #3  0x00007fce06a7344c in plugin_wrlock () at map.c:1242
>  #4  0x00007fce06a5d639 in backend_be_pre_write_cb (pb=<optimized out>)
>       at back-sch.c:2381
>  #5  0x00007fce16df8028 in plugin_call_func (list=0x55ddade18840,
> operation=operation@entry=453, pb=pb@entry=0x55ddaf08afc0, call_one=call_one@entry=0)
>       at ldap/servers/slapd/plugin.c:2028
>  #6  0x00007fce16df82e3 in plugin_call_list (pb=0x55ddaf08afc0, operation=453,
> list=<optimized out>) at ldap/servers/slapd/plugin.c:1972
>  #7  0x00007fce16df82e3 in plugin_call_plugins (pb=pb@entry=0x55ddaf08afc0,
> whichfunction=whichfunction@entry=453) at ldap/servers/slapd/plugin.c:442
>  #8  0x00007fce08a31c88 in ldbm_back_delete (pb=0x55ddaf08afc0)
>       at ldap/servers/slapd/back-ldbm/ldbm_delete.c:373
>  #9  0x00007fce16da8feb in op_shared_delete (pb=pb@entry=0x55ddaf08afc0)
>       at ldap/servers/slapd/delete.c:324
>  #10 0x00007fce16da9383 in do_delete (pb=pb@entry=0x55ddaf08afc0)
>       at ldap/servers/slapd/delete.c:97
>  #11 0x000055ddac64e62c in connection_dispatch_operation (pb=0x55ddaf08afc0,
> op=0x55ddadd17340, conn=0x55ddaf12ed00) at ldap/servers/slapd/connection.c:615
>  #12 0x000055ddac64e62c in connection_threadmain ()
>       at ldap/servers/slapd/connection.c:1790
>  #13 0x00007fce14babc5b in _pt_root (arg=0x55ddaf0186c0)
>       at ../../../nspr/pr/src/pthreads/ptthread.c:201
>  #14 0x00007fce1454be65 in start_thread (arg=0x7fcdf405d700)
>       at pthread_create.c:307
>  #15 0x00007fce13bf788d in clone ()
>       at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>
>
>
>  My thought when writing the patch was that calls to the pre delete callback
>  and the post delete callback should be balanced: either both of them should be
>  called or neither.  But you know the plugin abi better than I do, and if
>  you think it's ok to have the post callback called without a corresponding pre
>  who am I to argue?  I've just confirmed that this patch also prevents
>  the hangs I'm seeing:
>
>  --- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig	2020-01-31 07:28:04.085861174
> -0500
>  +++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c	2020-01-31 07:30:33.932947489 -0500
>  @@ -81,6 +81,7 @@
>        Connection *pb_conn;
>        int32_t parent_op = 0;
>        struct timespec parent_time;
>  +    int pre_delete_called = 0;
>    
>        if (slapi_pblock_get(pb, SLAPI_CONN_ID, &conn_id) < 0) {
>            conn_id = 0; /* connection is NULL */
>  @@ -371,6 +372,7 @@
>                    }
>                    if (retval == 0) {
>                        retval = plugin_call_plugins(pb, SLAPI_PLUGIN_BE_PRE_DELETE_FN);
>  +		    pre_delete_called = 1;
>                    }
>                    if (retval)
>                    {
>  @@ -1491,7 +1493,7 @@
>         * The bepostop is called even if the operation fails,
>         * but not if the operation is purging tombstones.
>         */
>  -    if (!delete_tombstone_entry) {
>  +    if (!delete_tombstone_entry || pre_delete_called) {
>            plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
>        }
>        /* Need to return to cache after post op plugins are called */
>
>
>  Signed-off-by: Jay Fenlason <ds389(a)nerd-marrow.com&gt;
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
Handelsregister: Amtsgericht München, HRB 153243,
Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage

[William Brown]

>  On 1 Feb 2020, at 12:10, Jay Fenlason <ds389(a)nerd-marrow.com&gt;
> wrote:
>  
>  I have a small FreeIPA deployment of ~6-8 servers running on Centos
>  7.7.  Do to the addition and removal of some of the servers, some
>  cruft (tombstones, replication conflicts, etc) have crept in to the
>  directory.  I noticed that when I attempted to delete some of the
>  cruft entries, ns-slapd would hang, failirg to process requests, or
>  even shut down. 
Can you tell us exactly what entries you noticed and how you attempted to delete them?
There are certainly some things like tombstones and such that you shouldn't be
touching as they are part of the internal replication state machine.

Knowing what you did will also help us to create a test case and reproducers to validate
your patch also.

Thanks for the report :) 


>   In my investigation I found that pre-write callback
>  of the plugins (including the nis plugin) were being called, but no
>  corresponding post-write call was happening.  The nis plugin takes a
>  write lock in the pre-write callback and releases it in the post-write
>  callback.  Never releasing the lock causes the next attempt to take it
>  to hang.  I wrote the attached patch, which simply ensures the
>  post-write callback is always called if the pre-write callback was
>  called.
>  
>  This is such a simple patch that it scarcely needs a signoff.
>  Nonetheless:
>  
>  Signed-off-by: Jay Fenlason <ds389(a)nerd-marrow.com&gt;
>  
>  --- a/ldap/servers/slapd/back-ldbm/ldbm_delete.c.orig	2020-01-31 07:28:04.085861174
> -0500
>  +++ b/ldap/servers/slapd/back-ldbm/ldbm_delete.c	2020-01-31 07:30:33.932947489 -0500
>  @@ -81,6 +81,7 @@
>      Connection *pb_conn;
>      int32_t parent_op = 0;
>      struct timespec parent_time;
>  +    int pre_delete_called = 0;
>  
>      if (slapi_pblock_get(pb, SLAPI_CONN_ID, &conn_id) < 0) {
>          conn_id = 0; /* connection is NULL */
>  @@ -371,6 +372,7 @@
>                  }
>                  if (retval == 0) {
>                      retval = plugin_call_plugins(pb, SLAPI_PLUGIN_BE_PRE_DELETE_FN);
>  +		    pre_delete_called = 1;
>                  }
>                  if (retval)
>                  {
>  @@ -1491,7 +1493,7 @@
>       * The bepostop is called even if the operation fails,
>       * but not if the operation is purging tombstones.
>       */
>  -    if (!delete_tombstone_entry) {
>  +    if (pre_delete_called) {
>          plugin_call_plugins(pb, SLAPI_PLUGIN_BE_POST_DELETE_FN);
>      }
>      /* Need to return to cache after post op plugins are called */
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Jay Fenlason]

On Mon, Feb 03, 2020 at 10:38:59AM +1000, William Brown wrote:
>  
>  
>  > On 1 Feb 2020, at 12:10, Jay Fenlason <ds389(a)nerd-marrow.com&gt; wrote:
>  > 
>  > I have a small FreeIPA deployment of ~6-8 servers running on Centos
>  > 7.7.  Do to the addition and removal of some of the servers, some
>  > cruft (tombstones, replication conflicts, etc) have crept in to the
>  > directory.  I noticed that when I attempted to delete some of the
>  > cruft entries, ns-slapd would hang, failirg to process requests, or
>  > even shut down. 
>  Can you tell us exactly what entries you noticed and how you
> attempted to delete them? There are certainly some things like tombstones and such that
> you shouldn't be touching as they are part of the internal replication state machine.
No, I don't remember what entries they were.  I was following
instructions from:
https://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/ipa-rep...
(or maybe elsewhere) using ldapdelete to remove tombstones for a truly
deleted server.

>  Knowing what you did will also help us to create a test case and
>  reproducers to validate your patch also. 
I found the bug by doing a series of "ipa-client-install" (with lots
of arguments, followed by
echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
echo [global] > /etc/ipa/installer.conf
echo ca_host = {ditto} >> /etc/ipa/installer.conf
echo {password} | kinit admin
ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}

followed by the replica install failing due to network issues,
misconfigured firewalls, etc, then
ipa-server-install --uninstall on the host
and ipa-replica-manage del {failed install host}
elsewhere in the mesh, sometimes with ldapdelete of the initial
replication agreement that ipa-replica-manage did not remove.

Rinse, repeat. . .

Until ipa-replica-install starts failing because the source LDAP
server hangs (because of this bug) during the "starting initial
replication" step.  It was while debugging that that I discovered that
ldapdelete on the tombstone entries also caused the LDAP servers to
lock up.


>  Thanks for the report :)  
Incidentally, there's another bug, which I have not investigated,
where attempting to ldapdelete a problematic tombstone entry
immediately after restarting the LDAP server returns an error, and
nothing is deleted on the server.  If you do an ldapsearch, and then
an ldapdelete, the entry is removed, but then slapd hangs (this bug
again) and does not respond to searches or deletes (or shutdown
requests) until you kill -9 it.  I don't know how it relates to this
bug.

    -- JF

[William Brown]

>  On 3 Feb 2020, at 23:43, Jay Fenlason <ds389(a)nerd-marrow.com&gt;
> wrote:
>  
>  On Mon, Feb 03, 2020 at 10:38:59AM +1000, William Brown wrote:
> > 
> > 
> >> On 1 Feb 2020, at 12:10, Jay Fenlason <ds389(a)nerd-marrow.com&gt; wrote:
> >> 
> >> I have a small FreeIPA deployment of ~6-8 servers running on Centos
> >> 7.7.  Do to the addition and removal of some of the servers, some
> >> cruft (tombstones, replication conflicts, etc) have crept in to the
> >> directory.  I noticed that when I attempted to delete some of the
> >> cruft entries, ns-slapd would hang, failirg to process requests, or
> >> even shut down.
>  
> > Can you tell us exactly what entries you noticed and how you attempted to delete
> them? There are certainly some things like tombstones and such that you shouldn't be
> touching as they are part of the internal replication state machine.
>  
>  No, I don't remember what entries they were.  I was following
>  instructions from:
>
> https://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/ipa-rep...
>  (or maybe elsewhere) using ldapdelete to remove tombstones for a truly
>  deleted server. 
I think this advice is quite out dated now, we have potentially got better tools to handle
this, but that's a really difficult issue to manage content ownership, management, and
getting google to show the latest content etc .... 

>  
> > Knowing what you did will also help us to create a test case and
> > reproducers to validate your patch also.
>  
>  I found the bug by doing a series of "ipa-client-install" (with lots
>  of arguments, followed by
>  echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
>  echo [global] > /etc/ipa/installer.conf
>  echo ca_host = {ditto} >> /etc/ipa/installer.conf
>  echo {password} | kinit admin
>  ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
>  ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
>  
>  followed by the replica install failing due to network issues,
>  misconfigured firewalls, etc, then
>  ipa-server-install --uninstall on the host
>  and ipa-replica-manage del {failed install host}
>  elsewhere in the mesh, sometimes with ldapdelete of the initial
>  replication agreement that ipa-replica-manage did not remove.
>  
>  Rinse, repeat. . .
>  
>  Until ipa-replica-install starts failing because the source LDAP
>  server hangs (because of this bug) during the "starting initial
>  replication" step.  It was while debugging that that I discovered that
>  ldapdelete on the tombstone entries also caused the LDAP servers to
>  lock up.
>  
>  
> > Thanks for the report :) 
>  
>  Incidentally, there's another bug, which I have not investigated,
>  where attempting to ldapdelete a problematic tombstone entry
>  immediately after restarting the LDAP server returns an error, and
>  nothing is deleted on the server.  If you do an ldapsearch, and then
>  an ldapdelete, the entry is removed, but then slapd hangs (this bug
>  again) and does not respond to searches or deletes (or shutdown
>  requests) until you kill -9 it.  I don't know how it relates to this
>  bug. 
So I think that deleting the tombstones is not the correct (or valid) course of action
here. Tombstones are a really important part of the replication lifecycle, so if anything
we need to taker stronger steps to prevent a client from being able to delete them at all.
This makes me question the patch you have provided, because you shouldn't be in a
position to delete tombstones in the first place, only the server internally should be
purging (deleting) these when replication is known to be in a consistent state. I am happy
to explain the functionality of tombstones further if you are interested. 

Deleting a conflict entry however, is just fine, so that shouldn't have caused the
issue. 

I wonder if a contributing factor here is if ipa-replica-install is re-using replica ids,
which could cause replication to have a problem.

Perhaps the solution here is to have ipa-replica-install to attempt a cleanallruv on any
replica id it's *about* to try to use, in case it has been re-used. 

My thinking at this point is that there is something else going on, and the issue may
reside in a series of interactions in the ipa replica steps you have taken. Have you
contacted the freeipa-users group about this at all? 

>  
>     -- JF
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Jay Fenlason]

Ok, let's take this from the top:

1: Defects that cause a server to become unresponsive are bad and must
be repaired as soon as possible.

2: Some #1 class defects are exploitable and require a CVE.  As far as
I can tell, this one does not, but you should keep an eye out for the
possibility.

3: The #1 class defect I have found can be triggered in at least two
ways.  One requires ipa-replica-install and is hard to reproduce in a
test environment.  The other requires ldapdelete and is easy to
reproduce in an isolated VM.

4: The defect is a mismatch between the plugin ABI as implemented by
389-ds, and the behavior the NIS plugin expects.

5: I have found no specification or documentation on said ABI, so I
can only guess what the correct behavior is here.

6: The ABI includes two functions in the plugin: pre_delete and
post_delete.

7: The NIS plugin expects that every call to pre_delete will be
followed by a call to post_delete.  It takes a lock in pre_delete and
releases it in post_delete.

8: Under some circumstances 389-ds can call pre_delete but fail to
call post_delete.  Because of #5, there is no way to tell if this is
expected behavior, but the NIS plugin clearly does not expect it.

9: My patch ensures that every call to pre_delete is followed by a
corresponding call to post_delete.  V1 of the patch also ensures that
every call to post_delete is after a corresponding pre_delete call.
V2 relaxes the second requirement, allowing post_delete calls without
a corresponding pre_delete call because someone expressed worry about
potential regressions.

10: You are refusing to merge my patch because of a doctrinal
objection to the use of ldapdelete in the simple reproducer.

[Ludwig Krispenz]

I think I can agree with 1-8, 9 is one solution to fix the problem you 
reported, but not yet validate that there are no other side effects, 
there are potential postop plugins which should NOT be called for 
tombstone delete, eg retro cl, we need to investigate side effects of 
the patch and evaluate other solutions, I suggested one in anotehr reply.

for 10, the claim that not crying hurrah and merging a patch without 
further investigation and testing is "doctrinal objection" is very 
strong and I have to reject this.

Regards,
Ludwig

On 02/04/2020 05:00 PM, Jay Fenlason wrote:
>  Ok, let's take this from the top:
>
>  1: Defects that cause a server to become unresponsive are bad and must
>  be repaired as soon as possible.
>
>  2: Some #1 class defects are exploitable and require a CVE.  As far as
>  I can tell, this one does not, but you should keep an eye out for the
>  possibility.
>
>  3: The #1 class defect I have found can be triggered in at least two
>  ways.  One requires ipa-replica-install and is hard to reproduce in a
>  test environment.  The other requires ldapdelete and is easy to
>  reproduce in an isolated VM.
>
>  4: The defect is a mismatch between the plugin ABI as implemented by
>  389-ds, and the behavior the NIS plugin expects.
>
>  5: I have found no specification or documentation on said ABI, so I
>  can only guess what the correct behavior is here.
>
>  6: The ABI includes two functions in the plugin: pre_delete and
>  post_delete.
>
>  7: The NIS plugin expects that every call to pre_delete will be
>  followed by a call to post_delete.  It takes a lock in pre_delete and
>  releases it in post_delete.
>
>  8: Under some circumstances 389-ds can call pre_delete but fail to
>  call post_delete.  Because of #5, there is no way to tell if this is
>  expected behavior, but the NIS plugin clearly does not expect it.
>
>  9: My patch ensures that every call to pre_delete is followed by a
>  corresponding call to post_delete.  V1 of the patch also ensures that
>  every call to post_delete is after a corresponding pre_delete call.
>  V2 relaxes the second requirement, allowing post_delete calls without
>  a corresponding pre_delete call because someone expressed worry about
>  potential regressions.
>
>  10: You are refusing to merge my patch because of a doctrinal
>  objection to the use of ldapdelete in the simple reproducer.
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
Handelsregister: Amtsgericht München, HRB 153243,
Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage

[William Brown]

>  On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> wrote:
>  
>  I think I can agree with 1-8, 9 is one solution to fix the problem you reported, but not
> yet validate that there are no other side effects, there are potential postop plugins
> which should NOT be called for tombstone delete, eg retro cl, we need to investigate side
> effects of the patch and evaluate other solutions, I suggested one in anotehr reply.
>  
>  for 10, the claim that not crying hurrah and merging a patch without further
> investigation and testing is "doctrinal objection" is very strong and I have to
> reject this.
>  
>  Regards,
>  Ludwig
>  
>  On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> > Ok, let's take this from the top:
> > 
> > 1: Defects that cause a server to become unresponsive are bad and must
> > be repaired as soon as possible. 
I'm not objecting to this.

> > 
> > 2: Some #1 class defects are exploitable and require a CVE.  As far as
> > I can tell, this one does not, but you should keep an eye out for the
> > possibility.
> > 
> > 3: The #1 class defect I have found can be triggered in at least two
> > ways.  One requires ipa-replica-install and is hard to reproduce in a
> > test environment.  The other requires ldapdelete and is easy to
> > reproduce in an isolated VM. 
Ipa replica install and ldapdelete of a tombstone/conflict both require cn=directory
manager, which would automatically cause any reported CVE to be void - cn=directory
manager is game over. 

> > 
> > 4: The defect is a mismatch between the plugin ABI as implemented by
> > 389-ds, and the behavior the NIS plugin expects.
> > 
> > 5: I have found no specification or documentation on said ABI, so I
> > can only guess what the correct behavior is here.
> > 
> > 6: The ABI includes two functions in the plugin: pre_delete and
> > post_delete.
> > 
> > 7: The NIS plugin expects that every call to pre_delete will be
> > followed by a call to post_delete.  It takes a lock in pre_delete and
> > releases it in post_delete. 
But you didn't report an issue in NIS? You reported an issue with ldapdelete on
tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa and if an issue
in it exists, please report it to them with reproduction steps. 

> > 
> > 8: Under some circumstances 389-ds can call pre_delete but fail to
> > call post_delete.  Because of #5, there is no way to tell if this is
> > expected behavior, but the NIS plugin clearly does not expect it.
> > 
> > 9: My patch ensures that every call to pre_delete is followed by a
> > corresponding call to post_delete.  V1 of the patch also ensures that
> > every call to post_delete is after a corresponding pre_delete call.
> > V2 relaxes the second requirement, allowing post_delete calls without
> > a corresponding pre_delete call because someone expressed worry about
> > potential regressions.
> > 
> > 10: You are refusing to merge my patch because of a doctrinal
> > objection to the use of ldapdelete in the simple reproducer. 
It's not really a doctrinal objection. Replication is a really complex topic, and
difficult to get correct. Ludwig knows a lot in this area, and has really good comments to
make on the topic. I understand that you have an issue you have found in your setup, and
you want to resolve it, but we have to consider the deployments of many thousands of other
users and their replication experience too. For us it's important to be able to
reproduce, and see the issue, to understand the consequences, and to make these changes
carefully. Accepting the patch without clear understanding of it's consequences is not
something we do.

At this time we still don't have a clear reproducer from you on "how" you
constructed the invalid directory state. You reported the following:

>  I found the bug by doing a series of "ipa-client-install"
> (with lots
>  of arguments, followed by
>  echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
>  echo [global] > /etc/ipa/installer.conf
>  echo ca_host = {ditto} >> /etc/ipa/installer.conf
>  echo {password} | kinit admin
>  ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
>  ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
>  
>  followed by the replica install failing due to network issues,
>  misconfigured firewalls, etc, then
>  ipa-server-install --uninstall on the host
>  and ipa-replica-manage del {failed install host}
>  elsewhere in the mesh, sometimes with ldapdelete of the initial
>  replication agreement that ipa-replica-manage did not remove. 
I'm certainly not able to produce a reproduction case from these steps ... This is why
I have questioned what ipa-replica-install is doing in a previous email, and I think that
we need a better series of steps to attempt to reproduce, as well as better and clearer
information about "what" has crashed/hung/failed in the directories in these
states.

We still want to help, but please be patient with us as we want to ensure our fixes are
the highest possible quality.

Thanks, 

—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[thierry bordaz]

On 2/5/20 2:30 AM, William Brown wrote:
> > On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt; wrote:
> >
> > I think I can agree with 1-8, 9 is one solution to fix the problem you reported, but
> not yet validate that there are no other side effects, there are potential postop plugins
> which should NOT be called for tombstone delete, eg retro cl, we need to investigate side
> effects of the patch and evaluate other solutions, I suggested one in anotehr reply.
> >
> > for 10, the claim that not crying hurrah and merging a patch without further
> investigation and testing is "doctrinal objection" is very strong and I have to
> reject this.
> >
> > Regards,
> > Ludwig
> >
> > On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> >> Ok, let's take this from the top:
> >>
> >> 1: Defects that cause a server to become unresponsive are bad and must
> >> be repaired as soon as possible.
>  I'm not objecting to this.
>
> >> 2: Some #1 class defects are exploitable and require a CVE.  As far as
> >> I can tell, this one does not, but you should keep an eye out for the
> >> possibility.
> >>
> >> 3: The #1 class defect I have found can be triggered in at least two
> >> ways.  One requires ipa-replica-install and is hard to reproduce in a
> >> test environment.  The other requires ldapdelete and is easy to
> >> reproduce in an isolated VM.
>  Ipa replica install and ldapdelete of a tombstone/conflict both require cn=directory
> manager, which would automatically cause any reported CVE to be void - cn=directory
> manager is game over.
>
> >> 4: The defect is a mismatch between the plugin ABI as implemented by
> >> 389-ds, and the behavior the NIS plugin expects.
> >>
> >> 5: I have found no specification or documentation on said ABI, so I
> >> can only guess what the correct behavior is here.
> >>
> >> 6: The ABI includes two functions in the plugin: pre_delete and
> >> post_delete.
> >>
> >> 7: The NIS plugin expects that every call to pre_delete will be
> >> followed by a call to post_delete.  It takes a lock in pre_delete and
> >> releases it in post_delete.
>  But you didn't report an issue in NIS? You reported an issue with ldapdelete on
> tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa and if an issue
> in it exists, please report it to them with reproduction steps. 
Hi Jay,

Thanks for your great investigations/patch/summary.

I made the change in NIS plugin to acquire/release map lock in pre/post 
ops. At that time I was assuming the pre/post callback were balanced.
I was wrong and already hit similar issue in #1694263. I updated NIS 
plugin to workaround that bug.

The new bug you are reporting is a new example that pre/post are 
sometime not balanced :(
The proposed fixes (prevent preop call on tombstone, force postop call 
even on tombstone) looks valid but will have wider impact and need 
additional investigation.
An other approach would be to make NIS plugin ignore operations on 
tombstone. It also require additional investigation.

best regards
thierry


> >> 8: Under some circumstances 389-ds can call pre_delete but
> fail to
> >> call post_delete.  Because of #5, there is no way to tell if this is
> >> expected behavior, but the NIS plugin clearly does not expect it.
> >>
> >> 9: My patch ensures that every call to pre_delete is followed by a
> >> corresponding call to post_delete.  V1 of the patch also ensures that
> >> every call to post_delete is after a corresponding pre_delete call.
> >> V2 relaxes the second requirement, allowing post_delete calls without
> >> a corresponding pre_delete call because someone expressed worry about
> >> potential regressions.
> >>
> >> 10: You are refusing to merge my patch because of a doctrinal
> >> objection to the use of ldapdelete in the simple reproducer.
>  It's not really a doctrinal objection. Replication is a really complex topic, and
> difficult to get correct. Ludwig knows a lot in this area, and has really good comments to
> make on the topic. I understand that you have an issue you have found in your setup, and
> you want to resolve it, but we have to consider the deployments of many thousands of other
> users and their replication experience too. For us it's important to be able to
> reproduce, and see the issue, to understand the consequences, and to make these changes
> carefully. Accepting the patch without clear understanding of it's consequences is not
> something we do.
>
>  At this time we still don't have a clear reproducer from you on "how" you
> constructed the invalid directory state. You reported the following:
>
> > I found the bug by doing a series of "ipa-client-install" (with lots
> > of arguments, followed by
> > echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> > echo [global] > /etc/ipa/installer.conf
> > echo ca_host = {ditto} >> /etc/ipa/installer.conf
> > echo {password} | kinit admin
> > ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> > ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
> >
> > followed by the replica install failing due to network issues,
> > misconfigured firewalls, etc, then
> > ipa-server-install --uninstall on the host
> > and ipa-replica-manage del {failed install host}
> > elsewhere in the mesh, sometimes with ldapdelete of the initial
> > replication agreement that ipa-replica-manage did not remove.
>  I'm certainly not able to produce a reproduction case from these steps ... This is
> why I have questioned what ipa-replica-install is doing in a previous email, and I think
> that we need a better series of steps to attempt to reproduce, as well as better and
> clearer information about "what" has crashed/hung/failed in the directories in
> these states.
>
>  We still want to help, but please be patient with us as we want to ensure our fixes are
> the highest possible quality.
>
>  Thanks,
>
>  —
>  Sincerely,
>
>  William Brown
>
>  Senior Software Engineer, 389 Directory Server
>  SUSE Labs
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...

[Ludwig Krispenz]

On 02/05/2020 10:53 AM, thierry bordaz wrote:
>  On 2/5/20 2:30 AM, William Brown wrote:
> >
> >> On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt; wrote:
> >>
> >> I think I can agree with 1-8, 9 is one solution to fix the problem 
> >> you reported, but not yet validate that there are no other side 
> >> effects, there are potential postop plugins which should NOT be 
> >> called for tombstone delete, eg retro cl, we need to investigate 
> >> side effects of the patch and evaluate other solutions, I suggested 
> >> one in anotehr reply.
> >>
> >> for 10, the claim that not crying hurrah and merging a patch without 
> >> further investigation and testing is "doctrinal objection" is very 
> >> strong and I have to reject this.
> >>
> >> Regards,
> >> Ludwig
> >>
> >> On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> >>> Ok, let's take this from the top:
> >>>
> >>> 1: Defects that cause a server to become unresponsive are bad and must
> >>> be repaired as soon as possible.
> > I'm not objecting to this.
> >
> >>> 2: Some #1 class defects are exploitable and require a CVE.  As far as
> >>> I can tell, this one does not, but you should keep an eye out for the
> >>> possibility.
> >>>
> >>> 3: The #1 class defect I have found can be triggered in at least two
> >>> ways.  One requires ipa-replica-install and is hard to reproduce in a
> >>> test environment.  The other requires ldapdelete and is easy to
> >>> reproduce in an isolated VM.
> > Ipa replica install and ldapdelete of a tombstone/conflict both 
> > require cn=directory manager, which would automatically cause any 
> > reported CVE to be void - cn=directory manager is game over.
> >
> >>> 4: The defect is a mismatch between the plugin ABI as implemented by
> >>> 389-ds, and the behavior the NIS plugin expects.
> >>>
> >>> 5: I have found no specification or documentation on said ABI, so I
> >>> can only guess what the correct behavior is here.
> >>>
> >>> 6: The ABI includes two functions in the plugin: pre_delete and
> >>> post_delete.
> >>>
> >>> 7: The NIS plugin expects that every call to pre_delete will be
> >>> followed by a call to post_delete.  It takes a lock in pre_delete and
> >>> releases it in post_delete.
> > But you didn't report an issue in NIS? You reported an issue with 
> > ldapdelete on tombstones and conflicts ... the slapi-nis plugin is 
> > maintained by freeipa and if an issue in it exists, please report it 
> > to them with reproduction steps.
>
>  Hi Jay,
>
>  Thanks for your great investigations/patch/summary.
>
>  I made the change in NIS plugin to acquire/release map lock in 
>  pre/post ops. At that time I was assuming the pre/post callback were 
>  balanced.
>  I was wrong and already hit similar issue in #1694263. I updated NIS 
>  plugin to workaround that bug.
>
>  The new bug you are reporting is a new example that pre/post are 
>  sometime not balanced :(
>  The proposed fixes (prevent preop call on tombstone, force postop call 
>  even on tombstone) looks valid but will have wider impact and need 
>  additional investigation.
>  An other approach would be to make NIS plugin ignore operations on 
>  tombstone. It also require additional investigation. 
I think that for tombstone
deletions we should also prevent the call of 
the preop plugins, no plugin should deal with these operations
>  best regards
>  thierry
>
>
> >>> 8: Under some circumstances 389-ds can call pre_delete but fail to
> >>> call post_delete.  Because of #5, there is no way to tell if this is
> >>> expected behavior, but the NIS plugin clearly does not expect it.
> >>>
> >>> 9: My patch ensures that every call to pre_delete is followed by a
> >>> corresponding call to post_delete.  V1 of the patch also ensures that
> >>> every call to post_delete is after a corresponding pre_delete call.
> >>> V2 relaxes the second requirement, allowing post_delete calls without
> >>> a corresponding pre_delete call because someone expressed worry about
> >>> potential regressions.
> >>>
> >>> 10: You are refusing to merge my patch because of a doctrinal
> >>> objection to the use of ldapdelete in the simple reproducer.
> > It's not really a doctrinal objection. Replication is a really 
> > complex topic, and difficult to get correct. Ludwig knows a lot in 
> > this area, and has really good comments to make on the topic. I 
> > understand that you have an issue you have found in your setup, and 
> > you want to resolve it, but we have to consider the deployments of 
> > many thousands of other users and their replication experience too. 
> > For us it's important to be able to reproduce, and see the issue, to 
> > understand the consequences, and to make these changes carefully. 
> > Accepting the patch without clear understanding of it's consequences 
> > is not something we do.
> >
> > At this time we still don't have a clear reproducer from you on "how" 
> > you constructed the invalid directory state. You reported the following:
> >
> >> I found the bug by doing a series of "ipa-client-install" (with lots
> >> of arguments, followed by
> >> echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> >> echo [global] > /etc/ipa/installer.conf
> >> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> >> echo {password} | kinit admin
> >> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> >> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
> >>
> >> followed by the replica install failing due to network issues,
> >> misconfigured firewalls, etc, then
> >> ipa-server-install --uninstall on the host
> >> and ipa-replica-manage del {failed install host}
> >> elsewhere in the mesh, sometimes with ldapdelete of the initial
> >> replication agreement that ipa-replica-manage did not remove.
> > I'm certainly not able to produce a reproduction case from these 
> > steps ... This is why I have questioned what ipa-replica-install is 
> > doing in a previous email, and I think that we need a better series 
> > of steps to attempt to reproduce, as well as better and clearer 
> > information about "what" has crashed/hung/failed in the directories 
> > in these states.
> >
> > We still want to help, but please be patient with us as we want to 
> > ensure our fixes are the highest possible quality.
> >
> > Thanks,
> >
> > —
> > Sincerely,
> >
> > William Brown
> >
> > Senior Software Engineer, 389 Directory Server
> > SUSE Labs
> > _______________________________________________
> > 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: 
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/[e-mail]...
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
Handelsregister: Amtsgericht München, HRB 153243,
Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage

[William Brown]

>  On 5 Feb 2020, at 20:08, Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> wrote:
>  
>  
>  On 02/05/2020 10:53 AM, thierry bordaz wrote:
> > 
> > 
> > On 2/5/20 2:30 AM, William Brown wrote:
> >> 
> >>> On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt; wrote:
> >>> 
> >>> I think I can agree with 1-8, 9 is one solution to fix the problem you
> reported, but not yet validate that there are no other side effects, there are potential
> postop plugins which should NOT be called for tombstone delete, eg retro cl, we need to
> investigate side effects of the patch and evaluate other solutions, I suggested one in
> anotehr reply.
> >>> 
> >>> for 10, the claim that not crying hurrah and merging a patch without further
> investigation and testing is "doctrinal objection" is very strong and I have to
> reject this.
> >>> 
> >>> Regards,
> >>> Ludwig
> >>> 
> >>> On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> >>>> Ok, let's take this from the top:
> >>>> 
> >>>> 1: Defects that cause a server to become unresponsive are bad and must
> >>>> be repaired as soon as possible.
> >> I'm not objecting to this.
> >> 
> >>>> 2: Some #1 class defects are exploitable and require a CVE.  As far as
> >>>> I can tell, this one does not, but you should keep an eye out for the
> >>>> possibility.
> >>>> 
> >>>> 3: The #1 class defect I have found can be triggered in at least two
> >>>> ways.  One requires ipa-replica-install and is hard to reproduce in a
> >>>> test environment.  The other requires ldapdelete and is easy to
> >>>> reproduce in an isolated VM.
> >> Ipa replica install and ldapdelete of a tombstone/conflict both require
> cn=directory manager, which would automatically cause any reported CVE to be void -
> cn=directory manager is game over.
> >> 
> >>>> 4: The defect is a mismatch between the plugin ABI as implemented by
> >>>> 389-ds, and the behavior the NIS plugin expects.
> >>>> 
> >>>> 5: I have found no specification or documentation on said ABI, so I
> >>>> can only guess what the correct behavior is here.
> >>>> 
> >>>> 6: The ABI includes two functions in the plugin: pre_delete and
> >>>> post_delete.
> >>>> 
> >>>> 7: The NIS plugin expects that every call to pre_delete will be
> >>>> followed by a call to post_delete.  It takes a lock in pre_delete and
> >>>> releases it in post_delete.
> >> But you didn't report an issue in NIS? You reported an issue with ldapdelete
> on tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa and if an
> issue in it exists, please report it to them with reproduction steps.
> > 
> > Hi Jay,
> > 
> > Thanks for your great investigations/patch/summary.
> > 
> > I made the change in NIS plugin to acquire/release map lock in pre/post ops. At that
> time I was assuming the pre/post callback were balanced.
> > I was wrong and already hit similar issue in #1694263. I updated NIS plugin to
> workaround that bug.
> > 
> > The new bug you are reporting is a new example that pre/post are sometime not
> balanced :(
> > The proposed fixes (prevent preop call on tombstone, force postop call even on
> tombstone) looks valid but will have wider impact and need additional investigation.
> > An other approach would be to make NIS plugin ignore operations on tombstone. It also
> require additional investigation.
>  I think that for tombstone deletions we should also prevent the call of the preop
> plugins, no plugin should deal with these operations 
That sounds reasonable. Can we create an issue for this? 

> > 
> > best regards
> > thierry
> > 
> > 
> >>>> 8: Under some circumstances 389-ds can call pre_delete but fail to
> >>>> call post_delete.  Because of #5, there is no way to tell if this is
> >>>> expected behavior, but the NIS plugin clearly does not expect it.
> >>>> 
> >>>> 9: My patch ensures that every call to pre_delete is followed by a
> >>>> corresponding call to post_delete.  V1 of the patch also ensures that
> >>>> every call to post_delete is after a corresponding pre_delete call.
> >>>> V2 relaxes the second requirement, allowing post_delete calls without
> >>>> a corresponding pre_delete call because someone expressed worry about
> >>>> potential regressions.
> >>>> 
> >>>> 10: You are refusing to merge my patch because of a doctrinal
> >>>> objection to the use of ldapdelete in the simple reproducer.
> >> It's not really a doctrinal objection. Replication is a really complex topic,
> and difficult to get correct. Ludwig knows a lot in this area, and has really good
> comments to make on the topic. I understand that you have an issue you have found in your
> setup, and you want to resolve it, but we have to consider the deployments of many
> thousands of other users and their replication experience too. For us it's important
> to be able to reproduce, and see the issue, to understand the consequences, and to make
> these changes carefully. Accepting the patch without clear understanding of it's
> consequences is not something we do.
> >> 
> >> At this time we still don't have a clear reproducer from you on
> "how" you constructed the invalid directory state. You reported the following:
> >> 
> >>> I found the bug by doing a series of "ipa-client-install" (with
> lots
> >>> of arguments, followed by
> >>> echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> >>> echo [global] > /etc/ipa/installer.conf
> >>> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> >>> echo {password} | kinit admin
> >>> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> >>> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
> >>> 
> >>> followed by the replica install failing due to network issues,
> >>> misconfigured firewalls, etc, then
> >>> ipa-server-install --uninstall on the host
> >>> and ipa-replica-manage del {failed install host}
> >>> elsewhere in the mesh, sometimes with ldapdelete of the initial
> >>> replication agreement that ipa-replica-manage did not remove.
> >> I'm certainly not able to produce a reproduction case from these steps ...
> This is why I have questioned what ipa-replica-install is doing in a previous email, and I
> think that we need a better series of steps to attempt to reproduce, as well as better and
> clearer information about "what" has crashed/hung/failed in the directories in
> these states.
> >> 
> >> We still want to help, but please be patient with us as we want to ensure our
> fixes are the highest possible quality.
> >> 
> >> Thanks,
> >> 
> >> —
> >> Sincerely,
> >> 
> >> William Brown
> >> 
> >> Senior Software Engineer, 389 Directory Server
> >> SUSE Labs
> >> _______________________________________________
> >> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> > _______________________________________________
> > 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  
>  -- 
>  Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
>  Handelsregister: Amtsgericht München, HRB 153243,
>  Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs

[Ludwig Krispenz]

On 02/06/2020 12:57 AM, William Brown wrote:
> > On 5 Feb 2020, at 20:08, Ludwig Krispenz <lkrispen(a)redhat.com&gt; wrote:
> >
> >
> > On 02/05/2020 10:53 AM, thierry bordaz wrote:
> >>
> >> On 2/5/20 2:30 AM, William Brown wrote:
> >>>> On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> wrote:
> >>>>
> >>>> I think I can agree with 1-8, 9 is one solution to fix the problem you
> reported, but not yet validate that there are no other side effects, there are potential
> postop plugins which should NOT be called for tombstone delete, eg retro cl, we need to
> investigate side effects of the patch and evaluate other solutions, I suggested one in
> anotehr reply.
> >>>>
> >>>> for 10, the claim that not crying hurrah and merging a patch without
> further investigation and testing is "doctrinal objection" is very strong and I
> have to reject this.
> >>>>
> >>>> Regards,
> >>>> Ludwig
> >>>>
> >>>> On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> >>>>> Ok, let's take this from the top:
> >>>>>
> >>>>> 1: Defects that cause a server to become unresponsive are bad and
> must
> >>>>> be repaired as soon as possible.
> >>> I'm not objecting to this.
> >>>
> >>>>> 2: Some #1 class defects are exploitable and require a CVE.  As far
> as
> >>>>> I can tell, this one does not, but you should keep an eye out for
> the
> >>>>> possibility.
> >>>>>
> >>>>> 3: The #1 class defect I have found can be triggered in at least two
> >>>>> ways.  One requires ipa-replica-install and is hard to reproduce in
> a
> >>>>> test environment.  The other requires ldapdelete and is easy to
> >>>>> reproduce in an isolated VM.
> >>> Ipa replica install and ldapdelete of a tombstone/conflict both require
> cn=directory manager, which would automatically cause any reported CVE to be void -
> cn=directory manager is game over.
> >>>
> >>>>> 4: The defect is a mismatch between the plugin ABI as implemented by
> >>>>> 389-ds, and the behavior the NIS plugin expects.
> >>>>>
> >>>>> 5: I have found no specification or documentation on said ABI, so I
> >>>>> can only guess what the correct behavior is here.
> >>>>>
> >>>>> 6: The ABI includes two functions in the plugin: pre_delete and
> >>>>> post_delete.
> >>>>>
> >>>>> 7: The NIS plugin expects that every call to pre_delete will be
> >>>>> followed by a call to post_delete.  It takes a lock in pre_delete
> and
> >>>>> releases it in post_delete.
> >>> But you didn't report an issue in NIS? You reported an issue with
> ldapdelete on tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa
> and if an issue in it exists, please report it to them with reproduction steps.
> >> Hi Jay,
> >>
> >> Thanks for your great investigations/patch/summary.
> >>
> >> I made the change in NIS plugin to acquire/release map lock in pre/post ops. At
> that time I was assuming the pre/post callback were balanced.
> >> I was wrong and already hit similar issue in #1694263. I updated NIS plugin to
> workaround that bug.
> >>
> >> The new bug you are reporting is a new example that pre/post are sometime not
> balanced :(
> >> The proposed fixes (prevent preop call on tombstone, force postop call even on
> tombstone) looks valid but will have wider impact and need additional investigation.
> >> An other approach would be to make NIS plugin ignore operations on tombstone. It
> also require additional investigation.
> > I think that for tombstone deletions we should also prevent the call of the preop
> plugins, no plugin should deal with these operations
>  That sounds reasonable. Can we create an issue for this? 
In fact discussing this
yesterday we did get a step further. The call 
for preop plugins is already prevented for tombstone deletions, BUT

the detection if it is a delete of a tombstone flag comes from two 
sources, an operation flag, passed by the internal tombstone purging 
thread and by checking the entry flag of the entry to be deleted, which 
would be used if an external operation deletes a tombstone. 
Unfortunately this check is after the call for preops, so for external 
deletes the preops are called and the postops are not called.  Thierry 
and I agreed that the solution should be to do the check of the entry 
flag earlier and use the already existing bypass for pre and post 
plugins when a tombstone is deleted.

And yes, we need an issue for this.
> >> best regards
> >> thierry
> >>
> >>
> >>>>> 8: Under some circumstances 389-ds can call pre_delete but fail to
> >>>>> call post_delete.  Because of #5, there is no way to tell if this is
> >>>>> expected behavior, but the NIS plugin clearly does not expect it.
> >>>>>
> >>>>> 9: My patch ensures that every call to pre_delete is followed by a
> >>>>> corresponding call to post_delete.  V1 of the patch also ensures
> that
> >>>>> every call to post_delete is after a corresponding pre_delete call.
> >>>>> V2 relaxes the second requirement, allowing post_delete calls
> without
> >>>>> a corresponding pre_delete call because someone expressed worry
> about
> >>>>> potential regressions.
> >>>>>
> >>>>> 10: You are refusing to merge my patch because of a doctrinal
> >>>>> objection to the use of ldapdelete in the simple reproducer.
> >>> It's not really a doctrinal objection. Replication is a really complex
> topic, and difficult to get correct. Ludwig knows a lot in this area, and has really good
> comments to make on the topic. I understand that you have an issue you have found in your
> setup, and you want to resolve it, but we have to consider the deployments of many
> thousands of other users and their replication experience too. For us it's important
> to be able to reproduce, and see the issue, to understand the consequences, and to make
> these changes carefully. Accepting the patch without clear understanding of it's
> consequences is not something we do.
> >>>
> >>> At this time we still don't have a clear reproducer from you on
> "how" you constructed the invalid directory state. You reported the following:
> >>>
> >>>> I found the bug by doing a series of "ipa-client-install" (with
> lots
> >>>> of arguments, followed by
> >>>> echo ca_host = {a not-firewalled IPA CA} >> /etc/ipa/default.conf
> >>>> echo [global] > /etc/ipa/installer.conf
> >>>> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> >>>> echo {password} | kinit admin
> >>>> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> >>>> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
> >>>>
> >>>> followed by the replica install failing due to network issues,
> >>>> misconfigured firewalls, etc, then
> >>>> ipa-server-install --uninstall on the host
> >>>> and ipa-replica-manage del {failed install host}
> >>>> elsewhere in the mesh, sometimes with ldapdelete of the initial
> >>>> replication agreement that ipa-replica-manage did not remove.
> >>> I'm certainly not able to produce a reproduction case from these steps
> ... This is why I have questioned what ipa-replica-install is doing in a previous email,
> and I think that we need a better series of steps to attempt to reproduce, as well as
> better and clearer information about "what" has crashed/hung/failed in the
> directories in these states.
> >>>
> >>> We still want to help, but please be patient with us as we want to ensure our
> fixes are the highest possible quality.
> >>>
> >>> Thanks,
> >>>
> >>> —
> >>> Sincerely,
> >>>
> >>> William Brown
> >>>
> >>> Senior Software Engineer, 389 Directory Server
> >>> SUSE Labs
> >>> _______________________________________________
> >>> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >>> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >>> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> >> _______________________________________________
> >> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> > -- 
> > Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
> > Handelsregister: Amtsgericht München, HRB 153243,
> > Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage
> > _______________________________________________
> > 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  —
>  Sincerely,
>
>  William Brown
>
>  Senior Software Engineer, 389 Directory Server
>  SUSE Labs
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
-- 
Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
Handelsregister: Amtsgericht München, HRB 153243,
Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage

[William Brown]

>  On 6 Feb 2020, at 18:40, Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> wrote:
>  
>  
>  On 02/06/2020 12:57 AM, William Brown wrote:
> > 
> >> On 5 Feb 2020, at 20:08, Ludwig Krispenz <lkrispen(a)redhat.com&gt; wrote:
> >> 
> >> 
> >> On 02/05/2020 10:53 AM, thierry bordaz wrote:
> >>> 
> >>> On 2/5/20 2:30 AM, William Brown wrote:
> >>>>> On 5 Feb 2020, at 03:10, Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> wrote:
> >>>>> 
> >>>>> I think I can agree with 1-8, 9 is one solution to fix the problem
> you reported, but not yet validate that there are no other side effects, there are
> potential postop plugins which should NOT be called for tombstone delete, eg retro cl, we
> need to investigate side effects of the patch and evaluate other solutions, I suggested
> one in anotehr reply.
> >>>>> 
> >>>>> for 10, the claim that not crying hurrah and merging a patch without
> further investigation and testing is "doctrinal objection" is very strong and I
> have to reject this.
> >>>>> 
> >>>>> Regards,
> >>>>> Ludwig
> >>>>> 
> >>>>> On 02/04/2020 05:00 PM, Jay Fenlason wrote:
> >>>>>> Ok, let's take this from the top:
> >>>>>> 
> >>>>>> 1: Defects that cause a server to become unresponsive are bad and
> must
> >>>>>> be repaired as soon as possible.
> >>>> I'm not objecting to this.
> >>>> 
> >>>>>> 2: Some #1 class defects are exploitable and require a CVE.  As
> far as
> >>>>>> I can tell, this one does not, but you should keep an eye out for
> the
> >>>>>> possibility.
> >>>>>> 
> >>>>>> 3: The #1 class defect I have found can be triggered in at least
> two
> >>>>>> ways.  One requires ipa-replica-install and is hard to reproduce
> in a
> >>>>>> test environment.  The other requires ldapdelete and is easy to
> >>>>>> reproduce in an isolated VM.
> >>>> Ipa replica install and ldapdelete of a tombstone/conflict both require
> cn=directory manager, which would automatically cause any reported CVE to be void -
> cn=directory manager is game over.
> >>>> 
> >>>>>> 4: The defect is a mismatch between the plugin ABI as implemented
> by
> >>>>>> 389-ds, and the behavior the NIS plugin expects.
> >>>>>> 
> >>>>>> 5: I have found no specification or documentation on said ABI, so
> I
> >>>>>> can only guess what the correct behavior is here.
> >>>>>> 
> >>>>>> 6: The ABI includes two functions in the plugin: pre_delete and
> >>>>>> post_delete.
> >>>>>> 
> >>>>>> 7: The NIS plugin expects that every call to pre_delete will be
> >>>>>> followed by a call to post_delete.  It takes a lock in pre_delete
> and
> >>>>>> releases it in post_delete.
> >>>> But you didn't report an issue in NIS? You reported an issue with
> ldapdelete on tombstones and conflicts ... the slapi-nis plugin is maintained by freeipa
> and if an issue in it exists, please report it to them with reproduction steps.
> >>> Hi Jay,
> >>> 
> >>> Thanks for your great investigations/patch/summary.
> >>> 
> >>> I made the change in NIS plugin to acquire/release map lock in pre/post ops.
> At that time I was assuming the pre/post callback were balanced.
> >>> I was wrong and already hit similar issue in #1694263. I updated NIS plugin
> to workaround that bug.
> >>> 
> >>> The new bug you are reporting is a new example that pre/post are sometime not
> balanced :(
> >>> The proposed fixes (prevent preop call on tombstone, force postop call even
> on tombstone) looks valid but will have wider impact and need additional investigation.
> >>> An other approach would be to make NIS plugin ignore operations on tombstone.
> It also require additional investigation.
> >> I think that for tombstone deletions we should also prevent the call of the preop
> plugins, no plugin should deal with these operations
> > That sounds reasonable. Can we create an issue for this?
>  In fact discussing this yesterday we did get a step further. The call for preop plugins
> is already prevented for tombstone deletions, BUT
>  
>  the detection if it is a delete of a tombstone flag comes from two sources, an operation
> flag, passed by the internal tombstone purging thread and by checking the entry flag of
> the entry to be deleted, which would be used if an external operation deletes a tombstone.
> Unfortunately this check is after the call for preops, so for external deletes the preops
> are called and the postops are not called.  Thierry and I agreed that the solution should
> be to do the check of the entry flag earlier and use the already existing bypass for pre
> and post plugins when a tombstone is deleted.
>  
>  And yes, we need an issue for this. 
I'll leave that to you and thierry since you both know more about this at the moment
than I do :) 


> > 
> >>> best regards
> >>> thierry
> >>> 
> >>> 
> >>>>>> 8: Under some circumstances 389-ds can call pre_delete but fail
> to
> >>>>>> call post_delete.  Because of #5, there is no way to tell if this
> is
> >>>>>> expected behavior, but the NIS plugin clearly does not expect
> it.
> >>>>>> 
> >>>>>> 9: My patch ensures that every call to pre_delete is followed by
> a
> >>>>>> corresponding call to post_delete.  V1 of the patch also ensures
> that
> >>>>>> every call to post_delete is after a corresponding pre_delete
> call.
> >>>>>> V2 relaxes the second requirement, allowing post_delete calls
> without
> >>>>>> a corresponding pre_delete call because someone expressed worry
> about
> >>>>>> potential regressions.
> >>>>>> 
> >>>>>> 10: You are refusing to merge my patch because of a doctrinal
> >>>>>> objection to the use of ldapdelete in the simple reproducer.
> >>>> It's not really a doctrinal objection. Replication is a really
> complex topic, and difficult to get correct. Ludwig knows a lot in this area, and has
> really good comments to make on the topic. I understand that you have an issue you have
> found in your setup, and you want to resolve it, but we have to consider the deployments
> of many thousands of other users and their replication experience too. For us it's
> important to be able to reproduce, and see the issue, to understand the consequences, and
> to make these changes carefully. Accepting the patch without clear understanding of
> it's consequences is not something we do.
> >>>> 
> >>>> At this time we still don't have a clear reproducer from you on
> "how" you constructed the invalid directory state. You reported the following:
> >>>> 
> >>>>> I found the bug by doing a series of "ipa-client-install"
> (with lots
> >>>>> of arguments, followed by
> >>>>> echo ca_host = {a not-firewalled IPA CA} >>
> /etc/ipa/default.conf
> >>>>> echo [global] > /etc/ipa/installer.conf
> >>>>> echo ca_host = {ditto} >> /etc/ipa/installer.conf
> >>>>> echo {password} | kinit admin
> >>>>> ipa hostgroup-add-member ipaservers --hosts $(hostname -f)
> >>>>> ipa-relica-install --setup-ca --setup-dns --forwarder={ip addr}
> >>>>> 
> >>>>> followed by the replica install failing due to network issues,
> >>>>> misconfigured firewalls, etc, then
> >>>>> ipa-server-install --uninstall on the host
> >>>>> and ipa-replica-manage del {failed install host}
> >>>>> elsewhere in the mesh, sometimes with ldapdelete of the initial
> >>>>> replication agreement that ipa-replica-manage did not remove.
> >>>> I'm certainly not able to produce a reproduction case from these
> steps ... This is why I have questioned what ipa-replica-install is doing in a previous
> email, and I think that we need a better series of steps to attempt to reproduce, as well
> as better and clearer information about "what" has crashed/hung/failed in the
> directories in these states.
> >>>> 
> >>>> We still want to help, but please be patient with us as we want to ensure
> our fixes are the highest possible quality.
> >>>> 
> >>>> Thanks,
> >>>> 
> >>>> —
> >>>> Sincerely,
> >>>> 
> >>>> William Brown
> >>>> 
> >>>> Senior Software Engineer, 389 Directory Server
> >>>> SUSE Labs
> >>>> _______________________________________________
> >>>> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >>>> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >>>> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> >>> _______________________________________________
> >>> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >>> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >>> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> >> -- 
> >> Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
> >> Handelsregister: Amtsgericht München, HRB 153243,
> >> Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas
> Savage
> >> _______________________________________________
> >> 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> >> To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> >> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
> > —
> > Sincerely,
> > 
> > William Brown
> > 
> > Senior Software Engineer, 389 Directory Server
> > SUSE Labs
> > _______________________________________________
> > 389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  
>  -- 
>  Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
>  Handelsregister: Amtsgericht München, HRB 153243,
>  Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill, Thomas Savage
>  _______________________________________________
>  389-devel mailing list -- 389-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to 389-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]... 
—
Sincerely,

William Brown

Senior Software Engineer, 389 Directory Server
SUSE Labs