glue entry problem

[Elizabeth Jones]

I have all kinds of borkage in my ldap today.

I created a new ou in one of my data centers,

ou=cdc,ou=service accts,ou=staff,ou=people,dc=mycompany,dc=com

under this I added 2 users. About 5 minutes later I got an alarm from my
monitoring system saying that replication had failed, and I discovered
that replication from this data center to my second data center had
failed, and more specifically this ou --

[22/Apr/2014:15:28:03 -0500] - Retry count exceeded in add
[22/Apr/2014:15:28:03 -0500] NSMMReplicationPlugin - conn=437731 op=4
csn=5356cc22000000010000: Can't created glue entry ou=CDC,ou=Service
Accts,ou=People,dc=mycompany,dc=com
uniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417, error 51

So I thought there was something wrong in the new ou I'd created so I went
back and deleted the two children, then tried to delete the ou.  But my
ldap thinks that the children still exist and won't let me delete the ou
--

[22/Apr/2014:15:45:17 -0500] entryrdn-index - _entryrdn_delete_key: Failed
to remove ou=cdc; has children
[22/Apr/2014:15:45:17 -0500] - database index operation failed BAD 1031,
err=-1 Unknown error: -1


Any thoughts on how to proceed with this?  I'm afraid to do anything else
on the first server now that I've managed to get it into this state.

thanks -
EJ

[Rich Megginson]

On 04/22/2014 03:07 PM, Elizabeth Jones wrote:
>  I have all kinds of borkage in my ldap today.
>
>  I created a new ou in one of my data centers,
>
>  ou=cdc,ou=service accts,ou=staff,ou=people,dc=mycompany,dc=com
>
>  under this I added 2 users. About 5 minutes later I got an alarm from my
>  monitoring system saying that replication had failed, and I discovered
>  that replication from this data center to my second data center had
>  failed, and more specifically this ou --
>
>  [22/Apr/2014:15:28:03 -0500] - Retry count exceeded in add
>  [22/Apr/2014:15:28:03 -0500] NSMMReplicationPlugin - conn=437731 op=4
>  csn=5356cc22000000010000: Can't created glue entry ou=CDC,ou=Service
>  Accts,ou=People,dc=mycompany,dc=com
>  uniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417, error 51
>
>  So I thought there was something wrong in the new ou I'd created so I went
>  back and deleted the two children, then tried to delete the ou.  But my
>  ldap thinks that the children still exist and won't let me delete the ou
>  --
>
>  [22/Apr/2014:15:45:17 -0500] entryrdn-index - _entryrdn_delete_key: Failed
>  to remove ou=cdc; has children
>  [22/Apr/2014:15:45:17 -0500] - database index operation failed BAD 1031,
>  err=-1 Unknown error: -1
>
>
>  Any thoughts on how to proceed with this?  I'm afraid to do anything else
>  on the first server now that I've managed to get it into this state. 
rpm -q
389-ds-base
>  thanks -
>  EJ
>
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Elizabeth Jones]

>  rpm -q 389-ds-base 
389-ds-base-1.2.11.25-1.el6.x86_64

[Elizabeth Jones]

As an added bonus, I now see this when I try to ldapsearch for this ou in
my first ldap and in its local consumer --

ldap1:
dn: nsuniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417,ou=CDC,ou=Service
Accts,ou=People,dc=mycompany,dc=com

ldap2 (local consumer):
dn: ou=CDC,ou=Service Accts,ou=People,dc=mycompany,dc=com


>  I have all kinds of borkage in my ldap today.
>
>  I created a new ou in one of my data centers,
>
>  ou=cdc,ou=service accts,ou=staff,ou=people,dc=mycompany,dc=com
>
>  under this I added 2 users. About 5 minutes later I got an alarm from my
>  monitoring system saying that replication had failed, and I discovered
>  that replication from this data center to my second data center had
>  failed, and more specifically this ou --
>
>  [22/Apr/2014:15:28:03 -0500] - Retry count exceeded in add
>  [22/Apr/2014:15:28:03 -0500] NSMMReplicationPlugin - conn=437731 op=4
>  csn=5356cc22000000010000: Can't created glue entry ou=CDC,ou=Service
>  Accts,ou=People,dc=mycompany,dc=com
>  uniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417, error 51
>
>  So I thought there was something wrong in the new ou I'd created so I went
>  back and deleted the two children, then tried to delete the ou.  But my
>  ldap thinks that the children still exist and won't let me delete the ou
>  --
>
>  [22/Apr/2014:15:45:17 -0500] entryrdn-index - _entryrdn_delete_key: Failed
>  to remove ou=cdc; has children
>  [22/Apr/2014:15:45:17 -0500] - database index operation failed BAD 1031,
>  err=-1 Unknown error: -1
>
>
>  Any thoughts on how to proceed with this?  I'm afraid to do anything else
>  on the first server now that I've managed to get it into this state.
>
>  thanks -
>  EJ
>
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Rich Megginson]

On 04/22/2014 03:18 PM, Elizabeth Jones wrote:
> > rpm -q 389-ds-base
>  389-ds-base-1.2.11.25-1.el6.x86_64 
I strongly encourage you to use the version of 389-ds-base included with 
the base EL6 OS.  If not, you might try upgrading to 1.2.11.29 from the 
copr repo
http://port389.org/wiki/Download


That being said, I'm not sure if this particular glue issue has been 
fixed, nor do I know how to fix your borked database, perhaps one of the 
other 389 devs will chime in.

>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Noriko Hosoi]

We have a couple of known bugs which you might have run into...  We are 
actively working on them.
#47696 <https://fedorahosted.org/389/ticket/47696> Large Searches Hang - 
Possibly entryrdn related <https://fedorahosted.org/389/ticket/47696>
#47750 <https://fedorahosted.org/389/ticket/47750> Creating a glue fails 
if one above level is a conflict or missing 
<https://fedorahosted.org/389/ticket/47750>
...

You mentioned 2 servers ldap1 and ldap2.  Are they both masters? You put 
"local consumer" to ldap2.  Does that mean ldap2 is a read only replica?

It looks to me ldap1 got broken and ldap2 is still healthy.  You may 
want to make ldap1 in sync with ldap2 and start from there. If ldap2 is 
a master,  you could re-initialize ldap1 from ldap2.

If ldap2 is a read only replica, you could export the contents with 
db2ldif -r -n <your_backend> command line utility on ldap2 and import 
the exported ldif file to ldap1.

Or if you don't mind losing the replication information such as 
tombstones and state info, you could export the contents without "-r" by 
db2ldif on ldap2 and import the ldif file to ldap1, then re-initialize 
ldap2 on ldap1.

Hope you could choose one of the 3 ways and it fixes the problem.
--noriko

  Elizabeth Jones wrote:
>  As an added bonus, I now see this when I try to ldapsearch for this
> ou in
>  my first ldap and in its local consumer --
>
>  ldap1:
>  dn: nsuniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417,ou=CDC,ou=Service
>  Accts,ou=People,dc=mycompany,dc=com
>
>  ldap2 (local consumer):
>  dn: ou=CDC,ou=Service Accts,ou=People,dc=mycompany,dc=com
>
>
> > I have all kinds of borkage in my ldap today.
> >
> > I created a new ou in one of my data centers,
> >
> > ou=cdc,ou=service accts,ou=staff,ou=people,dc=mycompany,dc=com
> >
> > under this I added 2 users. About 5 minutes later I got an alarm from my
> > monitoring system saying that replication had failed, and I discovered
> > that replication from this data center to my second data center had
> > failed, and more specifically this ou --
> >
> > [22/Apr/2014:15:28:03 -0500] - Retry count exceeded in add
> > [22/Apr/2014:15:28:03 -0500] NSMMReplicationPlugin - conn=437731 op=4
> > csn=5356cc22000000010000: Can't created glue entry ou=CDC,ou=Service
> > Accts,ou=People,dc=mycompany,dc=com
> > uniqueid=dde5bb01-ca5811e3-af3cad6b-9c050417, error 51
> >
> > So I thought there was something wrong in the new ou I'd created so I went
> > back and deleted the two children, then tried to delete the ou.  But my
> > ldap thinks that the children still exist and won't let me delete the ou
> > --
> >
> > [22/Apr/2014:15:45:17 -0500] entryrdn-index - _entryrdn_delete_key: Failed
> > to remove ou=cdc; has children
> > [22/Apr/2014:15:45:17 -0500] - database index operation failed BAD 1031,
> > err=-1 Unknown error: -1
> >
> >
> > Any thoughts on how to proceed with this?  I'm afraid to do anything else
> > on the first server now that I've managed to get it into this state.
> >
> > thanks -
> > EJ
> >
> >
> > --
> > 389 users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Elizabeth Jones]

>  You mentioned 2 servers ldap1 and ldap2.  Are they both masters? You put
>  "local consumer" to ldap2.  Does that mean ldap2 is a read only replica?
We have two data centers and each data center has an ldap1 and ldap2.  All
4 are masters, but we only ever send updates to DCA-ldap1.  That then
pushes to DCA-ldap2 and to DCB-ldap1.  DCB-ldap1 pushes to DCB-ldap2, so
the ldap2's are really functioning as consumers all the time.

>  It looks to me ldap1 got broken and ldap2 is still healthy.  You may
>  want to make ldap1 in sync with ldap2 and start from there. If ldap2 is
>  a master,  you could re-initialize ldap1 from ldap2.
>
>  If ldap2 is a read only replica, you could export the contents with
>  db2ldif -r -n <your_backend> command line utility on ldap2 and import
>  the exported ldif file to ldap1.
>
>  Or if you don't mind losing the replication information such as
>  tombstones and state info, you could export the contents without "-r" by
>  db2ldif on ldap2 and import the ldif file to ldap1, then re-initialize
>  ldap2 on ldap1.
>
>  Hope you could choose one of the 3 ways and it fixes the problem. 
I will give this a shot.  When we tried to initialize these last week the
initialization emptied out the receiving ldap and I believe that was a bug
that is fixed in 1.3.  I recovered by using a backup but I don't want to
try initializing again until we can upgrade to 1.3. Can I still do the
ldif on ldap2, since it is actually a master rather than a read only
replica?

Thanks,
EJ

[Noriko Hosoi]

Elizabeth Jones wrote:
>  I will give this a shot.  When we tried to initialize these last week
> the
>  initialization emptied out the receiving ldap and I believe that was a bug
>  that is fixed in 1.3.  I recovered by using a backup but I don't want to
>  try initializing again until we can upgrade to 1.3. Can I still do the
>  ldif on ldap2, since it is actually a master rather than a read only
>  replica?
>  do the ldif on ldap2 
means export the Directory Server contents to an ldif file on
ldap2?  If 
so, yes.

Thanks,
--noriko