Tombstones not deleting

[Jim Tyrrell]

I have noticed on my Fedora consumers there appear to be quite a few
tombstones going back months even thought the Purge delay is set to a week:

ldapsearch -x -b "cn=mapping tree,cn=config" -D "cn=Directory Manager"
-W cn=replica nsds5ReplicaPurgeDelay
# replica, o=blah.com, mapping tree, config
dn: cn=replica,cn="o=blah.com",cn=mapping tree, cn=config
nsds5ReplicaPurgeDelay: 604800

--- example tombstone ---
# ad82a101-1dd111b2-80a3f995-55bd0000, [e-mail], Blah, blah.com
dn: nsuniqueid=ad82a101-1dd111b2-80a3f995-55bd0000,
uid=[e-mail],ou=Blah, o=blah.com
objectClass: blahPerson
objectClass: nsTombstone
uid: [e-mail]
nsParentUniqueId: ccd21704-1dd111b2-80a6a51e-7dae0000
modifyTimestamp: 20090713210513Z

There seems to be hundreds of these dating back 6 months to when the
server was built.  Why are these old entries not being purged?

Thanks.

Jim.

[Rich Megginson]

[e-mail] wrote:
> I have noticed on my Fedora consumers there appear to be quite a few
> tombstones going back months even thought the Purge delay is set to a week:
>
> ldapsearch -x -b "cn=mapping tree,cn=config" -D "cn=Directory Manager"
> -W cn=replica nsds5ReplicaPurgeDelay
> # replica, o=blah.com, mapping tree, config
> dn: cn=replica,cn="o=blah.com",cn=mapping tree, cn=config
> nsds5ReplicaPurgeDelay: 604800
>
> --- example tombstone ---
> # ad82a101-1dd111b2-80a3f995-55bd0000, [e-mail], Blah, blah.com
> dn: nsuniqueid=ad82a101-1dd111b2-80a3f995-55bd0000,
> uid=[e-mail],ou=Blah, o=blah.com
> objectClass: blahPerson
> objectClass: nsTombstone
> uid: [e-mail]
> nsParentUniqueId: ccd21704-1dd111b2-80a6a51e-7dae0000
> modifyTimestamp: 20090713210513Z
>
> There seems to be hundreds of these dating back 6 months to when the
> server was built.  Why are these old entries not being purged?

The purge algorithm never purges everything.  How many are not purged?  
What's the oldest date?
> Thanks.
>
> Jim.
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Jim Tyrrell]

Rich Megginson wrote:
> [e-mail] wrote:
> > I have noticed on my Fedora consumers there appear to be quite a few
> > tombstones going back months even thought the Purge delay is set to a week:
> >
> > ldapsearch -x -b "cn=mapping tree,cn=config" -D "cn=Directory Manager"
> > -W cn=replica nsds5ReplicaPurgeDelay
> > # replica, o=blah.com, mapping tree, config
> > dn: cn=replica,cn="o=blah.com",cn=mapping tree, cn=config
> > nsds5ReplicaPurgeDelay: 604800
> >
> > --- example tombstone ---
> > # ad82a101-1dd111b2-80a3f995-55bd0000, [e-mail], Blah, blah.com
> > dn: nsuniqueid=ad82a101-1dd111b2-80a3f995-55bd0000,
> > uid=[e-mail],ou=Blah, o=blah.com
> > objectClass: blahPerson
> > objectClass: nsTombstone
> > uid: [e-mail]
> > nsParentUniqueId: ccd21704-1dd111b2-80a6a51e-7dae0000
> > modifyTimestamp: 20090713210513Z
> >
> > There seems to be hundreds of these dating back 6 months to when the
> > server was built.  Why are these old entries not being purged?
>
> The purge algorithm never purges everything.  How many are not purged?  
> What's the oldest date?

The oldest date is 13/07/2009 which was when the server was built and 
the database imported.  There are about 200 on that date, there are 
another 300 spread over the time since then.  This server wont be having 
that many changes so I'm not sure if this would reflect all tombstones 
since then or not - I suspect there would of been a few more than that 
if they were never removed.

Is there a way to see why these ones were not deleted?  Is it possible 
to manually force a purge of these?

Jim.

[Rich Megginson]

[e-mail] wrote:
> Rich Megginson wrote:
> > [e-mail] wrote:
> > > I have noticed on my Fedora consumers there appear to be quite a few
> > > tombstones going back months even thought the Purge delay is set to a week:
> > >
> > > ldapsearch -x -b "cn=mapping tree,cn=config" -D "cn=Directory Manager"
> > > -W cn=replica nsds5ReplicaPurgeDelay
> > > # replica, o=blah.com, mapping tree, config
> > > dn: cn=replica,cn="o=blah.com",cn=mapping tree, cn=config
> > > nsds5ReplicaPurgeDelay: 604800
> > >
> > > --- example tombstone ---
> > > # ad82a101-1dd111b2-80a3f995-55bd0000, [e-mail], Blah, blah.com
> > > dn: nsuniqueid=ad82a101-1dd111b2-80a3f995-55bd0000,
> > > uid=[e-mail],ou=Blah, o=blah.com
> > > objectClass: blahPerson
> > > objectClass: nsTombstone
> > > uid: [e-mail]
> > > nsParentUniqueId: ccd21704-1dd111b2-80a6a51e-7dae0000
> > > modifyTimestamp: 20090713210513Z
> > >
> > > There seems to be hundreds of these dating back 6 months to when the
> > > server was built.  Why are these old entries not being purged?
> >
> > The purge algorithm never purges everything.  How many are not purged?  
> > What's the oldest date?
>
> The oldest date is 13/07/2009 which was when the server was built and 
> the database imported.  There are about 200 on that date, there are 
> another 300 spread over the time since then.  This server wont be having 
> that many changes so I'm not sure if this would reflect all tombstones 
> since then or not - I suspect there would of been a few more than that 
> if they were never removed.
>
> Is there a way to see why these ones were not deleted?

If you turn on the replication log level, you can see when the tombstone 
reap thread runs, and see how it decides to clean up tombstones.  If you 
don't want to wait, you can set the tombstoneReapInterval to a lower value.
> Is it possible 
> to manually force a purge of these?

You can manually delete them with ldapdelete.  But there is currently a 
bug (fixed in the source) - if the tombstone entry you are deleting has 
the nsds5ReplConflict attribute, you will have to use ldapmodify to 
remove that attribute, then you can delete the entry.
> Jim.
>
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Jim Tyrrell]

Rich Megginson wrote:
> [e-mail] wrote:
> > Rich Megginson wrote:
> > > [e-mail] wrote:
> > > > I have noticed on my Fedora consumers there appear to be quite a few
> > > > tombstones going back months even thought the Purge delay is set to a week:
> > > >
> > > > ldapsearch -x -b "cn=mapping tree,cn=config" -D "cn=Directory Manager"
> > > > -W cn=replica nsds5ReplicaPurgeDelay
> > > > # replica, o=blah.com, mapping tree, config
> > > > dn: cn=replica,cn="o=blah.com",cn=mapping tree, cn=config
> > > > nsds5ReplicaPurgeDelay: 604800
> > > >
> > > > --- example tombstone ---
> > > > # ad82a101-1dd111b2-80a3f995-55bd0000, [e-mail], Blah, blah.com
> > > > dn: nsuniqueid=ad82a101-1dd111b2-80a3f995-55bd0000,
> > > > uid=[e-mail],ou=Blah, o=blah.com
> > > > objectClass: blahPerson
> > > > objectClass: nsTombstone
> > > > uid: [e-mail]
> > > > nsParentUniqueId: ccd21704-1dd111b2-80a6a51e-7dae0000
> > > > modifyTimestamp: 20090713210513Z
> > > >
> > > > There seems to be hundreds of these dating back 6 months to when the
> > > > server was built.  Why are these old entries not being purged?
> > >
> > > The purge algorithm never purges everything.  How many are not purged?  
> > > What's the oldest date?
> >
> > The oldest date is 13/07/2009 which was when the server was built and 
> > the database imported.  There are about 200 on that date, there are 
> > another 300 spread over the time since then.  This server wont be having 
> > that many changes so I'm not sure if this would reflect all tombstones 
> > since then or not - I suspect there would of been a few more than that 
> > if they were never removed.
> >
> > Is there a way to see why these ones were not deleted?
>
> If you turn on the replication log level, you can see when the tombstone 
> reap thread runs, and see how it decides to clean up tombstones.  If you 
> don't want to wait, you can set the tombstoneReapInterval to a lower value.
> > Is it possible 
> > to manually force a purge of these?
>
> You can manually delete them with ldapdelete.  But there is currently a 
> bug (fixed in the source) - if the tombstone entry you are deleting has 
> the nsds5ReplConflict attribute, you will have to use ldapmodify to 
> remove that attribute, then you can delete the entry.

OK - I have some debugging for the reaps:

[25/Mar/2010:08:28:40 +0000] NSMMReplicationPlugin - 
_replica_reap_tombstones: removing tombstone 
nsuniqueid=2a83f601-1dd211b2-8055f995-55bd0000, 
uid=[e-mail],ou=blah, o=blah.co.uk because its deletion 
csn (4ba16da5000000660000) is less than the purge csn 
(4ba248bf000000660000).
[25/Mar/2010:08:28:44 +0000] NSMMReplicationPlugin - 
_replica_reap_tombstones: NOT removing tombstone 
nsuniqueid=e173fa81-1dd111b2-809bf995-55bd0000, 
uid=[e-mail],ou=blah, o=blah.co.uk
[25/Mar/2010:08:28:44 +0000] NSMMReplicationPlugin - 
_replica_reap_tombstones: NOT removing tombstone 
nsuniqueid=ac72a282-1dd111b2-80b1f995-55bd0000, 
uid=[e-mail],ou=blah, o=blah.co.uk
[25/Mar/2010:08:28:44 +0000] NSMMReplicationPlugin - 
_replica_reap_tombstones: NOT removing tombstone 
nsuniqueid=f3f92e81-1dd111b2-80b1f995-55bd0000, 
uid=[e-mail],ou=blah, o=blah.co.uk

It looks like it is generally purging the tombstones, but there is a 
growing number that it is not.  I'm not sure what these deletion and 
purge csn's are and where I should find them?

Thanks.

Jim.

[Jim Tyrrell]

The Tombstones which are deleted have the following reason:

"because its deletion csn (4ba584d6000800660000) is less than the purge 
csn (4ba6a401000000660000)."

Can I find out what the deletion csn and purge csn is for an object 
which has not been deleted so I can verify that it should indeed of been 
deleted?  If so where do I find this info?

Thanks.

Jim.