Strange Disk IO issue

[Brad Schuetz]

I have recently upgraded our 389 servers from pretty old versions that
were a mix and match of 389 release and CentOS released versions (all on
centos 5) to the latest (on centos6) (specific RPMs listed below).

I did this though a full ldif dump of the original server and imported
into a freshly installed new master server.  Then I setup the
replication agreements with the 7 slave servers and everything was
running fine. 

After about a week I starting having a problem with the hubs servers
where all of them after (possibly exactly) 24 hours would start going
crazy on the disk IO (95-100% according to sysstat) of that server
making queries to ldap slow.  The master server does not exhibit this
problem, it will run completely fine.

A simple restart of the dirsrv process corrects the issue and then it
will run for another 24 hours before repeating the issue.

The hardware running each node is somewhat different with varying disk
speeds underlying, but all exhibit the same behavior.

This happens the same on the 2 nodes that get relatively little traffic
and the 5 nodes that get a lot of traffic.

I was originally on the 389-ds-base release that shipped with CentOS6
and have changed to the version from the
<http://repos.fedorapeople.org/repos/rmeggins/389-ds-base/epel-389-ds-base...
repo, both do the same thing.

Any thoughts/suggestions on how to fix or further diagnose this?  I've
had no luck with strace or error logs to find any issues.  At this point
I've unfortunately had to resort to a cron job to restart all of my LDAP
hubs.

Installed RPMs:
389-ds-console-1.2.6-1.el6.noarch
389-ds-1.2.2-1.el6.noarch
389-console-1.1.7-1.el6.noarch
389-admin-console-1.1.8-1.el6.noarch
389-ds-console-doc-1.2.6-1.el6.noarch
389-dsgw-1.1.9-1.el6.x86_64
389-admin-1.1.29-1.el6.x86_64
389-ds-base-1.2.10.7-1.el6.x86_64
389-adminutil-1.1.15-1.el6.x86_64
389-admin-console-doc-1.1.8-1.el6.noarch
389-ds-base-libs-1.2.10.7-1.el6.x86_64

--
Brad

[Paul Robert Marino]

That is odd behavior.
Do all of the replcas have the same applications connecting to them? Not
nessisarily the same instances but the same applcations configured in a
simmilar way. The reason I ask is I'm wondering if there might be a rouge
app sending heavy queries repeatedly to the servers. Is sounds like issues
I've seen in environments where nscd wasn't properly tuned (hundreds of
processes querying nscd but only the default low thread count limit) and
had stopped functioning correctly. The result was the ldap servers were
victims of hundreds of boxes launching an unintended dos atack.
On May 14, 2012 7:54 PM, "Brad Schuetz" <brad(a)omnis.com&gt; wrote:

>  I have recently upgraded our 389 servers from pretty old versions
> that
>  were a mix and match of 389 release and CentOS released versions (all on
>  centos 5) to the latest (on centos6) (specific RPMs listed below).
>
>  I did this though a full ldif dump of the original server and imported
>  into a freshly installed new master server.  Then I setup the
>  replication agreements with the 7 slave servers and everything was
>  running fine.
>
>  After about a week I starting having a problem with the hubs servers
>  where all of them after (possibly exactly) 24 hours would start going
>  crazy on the disk IO (95-100% according to sysstat) of that server
>  making queries to ldap slow.  The master server does not exhibit this
>  problem, it will run completely fine.
>
>  A simple restart of the dirsrv process corrects the issue and then it
>  will run for another 24 hours before repeating the issue.
>
>  The hardware running each node is somewhat different with varying disk
>  speeds underlying, but all exhibit the same behavior.
>
>  This happens the same on the 2 nodes that get relatively little traffic
>  and the 5 nodes that get a lot of traffic.
>
>  I was originally on the 389-ds-base release that shipped with CentOS6
>  and have changed to the version from the
>  <
>  http://repos.fedorapeople.org/repos/rmeggins/389-ds-base/epel-389-ds-base...
>  >
>  repo, both do the same thing.
>
>  Any thoughts/suggestions on how to fix or further diagnose this?  I've
>  had no luck with strace or error logs to find any issues.  At this point
>  I've unfortunately had to resort to a cron job to restart all of my LDAP
>  hubs.
>
>  Installed RPMs:
>  389-ds-console-1.2.6-1.el6.noarch
>  389-ds-1.2.2-1.el6.noarch
>  389-console-1.1.7-1.el6.noarch
>  389-admin-console-1.1.8-1.el6.noarch
>  389-ds-console-doc-1.2.6-1.el6.noarch
>  389-dsgw-1.1.9-1.el6.x86_64
>  389-admin-1.1.29-1.el6.x86_64
>  389-ds-base-1.2.10.7-1.el6.x86_64
>  389-adminutil-1.1.15-1.el6.x86_64
>  389-admin-console-doc-1.1.8-1.el6.noarch
>  389-ds-base-libs-1.2.10.7-1.el6.x86_64
>
>  --
>  Brad
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Brad Schuetz]

On 05/15/2012 06:18 AM, Paul Robert Marino wrote:
>  That is odd behavior.
>  Do all of the replcas have the same applications connecting to them?
>  Not nessisarily the same instances but the same applcations configured
>  in a simmilar way. The reason I ask is I'm wondering if there might be
>  a rouge app sending heavy queries repeatedly to the servers. Is sounds
>  like issues I've seen in environments where nscd wasn't properly tuned
>  (hundreds of processes querying nscd but only the default low thread
>  count limit) and had stopped functioning correctly. The result was the
>  ldap servers were victims of hundreds of boxes launching an unintended
>  dos atack.
>  
Of the 7 replicas, 5 are attached to one network, and 2 are on another
network.  The 5 are queried a LOT, the other 2 barely get any traffic at
all.  All, however, are getting the same traffic that they were getting
when I was using previous versions of the LDAP server.

The 2 that are barely used I've checked for excessive queries being run
at the point when load goes crazy and they are getting the usual minimal
load.

Also that doesn't explain why it's always 24 hours that it goes
haywire.  It doesn't matter when I restart the service, it could be
restarted at 2am, then in 24 hours it will go crazy IO load.

--
Brad

[Gregory Matthews]

On 15/05/12 19:22, Brad Schuetz wrote:
>  Of the 7 replicas, 5 are attached to one network, and 2 are on
> another
>  network.  The 5 are queried a LOT, the other 2 barely get any traffic at
>  all.  All, however, are getting the same traffic that they were getting
>  when I was using previous versions of the LDAP server.
>
>  The 2 that are barely used I've checked for excessive queries being run
>  at the point when load goes crazy and they are getting the usual minimal
>  load.
>
>  Also that doesn't explain why it's always 24 hours that it goes
>  haywire.  It doesn't matter when I restart the service, it could be
>  restarted at 2am, then in 24 hours it will go crazy IO load. 
do you know what the IO is? is it swapping? are you running collectl or 
similar so you can look at historic performance data?

G

>  --
>  Brad
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users 

-- 
Greg Matthews        01235 778658
Scientific Computing Group Leader
Diamond Light Source Ltd. OXON UK

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged
material, and are for the use of the intended addressee only. If you are not the intended
addressee or an authorised recipient of the addressee please notify us of receipt by
returning the e-mail and do not use, copy, retain, distribute or disclose the information
in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily
of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free
from viruses and we cannot accept liability for any damage which you may sustain as a
result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with
its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot,
Oxfordshire, OX11 0DE, United Kingdom

[Paul Robert Marino]

The exact timing of the issue is to strange is there a backup job running
at midnight. Or some other timed job that could be eating the ram or disk
IO. Possibly one that is reliant on ldap queries that would otherwise be
inocuious.
On May 16, 2012 7:51 AM, "Gregory Matthews" <greg.matthews(a)diamond.ac.uk&gt;
wrote:

>  On 15/05/12 19:22, Brad Schuetz wrote:
>
> > Of the 7 replicas, 5 are attached to one network, and 2 are on another
> > network.  The 5 are queried a LOT, the other 2 barely get any traffic at
> > all.  All, however, are getting the same traffic that they were getting
> > when I was using previous versions of the LDAP server.
> >
> > The 2 that are barely used I've checked for excessive queries being run
> > at the point when load goes crazy and they are getting the usual minimal
> > load.
> >
> > Also that doesn't explain why it's always 24 hours that it goes
> > haywire.  It doesn't matter when I restart the service, it could be
> > restarted at 2am, then in 24 hours it will go crazy IO load.
> >
>
>  do you know what the IO is? is it swapping? are you running collectl or
>  similar so you can look at historic performance data?
>
>  G
>
>
> > --
> > Brad
> > --
> > 389 users mailing list
> > 389-users(a)lists.fedoraproject.**org <389-users(a)lists.fedoraproject.org&gt;
> >
> https://admin.fedoraproject.**org/mailman/listinfo/389-users<https://a...
> >
>
>
>  --
>  Greg Matthews        01235 778658
>  Scientific Computing Group Leader
>  Diamond Light Source Ltd. OXON UK
>
>  --
>  This e-mail and any attachments may contain confidential, copyright and or
>  privileged material, and are for the use of the intended addressee only. If
>  you are not the intended addressee or an authorised recipient of the
>  addressee please notify us of receipt by returning the e-mail and do not
>  use, copy, retain, distribute or disclose the information in or attached to
>  the e-mail.
>  Any opinions expressed within this e-mail are those of the individual and
>  not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd.
>  cannot guarantee that this e-mail or any attachments are free from viruses
>  and we cannot accept liability for any damage which you may sustain as a
>  result of software viruses which may be transmitted in or with the message.
>  Diamond Light Source Limited (company no. 4375679). Registered in England
>  and Wales with its registered office at Diamond House, Harwell Science and
>  Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
>
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.**org <389-users(a)lists.fedoraproject.org&gt;
>
> https://admin.fedoraproject.**org/mailman/listinfo/389-users<https://a...

[Brad Schuetz]

On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
>  The exact timing of the issue is to strange is there a backup job
>  running at midnight. Or some other timed job that could be eating the
>  ram or disk IO. Possibly one that is reliant on ldap queries that
>  would otherwise be inocuious.
>
>  
It doesn't happen at midnight, it's 24 hours from when the process was
started, so I can restart dirsrv at 3:17pm on Wednesday and at right
around 3:17pm on Thursday that server will go to 100% disk IO usage.

I've restarted the servers at totally random times to reproduce this
issue, and currently restart, via cron, all my ldap servers twice per
day at randomly selected times of the day to make sure that both they
are restarted before the 24 hours hits, and so that no more than 1
dirsrv process is being restarted at the same time.

Keep in mind, the ldap queries load has not changed from the setup I was
running prior to this which was running (much) older versions of the 389
server software.  In fact, as part of this system upgrade, additional
servers were added to reduce the individual load on each server.

--
Brad

Audit log parsing

[Josh Ellsworth]

Does anyone have a system for parsing the 389 audit log? I have to periodically generate a
list of changes and validate that all of them were authorized via ticket. I had a python
script for our SunDS servers but thought I'd ask about a 389 version before
reinventing the wheel.

Josh

--
Joshua Ellsworth
System Administrator, Primatics Financial
Phone: [phone]
jellsworth(a)primaticsfinancial.com

Audit log parsing

[Rich Megginson]

On 05/16/2012 12:37 PM, Josh Ellsworth wrote:
>  Does anyone have a system for parsing the 389 audit log? I have to
> periodically generate a list of changes and validate that all of them were authorized via
> ticket. I had a python script for our SunDS servers but thought I'd ask about a 389
> version before reinventing the wheel. 
The audit log format should be almost identical between SunDS and 389.  
Does your python script work with 389?  Have you tried it?

>  Josh
>
>  --
>  Joshua Ellsworth
>  System Administrator, Primatics Financial
>  Phone: [phone]
>  jellsworth(a)primaticsfinancial.com
>
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Nathan Kinder]

On 05/16/2012 11:19 AM, Brad Schuetz wrote:
>  On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> > The exact timing of the issue is to strange is there a backup job
> > running at midnight. Or some other timed job that could be eating the
> > ram or disk IO. Possibly one that is reliant on ldap queries that
> > would otherwise be inocuious.
> >
> >
>  It doesn't happen at midnight, it's 24 hours from when the process was
>  started, so I can restart dirsrv at 3:17pm on Wednesday and at right
>  around 3:17pm on Thursday that server will go to 100% disk IO usage. 
The default
tombstone purge interval is 1 day, which seems to fit what 
you are seeing.  The tombstone reap thread will start every 24 hours to 
find tombstone entries that can be deleted.  The default retention 
period for tombstones is 1 week.  It is possible that you have a large 
number of tombstone entries that need to be deleted.  This will occur 
independently on all of your server instances.  This is controlled by 
the "nsDS5ReplicaTombstonePurgeInterval" and "nsDS5ReplicaPurgeDelay"

attributes in your "cn=replica,cn=<suffixDN>,cn=mapping tree,cn=config" 
entry.

You can search for "(objectclass=nstombstone)" as Directory Manager to 
see how many tombstone entries you have.
>  I've restarted the servers at totally random times to reproduce this
>  issue, and currently restart, via cron, all my ldap servers twice per
>  day at randomly selected times of the day to make sure that both they
>  are restarted before the 24 hours hits, and so that no more than 1
>  dirsrv process is being restarted at the same time.
>
>  Keep in mind, the ldap queries load has not changed from the setup I was
>  running prior to this which was running (much) older versions of the 389
>  server software.  In fact, as part of this system upgrade, additional
>  servers were added to reduce the individual load on each server.
>
>  --
>  Brad
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Brad Schuetz]

On 05/16/2012 11:54 AM, Nathan Kinder wrote:
>  On 05/16/2012 11:19 AM, Brad Schuetz wrote:
> >
> > On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >> The exact timing of the issue is to strange is there a backup job
> >> running at midnight. Or some other timed job that could be eating the
> >> ram or disk IO. Possibly one that is reliant on ldap queries that
> >> would otherwise be inocuious.
> >>
> >>
> > It doesn't happen at midnight, it's 24 hours from when the process was
> > started, so I can restart dirsrv at 3:17pm on Wednesday and at right
> > around 3:17pm on Thursday that server will go to 100% disk IO usage.
>  The default tombstone purge interval is 1 day, which seems to fit what
>  you are seeing.  The tombstone reap thread will start every 24 hours
>  to find tombstone entries that can be deleted.  The default retention
>  period for tombstones is 1 week.  It is possible that you have a large
>  number of tombstone entries that need to be deleted.  This will occur
>  independently on all of your server instances.  This is controlled by
>  the "nsDS5ReplicaTombstonePurgeInterval" and
> "nsDS5ReplicaPurgeDelay"
>  attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
>  tree,cn=config" entry.
>  
I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's
using
that default), and "nsDS5ReplicaPurgeDelay" is set to 3600


>  You can search for "(objectclass=nstombstone)" as Directory
> Manager to
>  see how many tombstone entries you have. 
I have a LOT of tombstone entries, over 200k on this one server (I'm
guessing since I've been restarting the process for over a week now, not
letting it run the cleanup process).

So, any suggestions on what can I do to fix this?  The process that's
reaping the entries is using too much IO making queries time out, older
versions of the software did not exhibit this behavior.  In fact, I can
reinitalize the entire replica faster than this thing is reaping the
entries, it takes 7 minutes to reinit a replica, but when this issue
first started I let the dirsrv run much longer before restarting it.

Should I make it purge more frequently so there are fewer entries to
reap?  Or is this just some weird bug?

--
Brad

[Nathan Kinder]

On 05/16/2012 01:09 PM, Brad Schuetz wrote:
>  On 05/16/2012 11:54 AM, Nathan Kinder wrote:
> > On 05/16/2012 11:19 AM, Brad Schuetz wrote:
> >> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >>> The exact timing of the issue is to strange is there a backup job
> >>> running at midnight. Or some other timed job that could be eating the
> >>> ram or disk IO. Possibly one that is reliant on ldap queries that
> >>> would otherwise be inocuious.
> >>>
> >>>
> >> It doesn't happen at midnight, it's 24 hours from when the process was
> >> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
> >> around 3:17pm on Thursday that server will go to 100% disk IO usage.
> > The default tombstone purge interval is 1 day, which seems to fit what
> > you are seeing.  The tombstone reap thread will start every 24 hours
> > to find tombstone entries that can be deleted.  The default retention
> > period for tombstones is 1 week.  It is possible that you have a large
> > number of tombstone entries that need to be deleted.  This will occur
> > independently on all of your server instances.  This is controlled by
> > the "nsDS5ReplicaTombstonePurgeInterval" and
> "nsDS5ReplicaPurgeDelay"
> > attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
> > tree,cn=config" entry.
> >
>  I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's using
>  that default), and "nsDS5ReplicaPurgeDelay" is set to 3600 
Ok, so this
means every 24 hours, the tombstone reap thread will look 
for tombstones older than 1 hour and remove them.
> > You can search for "(objectclass=nstombstone)" as Directory Manager to
> > see how many tombstone entries you have.
>  I have a LOT of tombstone entries, over 200k on this one server (I'm
>  guessing since I've been restarting the process for over a week now, not
>  letting it run the cleanup process). 
That's possible if you really do 200k
delete operations in 1 week, but 
that sounds like a lot.  It would seem that these tombstones have been 
building up for a longer time than 1 week.
>  So, any suggestions on what can I do to fix this?  The process that's
>  reaping the entries is using too much IO making queries time out, older
>  versions of the software did not exhibit this behavior.  In fact, I can
>  reinitalize the entire replica faster than this thing is reaping the
>  entries, it takes 7 minutes to reinit a replica, but when this issue
>  first started I let the dirsrv run much longer before restarting it. 
Due to the
number of matching entries for the tombstone search, it is 
having to walk your entire database, which is why you see the IO 
spiking.  What you could do is to export your database with "db2ldif 
-r".  This will preserve the replication related data in the LDIF.  You 
can then remove the tombstone entries in the LDIF file via a script and 
reimport it.  You would have to do this on each server, or do it on one 
master and then reinitialize the rest of your servers.  One thing to 
watch out for is that you do not want to remove the RUV entry, which 
will have the "nstombstone" objectclass.  This RUV entry will have a 
"nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff" value that you can use 
to distinguish if from the rest of the tombstones.
>  Should I make it purge more frequently so there are fewer entries to
>  reap?  Or is this just some weird bug? 
I'd leave the purge settings as they
are.
>  --
>  Brad
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Rich Megginson]

On 05/16/2012 04:06 PM, Nathan Kinder wrote:
>  On 05/16/2012 01:09 PM, Brad Schuetz wrote:
> > On 05/16/2012 11:54 AM, Nathan Kinder wrote:
> >> On 05/16/2012 11:19 AM, Brad Schuetz wrote:
> >>> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >>>> The exact timing of the issue is to strange is there a backup job
> >>>> running at midnight. Or some other timed job that could be eating the
> >>>> ram or disk IO. Possibly one that is reliant on ldap queries that
> >>>> would otherwise be inocuious.
> >>>>
> >>>>
> >>> It doesn't happen at midnight, it's 24 hours from when the process
> was
> >>> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
> >>> around 3:17pm on Thursday that server will go to 100% disk IO usage.
> >> The default tombstone purge interval is 1 day, which seems to fit what
> >> you are seeing.  The tombstone reap thread will start every 24 hours
> >> to find tombstone entries that can be deleted.  The default retention
> >> period for tombstones is 1 week.  It is possible that you have a large
> >> number of tombstone entries that need to be deleted.  This will occur
> >> independently on all of your server instances.  This is controlled by
> >> the "nsDS5ReplicaTombstonePurgeInterval" and
> "nsDS5ReplicaPurgeDelay"
> >> attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
> >> tree,cn=config" entry.
> >>
> > I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's
> using
> > that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
>  Ok, so this means every 24 hours, the tombstone reap thread will look 
>  for tombstones older than 1 hour and remove them.
> >
> >
> >> You can search for "(objectclass=nstombstone)" as Directory Manager to
> >> see how many tombstone entries you have.
> > I have a LOT of tombstone entries, over 200k on this one server (I'm
> > guessing since I've been restarting the process for over a week now, not
> > letting it run the cleanup process).
>  That's possible if you really do 200k delete operations in 1 week, but 
>  that sounds like a lot.  It would seem that these tombstones have been 
>  building up for a longer time than 1 week.
> >
> > So, any suggestions on what can I do to fix this?  The process that's
> > reaping the entries is using too much IO making queries time out, older
> > versions of the software did not exhibit this behavior.  In fact, I can
> > reinitalize the entire replica faster than this thing is reaping the
> > entries, it takes 7 minutes to reinit a replica, but when this issue
> > first started I let the dirsrv run much longer before restarting it.
>  Due to the number of matching entries for the tombstone search, it is 
>  having to walk your entire database, which is why you see the IO spiking. 
Perhaps also try increasing nsslapd-idlistscanlimit so that it can hold 
the entire candidate list of tombstones to delete -
http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Admin...
>  What you could do is to export your database with "db2ldif
> -r".  This 
>  will preserve the replication related data in the LDIF.  You can then 
>  remove the tombstone entries in the LDIF file via a script and 
>  reimport it.  You would have to do this on each server, or do it on 
>  one master and then reinitialize the rest of your servers.  One thing 
>  to watch out for is that you do not want to remove the RUV entry, 
>  which will have the "nstombstone" objectclass.  This RUV entry will 
>  have a "nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff" value that you 
>  can use to distinguish if from the rest of the tombstones.
> >
> > Should I make it purge more frequently so there are fewer entries to
> > reap?  Or is this just some weird bug?
>  I'd leave the purge settings as they are.
> >
> > -- 
> > Brad
> > -- 
> > 389 users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
>  -- 
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Brad Schuetz]

On 05/16/2012 04:01 PM, Rich Megginson wrote:
>  On 05/16/2012 04:06 PM, Nathan Kinder wrote:
> > On 05/16/2012 01:09 PM, Brad Schuetz wrote:
> >> On 05/16/2012 11:54 AM, Nathan Kinder wrote:
> >>> On 05/16/2012 11:19 AM, Brad Schuetz wrote:
> >>>> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >>>>> The exact timing of the issue is to strange is there a backup job
> >>>>> running at midnight. Or some other timed job that could be eating
> >>>>> the
> >>>>> ram or disk IO. Possibly one that is reliant on ldap queries that
> >>>>> would otherwise be inocuious.
> >>>>>
> >>>>>
> >>>> It doesn't happen at midnight, it's 24 hours from when the
> process
> >>>> was
> >>>> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
> >>>> around 3:17pm on Thursday that server will go to 100% disk IO usage.
> >>> The default tombstone purge interval is 1 day, which seems to fit what
> >>> you are seeing.  The tombstone reap thread will start every 24 hours
> >>> to find tombstone entries that can be deleted.  The default retention
> >>> period for tombstones is 1 week.  It is possible that you have a large
> >>> number of tombstone entries that need to be deleted.  This will occur
> >>> independently on all of your server instances.  This is controlled by
> >>> the "nsDS5ReplicaTombstonePurgeInterval" and
> "nsDS5ReplicaPurgeDelay"
> >>> attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
> >>> tree,cn=config" entry.
> >>>
> >> I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so it's
> using
> >> that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
> > Ok, so this means every 24 hours, the tombstone reap thread will look
> > for tombstones older than 1 hour and remove them.
> >>
> >>
> >>> You can search for "(objectclass=nstombstone)" as Directory Manager
> to
> >>> see how many tombstone entries you have.
> >> I have a LOT of tombstone entries, over 200k on this one server (I'm
> >> guessing since I've been restarting the process for over a week now,
> >> not
> >> letting it run the cleanup process).
> > That's possible if you really do 200k delete operations in 1 week,
> > but that sounds like a lot.  It would seem that these tombstones have
> > been building up for a longer time than 1 week.
> >>
> >> So, any suggestions on what can I do to fix this?  The process that's
> >> reaping the entries is using too much IO making queries time out, older
> >> versions of the software did not exhibit this behavior.  In fact, I can
> >> reinitalize the entire replica faster than this thing is reaping the
> >> entries, it takes 7 minutes to reinit a replica, but when this issue
> >> first started I let the dirsrv run much longer before restarting it.
> > Due to the number of matching entries for the tombstone search, it is
> > having to walk your entire database, which is why you see the IO
> > spiking.
>
>  Perhaps also try increasing nsslapd-idlistscanlimit so that it can
>  hold the entire candidate list of tombstones to delete -
>
> http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Admin...
>  
Is there any way that I can remove the nsTombstone entries from the
master server so I can get this under control?  I think I found out why
I have so many nsTombstone entries in the first place so I'd like to get
the current ones deleted and see how much delete activity I'll have
moving forward.

I saw this bug report,
<https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
indicate that I should be able to ldapdelete the nsTombstone entries
using their full dn, however it always says object not found.

I'd rather not fully export and reimport the master and then reinit all
the replicas if I can avoid it.

--
Brad

[Rich Megginson]

On 05/16/2012 06:48 PM, Brad Schuetz wrote:
>  On 05/16/2012 04:01 PM, Rich Megginson wrote:
> > On 05/16/2012 04:06 PM, Nathan Kinder wrote:
> >> On 05/16/2012 01:09 PM, Brad Schuetz wrote:
> >>> On 05/16/2012 11:54 AM, Nathan Kinder wrote:
> >>>> On 05/16/2012 11:19 AM, Brad Schuetz wrote:
> >>>>> On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >>>>>> The exact timing of the issue is to strange is there a backup
> job
> >>>>>> running at midnight. Or some other timed job that could be
> eating
> >>>>>> the
> >>>>>> ram or disk IO. Possibly one that is reliant on ldap queries
> that
> >>>>>> would otherwise be inocuious.
> >>>>>>
> >>>>>>
> >>>>> It doesn't happen at midnight, it's 24 hours from when the
> process
> >>>>> was
> >>>>> started, so I can restart dirsrv at 3:17pm on Wednesday and at right
> >>>>> around 3:17pm on Thursday that server will go to 100% disk IO usage.
> >>>> The default tombstone purge interval is 1 day, which seems to fit what
> >>>> you are seeing.  The tombstone reap thread will start every 24 hours
> >>>> to find tombstone entries that can be deleted.  The default retention
> >>>> period for tombstones is 1 week.  It is possible that you have a large
> >>>> number of tombstone entries that need to be deleted.  This will occur
> >>>> independently on all of your server instances.  This is controlled by
> >>>> the "nsDS5ReplicaTombstonePurgeInterval" and
> "nsDS5ReplicaPurgeDelay"
> >>>> attributes in your "cn=replica,cn=<suffixDN>,cn=mapping
> >>>> tree,cn=config" entry.
> >>>>
> >>> I have no "nsDS5ReplicaTombstonePurgeInterval" value set (so
> it's using
> >>> that default), and "nsDS5ReplicaPurgeDelay" is set to 3600
> >> Ok, so this means every 24 hours, the tombstone reap thread will look
> >> for tombstones older than 1 hour and remove them.
> >>>
> >>>> You can search for "(objectclass=nstombstone)" as Directory
> Manager to
> >>>> see how many tombstone entries you have.
> >>> I have a LOT of tombstone entries, over 200k on this one server (I'm
> >>> guessing since I've been restarting the process for over a week now,
> >>> not
> >>> letting it run the cleanup process).
> >> That's possible if you really do 200k delete operations in 1 week,
> >> but that sounds like a lot.  It would seem that these tombstones have
> >> been building up for a longer time than 1 week.
> >>> So, any suggestions on what can I do to fix this?  The process that's
> >>> reaping the entries is using too much IO making queries time out, older
> >>> versions of the software did not exhibit this behavior.  In fact, I can
> >>> reinitalize the entire replica faster than this thing is reaping the
> >>> entries, it takes 7 minutes to reinit a replica, but when this issue
> >>> first started I let the dirsrv run much longer before restarting it.
> >> Due to the number of matching entries for the tombstone search, it is
> >> having to walk your entire database, which is why you see the IO
> >> spiking.
> > Perhaps also try increasing nsslapd-idlistscanlimit so that it can
> > hold the entire candidate list of tombstones to delete -
> >
> http://docs.redhat.com/docs/en-US/Red_Hat_Directory_Server/9.0/html/Admin...
> >
>  Is there any way that I can remove the nsTombstone entries from the
>  master server so I can get this under control?  I think I found out why
>  I have so many nsTombstone entries in the first place so I'd like to get
>  the current ones deleted and see how much delete activity I'll have
>  moving forward.
>
>  I saw this bug report,
>  <https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
>  indicate that I should be able to ldapdelete the nsTombstone entries
>  using their full dn, however it always says object not found. 
Can you provide
1) the exact ldapdelete command line
2) the excerpts from the access and errors log from around the time of 
the deletion


>  I'd rather not fully export and reimport the master and then reinit all
>  the replicas if I can avoid it.
>
>  --
>  Brad
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Brad Schuetz]

On 05/16/2012 06:24 PM, Rich Megginson wrote:
>  On 05/16/2012 06:48 PM, Brad Schuetz wrote:
> > Is there any way that I can remove the nsTombstone entries from the
> > master server so I can get this under control?  I think I found out why
> > I have so many nsTombstone entries in the first place so I'd like to get
> > the current ones deleted and see how much delete activity I'll have
> > moving forward.
> >
> > I saw this bug report,
> > <https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
> > indicate that I should be able to ldapdelete the nsTombstone entries
> > using their full dn, however it always says object not found.
>
>  Can you provide
>  1) the exact ldapdelete command line
>  2) the excerpts from the access and errors log from around the time of
>  the deletion 
Here's the command with only the username part of the DN redacted and
the access log for the command, I confirmed that after the delete failed
that the entry was in fact still there by running the ldap search again
for nstombstone entries to be sure that it wasn't just a timing issue. 
Also attempted the same command with quotes around the DN to delete. 
The error log produced no entries around the time of the command being run.

ldapdelete -x -D 'cn=Directory Manager' -W
nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,dc=omnis,dc=com

This was the output from the above command:

ldap_delete: No such object (32)
        matched DN:
uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com

access log:
17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
::1 to ::1
[17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
[17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
Manager" method=128 version=3
[17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
nentries=0 etime=0 dn="cn=directory manager"
[17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
nentries=0 etime=0
[17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
[17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1

However I can report that the internal process seems to have finally
cleaned up the bulk of the entries, I had over 20k entries on the master
and it's finally down to less than 100.  I temporarily changed the
values for when it should reap the entries to try and get it to clear
them out internally after I fixed the issue that I believe was
responsible for the extraneous deletes.

--
Brad

[Rich Megginson]

On 05/16/2012 07:48 PM, Brad Schuetz wrote:
>  On 05/16/2012 06:24 PM, Rich Megginson wrote:
> > On 05/16/2012 06:48 PM, Brad Schuetz wrote:
> >> Is there any way that I can remove the nsTombstone entries from the
> >> master server so I can get this under control?  I think I found out why
> >> I have so many nsTombstone entries in the first place so I'd like to get
> >> the current ones deleted and see how much delete activity I'll have
> >> moving forward.
> >>
> >> I saw this bug report,
> >> <https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
> >> indicate that I should be able to ldapdelete the nsTombstone entries
> >> using their full dn, however it always says object not found.
> > Can you provide
> > 1) the exact ldapdelete command line
> > 2) the excerpts from the access and errors log from around the time of
> > the deletion
>  Here's the command with only the username part of the DN redacted and
>  the access log for the command, I confirmed that after the delete failed
>  that the entry was in fact still there by running the ldap search again
>  for nstombstone entries to be sure that it wasn't just a timing issue.
>  Also attempted the same command with quotes around the DN to delete.
>  The error log produced no entries around the time of the command being run.
>
>  ldapdelete -x -D 'cn=Directory Manager' -W
>
> nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,dc=omnis,dc=com
>
>  This was the output from the above command:
>
>  ldap_delete: No such object (32)
>           matched DN:
>  uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com 
Looks like 617862 is back
. . .
>  access log:
>  17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
>  ::1 to ::1
>  [17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
>  [17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
>  Manager" method=128 version=3
>  [17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
>  nentries=0 etime=0 dn="cn=directory manager"
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
>
> dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>  nentries=0 etime=0
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>  nentries=0 etime=0
>  [17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
>  [17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1
>
>  However I can report that the internal process seems to have finally
>  cleaned up the bulk of the entries, I had over 20k entries on the master
>  and it's finally down to less than 100.  I temporarily changed the
>  values for when it should reap the entries to try and get it to clear
>  them out internally after I fixed the issue that I believe was
>  responsible for the extraneous deletes.
>
>  --
>  Brad

[Paul Robert Marino]

By the way despite the recomendation against reducing the timer on cleaning
out the tombstones  I would do it any way.
My experience with many other database platforms is that if the cleanups of
stale entries is taking too long it means you aren't doing them often
enough for your environment.
If you do a lot of deletes more often than most other people you need to do
cleanups more often than other people no matter what the experts say.
Bulk cleanups are ment to make the process more optimal and allow for role
back of changes but they aren't always practical for every implementation.
 On May 16, 2012 10:07 PM, "Rich Megginson" <rmeggins(a)redhat.com&gt; wrote:

>  On 05/16/2012 07:48 PM, Brad Schuetz wrote:
>
> > On 05/16/2012 06:24 PM, Rich Megginson wrote:
> >
> >> On 05/16/2012 06:48 PM, Brad Schuetz wrote:
> >>
> >>> Is there any way that I can remove the nsTombstone entries from the
> >>> master server so I can get this under control?  I think I found out why
> >>> I have so many nsTombstone entries in the first place so I'd like to get
> >>> the current ones deleted and see how much delete activity I'll have
> >>> moving forward.
> >>>
> >>> I saw this bug report,
> >>>
> <https://bugzilla.redhat.com/**show_bug.cgi?id=617862<https://bugzilla....;,
> >>> that seems to
> >>> indicate that I should be able to ldapdelete the nsTombstone entries
> >>> using their full dn, however it always says object not found.
> >>>
> >> Can you provide
> >> 1) the exact ldapdelete command line
> >> 2) the excerpts from the access and errors log from around the time of
> >> the deletion
> >>
> > Here's the command with only the username part of the DN redacted and
> > the access log for the command, I confirmed that after the delete failed
> > that the entry was in fact still there by running the ldap search again
> > for nstombstone entries to be sure that it wasn't just a timing issue.
> > Also attempted the same command with quotes around the DN to delete.
> > The error log produced no entries around the time of the command being
> > run.
> >
> > ldapdelete -x -D 'cn=Directory Manager' -W
> > nsuniqueid=14403301-1dd211b2-**9635fc8a-5f520000,uid=**
> > aeria005,ou=Users,ou=System,**dc=cust,dc=omnis,dc=com
> >
> > This was the output from the above command:
> >
> > ldap_delete: No such object (32)
> >         matched DN:
> > uid=[redacted],ou=Users,ou=**System,dc=cust,dc=omnis,dc=com
> >
>  Looks like 617862 is back . . .
>
> >
> > access log:
> > 17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
> > ::1 to ::1
> > [17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
> > [17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
> > Manager" method=128 version=3
> > [17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
> > nentries=0 etime=0 dn="cn=directory manager"
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
> > dn="nsuniqueid=14403301-**1dd211b2-9635fc8a-5f520000,**
> > uid=[redacted],ou=Users,ou=**System,dc=cust,dc=omnis,dc=**com"
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
> > nentries=0 etime=0
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
> > nentries=0 etime=0
> > [17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
> > [17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1
> >
> > However I can report that the internal process seems to have finally
> > cleaned up the bulk of the entries, I had over 20k entries on the master
> > and it's finally down to less than 100.  I temporarily changed the
> > values for when it should reap the entries to try and get it to clear
> > them out internally after I fixed the issue that I believe was
> > responsible for the extraneous deletes.
> >
> > --
> > Brad
> >
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.**org <389-users(a)lists.fedoraproject.org&gt;
>
> https://admin.fedoraproject.**org/mailman/listinfo/389-users<https://a...

[Rich Megginson]

On 05/16/2012 07:48 PM, Brad Schuetz wrote:
>  On 05/16/2012 06:24 PM, Rich Megginson wrote:
> > On 05/16/2012 06:48 PM, Brad Schuetz wrote:
> >> Is there any way that I can remove the nsTombstone entries from the
> >> master server so I can get this under control?  I think I found out why
> >> I have so many nsTombstone entries in the first place so I'd like to get
> >> the current ones deleted and see how much delete activity I'll have
> >> moving forward.
> >>
> >> I saw this bug report,
> >> <https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
> >> indicate that I should be able to ldapdelete the nsTombstone entries
> >> using their full dn, however it always says object not found.
> > Can you provide
> > 1) the exact ldapdelete command line
> > 2) the excerpts from the access and errors log from around the time of
> > the deletion
>  Here's the command with only the username part of the DN redacted and
>  the access log for the command, I confirmed that after the delete failed
>  that the entry was in fact still there by running the ldap search again
>  for nstombstone entries to be sure that it wasn't just a timing issue.
>  Also attempted the same command with quotes around the DN to delete.
>  The error log produced no entries around the time of the command being run.
>
>  ldapdelete -x -D 'cn=Directory Manager' -W
>
> nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,dc=omnis,dc=com
>
>  This was the output from the above command:
>
>  ldap_delete: No such object (32)
>           matched DN:
>  uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com
>
>  access log:
>  17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
>  ::1 to ::1
>  [17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
>  [17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
>  Manager" method=128 version=3
>  [17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
>  nentries=0 etime=0 dn="cn=directory manager"
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
>
> dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>  nentries=0 etime=0
>  [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
>  nentries=0 etime=0
>  [17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
>  [17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1
>
>  However I can report that the internal process seems to have finally
>  cleaned up the bulk of the entries, I had over 20k entries on the master
>  and it's finally down to less than 100.  I temporarily changed the
>  values for when it should reap the entries to try and get it to clear
>  them out internally after I fixed the issue that I believe was
>  responsible for the extraneous deletes. 
Good.  I filed a ticket about the tombstone
deletion issue:
https://fedorahosted.org/389/ticket/376

The fact that there are two result lines for the operation is a good 
clue that something is going wrong.

>  --
>  Brad

[Brad Schuetz]

On 05/17/2012 08:30 AM, Rich Megginson wrote:
>  On 05/16/2012 07:48 PM, Brad Schuetz wrote:
> > On 05/16/2012 06:24 PM, Rich Megginson wrote:
> >> On 05/16/2012 06:48 PM, Brad Schuetz wrote:
> >>> Is there any way that I can remove the nsTombstone entries from the
> >>> master server so I can get this under control?  I think I found out
> >>> why
> >>> I have so many nsTombstone entries in the first place so I'd like
> >>> to get
> >>> the current ones deleted and see how much delete activity I'll have
> >>> moving forward.
> >>>
> >>> I saw this bug report,
> >>> <https://bugzilla.redhat.com/show_bug.cgi?id=617862>;, that seems to
> >>> indicate that I should be able to ldapdelete the nsTombstone entries
> >>> using their full dn, however it always says object not found.
> >> Can you provide
> >> 1) the exact ldapdelete command line
> >> 2) the excerpts from the access and errors log from around the time of
> >> the deletion
> > Here's the command with only the username part of the DN redacted and
> > the access log for the command, I confirmed that after the delete failed
> > that the entry was in fact still there by running the ldap search again
> > for nstombstone entries to be sure that it wasn't just a timing issue.
> > Also attempted the same command with quotes around the DN to delete.
> > The error log produced no entries around the time of the command
> > being run.
> >
> > ldapdelete -x -D 'cn=Directory Manager' -W
> >
> nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=aeria005,ou=Users,ou=System,dc=cust,dc=omnis,dc=com
> >
> >
> > This was the output from the above command:
> >
> > ldap_delete: No such object (32)
> >          matched DN:
> > uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com
> >
> > access log:
> > 17/May/2012:01:41:35 +0000] conn=947 fd=80 slot=80 SSL connection from
> > ::1 to ::1
> > [17/May/2012:01:41:35 +0000] conn=947 SSL 256-bit AES
> > [17/May/2012:01:41:35 +0000] conn=947 op=0 BIND dn="cn=Directory
> > Manager" method=128 version=3
> > [17/May/2012:01:41:35 +0000] conn=947 op=0 RESULT err=0 tag=97
> > nentries=0 etime=0 dn="cn=directory manager"
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 DEL
> >
> dn="nsuniqueid=14403301-1dd211b2-9635fc8a-5f520000,uid=[redacted],ou=Users,ou=System,dc=cust,dc=omnis,dc=com"
> >
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
> > nentries=0 etime=0
> > [17/May/2012:01:41:35 +0000] conn=947 op=1 RESULT err=32 tag=107
> > nentries=0 etime=0
> > [17/May/2012:01:41:35 +0000] conn=947 op=2 UNBIND
> > [17/May/2012:01:41:35 +0000] conn=947 op=2 fd=80 closed - U1
> >
> > However I can report that the internal process seems to have finally
> > cleaned up the bulk of the entries, I had over 20k entries on the master
> > and it's finally down to less than 100.  I temporarily changed the
> > values for when it should reap the entries to try and get it to clear
> > them out internally after I fixed the issue that I believe was
> > responsible for the extraneous deletes.
>  Good.  I filed a ticket about the tombstone deletion issue:
>  https://fedorahosted.org/389/ticket/376
>
>  The fact that there are two result lines for the operation is a good
>  clue that something is going wrong.
>  
Looks like my IO issue is resolved now that the tombstone entries aren't
going crazy.

Thanks for the help everyone.

--
Brad

[Nathan Kinder]

On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
>  The exact timing of the issue is to strange is there a backup job 
>  running at midnight. Or some other timed job that could be eating the 
>  ram or disk IO. Possibly one that is reliant on ldap queries that 
>  would otherwise be inocuious.
>  
It is possible that it is the tombstone reap thread, which runs 
periodically.  Do you do a lot of entry deletion operations throughout 
the typical day?
>  On May 16, 2012 7:51 AM, "Gregory Matthews" 
>  <greg.matthews(a)diamond.ac.uk <mailto:[e-mail]>> wrote:
>
>      On 15/05/12 19:22, Brad Schuetz wrote:
>
>          Of the 7 replicas, 5 are attached to one network, and 2 are on
>          another
>          network.  The 5 are queried a LOT, the other 2 barely get any
>          traffic at
>          all.  All, however, are getting the same traffic that they
>          were getting
>          when I was using previous versions of the LDAP server.
>
>          The 2 that are barely used I've checked for excessive queries
>          being run
>          at the point when load goes crazy and they are getting the
>          usual minimal
>          load.
>
>          Also that doesn't explain why it's always 24 hours that it goes
>          haywire.  It doesn't matter when I restart the service, it
>          could be
>          restarted at 2am, then in 24 hours it will go crazy IO load.
>
>
>      do you know what the IO is? is it swapping? are you running
>      collectl or similar so you can look at historic performance data?
>
>      G
>
>
>          --
>          Brad
>          --
>          389 users mailing list
>          389-users(a)lists.fedoraproject.org
>          <mailto:[e-mail]>
>          https://admin.fedoraproject.org/mailman/listinfo/389-users
>
>
>
>      -- 
>      Greg Matthews        01235 778658
>      Scientific Computing Group Leader
>      Diamond Light Source Ltd. OXON UK
>
>      -- 
>      This e-mail and any attachments may contain confidential,
>      copyright and or privileged material, and are for the use of the
>      intended addressee only. If you are not the intended addressee or
>      an authorised recipient of the addressee please notify us of
>      receipt by returning the e-mail and do not use, copy, retain,
>      distribute or disclose the information in or attached to the e-mail.
>      Any opinions expressed within this e-mail are those of the
>      individual and not necessarily of Diamond Light Source Ltd.
>      Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>      attachments are free from viruses and we cannot accept liability
>      for any damage which you may sustain as a result of software
>      viruses which may be transmitted in or with the message.
>      Diamond Light Source Limited (company no. 4375679). Registered in
>      England and Wales with its registered office at Diamond House,
>      Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11
>      0DE, United Kingdom
>
>
>
>
>      --
>      389 users mailing list
>      389-users(a)lists.fedoraproject.org
>      <mailto:[e-mail]>
>      https://admin.fedoraproject.org/mailman/listinfo/389-users
>
>
>
>  --
>  389 users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/389-users

[Brad Schuetz]

On 05/16/2012 11:19 AM, Nathan Kinder wrote:
>  On 05/16/2012 06:16 AM, Paul Robert Marino wrote:
> >
> > The exact timing of the issue is to strange is there a backup job
> > running at midnight. Or some other timed job that could be eating the
> > ram or disk IO. Possibly one that is reliant on ldap queries that
> > would otherwise be inocuious.
> >
>  It is possible that it is the tombstone reap thread, which runs
>  periodically.  Do you do a lot of entry deletion operations throughout
>  the typical day?
>  
I don't have the exact numbers, but yes, I would say there are a
significant amount of entry deletions on any given day.

--
Brad

[Brad Schuetz]

On 05/16/2012 04:51 AM, Gregory Matthews wrote:
>  On 15/05/12 19:22, Brad Schuetz wrote:
> > Of the 7 replicas, 5 are attached to one network, and 2 are on another
> > network.  The 5 are queried a LOT, the other 2 barely get any traffic at
> > all.  All, however, are getting the same traffic that they were getting
> > when I was using previous versions of the LDAP server.
> >
> > The 2 that are barely used I've checked for excessive queries being run
> > at the point when load goes crazy and they are getting the usual minimal
> > load.
> >
> > Also that doesn't explain why it's always 24 hours that it goes
> > haywire.  It doesn't matter when I restart the service, it could be
> > restarted at 2am, then in 24 hours it will go crazy IO load.
>
>  do you know what the IO is? is it swapping? are you running collectl
>  or similar so you can look at historic performance data?
>
>  G
>  
It's not swap, and happens regardless of the amount of the ram in the
server.

I've run "sysctl -x 1" for random periods of time both before and after
the issue hits and the IO is very low, < 20% most of the time with is
typically < 5% usage.

But when the problem happens sysctl reports > 95% usage.

--
Brad