in strace.log:
[pid 8088] 12:55:39.739539 poll([{fd=435, events=POLLOUT}], 1, 1800000
<unfinished ...>
[pid 8058] 12:55:39.739573 <... write resumed> ) = 1 <0.000087>
[pid 8088] 12:55:39.739723 <... poll resumed> ) = 1 ([{fd=435,
revents=POLLOUT}]) <0.000168>
[pid 8058] 12:55:39.739754 write(426, "dn: cn=ntU"..., 344 <unfinished
...>
[pid 8088] 12:55:39.739824 sendto(435, "0<\2\1\2d7\0043u"..., 62, 0,
NULL, 0 <unfinished ...>
Usually you need to do a few iterations on the strace data to get to the
bottom of things.
I took a look at the full file you uploaded. Couple of things :
1. With a truncated strace it helps to know the identity of the FDs. You
can use "lsof" or dump the data out of /proc /sys...somewhere.
2. You can tell strace to only output certain classes of syscalls. For
this problem, where filesystem I/O is the target, you can use : strace
-e trace=file and
-e trace=desc. More details here :
https://linux-audit.com/the-ultimate-strace-cheat-sheet/ and also in the
man page. This will remove the extra noise (gettimeofday() etc) in the
output.
That said, I saw this, which looks possibly like a smoking gun:
[pid 8058] 12:55:40.563393 open("/etc/dirsrv/slapd-ldap0/dse.ldif.tmp",
O_RDWR|O_CREAT|O_TRUNC, 0600 <unfinished ...>
[pid 3186] 12:55:40.563826 <... select resumed> ) = 0 (Timeout) <0.009394>
[pid 8058] 12:55:40.563855 <... open resumed> ) = 426 <0.000431>
It looks to have opened dse.ldif, then assuming that FD = 426 is from
the open() call, it goes on to write a bunch of data to that file :
[pid 8058] 12:55:40.567465 write(426, "dn: cn=mon"..., 413) = 413
<0.000209>
[pid 8058] 12:55:40.567826 write(426, "\n", 1) = 1 <0.000356>
[pid 8058] 12:55:40.568601 write(426, "dn: cn=cha"..., 318) = 318
<0.000058>
[pid 8058] 12:55:40.568727 write(426, "\n", 1) = 1 <0.000045>
[pid 8058] 12:55:40.568937 write(426, "dn: cn=enc"..., 321) = 321
<0.000042>
[pid 8058] 12:55:40.569040 write(426, "\n", 1) = 1 <0.000041>
[pid 8058] 12:55:40.569182 write(426, "dn: cn=fea"..., 100) = 100
<0.000042>
[pid 8058] 12:55:40.569281 write(426, "\n", 1) = 1 <0.000040>
[pid 8058] 12:55:40.569427 write(426, "dn: cn=kol"..., 409) = 409
<0.000042>
[pid 8058] 12:55:40.569528 write(426, "\n", 1) = 1 <0.000041>
dse.ldif is supposed to hold seldom-changing config data for the server.
So...one theory is that something the server is doing, or you are
unknowingly asking it to do, is causing dse.ldif to be re-written
constantly.
As you point out, the high filesystem I/O may not be the cause for the
poor search performance, but certainly the server should not be
constantly re-writing dse.ldif.