dirsrv becomes completely unresponsible

[Jan Kowalsky]

Hi all,

we still struggeling with multiple problems in our 389-ds setup.

I already opened the threads "disk i/o: very high write rates" and
"ldapsearch performance problem​".

Another (the biggest) problem is, that dirsrv becomes suddenly totally
unresponsive. The server is still running, no errors in error-log - but
ldap queries are not answered and run in timeouts.

The only way is to restart it. After restart everything is fine - for a
while. But sometimes it hangs immediately again after restart.

I logged a straces while dirsrv was unresponsive:
  https://nextcloud.datenkollektiv.net/s/qD8ojQZztMqbQtp

Maybe someone has an idea?

Thanks and regards.
Jan

[Rob Crittenden]

Jan Kowalsky wrote:
>  Hi all,
>  
>  we still struggeling with multiple problems in our 389-ds setup.
>  
>  I already opened the threads "disk i/o: very high write rates" and
>  "ldapsearch performance problem​".
>  
>  Another (the biggest) problem is, that dirsrv becomes suddenly totally
>  unresponsive. The server is still running, no errors in error-log - but
>  ldap queries are not answered and run in timeouts.
>  
>  The only way is to restart it. After restart everything is fine - for a
>  while. But sometimes it hangs immediately again after restart.
>  
>  I logged a straces while dirsrv was unresponsive:
>    https://nextcloud.datenkollektiv.net/s/qD8ojQZztMqbQtp 
You want to follow these instructions and file a bug with the output,
https://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs

rob

[Jan Kowalsky]

Hi Rob,

thanks for answer.

Am 21.08.2018 um 14:05 schrieb Rob Crittenden:
>  Jan Kowalsky wrote:
> > Hi all,
> >
> > we still struggeling with multiple problems in our 389-ds setup.
> >
> > I already opened the threads "disk i/o: very high write rates" and
> > "ldapsearch performance problem​".
> >
> > Another (the biggest) problem is, that dirsrv becomes suddenly totally
> > unresponsive. The server is still running, no errors in error-log - but
> > ldap queries are not answered and run in timeouts.
> >
> > The only way is to restart it. After restart everything is fine - for a
> > while. But sometimes it hangs immediately again after restart.
> >
> > I logged a straces while dirsrv was unresponsive:
> >   https://nextcloud.datenkollektiv.net/s/qD8ojQZztMqbQtp
>  
>  You want to follow these instructions and file a bug with the output,
>  https://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs 
Unfortunately debugging with gdb and pstack doesn't work in my
environment (Debian Jessie The debugging symbols are installed - but
something seems to be wrong:

gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
all
bt full' -ex 'quit' /usr/sbin/ns-slapd `pidof ns-slapd`

Thread 207 (Thread 0x7f2eb4f51700 (LWP 25429)):
#0  0x00007f2f661d7893 in select () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

....



dpkg -l|grep 389-ds

ii  389-ds                            1.3.3.5-4+deb8u1
  all          389 Directory Server suite - metapackage
ii  389-ds-base                       1.3.3.5-4+deb8u1
  amd64        389 Directory Server suite - server
ii  389-ds-base-dbg                   1.3.3.5-4+deb8u1
  amd64        389 Directory Server suite - server debugging symbols
ii  389-ds-base-libs                  1.3.3.5-4+deb8u1
  amd64        389 Directory Server suite - libraries
ii  389-ds-base-libs-dbg              1.3.3.5-4+deb8u1
  amd64        389 Directory Server suite - library debugging symbols
ii  389-ds-console                    1.2.7-1
  all          389 Directory Server Management Console
ii  389-dsgw                          1.1.11-1
  amd64        389 Directory Server Gateway

I'll try to update to stretch - maybe the dgb packages in jessie are
buggy. Or any other idea?

Thanks
Jan

[William Brown]

On Tue, 2018-08-21 at 14:38 +0200, Jan Kowalsky wrote:
>  Hi Rob,
>  
>  thanks for answer.
>  
>  Am 21.08.2018 um 14:05 schrieb Rob Crittenden:
>  > Jan Kowalsky wrote:
>  > > Hi all,
>  > > 
>  > > we still struggeling with multiple problems in our 389-ds setup.
>  > > 
>  > > I already opened the threads "disk i/o: very high write rates"
>  > > and
>  > > "ldapsearch performance problem".
>  > > 
>  > > Another (the biggest) problem is, that dirsrv becomes suddenly
>  > > totally
>  > > unresponsive. The server is still running, no errors in error-log 
>  > > - but
>  > > ldap queries are not answered and run in timeouts. 
Seeing an access log could help, and there are some tools to help dump
the in memory access log when the hang happens. 

I plan to write in profiling soon (if I get the freetime ....)  so that
could help too,



>  > > 
>  > > The only way is to restart it. After restart everything is fine -
>  > > for a
>  > > while. But sometimes it hangs immediately again after restart.
>  > > 
>  > > I logged a straces while dirsrv was unresponsive:
>  > >   https://nextcloud.datenkollektiv.net/s/qD8ojQZztMqbQtp
>  > 
>  > You want to follow these instructions and file a bug with the
>  > output,
>  >  
https://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#debugging-hangs
>  
>  Unfortunately debugging with gdb and pstack doesn't work in my
>  environment (Debian Jessie The debugging symbols are installed - but
>  something seems to be wrong:
>  
>  gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply
>  all
>  bt full' -ex 'quit' /usr/sbin/ns-slapd `pidof ns-slapd`
>  
>  Thread 207 (Thread 0x7f2eb4f51700 (LWP 25429)):
>  #0  0x00007f2f661d7893 in select () from /lib/x86_64-linux-
>  gnu/libc.so.6
>  No symbol table info available.
>  
>  ....
>  
>  
>  
>  dpkg -l|grep 389-ds
>  
>  ii  389-ds                            1.3.3.5-4+deb8u1
>    all          389 Directory Server suite - metapackage
>  ii  389-ds-base                       1.3.3.5-4+deb8u1
>    amd64        389 Directory Server suite - server
>  ii  389-ds-base-dbg                   1.3.3.5-4+deb8u1
>    amd64        389 Directory Server suite - server debugging symbols
>  ii  389-ds-base-libs                  1.3.3.5-4+deb8u1
>    amd64        389 Directory Server suite - libraries
>  ii  389-ds-base-libs-dbg              1.3.3.5-4+deb8u1
>    amd64        389 Directory Server suite - library debugging symbols
>  ii  389-ds-console                    1.2.7-1
>    all          389 Directory Server Management Console
>  ii  389-dsgw                          1.1.11-1
>    amd64        389 Directory Server Gateway 

That's a pretty old version now, we have a lot of fixes since there.
Are there some debian updates perhaps to help? 

>  
>  I'll try to update to stretch - maybe the dgb packages in jessie are
>  buggy. Or any other idea?
>  
>  Thanks
>  Jan
>  _______________________________________________
>  389-users mailing list -- 389-users(a)lists.fedoraproject.org
>  To unsubscribe send an email to 
>  389-users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
-- 
Sincerely,

William