Hi Kathy,

The procedure to diagnose hang looks nice. My understanding is that it assumes that in deadlock situation the more we have threads waiting on a resource, the more probable we have a hang/deadlock. Now because of the dynamic of the server itself, on the configuration, on the type of requests, on the monitoring of rwlock vs simple locks, on the impact on gdb prone to stop threads on locks, this scripts may fails to detect hang/deadlock.

ATM I see no specific 8.6 enhancements that can explain why the script is failing now.

If the server is no longer responsive to a client request (keepalive req), I suggest you collect pstacks, 'top -H -p `pidof ns-slapd`' and db_stat -N -CA. If it is eating CPU you may looks at the activity of thread consuming CPU. If it is not, it is possibly deadlock between backend/database access.

Best regards
thierry

On 8/30/22 11:20 PM, Kathy Zhu via FreeIPA-users wrote:
Hi Team, 

We used following to get the number of rwlocks for /usr/sbin/ns-slapd process in Centos 7.9 to catch deadlocks: 

PID=`pidof ns-slapd`

gdb -ex 'set confirm off' -ex 'set pagination off' -ex 'thread apply all bt full' -ex 'quit' /usr/sbin/ns-slapd $PID |& grep '^#0.*lock' | grep pthread_rwlock | sort -u 


That helped us to detect ns-slapd hang caused by deadlocks. 


After migrating to Red Hat 8.6, we had a lot of hangs (dirsvr is running but not responding) and could not find why. We use the same above method, however, we are not able to catch anything. I wonder if there is a different way to count the rwlocks in Red Hat 8.6? 


We realize that there are multiple reasons to cause hangs, however, we would like to rule out the possibility of the deadlock. 


The OS and packages: 


Red Hat Enterprise Linux release 8.6 (Ootpa)

ipa-server.x86_64 4.9.8-7.module+el8.6.0+14337+19b76db2 @rhel-8-for-x86_64-appstream-rpms

slapi-nis-0.56.6-4.module+el8.6.0+12936+736896b2.x86_64

389-ds-base-libs-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64

389-ds-base-1.4.3.28-6.module+el8.6.0+14129+983ceada.x86_64



Many thanks. 


Kathy. 



_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue