389-users June 2020

389-users@lists.fedoraproject.org

13 participants
19 discussions

Announcing 389 Directory Server 1.4.2.15
by Mark Reynolds 15 Jun '20

15 Jun '20

389 Directory Server 1.4.2.15 The 389 Directory Server team is proud to announce 389-ds-base version 1.4.2.15 Fedora packages are available on Fedora 31. https://koji.fedoraproject.org/koji/taskinfo?taskID=45761486 <https://koji.fedoraproject.org/koji/taskinfo?taskID=45761486> Bodhi https://bodhi.fedoraproject.org/updates/FEDORA-2020-7b44e02730 <https://bodhi.fedoraproject.org/updates/FEDORA-2020-7b44e02730> The new packages and versions are: * 389-ds-base-1.4.2.15-1 Source tarballs are available for download at Download 389-ds-base Source <https://releases.pagure.org/389-ds-base/389-ds-base-1.4.2.15.tar.bz2> Highlights in 1.4.2.15 * Bug fixes, and UI completion Installation and Upgrade See Download <https://www.port389.org/docs/389ds/download.html> for information about setting up your yum repositories. To install the server use *dnf install 389-ds-base* To install the Cockpit UI plugin use *dnf install cockpit-389-ds* After rpm install completes, run *dscreate interactive* For upgrades, simply install the package. There are no further steps required. There are no upgrade steps besides installing the new rpms See Install_Guide <https://www.port389.org/docs/389ds/howto/howto-install-389.html> for more information about the initial installation and setup See Source <https://www.port389.org/docs/389ds/development/source.html> for information about source tarballs and SCM (git) access. New UI Progress (Cockpit plugin) The new UI is complete and QE tested. Feedback We are very interested in your feedback! Please provide feedback and comments to the 389-users mailing list: https://lists.fedoraproject.org/admin/lists/389-users.lists.fedoraproject.o… If you find a bug, or would like to see a new feature, file it in our Pagure project: https://pagure.io/389-ds-base * Bump version to 1.4.2.15 * Issue 51072 - Set the default minimum worker threads * Issue 51100 - Correct numSubordinates value for cn=monitor * Issue 51136 - dsctl and dsidm do not errors correctly when using JSON * Issue 51132 - Winsync setting winSyncWindowsFilter not working as expected * Issue 51072 - improve autotune defaults * Issue 50746 - Add option to healthcheck to list all the lint reports * Issue 51118 - UI - improve modal validation when creating an instance -- 389 Directory Server Development Team

1 0

Announcing 389 Directory Server 1.4.3.10
by Mark Reynolds 15 Jun '20

15 Jun '20

389 Directory Server 1.4.3.10 The 389 Directory Server team is proud to announce 389-ds-base version 1.4.3.10 Fedora packages are available on Fedora 32. https://koji.fedoraproject.org/koji/taskinfo?taskID=45760593 <https://koji.fedoraproject.org/koji/taskinfo?taskID=45760593> - Fedora 32 https://bodhi.fedoraproject.org/updates/FEDORA-2020-f356a6e096 <https://bodhi.fedoraproject.org/updates/FEDORA-2020-f356a6e096> - Bodhi The new packages and versions are: * 389-ds-base-1.4.3.10-1 Source tarballs are available for download at Download 389-ds-base Source <https://releases.pagure.org/389-ds-base/389-ds-base-1.4.3.10.tar.bz2> Highlights in 1.4.3.10 * Bug fixes, and UI completion Installation and Upgrade See Download <https://www.port389.org/docs/389ds/download.html> for information about setting up your yum repositories. To install the server use *dnf install 389-ds-base* To install the Cockpit UI plugin use *dnf install cockpit-389-ds* After rpm install completes, run *dscreate interactive* For upgrades, simply install the package. There are no further steps required. There are no upgrade steps besides installing the new rpms See Install_Guide <https://www.port389.org/docs/389ds/howto/howto-install-389.html> for more information about the initial installation and setup See Source <https://www.port389.org/docs/389ds/development/source.html> for information about source tarballs and SCM (git) access. New UI Progress (Cockpit plugin) The new UI is complete and QE tested. Feedback We are very interested in your feedback! Please provide feedback and comments to the 389-users mailing list: https://lists.fedoraproject.org/admin/lists/389-users.lists.fedoraproject.o… If you find a bug, or would like to see a new feature, file it in our Pagure project: https://pagure.io/389-ds-base * Bump version to 1.4.3.10 * Issue 49859 - A distinguished value can be missing in an entry * Issue 50791 - Healthcheck should look for notes=A/F in access log * Issue 51072 - Set the default minimum worker threads * Issue 50912 - pwdReset can be modified by a user * Issue 51100 - Correct numSubordinates value for cn=monitor * Issue 51136 - dsctl and dsidm do not errors correctly when using JSON * Issue 51132 - Winsync setting winSyncWindowsFilter not working as expected * Issue 51072 - improve autotune defaults * Issue 50746 - Add option to healthcheck to list all the lint reports * Issue 51118 - UI - improve modal validation when creating an instance -- 389 Directory Server Development Team

1 0

Re: [EXTERNAL] Re: Re: Re: Re: new server setup hanging
by William Brown 11 Jun '20

11 Jun '20

> On 12 Jun 2020, at 03:12, Crocker, Deborah <crock(a)ua.edu> wrote: > > What is it about this newer version compared to the old where this is happening. Is it that our setup is not quite the same? We try to bring all settings forward (except now it is auto-tuning cache) but it is possible we missed something. It's hard to tell. Unindexed searches like this will always hurt performance. Unindexed searches have a tendancy to blow your cache out through evicts/includes. You should check also your db monitor to see if there are many cache evictions. That would tell you that autotuning is too low. We had to develop the cache auto-tune to work with FreeIPA in mind, and so by default it uses 10% of the system ram (25% as of 1.4.4 I think ....). FreeIPA comes with a lot of other daemons like dogtag and co, and they are are memory hungry, so DS has to "share the playground" with them. There were also issues with glibc fragmenting our address space, and that caused us to "appear" to leak (We have since improved this situation of course). When autotuning was added, DS would ship with out of the box, I think 100MB of entry cache only, and some people went to production with this. Auto tuning isn't designed to be perfect, it's designed to be "better than before". And yes we'll keep improving that, but sometimes you need to tweak it to use more of the resources you have for your workload. As yet, I haven't thought of a good way to make it so that a pure 389-ds instance gets more memory, but we tune for less in freeipa to share .... You could find that changing it to 25% or 40% will improve your situation, especially if you are seeing lots of inclusions and evictions. https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/h… And again, you *really really* should index all the attributes in that query, because any query that is "notes=F|A|U" is going to be bad, and you should configure SSSD to "play nice" ie ignore_group_members=true and enumerate=false to reduce load on your directory servervs, but also to improve your client login times (it used to take 5 minutes for me to sudo at my old workplace until I set ignore_group_members=true). Hope that helps, — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs

1 0

Re: [EXTERNAL] Re: Re: Re: new server setup hanging
by William Brown 11 Jun '20

11 Jun '20

> > We have a number of linux hosts authenticating to ldap. Some of them using SSSD had "enumerate=true", Yeah, you need to disable enumerate=true, because SSSD will do paged searches and that will get around some search limits that normally would block that. As well, you probably should look at turning on "ignore_group_members=true", because if you don't have that set, then SSSD will enumerate your whole directory too. > which means they run a search for everything every five minutes. Just one of those will tie up the host. The search is: > filter="(&(objectClass=posixAccount)(uid=*)(uidNumber=*)(uaNetgroupLinuxGid=*))" > only uaNetgroupLinuxGID is unindexed. Again, this causes no problem on our existing setup. > ... > > Thread 49 (Thread 0x7fce91cb8700 (LWP 2176)): > #0 0x00007fcf0b3929ff in comp_cmp (s1p=<optimized out>, s2p=s2p@entry=0x55955e6fa140 "uaUDCid") at ldap/servers/slapd/attr.c:88 > #1 0x00007fcf0b392bc9 in slapi_attr_type_cmp (a1=a1@entry=0x55945a2b7b90 "uaee121Shell", a2=0x55955e6fa140 "uaUDCid", opt=opt@entry=2) at ldap/servers/slapd/attr.c:122 > #2 0x00007fcf0b3944ff in attrlist_find_ex (a=<optimized out>, type=type@entry=0x55945a2b7b90 "uaee121Shell", type_name_disposition=type_name_disposition@entry=0x0, actual_type_name=actual_type_name@entry=0x0, hint=hint@entry=0x7fce91cb2488) at ldap/servers/slapd/attrlist.c:176 > #3 0x00007fcf0b3b7211 in test_presence_filter (pb=pb@entry=0x0, e=e@entry=0x55955e6ee300, type=0x55945a2b7b90 "uaee121Shell", verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb25c0) at ldap/servers/slapd/filterentry.c:355 > #4 0x00007fcf0b42997e in vattr_test_filter (pb=pb@entry=0x0, e=e@entry=0x55955e6ee300, f=f@entry=0x55947509ab80, filter_type=FILTER_TYPE_PRES, type=<optimized out>) at ldap/servers/slapd/vattr.c:753 > #5 0x00007fcf0b3b6ec4 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x0, e=0x55955e6ee300, f=0x55947509ab80, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2684) at ldap/servers/slapd/filterentry.c:823 > #6 0x00007fcf0b3b7ba6 in slapi_vattr_filter_test_ext (pb=pb@entry=0x0, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0) at ldap/servers/slapd/filterentry.c:771 > #7 0x00007fcf0b3b7bf8 in slapi_vattr_filter_test (pb=pb@entry=0x0, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=0) at ldap/servers/slapd/filterentry.c:715 > #8 0x00007fcf01599e02 in acl__resource_match_aci (aclpb=aclpb@entry=0x559474f16000, aci=aci@entry=0x55947509a880, skip_attrEval=skip_attrEval@entry=0, a_matched=a_matched@entry=0x7fce91cb2bd0) at ldap/servers/plugins/acl/acl.c:2422 > #9 0x00007fcf0159b280 in acl__scan_for_acis (err=<synthetic pointer>, aclpb=0x559474f16000) at ldap/servers/plugins/acl/acl.c:1974 > #10 0x00007fcf0159b280 in acl_access_allowed (pb=<optimized out>, e=e@entry=0x55955e6ee300, attr=attr@entry=0x5595925e2ea0 "uid", val=<optimized out>, access=access@entry=2) at ldap/servers/plugins/acl/acl.c:568 > #11 0x00007fcf015ae9f7 in acl_access_allowed_main (pb=<optimized out>, e=0x55955e6ee300, attrs=<optimized out>, val=<optimized out>, access=2, flags=<optimized out>, errbuf=0x0) at ldap/servers/plugins/acl/aclplugin.c:371 > #12 0x00007fcf0b3f0cbc in plugin_call_acl_plugin (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, attrs=attrs@entry=0x7fce91cb2d10, val=val@entry=0x0, access=access@entry=2, flags=flags@entry=0, errbuf=errbuf@entry=0x0) at ldap/servers/slapd/plugin_acl.c:62 > #13 0x00007fcf0b3b638d in test_filter_access (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, attr_type=<optimized out>, attr_val=attr_val@entry=0x0) at ldap/servers/slapd/filterentry.c:956 > #14 0x00007fcf0b3b7082 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, f=f@entry=0x559475f39000, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2de4) at ldap/servers/slapd/filterentry.c:855 > #15 0x00007fcf0b3b6d31 in vattr_test_filter_list_and (ftype=160, access_check_done=0x7fce91cb2de4, only_check_access=0, verify_access=1, flist=<optimized out>, e=0x55955e6ee300, pb=0x559475874000) at ldap/servers/slapd/filterentry.c:980 > #16 0x00007fcf0b3b6d31 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x559475874000, e=0x55955e6ee300, f=<optimized out>, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2de4) at ldap/servers/slapd/filterentry.c:885 > #17 0x00007fcf0b3b7ba6 in slapi_vattr_filter_test_ext (pb=pb@entry=0x559475874000, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0) at ldap/servers/slapd/filterentry.c:771 > #18 0x00007fcf0b3b7bf8 in slapi_vattr_filter_test (pb=pb@entry=0x559475874000, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=1) at ldap/servers/slapd/filterentry.c:715 > #19 0x00007fcf002c0df1 in ldbm_back_next_search_entry_ext (pb=0x559475874000, use_extension=0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1702 > #20 0x00007fcf0b3deca6 in iterate (send_result=1, be=0x559459ae7c70, pr_statp=0x7fce91cb30a4, pagesize=<optimized out>, pnentries=0x7fce91cb3138, pb=0x559475874000) at ldap/servers/slapd/opshared.c:1292 > #21 0x00007fcf0b3deca6 in send_results_ext (pb=pb@entry=0x559475874000, nentries=nentries@entry=0x7fce91cb3138, pagesize=1000, pr_stat=pr_stat@entry=0x7fce91cb30a4, send_result=1) at ldap/servers/slapd/opshared.c:1645 > #22 0x00007fcf0b3e0474 in op_shared_search (pb=pb@entry=0x559475874000, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:683 > #23 0x000055945722cc0e in do_search (pb=pb@entry=0x559475874000) at ldap/servers/slapd/search.c:352 > #24 0x000055945721a98a in connection_dispatch_operation (pb=0x559475874000, op=0x559592580b40, conn=0x559475186510) at ldap/servers/slapd/connection.c:651 > #25 0x000055945721a98a in connection_threadmain () at ldap/servers/slapd/connection.c:1793 > #26 0x00007fcf091a0c5b in _pt_root (arg=0x559459ba5200) at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #27 0x00007fcf08b40ea5 in start_thread (arg=0x7fce91cb8700) at pthread_create.c:307 > #28 0x00007fcf081ec8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Yep, it's holding the backend lock while applying the filter test. In a condition like: "(&(objectClass=posixAccount)(uid=*)(uidNumber=*)(uaNetgroupLinuxGid=*))" You really need everything indexed because here, this really is going to have to enumerate *everything* that is an objectClass posix account, and then apply the filtertest. So you should index uaNetgroupLinuxGid, then the test can be asserted in indexes only which is significantly faster. I recommend a presence and equality index to be safe. If you look at the access log and there is any "notes=A", "notes=F", or "notes=U", you should probably check those queries and ensure that all the elements of that filter are indexed, and that all the elements of that filter are present in schema. Hope that helps, — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs

2 1

Re: [EXTERNAL] Re: Re: new server setup hanging
by William Brown 10 Jun '20

10 Jun '20

> On 1 Jun 2020, at 11:43, Crocker, Deborah <crock(a)ua.edu> wrote: > > Is this sufficient? Again, this server has a light load and we don't think we saw the problem, although I do note that the CPU usage seems pretty high for such a light load. All threads are idle except thread 1 that is checking if there are new connections. I don't see anything obviously wrong here ... :( > > > Thread 26 (Thread 0x7f0600d32700 (LWP 11330)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a136bf7 in deadlock_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5b00) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f0600d32700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 25 (Thread 0x7f0600531700 (LWP 11331)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a13a7c7 in checkpoint_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f59e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f0600531700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 24 (Thread 0x7f05ffd30700 (LWP 11332)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a136e47 in trickle_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5c20) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f05ffd30700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > ---Type <return> to continue, or q <return> to quit--- > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 23 (Thread 0x7f05ff52f700 (LWP 11333)): > #0 0x00007f06420a09a3 in select () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f06452e0649 in DS_Sleep () > at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f063a1319f7 in perf_threadmain () > at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #3 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5440) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #4 0x00007f06429fdea5 in start_thread (arg=0x7f05ff52f700) > at pthread_create.c:307 > #5 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 22 (Thread 0x7f05fed2e700 (LWP 11334)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55e900, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f063abe515e in cos_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libcos-plugin.so > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b680) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fed2e700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 21 (Thread 0x7f05fe52d700 (LWP 11335)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55ebc0, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f06382041fd in roles_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libroles-plugin.so > ---Type <return> to continue, or q <return> to quit--- > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b440) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fe52d700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 20 (Thread 0x7f05fdd2c700 (LWP 11336)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b404d3c40, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x00007f06452ccf58 in slapi_wait_condvar () > at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f06382041fd in roles_cache_wait_on_change () > at /usr/lib64/dirsrv/plugins/libroles-plugin.so > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b320) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fdd2c700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 19 (Thread 0x7f05fd52b700 (LWP 11337)): > #0 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > #1 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b55ec08, ml=0x557b4055c160, timeout=timeout@entry=30000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > #2 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b55ec00, timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > #3 0x0000557b3a0be208 in housecleaning () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b0e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fd52b700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 18 (Thread 0x7f05fcd2a700 (LWP 11338)): > ---Type <return> to continue, or q <return> to quit--- > #0 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > #1 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b61fa08, ml=0x557b3b6709a0, timeout=timeout@entry=10000) > at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > #2 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b61fa00, timeout=10000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > #3 0x00007f064526ef23 in eq_loop () > at /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b200) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fcd2a700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 17 (Thread 0x7f05fc529700 (LWP 11339)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040ad80) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fc529700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 16 (Thread 0x7f05fbd28700 (LWP 11340)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b7a0) > ---Type <return> to continue, or q <return> to quit--- > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fbd28700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 15 (Thread 0x7f05fb527700 (LWP 11341)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b8c0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fb527700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 14 (Thread 0x7f05fad26700 (LWP 11342)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040be60) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fad26700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 13 (Thread 0x7f05fa525700 (LWP 11343)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > ---Type <return> to continue, or q <return> to quit--- > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bd40) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05fa525700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 12 (Thread 0x7f05f9d24700 (LWP 11344)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bc20) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f9d24700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 11 (Thread 0x7f05f9523700 (LWP 11345)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040bb00) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f9523700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 10 (Thread 0x7f05f8d22700 (LWP 11346)): > ---Type <return> to continue, or q <return> to quit--- > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040b9e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f8d22700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 9 (Thread 0x7f05f8521700 (LWP 11347)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a120) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f8521700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 8 (Thread 0x7f05f7d20700 (LWP 11348)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a240) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f7d20700) > at pthread_create.c:307 > ---Type <return> to continue, or q <return> to quit--- > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 7 (Thread 0x7f05f751f700 (LWP 11349)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a360) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f751f700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 6 (Thread 0x7f05f6d1e700 (LWP 11350)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a5a0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f6d1e700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 5 (Thread 0x7f05f651d700 (LWP 11351)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > ---Type <return> to continue, or q <return> to quit--- > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a480) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f651d700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 4 (Thread 0x7f05f5d1c700 (LWP 11352)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a7e0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f5d1c700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 3 (Thread 0x7f05f551b700 (LWP 11353)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a900) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f551b700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 2 (Thread 0x7f05f4d1a700 (LWP 11354)): > #0 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, timeout=---Type <return> to continue, or q <return> to quit--- > 4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > #2 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > #3 0x0000557b3a0b4941 in connection_threadmain () > #4 0x00007f064305dc5b in _pt_root (arg=0x557b4040a6c0) > at ../../../nspr/pr/src/pthreads/ptthread.c:201 > #5 0x00007f06429fdea5 in start_thread (arg=0x7f05f4d1a700) > at pthread_create.c:307 > #6 0x00007f06420a98dd in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > Thread 1 (Thread 0x7f064594c940 (LWP 11328)): > #0 0x00007f064209ec3d in poll () > at ../sysdeps/unix/syscall-template.S:81 > #1 0x00007f0643059ba7 in poll (__timeout=250, __nfds=123, __fds=0x557b404ae400) at /usr/include/bits/poll2.h:46 > #2 0x00007f0643059ba7 in _pr_poll_with_poll (pds=0x557b40bb8000, npds=123, timeout=<optimized out>) > at ../../../nspr/pr/src/pthreads/ptio.c:4023 > #3 0x0000557b3a0ba149 in slapd_daemon () > #4 0x0000557b3a0abb15 in main () > (gdb) > (gdb) > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > deborah.crocker(a)ua.edu > > > -----Original Message----- > From: William Brown <wbrown(a)suse.de> > Sent: Sunday, May 31, 2020 8:00 PM > To: 389-users(a)lists.fedoraproject.org > Subject: [EXTERNAL] [389-users] Re: Re: new server setup hanging > > > >> On 1 Jun 2020, at 10:54, Crocker, Deborah <crock(a)ua.edu> wrote: >> >> We had to roll it back. There is one host running with it but the load is so light we never saw a problem. We think it was a known bug, maybe this: >> >> https://pagure.io/389-ds-base/issue/50329 > > That issue is fixed in 1.3.9, you are running 1.3.10, so that seems unlikely? > >> >> Do you want any info off the running host? > > As before, I'd need to see a gdb -p "pid" and `thread apply all bt` thanks. When you gdb -p <pid>, it will tell you a command of what debug info packages you need to install and how. You should install those before you run the gdb commands. It will cause the server to "pause" when you attach gdb btw. > >> >> We'll now probably move into the 1.4.x trees. Any advice on which is the most stable? > > We try to make sure they are all stable - if you are using Red Hat/CentOS or SLE/Suse Leap, then whatever 389-ds version are in those platforms is the "best maintained" for that platform, and we'll resolve issues in them etc. I think that's 1.4.2.x or 1.4.3.x from the top of my head at the moment for RHEL 8.x and SLE 15.x. > >> >> -----Original Message----- >> From: William Brown <wbrown(a)suse.de> >> Sent: Sunday, May 31, 2020 6:25 PM >> To: 389-users(a)lists.fedoraproject.org >> Subject: [EXTERNAL] [389-users] Re: new server setup hanging >> >> Hey there, >> >> >> We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out? >> >> gdb -p "pid" >> >> thread apply all bt >> >> >> It'd be great if you can install debug info too to help. >> >> Thanks, >> >> >>> On 31 May 2020, at 05:09, Crocker, Deborah <crock(a)ua.edu> wrote: >>> >>> Some more information from a coworker: >>> >>> Yeah, this sounds like an LDAP server bug. I haven't figured out what to look at to pin it down, but when it's slow to connect, I can see with strace that the primary thread hasn't called accept() yet for the connection I'm trying to open. Once it does, the whole thing goes very quickly, and I usually see a burst of other connections accepted and handled at the same time. >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>> >>> >>> -----Original Message----- >>> From: Crocker, Deborah <crock(a)ua.edu> >>> Sent: Saturday, May 30, 2020 2:08 PM >>> To: General discussion list for the 389 Directory server project. >>> <389-users(a)lists.fedoraproject.org> >>> Subject: [EXTERNAL] [389-users] new server setup hanging >>> >>> We're trying to move into our new server setup. We have one that seems to be fine under a load but when we bring the next we're having trouble with it hanging. The second does have more clients (and different) so there could be something about what a client is doing. Here is the server: >>> 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, running on >>> CentOS Linux release 7.8.2003 >>> >>> And here is the pstack output listing the only thread that is not idle. Can anyone tell me what is going on? >>> >>> Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): >>> #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #2 0x00007f860a8d5179 in slapi_be_select () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #3 0x00007f860a9296a0 in vattr_test_filter () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #6 0x00007f8600a99e02 in acl__resource_match_aci () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #7 0x00007f8600a9b280 in acl_access_allowed () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #8 0x00007f8600aae9f7 in acl_access_allowed_main () at >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so >>> #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #10 0x00007f860a8b638d in test_filter_access () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so >>> #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #16 0x00007f860a8e0e09 in op_shared_search () at >>> /usr/lib64/dirsrv/libslapd.so.0 >>> #17 0x0000557410dd3c0e in do_search () >>> #18 0x0000557410dc198a in connection_threadmain () >>> #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so >>> #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 >>> #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>> >>> >>> -----Original Message----- >>> From: William Brown <wbrown(a)suse.de> >>> Sent: Wednesday, May 27, 2020 5:43 PM >>> To: 389-users(a)lists.fedoraproject.org >>> Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new >>> servers into production >>> >>> >>> >>>> On 27 May 2020, at 23:20, Crocker, Deborah <crock(a)ua.edu> wrote: >>>> >>>> Thanks - I think we have enough ideas in here to get this going. One last question: >>>> If replication is set up through the host name - how often does the directory server do a DNS look up, or does it do it once on startup (or creation of the rep agreement)? >>> >>> I "think" it's every time it initiates the new connection - but remember, for replication, that *is* quite different to a client doing a search, so I'd be pretty careful about this. IMO you should be standing up your replacement servers in parallel, joining them all, moving the IP's then decomission the old servers. Alternately, you'll need an outage window to shutdown your old servers, export the ldif, and then import and bring up the new ones. >>> >>> I think having "IP's are a limited resource" really does make this >>> whole process much much harder than it needs to be for you ... :( >>> >>>> >>>> -----Original Message----- >>>> From: William Brown <wbrown(a)suse.de> >>>> Sent: Tuesday, May 26, 2020 10:48 PM >>>> To: 389-users(a)lists.fedoraproject.org >>>> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers >>>> into production >>>> >>>> There are a few options. The best would be a load balancer which has the ip's so that it's transparent to your LDAP servers where they are. >>>> >>>> But also as mentioned, the virtual IP's honestly is the best way. Linux can have multiple IP's on an interface so you can just have two IP's on one interface, andthat's the best way to do this. >>>> >>>> Alternately, don't rely on the IP, lower your DNS ttl's to a very short time, change the DNS A/AAAA records, and then do it that way. >>>> >>>> >>>> >>>>> On 27 May 2020, at 06:17, Crocker, Deborah <crock(a)ua.edu> wrote: >>>>> >>>>> I’d like not to take up two ip addresses per host indefinitely. We have re-IP’d our hosts before so I know we can to do this but it was during a downtime when everything was restarted. Just trying to get away with not restarting the masters. >>>>> >>>>> Deborah Crocker, PhD >>>>> Systems Engineer III >>>>> Office of Information Technology >>>>> The University of Alabama >>>>> Box 870346 >>>>> Tuscaloosa, AL 36587 >>>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>>>> >>>>> From: Leo Pleiman <lpleiman(a)salsalabs.com> >>>>> Sent: Tuesday, May 26, 2020 3:08 PM >>>>> To: General discussion list for the 389 Directory server project. >>>>> <389-users(a)lists.fedoraproject.org> >>>>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers >>>>> into production >>>>> >>>>> My experience has been that the replicas and consumers have a unique id, more than just an IP address which creates the trust relationship with the master. If your goal is to simply maintain an IP so your clients don't have to be repointed, I would build each new LDAP host and replication agreement, and then as you decommission the old hosts use their IP address as a virtual IP address on the replacement host. It would take a quick restart od the LDAP service to start a listener on the virtual Ip address. >>>>> >>>>> >>>>> Leo Pleiman >>>>> Senior System Engineer >>>>> Direct 202-787-3622 >>>>> Cell 410-688-3873 >>>>> >>>>> >>>>> >>>>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <crock(a)ua.edu> wrote: >>>>> We have a setup with 2 multi-masters and 3 consumers. We are now building new host and want to put them in place ultimately at the same IP address as the original ones. I need some advice on how to do this quickly and cleanly. >>>>> >>>>> To add a new consumer the idea now is to set it up and set up replications agreements from each master using consumer DNS name (don't start continuous replication yet). After initializing new consumer from one master - turn off old consumer, remove old consumer agreement from each master, and re-IP new consumer. Do we need to restart masters to re-read DNS or will it pick that up when it starts the next replication? Is this the best way to do this? >>>>> >>>>> Thanks >>>>> >>>>> Deborah Crocker, PhD >>>>> Systems Engineer III >>>>> Office of Information Technology >>>>> The University of Alabama >>>>> Box 870346 >>>>> Tuscaloosa, AL 36587 >>>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>>>> >>>>> _______________________________________________ >>>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>>> unsubscribe send an email to >>>>> 389-users-leave(a)lists.fedoraproject.org >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedor >>>>> a p r oject.org _______________________________________________ >>>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>>> unsubscribe send an email to >>>>> 389-users-leave(a)lists.fedoraproject.org >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedor >>>>> a >>>>> p >>>>> r >>>>> oject.org >>>> >>>> — >>>> Sincerely, >>>> >>>> William Brown >>>> >>>> Senior Software Engineer, 389 Directory Server SUSE Labs >>>> _______________________________________________ >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora >>>> p r oject.org _______________________________________________ >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora >>>> p >>>> r >>>> oject.org >>> >>> — >>> Sincerely, >>> >>> William Brown >>> >>> Senior Software Engineer, 389 Directory Server SUSE Labs >>> _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r oject.org _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r oject.org _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r >>> oject.org >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server SUSE Labs >> _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs

2 2

Re: [EXTERNAL] Re: Re: Re: new server setup hanging
by thierry bordaz 05 Jun '20

05 Jun '20

Hi, Sorry to come late on this thread, my understanding is that your second server is looking like hanging. Is it consuming CPU ? does it accept new connections, new operations ? is it "hanging" because of bad response time ? The server being idle, are you sure connections are reaching the server ? Are you seeing any activity logged in access log ? regards thierry On 6/1/20 5:03 AM, Crocker, Deborah wrote: > This runs about 1/4 the load of the ones that had an issue. I don't > know if I can run this up to see it. > ------------------------------------------------------------------------ > *From:* William Brown <wbrown(a)suse.de> > *Sent:* Sunday, May 31, 2020 9:27:58 PM > *To:* 389-users(a)lists.fedoraproject.org > <389-users(a)lists.fedoraproject.org> > *Subject:* [EXTERNAL] [389-users] Re: Re: Re: new server setup hanging > > > > On 1 Jun 2020, at 11:43, Crocker, Deborah <crock(a)ua.edu> wrote: > > > > Is this sufficient? Again,Â this server has a light load and we > don't think we saw the problem, although I do note that the CPU usage > seems pretty high for such a light load. > > All threads are idle except thread 1 that is checking if there are new > connections. I don't see anything obviously wrong here ... :( > > > > > > > > Thread 26 (Thread 0x7f0600d32700 (LWP 11330)): > > #0Â 0x00007f06420a09a3 in select () > >Â Â Â at ../sysdeps/unix/syscall-template.S:81 > > #1Â 0x00007f06452e0649 in DS_Sleep () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #2Â 0x00007f063a136bf7 in deadlock_threadmain () > >Â Â Â at /usr/lib64/dirsrv/plugins/libback-ldbm.so > > #3Â 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5b00) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #4Â 0x00007f06429fdea5 in start_thread (arg=0x7f0600d32700) > >Â Â Â at pthread_create.c:307 > > #5Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 25 (Thread 0x7f0600531700 (LWP 11331)): > > #0Â 0x00007f06420a09a3 in select () > >Â Â Â at ../sysdeps/unix/syscall-template.S:81 > > #1Â 0x00007f06452e0649 in DS_Sleep () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #2Â 0x00007f063a13a7c7 in checkpoint_threadmain () > >Â Â Â at /usr/lib64/dirsrv/plugins/libback-ldbm.so > > #3Â 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f59e0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #4Â 0x00007f06429fdea5 in start_thread (arg=0x7f0600531700) > >Â Â Â at pthread_create.c:307 > > #5Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 24 (Thread 0x7f05ffd30700 (LWP 11332)): > > #0Â 0x00007f06420a09a3 in select () > >Â Â Â at ../sysdeps/unix/syscall-template.S:81 > > #1Â 0x00007f06452e0649 in DS_Sleep () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #2Â 0x00007f063a136e47 in trickle_threadmain () > >Â Â Â at /usr/lib64/dirsrv/plugins/libback-ldbm.so > > #3Â 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5c20) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #4Â 0x00007f06429fdea5 in start_thread (arg=0x7f05ffd30700) > >Â Â Â at pthread_create.c:307 > > #5Â 0x00007f06420a98dd in clone () > > ---Type <return> to continue, or q <return> to quit--- > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 23 (Thread 0x7f05ff52f700 (LWP 11333)): > > #0Â 0x00007f06420a09a3 in select () > >Â Â Â at ../sysdeps/unix/syscall-template.S:81 > > #1Â 0x00007f06452e0649 in DS_Sleep () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #2Â 0x00007f063a1319f7 in perf_threadmain () > >Â Â Â at /usr/lib64/dirsrv/plugins/libback-ldbm.so > > #3Â 0x00007f064305dc5b in _pt_root (arg=0x557b3b2f5440) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #4Â 0x00007f06429fdea5 in start_thread (arg=0x7f05ff52f700) > >Â Â Â at pthread_create.c:307 > > #5Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 22 (Thread 0x7f05fed2e700 (LWP 11334)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55e900, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x00007f06452ccf58 in slapi_wait_condvar () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #3Â 0x00007f063abe515e in cos_cache_wait_on_change () > >Â Â Â at /usr/lib64/dirsrv/plugins/libcos-plugin.so > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b680) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fed2e700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 21 (Thread 0x7f05fe52d700 (LWP 11335)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55ebc0, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x00007f06452ccf58 in slapi_wait_condvar () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #3Â 0x00007f06382041fd in roles_cache_wait_on_change () > >Â Â Â at /usr/lib64/dirsrv/plugins/libroles-plugin.so > > ---Type <return> to continue, or q <return> to quit--- > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b440) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fe52d700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 20 (Thread 0x7f05fdd2c700 (LWP 11336)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b404d3c40, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x00007f06452ccf58 in slapi_wait_condvar () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #3Â 0x00007f06382041fd in roles_cache_wait_on_change () > >Â Â Â at /usr/lib64/dirsrv/plugins/libroles-plugin.so > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b320) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fdd2c700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 19 (Thread 0x7f05fd52b700 (LWP 11337)): > > #0Â 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > >Â Â Â at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > > #1Â 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b55ec08, > ml=0x557b4055c160, timeout=timeout@entry=30000) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > > #2Â 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b55ec00, > timeout=30000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > > #3Â 0x0000557b3a0be208 in housecleaning () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b0e0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fd52b700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 18 (Thread 0x7f05fcd2a700 (LWP 11338)): > > ---Type <return> to continue, or q <return> to quit--- > > #0Â 0x00007f0642a01de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > >Â Â Â at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 > > #1Â 0x00007f0643057d07 in pt_TimedWait (cv=cv@entry=0x557b3b61fa08, > ml=0x557b3b6709a0, timeout=timeout@entry=10000) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptsynch.c:258 > > #2Â 0x00007f06430581ee in PR_WaitCondVar (cvar=0x557b3b61fa00, > timeout=10000) at ../../../nspr/pr/src/pthreads/ptsynch.c:387 > > #3Â 0x00007f064526ef23 in eq_loop () > >Â Â Â at /usr/lib64/dirsrv/libslapd.so.0 > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b200) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fcd2a700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 17 (Thread 0x7f05fc529700 (LWP 11339)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040ad80) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fc529700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 16 (Thread 0x7f05fbd28700 (LWP 11340)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b7a0) > > ---Type <return> to continue, or q <return> to quit--- > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fbd28700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 15 (Thread 0x7f05fb527700 (LWP 11341)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b8c0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fb527700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 14 (Thread 0x7f05fad26700 (LWP 11342)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040be60) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fad26700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 13 (Thread 0x7f05fa525700 (LWP 11343)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > ---Type <return> to continue, or q <return> to quit--- > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040bd40) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05fa525700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 12 (Thread 0x7f05f9d24700 (LWP 11344)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040bc20) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f9d24700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 11 (Thread 0x7f05f9523700 (LWP 11345)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040bb00) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f9523700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 10 (Thread 0x7f05f8d22700 (LWP 11346)): > > ---Type <return> to continue, or q <return> to quit--- > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040b9e0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f8d22700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 9 (Thread 0x7f05f8521700 (LWP 11347)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a120) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f8521700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 8 (Thread 0x7f05f7d20700 (LWP 11348)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a240) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f7d20700) > >Â Â Â at pthread_create.c:307 > > ---Type <return> to continue, or q <return> to quit--- > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 7 (Thread 0x7f05f751f700 (LWP 11349)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a360) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f751f700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 6 (Thread 0x7f05f6d1e700 (LWP 11350)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a5a0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f6d1e700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 5 (Thread 0x7f05f651d700 (LWP 11351)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > ---Type <return> to continue, or q <return> to quit--- > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a480) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f651d700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 4 (Thread 0x7f05f5d1c700 (LWP 11352)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a7e0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f5d1c700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 3 (Thread 0x7f05f551b700 (LWP 11353)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a900) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f551b700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 2 (Thread 0x7f05f4d1a700 (LWP 11354)): > > #0Â 0x00007f0642a01a35 in pthread_cond_wait@@GLIBC_2.3.2 () > >Â Â Â at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1Â 0x00007f0643058270 in PR_WaitCondVar (cvar=0x557b3b55fd80, > timeout=---Type <return> to continue, or q <return> to quit--- > > 4294967295) at ../../../nspr/pr/src/pthreads/ptsynch.c:385 > > #2Â 0x0000557b3a0b32fe in [IDLE THREAD] connection_wait_for_new_work () > > #3Â 0x0000557b3a0b4941 in connection_threadmain () > > #4Â 0x00007f064305dc5b in _pt_root (arg=0x557b4040a6c0) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptthread.c:201 > > #5Â 0x00007f06429fdea5 in start_thread (arg=0x7f05f4d1a700) > >Â Â Â at pthread_create.c:307 > > #6Â 0x00007f06420a98dd in clone () > >Â Â Â at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > > > > Thread 1 (Thread 0x7f064594c940 (LWP 11328)): > > #0Â 0x00007f064209ec3d in poll () > >Â Â Â at ../sysdeps/unix/syscall-template.S:81 > > #1Â 0x00007f0643059ba7 in poll (__timeout=250, __nfds=123, > __fds=0x557b404ae400) at /usr/include/bits/poll2.h:46 > > #2Â 0x00007f0643059ba7 in _pr_poll_with_poll (pds=0x557b40bb8000, > npds=123, timeout=<optimized out>) > >Â Â Â at ../../../nspr/pr/src/pthreads/ptio.c:4023 > > #3Â 0x0000557b3a0ba149 in slapd_daemon () > > #4Â 0x0000557b3a0abb15 in main () > > (gdb) > > (gdb) > > > > Deborah Crocker, PhD > > Systems Engineer III > > Office of Information Technology > > The University of Alabama > > Box 870346 > > Tuscaloosa, AL 36587 > > Office 205-348-3758 | Fax 205-348-9393 > > deborah.crocker(a)ua.edu > > > > > > -----Original Message----- > > From: William Brown <wbrown(a)suse.de> > > Sent: Sunday, May 31, 2020 8:00 PM > > To: 389-users(a)lists.fedoraproject.org > > Subject: [EXTERNAL] [389-users] Re: Re: new server setup hanging > > > > > > > >> On 1 Jun 2020, at 10:54, Crocker, Deborah <crock(a)ua.edu> wrote: > >> > >> We had to roll it back. There is one host running with it but the > load is so light we never saw a problem. We think it was a known bug, > maybe this: > >> > >> https://pagure.io/389-ds-base/issue/50329 > > > > That issue is fixed in 1.3.9, you are running 1.3.10, so that seems > unlikely? > > > >> > >> Do you want any info off the running host? > > > > As before, I'd need to see a gdb -p "pid" and `thread apply all bt` > thanks. When you gdb -p <pid>, it will tell you a command of what > debug info packages you need to install and how. You should install > those before you run the gdb commands. It will cause the server to > "pause" when you attach gdb btw. > > > >> > >> We'll now probably move into the 1.4.x trees. Any advice on which > is the most stable? > > > > We try to make sure they are all stable - if you are using Red > Hat/CentOS or SLE/Suse Leap, then whatever 389-ds version are in those > platforms is the "best maintained" for that platform, and we'll > resolve issues in them etc. I think that's 1.4.2.x or 1.4.3.x from the > top of my head at the moment for RHEL 8.x and SLE 15.x. > > > >> > >> -----Original Message----- > >> From: William Brown <wbrown(a)suse.de> > >> Sent: Sunday, May 31, 2020 6:25 PM > >> To: 389-users(a)lists.fedoraproject.org > >> Subject: [EXTERNAL] [389-users] Re: new server setup hanging > >> > >> Hey there, > >> > >> > >> We need to see the pstacks from all threads to really determine the > cause here. Can you send us a complete read out? > >> > >> gdb -p "pid" > >> > >> thread apply all bt > >> > >> > >> It'd be great if you can install debug info too to help. > >> > >> Thanks, > >> > >> > >>> On 31 May 2020, at 05:09, Crocker, Deborah <crock(a)ua.edu> wrote: > >>> > >>> Some more information from a coworker: > >>> > >>> Yeah, this sounds like an LDAP server bug. I haven't figured out > what to look at to pin it down, but when it's slow to connect, I can > see with strace that the primary thread hasn't called accept() yet for > the connection I'm trying to open. Once it does, the whole thing goes > very quickly, and I usually see a burst of other connections accepted > and handled at the same time. > >>> > >>> Deborah Crocker, PhD > >>> Systems Engineer III > >>> Office of Information Technology > >>> The University of Alabama > >>> Box 870346 > >>> Tuscaloosa, AL 36587 > >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu > >>> > >>> > >>> -----Original Message----- > >>> From: Crocker, Deborah <crock(a)ua.edu> > >>> Sent: Saturday, May 30, 2020 2:08 PM > >>> To: General discussion list for the 389 Directory server project. > >>> <389-users(a)lists.fedoraproject.org> > >>> Subject: [EXTERNAL] [389-users] new server setup hanging > >>> > >>> We're trying to move into our new server setup. We have one that > seems to be fine under a load but when we bring the next we're having > trouble with it hanging. The second does have more clients (and > different) so there could be something about what a client is doing. > Here is the server: > >>>Â Â Â Â Â 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, > running on > >>>Â Â Â Â Â CentOS Linux release 7.8.2003 > >>> > >>> And here is the pstack output listing the only thread that is not > idle. Can anyone tell me what is going on? > >>> > >>> Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): > >>> #0Â 0x00007f860a90fe02 in slapi_atomic_load_32 () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #1Â 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #2Â 0x00007f860a8d5179 in slapi_be_select () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #3Â 0x00007f860a9296a0 in vattr_test_filter () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #4Â 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #5Â 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #6Â 0x00007f8600a99e02 in acl__resource_match_aci () at > >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so > >>> #7Â 0x00007f8600a9b280 in acl_access_allowed () at > >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so > >>> #8Â 0x00007f8600aae9f7 in acl_access_allowed_main () at > >>> /usr/lib64/dirsrv/plugins/libacl-plugin.so > >>> #9Â 0x00007f860a8f0cbc in plugin_call_acl_plugin () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #10 0x00007f860a8b638d in test_filter_access () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at > >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so > >>> #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #16 0x00007f860a8e0e09 in op_shared_search () at > >>> /usr/lib64/dirsrv/libslapd.so.0 > >>> #17 0x0000557410dd3c0e in do_search () > >>> #18 0x0000557410dc198a in connection_threadmain () > >>> #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so > >>> #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 > >>> #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 > >>> > >>> Deborah Crocker, PhD > >>> Systems Engineer III > >>> Office of Information Technology > >>> The University of Alabama > >>> Box 870346 > >>> Tuscaloosa, AL 36587 > >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu > >>> > >>> > >>> -----Original Message----- > >>> From: William Brown <wbrown(a)suse.de> > >>> Sent: Wednesday, May 27, 2020 5:43 PM > >>> To: 389-users(a)lists.fedoraproject.org > >>> Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new > >>> servers into production > >>> > >>> > >>> > >>>> On 27 May 2020, at 23:20, Crocker, Deborah <crock(a)ua.edu> wrote: > >>>> > >>>> Thanks - I think we have enough ideas in here to get this going. > One last question: > >>>> If replication is set up through the host name - how often does > the directory server do a DNS look up, or does it do it once on > startup (or creation of the rep agreement)? > >>> > >>> I "think" it's every time it initiates the new connection - but > remember, for replication, that *is* quite different to a client doing > a search, so I'd be pretty careful about this. IMO you should be > standing up your replacement servers in parallel, joining them all, > moving the IP's then decomission the old servers. Alternately, you'll > need an outage window to shutdown your old servers, export the ldif, > and then import and bring up the new ones. > >>> > >>> I think having "IP's are a limited resource" really does make this > >>> whole process much much harder than it needs to be for you ... :( > >>> > >>>> > >>>> -----Original Message----- > >>>> From: William Brown <wbrown(a)suse.de> > >>>> Sent: Tuesday, May 26, 2020 10:48 PM > >>>> To: 389-users(a)lists.fedoraproject.org > >>>> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers > >>>> into production > >>>> > >>>> There are a few options. The best would be a load balancer which > has the ip's so that it's transparent to your LDAP servers where they are. > >>>> > >>>> But also as mentioned, the virtual IP's honestly is the best way. > Linux can have multiple IP's on an interface so you can just have two > IP's on one interface, andthat's the best way to do this. > >>>> > >>>> Alternately, don't rely on the IP, lower your DNS ttl's to a very > short time, change the DNS A/AAAA records, and then do it that way. > >>>> > >>>> > >>>> > >>>>> On 27 May 2020, at 06:17, Crocker, Deborah <crock(a)ua.edu> wrote: > >>>>> > >>>>> Iâ€™d like not to take up two ip addresses per host indefinitely. > We have re-IPâ€™d our hosts before so I know we can to do this but it > was during a downtime when everything was restarted. Just trying to > get away with not restarting the masters. > >>>>> > >>>>> Deborah Crocker, PhD > >>>>> Systems Engineer III > >>>>> Office of Information Technology > >>>>> The University of Alabama > >>>>> Box 870346 > >>>>> Tuscaloosa, AL 36587 > >>>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu > >>>>> > >>>>> From: Leo Pleiman <lpleiman(a)salsalabs.com> > >>>>> Sent: Tuesday, May 26, 2020 3:08 PM > >>>>> To: General discussion list for the 389 Directory server project. > >>>>> <389-users(a)lists.fedoraproject.org> > >>>>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers > >>>>> into production > >>>>> > >>>>> My experience has been that the replicas and consumers have a > unique id, more than just an IP address which creates the trust > relationship with the master. If your goal is to simply maintain an IP > so your clients don't have to be repointed, I would build each new > LDAP host and replication agreement, and then as you decommission the > old hosts use their IP address as a virtual IP address on the > replacement host. It would take a quick restart od the LDAP service to > start a listener on the virtual Ip address. > >>>>> > >>>>> > >>>>> Leo Pleiman > >>>>> Senior System Engineer > >>>>> Direct 202-787-3622 > >>>>> Cell 410-688-3873 > >>>>> > >>>>> > >>>>> > >>>>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <crock(a)ua.edu> > wrote: > >>>>> We have a setup with 2 multi-masters and 3 consumers. We are now > building new host and want to put them in place ultimately at the same > IP address as the original ones. I need some advice on how to do this > quickly and cleanly. > >>>>> > >>>>> To add a new consumer the idea now is to set it up and set up > replications agreements from each master using consumer DNS name > (don't start continuous replication yet). After initializing new > consumer from one master - turn off old consumer, remove old consumer > agreement from each master, and re-IP new consumer. Do we need to > restart masters to re-read DNS or will it pick that up when it starts > the next replication? Is this the best way to do this? > >>>>> > >>>>> Thanks > >>>>> > >>>>> Deborah Crocker, PhD > >>>>> Systems Engineer III > >>>>> Office of Information Technology > >>>>> The University of Alabama > >>>>> Box 870346 > >>>>> Tuscaloosa, AL 36587 > >>>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu > >>>>> > >>>>> _______________________________________________ > >>>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>>>> unsubscribe send an email to > >>>>> 389-users-leave(a)lists.fedoraproject.org > >>>>> Fedora Code of Conduct: > >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>>>> List Guidelines: > >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>>>> List Archives: > >>>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedor > >>>>> a p r oject.org _______________________________________________ > >>>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>>>> unsubscribe send an email to > >>>>> 389-users-leave(a)lists.fedoraproject.org > >>>>> Fedora Code of Conduct: > >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>>>> List Guidelines: > >>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>>>> List Archives: > >>>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedor > >>>>> a > >>>>> p > >>>>> r > >>>>> oject.org > >>>> > >>>> â€” > >>>> Sincerely, > >>>> > >>>> William Brown > >>>> > >>>> Senior Software Engineer, 389 Directory Server SUSE Labs > >>>> _______________________________________________ > >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >>>> Fedora Code of Conduct: > >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>>> List Guidelines: > >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>>> List Archives: > >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora > >>>> p r oject.org _______________________________________________ > >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >>>> Fedora Code of Conduct: > >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>>> List Guidelines: > >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>>> List Archives: > >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora > >>>> p > >>>> r > >>>> oject.org > >>> > >>> â€” > >>> Sincerely, > >>> > >>> William Brown > >>> > >>> Senior Software Engineer, 389 Directory Server SUSE Labs > >>> _______________________________________________ > >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >>> Fedora Code of Conduct: > >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>> List Guidelines: > >>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>> List Archives: > >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap > >>> r oject.org _______________________________________________ > >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >>> Fedora Code of Conduct: > >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>> List Guidelines: > >>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>> List Archives: > >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap > >>> r oject.org _______________________________________________ > >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >>> Fedora Code of Conduct: > >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >>> List Guidelines: > >>> https://fedoraproject.org/wiki/Mailing_list_guidelines > >>> List Archives: > >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap > >>> r > >>> oject.org > >> > >> â€” > >> Sincerely, > >> > >> William Brown > >> > >> Senior Software Engineer, 389 Directory Server SUSE Labs > >> _______________________________________________ > >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: > >> https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr > >> oject.org _______________________________________________ > >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: > >> https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr > >> oject.org > > > > â€” > > Sincerely, > > > > William Brown > > > > Senior Software Engineer, 389 Directory Server SUSE Labs > _______________________________________________ > > 389-users mailing list -- 389-users(a)lists.fedoraproject.org To > unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > <https://fedoraproject.org/wiki/Mailing_list_guidelines> > > List Archives: > https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > > _______________________________________________ > > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > <https://fedoraproject.org/wiki/Mailing_list_guidelines> > > List Archives: > https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > > â€” > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > <https://fedoraproject.org/wiki/Mailing_list_guidelines> > List Archives: > https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject…

1 0

Max Attribute Length
by Joshua Brodie 01 Jun '20

01 Jun '20

is there a way to find the max attribute name length allowed? Thank you.

2 1

Re: [EXTERNAL] Re: new server setup hanging
by William Brown 31 May '20

31 May '20

> On 1 Jun 2020, at 10:54, Crocker, Deborah <crock(a)ua.edu> wrote: > > We had to roll it back. There is one host running with it but the load is so light we never saw a problem. We think it was a known bug, maybe this: > > https://pagure.io/389-ds-base/issue/50329 That issue is fixed in 1.3.9, you are running 1.3.10, so that seems unlikely? > > Do you want any info off the running host? As before, I'd need to see a gdb -p "pid" and `thread apply all bt` thanks. When you gdb -p <pid>, it will tell you a command of what debug info packages you need to install and how. You should install those before you run the gdb commands. It will cause the server to "pause" when you attach gdb btw. > > We'll now probably move into the 1.4.x trees. Any advice on which is the most stable? We try to make sure they are all stable - if you are using Red Hat/CentOS or SLE/Suse Leap, then whatever 389-ds version are in those platforms is the "best maintained" for that platform, and we'll resolve issues in them etc. I think that's 1.4.2.x or 1.4.3.x from the top of my head at the moment for RHEL 8.x and SLE 15.x. > > -----Original Message----- > From: William Brown <wbrown(a)suse.de> > Sent: Sunday, May 31, 2020 6:25 PM > To: 389-users(a)lists.fedoraproject.org > Subject: [EXTERNAL] [389-users] Re: new server setup hanging > > Hey there, > > > We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out? > > gdb -p "pid" > > thread apply all bt > > > It'd be great if you can install debug info too to help. > > Thanks, > > >> On 31 May 2020, at 05:09, Crocker, Deborah <crock(a)ua.edu> wrote: >> >> Some more information from a coworker: >> >> Yeah, this sounds like an LDAP server bug. I haven't figured out what to look at to pin it down, but when it's slow to connect, I can see with strace that the primary thread hasn't called accept() yet for the connection I'm trying to open. Once it does, the whole thing goes very quickly, and I usually see a burst of other connections accepted and handled at the same time. >> >> Deborah Crocker, PhD >> Systems Engineer III >> Office of Information Technology >> The University of Alabama >> Box 870346 >> Tuscaloosa, AL 36587 >> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >> >> >> -----Original Message----- >> From: Crocker, Deborah <crock(a)ua.edu> >> Sent: Saturday, May 30, 2020 2:08 PM >> To: General discussion list for the 389 Directory server project. >> <389-users(a)lists.fedoraproject.org> >> Subject: [EXTERNAL] [389-users] new server setup hanging >> >> We're trying to move into our new server setup. We have one that seems to be fine under a load but when we bring the next we're having trouble with it hanging. The second does have more clients (and different) so there could be something about what a client is doing. Here is the server: >> 389-Directory/1.3.10.1 B2020.133.1625 Installed from EPEL, running on >> CentOS Linux release 7.8.2003 >> >> And here is the pstack output listing the only thread that is not idle. Can anyone tell me what is going on? >> >> Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): >> #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #2 0x00007f860a8d5179 in slapi_be_select () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #3 0x00007f860a9296a0 in vattr_test_filter () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #6 0x00007f8600a99e02 in acl__resource_match_aci () at >> /usr/lib64/dirsrv/plugins/libacl-plugin.so >> #7 0x00007f8600a9b280 in acl_access_allowed () at >> /usr/lib64/dirsrv/plugins/libacl-plugin.so >> #8 0x00007f8600aae9f7 in acl_access_allowed_main () at >> /usr/lib64/dirsrv/plugins/libacl-plugin.so >> #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #10 0x00007f860a8b638d in test_filter_access () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at >> /usr/lib64/dirsrv/plugins/libback-ldbm.so >> #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #16 0x00007f860a8e0e09 in op_shared_search () at >> /usr/lib64/dirsrv/libslapd.so.0 >> #17 0x0000557410dd3c0e in do_search () >> #18 0x0000557410dc198a in connection_threadmain () >> #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so >> #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 >> #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 >> >> Deborah Crocker, PhD >> Systems Engineer III >> Office of Information Technology >> The University of Alabama >> Box 870346 >> Tuscaloosa, AL 36587 >> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >> >> >> -----Original Message----- >> From: William Brown <wbrown(a)suse.de> >> Sent: Wednesday, May 27, 2020 5:43 PM >> To: 389-users(a)lists.fedoraproject.org >> Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new >> servers into production >> >> >> >>> On 27 May 2020, at 23:20, Crocker, Deborah <crock(a)ua.edu> wrote: >>> >>> Thanks - I think we have enough ideas in here to get this going. One last question: >>> If replication is set up through the host name - how often does the directory server do a DNS look up, or does it do it once on startup (or creation of the rep agreement)? >> >> I "think" it's every time it initiates the new connection - but remember, for replication, that *is* quite different to a client doing a search, so I'd be pretty careful about this. IMO you should be standing up your replacement servers in parallel, joining them all, moving the IP's then decomission the old servers. Alternately, you'll need an outage window to shutdown your old servers, export the ldif, and then import and bring up the new ones. >> >> I think having "IP's are a limited resource" really does make this >> whole process much much harder than it needs to be for you ... :( >> >>> >>> -----Original Message----- >>> From: William Brown <wbrown(a)suse.de> >>> Sent: Tuesday, May 26, 2020 10:48 PM >>> To: 389-users(a)lists.fedoraproject.org >>> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers >>> into production >>> >>> There are a few options. The best would be a load balancer which has the ip's so that it's transparent to your LDAP servers where they are. >>> >>> But also as mentioned, the virtual IP's honestly is the best way. Linux can have multiple IP's on an interface so you can just have two IP's on one interface, andthat's the best way to do this. >>> >>> Alternately, don't rely on the IP, lower your DNS ttl's to a very short time, change the DNS A/AAAA records, and then do it that way. >>> >>> >>> >>>> On 27 May 2020, at 06:17, Crocker, Deborah <crock(a)ua.edu> wrote: >>>> >>>> I’d like not to take up two ip addresses per host indefinitely. We have re-IP’d our hosts before so I know we can to do this but it was during a downtime when everything was restarted. Just trying to get away with not restarting the masters. >>>> >>>> Deborah Crocker, PhD >>>> Systems Engineer III >>>> Office of Information Technology >>>> The University of Alabama >>>> Box 870346 >>>> Tuscaloosa, AL 36587 >>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>>> >>>> From: Leo Pleiman <lpleiman(a)salsalabs.com> >>>> Sent: Tuesday, May 26, 2020 3:08 PM >>>> To: General discussion list for the 389 Directory server project. >>>> <389-users(a)lists.fedoraproject.org> >>>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers into >>>> production >>>> >>>> My experience has been that the replicas and consumers have a unique id, more than just an IP address which creates the trust relationship with the master. If your goal is to simply maintain an IP so your clients don't have to be repointed, I would build each new LDAP host and replication agreement, and then as you decommission the old hosts use their IP address as a virtual IP address on the replacement host. It would take a quick restart od the LDAP service to start a listener on the virtual Ip address. >>>> >>>> >>>> Leo Pleiman >>>> Senior System Engineer >>>> Direct 202-787-3622 >>>> Cell 410-688-3873 >>>> >>>> >>>> >>>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <crock(a)ua.edu> wrote: >>>> We have a setup with 2 multi-masters and 3 consumers. We are now building new host and want to put them in place ultimately at the same IP address as the original ones. I need some advice on how to do this quickly and cleanly. >>>> >>>> To add a new consumer the idea now is to set it up and set up replications agreements from each master using consumer DNS name (don't start continuous replication yet). After initializing new consumer from one master - turn off old consumer, remove old consumer agreement from each master, and re-IP new consumer. Do we need to restart masters to re-read DNS or will it pick that up when it starts the next replication? Is this the best way to do this? >>>> >>>> Thanks >>>> >>>> Deborah Crocker, PhD >>>> Systems Engineer III >>>> Office of Information Technology >>>> The University of Alabama >>>> Box 870346 >>>> Tuscaloosa, AL 36587 >>>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>>> >>>> _______________________________________________ >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora >>>> p r oject.org _______________________________________________ >>>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: >>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedora >>>> p >>>> r >>>> oject.org >>> >>> — >>> Sincerely, >>> >>> William Brown >>> >>> Senior Software Engineer, 389 Directory Server SUSE Labs >>> _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r oject.org _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r >>> oject.org >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server SUSE Labs >> _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs

2 2

Re: new server setup hanging
by William Brown 31 May '20

31 May '20

Hey there, We need to see the pstacks from all threads to really determine the cause here. Can you send us a complete read out? gdb -p "pid" thread apply all bt It'd be great if you can install debug info too to help. Thanks, > On 31 May 2020, at 05:09, Crocker, Deborah <crock(a)ua.edu> wrote: > > Some more information from a coworker: > > Yeah, this sounds like an LDAP server bug. I haven't figured out what to look at to pin it down, but when it's slow to connect, I can see with strace that the primary thread hasn't called accept() yet for the connection I'm trying to open. Once it does, the whole thing goes very quickly, and I usually see a burst of other connections accepted and handled at the same time. > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > deborah.crocker(a)ua.edu > > > -----Original Message----- > From: Crocker, Deborah <crock(a)ua.edu> > Sent: Saturday, May 30, 2020 2:08 PM > To: General discussion list for the 389 Directory server project. <389-users(a)lists.fedoraproject.org> > Subject: [EXTERNAL] [389-users] new server setup hanging > > We're trying to move into our new server setup. We have one that seems to be fine under a load but when we bring the next we're having trouble with it hanging. The second does have more clients (and different) so there could be something about what a client is doing. Here is the server: > 389-Directory/1.3.10.1 B2020.133.1625 > Installed from EPEL, running on > CentOS Linux release 7.8.2003 > > And here is the pstack output listing the only thread that is not idle. Can anyone tell me what is going on? > > Thread 44 (Thread 0x7f858e9b3700 (LWP 2515)): > #0 0x00007f860a90fe02 in slapi_atomic_load_32 () at /usr/lib64/dirsrv/libslapd.so.0 > #1 0x00007f860a8d4e8e in slapi_get_mapping_tree_node_by_dn () at /usr/lib64/dirsrv/libslapd.so.0 > #2 0x00007f860a8d5179 in slapi_be_select () at /usr/lib64/dirsrv/libslapd.so.0 > #3 0x00007f860a9296a0 in vattr_test_filter () at /usr/lib64/dirsrv/libslapd.so.0 > #4 0x00007f860a8b6ec4 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 > #5 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 > #6 0x00007f8600a99e02 in acl__resource_match_aci () at /usr/lib64/dirsrv/plugins/libacl-plugin.so > #7 0x00007f8600a9b280 in acl_access_allowed () at /usr/lib64/dirsrv/plugins/libacl-plugin.so > #8 0x00007f8600aae9f7 in acl_access_allowed_main () at /usr/lib64/dirsrv/plugins/libacl-plugin.so > #9 0x00007f860a8f0cbc in plugin_call_acl_plugin () at /usr/lib64/dirsrv/libslapd.so.0 > #10 0x00007f860a8b638d in test_filter_access () at /usr/lib64/dirsrv/libslapd.so.0 > #11 0x00007f860a8b6fb5 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 > #12 0x00007f860a8b6d31 in slapi_vattr_filter_test_ext_internal () at /usr/lib64/dirsrv/libslapd.so.0 > #13 0x00007f860a8b7ba6 in slapi_vattr_filter_test_ext () at /usr/lib64/dirsrv/libslapd.so.0 > #14 0x00007f85ff7c0df1 in ldbm_back_next_search_entry_ext () at /usr/lib64/dirsrv/plugins/libback-ldbm.so > #15 0x00007f860a8deca6 in send_results_ext.constprop.5 () at /usr/lib64/dirsrv/libslapd.so.0 > #16 0x00007f860a8e0e09 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0 > #17 0x0000557410dd3c0e in do_search () > #18 0x0000557410dc198a in connection_threadmain () > #19 0x00007f86086a0c5b in _pt_root () at /lib64/libnspr4.so > #20 0x00007f8608040ea5 in start_thread () at /lib64/libpthread.so.0 > #21 0x00007f86076ec8dd in clone () at /lib64/libc.so.6 > > Deborah Crocker, PhD > Systems Engineer III > Office of Information Technology > The University of Alabama > Box 870346 > Tuscaloosa, AL 36587 > Office 205-348-3758 | Fax 205-348-9393 > deborah.crocker(a)ua.edu > > > -----Original Message----- > From: William Brown <wbrown(a)suse.de> > Sent: Wednesday, May 27, 2020 5:43 PM > To: 389-users(a)lists.fedoraproject.org > Subject: [EXTERNAL] [389-users] Re: Re: Re: Advice to bring new servers into production > > > >> On 27 May 2020, at 23:20, Crocker, Deborah <crock(a)ua.edu> wrote: >> >> Thanks - I think we have enough ideas in here to get this going. One last question: >> If replication is set up through the host name - how often does the directory server do a DNS look up, or does it do it once on startup (or creation of the rep agreement)? > > I "think" it's every time it initiates the new connection - but remember, for replication, that *is* quite different to a client doing a search, so I'd be pretty careful about this. IMO you should be standing up your replacement servers in parallel, joining them all, moving the IP's then decomission the old servers. Alternately, you'll need an outage window to shutdown your old servers, export the ldif, and then import and bring up the new ones. > > I think having "IP's are a limited resource" really does make this whole process much much harder than it needs to be for you ... :( > >> >> -----Original Message----- >> From: William Brown <wbrown(a)suse.de> >> Sent: Tuesday, May 26, 2020 10:48 PM >> To: 389-users(a)lists.fedoraproject.org >> Subject: [EXTERNAL] [389-users] Re: Re: Advice to bring new servers >> into production >> >> There are a few options. The best would be a load balancer which has the ip's so that it's transparent to your LDAP servers where they are. >> >> But also as mentioned, the virtual IP's honestly is the best way. Linux can have multiple IP's on an interface so you can just have two IP's on one interface, andthat's the best way to do this. >> >> Alternately, don't rely on the IP, lower your DNS ttl's to a very short time, change the DNS A/AAAA records, and then do it that way. >> >> >> >>> On 27 May 2020, at 06:17, Crocker, Deborah <crock(a)ua.edu> wrote: >>> >>> I’d like not to take up two ip addresses per host indefinitely. We have re-IP’d our hosts before so I know we can to do this but it was during a downtime when everything was restarted. Just trying to get away with not restarting the masters. >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>> >>> From: Leo Pleiman <lpleiman(a)salsalabs.com> >>> Sent: Tuesday, May 26, 2020 3:08 PM >>> To: General discussion list for the 389 Directory server project. >>> <389-users(a)lists.fedoraproject.org> >>> Subject: [EXTERNAL] [389-users] Re: Advice to bring new servers into >>> production >>> >>> My experience has been that the replicas and consumers have a unique id, more than just an IP address which creates the trust relationship with the master. If your goal is to simply maintain an IP so your clients don't have to be repointed, I would build each new LDAP host and replication agreement, and then as you decommission the old hosts use their IP address as a virtual IP address on the replacement host. It would take a quick restart od the LDAP service to start a listener on the virtual Ip address. >>> >>> >>> Leo Pleiman >>> Senior System Engineer >>> Direct 202-787-3622 >>> Cell 410-688-3873 >>> >>> >>> >>> On Tue, May 26, 2020 at 3:57 PM Crocker, Deborah <crock(a)ua.edu> wrote: >>> We have a setup with 2 multi-masters and 3 consumers. We are now building new host and want to put them in place ultimately at the same IP address as the original ones. I need some advice on how to do this quickly and cleanly. >>> >>> To add a new consumer the idea now is to set it up and set up replications agreements from each master using consumer DNS name (don't start continuous replication yet). After initializing new consumer from one master - turn off old consumer, remove old consumer agreement from each master, and re-IP new consumer. Do we need to restart masters to re-read DNS or will it pick that up when it starts the next replication? Is this the best way to do this? >>> >>> Thanks >>> >>> Deborah Crocker, PhD >>> Systems Engineer III >>> Office of Information Technology >>> The University of Alabama >>> Box 870346 >>> Tuscaloosa, AL 36587 >>> Office 205-348-3758 | Fax 205-348-9393 deborah.crocker(a)ua.edu >>> >>> _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r oject.org _______________________________________________ >>> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >>> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorap >>> r >>> oject.org >> >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server SUSE Labs >> _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org _______________________________________________ >> 389-users mailing list -- 389-users(a)lists.fedoraproject.org To >> unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-users@lists.fedorapr >> oject.org > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… > _______________________________________________ > 389-users mailing list -- 389-users(a)lists.fedoraproject.org > To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject… — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

389-users June 2020