Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)

[SOLER SANGUESA Miguel]

hello,

I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5). An hour later
I tried to do the same with the unique replica I have, but after update dirsrv is not
starting.
It says it is needed run "ipa-server-upgrade", but it also fails:
# ipactl start
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/8]: saving configuration
  [2/8]: disabling listeners
  [3/8]: enabling DS global lock
  [4/8]: starting directory server
  [error] CalledProcessError: Command '/bin/systemctl start
dirsrv(a)IPA-EXAMOLE-ORG.service' returned non-zero exit status 1
  [cleanup]: stopping directory server
  [cleanup]: restoring configuration
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service'
returned non-zero exit status 1
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

On the log I can see:
2018-04-30T14:36:15Z DEBUG Starting external process
2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active dirsrv(a)IPA-EXAMPLE-ORG.service
2018-04-30T14:36:15Z DEBUG Process finished, return code=3
2018-04-30T14:36:15Z DEBUG stdout=failed
...
2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
2018-04-30T14:36:15Z DEBUG Starting external process
2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
2018-04-30T14:36:15Z DEBUG Process finished, return code=1
2018-04-30T14:36:15Z DEBUG stdout=
2018-04-30T14:36:15Z DEBUG stderr=Job for dirsrv(a)IPA-EXAMPLE-ORG.service failed because
the control process exited with error code. See "systemctl status
dirsrv(a)IPA-EXAMPLE-ORG.service" and "journalctl -xe" for details.

2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py",
line 95, in __start
    srv.start(self.serverid, ldapi=True)
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
161, in start
    instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
294, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 542, in
run
    raise CalledProcessError(p.returncode, arg_string, str(output))
CalledProcessError: Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service'
returned non-zero exit status 1

2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command '/bin/systemctl start
dirsrv(a)IPA-EXAMPLE-ORG.service' returned non-zero exit status 1

Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
[30/Apr/2018:16:04:52.584220922 +0200] - ERR - slapd_bootstrap_config - The default
password storage scheme could not be read or was not found in the file
/etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.

Checking on internet I show that "dse.ldif" could be corrupted, so I changed
with "dse.ldif.startOK" without any change and then I changed with
"dse.ldif.bak". The problem persist but the error has changed:
[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port - Non-Secure Port
Disabled
[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error - Netscape Portable
Runtime error -5975: /usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
replication_legacy_plugin_init
[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error - Could not load
symbol "replication_legacy_plugin_init" from
"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy
Replication Plugin
[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - Unable to load plugin
"cn=Legacy Replication Plugin,cn=plugins,cn=config"

I saw a bug about this problem, but it is still opened:
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442

Any idea how to fix the issue?

If it is not possible to fix it, can I remove the replica from IPA and install it again
with the same name?

Thanks & Regards.
______________________________

[Alexander Bokovoy]

On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> hello,
>
> I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5). An hour later
> I tried to do the same with the unique replica I have, but after update dirsrv is not
> starting.
> It says it is needed run "ipa-server-upgrade", but it also fails:
> # ipactl start
> Upgrade required: please run ipa-server-upgrade command
> Aborting ipactl
>
> # ipa-server-upgrade
> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>   [1/8]: saving configuration
>   [2/8]: disabling listeners
>   [3/8]: enabling DS global lock
>   [4/8]: starting directory server
>   [error] CalledProcessError: Command '/bin/systemctl start
> dirsrv(a)IPA-EXAMOLE-ORG.service' returned non-zero exit status 1
>   [cleanup]: stopping directory server
>   [cleanup]: restoring configuration
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
> ipa-server-upgrade manually.
> Unexpected error - see /var/log/ipaupgrade.log for details:
> CalledProcessError: Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service'
> returned non-zero exit status 1
> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
>
> On the log I can see:
> 2018-04-30T14:36:15Z DEBUG Starting external process
> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active dirsrv(a)IPA-EXAMPLE-ORG.service
> 2018-04-30T14:36:15Z DEBUG Process finished, return code=3
> 2018-04-30T14:36:15Z DEBUG stdout=failed
> ...
> 2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> 2018-04-30T14:36:15Z DEBUG Starting external process
> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> 2018-04-30T14:36:15Z DEBUG Process finished, return code=1
> 2018-04-30T14:36:15Z DEBUG stdout=
> 2018-04-30T14:36:15Z DEBUG stderr=Job for dirsrv(a)IPA-EXAMPLE-ORG.service failed because
> the control process exited with error code. See "systemctl status
> dirsrv(a)IPA-EXAMPLE-ORG.service" and "journalctl -xe" for details.
>
> 2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> 504, in start_creation
>     run_step(full_msg, method)
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> 494, in run_step
>     method()
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py",
> line 95, in __start
>     srv.start(self.serverid, ldapi=True)
>   File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
> 161, in start
>     instance_name, capture_output=capture_output, wait=wait)
>   File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> 294, in start
>     skip_output=not capture_output)
>   File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 542, in
> run
>     raise CalledProcessError(p.returncode, arg_string, str(output))
> CalledProcessError: Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service'
> returned non-zero exit status 1
>
> 2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command '/bin/systemctl start
> dirsrv(a)IPA-EXAMPLE-ORG.service' returned non-zero exit status 1
>
> Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> [30/Apr/2018:16:04:52.584220922 +0200] - ERR - slapd_bootstrap_config - The default
> password storage scheme could not be read or was not found in the file
> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>
> Checking on internet I show that "dse.ldif" could be corrupted, so I changed
> with "dse.ldif.startOK" without any change and then I changed with
> "dse.ldif.bak". The problem persist but the error has changed:
> [30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port - Non-Secure Port
> Disabled
> [30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error - Netscape Portable
> Runtime error -5975: /usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
> replication_legacy_plugin_init
> [30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error - Could not load
> symbol "replication_legacy_plugin_init" from
> "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy
> Replication Plugin
> [30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - Unable to load plugin
> "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>
> I saw a bug about this problem, but it is still opened:
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>
> Any idea how to fix the issue?
>
> If it is not possible to fix it, can I remove the replica from IPA and install it again
> with the same name? 
A quick fix could be to remove an entry for cn=Legacy
Replication
Plugin,cn=plugins,cn=config from /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
when dirsrv is down.

-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[SOLER SANGUESA Miguel]

Hello,



Thank you for your answer, now dirsrv can start, but after running the "
dirsrv(a)IPA-UNICC-ORG.service " it hungs:

# ipa-server-upgrade

Upgrading IPA:. Estimated time: 1 minute 30 seconds

  [1/8]: saving configuration

  [2/8]: disabling listeners

  [3/8]: enabling DS global lock

  [4/8]: starting directory server

  [5/8]: updating schema

  [6/8]: upgrading server



The latest messages on the log is (there are no logs on the lasts 2h):

2018-05-01T08:13:14Z DEBUG ---------------------------------------------

2018-05-01T08:13:14Z DEBUG Final value after applying updates

2018-05-01T08:13:14Z DEBUG dn: cn=description,cn=index,cn=userroot,cn=ldbm
database,cn=plugins,cn=config

2018-05-01T08:13:14Z DEBUG objectclass:

2018-05-01T08:13:14Z DEBUG      top

2018-05-01T08:13:14Z DEBUG      nsindex

2018-05-01T08:13:14Z DEBUG nsindextype:

2018-05-01T08:13:14Z DEBUG      eq

2018-05-01T08:13:14Z DEBUG      sub

2018-05-01T08:13:14Z DEBUG cn:

2018-05-01T08:13:14Z DEBUG      description

2018-05-01T08:13:14Z DEBUG nssystemindex:

2018-05-01T08:13:14Z DEBUG      false

2018-05-01T08:13:19Z DEBUG Creating task to index attribute: description

2018-05-01T08:13:19Z DEBUG Task id:
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config

If I check the ipa services, dirsrv is the only one running:

# ipactl status

Directory Service: RUNNING

krb5kdc Service: STOPPED

kadmin Service: STOPPED

named Service: STOPPED

httpd Service: STOPPED

ipa-custodia Service: STOPPED

ntpd Service: RUNNING

pki-tomcatd Service: STOPPED

smb Service: STOPPED

winbind Service: STOPPED

ipa-otpd Service: STOPPED

ipa-dnskeysyncd Service: STOPPED

ipa: INFO: The ipactl command was successful



Should I stop the ipa-server-upgrade and start ipa services?



Thanks.



-----Original Message-----
From: Alexander Bokovoy <abokovoy(a)redhat.com
>  
Sent:
Tuesday, May 01, 2018 9:56
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org
>  
Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org
>  
Subject: Re: [Freeipa-users] Problem on dirsrv when updating
from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)



On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:

> hello, 
>  
> I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL
> 7.5). An hour later I tried to do the same with the unique replica I have, but after
> update dirsrv is not starting. 
> It says it is needed run "ipa-server-upgrade", but it also
> fails: 
> # ipactl start 
> Upgrade required: please run ipa-server-upgrade command Aborting
> ipactl 
>  
> # ipa-server-upgrade 
> Upgrading IPA:. Estimated time: 1 minute 30 seconds 
>   [1/8]: saving configuration 
>   [2/8]: disabling listeners 
>   [3/8]: enabling DS global lock 
>   [4/8]: starting directory server 
>   [error] CalledProcessError: Command '/bin/systemctl start
> [e-mail]<mailto:[e-mail]>'
> returned non-zero exit status 1 
>   [cleanup]: stopping directory server 
>   [cleanup]: restoring configuration 
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> command ipa-server-upgrade manually. 
> Unexpected error - see /var/log/ipaupgrade.log for details: 
> CalledProcessError: Command '/bin/systemctl start 
> [e-mail]<mailto:[e-mail]>'
> returned non-zero exit status 1 The 
> ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> more 
> information 
>  
> On the log I can see: 
> 2018-04-30T14:36:15Z DEBUG Starting external process 
> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active 
> [e-mail]<mailto:[e-mail]>
> 2018-04-30T14:36:15Z DEBUG Process 
> finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> 2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server 
> 2018-04-30T14:36:15Z DEBUG Starting external process 
> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start 
> [e-mail]<mailto:[e-mail]>
> 2018-04-30T14:36:15Z DEBUG Process 
> finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout= 
> 2018-04-30T14:36:15Z DEBUG stderr=Job for
> [e-mail]<mailto:[e-mail]> failed because
> the control process exited with error code. See "systemctl status
> [e-mail]<mailto:[e-mail]>" and
> "journalctl -xe" for details. 
>  
> 2018-04-30T14:36:15Z DEBUG Traceback (most recent call last): 
>   File
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in
> start_creation 
>     run_step(full_msg, method) 
>   File
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in
> run_step 
>     method() 
>   File
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line
> 95, in __start 
>     srv.start(self.serverid, ldapi=True) 
>   File
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 161, in
> start 
>     instance_name, capture_output=capture_output, wait=wait) 
>   File
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in
> start 
>     skip_output=not capture_output) 
>   File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 542, in run
>     raise CalledProcessError(p.returncode, arg_string, str(output))
> CalledProcessError: Command '/bin/systemctl start 
> [e-mail]<mailto:[e-mail]>'
> returned non-zero exit status 1 
>  
> 2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> '/bin/systemctl start
> [e-mail]<mailto:[e-mail]>' returned
> non-zero exit status 1 
>  
> Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show: 
> [30/Apr/2018:16:04:52.584220922 +0200] - ERR - slapd_bootstrap_config
> - The default password storage scheme could not be read or was not found in the file
> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory. 
>  
> Checking on internet I show that "dse.ldif" could be
> corrupted, so I changed with "dse.ldif.startOK" without any change and then I
> changed with "dse.ldif.bak". The problem persist but the error has changed:
> [30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
> Non-Secure Port Disabled 
> [30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error -
> Netscape Portable Runtime error -5975: 
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
> replication_legacy_plugin_init 
> [30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error -
> Could not load symbol "replication_legacy_plugin_init" from
> "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> plugin Legacy 
> Replication Plugin 
> [30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry -
> Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>  
> I saw a bug about this problem, but it is still opened: 
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>  
> Any idea how to fix the issue? 
>  
> If it is not possible to fix it, can I remove the replica from IPA and
> install it again with the same name? 
A quick fix could be to remove an entry for cn=Legacy Replication
Plugin,cn=plugins,cn=config from /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif

when dirsrv is down.



--

/ Alexander Bokovoy

Sr. Principal Software Engineer

Security / Identity Management Engineering Red Hat Limited, Finland

[Rob Crittenden]

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  Thank you for your answer, now dirsrv can start, but after running the " 
>  dirsrv(a)IPA-UNICC-ORG.service " it hungs:
>  
>  # ipa-server-upgrade
>  
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>     [1/8]: saving configuration
>  
>     [2/8]: disabling listeners
>  
>     [3/8]: enabling DS global lock
>  
>     [4/8]: starting directory server
>  
>     [5/8]: updating schema
>  
>     [6/8]: upgrading server
>  
>  The latest messages on the log is (there are no logs on the lasts 2h):
>  
>  2018-05-01T08:13:14Z DEBUG ---------------------------------------------
>  
>  2018-05-01T08:13:14Z DEBUG Final value after applying updates
>  
>  2018-05-01T08:13:14Z DEBUG dn: 
>  cn=description,cn=index,cn=userroot,cn=ldbm database,cn=plugins,cn=config
>  
>  2018-05-01T08:13:14Z DEBUG objectclass:
>  
>  2018-05-01T08:13:14Z DEBUG      top
>  
>  2018-05-01T08:13:14Z DEBUG      nsindex
>  
>  2018-05-01T08:13:14Z DEBUG nsindextype:
>  
>  2018-05-01T08:13:14Z DEBUG      eq
>  
>  2018-05-01T08:13:14Z DEBUG      sub
>  
>  2018-05-01T08:13:14Z DEBUG cn:
>  
>  2018-05-01T08:13:14Z DEBUG      description
>  
>  2018-05-01T08:13:14Z DEBUG nssystemindex:
>  
>  2018-05-01T08:13:14Z DEBUG      false
>  
>  2018-05-01T08:13:19Z DEBUG Creating task to index attribute: description
>  
>  2018-05-01T08:13:19Z DEBUG Task id: 
>  cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config
>  
>  If I check the ipa services, dirsrv is the only one running:
>  
>  # ipactl status
>  
>  Directory Service: RUNNING
>  
>  krb5kdc Service: STOPPED
>  
>  kadmin Service: STOPPED
>  
>  named Service: STOPPED
>  
>  httpd Service: STOPPED
>  
>  ipa-custodia Service: STOPPED
>  
>  ntpd Service: RUNNING
>  
>  pki-tomcatd Service: STOPPED
>  
>  smb Service: STOPPED
>  
>  winbind Service: STOPPED
>  
>  ipa-otpd Service: STOPPED
>  
>  ipa-dnskeysyncd Service: STOPPED
>  
>  ipa: INFO: The ipactl command was successful
>  
>  Should I stop the ipa-server-upgrade and start ipa services? 
It depends on how many entries you have. This is an indexing task and 
while 2 hours seems a bit excessive, it depends on how much work it has 
to do.

The 389-ds access and/or error logs may provide details.

rob

>  
>  Thanks.
>  
>  -----Original Message-----
>  From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
>  Sent: Tuesday, May 01, 2018 9:56
>  To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>  Subject: Re: [Freeipa-users] Problem on dirsrv when updating from 4.5.0 
>  (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>  
>  On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  
>   >hello,
>  
>   >
>  
>   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 
>  7.5). An hour later I tried to do the same with the unique replica I 
>  have, but after update dirsrv is not starting.
>  
>   >It says it is needed run "ipa-server-upgrade", but it also fails:
>  
>   ># ipactl start
>  
>   >Upgrade required: please run ipa-server-upgrade command Aborting ipactl
>  
>   >
>  
>   ># ipa-server-upgrade
>  
>   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>   >  [1/8]: saving configuration
>  
>   >  [2/8]: disabling listeners
>  
>   >  [3/8]: enabling DS global lock
>  
>   >  [4/8]: starting directory server
>  
>   >  [error] CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 1
>  
>   >  [cleanup]: stopping directory server
>  
>   >  [cleanup]: restoring configuration
>  
>   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run 
>  command ipa-server-upgrade manually.
>  
>   >Unexpected error - see /var/log/ipaupgrade.log for details:
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 1 
>  The
>  
>   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more
>  
>   >information
>  
>   >
>  
>   >On the log I can see:
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]> 
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
>  
>   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]> 
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
>  
>   >2018-04-30T14:36:15Z DEBUG stderr=Job for 
>  dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]> 
>  failed because the control process exited with error code. See 
>  "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>" and "journalctl -xe" for
> details.
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>  
>   >  File 
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  504, in start_creation
>  
>   >    run_step(full_msg, method)
>  
>   >  File 
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  494, in run_step
>  
>   >    method()
>  
>   >  File 
>  "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", 
>  line 95, in __start
>  
>   >    srv.start(self.serverid, ldapi=True)
>  
>   >  File 
>  "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 
>  161, in start
>  
>   >    instance_name, capture_output=capture_output, wait=wait)
>  
>   >  File 
>  "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 
>  294, in start
>  
>   >    skip_output=not capture_output)
>  
>   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 
>  542, in run
>  
>   >    raise CalledProcessError(p.returncode, arg_string, str(output))
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 1
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command 
>  '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 1
>  
>   >
>  
>   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
>  
>   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - slapd_bootstrap_config 
>  - The default password storage scheme could not be read or was not found 
>  in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>  
>   >
>  
>   >Checking on internet I show that "dse.ldif" could be corrupted, so I 
>  changed with "dse.ldif.startOK" without any change and then I changed 
>  with "dse.ldif.bak". The problem persist but the error has changed:
>  
>   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
>  
>   >Non-Secure Port Disabled
>  
>   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error -
>  
>   >Netscape Portable Runtime error -5975:
>  
>   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
>  
>   >replication_legacy_plugin_init
>  
>   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error -
>  
>   >Could not load symbol "replication_legacy_plugin_init" from
>  
>   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy
>  
>   >Replication Plugin
>  
>   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - 
>  Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>  
>   >
>  
>   >I saw a bug about this problem, but it is still opened:
>  
>   >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>  
>   >
>  
>   >Any idea how to fix the issue?
>  
>   >
>  
>   >If it is not possible to fix it, can I remove the replica from IPA and 
>  install it again with the same name?
>  
>  A quick fix could be to remove an entry for cn=Legacy Replication 
>  Plugin,cn=plugins,cn=config from /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
>  
>  when dirsrv is down.
>  
>  --
>  
>  / Alexander Bokovoy
>  
>  Sr. Principal Software Engineer
>  
>  Security / Identity Management Engineering Red Hat Limited, Finland
>  
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>

[SOLER SANGUESA Miguel]

Hello,

on the errors log I show this error every 5 min:
[01/May/2018:15:59:25.956271320 +0200] - ERR - set_krb5_creds - Could not get initial
credentials for principal [ldap/ipareplica.ipa.example.org(a)IPA.EXAMPLE.ORG] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)

I have started the krb5kdc Service and now it doesn't complain.

On the access log I have thousands of lines with (op value is increasing by 1 every log):
[01/May/2018:16:49:11.011904150 +0200] conn=5 op=23845 SRCH
base="cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config"
scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
[01/May/2018:16:49:11.012135091 +0200] conn=5 op=23845 RESULT err=0 tag=101 nentries=1
etime=1.0001333670

rigth now I have the folloging services up:
# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: STOPPED
named Service: STOPPED
httpd Service: STOPPED
ipa-custodia Service: STOPPED
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
ipa: INFO: The ipactl command was successful

I haven't tried to start all services because I don't know if it would affect the
upgrade.

The /var/log/ipaupgrade.log doen't have any new log for the last 7 hours, the last one
is what I posted on the previous email.
Should I still wait, or should I start all services?

Thanks & Regards.
______________________________

-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com&gt; 
Sent: Tuesday, May 01, 2018 15:18
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  Thank you for your answer, now dirsrv can start, but after running the " 
>  dirsrv(a)IPA-UNICC-ORG.service " it hungs:
>  
>  # ipa-server-upgrade
>  
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>     [1/8]: saving configuration
>  
>     [2/8]: disabling listeners
>  
>     [3/8]: enabling DS global lock
>  
>     [4/8]: starting directory server
>  
>     [5/8]: updating schema
>  
>     [6/8]: upgrading server
>  
>  The latest messages on the log is (there are no logs on the lasts 2h):
>  
>  2018-05-01T08:13:14Z DEBUG 
>  ---------------------------------------------
>  
>  2018-05-01T08:13:14Z DEBUG Final value after applying updates
>  
>  2018-05-01T08:13:14Z DEBUG dn: 
>  cn=description,cn=index,cn=userroot,cn=ldbm 
>  database,cn=plugins,cn=config
>  
>  2018-05-01T08:13:14Z DEBUG objectclass:
>  
>  2018-05-01T08:13:14Z DEBUG      top
>  
>  2018-05-01T08:13:14Z DEBUG      nsindex
>  
>  2018-05-01T08:13:14Z DEBUG nsindextype:
>  
>  2018-05-01T08:13:14Z DEBUG      eq
>  
>  2018-05-01T08:13:14Z DEBUG      sub
>  
>  2018-05-01T08:13:14Z DEBUG cn:
>  
>  2018-05-01T08:13:14Z DEBUG      description
>  
>  2018-05-01T08:13:14Z DEBUG nssystemindex:
>  
>  2018-05-01T08:13:14Z DEBUG      false
>  
>  2018-05-01T08:13:19Z DEBUG Creating task to index attribute: 
>  description
>  
>  2018-05-01T08:13:19Z DEBUG Task id: 
>  cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
>  config
>  
>  If I check the ipa services, dirsrv is the only one running:
>  
>  # ipactl status
>  
>  Directory Service: RUNNING
>  
>  krb5kdc Service: STOPPED
>  
>  kadmin Service: STOPPED
>  
>  named Service: STOPPED
>  
>  httpd Service: STOPPED
>  
>  ipa-custodia Service: STOPPED
>  
>  ntpd Service: RUNNING
>  
>  pki-tomcatd Service: STOPPED
>  
>  smb Service: STOPPED
>  
>  winbind Service: STOPPED
>  
>  ipa-otpd Service: STOPPED
>  
>  ipa-dnskeysyncd Service: STOPPED
>  
>  ipa: INFO: The ipactl command was successful
>  
>  Should I stop the ipa-server-upgrade and start ipa services? 
It depends on how many entries you have. This is an indexing task and while 2 hours seems
a bit excessive, it depends on how much work it has to do.

The 389-ds access and/or error logs may provide details.

rob

>  
>  Thanks.
>  
>  -----Original Message-----
>  From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
>  Sent: Tuesday, May 01, 2018 9:56
>  To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>  Subject: Re: [Freeipa-users] Problem on dirsrv when updating from 
>  4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>  
>  On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  
>   >hello,
>  
>   >
>  
>   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 
>  (RHEL 7.5). An hour later I tried to do the same with the unique 
>  replica I have, but after update dirsrv is not starting.
>  
>   >It says it is needed run "ipa-server-upgrade", but it also fails:
>  
>   ># ipactl start
>  
>   >Upgrade required: please run ipa-server-upgrade command Aborting 
>  ipactl
>  
>   >
>  
>   ># ipa-server-upgrade
>  
>   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>   >  [1/8]: saving configuration
>  
>   >  [2/8]: disabling listeners
>  
>   >  [3/8]: enabling DS global lock
>  
>   >  [4/8]: starting directory server
>  
>   >  [error] CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >  [cleanup]: stopping directory server
>  
>   >  [cleanup]: restoring configuration
>  
>   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run 
>  command ipa-server-upgrade manually.
>  
>   >Unexpected error - see /var/log/ipaupgrade.log for details:
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1 The
>  
>   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for 
>  more
>  
>   >information
>  
>   >
>  
>   >On the log I can see:
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
>  
>   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
>  
>   >2018-04-30T14:36:15Z DEBUG stderr=Job for 
>  dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
>  failed because the control process exited with error code. See 
>  "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>" and "journalctl -xe" for
> details.
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  504, in start_creation
>  
>   >    run_step(full_msg, method)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  494, in run_step
>  
>   >    method()
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
>  ",
>  line 95, in __start
>  
>   >    srv.start(self.serverid, ldapi=True)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", 
>  line 161, in start
>  
>   >    instance_name, capture_output=capture_output, wait=wait)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 
>  294, in start
>  
>   >    skip_output=not capture_output)
>  
>   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 
>  542, in run
>  
>   >    raise CalledProcessError(p.returncode, arg_string, str(output))
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command 
>  '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >
>  
>   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
>  
>   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
>  slapd_bootstrap_config
>  - The default password storage scheme could not be read or was not 
>  found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>  
>   >
>  
>   >Checking on internet I show that "dse.ldif" could be corrupted, so I 
>  changed with "dse.ldif.startOK" without any change and then I changed 
>  with "dse.ldif.bak". The problem persist but the error has changed:
>  
>   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
>  
>   >Non-Secure Port Disabled
>  
>   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error 
>  -
>  
>   >Netscape Portable Runtime error -5975:
>  
>   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
>  
>   >replication_legacy_plugin_init
>  
>   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error 
>  -
>  
>   >Could not load symbol "replication_legacy_plugin_init" from
>  
>   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin 
>  Legacy
>  
>   >Replication Plugin
>  
>   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - 
>  Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>  
>   >
>  
>   >I saw a bug about this problem, but it is still opened:
>  
>   >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>  
>   >
>  
>   >Any idea how to fix the issue?
>  
>   >
>  
>   >If it is not possible to fix it, can I remove the replica from IPA 
>  and install it again with the same name?
>  
>  A quick fix could be to remove an entry for cn=Legacy Replication 
>  Plugin,cn=plugins,cn=config from 
>  /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
>  
>  when dirsrv is down.
>  
>  --
>  
>  / Alexander Bokovoy
>  
>  Sr. Principal Software Engineer
>  
>  Security / Identity Management Engineering Red Hat Limited, Finland
>  
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to 
>  freeipa-users-leave(a)lists.fedorahosted.org
>

[Rob Crittenden]

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  on the errors log I show this error every 5 min:
>  [01/May/2018:15:59:25.956271320 +0200] - ERR - set_krb5_creds - Could not get initial
> credentials for principal [ldap/ipareplica.ipa.example.org(a)IPA.EXAMPLE.ORG] in keytab
> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for requested realm)
>  
>  I have started the krb5kdc Service and now it doesn't complain. 
Either way it's fine and unrelated to your issue.

>  On the access log I have thousands of lines with (op value is
> increasing by 1 every log):
>  [01/May/2018:16:49:11.011904150 +0200] conn=5 op=23845 SRCH
> base="cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config"
> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
>  [01/May/2018:16:49:11.012135091 +0200] conn=5 op=23845 RESULT err=0 tag=101 nentries=1
> etime=1.0001333670 
The upgrader is not detecting that the indexing is finished. Can you see 
what the status is:

# ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA.EXAMPLE.ORG.socket -b 
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config 
-s base

rob


>  rigth now I have the folloging services up:
>  # ipactl status
>  Directory Service: RUNNING
>  krb5kdc Service: RUNNING
>  kadmin Service: STOPPED
>  named Service: STOPPED
>  httpd Service: STOPPED
>  ipa-custodia Service: STOPPED
>  ntpd Service: RUNNING
>  pki-tomcatd Service: STOPPED
>  smb Service: STOPPED
>  winbind Service: STOPPED
>  ipa-otpd Service: STOPPED
>  ipa-dnskeysyncd Service: STOPPED
>  ipa: INFO: The ipactl command was successful
>  
>  I haven't tried to start all services because I don't know if it would affect the
> upgrade.
>  
>  The /var/log/ipaupgrade.log doen't have any new log for the last 7 hours, the last
> one is what I posted on the previous email.
>  Should I still wait, or should I start all services?
>  
>  Thanks & Regards.
>  ______________________________
>  
>  -----Original Message-----
>  From: Rob Crittenden <rcritten(a)redhat.com&gt;
>  Sent: Tuesday, May 01, 2018 15:18
>  To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>  
>  SOLER SANGUESA Miguel via FreeIPA-users wrote:
> > Hello,
> >
> > Thank you for your answer, now dirsrv can start, but after running the "
> > dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >
> > # ipa-server-upgrade
> >
> > Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >
> >     [1/8]: saving configuration
> >
> >     [2/8]: disabling listeners
> >
> >     [3/8]: enabling DS global lock
> >
> >     [4/8]: starting directory server
> >
> >     [5/8]: updating schema
> >
> >     [6/8]: upgrading server
> >
> > The latest messages on the log is (there are no logs on the lasts 2h):
> >
> > 2018-05-01T08:13:14Z DEBUG
> > ---------------------------------------------
> >
> > 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >
> > 2018-05-01T08:13:14Z DEBUG dn:
> > cn=description,cn=index,cn=userroot,cn=ldbm
> > database,cn=plugins,cn=config
> >
> > 2018-05-01T08:13:14Z DEBUG objectclass:
> >
> > 2018-05-01T08:13:14Z DEBUG      top
> >
> > 2018-05-01T08:13:14Z DEBUG      nsindex
> >
> > 2018-05-01T08:13:14Z DEBUG nsindextype:
> >
> > 2018-05-01T08:13:14Z DEBUG      eq
> >
> > 2018-05-01T08:13:14Z DEBUG      sub
> >
> > 2018-05-01T08:13:14Z DEBUG cn:
> >
> > 2018-05-01T08:13:14Z DEBUG      description
> >
> > 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >
> > 2018-05-01T08:13:14Z DEBUG      false
> >
> > 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> > description
> >
> > 2018-05-01T08:13:19Z DEBUG Task id:
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
> > config
> >
> > If I check the ipa services, dirsrv is the only one running:
> >
> > # ipactl status
> >
> > Directory Service: RUNNING
> >
> > krb5kdc Service: STOPPED
> >
> > kadmin Service: STOPPED
> >
> > named Service: STOPPED
> >
> > httpd Service: STOPPED
> >
> > ipa-custodia Service: STOPPED
> >
> > ntpd Service: RUNNING
> >
> > pki-tomcatd Service: STOPPED
> >
> > smb Service: STOPPED
> >
> > winbind Service: STOPPED
> >
> > ipa-otpd Service: STOPPED
> >
> > ipa-dnskeysyncd Service: STOPPED
> >
> > ipa: INFO: The ipactl command was successful
> >
> > Should I stop the ipa-server-upgrade and start ipa services?
>  
>  It depends on how many entries you have. This is an indexing task and while 2 hours seems
> a bit excessive, it depends on how much work it has to do.
>  
>  The 389-ds access and/or error logs may provide details.
>  
>  rob
>  
> >
> > Thanks.
> >
> > -----Original Message-----
> > From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> > Sent: Tuesday, May 01, 2018 9:56
> > To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> > Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >
> >   >hello,
> >
> >   >
> >
> >   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4
> > (RHEL 7.5). An hour later I tried to do the same with the unique
> > replica I have, but after update dirsrv is not starting.
> >
> >   >It says it is needed run "ipa-server-upgrade", but it also fails:
> >
> >   ># ipactl start
> >
> >   >Upgrade required: please run ipa-server-upgrade command Aborting
> > ipactl
> >
> >   >
> >
> >   ># ipa-server-upgrade
> >
> >   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >
> >   >  [1/8]: saving configuration
> >
> >   >  [2/8]: disabling listeners
> >
> >   >  [3/8]: enabling DS global lock
> >
> >   >  [4/8]: starting directory server
> >
> >   >  [error] CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >  [cleanup]: stopping directory server
> >
> >   >  [cleanup]: restoring configuration
> >
> >   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> > command ipa-server-upgrade manually.
> >
> >   >Unexpected error - see /var/log/ipaupgrade.log for details:
> >
> >   >CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1 The
> >
> >   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> > more
> >
> >   >information
> >
> >   >
> >
> >   >On the log I can see:
> >
> >   >2018-04-30T14:36:15Z DEBUG Starting external process
> >
> >   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>
> > 2018-04-30T14:36:15Z DEBUG Process
> >
> >   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >
> >   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >
> >   >2018-04-30T14:36:15Z DEBUG Starting external process
> >
> >   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>
> > 2018-04-30T14:36:15Z DEBUG Process
> >
> >   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >
> >   >2018-04-30T14:36:15Z DEBUG stderr=Job for
> > dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
> > failed because the control process exited with error code. See
> > "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>" and "journalctl -xe"
> for details.
> >
> >   >
> >
> >   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> > 504, in start_creation
> >
> >   >    run_step(full_msg, method)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> > 494, in run_step
> >
> >   >    method()
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
> > ",
> > line 95, in __start
> >
> >   >    srv.start(self.serverid, ldapi=True)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> > line 161, in start
> >
> >   >    instance_name, capture_output=capture_output, wait=wait)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> > 294, in start
> >
> >   >    skip_output=not capture_output)
> >
> >   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
> > 542, in run
> >
> >   >    raise CalledProcessError(p.returncode, arg_string, str(output))
> >
> >   >CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >
> >
> >   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> > '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >
> >
> >   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >
> >   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> > slapd_bootstrap_config
> > - The default password storage scheme could not be read or was not
> > found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
> >
> >   >
> >
> >   >Checking on internet I show that "dse.ldif" could be corrupted, so I
> > changed with "dse.ldif.startOK" without any change and then I changed
> > with "dse.ldif.bak". The problem persist but the error has changed:
> >
> >   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
> >
> >   >Non-Secure Port Disabled
> >
> >   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error
> > -
> >
> >   >Netscape Portable Runtime error -5975:
> >
> >   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
> >
> >   >replication_legacy_plugin_init
> >
> >   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error
> > -
> >
> >   >Could not load symbol "replication_legacy_plugin_init" from
> >
> >   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> > Legacy
> >
> >   >Replication Plugin
> >
> >   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry -
> > Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
> >
> >   >
> >
> >   >I saw a bug about this problem, but it is still opened:
> >
> >   >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >
> >   >
> >
> >   >Any idea how to fix the issue?
> >
> >   >
> >
> >   >If it is not possible to fix it, can I remove the replica from IPA
> > and install it again with the same name?
> >
> > A quick fix could be to remove an entry for cn=Legacy Replication
> > Plugin,cn=plugins,cn=config from
> > /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >
> > when dirsrv is down.
> >
> > --
> >
> > / Alexander Bokovoy
> >
> > Sr. Principal Software Engineer
> >
> > Security / Identity Management Engineering Red Hat Limited, Finland
> >
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> >
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>

[SOLER SANGUESA Miguel]

Hello,

This is the output of the command (seems that is not complete):

# ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config -s base
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
# extended LDIF
#
# LDAPv3
# base
<cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config> with
scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# indextask_description_137444551994158920_5958, index, tasks, config
dn: cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=conf
 ig
objectClass: top
objectClass: extensibleObject
nsindexattribute: description
nsinstance: userRoot
cn: indextask_description_137444551994158920_5958
nstaskcurrentitem: 0
nstasktotalitems: 1

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

Thanks & Regards.
______________________________
-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com&gt; 
Sent: Tuesday, May 01, 2018 15:18
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  Thank you for your answer, now dirsrv can start, but after running the " 
>  dirsrv(a)IPA-UNICC-ORG.service " it hungs:
>  
>  # ipa-server-upgrade
>  
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>     [1/8]: saving configuration
>  
>     [2/8]: disabling listeners
>  
>     [3/8]: enabling DS global lock
>  
>     [4/8]: starting directory server
>  
>     [5/8]: updating schema
>  
>     [6/8]: upgrading server
>  
>  The latest messages on the log is (there are no logs on the lasts 2h):
>  
>  2018-05-01T08:13:14Z DEBUG 
>  ---------------------------------------------
>  
>  2018-05-01T08:13:14Z DEBUG Final value after applying updates
>  
>  2018-05-01T08:13:14Z DEBUG dn: 
>  cn=description,cn=index,cn=userroot,cn=ldbm 
>  database,cn=plugins,cn=config
>  
>  2018-05-01T08:13:14Z DEBUG objectclass:
>  
>  2018-05-01T08:13:14Z DEBUG      top
>  
>  2018-05-01T08:13:14Z DEBUG      nsindex
>  
>  2018-05-01T08:13:14Z DEBUG nsindextype:
>  
>  2018-05-01T08:13:14Z DEBUG      eq
>  
>  2018-05-01T08:13:14Z DEBUG      sub
>  
>  2018-05-01T08:13:14Z DEBUG cn:
>  
>  2018-05-01T08:13:14Z DEBUG      description
>  
>  2018-05-01T08:13:14Z DEBUG nssystemindex:
>  
>  2018-05-01T08:13:14Z DEBUG      false
>  
>  2018-05-01T08:13:19Z DEBUG Creating task to index attribute: 
>  description
>  
>  2018-05-01T08:13:19Z DEBUG Task id: 
>  cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
>  config
>  
>  If I check the ipa services, dirsrv is the only one running:
>  
>  # ipactl status
>  
>  Directory Service: RUNNING
>  
>  krb5kdc Service: STOPPED
>  
>  kadmin Service: STOPPED
>  
>  named Service: STOPPED
>  
>  httpd Service: STOPPED
>  
>  ipa-custodia Service: STOPPED
>  
>  ntpd Service: RUNNING
>  
>  pki-tomcatd Service: STOPPED
>  
>  smb Service: STOPPED
>  
>  winbind Service: STOPPED
>  
>  ipa-otpd Service: STOPPED
>  
>  ipa-dnskeysyncd Service: STOPPED
>  
>  ipa: INFO: The ipactl command was successful
>  
>  Should I stop the ipa-server-upgrade and start ipa services? 
It depends on how many entries you have. This is an indexing task and while 2 hours seems
a bit excessive, it depends on how much work it has to do.

The 389-ds access and/or error logs may provide details.

rob

>  
>  Thanks.
>  
>  -----Original Message-----
>  From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
>  Sent: Tuesday, May 01, 2018 9:56
>  To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>  Subject: Re: [Freeipa-users] Problem on dirsrv when updating from 
>  4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>  
>  On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  
>   >hello,
>  
>   >
>  
>   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 
>  (RHEL 7.5). An hour later I tried to do the same with the unique 
>  replica I have, but after update dirsrv is not starting.
>  
>   >It says it is needed run "ipa-server-upgrade", but it also fails:
>  
>   ># ipactl start
>  
>   >Upgrade required: please run ipa-server-upgrade command Aborting 
>  ipactl
>  
>   >
>  
>   ># ipa-server-upgrade
>  
>   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
>  
>   >  [1/8]: saving configuration
>  
>   >  [2/8]: disabling listeners
>  
>   >  [3/8]: enabling DS global lock
>  
>   >  [4/8]: starting directory server
>  
>   >  [error] CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >  [cleanup]: stopping directory server
>  
>   >  [cleanup]: restoring configuration
>  
>   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run 
>  command ipa-server-upgrade manually.
>  
>   >Unexpected error - see /var/log/ipaupgrade.log for details:
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1 The
>  
>   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for 
>  more
>  
>   >information
>  
>   >
>  
>   >On the log I can see:
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
>  
>   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
>  
>   >2018-04-30T14:36:15Z DEBUG Starting external process
>  
>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>
>  2018-04-30T14:36:15Z DEBUG Process
>  
>   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
>  
>   >2018-04-30T14:36:15Z DEBUG stderr=Job for 
>  dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
>  failed because the control process exited with error code. See 
>  "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>" and "journalctl -xe" for
> details.
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  504, in start_creation
>  
>   >    run_step(full_msg, method)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>  494, in run_step
>  
>   >    method()
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
>  ",
>  line 95, in __start
>  
>   >    srv.start(self.serverid, ldapi=True)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", 
>  line 161, in start
>  
>   >    instance_name, capture_output=capture_output, wait=wait)
>  
>   >  File
>  "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 
>  294, in start
>  
>   >    skip_output=not capture_output)
>  
>   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 
>  542, in run
>  
>   >    raise CalledProcessError(p.returncode, arg_string, str(output))
>  
>   >CalledProcessError: Command '/bin/systemctl start
>  
>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >
>  
>   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command 
>  '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
>  <mailto:[e-mail]>' returned non-zero exit status 
>  1
>  
>   >
>  
>   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
>  
>   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
>  slapd_bootstrap_config
>  - The default password storage scheme could not be read or was not 
>  found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>  
>   >
>  
>   >Checking on internet I show that "dse.ldif" could be corrupted, so I 
>  changed with "dse.ldif.startOK" without any change and then I changed 
>  with "dse.ldif.bak". The problem persist but the error has changed:
>  
>   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
>  
>   >Non-Secure Port Disabled
>  
>   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error 
>  -
>  
>   >Netscape Portable Runtime error -5975:
>  
>   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
>  
>   >replication_legacy_plugin_init
>  
>   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error 
>  -
>  
>   >Could not load symbol "replication_legacy_plugin_init" from
>  
>   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin 
>  Legacy
>  
>   >Replication Plugin
>  
>   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - 
>  Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>  
>   >
>  
>   >I saw a bug about this problem, but it is still opened:
>  
>   >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>  
>   >
>  
>   >Any idea how to fix the issue?
>  
>   >
>  
>   >If it is not possible to fix it, can I remove the replica from IPA 
>  and install it again with the same name?
>  
>  A quick fix could be to remove an entry for cn=Legacy Replication 
>  Plugin,cn=plugins,cn=config from 
>  /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
>  
>  when dirsrv is down.
>  
>  --
>  
>  / Alexander Bokovoy
>  
>  Sr. Principal Software Engineer
>  
>  Security / Identity Management Engineering Red Hat Limited, Finland
>  
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to 
>  freeipa-users-leave(a)lists.fedorahosted.org
>

[shisheer guragain]

Hello,
           Did you have any issues with login to Web UI after upgrading to 7.5 new
version? 

Sent from my iPhone

> On May 2, 2018, at 3:00 AM, SOLER SANGUESA Miguel via FreeIPA-users
<freeipa-users(a)lists.fedorahosted.org&gt; wrote:
> 
> Hello,
> 
> This is the output of the command (seems that is not complete):
> 
> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config -s base
> SASL/EXTERNAL authentication started
> SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> SASL SSF: 0
> # extended LDIF
> #
> # LDAPv3
> # base
<cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config> with
scope baseObject
> # filter: (objectclass=*)
> # requesting: ALL
> #
> 
> # indextask_description_137444551994158920_5958, index, tasks, config
> dn: cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=conf
> ig
> objectClass: top
> objectClass: extensibleObject
> nsindexattribute: description
> nsinstance: userRoot
> cn: indextask_description_137444551994158920_5958
> nstaskcurrentitem: 0
> nstasktotalitems: 1
> 
> # search result
> search: 3
> result: 0 Success
> 
> # numResponses: 2
> # numEntries: 1
> 
> Thanks & Regards.
> ______________________________
> -----Original Message-----
> From: Rob Crittenden <rcritten(a)redhat.com&gt; 
> Sent: Tuesday, May 01, 2018 15:18
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL
7.4) to 4.5.4 (RHEL 7.5)
> 
> SOLER SANGUESA Miguel via FreeIPA-users wrote:
>> Hello,
>> 
>> Thank you for your answer, now dirsrv can start, but after running the " 
>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
>> 
>> # ipa-server-upgrade
>> 
>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>> 
>>   [1/8]: saving configuration
>> 
>>   [2/8]: disabling listeners
>> 
>>   [3/8]: enabling DS global lock
>> 
>>   [4/8]: starting directory server
>> 
>>   [5/8]: updating schema
>> 
>>   [6/8]: upgrading server
>> 
>> The latest messages on the log is (there are no logs on the lasts 2h):
>> 
>> 2018-05-01T08:13:14Z DEBUG 
>> ---------------------------------------------
>> 
>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
>> 
>> 2018-05-01T08:13:14Z DEBUG dn: 
>> cn=description,cn=index,cn=userroot,cn=ldbm 
>> database,cn=plugins,cn=config
>> 
>> 2018-05-01T08:13:14Z DEBUG objectclass:
>> 
>> 2018-05-01T08:13:14Z DEBUG      top
>> 
>> 2018-05-01T08:13:14Z DEBUG      nsindex
>> 
>> 2018-05-01T08:13:14Z DEBUG nsindextype:
>> 
>> 2018-05-01T08:13:14Z DEBUG      eq
>> 
>> 2018-05-01T08:13:14Z DEBUG      sub
>> 
>> 2018-05-01T08:13:14Z DEBUG cn:
>> 
>> 2018-05-01T08:13:14Z DEBUG      description
>> 
>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
>> 
>> 2018-05-01T08:13:14Z DEBUG      false
>> 
>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute: 
>> description
>> 
>> 2018-05-01T08:13:19Z DEBUG Task id: 
>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
>> config
>> 
>> If I check the ipa services, dirsrv is the only one running:
>> 
>> # ipactl status
>> 
>> Directory Service: RUNNING
>> 
>> krb5kdc Service: STOPPED
>> 
>> kadmin Service: STOPPED
>> 
>> named Service: STOPPED
>> 
>> httpd Service: STOPPED
>> 
>> ipa-custodia Service: STOPPED
>> 
>> ntpd Service: RUNNING
>> 
>> pki-tomcatd Service: STOPPED
>> 
>> smb Service: STOPPED
>> 
>> winbind Service: STOPPED
>> 
>> ipa-otpd Service: STOPPED
>> 
>> ipa-dnskeysyncd Service: STOPPED
>> 
>> ipa: INFO: The ipactl command was successful
>> 
>> Should I stop the ipa-server-upgrade and start ipa services?
> 
> It depends on how many entries you have. This is an indexing task and while 2 hours
seems a bit excessive, it depends on how much work it has to do.
> 
> The 389-ds access and/or error logs may provide details.
> 
> rob
> 
>> 
>> Thanks.
>> 
>> -----Original Message-----
>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
>> Sent: Tuesday, May 01, 2018 9:56
>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from 
>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>> 
>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
>>> 
>>> hello,
>> 
>>> 
>> 
>>> I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 
>> (RHEL 7.5). An hour later I tried to do the same with the unique 
>> replica I have, but after update dirsrv is not starting.
>> 
>>> It says it is needed run "ipa-server-upgrade", but it also fails:
>> 
>>> # ipactl start
>> 
>>> Upgrade required: please run ipa-server-upgrade command Aborting 
>> ipactl
>> 
>>> 
>> 
>>> # ipa-server-upgrade
>> 
>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>> 
>>>   [1/8]: saving configuration
>> 
>>>   [2/8]: disabling listeners
>> 
>>>   [3/8]: enabling DS global lock
>> 
>>>   [4/8]: starting directory server
>> 
>>>   [error] CalledProcessError: Command '/bin/systemctl start
>> 
>>> dirsrv(a)IPA-EXAMOLE-ORG.service
>> <mailto:[e-mail]>' returned non-zero exit status

>> 1
>> 
>>>   [cleanup]: stopping directory server
>> 
>>>   [cleanup]: restoring configuration
>> 
>>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run 
>> command ipa-server-upgrade manually.
>> 
>>> Unexpected error - see /var/log/ipaupgrade.log for details:
>> 
>>> CalledProcessError: Command '/bin/systemctl start
>> 
>>> dirsrv(a)IPA-EXAMPLE-ORG.service
>> <mailto:[e-mail]>' returned non-zero exit status

>> 1 The
>> 
>>> ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for 
>> more
>> 
>>> information
>> 
>>> 
>> 
>>> On the log I can see:
>> 
>>> 2018-04-30T14:36:15Z DEBUG Starting external process
>> 
>>> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
>> 
>>> dirsrv(a)IPA-EXAMPLE-ORG.service 
>> <mailto:[e-mail]>
>> 2018-04-30T14:36:15Z DEBUG Process
>> 
>>> finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
>> 
>>> 2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
>> 
>>> 2018-04-30T14:36:15Z DEBUG Starting external process
>> 
>>> 2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>> 
>>> dirsrv(a)IPA-EXAMPLE-ORG.service 
>> <mailto:[e-mail]>
>> 2018-04-30T14:36:15Z DEBUG Process
>> 
>>> finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
>> 
>>> 2018-04-30T14:36:15Z DEBUG stderr=Job for 
>> dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
>> failed because the control process exited with error code. See 
>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
>> <mailto:[e-mail]>" and "journalctl
-xe" for details.
>> 
>>> 
>> 
>>> 2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>> 
>>>   File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>> 504, in start_creation
>> 
>>>     run_step(full_msg, method)
>> 
>>>   File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 
>> 494, in run_step
>> 
>>>     method()
>> 
>>>   File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
>> ",
>> line 95, in __start
>> 
>>>     srv.start(self.serverid, ldapi=True)
>> 
>>>   File
>> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", 
>> line 161, in start
>> 
>>>     instance_name, capture_output=capture_output, wait=wait)
>> 
>>>   File
>> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 
>> 294, in start
>> 
>>>     skip_output=not capture_output)
>> 
>>>   File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
line 
>> 542, in run
>> 
>>>     raise CalledProcessError(p.returncode, arg_string, str(output))
>> 
>>> CalledProcessError: Command '/bin/systemctl start
>> 
>>> dirsrv(a)IPA-EXAMPLE-ORG.service
>> <mailto:[e-mail]>' returned non-zero exit status

>> 1
>> 
>>> 
>> 
>>> 2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command 
>> '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
>> <mailto:[e-mail]>' returned non-zero exit status

>> 1
>> 
>>> 
>> 
>>> Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
>> 
>>> [30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
>> slapd_bootstrap_config
>> - The default password storage scheme could not be read or was not 
>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>> 
>>> 
>> 
>>> Checking on internet I show that "dse.ldif" could be corrupted, so
I 
>> changed with "dse.ldif.startOK" without any change and then I changed 
>> with "dse.ldif.bak". The problem persist but the error has changed:
>> 
>>> [30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
>> 
>>> Non-Secure Port Disabled
>> 
>>> [30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error 
>> -
>> 
>>> Netscape Portable Runtime error -5975:
>> 
>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
>> 
>>> replication_legacy_plugin_init
>> 
>>> [30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error 
>> -
>> 
>>> Could not load symbol "replication_legacy_plugin_init" from
>> 
>>> "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin 
>> Legacy
>> 
>>> Replication Plugin
>> 
>>> [30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - 
>> Unable to load plugin "cn=Legacy Replication
Plugin,cn=plugins,cn=config"
>> 
>>> 
>> 
>>> I saw a bug about this problem, but it is still opened:
>> 
>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>> 
>>> 
>> 
>>> Any idea how to fix the issue?
>> 
>>> 
>> 
>>> If it is not possible to fix it, can I remove the replica from IPA 
>> and install it again with the same name?
>> 
>> A quick fix could be to remove an entry for cn=Legacy Replication 
>> Plugin,cn=plugins,cn=config from 
>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
>> 
>> when dirsrv is down.
>> 
>> --
>> 
>> / Alexander Bokovoy
>> 
>> Sr. Principal Software Engineer
>> 
>> Security / Identity Management Engineering Red Hat Limited, Finland
>> 
>> 
>> 
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to 
>> freeipa-users-leave(a)lists.fedorahosted.org
>> 
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org

[Rob Crittenden]

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  This is the output of the command (seems that is not complete):
>  
>  # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config -s base
>  SASL/EXTERNAL authentication started
>  SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>  SASL SSF: 0
>  # extended LDIF
>  #
>  # LDAPv3
>  # base
> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config> with
> scope baseObject
>  # filter: (objectclass=*)
>  # requesting: ALL
>  #
>  
>  # indextask_description_137444551994158920_5958, index, tasks, config
>  dn: cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=conf
>    ig
>  objectClass: top
>  objectClass: extensibleObject
>  nsindexattribute: description
>  nsinstance: userRoot
>  cn: indextask_description_137444551994158920_5958
>  nstaskcurrentitem: 0
>  nstasktotalitems: 1 
IPA is looking for nstaskstatus and since that isn't there it is looping 
forever.

I've cc'd a couple of the 389-ds devs to see what they think about it.

rob

>  
>  # search result
>  search: 3
>  result: 0 Success
>  
>  # numResponses: 2
>  # numEntries: 1
>  
>  Thanks & Regards.
>  ______________________________
>  -----Original Message-----
>  From: Rob Crittenden <rcritten(a)redhat.com&gt;
>  Sent: Tuesday, May 01, 2018 15:18
>  To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>  
>  SOLER SANGUESA Miguel via FreeIPA-users wrote:
> > Hello,
> >
> > Thank you for your answer, now dirsrv can start, but after running the "
> > dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >
> > # ipa-server-upgrade
> >
> > Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >
> >     [1/8]: saving configuration
> >
> >     [2/8]: disabling listeners
> >
> >     [3/8]: enabling DS global lock
> >
> >     [4/8]: starting directory server
> >
> >     [5/8]: updating schema
> >
> >     [6/8]: upgrading server
> >
> > The latest messages on the log is (there are no logs on the lasts 2h):
> >
> > 2018-05-01T08:13:14Z DEBUG
> > ---------------------------------------------
> >
> > 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >
> > 2018-05-01T08:13:14Z DEBUG dn:
> > cn=description,cn=index,cn=userroot,cn=ldbm
> > database,cn=plugins,cn=config
> >
> > 2018-05-01T08:13:14Z DEBUG objectclass:
> >
> > 2018-05-01T08:13:14Z DEBUG      top
> >
> > 2018-05-01T08:13:14Z DEBUG      nsindex
> >
> > 2018-05-01T08:13:14Z DEBUG nsindextype:
> >
> > 2018-05-01T08:13:14Z DEBUG      eq
> >
> > 2018-05-01T08:13:14Z DEBUG      sub
> >
> > 2018-05-01T08:13:14Z DEBUG cn:
> >
> > 2018-05-01T08:13:14Z DEBUG      description
> >
> > 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >
> > 2018-05-01T08:13:14Z DEBUG      false
> >
> > 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> > description
> >
> > 2018-05-01T08:13:19Z DEBUG Task id:
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
> > config
> >
> > If I check the ipa services, dirsrv is the only one running:
> >
> > # ipactl status
> >
> > Directory Service: RUNNING
> >
> > krb5kdc Service: STOPPED
> >
> > kadmin Service: STOPPED
> >
> > named Service: STOPPED
> >
> > httpd Service: STOPPED
> >
> > ipa-custodia Service: STOPPED
> >
> > ntpd Service: RUNNING
> >
> > pki-tomcatd Service: STOPPED
> >
> > smb Service: STOPPED
> >
> > winbind Service: STOPPED
> >
> > ipa-otpd Service: STOPPED
> >
> > ipa-dnskeysyncd Service: STOPPED
> >
> > ipa: INFO: The ipactl command was successful
> >
> > Should I stop the ipa-server-upgrade and start ipa services?
>  
>  It depends on how many entries you have. This is an indexing task and while 2 hours seems
> a bit excessive, it depends on how much work it has to do.
>  
>  The 389-ds access and/or error logs may provide details.
>  
>  rob
>  
> >
> > Thanks.
> >
> > -----Original Message-----
> > From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> > Sent: Tuesday, May 01, 2018 9:56
> > To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> > Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >
> >   >hello,
> >
> >   >
> >
> >   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4
> > (RHEL 7.5). An hour later I tried to do the same with the unique
> > replica I have, but after update dirsrv is not starting.
> >
> >   >It says it is needed run "ipa-server-upgrade", but it also fails:
> >
> >   ># ipactl start
> >
> >   >Upgrade required: please run ipa-server-upgrade command Aborting
> > ipactl
> >
> >   >
> >
> >   ># ipa-server-upgrade
> >
> >   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >
> >   >  [1/8]: saving configuration
> >
> >   >  [2/8]: disabling listeners
> >
> >   >  [3/8]: enabling DS global lock
> >
> >   >  [4/8]: starting directory server
> >
> >   >  [error] CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >  [cleanup]: stopping directory server
> >
> >   >  [cleanup]: restoring configuration
> >
> >   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> > command ipa-server-upgrade manually.
> >
> >   >Unexpected error - see /var/log/ipaupgrade.log for details:
> >
> >   >CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1 The
> >
> >   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> > more
> >
> >   >information
> >
> >   >
> >
> >   >On the log I can see:
> >
> >   >2018-04-30T14:36:15Z DEBUG Starting external process
> >
> >   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>
> > 2018-04-30T14:36:15Z DEBUG Process
> >
> >   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >
> >   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >
> >   >2018-04-30T14:36:15Z DEBUG Starting external process
> >
> >   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>
> > 2018-04-30T14:36:15Z DEBUG Process
> >
> >   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >
> >   >2018-04-30T14:36:15Z DEBUG stderr=Job for
> > dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
> > failed because the control process exited with error code. See
> > "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>" and "journalctl -xe"
> for details.
> >
> >   >
> >
> >   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> > 504, in start_creation
> >
> >   >    run_step(full_msg, method)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> > 494, in run_step
> >
> >   >    method()
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
> > ",
> > line 95, in __start
> >
> >   >    srv.start(self.serverid, ldapi=True)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> > line 161, in start
> >
> >   >    instance_name, capture_output=capture_output, wait=wait)
> >
> >   >  File
> > "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> > 294, in start
> >
> >   >    skip_output=not capture_output)
> >
> >   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line
> > 542, in run
> >
> >   >    raise CalledProcessError(p.returncode, arg_string, str(output))
> >
> >   >CalledProcessError: Command '/bin/systemctl start
> >
> >   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >
> >
> >   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> > '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> > <mailto:[e-mail]>' returned non-zero exit status
> > 1
> >
> >   >
> >
> >   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >
> >   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> > slapd_bootstrap_config
> > - The default password storage scheme could not be read or was not
> > found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
> >
> >   >
> >
> >   >Checking on internet I show that "dse.ldif" could be corrupted, so I
> > changed with "dse.ldif.startOK" without any change and then I changed
> > with "dse.ldif.bak". The problem persist but the error has changed:
> >
> >   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
> >
> >   >Non-Secure Port Disabled
> >
> >   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error
> > -
> >
> >   >Netscape Portable Runtime error -5975:
> >
> >   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined symbol:
> >
> >   >replication_legacy_plugin_init
> >
> >   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error
> > -
> >
> >   >Could not load symbol "replication_legacy_plugin_init" from
> >
> >   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> > Legacy
> >
> >   >Replication Plugin
> >
> >   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry -
> > Unable to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
> >
> >   >
> >
> >   >I saw a bug about this problem, but it is still opened:
> >
> >   >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >
> >   >
> >
> >   >Any idea how to fix the issue?
> >
> >   >
> >
> >   >If it is not possible to fix it, can I remove the replica from IPA
> > and install it again with the same name?
> >
> > A quick fix could be to remove an entry for cn=Legacy Replication
> > Plugin,cn=plugins,cn=config from
> > /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >
> > when dirsrv is down.
> >
> > --
> >
> > / Alexander Bokovoy
> >
> > Sr. Principal Software Engineer
> >
> > Security / Identity Management Engineering Red Hat Limited, Finland
> >
> >
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> >
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>

[thierry bordaz]

Hi,

During indexing task we should see in the task status the periodic 
progression of the indexing.
May be the indexing is hanging somewhere. When the problem occurs could 
you provide a pstack of the dirsrv server ?

best regards
thierry

On 05/02/2018 10:27 PM, Rob Crittenden wrote:
>  SOLER SANGUESA Miguel via FreeIPA-users wrote:
> > Hello,
> >
> > This is the output of the command (seems that is not complete):
> >
> > # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket 
> > -b 
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config 
> > -s base
> > SASL/EXTERNAL authentication started
> > SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> > SASL SSF: 0
> > # extended LDIF
> > #
> > # LDAPv3
> > # base 
> > <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=config>
>
> > with scope baseObject
> > # filter: (objectclass=*)
> > # requesting: ALL
> > #
> >
> > # indextask_description_137444551994158920_5958, index, tasks, config
> > dn: 
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=conf
> >   ig
> > objectClass: top
> > objectClass: extensibleObject
> > nsindexattribute: description
> > nsinstance: userRoot
> > cn: indextask_description_137444551994158920_5958
> > nstaskcurrentitem: 0
> > nstasktotalitems: 1
>
>  IPA is looking for nstaskstatus and since that isn't there it is 
>  looping forever.
>
>  I've cc'd a couple of the 389-ds devs to see what they think about it.
>
>  rob
>
> >
> > # search result
> > search: 3
> > result: 0 Success
> >
> > # numResponses: 2
> > # numEntries: 1
> >
> > Thanks & Regards.
> > ______________________________
> > -----Original Message-----
> > From: Rob Crittenden <rcritten(a)redhat.com&gt;
> > Sent: Tuesday, May 01, 2018 15:18
> > To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >> Hello,
> >>
> >> Thank you for your answer, now dirsrv can start, but after running 
> >> the "
> >> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>
> >> # ipa-server-upgrade
> >>
> >> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>
> >>     [1/8]: saving configuration
> >>
> >>     [2/8]: disabling listeners
> >>
> >>     [3/8]: enabling DS global lock
> >>
> >>     [4/8]: starting directory server
> >>
> >>     [5/8]: updating schema
> >>
> >>     [6/8]: upgrading server
> >>
> >> The latest messages on the log is (there are no logs on the lasts 2h):
> >>
> >> 2018-05-01T08:13:14Z DEBUG
> >> ---------------------------------------------
> >>
> >> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>
> >> 2018-05-01T08:13:14Z DEBUG dn:
> >> cn=description,cn=index,cn=userroot,cn=ldbm
> >> database,cn=plugins,cn=config
> >>
> >> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      top
> >>
> >> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>
> >> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      eq
> >>
> >> 2018-05-01T08:13:14Z DEBUG      sub
> >>
> >> 2018-05-01T08:13:14Z DEBUG cn:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      description
> >>
> >> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      false
> >>
> >> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >> description
> >>
> >> 2018-05-01T08:13:19Z DEBUG Task id:
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn=
> >> config
> >>
> >> If I check the ipa services, dirsrv is the only one running:
> >>
> >> # ipactl status
> >>
> >> Directory Service: RUNNING
> >>
> >> krb5kdc Service: STOPPED
> >>
> >> kadmin Service: STOPPED
> >>
> >> named Service: STOPPED
> >>
> >> httpd Service: STOPPED
> >>
> >> ipa-custodia Service: STOPPED
> >>
> >> ntpd Service: RUNNING
> >>
> >> pki-tomcatd Service: STOPPED
> >>
> >> smb Service: STOPPED
> >>
> >> winbind Service: STOPPED
> >>
> >> ipa-otpd Service: STOPPED
> >>
> >> ipa-dnskeysyncd Service: STOPPED
> >>
> >> ipa: INFO: The ipactl command was successful
> >>
> >> Should I stop the ipa-server-upgrade and start ipa services?
> >
> > It depends on how many entries you have. This is an indexing task and 
> > while 2 hours seems a bit excessive, it depends on how much work it 
> > has to do.
> >
> > The 389-ds access and/or error logs may provide details.
> >
> > rob
> >
> >>
> >> Thanks.
> >>
> >> -----Original Message-----
> >> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >> Sent: Tuesday, May 01, 2018 9:56
> >> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>
> >>   >hello,
> >>
> >>   >
> >>
> >>   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4
> >> (RHEL 7.5). An hour later I tried to do the same with the unique
> >> replica I have, but after update dirsrv is not starting.
> >>
> >>   >It says it is needed run "ipa-server-upgrade", but it also
> fails:
> >>
> >>   ># ipactl start
> >>
> >>   >Upgrade required: please run ipa-server-upgrade command Aborting
> >> ipactl
> >>
> >>   >
> >>
> >>   ># ipa-server-upgrade
> >>
> >>   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>
> >>   >  [1/8]: saving configuration
> >>
> >>   >  [2/8]: disabling listeners
> >>
> >>   >  [3/8]: enabling DS global lock
> >>
> >>   >  [4/8]: starting directory server
> >>
> >>   >  [error] CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit status
> >> 1
> >>
> >>   >  [cleanup]: stopping directory server
> >>
> >>   >  [cleanup]: restoring configuration
> >>
> >>   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> >> command ipa-server-upgrade manually.
> >>
> >>   >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>
> >>   >CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit status
> >> 1 The
> >>
> >>   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> >> more
> >>
> >>   >information
> >>
> >>   >
> >>
> >>   >On the log I can see:
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Starting external process
> >>
> >>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>
> >> 2018-04-30T14:36:15Z DEBUG Process
> >>
> >>   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >>
> >>   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Starting external process
> >>
> >>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>
> >> 2018-04-30T14:36:15Z DEBUG Process
> >>
> >>   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>
> >>   >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >> dirsrv(a)IPA-EXAMPLE-ORG.service <mailto:[e-mail]>
> >> failed because the control process exited with error code. See
> >> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>" and "journalctl
> -xe" for 
> >> details.
> >>
> >>   >
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> >> 504, in start_creation
> >>
> >>   >    run_step(full_msg, method)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> >> 494, in run_step
> >>
> >>   >    method()
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py
> >> ",
> >> line 95, in __start
> >>
> >>   >    srv.start(self.serverid, ldapi=True)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >> line 161, in start
> >>
> >>   >    instance_name, capture_output=capture_output, wait=wait)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> >> 294, in start
> >>
> >>   >    skip_output=not capture_output)
> >>
> >>   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> line
> >> 542, in run
> >>
> >>   >    raise CalledProcessError(p.returncode, arg_string, str(output))
> >>
> >>   >CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit status
> >> 1
> >>
> >>   >
> >>
> >>   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> >> '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit status
> >> 1
> >>
> >>   >
> >>
> >>   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>
> >>   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >> slapd_bootstrap_config
> >> - The default password storage scheme could not be read or was not
> >> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is 
> >> mandatory.
> >>
> >>   >
> >>
> >>   >Checking on internet I show that "dse.ldif" could be corrupted,
> so I
> >> changed with "dse.ldif.startOK" without any change and then I changed
> >> with "dse.ldif.bak". The problem persist but the error has changed:
> >>
> >>   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
> >>
> >>   >Non-Secure Port Disabled
> >>
> >>   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error
> >> -
> >>
> >>   >Netscape Portable Runtime error -5975:
> >>
> >>   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined 
> >> symbol:
> >>
> >>   >replication_legacy_plugin_init
> >>
> >>   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error
> >> -
> >>
> >>   >Could not load symbol "replication_legacy_plugin_init" from
> >>
> >>   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> >> Legacy
> >>
> >>   >Replication Plugin
> >>
> >>   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry -
> >> Unable to load plugin "cn=Legacy Replication 
> >> Plugin,cn=plugins,cn=config"
> >>
> >>   >
> >>
> >>   >I saw a bug about this problem, but it is still opened:
> >>
> >> >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >>
> >>   >
> >>
> >>   >Any idea how to fix the issue?
> >>
> >>   >
> >>
> >>   >If it is not possible to fix it, can I remove the replica from IPA
> >> and install it again with the same name?
> >>
> >> A quick fix could be to remove an entry for cn=Legacy Replication
> >> Plugin,cn=plugins,cn=config from
> >> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>
> >> when dirsrv is down.
> >>
> >> -- 
> >>
> >> / Alexander Bokovoy
> >>
> >> Sr. Principal Software Engineer
> >>
> >> Security / Identity Management Engineering Red Hat Limited, Finland
> >>
> >>
> >>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >>
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to 
> > freeipa-users-leave(a)lists.fedorahosted.org
> >
>

[SOLER SANGUESA Miguel]

hello,

Yesterday my ssh console closed the connection, so I had to start again the
"ipa-server-upgrade", but the result is more or less the same:
# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/10]: stopping directory server
  [2/10]: saving configuration
  [3/10]: disabling listeners
  [4/10]: enabling DS global lock
  [5/10]: starting directory server
  [6/10]: updating schema
  [7/10]: upgrading server

But now, the lines that are repeated on the access log are:
[03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
[03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0 tag=101 nentries=1
etime=1.0000687740

And it is on the same state than the other:
# ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
# extended LDIF
#
# LDAPv3
# base <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config> with
scope baseObject
# filter: (objectclass=*)
# requesting: ALL
#

# indextask_l_137445500911864330_4055, index, tasks, config
dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
objectClass: top
objectClass: extensibleObject
nsindexattribute: l
nsinstance: userRoot
cn: indextask_l_137445500911864330_4055
nstaskcurrentitem: 0
nstasktotalitems: 1

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

The pstack of the proces: /usr/bin/python2 /usr/sbin/ipa-server-upgrade
# pstack 9805
#0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
#1  0x00007fb8ff5e0000 in time_sleep () from
/usr/lib64/python2.7/lib-dynload/timemodule.so
#2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#6  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#7  0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#11 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#12 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#15 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#20 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#21 0x00007fb90730d142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
#23 0x00007fb90732773e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
#26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
#27 0x000000000040066e in _start ()


And the pstack from the process: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-IPA-EXAMPLE-ORG
-i /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
# pstack 15536
Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 49 (Thread 0x7f5bc02e8700 (LWP 15540)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 48 (Thread 0x7f5bbfae7700 (LWP 15541)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 47 (Thread 0x7f5bbf2e6700 (LWP 15542)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 46 (Thread 0x7f5bbeae5700 (LWP 15543)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 45 (Thread 0x7f5bbe2e4700 (LWP 15544)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 44 (Thread 0x7f5bbdae3700 (LWP 15545)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 43 (Thread 0x7f5bbd2e2700 (LWP 15546)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 42 (Thread 0x7f5bbcae1700 (LWP 15547)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 41 (Thread 0x7f5bbc2e0700 (LWP 15548)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 40 (Thread 0x7f5bbbadf700 (LWP 15549)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 39 (Thread 0x7f5bbb2de700 (LWP 15550)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 38 (Thread 0x7f5bbaadd700 (LWP 15551)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 37 (Thread 0x7f5bba2dc700 (LWP 15552)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 36 (Thread 0x7f5bb9adb700 (LWP 15553)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 35 (Thread 0x7f5bb92da700 (LWP 15554)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 34 (Thread 0x7f5bb8ad9700 (LWP 15555)):
#0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
#1  0x00007f5bd1d9c803 in epoll_dispatch () at /lib64/libevent-2.0.so.5
#2  0x00007f5bd1d883ea in event_base_loop () at /lib64/libevent-2.0.so.5
#3  0x00007f5bd4d34eae in ns_event_fw_loop () at /usr/lib64/dirsrv/libnunc-stans.so.0
#4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
/usr/lib64/dirsrv/libnunc-stans.so.0
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 33 (Thread 0x7f5bb2b84700 (LWP 15559)):
#0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
#1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc3e41e77 in deadlock_threadmain () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 32 (Thread 0x7f5bb2383700 (LWP 15560)):
#0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
#1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 31 (Thread 0x7f5bb1b82700 (LWP 15561)):
#0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
#1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc3e420c7 in trickle_threadmain () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 30 (Thread 0x7f5bb1381700 (LWP 15562)):
#0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
#1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc3e3ce47 in perf_threadmain () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 29 (Thread 0x7f5bb0b80700 (LWP 15563)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f5bc920213e in cos_cache_wait_on_change () at
/usr/lib64/dirsrv/plugins/libcos-plugin.so
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 28 (Thread 0x7f5bb037f700 (LWP 15564)):
#0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
#1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
/usr/lib64/dirsrv/plugins/libipa_cldap.so
#2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 27 (Thread 0x7f5bd5132700 (LWP 15566)):
#0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
/lib64/libpthread.so.0
#1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
#2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
#3  0x00007f5bc3572033 in _cl5TrimMain () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 26 (Thread 0x7f5bafb7e700 (LWP 15567)):
#0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
/lib64/libpthread.so.0
#1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
#2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
#3  0x00007f5bc3587fa4 in protocol_sleep () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007f5bc358b75a in repl5_inc_run () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#5  0x00007f5bc358fbec in prot_thread_main () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 25 (Thread 0x7f5baf37d700 (LWP 15568)):
#0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
/lib64/libpthread.so.0
#1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
#2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
#3  0x00007f5bc3587fa4 in protocol_sleep () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#4  0x00007f5bc358b75a in repl5_inc_run () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#5  0x00007f5bc358fbec in prot_thread_main () at
/usr/lib64/dirsrv/plugins/libreplication-plugin.so
#6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 24 (Thread 0x7f5bae759700 (LWP 15569)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 23 (Thread 0x7f5badf58700 (LWP 15570)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 22 (Thread 0x7f5bad757700 (LWP 15571)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
#3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
/usr/lib64/dirsrv/plugins/libroles-plugin.so
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 21 (Thread 0x7f5bacd4f700 (LWP 15572)):
#0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
/lib64/libpthread.so.0
#1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
#2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
#3  0x0000564591e10b08 in housecleaning ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 20 (Thread 0x7f5bac54e700 (LWP 15573)):
#0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
/lib64/libpthread.so.0
#1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
#2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
#3  0x00007f5bd4a5ba63 in eq_loop () at /usr/lib64/dirsrv/libslapd.so.0
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 19 (Thread 0x7f5babd4d700 (LWP 15574)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 18 (Thread 0x7f5bab54c700 (LWP 15575)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 17 (Thread 0x7f5baad4b700 (LWP 15576)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 16 (Thread 0x7f5baa54a700 (LWP 15577)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 15 (Thread 0x7f5ba9d49700 (LWP 15578)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 14 (Thread 0x7f5ba9548700 (LWP 15579)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 13 (Thread 0x7f5ba8d47700 (LWP 15580)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 12 (Thread 0x7f5ba8546700 (LWP 15581)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 11 (Thread 0x7f5ba7d45700 (LWP 15582)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 10 (Thread 0x7f5ba7544700 (LWP 15583)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 9 (Thread 0x7f5ba6d43700 (LWP 15584)):
#0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
#1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
#2  0x0000564591e07049 in connection_read_operation ()
#3  0x0000564591e0765d in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 8 (Thread 0x7f5ba6542700 (LWP 15585)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 7 (Thread 0x7f5ba5d41700 (LWP 15586)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 6 (Thread 0x7f5ba5540700 (LWP 15587)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 5 (Thread 0x7f5ba4d3f700 (LWP 15588)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 4 (Thread 0x7f5ba453e700 (LWP 15589)):
#0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
#1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
#2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
#3  0x0000564591e075d7 in connection_threadmain ()
#4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 3 (Thread 0x7f5ba3900700 (LWP 15590)):
#0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
#1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
/usr/lib64/dirsrv/libslapd.so.0
#5  0x00007f5bd4aace0c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
#6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
#7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
/usr/lib64/dirsrv/libslapd.so.0
#8  0x00007f5bd4a8a0d1 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
#9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
/usr/lib64/dirsrv/libslapd.so.0
#10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#13 0x00007f5bc1eba597 in wrap_pthread_starter () at
/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
#14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 2 (Thread 0x7f5ba30ff700 (LWP 15606)):
#0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at /lib64/libpthread.so.0
#1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
/usr/lib64/dirsrv/libslapd.so.0
#2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#4  0x00007f5bd4ac5942 in task_index_thread () at /usr/lib64/dirsrv/libslapd.so.0
#5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
#6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
#7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
Thread 1 (Thread 0x7f5bd5135940 (LWP 15536)):
#0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
#1  0x00007f5bd4d34b2f in ns_thrpool_wait () at /usr/lib64/dirsrv/libnunc-stans.so.0
#2  0x0000564591e0d862 in slapd_daemon ()
#3  0x0000564591dfe86f in main ()

Thanks & Regards.
______________________________

-----Original Message-----
From: thierry bordaz <tbordaz(a)redhat.com&gt; 
Sent: Thursday, May 03, 2018 9:31
To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
<solerm(a)unicc.org&gt;
Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

Hi,

During indexing task we should see in the task status the periodic progression of the
indexing.
May be the indexing is hanging somewhere. When the problem occurs could you provide a
pstack of the dirsrv server ?

best regards
thierry

On 05/02/2018 10:27 PM, Rob Crittenden wrote:
>  SOLER SANGUESA Miguel via FreeIPA-users wrote:
> > Hello,
> >
> > This is the output of the command (seems that is not complete):
> >
> > # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> > -b
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> > =config
> > -s base
> > SASL/EXTERNAL authentication started
> > SASL username: 
> > gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> > SASL SSF: 0
> > # extended LDIF
> > #
> > # LDAPv3
> > # base
> > <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> > n=config>
> > with scope baseObject
> > # filter: (objectclass=*)
> > # requesting: ALL
> > #
> >
> > # indextask_description_137444551994158920_5958, index, tasks, config
> > dn: 
> > cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> > =conf
> >   ig
> > objectClass: top
> > objectClass: extensibleObject
> > nsindexattribute: description
> > nsinstance: userRoot
> > cn: indextask_description_137444551994158920_5958
> > nstaskcurrentitem: 0
> > nstasktotalitems: 1
>
>  IPA is looking for nstaskstatus and since that isn't there it is 
>  looping forever.
>
>  I've cc'd a couple of the 389-ds devs to see what they think about it.
>
>  rob
>
> >
> > # search result
> > search: 3
> > result: 0 Success
> >
> > # numResponses: 2
> > # numEntries: 1
> >
> > Thanks & Regards.
> > ______________________________
> > -----Original Message-----
> > From: Rob Crittenden <rcritten(a)redhat.com&gt;
> > Sent: Tuesday, May 01, 2018 15:18
> > To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >> Hello,
> >>
> >> Thank you for your answer, now dirsrv can start, but after running 
> >> the "
> >> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>
> >> # ipa-server-upgrade
> >>
> >> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>
> >>     [1/8]: saving configuration
> >>
> >>     [2/8]: disabling listeners
> >>
> >>     [3/8]: enabling DS global lock
> >>
> >>     [4/8]: starting directory server
> >>
> >>     [5/8]: updating schema
> >>
> >>     [6/8]: upgrading server
> >>
> >> The latest messages on the log is (there are no logs on the lasts 2h):
> >>
> >> 2018-05-01T08:13:14Z DEBUG
> >> ---------------------------------------------
> >>
> >> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>
> >> 2018-05-01T08:13:14Z DEBUG dn:
> >> cn=description,cn=index,cn=userroot,cn=ldbm
> >> database,cn=plugins,cn=config
> >>
> >> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      top
> >>
> >> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>
> >> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      eq
> >>
> >> 2018-05-01T08:13:14Z DEBUG      sub
> >>
> >> 2018-05-01T08:13:14Z DEBUG cn:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      description
> >>
> >> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>
> >> 2018-05-01T08:13:14Z DEBUG      false
> >>
> >> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >> description
> >>
> >> 2018-05-01T08:13:19Z DEBUG Task id:
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >> n=
> >> config
> >>
> >> If I check the ipa services, dirsrv is the only one running:
> >>
> >> # ipactl status
> >>
> >> Directory Service: RUNNING
> >>
> >> krb5kdc Service: STOPPED
> >>
> >> kadmin Service: STOPPED
> >>
> >> named Service: STOPPED
> >>
> >> httpd Service: STOPPED
> >>
> >> ipa-custodia Service: STOPPED
> >>
> >> ntpd Service: RUNNING
> >>
> >> pki-tomcatd Service: STOPPED
> >>
> >> smb Service: STOPPED
> >>
> >> winbind Service: STOPPED
> >>
> >> ipa-otpd Service: STOPPED
> >>
> >> ipa-dnskeysyncd Service: STOPPED
> >>
> >> ipa: INFO: The ipactl command was successful
> >>
> >> Should I stop the ipa-server-upgrade and start ipa services?
> >
> > It depends on how many entries you have. This is an indexing task and 
> > while 2 hours seems a bit excessive, it depends on how much work it 
> > has to do.
> >
> > The 389-ds access and/or error logs may provide details.
> >
> > rob
> >
> >>
> >> Thanks.
> >>
> >> -----Original Message-----
> >> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >> Sent: Tuesday, May 01, 2018 9:56
> >> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>
> >>   >hello,
> >>
> >>   >
> >>
> >>   >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 
> >> (RHEL 7.5). An hour later I tried to do the same with the unique 
> >> replica I have, but after update dirsrv is not starting.
> >>
> >>   >It says it is needed run "ipa-server-upgrade", but it also
> fails:
> >>
> >>   ># ipactl start
> >>
> >>   >Upgrade required: please run ipa-server-upgrade command Aborting 
> >> ipactl
> >>
> >>   >
> >>
> >>   ># ipa-server-upgrade
> >>
> >>   >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>
> >>   >  [1/8]: saving configuration
> >>
> >>   >  [2/8]: disabling listeners
> >>
> >>   >  [3/8]: enabling DS global lock
> >>
> >>   >  [4/8]: starting directory server
> >>
> >>   >  [error] CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit 
> >> status
> >> 1
> >>
> >>   >  [cleanup]: stopping directory server
> >>
> >>   >  [cleanup]: restoring configuration
> >>
> >>   >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and 
> >> run command ipa-server-upgrade manually.
> >>
> >>   >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>
> >>   >CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit 
> >> status
> >> 1 The
> >>
> >>   >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log 
> >> for more
> >>
> >>   >information
> >>
> >>   >
> >>
> >>   >On the log I can see:
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Starting external process
> >>
> >>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>
> >> 2018-04-30T14:36:15Z DEBUG Process
> >>
> >>   >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >>
> >>   >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Starting external process
> >>
> >>   >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>
> >> 2018-04-30T14:36:15Z DEBUG Process
> >>
> >>   >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>
> >>   >2018-04-30T14:36:15Z DEBUG stderr=Job for 
> >> dirsrv(a)IPA-EXAMPLE-ORG.service 
> >> <mailto:[e-mail]>
> >> failed because the control process exited with error code. See 
> >> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
> >> <mailto:[e-mail]>" and "journalctl
> -xe" for 
> >> details.
> >>
> >>   >
> >>
> >>   >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", 
> >> line 504, in start_creation
> >>
> >>   >    run_step(full_msg, method)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", 
> >> line 494, in run_step
> >>
> >>   >    method()
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >> py
> >> ",
> >> line 95, in __start
> >>
> >>   >    srv.start(self.serverid, ldapi=True)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >> line 161, in start
> >>
> >>   >    instance_name, capture_output=capture_output, wait=wait)
> >>
> >>   >  File
> >> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", 
> >> line 294, in start
> >>
> >>   >    skip_output=not capture_output)
> >>
> >>   >  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
>
> >> line 542, in run
> >>
> >>   >    raise CalledProcessError(p.returncode, arg_string, 
> >> str(output))
> >>
> >>   >CalledProcessError: Command '/bin/systemctl start
> >>
> >>   &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >> <mailto:[e-mail]>' returned non-zero exit 
> >> status
> >> 1
> >>
> >>   >
> >>
> >>   >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command 
> >> '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
> >> <mailto:[e-mail]>' returned non-zero exit 
> >> status
> >> 1
> >>
> >>   >
> >>
> >>   >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>
> >>   >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
> >> slapd_bootstrap_config
> >> - The default password storage scheme could not be read or was not 
> >> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is 
> >> mandatory.
> >>
> >>   >
> >>
> >>   >Checking on internet I show that "dse.ldif" could be corrupted,
>
> >> so I changed with "dse.ldif.startOK" without any change and then I 
> >> changed with "dse.ldif.bak". The problem persist but the error has
> changed:
> >>
> >>   >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port 
> >> -
> >>
> >>   >Non-Secure Port Disabled
> >>
> >>   >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - 
> >> symload_report_error
> >> -
> >>
> >>   >Netscape Portable Runtime error -5975:
> >>
> >>   >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined
> >> symbol:
> >>
> >>   >replication_legacy_plugin_init
> >>
> >>   >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - 
> >> symload_report_error
> >> -
> >>
> >>   >Could not load symbol "replication_legacy_plugin_init" from
> >>
> >>   >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
>
> >> Legacy
> >>
> >>   >Replication Plugin
> >>
> >>   >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry 
> >> - Unable to load plugin "cn=Legacy Replication 
> >> Plugin,cn=plugins,cn=config"
> >>
> >>   >
> >>
> >>   >I saw a bug about this problem, but it is still opened:
> >>
> >> >https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >>
> >>   >
> >>
> >>   >Any idea how to fix the issue?
> >>
> >>   >
> >>
> >>   >If it is not possible to fix it, can I remove the replica from 
> >> IPA and install it again with the same name?
> >>
> >> A quick fix could be to remove an entry for cn=Legacy Replication 
> >> Plugin,cn=plugins,cn=config from 
> >> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>
> >> when dirsrv is down.
> >>
> >> --
> >>
> >> / Alexander Bokovoy
> >>
> >> Sr. Principal Software Engineer
> >>
> >> Security / Identity Management Engineering Red Hat Limited, Finland
> >>
> >>
> >>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >>
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> >
>
>

[thierry bordaz]

On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
>  hello,
>
>  Yesterday my ssh console closed the connection, so I had to start again the
> "ipa-server-upgrade", but the result is more or less the same:
>  # ipa-server-upgrade
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>     [1/10]: stopping directory server
>     [2/10]: saving configuration
>     [3/10]: disabling listeners
>     [4/10]: enabling DS global lock
>     [5/10]: starting directory server
>     [6/10]: updating schema
>     [7/10]: upgrading server
>
>  But now, the lines that are repeated on the access log are:
>  [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
>  [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0 tag=101 nentries=1
> etime=1.0000687740
>
>  And it is on the same state than the other:
>  # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
>  SASL/EXTERNAL authentication started
>  SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>  SASL SSF: 0
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config> with
> scope baseObject
>  # filter: (objectclass=*)
>  # requesting: ALL
>  #
>
>  # indextask_l_137445500911864330_4055, index, tasks, config
>  dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
>  objectClass: top
>  objectClass: extensibleObject
>  nsindexattribute: l
>  nsinstance: userRoot
>  cn: indextask_l_137445500911864330_4055
>  nstaskcurrentitem: 0
>  nstasktotalitems: 1
>
>  # search result
>  search: 3
>  result: 0 Success
>
>  # numResponses: 2
>  # numEntries: 1
>
>  The pstack of the proces: /usr/bin/python2 /usr/sbin/ipa-server-upgrade
>  # pstack 9805
>  #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
>  #1  0x00007fb8ff5e0000 in time_sleep () from
> /usr/lib64/python2.7/lib-dynload/timemodule.so
>  #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #21 0x00007fb90730d142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
>  #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
>  #23 0x00007fb90732773e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
>  #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
>  #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
>  #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
>  #27 0x000000000040066e in _start ()
>
>
>  And the pstack from the process: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-IPA-EXAMPLE-ORG
> -i /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
>  # pstack 15536
>  Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 49 (Thread 0x7f5bc02e8700 (LWP 15540)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 48 (Thread 0x7f5bbfae7700 (LWP 15541)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 47 (Thread 0x7f5bbf2e6700 (LWP 15542)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 46 (Thread 0x7f5bbeae5700 (LWP 15543)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 45 (Thread 0x7f5bbe2e4700 (LWP 15544)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 44 (Thread 0x7f5bbdae3700 (LWP 15545)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 43 (Thread 0x7f5bbd2e2700 (LWP 15546)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 42 (Thread 0x7f5bbcae1700 (LWP 15547)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 41 (Thread 0x7f5bbc2e0700 (LWP 15548)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 40 (Thread 0x7f5bbbadf700 (LWP 15549)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 39 (Thread 0x7f5bbb2de700 (LWP 15550)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 38 (Thread 0x7f5bbaadd700 (LWP 15551)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 37 (Thread 0x7f5bba2dc700 (LWP 15552)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 36 (Thread 0x7f5bb9adb700 (LWP 15553)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 35 (Thread 0x7f5bb92da700 (LWP 15554)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 34 (Thread 0x7f5bb8ad9700 (LWP 15555)):
>  #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
>  #1  0x00007f5bd1d9c803 in epoll_dispatch () at /lib64/libevent-2.0.so.5
>  #2  0x00007f5bd1d883ea in event_base_loop () at /lib64/libevent-2.0.so.5
>  #3  0x00007f5bd4d34eae in ns_event_fw_loop () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> /usr/lib64/dirsrv/libnunc-stans.so.0
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 33 (Thread 0x7f5bb2b84700 (LWP 15559)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 32 (Thread 0x7f5bb2383700 (LWP 15560)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 31 (Thread 0x7f5bb1b82700 (LWP 15561)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 30 (Thread 0x7f5bb1381700 (LWP 15562)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 29 (Thread 0x7f5bb0b80700 (LWP 15563)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 28 (Thread 0x7f5bb037f700 (LWP 15564)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> /usr/lib64/dirsrv/plugins/libipa_cldap.so
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 27 (Thread 0x7f5bd5132700 (LWP 15566)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3572033 in _cl5TrimMain () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 26 (Thread 0x7f5bafb7e700 (LWP 15567)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 25 (Thread 0x7f5baf37d700 (LWP 15568)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 24 (Thread 0x7f5bae759700 (LWP 15569)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 23 (Thread 0x7f5badf58700 (LWP 15570)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 22 (Thread 0x7f5bad757700 (LWP 15571)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 21 (Thread 0x7f5bacd4f700 (LWP 15572)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x0000564591e10b08 in housecleaning ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 20 (Thread 0x7f5bac54e700 (LWP 15573)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bd4a5ba63 in eq_loop () at /usr/lib64/dirsrv/libslapd.so.0
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 19 (Thread 0x7f5babd4d700 (LWP 15574)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 18 (Thread 0x7f5bab54c700 (LWP 15575)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 17 (Thread 0x7f5baad4b700 (LWP 15576)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 16 (Thread 0x7f5baa54a700 (LWP 15577)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 15 (Thread 0x7f5ba9d49700 (LWP 15578)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 14 (Thread 0x7f5ba9548700 (LWP 15579)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 13 (Thread 0x7f5ba8d47700 (LWP 15580)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 12 (Thread 0x7f5ba8546700 (LWP 15581)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 11 (Thread 0x7f5ba7d45700 (LWP 15582)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 10 (Thread 0x7f5ba7544700 (LWP 15583)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 9 (Thread 0x7f5ba6d43700 (LWP 15584)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
>  #2  0x0000564591e07049 in connection_read_operation ()
>  #3  0x0000564591e0765d in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 8 (Thread 0x7f5ba6542700 (LWP 15585)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 7 (Thread 0x7f5ba5d41700 (LWP 15586)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 6 (Thread 0x7f5ba5540700 (LWP 15587)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 5 (Thread 0x7f5ba4d3f700 (LWP 15588)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 4 (Thread 0x7f5ba453e700 (LWP 15589)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 
The indexing is hanging
because the Schema-compat cache initialization 
seems to sleep (while holding the backend lock in read).
This stack does not match my source, what is the version of slapi-nis rpm ?


regards
thierry
>  Thread 3 (Thread 0x7f5ba3900700 (LWP 15590)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd4aace0c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
>  #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
>  #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #8  0x00007f5bd4a8a0d1 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
>  #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 2 (Thread 0x7f5ba30ff700 (LWP 15606)):
>  #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #4  0x00007f5bd4ac5942 in task_index_thread () at /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 1 (Thread 0x7f5bd5135940 (LWP 15536)):
>  #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x0000564591e0d862 in slapd_daemon ()
>  #3  0x0000564591dfe86f in main ()
>
>  Thanks & Regards.
>  ______________________________
>
>  -----Original Message-----
>  From: thierry bordaz <tbordaz(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 9:31
>  To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> <solerm(a)unicc.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>
>  Hi,
>
>  During indexing task we should see in the task status the periodic progression of the
> indexing.
>  May be the indexing is hanging somewhere. When the problem occurs could you provide a
> pstack of the dirsrv server ?
>
>  best regards
>  thierry
>
>  On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> > SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >> Hello,
> >>
> >> This is the output of the command (seems that is not complete):
> >>
> >> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >> -b
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> >> =config
> >> -s base
> >> SASL/EXTERNAL authentication started
> >> SASL username:
> >> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >> SASL SSF: 0
> >> # extended LDIF
> >> #
> >> # LDAPv3
> >> # base
> >> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >> n=config>
> >> with scope baseObject
> >> # filter: (objectclass=*)
> >> # requesting: ALL
> >> #
> >>
> >> # indextask_description_137444551994158920_5958, index, tasks, config
> >> dn:
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> >> =conf
> >>    ig
> >> objectClass: top
> >> objectClass: extensibleObject
> >> nsindexattribute: description
> >> nsinstance: userRoot
> >> cn: indextask_description_137444551994158920_5958
> >> nstaskcurrentitem: 0
> >> nstasktotalitems: 1
> > IPA is looking for nstaskstatus and since that isn't there it is
> > looping forever.
> >
> > I've cc'd a couple of the 389-ds devs to see what they think about it.
> >
> > rob
> >
> >> # search result
> >> search: 3
> >> result: 0 Success
> >>
> >> # numResponses: 2
> >> # numEntries: 1
> >>
> >> Thanks & Regards.
> >> ______________________________
> >> -----Original Message-----
> >> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >> Sent: Tuesday, May 01, 2018 15:18
> >> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>> Hello,
> >>>
> >>> Thank you for your answer, now dirsrv can start, but after running
> >>> the "
> >>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>
> >>> # ipa-server-upgrade
> >>>
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>      [1/8]: saving configuration
> >>>
> >>>      [2/8]: disabling listeners
> >>>
> >>>      [3/8]: enabling DS global lock
> >>>
> >>>      [4/8]: starting directory server
> >>>
> >>>      [5/8]: updating schema
> >>>
> >>>      [6/8]: upgrading server
> >>>
> >>> The latest messages on the log is (there are no logs on the lasts 2h):
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG
> >>> ---------------------------------------------
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG dn:
> >>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>> database,cn=plugins,cn=config
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      top
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      description
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      false
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>> description
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>> n=
> >>> config
> >>>
> >>> If I check the ipa services, dirsrv is the only one running:
> >>>
> >>> # ipactl status
> >>>
> >>> Directory Service: RUNNING
> >>>
> >>> krb5kdc Service: STOPPED
> >>>
> >>> kadmin Service: STOPPED
> >>>
> >>> named Service: STOPPED
> >>>
> >>> httpd Service: STOPPED
> >>>
> >>> ipa-custodia Service: STOPPED
> >>>
> >>> ntpd Service: RUNNING
> >>>
> >>> pki-tomcatd Service: STOPPED
> >>>
> >>> smb Service: STOPPED
> >>>
> >>> winbind Service: STOPPED
> >>>
> >>> ipa-otpd Service: STOPPED
> >>>
> >>> ipa-dnskeysyncd Service: STOPPED
> >>>
> >>> ipa: INFO: The ipactl command was successful
> >>>
> >>> Should I stop the ipa-server-upgrade and start ipa services?
> >> It depends on how many entries you have. This is an indexing task and
> >> while 2 hours seems a bit excessive, it depends on how much work it
> >> has to do.
> >>
> >> The 389-ds access and/or error logs may provide details.
> >>
> >> rob
> >>
> >>> Thanks.
> >>>
> >>> -----Original Message-----
> >>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>> Sent: Tuesday, May 01, 2018 9:56
> >>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>
> >>>    >hello,
> >>>
> >>>    >
> >>>
> >>>    >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4
> >>> (RHEL 7.5). An hour later I tried to do the same with the unique
> >>> replica I have, but after update dirsrv is not starting.
> >>>
> >>>    >It says it is needed run "ipa-server-upgrade", but it also
> fails:
> >>>
> >>>    ># ipactl start
> >>>
> >>>    >Upgrade required: please run ipa-server-upgrade command Aborting
> >>> ipactl
> >>>
> >>>    >
> >>>
> >>>    ># ipa-server-upgrade
> >>>
> >>>    >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>    >  [1/8]: saving configuration
> >>>
> >>>    >  [2/8]: disabling listeners
> >>>
> >>>    >  [3/8]: enabling DS global lock
> >>>
> >>>    >  [4/8]: starting directory server
> >>>
> >>>    >  [error] CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >  [cleanup]: stopping directory server
> >>>
> >>>    >  [cleanup]: restoring configuration
> >>>
> >>>    >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> >>> run command ipa-server-upgrade manually.
> >>>
> >>>    >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1 The
> >>>
> >>>    >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
> >>> for more
> >>>
> >>>    >information
> >>>
> >>>    >
> >>>
> >>>    >On the log I can see:
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> failed because the control process exited with error code. See
> >>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>" and "journalctl
> -xe" for
> >>> details.
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 504, in start_creation
> >>>
> >>>    >    run_step(full_msg, method)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 494, in run_step
> >>>
> >>>    >    method()
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>> py
> >>> ",
> >>> line 95, in __start
> >>>
> >>>    >    srv.start(self.serverid, ldapi=True)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>> line 161, in start
> >>>
> >>>    >    instance_name, capture_output=capture_output, wait=wait)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>> line 294, in start
> >>>
> >>>    >    skip_output=not capture_output)
> >>>
> >>>    >  File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>> line 542, in run
> >>>
> >>>    >    raise CalledProcessError(p.returncode, arg_string,
> >>> str(output))
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> >>> '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>>
> >>>    >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>> slapd_bootstrap_config
> >>> - The default password storage scheme could not be read or was not
> >>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is
> >>> mandatory.
> >>>
> >>>    >
> >>>
> >>>    >Checking on internet I show that "dse.ldif" could be
> corrupted,
> >>> so I changed with "dse.ldif.startOK" without any change and then I
> >>> changed with "dse.ldif.bak". The problem persist but the error has
> changed:
> >>>
> >>>    >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port
> >>> -
> >>>
> >>>    >Non-Secure Port Disabled
> >>>
> >>>    >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Netscape Portable Runtime error -5975:
> >>>
> >>>    >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined
> >>> symbol:
> >>>
> >>>    >replication_legacy_plugin_init
> >>>
> >>>    >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Could not load symbol "replication_legacy_plugin_init" from
> >>>
> >>>    >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> plugin
> >>> Legacy
> >>>
> >>>    >Replication Plugin
> >>>
> >>>    >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry
> >>> - Unable to load plugin "cn=Legacy Replication
> >>> Plugin,cn=plugins,cn=config"
> >>>
> >>>    >
> >>>
> >>>    >I saw a bug about this problem, but it is still opened:
> >>>
> >>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >>>    >
> >>>
> >>>    >Any idea how to fix the issue?
> >>>
> >>>    >
> >>>
> >>>    >If it is not possible to fix it, can I remove the replica from
> >>> IPA and install it again with the same name?
> >>>
> >>> A quick fix could be to remove an entry for cn=Legacy Replication
> >>> Plugin,cn=plugins,cn=config from
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>
> >>> when dirsrv is down.
> >>>
> >>> --
> >>>
> >>> / Alexander Bokovoy
> >>>
> >>> Sr. Principal Software Engineer
> >>>
> >>> Security / Identity Management Engineering Red Hat Limited, Finland
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to
> >>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >>

[thierry bordaz]

me again ...

If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the 
hanging period it could also indicate the reason what Schema compat was 
waiting for (SSSD ?)

On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
>  hello,
>
>  Yesterday my ssh console closed the connection, so I had to start again the
> "ipa-server-upgrade", but the result is more or less the same:
>  # ipa-server-upgrade
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>     [1/10]: stopping directory server
>     [2/10]: saving configuration
>     [3/10]: disabling listeners
>     [4/10]: enabling DS global lock
>     [5/10]: starting directory server
>     [6/10]: updating schema
>     [7/10]: upgrading server
>
>  But now, the lines that are repeated on the access log are:
>  [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
>  [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0 tag=101 nentries=1
> etime=1.0000687740
>
>  And it is on the same state than the other:
>  # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
>  SASL/EXTERNAL authentication started
>  SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>  SASL SSF: 0
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config> with
> scope baseObject
>  # filter: (objectclass=*)
>  # requesting: ALL
>  #
>
>  # indextask_l_137445500911864330_4055, index, tasks, config
>  dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
>  objectClass: top
>  objectClass: extensibleObject
>  nsindexattribute: l
>  nsinstance: userRoot
>  cn: indextask_l_137445500911864330_4055
>  nstaskcurrentitem: 0
>  nstasktotalitems: 1
>
>  # search result
>  search: 3
>  result: 0 Success
>
>  # numResponses: 2
>  # numEntries: 1
>
>  The pstack of the proces: /usr/bin/python2 /usr/sbin/ipa-server-upgrade
>  # pstack 9805
>  #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
>  #1  0x00007fb8ff5e0000 in time_sleep () from
> /usr/lib64/python2.7/lib-dynload/timemodule.so
>  #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
>  #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
>  #21 0x00007fb90730d142 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
>  #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
>  #23 0x00007fb90732773e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
>  #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
>  #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
>  #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
>  #27 0x000000000040066e in _start ()
>
>
>  And the pstack from the process: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-IPA-EXAMPLE-ORG
> -i /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
>  # pstack 15536
>  Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 49 (Thread 0x7f5bc02e8700 (LWP 15540)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 48 (Thread 0x7f5bbfae7700 (LWP 15541)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 47 (Thread 0x7f5bbf2e6700 (LWP 15542)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 46 (Thread 0x7f5bbeae5700 (LWP 15543)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 45 (Thread 0x7f5bbe2e4700 (LWP 15544)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 44 (Thread 0x7f5bbdae3700 (LWP 15545)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 43 (Thread 0x7f5bbd2e2700 (LWP 15546)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 42 (Thread 0x7f5bbcae1700 (LWP 15547)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 41 (Thread 0x7f5bbc2e0700 (LWP 15548)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 40 (Thread 0x7f5bbbadf700 (LWP 15549)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 39 (Thread 0x7f5bbb2de700 (LWP 15550)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 38 (Thread 0x7f5bbaadd700 (LWP 15551)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 37 (Thread 0x7f5bba2dc700 (LWP 15552)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 36 (Thread 0x7f5bb9adb700 (LWP 15553)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 35 (Thread 0x7f5bb92da700 (LWP 15554)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 34 (Thread 0x7f5bb8ad9700 (LWP 15555)):
>  #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
>  #1  0x00007f5bd1d9c803 in epoll_dispatch () at /lib64/libevent-2.0.so.5
>  #2  0x00007f5bd1d883ea in event_base_loop () at /lib64/libevent-2.0.so.5
>  #3  0x00007f5bd4d34eae in ns_event_fw_loop () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> /usr/lib64/dirsrv/libnunc-stans.so.0
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 33 (Thread 0x7f5bb2b84700 (LWP 15559)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 32 (Thread 0x7f5bb2383700 (LWP 15560)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 31 (Thread 0x7f5bb1b82700 (LWP 15561)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 30 (Thread 0x7f5bb1381700 (LWP 15562)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 29 (Thread 0x7f5bb0b80700 (LWP 15563)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 28 (Thread 0x7f5bb037f700 (LWP 15564)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> /usr/lib64/dirsrv/plugins/libipa_cldap.so
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 27 (Thread 0x7f5bd5132700 (LWP 15566)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3572033 in _cl5TrimMain () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 26 (Thread 0x7f5bafb7e700 (LWP 15567)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 25 (Thread 0x7f5baf37d700 (LWP 15568)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 24 (Thread 0x7f5bae759700 (LWP 15569)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 23 (Thread 0x7f5badf58700 (LWP 15570)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 22 (Thread 0x7f5bad757700 (LWP 15571)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 21 (Thread 0x7f5bacd4f700 (LWP 15572)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x0000564591e10b08 in housecleaning ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 20 (Thread 0x7f5bac54e700 (LWP 15573)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bd4a5ba63 in eq_loop () at /usr/lib64/dirsrv/libslapd.so.0
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 19 (Thread 0x7f5babd4d700 (LWP 15574)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 18 (Thread 0x7f5bab54c700 (LWP 15575)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 17 (Thread 0x7f5baad4b700 (LWP 15576)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 16 (Thread 0x7f5baa54a700 (LWP 15577)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 15 (Thread 0x7f5ba9d49700 (LWP 15578)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 14 (Thread 0x7f5ba9548700 (LWP 15579)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 13 (Thread 0x7f5ba8d47700 (LWP 15580)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 12 (Thread 0x7f5ba8546700 (LWP 15581)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 11 (Thread 0x7f5ba7d45700 (LWP 15582)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 10 (Thread 0x7f5ba7544700 (LWP 15583)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 9 (Thread 0x7f5ba6d43700 (LWP 15584)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
>  #2  0x0000564591e07049 in connection_read_operation ()
>  #3  0x0000564591e0765d in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 8 (Thread 0x7f5ba6542700 (LWP 15585)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 7 (Thread 0x7f5ba5d41700 (LWP 15586)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 6 (Thread 0x7f5ba5540700 (LWP 15587)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 5 (Thread 0x7f5ba4d3f700 (LWP 15588)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 4 (Thread 0x7f5ba453e700 (LWP 15589)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 3 (Thread 0x7f5ba3900700 (LWP 15590)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd4aace0c in send_ldap_search_entry () at /usr/lib64/dirsrv/libslapd.so.0
>  #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at /usr/lib64/dirsrv/libslapd.so.0
>  #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #8  0x00007f5bd4a8a0d1 in op_shared_search () at /usr/lib64/dirsrv/libslapd.so.0
>  #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 2 (Thread 0x7f5ba30ff700 (LWP 15606)):
>  #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #4  0x00007f5bd4ac5942 in task_index_thread () at /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6
>  Thread 1 (Thread 0x7f5bd5135940 (LWP 15536)):
>  #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x0000564591e0d862 in slapd_daemon ()
>  #3  0x0000564591dfe86f in main ()
>
>  Thanks & Regards.
>  ______________________________
>
>  -----Original Message-----
>  From: thierry bordaz <tbordaz(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 9:31
>  To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> <solerm(a)unicc.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>
>  Hi,
>
>  During indexing task we should see in the task status the periodic progression of the
> indexing.
>  May be the indexing is hanging somewhere. When the problem occurs could you provide a
> pstack of the dirsrv server ?
>
>  best regards
>  thierry
>
>  On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> > SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >> Hello,
> >>
> >> This is the output of the command (seems that is not complete):
> >>
> >> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >> -b
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> >> =config
> >> -s base
> >> SASL/EXTERNAL authentication started
> >> SASL username:
> >> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >> SASL SSF: 0
> >> # extended LDIF
> >> #
> >> # LDAPv3
> >> # base
> >> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >> n=config>
> >> with scope baseObject
> >> # filter: (objectclass=*)
> >> # requesting: ALL
> >> #
> >>
> >> # indextask_description_137444551994158920_5958, index, tasks, config
> >> dn:
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,cn
> >> =conf
> >>    ig
> >> objectClass: top
> >> objectClass: extensibleObject
> >> nsindexattribute: description
> >> nsinstance: userRoot
> >> cn: indextask_description_137444551994158920_5958
> >> nstaskcurrentitem: 0
> >> nstasktotalitems: 1
> > IPA is looking for nstaskstatus and since that isn't there it is
> > looping forever.
> >
> > I've cc'd a couple of the 389-ds devs to see what they think about it.
> >
> > rob
> >
> >> # search result
> >> search: 3
> >> result: 0 Success
> >>
> >> # numResponses: 2
> >> # numEntries: 1
> >>
> >> Thanks & Regards.
> >> ______________________________
> >> -----Original Message-----
> >> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >> Sent: Tuesday, May 01, 2018 15:18
> >> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>> Hello,
> >>>
> >>> Thank you for your answer, now dirsrv can start, but after running
> >>> the "
> >>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>
> >>> # ipa-server-upgrade
> >>>
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>      [1/8]: saving configuration
> >>>
> >>>      [2/8]: disabling listeners
> >>>
> >>>      [3/8]: enabling DS global lock
> >>>
> >>>      [4/8]: starting directory server
> >>>
> >>>      [5/8]: updating schema
> >>>
> >>>      [6/8]: upgrading server
> >>>
> >>> The latest messages on the log is (there are no logs on the lasts 2h):
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG
> >>> ---------------------------------------------
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG dn:
> >>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>> database,cn=plugins,cn=config
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      top
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      description
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      false
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>> description
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>> n=
> >>> config
> >>>
> >>> If I check the ipa services, dirsrv is the only one running:
> >>>
> >>> # ipactl status
> >>>
> >>> Directory Service: RUNNING
> >>>
> >>> krb5kdc Service: STOPPED
> >>>
> >>> kadmin Service: STOPPED
> >>>
> >>> named Service: STOPPED
> >>>
> >>> httpd Service: STOPPED
> >>>
> >>> ipa-custodia Service: STOPPED
> >>>
> >>> ntpd Service: RUNNING
> >>>
> >>> pki-tomcatd Service: STOPPED
> >>>
> >>> smb Service: STOPPED
> >>>
> >>> winbind Service: STOPPED
> >>>
> >>> ipa-otpd Service: STOPPED
> >>>
> >>> ipa-dnskeysyncd Service: STOPPED
> >>>
> >>> ipa: INFO: The ipactl command was successful
> >>>
> >>> Should I stop the ipa-server-upgrade and start ipa services?
> >> It depends on how many entries you have. This is an indexing task and
> >> while 2 hours seems a bit excessive, it depends on how much work it
> >> has to do.
> >>
> >> The 389-ds access and/or error logs may provide details.
> >>
> >> rob
> >>
> >>> Thanks.
> >>>
> >>> -----Original Message-----
> >>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>> Sent: Tuesday, May 01, 2018 9:56
> >>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>
> >>>    >hello,
> >>>
> >>>    >
> >>>
> >>>    >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4
> >>> (RHEL 7.5). An hour later I tried to do the same with the unique
> >>> replica I have, but after update dirsrv is not starting.
> >>>
> >>>    >It says it is needed run "ipa-server-upgrade", but it also
> fails:
> >>>
> >>>    ># ipactl start
> >>>
> >>>    >Upgrade required: please run ipa-server-upgrade command Aborting
> >>> ipactl
> >>>
> >>>    >
> >>>
> >>>    ># ipa-server-upgrade
> >>>
> >>>    >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>    >  [1/8]: saving configuration
> >>>
> >>>    >  [2/8]: disabling listeners
> >>>
> >>>    >  [3/8]: enabling DS global lock
> >>>
> >>>    >  [4/8]: starting directory server
> >>>
> >>>    >  [error] CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >  [cleanup]: stopping directory server
> >>>
> >>>    >  [cleanup]: restoring configuration
> >>>
> >>>    >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> >>> run command ipa-server-upgrade manually.
> >>>
> >>>    >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1 The
> >>>
> >>>    >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
> >>> for more
> >>>
> >>>    >information
> >>>
> >>>    >
> >>>
> >>>    >On the log I can see:
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> failed because the control process exited with error code. See
> >>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>" and "journalctl
> -xe" for
> >>> details.
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 504, in start_creation
> >>>
> >>>    >    run_step(full_msg, method)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 494, in run_step
> >>>
> >>>    >    method()
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>> py
> >>> ",
> >>> line 95, in __start
> >>>
> >>>    >    srv.start(self.serverid, ldapi=True)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>> line 161, in start
> >>>
> >>>    >    instance_name, capture_output=capture_output, wait=wait)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>> line 294, in start
> >>>
> >>>    >    skip_output=not capture_output)
> >>>
> >>>    >  File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>> line 542, in run
> >>>
> >>>    >    raise CalledProcessError(p.returncode, arg_string,
> >>> str(output))
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
> >>> '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>>
> >>>    >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>> slapd_bootstrap_config
> >>> - The default password storage scheme could not be read or was not
> >>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is
> >>> mandatory.
> >>>
> >>>    >
> >>>
> >>>    >Checking on internet I show that "dse.ldif" could be
> corrupted,
> >>> so I changed with "dse.ldif.startOK" without any change and then I
> >>> changed with "dse.ldif.bak". The problem persist but the error has
> changed:
> >>>
> >>>    >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port
> >>> -
> >>>
> >>>    >Non-Secure Port Disabled
> >>>
> >>>    >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Netscape Portable Runtime error -5975:
> >>>
> >>>    >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined
> >>> symbol:
> >>>
> >>>    >replication_legacy_plugin_init
> >>>
> >>>    >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Could not load symbol "replication_legacy_plugin_init" from
> >>>
> >>>    >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> plugin
> >>> Legacy
> >>>
> >>>    >Replication Plugin
> >>>
> >>>    >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry
> >>> - Unable to load plugin "cn=Legacy Replication
> >>> Plugin,cn=plugins,cn=config"
> >>>
> >>>    >
> >>>
> >>>    >I saw a bug about this problem, but it is still opened:
> >>>
> >>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
> >>>    >
> >>>
> >>>    >Any idea how to fix the issue?
> >>>
> >>>    >
> >>>
> >>>    >If it is not possible to fix it, can I remove the replica from
> >>> IPA and install it again with the same name?
> >>>
> >>> A quick fix could be to remove an entry for cn=Legacy Replication
> >>> Plugin,cn=plugins,cn=config from
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>
> >>> when dirsrv is down.
> >>>
> >>> --
> >>>
> >>> / Alexander Bokovoy
> >>>
> >>> Sr. Principal Software Engineer
> >>>
> >>> Security / Identity Management Engineering Red Hat Limited, Finland
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to
> >>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >>

[SOLER SANGUESA Miguel]

Hello Thierry,

The version is: slapi-nis-0.56.0-8.el7.x86_64

And the errors are:
[02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin - group
"XXXXXX(a)ipa.example.org" does not exist because SSSD is offline.
[02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin - waiting for SSSD to
become online...

SSSD service is up, but as the ipa services are not (all) running, the ipa domain is down,
logs when started sssd shows:
sssd[be[ipa.example.org]][823]: Backend is offline

and logs sssd_nss.log & sssd_sudo.log have hundreds of:
 [sss_dp_get_reply] (0x0010): The Data Provider returned an error
[org.freedesktop.sssd.Error.DataProvider.Offline]

Thanks & Regards.

-----Original Message-----
From: thierry bordaz <tbordaz(a)redhat.com&gt; 
Sent: Thursday, May 03, 2018 11:25
To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden
<rcritten(a)redhat.com>; FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org&gt;
Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

me again ...

If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the hanging period
it could also indicate the reason what Schema compat was waiting for (SSSD ?)

On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
>  hello,
>
>  Yesterday my ssh console closed the connection, so I had to start again the
> "ipa-server-upgrade", but the result is more or less the same:
>  # ipa-server-upgrade
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>     [1/10]: stopping directory server
>     [2/10]: saving configuration
>     [3/10]: disabling listeners
>     [4/10]: enabling DS global lock
>     [5/10]: starting directory server
>     [6/10]: updating schema
>     [7/10]: upgrading server
>
>  But now, the lines that are repeated on the access log are:
>  [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
>  [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0 
>  tag=101 nentries=1 etime=1.0000687740
>
>  And it is on the same state than the other:
>  # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b 
>  cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s 
>  base SASL/EXTERNAL authentication started SASL username: 
>  gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>  SASL SSF: 0
>  # extended LDIF
>  #
>  # LDAPv3
>  # base 
>  <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config> 
>  with scope baseObject # filter: (objectclass=*) # requesting: ALL #
>
>  # indextask_l_137445500911864330_4055, index, tasks, config
>  dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
>  objectClass: top
>  objectClass: extensibleObject
>  nsindexattribute: l
>  nsinstance: userRoot
>  cn: indextask_l_137445500911864330_4055
>  nstaskcurrentitem: 0
>  nstasktotalitems: 1
>
>  # search result
>  search: 3
>  result: 0 Success
>
>  # numResponses: 2
>  # numEntries: 1
>
>  The pstack of the proces: /usr/bin/python2 
>  /usr/sbin/ipa-server-upgrade # pstack 9805
>  #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
>  #1  0x00007fb8ff5e0000 in time_sleep () from 
>  /usr/lib64/python2.7/lib-dynload/timemodule.so
>  #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from 
>  /lib64/libpython2.7.so.1.0
>  #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from 
>  /lib64/libpython2.7.so.1.0
>  #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from 
>  /lib64/libpython2.7.so.1.0
>  #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from 
>  /lib64/libpython2.7.so.1.0
>  #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from 
>  /lib64/libpython2.7.so.1.0
>  #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from 
>  /lib64/libpython2.7.so.1.0
>  #21 0x00007fb90730d142 in PyEval_EvalCode () from 
>  /lib64/libpython2.7.so.1.0
>  #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
>  #23 0x00007fb90732773e in PyRun_FileExFlags () from 
>  /lib64/libpython2.7.so.1.0
>  #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from 
>  /lib64/libpython2.7.so.1.0
>  #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
>  #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
>  #27 0x000000000040066e in _start ()
>
>
>  And the pstack from the process: /usr/sbin/ns-slapd -D 
>  /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i 
>  /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
>  # pstack 15536
>  Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49 
>  (Thread 0x7f5bc02e8700 (LWP 15540)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48 
>  (Thread 0x7f5bbfae7700 (LWP 15541)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47 
>  (Thread 0x7f5bbf2e6700 (LWP 15542)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46 
>  (Thread 0x7f5bbeae5700 (LWP 15543)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45 
>  (Thread 0x7f5bbe2e4700 (LWP 15544)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44 
>  (Thread 0x7f5bbdae3700 (LWP 15545)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43 
>  (Thread 0x7f5bbd2e2700 (LWP 15546)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42 
>  (Thread 0x7f5bbcae1700 (LWP 15547)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41 
>  (Thread 0x7f5bbc2e0700 (LWP 15548)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40 
>  (Thread 0x7f5bbbadf700 (LWP 15549)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39 
>  (Thread 0x7f5bbb2de700 (LWP 15550)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38 
>  (Thread 0x7f5bbaadd700 (LWP 15551)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37 
>  (Thread 0x7f5bba2dc700 (LWP 15552)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36 
>  (Thread 0x7f5bb9adb700 (LWP 15553)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35 
>  (Thread 0x7f5bb92da700 (LWP 15554)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4d33e63 in worker_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34 
>  (Thread 0x7f5bb8ad9700 (LWP 15555)):
>  #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
>  #1  0x00007f5bd1d9c803 in epoll_dispatch () at 
>  /lib64/libevent-2.0.so.5
>  #2  0x00007f5bd1d883ea in event_base_loop () at 
>  /lib64/libevent-2.0.so.5
>  #3  0x00007f5bd4d34eae in ns_event_fw_loop () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33 
>  (Thread 0x7f5bb2b84700 (LWP 15559)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e41e77 in deadlock_threadmain () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32 
>  (Thread 0x7f5bb2383700 (LWP 15560)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31 
>  (Thread 0x7f5bb1b82700 (LWP 15561)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e420c7 in trickle_threadmain () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30 
>  (Thread 0x7f5bb1381700 (LWP 15562)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e3ce47 in perf_threadmain () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29 
>  (Thread 0x7f5bb0b80700 (LWP 15563)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc920213e in cos_cache_wait_on_change () at 
>  /usr/lib64/dirsrv/plugins/libcos-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28 
>  (Thread 0x7f5bb037f700 (LWP 15564)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at 
>  /usr/lib64/dirsrv/plugins/libipa_cldap.so
>  #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27 
>  (Thread 0x7f5bd5132700 (LWP 15566)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3572033 in _cl5TrimMain () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26 
>  (Thread 0x7f5bafb7e700 (LWP 15567)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25 
>  (Thread 0x7f5baf37d700 (LWP 15568)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bc3587fa4 in protocol_sleep () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #4  0x00007f5bc358b75a in repl5_inc_run () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #5  0x00007f5bc358fbec in prot_thread_main () at 
>  /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>  #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24 
>  (Thread 0x7f5bae759700 (LWP 15569)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
>  /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23 
>  (Thread 0x7f5badf58700 (LWP 15570)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
>  /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22 
>  (Thread 0x7f5bad757700 (LWP 15571)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
>  /usr/lib64/dirsrv/plugins/libroles-plugin.so
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21 
>  (Thread 0x7f5bacd4f700 (LWP 15572)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x0000564591e10b08 in housecleaning ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20 
>  (Thread 0x7f5bac54e700 (LWP 15573)):
>  #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>  #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>  #3  0x00007f5bd4a5ba63 in eq_loop () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19 
>  (Thread 0x7f5babd4d700 (LWP 15574)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18 
>  (Thread 0x7f5bab54c700 (LWP 15575)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17 
>  (Thread 0x7f5baad4b700 (LWP 15576)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16 
>  (Thread 0x7f5baa54a700 (LWP 15577)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15 
>  (Thread 0x7f5ba9d49700 (LWP 15578)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14 
>  (Thread 0x7f5ba9548700 (LWP 15579)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13 
>  (Thread 0x7f5ba8d47700 (LWP 15580)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12 
>  (Thread 0x7f5ba8546700 (LWP 15581)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11 
>  (Thread 0x7f5ba7d45700 (LWP 15582)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10 
>  (Thread 0x7f5ba7544700 (LWP 15583)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9 
>  (Thread 0x7f5ba6d43700 (LWP 15584)):
>  #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>  #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
>  #2  0x0000564591e07049 in connection_read_operation ()
>  #3  0x0000564591e0765d in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8 
>  (Thread 0x7f5ba6542700 (LWP 15585)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7 
>  (Thread 0x7f5ba5d41700 (LWP 15586)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6 
>  (Thread 0x7f5ba5540700 (LWP 15587)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5 
>  (Thread 0x7f5ba4d3f700 (LWP 15588)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4 
>  (Thread 0x7f5ba453e700 (LWP 15589)):
>  #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>  #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work 
>  ()
>  #3  0x0000564591e075d7 in connection_threadmain ()
>  #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3 
>  (Thread 0x7f5ba3900700 (LWP 15590)):
>  #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>  #1  0x00007f5bd4acc599 in DS_Sleep () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd4aace0c in send_ldap_search_entry () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #8  0x00007f5bd4a8a0d1 in op_shared_search () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #13 0x00007f5bc1eba597 in wrap_pthread_starter () at 
>  /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>  #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2 
>  (Thread 0x7f5ba30ff700 (LWP 15606)):
>  #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at 
>  /lib64/libpthread.so.0
>  #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #4  0x00007f5bd4ac5942 in task_index_thread () at 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>  #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>  #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1 
>  (Thread 0x7f5bd5135940 (LWP 15536)):
>  #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
>  #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at 
>  /usr/lib64/dirsrv/libnunc-stans.so.0
>  #2  0x0000564591e0d862 in slapd_daemon ()
>  #3  0x0000564591dfe86f in main ()
>
>  Thanks & Regards.
>  ______________________________
>
>  -----Original Message-----
>  From: thierry bordaz <tbordaz(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 9:31
>  To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list 
>  <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel 
>  <solerm(a)unicc.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 
>  4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>
>  Hi,
>
>  During indexing task we should see in the task status the periodic progression of the
> indexing.
>  May be the indexing is hanging somewhere. When the problem occurs could you provide a
> pstack of the dirsrv server ?
>
>  best regards
>  thierry
>
>  On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> > SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >> Hello,
> >>
> >> This is the output of the command (seems that is not complete):
> >>
> >> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >> -b
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >> n
> >> =config
> >> -s base
> >> SASL/EXTERNAL authentication started SASL username:
> >> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >> SASL SSF: 0
> >> # extended LDIF
> >> #
> >> # LDAPv3
> >> # base
> >> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >> c
> >> n=config>
> >> with scope baseObject
> >> # filter: (objectclass=*)
> >> # requesting: ALL
> >> #
> >>
> >> # indextask_description_137444551994158920_5958, index, tasks, 
> >> config
> >> dn:
> >> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >> n
> >> =conf
> >>    ig
> >> objectClass: top
> >> objectClass: extensibleObject
> >> nsindexattribute: description
> >> nsinstance: userRoot
> >> cn: indextask_description_137444551994158920_5958
> >> nstaskcurrentitem: 0
> >> nstasktotalitems: 1
> > IPA is looking for nstaskstatus and since that isn't there it is 
> > looping forever.
> >
> > I've cc'd a couple of the 389-ds devs to see what they think about it.
> >
> > rob
> >
> >> # search result
> >> search: 3
> >> result: 0 Success
> >>
> >> # numResponses: 2
> >> # numEntries: 1
> >>
> >> Thanks & Regards.
> >> ______________________________
> >> -----Original Message-----
> >> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >> Sent: Tuesday, May 01, 2018 15:18
> >> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating 
> >> from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>> Hello,
> >>>
> >>> Thank you for your answer, now dirsrv can start, but after running 
> >>> the "
> >>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>
> >>> # ipa-server-upgrade
> >>>
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>      [1/8]: saving configuration
> >>>
> >>>      [2/8]: disabling listeners
> >>>
> >>>      [3/8]: enabling DS global lock
> >>>
> >>>      [4/8]: starting directory server
> >>>
> >>>      [5/8]: updating schema
> >>>
> >>>      [6/8]: upgrading server
> >>>
> >>> The latest messages on the log is (there are no logs on the lasts 2h):
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG
> >>> ---------------------------------------------
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG dn:
> >>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>> database,cn=plugins,cn=config
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      top
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      description
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>
> >>> 2018-05-01T08:13:14Z DEBUG      false
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>> description
> >>>
> >>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>> c
> >>> n=
> >>> config
> >>>
> >>> If I check the ipa services, dirsrv is the only one running:
> >>>
> >>> # ipactl status
> >>>
> >>> Directory Service: RUNNING
> >>>
> >>> krb5kdc Service: STOPPED
> >>>
> >>> kadmin Service: STOPPED
> >>>
> >>> named Service: STOPPED
> >>>
> >>> httpd Service: STOPPED
> >>>
> >>> ipa-custodia Service: STOPPED
> >>>
> >>> ntpd Service: RUNNING
> >>>
> >>> pki-tomcatd Service: STOPPED
> >>>
> >>> smb Service: STOPPED
> >>>
> >>> winbind Service: STOPPED
> >>>
> >>> ipa-otpd Service: STOPPED
> >>>
> >>> ipa-dnskeysyncd Service: STOPPED
> >>>
> >>> ipa: INFO: The ipactl command was successful
> >>>
> >>> Should I stop the ipa-server-upgrade and start ipa services?
> >> It depends on how many entries you have. This is an indexing task 
> >> and while 2 hours seems a bit excessive, it depends on how much work 
> >> it has to do.
> >>
> >> The 389-ds access and/or error logs may provide details.
> >>
> >> rob
> >>
> >>> Thanks.
> >>>
> >>> -----Original Message-----
> >>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>> Sent: Tuesday, May 01, 2018 9:56
> >>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>
> >>>    >hello,
> >>>
> >>>    >
> >>>
> >>>    >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 
> >>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the 
> >>> unique replica I have, but after update dirsrv is not starting.
> >>>
> >>>    >It says it is needed run "ipa-server-upgrade", but it also
> fails:
> >>>
> >>>    ># ipactl start
> >>>
> >>>    >Upgrade required: please run ipa-server-upgrade command 
> >>> Aborting ipactl
> >>>
> >>>    >
> >>>
> >>>    ># ipa-server-upgrade
> >>>
> >>>    >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>
> >>>    >  [1/8]: saving configuration
> >>>
> >>>    >  [2/8]: disabling listeners
> >>>
> >>>    >  [3/8]: enabling DS global lock
> >>>
> >>>    >  [4/8]: starting directory server
> >>>
> >>>    >  [error] CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit 
> >>> status
> >>> 1
> >>>
> >>>    >  [cleanup]: stopping directory server
> >>>
> >>>    >  [cleanup]: restoring configuration
> >>>
> >>>    >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and 
> >>> run command ipa-server-upgrade manually.
> >>>
> >>>    >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit 
> >>> status
> >>> 1 The
> >>>
> >>>    >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log 
> >>> for more
> >>>
> >>>    >information
> >>>
> >>>    >
> >>>
> >>>    >On the log I can see:
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed ...
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>
> >>> 2018-04-30T14:36:15Z DEBUG Process
> >>>
> >>>    >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG stderr=Job for 
> >>> dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>> <mailto:[e-mail]>
> >>> failed because the control process exited with error code. See 
> >>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>> <mailto:[e-mail]>" and "journalctl
> -xe" for 
> >>> details.
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 504, in start_creation
> >>>
> >>>    >    run_step(full_msg, method)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>> line 494, in run_step
> >>>
> >>>    >    method()
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>> py
> >>> ",
> >>> line 95, in __start
> >>>
> >>>    >    srv.start(self.serverid, ldapi=True)
> >>>
> >>>    >  File
> >>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>> line 161, in start
> >>>
> >>>    >    instance_name, capture_output=capture_output, wait=wait)
> >>>
> >>>    >  File
> >>> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>> line 294, in start
> >>>
> >>>    >    skip_output=not capture_output)
> >>>
> >>>    >  File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>> line 542, in run
> >>>
> >>>    >    raise CalledProcessError(p.returncode, arg_string,
> >>> str(output))
> >>>
> >>>    >CalledProcessError: Command '/bin/systemctl start
> >>>
> >>>    &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>> <mailto:[e-mail]>' returned non-zero exit 
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: 
> >>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>> <mailto:[e-mail]>' returned non-zero exit 
> >>> status
> >>> 1
> >>>
> >>>    >
> >>>
> >>>    >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>>
> >>>    >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
> >>> slapd_bootstrap_config
> >>> - The default password storage scheme could not be read or was not 
> >>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is 
> >>> mandatory.
> >>>
> >>>    >
> >>>
> >>>    >Checking on internet I show that "dse.ldif" could be
> corrupted, 
> >>> so I changed with "dse.ldif.startOK" without any change and then I
>
> >>> changed with "dse.ldif.bak". The problem persist but the error has
> changed:
> >>>
> >>>    >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - 
> >>> config_set_port
> >>> -
> >>>
> >>>    >Non-Secure Port Disabled
> >>>
> >>>    >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - 
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Netscape Portable Runtime error -5975:
> >>>
> >>>    >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined
> >>> symbol:
> >>>
> >>>    >replication_legacy_plugin_init
> >>>
> >>>    >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - 
> >>> symload_report_error
> >>> -
> >>>
> >>>    >Could not load symbol "replication_legacy_plugin_init"
> from
> >>>
> >>>    >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> plugin 
> >>> Legacy
> >>>
> >>>    >Replication Plugin
> >>>
> >>>    >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - 
> >>> load_plugin_entry
> >>> - Unable to load plugin "cn=Legacy Replication 
> >>> Plugin,cn=plugins,cn=config"
> >>>
> >>>    >
> >>>
> >>>    >I saw a bug about this problem, but it is still opened:
> >>>
> >>>> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
> >>>> 2
> >>>    >
> >>>
> >>>    >Any idea how to fix the issue?
> >>>
> >>>    >
> >>>
> >>>    >If it is not possible to fix it, can I remove the replica from 
> >>> IPA and install it again with the same name?
> >>>
> >>> A quick fix could be to remove an entry for cn=Legacy Replication 
> >>> Plugin,cn=plugins,cn=config from 
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>
> >>> when dirsrv is down.
> >>>
> >>> --
> >>>
> >>> / Alexander Bokovoy
> >>>
> >>> Sr. Principal Software Engineer
> >>>
> >>> Security / Identity Management Engineering Red Hat Limited, Finland
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to
> >>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >>
>

[thierry bordaz]

Hi Soler,

Thanks for the information.
So indexing is hanging because SC cache_init is running, the SC 
cache_init is hanging because SSSD is not started, SSSD is not started 
possibly because indexing prevents to get read access to the backend 
("Backend is offline" TBC).

An option would be to disable SC during upgrade phase ? Rob ?

regards
thierry

On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
>  Hello Thierry,
>
>  The version is: slapi-nis-0.56.0-8.el7.x86_64
>
>  And the errors are:
>  [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin - group
> "XXXXXX(a)ipa.example.org" does not exist because SSSD is offline.
>  [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin - waiting for SSSD to
> become online...
>
>  SSSD service is up, but as the ipa services are not (all) running, the ipa domain is
> down, logs when started sssd shows:
>  sssd[be[ipa.example.org]][823]: Backend is offline
>
>  and logs sssd_nss.log & sssd_sudo.log have hundreds of:
>    [sss_dp_get_reply] (0x0010): The Data Provider returned an error
> [org.freedesktop.sssd.Error.DataProvider.Offline]
>
>  Thanks & Regards.
>
>  -----Original Message-----
>  From: thierry bordaz <tbordaz(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 11:25
>  To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden
> <rcritten(a)redhat.com>; FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>
>  me again ...
>
>  If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the hanging
> period it could also indicate the reason what Schema compat was waiting for (SSSD ?)
>
>  On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> > hello,
> >
> > Yesterday my ssh console closed the connection, so I had to start again the
> "ipa-server-upgrade", but the result is more or less the same:
> > # ipa-server-upgrade
> > Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >     [1/10]: stopping directory server
> >     [2/10]: saving configuration
> >     [3/10]: disabling listeners
> >     [4/10]: enabling DS global lock
> >     [5/10]: starting directory server
> >     [6/10]: updating schema
> >     [7/10]: upgrading server
> >
> > But now, the lines that are repeated on the access log are:
> > [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> scope=0 filter="(objectClass=*)" attrs="nstaskstatus nstaskexitcode"
> > [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> > tag=101 nentries=1 etime=1.0000687740
> >
> > And it is on the same state than the other:
> > # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> > cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s
> > base SASL/EXTERNAL authentication started SASL username:
> > gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> > SASL SSF: 0
> > # extended LDIF
> > #
> > # LDAPv3
> > # base
> > <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> > with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >
> > # indextask_l_137445500911864330_4055, index, tasks, config
> > dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> > objectClass: top
> > objectClass: extensibleObject
> > nsindexattribute: l
> > nsinstance: userRoot
> > cn: indextask_l_137445500911864330_4055
> > nstaskcurrentitem: 0
> > nstasktotalitems: 1
> >
> > # search result
> > search: 3
> > result: 0 Success
> >
> > # numResponses: 2
> > # numEntries: 1
> >
> > The pstack of the proces: /usr/bin/python2
> > /usr/sbin/ipa-server-upgrade # pstack 9805
> > #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> > #1  0x00007fb8ff5e0000 in time_sleep () from
> > /usr/lib64/python2.7/lib-dynload/timemodule.so
> > #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> > /lib64/libpython2.7.so.1.0
> > #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> > /lib64/libpython2.7.so.1.0
> > #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> > /lib64/libpython2.7.so.1.0
> > #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> > /lib64/libpython2.7.so.1.0
> > #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> > /lib64/libpython2.7.so.1.0
> > #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> > /lib64/libpython2.7.so.1.0
> > #21 0x00007fb90730d142 in PyEval_EvalCode () from
> > /lib64/libpython2.7.so.1.0
> > #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> > #23 0x00007fb90732773e in PyRun_FileExFlags () from
> > /lib64/libpython2.7.so.1.0
> > #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> > /lib64/libpython2.7.so.1.0
> > #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> > #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> > #27 0x000000000040066e in _start ()
> >
> >
> > And the pstack from the process: /usr/sbin/ns-slapd -D
> > /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> > /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> > # pstack 15536
> > Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> > (Thread 0x7f5bc02e8700 (LWP 15540)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> > (Thread 0x7f5bbfae7700 (LWP 15541)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> > (Thread 0x7f5bbf2e6700 (LWP 15542)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> > (Thread 0x7f5bbeae5700 (LWP 15543)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> > (Thread 0x7f5bbe2e4700 (LWP 15544)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> > (Thread 0x7f5bbdae3700 (LWP 15545)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> > (Thread 0x7f5bbd2e2700 (LWP 15546)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> > (Thread 0x7f5bbcae1700 (LWP 15547)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> > (Thread 0x7f5bbc2e0700 (LWP 15548)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> > (Thread 0x7f5bbbadf700 (LWP 15549)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> > (Thread 0x7f5bbb2de700 (LWP 15550)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> > (Thread 0x7f5bbaadd700 (LWP 15551)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> > (Thread 0x7f5bba2dc700 (LWP 15552)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> > (Thread 0x7f5bb9adb700 (LWP 15553)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> > (Thread 0x7f5bb92da700 (LWP 15554)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4d33e63 in worker_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> > (Thread 0x7f5bb8ad9700 (LWP 15555)):
> > #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> > #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> > /lib64/libevent-2.0.so.5
> > #2  0x00007f5bd1d883ea in event_base_loop () at
> > /lib64/libevent-2.0.so.5
> > #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> > (Thread 0x7f5bb2b84700 (LWP 15559)):
> > #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> > #1  0x00007f5bd4acc599 in DS_Sleep () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> > (Thread 0x7f5bb2383700 (LWP 15560)):
> > #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> > #1  0x00007f5bd4acc599 in DS_Sleep () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> > (Thread 0x7f5bb1b82700 (LWP 15561)):
> > #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> > #1  0x00007f5bd4acc599 in DS_Sleep () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> > (Thread 0x7f5bb1381700 (LWP 15562)):
> > #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> > #1  0x00007f5bd4acc599 in DS_Sleep () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> > (Thread 0x7f5bb0b80700 (LWP 15563)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> > /usr/lib64/dirsrv/plugins/libcos-plugin.so
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> > (Thread 0x7f5bb037f700 (LWP 15564)):
> > #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> > #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> > /usr/lib64/dirsrv/plugins/libipa_cldap.so
> > #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> > (Thread 0x7f5bd5132700 (LWP 15566)):
> > #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> > #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> > #3  0x00007f5bc3572033 in _cl5TrimMain () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> > (Thread 0x7f5bafb7e700 (LWP 15567)):
> > #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> > #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> > #3  0x00007f5bc3587fa4 in protocol_sleep () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #4  0x00007f5bc358b75a in repl5_inc_run () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #5  0x00007f5bc358fbec in prot_thread_main () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> > (Thread 0x7f5baf37d700 (LWP 15568)):
> > #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> > #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> > #3  0x00007f5bc3587fa4 in protocol_sleep () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #4  0x00007f5bc358b75a in repl5_inc_run () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #5  0x00007f5bc358fbec in prot_thread_main () at
> > /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> > #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> > (Thread 0x7f5bae759700 (LWP 15569)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> > /usr/lib64/dirsrv/plugins/libroles-plugin.so
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> > (Thread 0x7f5badf58700 (LWP 15570)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> > /usr/lib64/dirsrv/plugins/libroles-plugin.so
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> > (Thread 0x7f5bad757700 (LWP 15571)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> > /usr/lib64/dirsrv/plugins/libroles-plugin.so
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> > (Thread 0x7f5bacd4f700 (LWP 15572)):
> > #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> > #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> > #3  0x0000564591e10b08 in housecleaning ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> > (Thread 0x7f5bac54e700 (LWP 15573)):
> > #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> > #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> > #3  0x00007f5bd4a5ba63 in eq_loop () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> > (Thread 0x7f5babd4d700 (LWP 15574)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> > (Thread 0x7f5bab54c700 (LWP 15575)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> > (Thread 0x7f5baad4b700 (LWP 15576)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> > (Thread 0x7f5baa54a700 (LWP 15577)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> > (Thread 0x7f5ba9d49700 (LWP 15578)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> > (Thread 0x7f5ba9548700 (LWP 15579)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> > (Thread 0x7f5ba8d47700 (LWP 15580)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> > (Thread 0x7f5ba8546700 (LWP 15581)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> > (Thread 0x7f5ba7d45700 (LWP 15582)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> > (Thread 0x7f5ba7544700 (LWP 15583)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> > (Thread 0x7f5ba6d43700 (LWP 15584)):
> > #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> > #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
> > #2  0x0000564591e07049 in connection_read_operation ()
> > #3  0x0000564591e0765d in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> > (Thread 0x7f5ba6542700 (LWP 15585)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> > (Thread 0x7f5ba5d41700 (LWP 15586)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> > (Thread 0x7f5ba5540700 (LWP 15587)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> > (Thread 0x7f5ba4d3f700 (LWP 15588)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> > (Thread 0x7f5ba453e700 (LWP 15589)):
> > #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> > #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> > ()
> > #3  0x0000564591e075d7 in connection_threadmain ()
> > #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> > (Thread 0x7f5ba3900700 (LWP 15590)):
> > #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> > #1  0x00007f5bd4acc599 in DS_Sleep () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> > /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> > #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> > (Thread 0x7f5ba30ff700 (LWP 15606)):
> > #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> > /lib64/libpthread.so.0
> > #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #4  0x00007f5bd4ac5942 in task_index_thread () at
> > /usr/lib64/dirsrv/libslapd.so.0
> > #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> > #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> > #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> > (Thread 0x7f5bd5135940 (LWP 15536)):
> > #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> > #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> > /usr/lib64/dirsrv/libnunc-stans.so.0
> > #2  0x0000564591e0d862 in slapd_daemon ()
> > #3  0x0000564591dfe86f in main ()
> >
> > Thanks & Regards.
> > ______________________________
> >
> > -----Original Message-----
> > From: thierry bordaz <tbordaz(a)redhat.com&gt;
> > Sent: Thursday, May 03, 2018 9:31
> > To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> > <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> > <solerm(a)unicc.org&gt;
> > Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > Hi,
> >
> > During indexing task we should see in the task status the periodic progression of the
> indexing.
> > May be the indexing is hanging somewhere. When the problem occurs could you provide a
> pstack of the dirsrv server ?
> >
> > best regards
> > thierry
> >
> > On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>> Hello,
> >>>
> >>> This is the output of the command (seems that is not complete):
> >>>
> >>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>> -b
> >>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>> n
> >>> =config
> >>> -s base
> >>> SASL/EXTERNAL authentication started SASL username:
> >>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>> SASL SSF: 0
> >>> # extended LDIF
> >>> #
> >>> # LDAPv3
> >>> # base
> >>> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>> c
> >>> n=config>
> >>> with scope baseObject
> >>> # filter: (objectclass=*)
> >>> # requesting: ALL
> >>> #
> >>>
> >>> # indextask_description_137444551994158920_5958, index, tasks,
> >>> config
> >>> dn:
> >>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>> n
> >>> =conf
> >>>     ig
> >>> objectClass: top
> >>> objectClass: extensibleObject
> >>> nsindexattribute: description
> >>> nsinstance: userRoot
> >>> cn: indextask_description_137444551994158920_5958
> >>> nstaskcurrentitem: 0
> >>> nstasktotalitems: 1
> >> IPA is looking for nstaskstatus and since that isn't there it is
> >> looping forever.
> >>
> >> I've cc'd a couple of the 389-ds devs to see what they think about it.
> >>
> >> rob
> >>
> >>> # search result
> >>> search: 3
> >>> result: 0 Success
> >>>
> >>> # numResponses: 2
> >>> # numEntries: 1
> >>>
> >>> Thanks & Regards.
> >>> ______________________________
> >>> -----Original Message-----
> >>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>> Sent: Tuesday, May 01, 2018 15:18
> >>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>> from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>> Hello,
> >>>>
> >>>> Thank you for your answer, now dirsrv can start, but after running
> >>>> the "
> >>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>
> >>>> # ipa-server-upgrade
> >>>>
> >>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>
> >>>>       [1/8]: saving configuration
> >>>>
> >>>>       [2/8]: disabling listeners
> >>>>
> >>>>       [3/8]: enabling DS global lock
> >>>>
> >>>>       [4/8]: starting directory server
> >>>>
> >>>>       [5/8]: updating schema
> >>>>
> >>>>       [6/8]: upgrading server
> >>>>
> >>>> The latest messages on the log is (there are no logs on the lasts 2h):
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG
> >>>> ---------------------------------------------
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>> database,cn=plugins,cn=config
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>
> >>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>
> >>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>> description
> >>>>
> >>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>> c
> >>>> n=
> >>>> config
> >>>>
> >>>> If I check the ipa services, dirsrv is the only one running:
> >>>>
> >>>> # ipactl status
> >>>>
> >>>> Directory Service: RUNNING
> >>>>
> >>>> krb5kdc Service: STOPPED
> >>>>
> >>>> kadmin Service: STOPPED
> >>>>
> >>>> named Service: STOPPED
> >>>>
> >>>> httpd Service: STOPPED
> >>>>
> >>>> ipa-custodia Service: STOPPED
> >>>>
> >>>> ntpd Service: RUNNING
> >>>>
> >>>> pki-tomcatd Service: STOPPED
> >>>>
> >>>> smb Service: STOPPED
> >>>>
> >>>> winbind Service: STOPPED
> >>>>
> >>>> ipa-otpd Service: STOPPED
> >>>>
> >>>> ipa-dnskeysyncd Service: STOPPED
> >>>>
> >>>> ipa: INFO: The ipactl command was successful
> >>>>
> >>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>> It depends on how many entries you have. This is an indexing task
> >>> and while 2 hours seems a bit excessive, it depends on how much work
> >>> it has to do.
> >>>
> >>> The 389-ds access and/or error logs may provide details.
> >>>
> >>> rob
> >>>
> >>>> Thanks.
> >>>>
> >>>> -----Original Message-----
> >>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>> Sent: Tuesday, May 01, 2018 9:56
> >>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>
> >>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>
> >>>>     >hello,
> >>>>
> >>>>     >
> >>>>
> >>>>     >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to
> >>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the
> >>>> unique replica I have, but after update dirsrv is not starting.
> >>>>
> >>>>     >It says it is needed run "ipa-server-upgrade", but it
> also fails:
> >>>>
> >>>>     ># ipactl start
> >>>>
> >>>>     >Upgrade required: please run ipa-server-upgrade command
> >>>> Aborting ipactl
> >>>>
> >>>>     >
> >>>>
> >>>>     ># ipa-server-upgrade
> >>>>
> >>>>     >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>
> >>>>     >  [1/8]: saving configuration
> >>>>
> >>>>     >  [2/8]: disabling listeners
> >>>>
> >>>>     >  [3/8]: enabling DS global lock
> >>>>
> >>>>     >  [4/8]: starting directory server
> >>>>
> >>>>     >  [error] CalledProcessError: Command '/bin/systemctl start
> >>>>
> >>>>     &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>> status
> >>>> 1
> >>>>
> >>>>     >  [cleanup]: stopping directory server
> >>>>
> >>>>     >  [cleanup]: restoring configuration
> >>>>
> >>>>     >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> >>>> run command ipa-server-upgrade manually.
> >>>>
> >>>>     >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>>
> >>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>
> >>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>> status
> >>>> 1 The
> >>>>
> >>>>     >ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
> >>>> for more
> >>>>
> >>>>     >information
> >>>>
> >>>>     >
> >>>>
> >>>>     >On the log I can see:
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>>
> >>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>
> >>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>
> >>>>     >finished, return code=3 2018-04-30T14:36:15Z DEBUG stdout=failed
> ...
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>
> >>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>
> >>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>
> >>>>     >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>
> >>>> failed because the control process exited with error code. See
> >>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>" and "journalctl
> -xe" for
> >>>> details.
> >>>>
> >>>>     >
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
> >>>>
> >>>>     >  File
> >>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>> line 504, in start_creation
> >>>>
> >>>>     >    run_step(full_msg, method)
> >>>>
> >>>>     >  File
> >>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>> line 494, in run_step
> >>>>
> >>>>     >    method()
> >>>>
> >>>>     >  File
> >>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>>> py
> >>>> ",
> >>>> line 95, in __start
> >>>>
> >>>>     >    srv.start(self.serverid, ldapi=True)
> >>>>
> >>>>     >  File
> >>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>>> line 161, in start
> >>>>
> >>>>     >    instance_name, capture_output=capture_output, wait=wait)
> >>>>
> >>>>     >  File
> >>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>> line 294, in start
> >>>>
> >>>>     >    skip_output=not capture_output)
> >>>>
> >>>>     >  File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>> line 542, in run
> >>>>
> >>>>     >    raise CalledProcessError(p.returncode, arg_string,
> >>>> str(output))
> >>>>
> >>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>
> >>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>> status
> >>>> 1
> >>>>
> >>>>     >
> >>>>
> >>>>     >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError:
> >>>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>> status
> >>>> 1
> >>>>
> >>>>     >
> >>>>
> >>>>     >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
> >>>>
> >>>>     >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>> slapd_bootstrap_config
> >>>> - The default password storage scheme could not be read or was not
> >>>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is
> >>>> mandatory.
> >>>>
> >>>>     >
> >>>>
> >>>>     >Checking on internet I show that "dse.ldif" could be
> corrupted,
> >>>> so I changed with "dse.ldif.startOK" without any change and
> then I
> >>>> changed with "dse.ldif.bak". The problem persist but the error
> has changed:
> >>>>
> >>>>     >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>> config_set_port
> >>>> -
> >>>>
> >>>>     >Non-Secure Port Disabled
> >>>>
> >>>>     >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>> symload_report_error
> >>>> -
> >>>>
> >>>>     >Netscape Portable Runtime error -5975:
> >>>>
> >>>>     >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: undefined
> >>>> symbol:
> >>>>
> >>>>     >replication_legacy_plugin_init
> >>>>
> >>>>     >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>> symload_report_error
> >>>> -
> >>>>
> >>>>     >Could not load symbol "replication_legacy_plugin_init"
> from
> >>>>
> >>>>     >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so"
> for plugin
> >>>> Legacy
> >>>>
> >>>>     >Replication Plugin
> >>>>
> >>>>     >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>> load_plugin_entry
> >>>> - Unable to load plugin "cn=Legacy Replication
> >>>> Plugin,cn=plugins,cn=config"
> >>>>
> >>>>     >
> >>>>
> >>>>     >I saw a bug about this problem, but it is still opened:
> >>>>
> >>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
> >>>>> 2
> >>>>     >
> >>>>
> >>>>     >Any idea how to fix the issue?
> >>>>
> >>>>     >
> >>>>
> >>>>     >If it is not possible to fix it, can I remove the replica from
> >>>> IPA and install it again with the same name?
> >>>>
> >>>> A quick fix could be to remove an entry for cn=Legacy Replication
> >>>> Plugin,cn=plugins,cn=config from
> >>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>
> >>>> when dirsrv is down.
> >>>>
> >>>> --
> >>>>
> >>>> / Alexander Bokovoy
> >>>>
> >>>> Sr. Principal Software Engineer
> >>>>
> >>>> Security / Identity Management Engineering Red Hat Limited, Finland
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>> To unsubscribe send an email to
> >>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>
> >>> _______________________________________________
> >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>> To unsubscribe send an email to
> >>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>

[Rob Crittenden]

thierry bordaz wrote:
>  Hi Soler,
>  
>  Thanks for the information.
>  So indexing is hanging because SC cache_init is running, the SC 
>  cache_init is hanging because SSSD is not started, SSSD is not started 
>  possibly because indexing prevents to get read access to the backend 
>  ("Backend is offline" TBC).
>  
>  An option would be to disable SC during upgrade phase ? Rob ? 
Sure, disabling SC would be a way to get things running again.

To do this:

1. Be sure dirsrv is not running
2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
4. change nsslapd-pluginEnabled from on to off
5. try the upgrade again

Assuming the upgrade is successful you can re-enable it post-install with:

# ipa-compat-manage enable

rob

>  
>  regards
>  thierry
>  
>  On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> > Hello Thierry,
> >
> > The version is: slapi-nis-0.56.0-8.el7.x86_64
> >
> > And the errors are:
> > [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin - 
> > group "XXXXXX(a)ipa.example.org" does not exist because SSSD is offline.
> > [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin - 
> > waiting for SSSD to become online...
> >
> > SSSD service is up, but as the ipa services are not (all) running, the 
> > ipa domain is down, logs when started sssd shows:
> > sssd[be[ipa.example.org]][823]: Backend is offline
> >
> > and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >   [sss_dp_get_reply] (0x0010): The Data Provider returned an error 
> > [org.freedesktop.sssd.Error.DataProvider.Offline]
> >
> > Thanks & Regards.
> >
> > -----Original Message-----
> > From: thierry bordaz <tbordaz(a)redhat.com&gt;
> > Sent: Thursday, May 03, 2018 11:25
> > To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden 
> > <rcritten(a)redhat.com>; FreeIPA users list 
> > <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > me again ...
> >
> > If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the 
> > hanging period it could also indicate the reason what Schema compat 
> > was waiting for (SSSD ?)
> >
> > On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >> hello,
> >>
> >> Yesterday my ssh console closed the connection, so I had to start 
> >> again the "ipa-server-upgrade", but the result is more or less the
> same:
> >> # ipa-server-upgrade
> >> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>     [1/10]: stopping directory server
> >>     [2/10]: saving configuration
> >>     [3/10]: disabling listeners
> >>     [4/10]: enabling DS global lock
> >>     [5/10]: starting directory server
> >>     [6/10]: updating schema
> >>     [7/10]: upgrading server
> >>
> >> But now, the lines that are repeated on the access log are:
> >> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH 
> >>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config" 
> >> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >> tag=101 nentries=1 etime=1.0000687740
> >>
> >> And it is on the same state than the other:
> >> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> >> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s
> >> base SASL/EXTERNAL authentication started SASL username:
> >> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >> SASL SSF: 0
> >> # extended LDIF
> >> #
> >> # LDAPv3
> >> # base
> >> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>
> >> # indextask_l_137445500911864330_4055, index, tasks, config
> >> dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >> objectClass: top
> >> objectClass: extensibleObject
> >> nsindexattribute: l
> >> nsinstance: userRoot
> >> cn: indextask_l_137445500911864330_4055
> >> nstaskcurrentitem: 0
> >> nstasktotalitems: 1
> >>
> >> # search result
> >> search: 3
> >> result: 0 Success
> >>
> >> # numResponses: 2
> >> # numEntries: 1
> >>
> >> The pstack of the proces: /usr/bin/python2
> >> /usr/sbin/ipa-server-upgrade # pstack 9805
> >> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >> #1  0x00007fb8ff5e0000 in time_sleep () from
> >> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >> /lib64/libpython2.7.so.1.0
> >> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >> /lib64/libpython2.7.so.1.0
> >> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >> /lib64/libpython2.7.so.1.0
> >> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >> #27 0x000000000040066e in _start ()
> >>
> >>
> >> And the pstack from the process: /usr/sbin/ns-slapd -D
> >> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> >> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >> # pstack 15536
> >> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> >> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> >> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> >> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> >> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> >> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> >> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> >> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> >> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> >> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> >> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> >> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> >> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> >> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> >> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> >> (Thread 0x7f5bb92da700 (LWP 15554)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> >> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >> /lib64/libevent-2.0.so.5
> >> #2  0x00007f5bd1d883ea in event_base_loop () at
> >> /lib64/libevent-2.0.so.5
> >> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> >> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> >> (Thread 0x7f5bb2383700 (LWP 15560)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> >> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> >> (Thread 0x7f5bb1381700 (LWP 15562)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> >> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> >> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> >> (Thread 0x7f5bb037f700 (LWP 15564)):
> >> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> >> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> >> (Thread 0x7f5bd5132700 (LWP 15566)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3572033 in _cl5TrimMain () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> >> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #5  0x00007f5bc358fbec in prot_thread_main () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> >> (Thread 0x7f5baf37d700 (LWP 15568)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #5  0x00007f5bc358fbec in prot_thread_main () at
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> >> (Thread 0x7f5bae759700 (LWP 15569)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> >> (Thread 0x7f5badf58700 (LWP 15570)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> >> (Thread 0x7f5bad757700 (LWP 15571)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> >> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x0000564591e10b08 in housecleaning ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> >> (Thread 0x7f5bac54e700 (LWP 15573)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> >> (Thread 0x7f5babd4d700 (LWP 15574)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> >> (Thread 0x7f5bab54c700 (LWP 15575)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> >> (Thread 0x7f5baad4b700 (LWP 15576)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> >> (Thread 0x7f5baa54a700 (LWP 15577)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> >> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> >> (Thread 0x7f5ba9548700 (LWP 15579)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> >> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> >> (Thread 0x7f5ba8546700 (LWP 15581)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> >> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> >> (Thread 0x7f5ba7544700 (LWP 15583)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> >> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
> >> #2  0x0000564591e07049 in connection_read_operation ()
> >> #3  0x0000564591e0765d in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> >> (Thread 0x7f5ba6542700 (LWP 15585)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> >> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> >> (Thread 0x7f5ba5540700 (LWP 15587)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> >> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> >> (Thread 0x7f5ba453e700 (LWP 15589)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> >> (Thread 0x7f5ba3900700 (LWP 15590)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> >> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> >> (Thread 0x7f5bd5135940 (LWP 15536)):
> >> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x0000564591e0d862 in slapd_daemon ()
> >> #3  0x0000564591dfe86f in main ()
> >>
> >> Thanks & Regards.
> >> ______________________________
> >>
> >> -----Original Message-----
> >> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >> Sent: Thursday, May 03, 2018 9:31
> >> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> >> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> >> <solerm(a)unicc.org&gt;
> >> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> Hi,
> >>
> >> During indexing task we should see in the task status the periodic 
> >> progression of the indexing.
> >> May be the indexing is hanging somewhere. When the problem occurs 
> >> could you provide a pstack of the dirsrv server ?
> >>
> >> best regards
> >> thierry
> >>
> >> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>> Hello,
> >>>>
> >>>> This is the output of the command (seems that is not complete):
> >>>>
> >>>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>> -b
> >>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>> n
> >>>> =config
> >>>> -s base
> >>>> SASL/EXTERNAL authentication started SASL username:
> >>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>> SASL SSF: 0
> >>>> # extended LDIF
> >>>> #
> >>>> # LDAPv3
> >>>> # base
> >>>> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>> c
> >>>> n=config>
> >>>> with scope baseObject
> >>>> # filter: (objectclass=*)
> >>>> # requesting: ALL
> >>>> #
> >>>>
> >>>> # indextask_description_137444551994158920_5958, index, tasks,
> >>>> config
> >>>> dn:
> >>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>> n
> >>>> =conf
> >>>>     ig
> >>>> objectClass: top
> >>>> objectClass: extensibleObject
> >>>> nsindexattribute: description
> >>>> nsinstance: userRoot
> >>>> cn: indextask_description_137444551994158920_5958
> >>>> nstaskcurrentitem: 0
> >>>> nstasktotalitems: 1
> >>> IPA is looking for nstaskstatus and since that isn't there it is
> >>> looping forever.
> >>>
> >>> I've cc'd a couple of the 389-ds devs to see what they think about
> it.
> >>>
> >>> rob
> >>>
> >>>> # search result
> >>>> search: 3
> >>>> result: 0 Success
> >>>>
> >>>> # numResponses: 2
> >>>> # numEntries: 1
> >>>>
> >>>> Thanks & Regards.
> >>>> ______________________________
> >>>> -----Original Message-----
> >>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>> Sent: Tuesday, May 01, 2018 15:18
> >>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>>> from
> >>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>
> >>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>> Hello,
> >>>>>
> >>>>> Thank you for your answer, now dirsrv can start, but after running
> >>>>> the "
> >>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>
> >>>>> # ipa-server-upgrade
> >>>>>
> >>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>
> >>>>>       [1/8]: saving configuration
> >>>>>
> >>>>>       [2/8]: disabling listeners
> >>>>>
> >>>>>       [3/8]: enabling DS global lock
> >>>>>
> >>>>>       [4/8]: starting directory server
> >>>>>
> >>>>>       [5/8]: updating schema
> >>>>>
> >>>>>       [6/8]: upgrading server
> >>>>>
> >>>>> The latest messages on the log is (there are no logs on the lasts 
> >>>>> 2h):
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>> database,cn=plugins,cn=config
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>
> >>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>> description
> >>>>>
> >>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>>> c
> >>>>> n=
> >>>>> config
> >>>>>
> >>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>
> >>>>> # ipactl status
> >>>>>
> >>>>> Directory Service: RUNNING
> >>>>>
> >>>>> krb5kdc Service: STOPPED
> >>>>>
> >>>>> kadmin Service: STOPPED
> >>>>>
> >>>>> named Service: STOPPED
> >>>>>
> >>>>> httpd Service: STOPPED
> >>>>>
> >>>>> ipa-custodia Service: STOPPED
> >>>>>
> >>>>> ntpd Service: RUNNING
> >>>>>
> >>>>> pki-tomcatd Service: STOPPED
> >>>>>
> >>>>> smb Service: STOPPED
> >>>>>
> >>>>> winbind Service: STOPPED
> >>>>>
> >>>>> ipa-otpd Service: STOPPED
> >>>>>
> >>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>
> >>>>> ipa: INFO: The ipactl command was successful
> >>>>>
> >>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>> It depends on how many entries you have. This is an indexing task
> >>>> and while 2 hours seems a bit excessive, it depends on how much work
> >>>> it has to do.
> >>>>
> >>>> The 389-ds access and/or error logs may provide details.
> >>>>
> >>>> rob
> >>>>
> >>>>> Thanks.
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>
> >>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>
> >>>>>     >hello,
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to
> >>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the
> >>>>> unique replica I have, but after update dirsrv is not starting.
> >>>>>
> >>>>>     >It says it is needed run "ipa-server-upgrade", but
> it also 
> >>>>> fails:
> >>>>>
> >>>>>     ># ipactl start
> >>>>>
> >>>>>     >Upgrade required: please run ipa-server-upgrade command
> >>>>> Aborting ipactl
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     ># ipa-server-upgrade
> >>>>>
> >>>>>     >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>
> >>>>>     >  [1/8]: saving configuration
> >>>>>
> >>>>>     >  [2/8]: disabling listeners
> >>>>>
> >>>>>     >  [3/8]: enabling DS global lock
> >>>>>
> >>>>>     >  [4/8]: starting directory server
> >>>>>
> >>>>>     >  [error] CalledProcessError: Command '/bin/systemctl
> start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >  [cleanup]: stopping directory server
> >>>>>
> >>>>>     >  [cleanup]: restoring configuration
> >>>>>
> >>>>>     >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log
> and
> >>>>> run command ipa-server-upgrade manually.
> >>>>>
> >>>>>     >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>>>
> >>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>>> status
> >>>>> 1 The
> >>>>>
> >>>>>     >ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log
> >>>>> for more
> >>>>>
> >>>>>     >information
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >On the log I can see:
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>
> >>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>
> >>>>>     >finished, return code=3 2018-04-30T14:36:15Z DEBUG 
> >>>>> stdout=failed ...
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory
> server
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>
> >>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>
> >>>>>     >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>
> >>>>> failed because the control process exited with error code. See
> >>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for
> >>>>> details.
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Traceback (most recent call
> last):
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>> line 504, in start_creation
> >>>>>
> >>>>>     >    run_step(full_msg, method)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>> line 494, in run_step
> >>>>>
> >>>>>     >    method()
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>>>> py
> >>>>> ",
> >>>>> line 95, in __start
> >>>>>
> >>>>>     >    srv.start(self.serverid, ldapi=True)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>>>> line 161, in start
> >>>>>
> >>>>>     >    instance_name, capture_output=capture_output, wait=wait)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>> line 294, in start
> >>>>>
> >>>>>     >    skip_output=not capture_output)
> >>>>>
> >>>>>     >  File
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>> line 542, in run
> >>>>>
> >>>>>     >    raise CalledProcessError(p.returncode, arg_string,
> >>>>> str(output))
> >>>>>
> >>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError:
> >>>>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I
> show:
> >>>>>
> >>>>>     >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>>> slapd_bootstrap_config
> >>>>> - The default password storage scheme could not be read or was not
> >>>>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is
> >>>>> mandatory.
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >Checking on internet I show that "dse.ldif" could
> be corrupted,
> >>>>> so I changed with "dse.ldif.startOK" without any change and
> then I
> >>>>> changed with "dse.ldif.bak". The problem persist but the
> error has 
> >>>>> changed:
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>>> config_set_port
> >>>>> -
> >>>>>
> >>>>>     >Non-Secure Port Disabled
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>>> symload_report_error
> >>>>> -
> >>>>>
> >>>>>     >Netscape Portable Runtime error -5975:
> >>>>>
> >>>>>     >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
> undefined
> >>>>> symbol:
> >>>>>
> >>>>>     >replication_legacy_plugin_init
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>>> symload_report_error
> >>>>> -
> >>>>>
> >>>>>     >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>
> >>>>>    
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> >>>>> Legacy
> >>>>>
> >>>>>     >Replication Plugin
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>>> load_plugin_entry
> >>>>> - Unable to load plugin "cn=Legacy Replication
> >>>>> Plugin,cn=plugins,cn=config"
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >I saw a bug about this problem, but it is still opened:
> >>>>>
> >>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
> >>>>>> 2
> >>>>>     >
> >>>>>
> >>>>>     >Any idea how to fix the issue?
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >If it is not possible to fix it, can I remove the replica
> from
> >>>>> IPA and install it again with the same name?
> >>>>>
> >>>>> A quick fix could be to remove an entry for cn=Legacy Replication
> >>>>> Plugin,cn=plugins,cn=config from
> >>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>
> >>>>> when dirsrv is down.
> >>>>>
> >>>>> -- 
> >>>>>
> >>>>> / Alexander Bokovoy
> >>>>>
> >>>>> Sr. Principal Software Engineer
> >>>>>
> >>>>> Security / Identity Management Engineering Red Hat Limited, Finland
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>> To unsubscribe send an email to
> >>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>
>

[thierry bordaz]

Hi,

The deadlock should happen quite rarely but remain a possibility.
Shouldn't we address this deadlock scenario in others ipa scripts that 
are doing DS admin tasks (import, index). For example disabling/enabling 
during those tasks ?

regards
thierry

On 05/03/2018 09:16 PM, Rob Crittenden via FreeIPA-users wrote:
>  thierry bordaz wrote:
> > Hi Soler,
> >
> > Thanks for the information.
> > So indexing is hanging because SC cache_init is running, the SC 
> > cache_init is hanging because SSSD is not started, SSSD is not 
> > started possibly because indexing prevents to get read access to the 
> > backend ("Backend is offline" TBC).
> >
> > An option would be to disable SC during upgrade phase ? Rob ?
>
>  Sure, disabling SC would be a way to get things running again.
>
>  To do this:
>
>  1. Be sure dirsrv is not running
>  2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
>  3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
>  4. change nsslapd-pluginEnabled from on to off
>  5. try the upgrade again
>
>  Assuming the upgrade is successful you can re-enable it post-install 
>  with:
>
>  # ipa-compat-manage enable
>
>  rob
>
> >
> > regards
> > thierry
> >
> > On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> >> Hello Thierry,
> >>
> >> The version is: slapi-nis-0.56.0-8.el7.x86_64
> >>
> >> And the errors are:
> >> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin 
> >> - group "XXXXXX(a)ipa.example.org" does not exist because SSSD is 
> >> offline.
> >> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin 
> >> - waiting for SSSD to become online...
> >>
> >> SSSD service is up, but as the ipa services are not (all) running, 
> >> the ipa domain is down, logs when started sssd shows:
> >> sssd[be[ipa.example.org]][823]: Backend is offline
> >>
> >> and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >>   [sss_dp_get_reply] (0x0010): The Data Provider returned an error 
> >> [org.freedesktop.sssd.Error.DataProvider.Offline]
> >>
> >> Thanks & Regards.
> >>
> >> -----Original Message-----
> >> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >> Sent: Thursday, May 03, 2018 11:25
> >> To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden 
> >> <rcritten(a)redhat.com>; FreeIPA users list 
> >> <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating 
> >> from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> me again ...
> >>
> >> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during 
> >> the hanging period it could also indicate the reason what Schema 
> >> compat was waiting for (SSSD ?)
> >>
> >> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >>> hello,
> >>>
> >>> Yesterday my ssh console closed the connection, so I had to start 
> >>> again the "ipa-server-upgrade", but the result is more or less the
>
> >>> same:
> >>> # ipa-server-upgrade
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>     [1/10]: stopping directory server
> >>>     [2/10]: saving configuration
> >>>     [3/10]: disabling listeners
> >>>     [4/10]: enabling DS global lock
> >>>     [5/10]: starting directory server
> >>>     [6/10]: updating schema
> >>>     [7/10]: upgrading server
> >>>
> >>> But now, the lines that are repeated on the access log are:
> >>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH 
> >>>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config" 
> >>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >>> tag=101 nentries=1 etime=1.0000687740
> >>>
> >>> And it is on the same state than the other:
> >>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> >>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s
> >>> base SASL/EXTERNAL authentication started SASL username:
> >>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>> SASL SSF: 0
> >>> # extended LDIF
> >>> #
> >>> # LDAPv3
> >>> # base
> >>> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >>> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>>
> >>> # indextask_l_137445500911864330_4055, index, tasks, config
> >>> dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >>> objectClass: top
> >>> objectClass: extensibleObject
> >>> nsindexattribute: l
> >>> nsinstance: userRoot
> >>> cn: indextask_l_137445500911864330_4055
> >>> nstaskcurrentitem: 0
> >>> nstasktotalitems: 1
> >>>
> >>> # search result
> >>> search: 3
> >>> result: 0 Success
> >>>
> >>> # numResponses: 2
> >>> # numEntries: 1
> >>>
> >>> The pstack of the proces: /usr/bin/python2
> >>> /usr/sbin/ipa-server-upgrade # pstack 9805
> >>> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >>> #1  0x00007fb8ff5e0000 in time_sleep () from
> >>> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >>> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >>> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >>> #27 0x000000000040066e in _start ()
> >>>
> >>>
> >>> And the pstack from the process: /usr/sbin/ns-slapd -D
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> >>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >>> # pstack 15536
> >>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> >>> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> >>> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> >>> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> >>> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> >>> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> >>> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> >>> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> >>> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> >>> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> >>> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> >>> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> >>> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> >>> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> >>> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> >>> (Thread 0x7f5bb92da700 (LWP 15554)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> >>> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >>> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >>> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >>> /lib64/libevent-2.0.so.5
> >>> #2  0x00007f5bd1d883ea in event_base_loop () at
> >>> /lib64/libevent-2.0.so.5
> >>> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> >>> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> >>> (Thread 0x7f5bb2383700 (LWP 15560)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> >>> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> >>> (Thread 0x7f5bb1381700 (LWP 15562)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> >>> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> >>> (Thread 0x7f5bb037f700 (LWP 15564)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> >>> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> >>> (Thread 0x7f5bd5132700 (LWP 15566)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3572033 in _cl5TrimMain () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> >>> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> >>> (Thread 0x7f5baf37d700 (LWP 15568)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> >>> (Thread 0x7f5bae759700 (LWP 15569)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> >>> (Thread 0x7f5badf58700 (LWP 15570)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> >>> (Thread 0x7f5bad757700 (LWP 15571)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> >>> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x0000564591e10b08 in housecleaning ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> >>> (Thread 0x7f5bac54e700 (LWP 15573)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> >>> (Thread 0x7f5babd4d700 (LWP 15574)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> >>> (Thread 0x7f5bab54c700 (LWP 15575)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> >>> (Thread 0x7f5baad4b700 (LWP 15576)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> >>> (Thread 0x7f5baa54a700 (LWP 15577)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> >>> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> >>> (Thread 0x7f5ba9548700 (LWP 15579)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> >>> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> >>> (Thread 0x7f5ba8546700 (LWP 15581)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> >>> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> >>> (Thread 0x7f5ba7544700 (LWP 15583)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> >>> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
> >>> #2  0x0000564591e07049 in connection_read_operation ()
> >>> #3  0x0000564591e0765d in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> >>> (Thread 0x7f5ba6542700 (LWP 15585)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> >>> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> >>> (Thread 0x7f5ba5540700 (LWP 15587)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> >>> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> >>> (Thread 0x7f5ba453e700 (LWP 15589)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> >>> (Thread 0x7f5ba3900700 (LWP 15590)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> >>> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >>> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> >>> (Thread 0x7f5bd5135940 (LWP 15536)):
> >>> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x0000564591e0d862 in slapd_daemon ()
> >>> #3  0x0000564591dfe86f in main ()
> >>>
> >>> Thanks & Regards.
> >>> ______________________________
> >>>
> >>> -----Original Message-----
> >>> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >>> Sent: Thursday, May 03, 2018 9:31
> >>> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> >>> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> >>> <solerm(a)unicc.org&gt;
> >>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> Hi,
> >>>
> >>> During indexing task we should see in the task status the periodic 
> >>> progression of the indexing.
> >>> May be the indexing is hanging somewhere. When the problem occurs 
> >>> could you provide a pstack of the dirsrv server ?
> >>>
> >>> best regards
> >>> thierry
> >>>
> >>> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>> Hello,
> >>>>>
> >>>>> This is the output of the command (seems that is not complete):
> >>>>>
> >>>>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>>> -b
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>>> n
> >>>>> =config
> >>>>> -s base
> >>>>> SASL/EXTERNAL authentication started SASL username:
> >>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>>> SASL SSF: 0
> >>>>> # extended LDIF
> >>>>> #
> >>>>> # LDAPv3
> >>>>> # base
> >>>>>
> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>>> c
> >>>>> n=config>
> >>>>> with scope baseObject
> >>>>> # filter: (objectclass=*)
> >>>>> # requesting: ALL
> >>>>> #
> >>>>>
> >>>>> # indextask_description_137444551994158920_5958, index, tasks,
> >>>>> config
> >>>>> dn:
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>>> n
> >>>>> =conf
> >>>>>     ig
> >>>>> objectClass: top
> >>>>> objectClass: extensibleObject
> >>>>> nsindexattribute: description
> >>>>> nsinstance: userRoot
> >>>>> cn: indextask_description_137444551994158920_5958
> >>>>> nstaskcurrentitem: 0
> >>>>> nstasktotalitems: 1
> >>>> IPA is looking for nstaskstatus and since that isn't there it is
> >>>> looping forever.
> >>>>
> >>>> I've cc'd a couple of the 389-ds devs to see what they think
> about 
> >>>> it.
> >>>>
> >>>> rob
> >>>>
> >>>>> # search result
> >>>>> search: 3
> >>>>> result: 0 Success
> >>>>>
> >>>>> # numResponses: 2
> >>>>> # numEntries: 1
> >>>>>
> >>>>> Thanks & Regards.
> >>>>> ______________________________
> >>>>> -----Original Message-----
> >>>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>>> Sent: Tuesday, May 01, 2018 15:18
> >>>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>>>> from
> >>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>
> >>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> Thank you for your answer, now dirsrv can start, but after
> running
> >>>>>> the "
> >>>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>>
> >>>>>> # ipa-server-upgrade
> >>>>>>
> >>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>       [1/8]: saving configuration
> >>>>>>
> >>>>>>       [2/8]: disabling listeners
> >>>>>>
> >>>>>>       [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>       [4/8]: starting directory server
> >>>>>>
> >>>>>>       [5/8]: updating schema
> >>>>>>
> >>>>>>       [6/8]: upgrading server
> >>>>>>
> >>>>>> The latest messages on the log is (there are no logs on the 
> >>>>>> lasts 2h):
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>>> ---------------------------------------------
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>>> database,cn=plugins,cn=config
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>>> description
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>>>> c
> >>>>>> n=
> >>>>>> config
> >>>>>>
> >>>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>>
> >>>>>> # ipactl status
> >>>>>>
> >>>>>> Directory Service: RUNNING
> >>>>>>
> >>>>>> krb5kdc Service: STOPPED
> >>>>>>
> >>>>>> kadmin Service: STOPPED
> >>>>>>
> >>>>>> named Service: STOPPED
> >>>>>>
> >>>>>> httpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-custodia Service: STOPPED
> >>>>>>
> >>>>>> ntpd Service: RUNNING
> >>>>>>
> >>>>>> pki-tomcatd Service: STOPPED
> >>>>>>
> >>>>>> smb Service: STOPPED
> >>>>>>
> >>>>>> winbind Service: STOPPED
> >>>>>>
> >>>>>> ipa-otpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>>
> >>>>>> ipa: INFO: The ipactl command was successful
> >>>>>>
> >>>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>>> It depends on how many entries you have. This is an indexing task
> >>>>> and while 2 hours seems a bit excessive, it depends on how much work
> >>>>> it has to do.
> >>>>>
> >>>>> The 389-ds access and/or error logs may provide details.
> >>>>>
> >>>>> rob
> >>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>>> To: FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating
> from
> >>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>>
> >>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users 
> >>>>>> wrote:
> >>>>>>
> >>>>>>     >hello,
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >I have an IPA master that updated from 4.5.0 (RHEL 7.4)
> to
> >>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the
> >>>>>> unique replica I have, but after update dirsrv is not starting.
> >>>>>>
> >>>>>>     >It says it is needed run "ipa-server-upgrade",
> but it also 
> >>>>>> fails:
> >>>>>>
> >>>>>>     ># ipactl start
> >>>>>>
> >>>>>>     >Upgrade required: please run ipa-server-upgrade command
> >>>>>> Aborting ipactl
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     ># ipa-server-upgrade
> >>>>>>
> >>>>>>     >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>     >  [1/8]: saving configuration
> >>>>>>
> >>>>>>     >  [2/8]: disabling listeners
> >>>>>>
> >>>>>>     >  [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>     >  [4/8]: starting directory server
> >>>>>>
> >>>>>>     >  [error] CalledProcessError: Command '/bin/systemctl
> start
> >>>>>>
> >>>>>>     &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>     >  [cleanup]: stopping directory server
> >>>>>>
> >>>>>>     >  [cleanup]: restoring configuration
> >>>>>>
> >>>>>>     >IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and
> >>>>>> run command ipa-server-upgrade manually.
> >>>>>>
> >>>>>>     >Unexpected error - see /var/log/ipaupgrade.log for
> details:
> >>>>>>
> >>>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1 The
> >>>>>>
> >>>>>>     >ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log
> >>>>>> for more
> >>>>>>
> >>>>>>     >information
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >On the log I can see:
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>>>>
> >>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>     >finished, return code=3 2018-04-30T14:36:15Z DEBUG 
> >>>>>> stdout=failed ...
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory
> server
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>>
> >>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>     >finished, return code=1 2018-04-30T14:36:15Z DEBUG
> stdout=
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> failed because the control process exited with error code. See
> >>>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for
> >>>>>> details.
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG Traceback (most recent call
> last):
> >>>>>>
> >>>>>>     >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 504, in start_creation
> >>>>>>
> >>>>>>     >    run_step(full_msg, method)
> >>>>>>
> >>>>>>     >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 494, in run_step
> >>>>>>
> >>>>>>     >    method()
> >>>>>>
> >>>>>>     >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance. 
> >>>>>>
> >>>>>> py
> >>>>>> ",
> >>>>>> line 95, in __start
> >>>>>>
> >>>>>>     >    srv.start(self.serverid, ldapi=True)
> >>>>>>
> >>>>>>     >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>>>>> line 161, in start
> >>>>>>
> >>>>>>     >    instance_name, capture_output=capture_output,
> wait=wait)
> >>>>>>
> >>>>>>     >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>>> line 294, in start
> >>>>>>
> >>>>>>     >    skip_output=not capture_output)
> >>>>>>
> >>>>>>     >  File 
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>>> line 542, in run
> >>>>>>
> >>>>>>     >    raise CalledProcessError(p.returncode, arg_string,
> >>>>>> str(output))
> >>>>>>
> >>>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError:
> >>>>>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I
> show:
> >>>>>>
> >>>>>>     >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>>>> slapd_bootstrap_config
> >>>>>> - The default password storage scheme could not be read or was
> not
> >>>>>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It
> is
> >>>>>> mandatory.
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >Checking on internet I show that "dse.ldif"
> could be 
> >>>>>> corrupted,
> >>>>>> so I changed with "dse.ldif.startOK" without any change
> and then I
> >>>>>> changed with "dse.ldif.bak". The problem persist but
> the error 
> >>>>>> has changed:
> >>>>>>
> >>>>>>     >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>>>> config_set_port
> >>>>>> -
> >>>>>>
> >>>>>>     >Non-Secure Port Disabled
> >>>>>>
> >>>>>>     >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>     >Netscape Portable Runtime error -5975:
> >>>>>>
> >>>>>> >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
> undefined
> >>>>>> symbol:
> >>>>>>
> >>>>>>     >replication_legacy_plugin_init
> >>>>>>
> >>>>>>     >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>     >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>>
> >>>>>>
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> >>>>>> Legacy
> >>>>>>
> >>>>>>     >Replication Plugin
> >>>>>>
> >>>>>>     >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>>>> load_plugin_entry
> >>>>>> - Unable to load plugin "cn=Legacy Replication
> >>>>>> Plugin,cn=plugins,cn=config"
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >I saw a bug about this problem, but it is still opened:
> >>>>>>
> >>>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
> >>>>>>> 2
> >>>>>>     >
> >>>>>>
> >>>>>>     >Any idea how to fix the issue?
> >>>>>>
> >>>>>>     >
> >>>>>>
> >>>>>>     >If it is not possible to fix it, can I remove the replica
> from
> >>>>>> IPA and install it again with the same name?
> >>>>>>
> >>>>>> A quick fix could be to remove an entry for cn=Legacy
> Replication
> >>>>>> Plugin,cn=plugins,cn=config from
> >>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>>
> >>>>>> when dirsrv is down.
> >>>>>>
> >>>>>> -- 
> >>>>>>
> >>>>>> / Alexander Bokovoy
> >>>>>>
> >>>>>> Sr. Principal Software Engineer
> >>>>>>
> >>>>>> Security / Identity Management Engineering Red Hat Limited,
> Finland
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> FreeIPA-users mailing list --
> freeipa-users(a)lists.fedorahosted.org
> >>>>>> To unsubscribe send an email to
> >>>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>
> >
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to 
>  freeipa-users-leave(a)lists.fedorahosted.org

[Rob Crittenden]

thierry bordaz via FreeIPA-users wrote:
>  Hi,
>  
>  The deadlock should happen quite rarely but remain a possibility.
>  Shouldn't we address this deadlock scenario in others ipa scripts that 
>  are doing DS admin tasks (import, index). For example disabling/enabling 
>  during those tasks ? 
If it can cause problems then yes, https://pagure.io/freeipa/issue/7531

rob

>  
>  regards
>  thierry
>  
>  On 05/03/2018 09:16 PM, Rob Crittenden via FreeIPA-users wrote:
> > thierry bordaz wrote:
> >> Hi Soler,
> >>
> >> Thanks for the information.
> >> So indexing is hanging because SC cache_init is running, the SC 
> >> cache_init is hanging because SSSD is not started, SSSD is not 
> >> started possibly because indexing prevents to get read access to the 
> >> backend ("Backend is offline" TBC).
> >>
> >> An option would be to disable SC during upgrade phase ? Rob ?
> >
> > Sure, disabling SC would be a way to get things running again.
> >
> > To do this:
> >
> > 1. Be sure dirsrv is not running
> > 2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
> > 3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
> > 4. change nsslapd-pluginEnabled from on to off
> > 5. try the upgrade again
> >
> > Assuming the upgrade is successful you can re-enable it post-install 
> > with:
> >
> > # ipa-compat-manage enable
> >
> > rob
> >
> >>
> >> regards
> >> thierry
> >>
> >> On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> >>> Hello Thierry,
> >>>
> >>> The version is: slapi-nis-0.56.0-8.el7.x86_64
> >>>
> >>> And the errors are:
> >>> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin 
> >>> - group "XXXXXX(a)ipa.example.org" does not exist because SSSD is 
> >>> offline.
> >>> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin 
> >>> - waiting for SSSD to become online...
> >>>
> >>> SSSD service is up, but as the ipa services are not (all) running, 
> >>> the ipa domain is down, logs when started sssd shows:
> >>> sssd[be[ipa.example.org]][823]: Backend is offline
> >>>
> >>> and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >>>   [sss_dp_get_reply] (0x0010): The Data Provider returned an error 
> >>> [org.freedesktop.sssd.Error.DataProvider.Offline]
> >>>
> >>> Thanks & Regards.
> >>>
> >>> -----Original Message-----
> >>> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >>> Sent: Thursday, May 03, 2018 11:25
> >>> To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden 
> >>> <rcritten(a)redhat.com>; FreeIPA users list 
> >>> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating 
> >>> from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> me again ...
> >>>
> >>> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during 
> >>> the hanging period it could also indicate the reason what Schema 
> >>> compat was waiting for (SSSD ?)
> >>>
> >>> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >>>> hello,
> >>>>
> >>>> Yesterday my ssh console closed the connection, so I had to start 
> >>>> again the "ipa-server-upgrade", but the result is more or less
> the 
> >>>> same:
> >>>> # ipa-server-upgrade
> >>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>     [1/10]: stopping directory server
> >>>>     [2/10]: saving configuration
> >>>>     [3/10]: disabling listeners
> >>>>     [4/10]: enabling DS global lock
> >>>>     [5/10]: starting directory server
> >>>>     [6/10]: updating schema
> >>>>     [7/10]: upgrading server
> >>>>
> >>>> But now, the lines that are repeated on the access log are:
> >>>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH 
> >>>>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config" 
> >>>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >>>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >>>> tag=101 nentries=1 etime=1.0000687740
> >>>>
> >>>> And it is on the same state than the other:
> >>>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
> >>>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s
> >>>> base SASL/EXTERNAL authentication started SASL username:
> >>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>> SASL SSF: 0
> >>>> # extended LDIF
> >>>> #
> >>>> # LDAPv3
> >>>> # base
> >>>>
> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >>>> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>>>
> >>>> # indextask_l_137445500911864330_4055, index, tasks, config
> >>>> dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >>>> objectClass: top
> >>>> objectClass: extensibleObject
> >>>> nsindexattribute: l
> >>>> nsinstance: userRoot
> >>>> cn: indextask_l_137445500911864330_4055
> >>>> nstaskcurrentitem: 0
> >>>> nstasktotalitems: 1
> >>>>
> >>>> # search result
> >>>> search: 3
> >>>> result: 0 Success
> >>>>
> >>>> # numResponses: 2
> >>>> # numEntries: 1
> >>>>
> >>>> The pstack of the proces: /usr/bin/python2
> >>>> /usr/sbin/ipa-server-upgrade # pstack 9805
> >>>> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >>>> #1  0x00007fb8ff5e0000 in time_sleep () from
> >>>> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >>>> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >>>> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >>>> /lib64/libpython2.7.so.1.0
> >>>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >>>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >>>> #27 0x000000000040066e in _start ()
> >>>>
> >>>>
> >>>> And the pstack from the process: /usr/sbin/ns-slapd -D
> >>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> >>>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >>>> # pstack 15536
> >>>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> >>>> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> >>>> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> >>>> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> >>>> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> >>>> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> >>>> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> >>>> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> >>>> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> >>>> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> >>>> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> >>>> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> >>>> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> >>>> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> >>>> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> >>>> (Thread 0x7f5bb92da700 (LWP 15554)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> >>>> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >>>> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >>>> /lib64/libevent-2.0.so.5
> >>>> #2  0x00007f5bd1d883ea in event_base_loop () at
> >>>> /lib64/libevent-2.0.so.5
> >>>> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> >>>> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >>>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> >>>> (Thread 0x7f5bb2383700 (LWP 15560)):
> >>>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> >>>> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >>>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> >>>> (Thread 0x7f5bb1381700 (LWP 15562)):
> >>>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> >>>> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> >>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> >>>> (Thread 0x7f5bb037f700 (LWP 15564)):
> >>>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>>> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> >>>> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >>>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> >>>> (Thread 0x7f5bd5132700 (LWP 15566)):
> >>>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #3  0x00007f5bc3572033 in _cl5TrimMain () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> >>>> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >>>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> >>>> (Thread 0x7f5baf37d700 (LWP 15568)):
> >>>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> >>>> (Thread 0x7f5bae759700 (LWP 15569)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> >>>> (Thread 0x7f5badf58700 (LWP 15570)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> >>>> (Thread 0x7f5bad757700 (LWP 15571)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> >>>> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >>>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #3  0x0000564591e10b08 in housecleaning ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> >>>> (Thread 0x7f5bac54e700 (LWP 15573)):
> >>>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> >>>> (Thread 0x7f5babd4d700 (LWP 15574)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> >>>> (Thread 0x7f5bab54c700 (LWP 15575)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> >>>> (Thread 0x7f5baad4b700 (LWP 15576)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> >>>> (Thread 0x7f5baa54a700 (LWP 15577)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> >>>> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> >>>> (Thread 0x7f5ba9548700 (LWP 15579)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> >>>> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> >>>> (Thread 0x7f5ba8546700 (LWP 15581)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> >>>> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> >>>> (Thread 0x7f5ba7544700 (LWP 15583)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> >>>> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >>>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e07049 in connection_read_operation ()
> >>>> #3  0x0000564591e0765d in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> >>>> (Thread 0x7f5ba6542700 (LWP 15585)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> >>>> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> >>>> (Thread 0x7f5ba5540700 (LWP 15587)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> >>>> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> >>>> (Thread 0x7f5ba453e700 (LWP 15589)):
> >>>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>>> ()
> >>>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> >>>> (Thread 0x7f5ba3900700 (LWP 15590)):
> >>>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> >>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> >>>> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >>>> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >>>> /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> >>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>>> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >>>> /usr/lib64/dirsrv/libslapd.so.0
> >>>> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>>> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>>> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> >>>> (Thread 0x7f5bd5135940 (LWP 15536)):
> >>>> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >>>> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >>>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>>> #2  0x0000564591e0d862 in slapd_daemon ()
> >>>> #3  0x0000564591dfe86f in main ()
> >>>>
> >>>> Thanks & Regards.
> >>>> ______________________________
> >>>>
> >>>> -----Original Message-----
> >>>> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >>>> Sent: Thursday, May 03, 2018 9:31
> >>>> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> >>>> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> >>>> <solerm(a)unicc.org&gt;
> >>>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>
> >>>> Hi,
> >>>>
> >>>> During indexing task we should see in the task status the periodic 
> >>>> progression of the indexing.
> >>>> May be the indexing is hanging somewhere. When the problem occurs 
> >>>> could you provide a pstack of the dirsrv server ?
> >>>>
> >>>> best regards
> >>>> thierry
> >>>>
> >>>> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> This is the output of the command (seems that is not complete):
> >>>>>>
> >>>>>> # ldapsearch -H
> ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>>>> -b
> >>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>>>> n
> >>>>>> =config
> >>>>>> -s base
> >>>>>> SASL/EXTERNAL authentication started SASL username:
> >>>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>>>> SASL SSF: 0
> >>>>>> # extended LDIF
> >>>>>> #
> >>>>>> # LDAPv3
> >>>>>> # base
> >>>>>>
> <cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>>>> c
> >>>>>> n=config>
> >>>>>> with scope baseObject
> >>>>>> # filter: (objectclass=*)
> >>>>>> # requesting: ALL
> >>>>>> #
> >>>>>>
> >>>>>> # indextask_description_137444551994158920_5958, index, tasks,
> >>>>>> config
> >>>>>> dn:
> >>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
> >>>>>> n
> >>>>>> =conf
> >>>>>>     ig
> >>>>>> objectClass: top
> >>>>>> objectClass: extensibleObject
> >>>>>> nsindexattribute: description
> >>>>>> nsinstance: userRoot
> >>>>>> cn: indextask_description_137444551994158920_5958
> >>>>>> nstaskcurrentitem: 0
> >>>>>> nstasktotalitems: 1
> >>>>> IPA is looking for nstaskstatus and since that isn't there it is
> >>>>> looping forever.
> >>>>>
> >>>>> I've cc'd a couple of the 389-ds devs to see what they think
> about 
> >>>>> it.
> >>>>>
> >>>>> rob
> >>>>>
> >>>>>> # search result
> >>>>>> search: 3
> >>>>>> result: 0 Success
> >>>>>>
> >>>>>> # numResponses: 2
> >>>>>> # numEntries: 1
> >>>>>>
> >>>>>> Thanks & Regards.
> >>>>>> ______________________________
> >>>>>> -----Original Message-----
> >>>>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>>>> Sent: Tuesday, May 01, 2018 15:18
> >>>>>> To: FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>>>>> from
> >>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>>
> >>>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> Thank you for your answer, now dirsrv can start, but after
> running
> >>>>>>> the "
> >>>>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>>>
> >>>>>>> # ipa-server-upgrade
> >>>>>>>
> >>>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>>
> >>>>>>>       [1/8]: saving configuration
> >>>>>>>
> >>>>>>>       [2/8]: disabling listeners
> >>>>>>>
> >>>>>>>       [3/8]: enabling DS global lock
> >>>>>>>
> >>>>>>>       [4/8]: starting directory server
> >>>>>>>
> >>>>>>>       [5/8]: updating schema
> >>>>>>>
> >>>>>>>       [6/8]: upgrading server
> >>>>>>>
> >>>>>>> The latest messages on the log is (there are no logs on the 
> >>>>>>> lasts 2h):
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>>>> ---------------------------------------------
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying
> updates
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>>>> database,cn=plugins,cn=config
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>>>> description
> >>>>>>>
> >>>>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
> >>>>>>> c
> >>>>>>> n=
> >>>>>>> config
> >>>>>>>
> >>>>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>>>
> >>>>>>> # ipactl status
> >>>>>>>
> >>>>>>> Directory Service: RUNNING
> >>>>>>>
> >>>>>>> krb5kdc Service: STOPPED
> >>>>>>>
> >>>>>>> kadmin Service: STOPPED
> >>>>>>>
> >>>>>>> named Service: STOPPED
> >>>>>>>
> >>>>>>> httpd Service: STOPPED
> >>>>>>>
> >>>>>>> ipa-custodia Service: STOPPED
> >>>>>>>
> >>>>>>> ntpd Service: RUNNING
> >>>>>>>
> >>>>>>> pki-tomcatd Service: STOPPED
> >>>>>>>
> >>>>>>> smb Service: STOPPED
> >>>>>>>
> >>>>>>> winbind Service: STOPPED
> >>>>>>>
> >>>>>>> ipa-otpd Service: STOPPED
> >>>>>>>
> >>>>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>>>
> >>>>>>> ipa: INFO: The ipactl command was successful
> >>>>>>>
> >>>>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>>>> It depends on how many entries you have. This is an indexing
> task
> >>>>>> and while 2 hours seems a bit excessive, it depends on how much
> work
> >>>>>> it has to do.
> >>>>>>
> >>>>>> The 389-ds access and/or error logs may provide details.
> >>>>>>
> >>>>>> rob
> >>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>>>> To: FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating
> from
> >>>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>>>
> >>>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users
>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>     >hello,
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >I have an IPA master that updated from 4.5.0 (RHEL
> 7.4) to
> >>>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with
> the
> >>>>>>> unique replica I have, but after update dirsrv is not
> starting.
> >>>>>>>
> >>>>>>>     >It says it is needed run
> "ipa-server-upgrade", but it also 
> >>>>>>> fails:
> >>>>>>>
> >>>>>>>     ># ipactl start
> >>>>>>>
> >>>>>>>     >Upgrade required: please run ipa-server-upgrade
> command
> >>>>>>> Aborting ipactl
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     ># ipa-server-upgrade
> >>>>>>>
> >>>>>>>     >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>>
> >>>>>>>     >  [1/8]: saving configuration
> >>>>>>>
> >>>>>>>     >  [2/8]: disabling listeners
> >>>>>>>
> >>>>>>>     >  [3/8]: enabling DS global lock
> >>>>>>>
> >>>>>>>     >  [4/8]: starting directory server
> >>>>>>>
> >>>>>>>     >  [error] CalledProcessError: Command
> '/bin/systemctl start
> >>>>>>>
> >>>>>>>     &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>>> status
> >>>>>>> 1
> >>>>>>>
> >>>>>>>     >  [cleanup]: stopping directory server
> >>>>>>>
> >>>>>>>     >  [cleanup]: restoring configuration
> >>>>>>>
> >>>>>>>     >IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and
> >>>>>>> run command ipa-server-upgrade manually.
> >>>>>>>
> >>>>>>>     >Unexpected error - see /var/log/ipaupgrade.log for
> details:
> >>>>>>>
> >>>>>>>     >CalledProcessError: Command '/bin/systemctl
> start
> >>>>>>>
> >>>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>>> status
> >>>>>>> 1 The
> >>>>>>>
> >>>>>>>     >ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log
> >>>>>>> for more
> >>>>>>>
> >>>>>>>     >information
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >On the log I can see:
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl
> is-active
> >>>>>>>
> >>>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>
> >>>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>>
> >>>>>>>     >finished, return code=3 2018-04-30T14:36:15Z DEBUG 
> >>>>>>> stdout=failed ...
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG   [4/8]: starting
> directory server
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>>>
> >>>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>
> >>>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>>
> >>>>>>>     >finished, return code=1 2018-04-30T14:36:15Z DEBUG
> stdout=
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>
> >>>>>>> failed because the control process exited with error code.
> See
> >>>>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for
> >>>>>>> details.
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG Traceback (most recent
> call last):
> >>>>>>>
> >>>>>>>     >  File
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>>> line 504, in start_creation
> >>>>>>>
> >>>>>>>     >    run_step(full_msg, method)
> >>>>>>>
> >>>>>>>     >  File
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>>> line 494, in run_step
> >>>>>>>
> >>>>>>>     >    method()
> >>>>>>>
> >>>>>>>     >  File
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance. 
> >>>>>>>
> >>>>>>> py
> >>>>>>> ",
> >>>>>>> line 95, in __start
> >>>>>>>
> >>>>>>>     >    srv.start(self.serverid, ldapi=True)
> >>>>>>>
> >>>>>>>     >  File
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
> >>>>>>> line 161, in start
> >>>>>>>
> >>>>>>>     >    instance_name, capture_output=capture_output,
> wait=wait)
> >>>>>>>
> >>>>>>>     >  File
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>>>> line 294, in start
> >>>>>>>
> >>>>>>>     >    skip_output=not capture_output)
> >>>>>>>
> >>>>>>>     >  File 
> >>>>>>>
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>>>> line 542, in run
> >>>>>>>
> >>>>>>>     >    raise CalledProcessError(p.returncode,
> arg_string,
> >>>>>>> str(output))
> >>>>>>>
> >>>>>>>     >CalledProcessError: Command '/bin/systemctl
> start
> >>>>>>>
> >>>>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>>> status
> >>>>>>> 1
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >2018-04-30T14:36:15Z DEBUG   [error]
> CalledProcessError:
> >>>>>>> Command '/bin/systemctl start
> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>>> status
> >>>>>>> 1
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors
> I show:
> >>>>>>>
> >>>>>>>     >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>>>>> slapd_bootstrap_config
> >>>>>>> - The default password storage scheme could not be read or
> was not
> >>>>>>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif.
> It is
> >>>>>>> mandatory.
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >Checking on internet I show that "dse.ldif"
> could be 
> >>>>>>> corrupted,
> >>>>>>> so I changed with "dse.ldif.startOK" without any
> change and then I
> >>>>>>> changed with "dse.ldif.bak". The problem persist
> but the error 
> >>>>>>> has changed:
> >>>>>>>
> >>>>>>>     >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>>>>> config_set_port
> >>>>>>> -
> >>>>>>>
> >>>>>>>     >Non-Secure Port Disabled
> >>>>>>>
> >>>>>>>     >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>>>>> symload_report_error
> >>>>>>> -
> >>>>>>>
> >>>>>>>     >Netscape Portable Runtime error -5975:
> >>>>>>>
> >>>>>>> >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
> undefined
> >>>>>>> symbol:
> >>>>>>>
> >>>>>>>     >replication_legacy_plugin_init
> >>>>>>>
> >>>>>>>     >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>>>>> symload_report_error
> >>>>>>> -
> >>>>>>>
> >>>>>>>     >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>>>
> >>>>>>>
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
> >>>>>>> Legacy
> >>>>>>>
> >>>>>>>     >Replication Plugin
> >>>>>>>
> >>>>>>>     >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>>>>> load_plugin_entry
> >>>>>>> - Unable to load plugin "cn=Legacy Replication
> >>>>>>> Plugin,cn=plugins,cn=config"
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >I saw a bug about this problem, but it is still
> opened:
> >>>>>>>
> >>>>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
> >>>>>>>> 2
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >Any idea how to fix the issue?
> >>>>>>>
> >>>>>>>     >
> >>>>>>>
> >>>>>>>     >If it is not possible to fix it, can I remove the
> replica from
> >>>>>>> IPA and install it again with the same name?
> >>>>>>>
> >>>>>>> A quick fix could be to remove an entry for cn=Legacy
> Replication
> >>>>>>> Plugin,cn=plugins,cn=config from
> >>>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>>>
> >>>>>>> when dirsrv is down.
> >>>>>>>
> >>>>>>> -- 
> >>>>>>>
> >>>>>>> / Alexander Bokovoy
> >>>>>>>
> >>>>>>> Sr. Principal Software Engineer
> >>>>>>>
> >>>>>>> Security / Identity Management Engineering Red Hat Limited,
> Finland
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> FreeIPA-users mailing list --
> freeipa-users(a)lists.fedorahosted.org
> >>>>>>> To unsubscribe send an email to
> >>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> FreeIPA-users mailing list --
> freeipa-users(a)lists.fedorahosted.org
> >>>>>> To unsubscribe send an email to
> >>>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>
> >>
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to 
> > freeipa-users-leave(a)lists.fedorahosted.org
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org

[SOLER SANGUESA Miguel]

Hello,

Thanks for your instructions, step to step seems it is improving. Unfoutunately it is not
solve yet, now I have this when I do an upgrade following your instructions:
# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/8]: saving configuration
  [2/8]: disabling listeners
  [3/8]: enabling DS global lock
  [4/8]: starting directory server
  [5/8]: updating schema
  [6/8]: upgrading server
  [7/8]: stopping directory server
  [8/8]: restoring configuration
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
Timeout exceeded
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information


On ipaupgrade.log I have:
....
2018-05-04T06:54:01Z DEBUG wait_for_open_ports: localhost [389] timeout 300
2018-05-04T06:54:01Z DEBUG waiting for port: 389
2018-05-04T06:54:01Z DEBUG Failed to connect to port 389 tcp on 127.0.0.1
2018-05-04T06:59:01Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
run command ipa-server-upgrade manually.
2018-05-04T06:59:01Z DEBUG   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
46, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1919, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
line 1586, in upgrade_configuration
    ds.start(ds_serverid)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py", line
641, in start
    super(DsInstance, self).start(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
401, in start
    self.service.start(instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
161, in start
    instance_name, capture_output=capture_output, wait=wait)
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
300, in start
    self.wait_for_open_ports(self.service_instance(instance_name))
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
270, in wait_for_open_ports
    self.api.env.startup_timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1256, in
wait_for_open_ports
    raise socket.timeout("Timeout exceeded")

2018-05-04T06:59:01Z DEBUG The ipa-server-upgrade command failed, exception: timeout:
Timeout exceeded
2018-05-04T06:59:01Z ERROR Timeout exceeded
2018-05-04T06:59:01Z ERROR The ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log for more information

And if I get the ports listening by ns-slapd, only upd port is listening:
# netstat -putan | grep 389
udp6       0      0 :::389                  :::*                               
17469/ns-slapd

Thanks & Regards.
______________________________
-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com&gt; 
Sent: Thursday, May 03, 2018 21:16
To: thierry bordaz <tbordaz(a)redhat.com>; SOLER SANGUESA Miguel
<solerm(a)unicc.org>; FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org&gt;
Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

thierry bordaz wrote:
>  Hi Soler,
>  
>  Thanks for the information.
>  So indexing is hanging because SC cache_init is running, the SC 
>  cache_init is hanging because SSSD is not started, SSSD is not started 
>  possibly because indexing prevents to get read access to the backend 
>  ("Backend is offline" TBC).
>  
>  An option would be to disable SC during upgrade phase ? Rob ?
>  
Sure, disabling SC would be a way to get things running again.

To do this:

1. Be sure dirsrv is not running
2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
4. change nsslapd-pluginEnabled from on to off 5. try the upgrade again

Assuming the upgrade is successful you can re-enable it post-install with:

# ipa-compat-manage enable

rob

>  
>  regards
>  thierry
>  
>  On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> > Hello Thierry,
> >
> > The version is: slapi-nis-0.56.0-8.el7.x86_64
> >
> > And the errors are:
> > [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin - 
> > group &quot;XXXXXX(a)ipa.example.org&quot; does not exist because SSSD is offline.
> > [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin - 
> > waiting for SSSD to become online...
> >
> > SSSD service is up, but as the ipa services are not (all) running, 
> > the ipa domain is down, logs when started sssd shows:
> > sssd[be[ipa.example.org]][823]: Backend is offline
> >
> > and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >   [sss_dp_get_reply] (0x0010): The Data Provider returned an error 
> > [org.freedesktop.sssd.Error.DataProvider.Offline]
> >
> > Thanks & Regards.
> >
> > -----Original Message-----
> > From: thierry bordaz <tbordaz(a)redhat.com&gt;
> > Sent: Thursday, May 03, 2018 11:25
> > To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden 
> > <rcritten(a)redhat.com>; FreeIPA users list 
> > <freeipa-users(a)lists.fedorahosted.org&gt;
> > Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> > Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> > 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >
> > me again ...
> >
> > If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the 
> > hanging period it could also indicate the reason what Schema compat 
> > was waiting for (SSSD ?)
> >
> > On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >> hello,
> >>
> >> Yesterday my ssh console closed the connection, so I had to start 
> >> again the "ipa-server-upgrade", but the result is more or less the
> same:
> >> # ipa-server-upgrade
> >> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>     [1/10]: stopping directory server
> >>     [2/10]: saving configuration
> >>     [3/10]: disabling listeners
> >>     [4/10]: enabling DS global lock
> >>     [5/10]: starting directory server
> >>     [6/10]: updating schema
> >>     [7/10]: upgrading server
> >>
> >> But now, the lines that are repeated on the access log are:
> >> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH 
> >>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> >> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >> tag=101 nentries=1 etime=1.0000687740
> >>
> >> And it is on the same state than the other:
> >> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket 
> >> -b 
> >> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
> SASL/EXTERNAL authentication started SASL username:
> >> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >> SASL SSF: 0
> >> # extended LDIF
> >> #
> >> # LDAPv3
> >> # base
> >> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>
> >> # indextask_l_137445500911864330_4055, index, tasks, config
> >> dn: 
> >> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >> objectClass: top
> >> objectClass: extensibleObject
> >> nsindexattribute: l
> >> nsinstance: userRoot
> >> cn: indextask_l_137445500911864330_4055
> >> nstaskcurrentitem: 0
> >> nstasktotalitems: 1
> >>
> >> # search result
> >> search: 3
> >> result: 0 Success
> >>
> >> # numResponses: 2
> >> # numEntries: 1
> >>
> >> The pstack of the proces: /usr/bin/python2 
> >> /usr/sbin/ipa-server-upgrade # pstack 9805
> >> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >> #1  0x00007fb8ff5e0000 in time_sleep () from 
> >> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >> /lib64/libpython2.7.so.1.0
> >> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >> /lib64/libpython2.7.so.1.0
> >> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >> /lib64/libpython2.7.so.1.0
> >> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >> /lib64/libpython2.7.so.1.0
> >> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >> #27 0x000000000040066e in _start ()
> >>
> >>
> >> And the pstack from the process: /usr/sbin/ns-slapd -D 
> >> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i 
> >> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >> # pstack 15536
> >> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49 
> >> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48 
> >> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47 
> >> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46 
> >> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45 
> >> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44 
> >> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43 
> >> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42 
> >> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41 
> >> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40 
> >> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39 
> >> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38 
> >> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37 
> >> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36 
> >> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35 
> >> (Thread 0x7f5bb92da700 (LWP 15554)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34 
> >> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >> /lib64/libevent-2.0.so.5
> >> #2  0x00007f5bd1d883ea in event_base_loop () at
> >> /lib64/libevent-2.0.so.5
> >> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33 
> >> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32 
> >> (Thread 0x7f5bb2383700 (LWP 15560)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31 
> >> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e420c7 in trickle_threadmain () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30 
> >> (Thread 0x7f5bb1381700 (LWP 15562)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e3ce47 in perf_threadmain () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29 
> >> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at 
> >> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28 
> >> (Thread 0x7f5bb037f700 (LWP 15564)):
> >> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at 
> >> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27 
> >> (Thread 0x7f5bd5132700 (LWP 15566)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3572033 in _cl5TrimMain () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26 
> >> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3587fa4 in protocol_sleep () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bc358b75a in repl5_inc_run () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #5  0x00007f5bc358fbec in prot_thread_main () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25 
> >> (Thread 0x7f5baf37d700 (LWP 15568)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bc3587fa4 in protocol_sleep () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #4  0x00007f5bc358b75a in repl5_inc_run () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #5  0x00007f5bc358fbec in prot_thread_main () at 
> >> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24 
> >> (Thread 0x7f5bae759700 (LWP 15569)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23 
> >> (Thread 0x7f5badf58700 (LWP 15570)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22 
> >> (Thread 0x7f5bad757700 (LWP 15571)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at 
> >> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21 
> >> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x0000564591e10b08 in housecleaning ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20 
> >> (Thread 0x7f5bac54e700 (LWP 15573)):
> >> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19 
> >> (Thread 0x7f5babd4d700 (LWP 15574)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18 
> >> (Thread 0x7f5bab54c700 (LWP 15575)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17 
> >> (Thread 0x7f5baad4b700 (LWP 15576)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16 
> >> (Thread 0x7f5baa54a700 (LWP 15577)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15 
> >> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14 
> >> (Thread 0x7f5ba9548700 (LWP 15579)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13 
> >> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12 
> >> (Thread 0x7f5ba8546700 (LWP 15581)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11 
> >> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10 
> >> (Thread 0x7f5ba7544700 (LWP 15583)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9 
> >> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at 
> >> /lib64/libnspr4.so
> >> #2  0x0000564591e07049 in connection_read_operation ()
> >> #3  0x0000564591e0765d in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8 
> >> (Thread 0x7f5ba6542700 (LWP 15585)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7 
> >> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6 
> >> (Thread 0x7f5ba5540700 (LWP 15587)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5 
> >> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4 
> >> (Thread 0x7f5ba453e700 (LWP 15589)):
> >> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >> ()
> >> #3  0x0000564591e075d7 in connection_threadmain ()
> >> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3 
> >> (Thread 0x7f5ba3900700 (LWP 15590)):
> >> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at 
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at 
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at 
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at 
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () 
> >> at /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at 
> >> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2 
> >> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >> /lib64/libpthread.so.0
> >> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1 
> >> (Thread 0x7f5bd5135940 (LWP 15536)):
> >> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >> /usr/lib64/dirsrv/libnunc-stans.so.0
> >> #2  0x0000564591e0d862 in slapd_daemon ()
> >> #3  0x0000564591dfe86f in main ()
> >>
> >> Thanks & Regards.
> >> ______________________________
> >>
> >> -----Original Message-----
> >> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >> Sent: Thursday, May 03, 2018 9:31
> >> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list 
> >> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel 
> >> <solerm(a)unicc.org&gt;
> >> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating 
> >> from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> Hi,
> >>
> >> During indexing task we should see in the task status the periodic 
> >> progression of the indexing.
> >> May be the indexing is hanging somewhere. When the problem occurs 
> >> could you provide a pstack of the dirsrv server ?
> >>
> >> best regards
> >> thierry
> >>
> >> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>> Hello,
> >>>>
> >>>> This is the output of the command (seems that is not complete):
> >>>>
> >>>> # ldapsearch -H 
> >>>> ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>> -b
> >>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>> ,c
> >>>> n
> >>>> =config
> >>>> -s base
> >>>> SASL/EXTERNAL authentication started SASL username:
> >>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>> SASL SSF: 0
> >>>> # extended LDIF
> >>>> #
> >>>> # LDAPv3
> >>>> # base
> >>>> <cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>> s,
> >>>> c
> >>>> n=config>
> >>>> with scope baseObject
> >>>> # filter: (objectclass=*)
> >>>> # requesting: ALL
> >>>> #
> >>>>
> >>>> # indextask_description_137444551994158920_5958, index, tasks, 
> >>>> config
> >>>> dn:
> >>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>> ,c
> >>>> n
> >>>> =conf
> >>>>     ig
> >>>> objectClass: top
> >>>> objectClass: extensibleObject
> >>>> nsindexattribute: description
> >>>> nsinstance: userRoot
> >>>> cn: indextask_description_137444551994158920_5958
> >>>> nstaskcurrentitem: 0
> >>>> nstasktotalitems: 1
> >>> IPA is looking for nstaskstatus and since that isn't there it is 
> >>> looping forever.
> >>>
> >>> I've cc'd a couple of the 389-ds devs to see what they think about
> it.
> >>>
> >>> rob
> >>>
> >>>> # search result
> >>>> search: 3
> >>>> result: 0 Success
> >>>>
> >>>> # numResponses: 2
> >>>> # numEntries: 1
> >>>>
> >>>> Thanks & Regards.
> >>>> ______________________________
> >>>> -----Original Message-----
> >>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>> Sent: Tuesday, May 01, 2018 15:18
> >>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating 
> >>>> from
> >>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>
> >>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>> Hello,
> >>>>>
> >>>>> Thank you for your answer, now dirsrv can start, but after 
> >>>>> running the "
> >>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>
> >>>>> # ipa-server-upgrade
> >>>>>
> >>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>
> >>>>>       [1/8]: saving configuration
> >>>>>
> >>>>>       [2/8]: disabling listeners
> >>>>>
> >>>>>       [3/8]: enabling DS global lock
> >>>>>
> >>>>>       [4/8]: starting directory server
> >>>>>
> >>>>>       [5/8]: updating schema
> >>>>>
> >>>>>       [6/8]: upgrading server
> >>>>>
> >>>>> The latest messages on the log is (there are no logs on the lasts
> >>>>> 2h):
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>> database,cn=plugins,cn=config
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>
> >>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>
> >>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>> description
> >>>>>
> >>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>>> s,
> >>>>> c
> >>>>> n=
> >>>>> config
> >>>>>
> >>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>
> >>>>> # ipactl status
> >>>>>
> >>>>> Directory Service: RUNNING
> >>>>>
> >>>>> krb5kdc Service: STOPPED
> >>>>>
> >>>>> kadmin Service: STOPPED
> >>>>>
> >>>>> named Service: STOPPED
> >>>>>
> >>>>> httpd Service: STOPPED
> >>>>>
> >>>>> ipa-custodia Service: STOPPED
> >>>>>
> >>>>> ntpd Service: RUNNING
> >>>>>
> >>>>> pki-tomcatd Service: STOPPED
> >>>>>
> >>>>> smb Service: STOPPED
> >>>>>
> >>>>> winbind Service: STOPPED
> >>>>>
> >>>>> ipa-otpd Service: STOPPED
> >>>>>
> >>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>
> >>>>> ipa: INFO: The ipactl command was successful
> >>>>>
> >>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>> It depends on how many entries you have. This is an indexing task 
> >>>> and while 2 hours seems a bit excessive, it depends on how much 
> >>>> work it has to do.
> >>>>
> >>>> The 389-ds access and/or error logs may provide details.
> >>>>
> >>>> rob
> >>>>
> >>>>> Thanks.
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating from
> >>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>
> >>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users
> wrote:
> >>>>>
> >>>>>     >hello,
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >I have an IPA master that updated from 4.5.0 (RHEL 7.4) to
> >>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the 
> >>>>> unique replica I have, but after update dirsrv is not starting.
> >>>>>
> >>>>>     >It says it is needed run "ipa-server-upgrade", but
> it also
> >>>>> fails:
> >>>>>
> >>>>>     ># ipactl start
> >>>>>
> >>>>>     >Upgrade required: please run ipa-server-upgrade command 
> >>>>> Aborting ipactl
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     ># ipa-server-upgrade
> >>>>>
> >>>>>     >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>
> >>>>>     >  [1/8]: saving configuration
> >>>>>
> >>>>>     >  [2/8]: disabling listeners
> >>>>>
> >>>>>     >  [3/8]: enabling DS global lock
> >>>>>
> >>>>>     >  [4/8]: starting directory server
> >>>>>
> >>>>>     >  [error] CalledProcessError: Command '/bin/systemctl
> start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMOLE-ORG.service 
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit 
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >  [cleanup]: stopping directory server
> >>>>>
> >>>>>     >  [cleanup]: restoring configuration
> >>>>>
> >>>>>     >IPA server upgrade failed: Inspect /var/log/ipaupgrade.log 
> >>>>> and run command ipa-server-upgrade manually.
> >>>>>
> >>>>>     >Unexpected error - see /var/log/ipaupgrade.log for details:
> >>>>>
> >>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit 
> >>>>> status
> >>>>> 1 The
> >>>>>
> >>>>>     >ipa-server-upgrade command failed. See 
> >>>>> /var/log/ipaupgrade.log for more
> >>>>>
> >>>>>     >information
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >On the log I can see:
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>
> >>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>
> >>>>>     >finished, return code=3 2018-04-30T14:36:15Z DEBUG 
> >>>>> stdout=failed ...
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory 
> >>>>> server
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>
> >>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>
> >>>>>     >finished, return code=1 2018-04-30T14:36:15Z DEBUG stdout=
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG stderr=Job for 
> >>>>> dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>
> >>>>> failed because the control process exited with error code. See 
> >>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for 
> >>>>> details.
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG Traceback (most recent call
> last):
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>> line 504, in start_creation
> >>>>>
> >>>>>     >    run_step(full_msg, method)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>> line 494, in run_step
> >>>>>
> >>>>>     >    method()
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>>>> py
> >>>>> ",
> >>>>> line 95, in __start
> >>>>>
> >>>>>     >    srv.start(self.serverid, ldapi=True)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py"
> >>>>> ,
> >>>>> line 161, in start
> >>>>>
> >>>>>     >    instance_name, capture_output=capture_output,
> wait=wait)
> >>>>>
> >>>>>     >  File
> >>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>> line 294, in start
> >>>>>
> >>>>>     >    skip_output=not capture_output)
> >>>>>
> >>>>>     >  File 
> >>>>> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>> line 542, in run
> >>>>>
> >>>>>     >    raise CalledProcessError(p.returncode, arg_string,
> >>>>> str(output))
> >>>>>
> >>>>>     >CalledProcessError: Command '/bin/systemctl start
> >>>>>
> >>>>>     &gt;dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit 
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError:
> >>>>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service 
> >>>>> <mailto:[e-mail]>' returned non-zero
> exit 
> >>>>> status
> >>>>> 1
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I
> show:
> >>>>>
> >>>>>     >[30/Apr/2018:16:04:52.584220922 +0200] - ERR - 
> >>>>> slapd_bootstrap_config
> >>>>> - The default password storage scheme could not be read or was 
> >>>>> not found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. 
> >>>>> It is mandatory.
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >Checking on internet I show that "dse.ldif" could
> be 
> >>>>> corrupted, so I changed with "dse.ldif.startOK" without any
>
> >>>>> change and then I changed with "dse.ldif.bak". The problem
>
> >>>>> persist but the error has
> >>>>> changed:
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - 
> >>>>> config_set_port
> >>>>> -
> >>>>>
> >>>>>     >Non-Secure Port Disabled
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.556581301 +0200] - ERR - 
> >>>>> symload_report_error
> >>>>> -
> >>>>>
> >>>>>     >Netscape Portable Runtime error -5975:
> >>>>>
> >>>>>     >/usr/lib64/dirsrv/plugins/libreplication-plugin.so: 
> >>>>> undefined
> >>>>> symbol:
> >>>>>
> >>>>>     >replication_legacy_plugin_init
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.561590553 +0200] - ERR - 
> >>>>> symload_report_error
> >>>>> -
> >>>>>
> >>>>>     >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>
> >>>>>    
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for 
> >>>>> plugin Legacy
> >>>>>
> >>>>>     >Replication Plugin
> >>>>>
> >>>>>     >[30/Apr/2018:16:32:13.564590264 +0200] - ERR - 
> >>>>> load_plugin_entry
> >>>>> - Unable to load plugin "cn=Legacy Replication 
> >>>>> Plugin,cn=plugins,cn=config"
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >I saw a bug about this problem, but it is still opened:
> >>>>>
> >>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529
> >>>>>> 44
> >>>>>> 2
> >>>>>     >
> >>>>>
> >>>>>     >Any idea how to fix the issue?
> >>>>>
> >>>>>     >
> >>>>>
> >>>>>     >If it is not possible to fix it, can I remove the replica 
> >>>>> from IPA and install it again with the same name?
> >>>>>
> >>>>> A quick fix could be to remove an entry for cn=Legacy Replication 
> >>>>> Plugin,cn=plugins,cn=config from 
> >>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>
> >>>>> when dirsrv is down.
> >>>>>
> >>>>> --
> >>>>>
> >>>>> / Alexander Bokovoy
> >>>>>
> >>>>> Sr. Principal Software Engineer
> >>>>>
> >>>>> Security / Identity Management Engineering Red Hat Limited, 
> >>>>> Finland
> >>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- 
> >>>>> freeipa-users(a)lists.fedorahosted.org
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>> To unsubscribe send an email to
> >>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>
>  
>

[Rob Crittenden]

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  Thanks for your instructions, step to step seems it is improving. Unfoutunately it is not
> solve yet, now I have this when I do an upgrade following your instructions:
>  # ipa-server-upgrade
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>     [1/8]: saving configuration
>     [2/8]: disabling listeners
>     [3/8]: enabling DS global lock
>     [4/8]: starting directory server
>     [5/8]: updating schema
>     [6/8]: upgrading server
>     [7/8]: stopping directory server
>     [8/8]: restoring configuration
>  Done.
>  Update complete
>  Upgrading IPA services
>  Upgrading the configuration of the IPA services
>  IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
> ipa-server-upgrade manually.
>  Timeout exceeded
>  The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Ok, make sure dirsrv is not running (VERY important), then:

1. edit /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
2. set nsslapd-port to 389
3. set nsslapd-security to on
4. set nsslapd-global-backend-lock to off (if you have this attribute at 
all)
5. ipa-server-upgrade

rob

>  
>  
>  On ipaupgrade.log I have:
>  ....
>  2018-05-04T06:54:01Z DEBUG wait_for_open_ports: localhost [389] timeout 300
>  2018-05-04T06:54:01Z DEBUG waiting for port: 389
>  2018-05-04T06:54:01Z DEBUG Failed to connect to port 389 tcp on 127.0.0.1
>  2018-05-04T06:59:01Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> run command ipa-server-upgrade manually.
>  2018-05-04T06:59:01Z DEBUG   File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
>       return_value = self.run()
>     File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
> 46, in run
>       server.upgrade()
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1919, in upgrade
>       upgrade_configuration()
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1586, in upgrade_configuration
>       ds.start(ds_serverid)
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py",
> line 641, in start
>       super(DsInstance, self).start(*args, **kwargs)
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> 401, in start
>       self.service.start(instance_name, capture_output=capture_output, wait=wait)
>     File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
> 161, in start
>       instance_name, capture_output=capture_output, wait=wait)
>     File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> 300, in start
>       self.wait_for_open_ports(self.service_instance(instance_name))
>     File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> 270, in wait_for_open_ports
>       self.api.env.startup_timeout)
>     File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1256, in
> wait_for_open_ports
>       raise socket.timeout("Timeout exceeded")
>  
>  2018-05-04T06:59:01Z DEBUG The ipa-server-upgrade command failed, exception: timeout:
> Timeout exceeded
>  2018-05-04T06:59:01Z ERROR Timeout exceeded
>  2018-05-04T06:59:01Z ERROR The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>  
>  And if I get the ports listening by ns-slapd, only upd port is listening:
>  # netstat -putan | grep 389
>  udp6       0      0 :::389                  :::*                               
> 17469/ns-slapd
>  
>  Thanks & Regards.
>  ______________________________
>  -----Original Message-----
>  From: Rob Crittenden <rcritten(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 21:16
>  To: thierry bordaz <tbordaz(a)redhat.com>; SOLER SANGUESA Miguel
> <solerm(a)unicc.org>; FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>  
>  thierry bordaz wrote:
> > Hi Soler,
> >
> > Thanks for the information.
> > So indexing is hanging because SC cache_init is running, the SC
> > cache_init is hanging because SSSD is not started, SSSD is not started
> > possibly because indexing prevents to get read access to the backend
> > ("Backend is offline" TBC).
> >
> > An option would be to disable SC during upgrade phase ? Rob ?
>  
>  Sure, disabling SC would be a way to get things running again.
>  
>  To do this:
>  
>  1. Be sure dirsrv is not running
>  2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
>  3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
>  4. change nsslapd-pluginEnabled from on to off 5. try the upgrade again
>  
>  Assuming the upgrade is successful you can re-enable it post-install with:
>  
>  # ipa-compat-manage enable
>  
>  rob
>  
> >
> > regards
> > thierry
> >
> > On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> >> Hello Thierry,
> >>
> >> The version is: slapi-nis-0.56.0-8.el7.x86_64
> >>
> >> And the errors are:
> >> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin -
> >> group &quot;XXXXXX(a)ipa.example.org&quot; does not exist because SSSD is offline.
> >> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin -
> >> waiting for SSSD to become online...
> >>
> >> SSSD service is up, but as the ipa services are not (all) running,
> >> the ipa domain is down, logs when started sssd shows:
> >> sssd[be[ipa.example.org]][823]: Backend is offline
> >>
> >> and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >>    [sss_dp_get_reply] (0x0010): The Data Provider returned an error
> >> [org.freedesktop.sssd.Error.DataProvider.Offline]
> >>
> >> Thanks & Regards.
> >>
> >> -----Original Message-----
> >> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >> Sent: Thursday, May 03, 2018 11:25
> >> To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden
> >> <rcritten(a)redhat.com>; FreeIPA users list
> >> <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> me again ...
> >>
> >> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the
> >> hanging period it could also indicate the reason what Schema compat
> >> was waiting for (SSSD ?)
> >>
> >> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >>> hello,
> >>>
> >>> Yesterday my ssh console closed the connection, so I had to start
> >>> again the "ipa-server-upgrade", but the result is more or less the
> same:
> >>> # ipa-server-upgrade
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>      [1/10]: stopping directory server
> >>>      [2/10]: saving configuration
> >>>      [3/10]: disabling listeners
> >>>      [4/10]: enabling DS global lock
> >>>      [5/10]: starting directory server
> >>>      [6/10]: updating schema
> >>>      [7/10]: upgrading server
> >>>
> >>> But now, the lines that are repeated on the access log are:
> >>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> >>>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> >>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >>> tag=101 nentries=1 etime=1.0000687740
> >>>
> >>> And it is on the same state than the other:
> >>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>> -b
> >>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
> SASL/EXTERNAL authentication started SASL username:
> >>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>> SASL SSF: 0
> >>> # extended LDIF
> >>> #
> >>> # LDAPv3
> >>> # base
> >>> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >>> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>>
> >>> # indextask_l_137445500911864330_4055, index, tasks, config
> >>> dn:
> >>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >>> objectClass: top
> >>> objectClass: extensibleObject
> >>> nsindexattribute: l
> >>> nsinstance: userRoot
> >>> cn: indextask_l_137445500911864330_4055
> >>> nstaskcurrentitem: 0
> >>> nstasktotalitems: 1
> >>>
> >>> # search result
> >>> search: 3
> >>> result: 0 Success
> >>>
> >>> # numResponses: 2
> >>> # numEntries: 1
> >>>
> >>> The pstack of the proces: /usr/bin/python2
> >>> /usr/sbin/ipa-server-upgrade # pstack 9805
> >>> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >>> #1  0x00007fb8ff5e0000 in time_sleep () from
> >>> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >>> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >>> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >>> #27 0x000000000040066e in _start ()
> >>>
> >>>
> >>> And the pstack from the process: /usr/sbin/ns-slapd -D
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> >>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >>> # pstack 15536
> >>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> >>> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> >>> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> >>> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> >>> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> >>> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> >>> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> >>> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> >>> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> >>> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> >>> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> >>> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> >>> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> >>> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> >>> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> >>> (Thread 0x7f5bb92da700 (LWP 15554)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> >>> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >>> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >>> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >>> /lib64/libevent-2.0.so.5
> >>> #2  0x00007f5bd1d883ea in event_base_loop () at
> >>> /lib64/libevent-2.0.so.5
> >>> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> >>> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> >>> (Thread 0x7f5bb2383700 (LWP 15560)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> >>> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> >>> (Thread 0x7f5bb1381700 (LWP 15562)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> >>> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> >>> (Thread 0x7f5bb037f700 (LWP 15564)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> >>> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> >>> (Thread 0x7f5bd5132700 (LWP 15566)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3572033 in _cl5TrimMain () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> >>> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> >>> (Thread 0x7f5baf37d700 (LWP 15568)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> >>> (Thread 0x7f5bae759700 (LWP 15569)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> >>> (Thread 0x7f5badf58700 (LWP 15570)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> >>> (Thread 0x7f5bad757700 (LWP 15571)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> >>> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x0000564591e10b08 in housecleaning ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> >>> (Thread 0x7f5bac54e700 (LWP 15573)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> >>> (Thread 0x7f5babd4d700 (LWP 15574)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> >>> (Thread 0x7f5bab54c700 (LWP 15575)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> >>> (Thread 0x7f5baad4b700 (LWP 15576)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> >>> (Thread 0x7f5baa54a700 (LWP 15577)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> >>> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> >>> (Thread 0x7f5ba9548700 (LWP 15579)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> >>> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> >>> (Thread 0x7f5ba8546700 (LWP 15581)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> >>> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> >>> (Thread 0x7f5ba7544700 (LWP 15583)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> >>> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at
> >>> /lib64/libnspr4.so
> >>> #2  0x0000564591e07049 in connection_read_operation ()
> >>> #3  0x0000564591e0765d in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> >>> (Thread 0x7f5ba6542700 (LWP 15585)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> >>> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> >>> (Thread 0x7f5ba5540700 (LWP 15587)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> >>> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> >>> (Thread 0x7f5ba453e700 (LWP 15589)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> >>> (Thread 0x7f5ba3900700 (LWP 15590)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb ()
> >>> at /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> >>> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >>> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> >>> (Thread 0x7f5bd5135940 (LWP 15536)):
> >>> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x0000564591e0d862 in slapd_daemon ()
> >>> #3  0x0000564591dfe86f in main ()
> >>>
> >>> Thanks & Regards.
> >>> ______________________________
> >>>
> >>> -----Original Message-----
> >>> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >>> Sent: Thursday, May 03, 2018 9:31
> >>> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> >>> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> >>> <solerm(a)unicc.org&gt;
> >>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>> from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> Hi,
> >>>
> >>> During indexing task we should see in the task status the periodic
> >>> progression of the indexing.
> >>> May be the indexing is hanging somewhere. When the problem occurs
> >>> could you provide a pstack of the dirsrv server ?
> >>>
> >>> best regards
> >>> thierry
> >>>
> >>> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>> Hello,
> >>>>>
> >>>>> This is the output of the command (seems that is not complete):
> >>>>>
> >>>>> # ldapsearch -H
> >>>>> ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>>> -b
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>>> ,c
> >>>>> n
> >>>>> =config
> >>>>> -s base
> >>>>> SASL/EXTERNAL authentication started SASL username:
> >>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>>> SASL SSF: 0
> >>>>> # extended LDIF
> >>>>> #
> >>>>> # LDAPv3
> >>>>> # base
> >>>>>
> <cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>>> s,
> >>>>> c
> >>>>> n=config>
> >>>>> with scope baseObject
> >>>>> # filter: (objectclass=*)
> >>>>> # requesting: ALL
> >>>>> #
> >>>>>
> >>>>> # indextask_description_137444551994158920_5958, index, tasks,
> >>>>> config
> >>>>> dn:
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>>> ,c
> >>>>> n
> >>>>> =conf
> >>>>>      ig
> >>>>> objectClass: top
> >>>>> objectClass: extensibleObject
> >>>>> nsindexattribute: description
> >>>>> nsinstance: userRoot
> >>>>> cn: indextask_description_137444551994158920_5958
> >>>>> nstaskcurrentitem: 0
> >>>>> nstasktotalitems: 1
> >>>> IPA is looking for nstaskstatus and since that isn't there it is
> >>>> looping forever.
> >>>>
> >>>> I've cc'd a couple of the 389-ds devs to see what they think
> about it.
> >>>>
> >>>> rob
> >>>>
> >>>>> # search result
> >>>>> search: 3
> >>>>> result: 0 Success
> >>>>>
> >>>>> # numResponses: 2
> >>>>> # numEntries: 1
> >>>>>
> >>>>> Thanks & Regards.
> >>>>> ______________________________
> >>>>> -----Original Message-----
> >>>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>>> Sent: Tuesday, May 01, 2018 15:18
> >>>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>>>> from
> >>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>
> >>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> Thank you for your answer, now dirsrv can start, but after
> >>>>>> running the "
> >>>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>>
> >>>>>> # ipa-server-upgrade
> >>>>>>
> >>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>        [1/8]: saving configuration
> >>>>>>
> >>>>>>        [2/8]: disabling listeners
> >>>>>>
> >>>>>>        [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>        [4/8]: starting directory server
> >>>>>>
> >>>>>>        [5/8]: updating schema
> >>>>>>
> >>>>>>        [6/8]: upgrading server
> >>>>>>
> >>>>>> The latest messages on the log is (there are no logs on the
> lasts
> >>>>>> 2h):
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>>> ---------------------------------------------
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>>> database,cn=plugins,cn=config
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>>> description
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>>>> s,
> >>>>>> c
> >>>>>> n=
> >>>>>> config
> >>>>>>
> >>>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>>
> >>>>>> # ipactl status
> >>>>>>
> >>>>>> Directory Service: RUNNING
> >>>>>>
> >>>>>> krb5kdc Service: STOPPED
> >>>>>>
> >>>>>> kadmin Service: STOPPED
> >>>>>>
> >>>>>> named Service: STOPPED
> >>>>>>
> >>>>>> httpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-custodia Service: STOPPED
> >>>>>>
> >>>>>> ntpd Service: RUNNING
> >>>>>>
> >>>>>> pki-tomcatd Service: STOPPED
> >>>>>>
> >>>>>> smb Service: STOPPED
> >>>>>>
> >>>>>> winbind Service: STOPPED
> >>>>>>
> >>>>>> ipa-otpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>>
> >>>>>> ipa: INFO: The ipactl command was successful
> >>>>>>
> >>>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>>> It depends on how many entries you have. This is an indexing task
> >>>>> and while 2 hours seems a bit excessive, it depends on how much
> >>>>> work it has to do.
> >>>>>
> >>>>> The 389-ds access and/or error logs may provide details.
> >>>>>
> >>>>> rob
> >>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>>> To: FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating
> from
> >>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>>
> >>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users
> wrote:
> >>>>>>
> >>>>>>      >hello,
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >I have an IPA master that updated from 4.5.0 (RHEL 7.4)
> to
> >>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the
> >>>>>> unique replica I have, but after update dirsrv is not starting.
> >>>>>>
> >>>>>>      >It says it is needed run "ipa-server-upgrade",
> but it also
> >>>>>> fails:
> >>>>>>
> >>>>>>      ># ipactl start
> >>>>>>
> >>>>>>      >Upgrade required: please run ipa-server-upgrade command
> >>>>>> Aborting ipactl
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      ># ipa-server-upgrade
> >>>>>>
> >>>>>>      >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>      >  [1/8]: saving configuration
> >>>>>>
> >>>>>>      >  [2/8]: disabling listeners
> >>>>>>
> >>>>>>      >  [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>      >  [4/8]: starting directory server
> >>>>>>
> >>>>>>      >  [error] CalledProcessError: Command
> '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >  [cleanup]: stopping directory server
> >>>>>>
> >>>>>>      >  [cleanup]: restoring configuration
> >>>>>>
> >>>>>>      >IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log
> >>>>>> and run command ipa-server-upgrade manually.
> >>>>>>
> >>>>>>      >Unexpected error - see /var/log/ipaupgrade.log for
> details:
> >>>>>>
> >>>>>>      >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1 The
> >>>>>>
> >>>>>>      >ipa-server-upgrade command failed. See
> >>>>>> /var/log/ipaupgrade.log for more
> >>>>>>
> >>>>>>      >information
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >On the log I can see:
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl
> is-active
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>      >finished, return code=3 2018-04-30T14:36:15Z DEBUG
> >>>>>> stdout=failed ...
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory
> >>>>>> server
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>      >finished, return code=1 2018-04-30T14:36:15Z DEBUG
> stdout=
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> failed because the control process exited with error code. See
> >>>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for
> >>>>>> details.
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Traceback (most recent call
> last):
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 504, in start_creation
> >>>>>>
> >>>>>>      >    run_step(full_msg, method)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 494, in run_step
> >>>>>>
> >>>>>>      >    method()
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>>>>> py
> >>>>>> ",
> >>>>>> line 95, in __start
> >>>>>>
> >>>>>>      >    srv.start(self.serverid, ldapi=True)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py"
> >>>>>> ,
> >>>>>> line 161, in start
> >>>>>>
> >>>>>>      >    instance_name, capture_output=capture_output,
> wait=wait)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>>> line 294, in start
> >>>>>>
> >>>>>>      >    skip_output=not capture_output)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>>> line 542, in run
> >>>>>>
> >>>>>>      >    raise CalledProcessError(p.returncode, arg_string,
> >>>>>> str(output))
> >>>>>>
> >>>>>>      >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG   [error]
> CalledProcessError:
> >>>>>> Command '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I
> show:
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>>>> slapd_bootstrap_config
> >>>>>> - The default password storage scheme could not be read or was
> >>>>>> not found in the file
> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif.
> >>>>>> It is mandatory.
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >Checking on internet I show that "dse.ldif"
> could be
> >>>>>> corrupted, so I changed with "dse.ldif.startOK" without
> any
> >>>>>> change and then I changed with "dse.ldif.bak". The
> problem
> >>>>>> persist but the error has
> >>>>>> changed:
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>>>> config_set_port
> >>>>>> -
> >>>>>>
> >>>>>>      >Non-Secure Port Disabled
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>      >Netscape Portable Runtime error -5975:
> >>>>>>
> >>>>>>      >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
> >>>>>> undefined
> >>>>>> symbol:
> >>>>>>
> >>>>>>      >replication_legacy_plugin_init
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>      >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>>
> >>>>>>     
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> >>>>>> plugin Legacy
> >>>>>>
> >>>>>>      >Replication Plugin
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>>>> load_plugin_entry
> >>>>>> - Unable to load plugin "cn=Legacy Replication
> >>>>>> Plugin,cn=plugins,cn=config"
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >I saw a bug about this problem, but it is still opened:
> >>>>>>
> >>>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529
> >>>>>>> 44
> >>>>>>> 2
> >>>>>>      >
> >>>>>>
> >>>>>>      >Any idea how to fix the issue?
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >If it is not possible to fix it, can I remove the
> replica
> >>>>>> from IPA and install it again with the same name?
> >>>>>>
> >>>>>> A quick fix could be to remove an entry for cn=Legacy
> Replication
> >>>>>> Plugin,cn=plugins,cn=config from
> >>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>>
> >>>>>> when dirsrv is down.
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> / Alexander Bokovoy
> >>>>>>
> >>>>>> Sr. Principal Software Engineer
> >>>>>>
> >>>>>> Security / Identity Management Engineering Red Hat Limited,
> >>>>>> Finland
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> FreeIPA-users mailing list --
> >>>>>> freeipa-users(a)lists.fedorahosted.org
> >>>>>> To unsubscribe send an email to
> >>>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>
> >
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>

[SOLER SANGUESA Miguel]

Thank you so much. It worked now, and now I can start ipa services.

Thanks & Regards.
______________________________

-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com&gt; 
Sent: Friday, May 04, 2018 15:01
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
Cc: Ludwig Krispenz <lkrispen(a)redhat.com>; thierry bordaz
<tbordaz(a)redhat.com>; SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
4.5.4 (RHEL 7.5)

SOLER SANGUESA Miguel via FreeIPA-users wrote:
>  Hello,
>  
>  Thanks for your instructions, step to step seems it is improving. Unfoutunately it is not
> solve yet, now I have this when I do an upgrade following your instructions:
>  # ipa-server-upgrade
>  Upgrading IPA:. Estimated time: 1 minute 30 seconds
>     [1/8]: saving configuration
>     [2/8]: disabling listeners
>     [3/8]: enabling DS global lock
>     [4/8]: starting directory server
>     [5/8]: updating schema
>     [6/8]: upgrading server
>     [7/8]: stopping directory server
>     [8/8]: restoring configuration
>  Done.
>  Update complete
>  Upgrading IPA services
>  Upgrading the configuration of the IPA services
>  IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
> ipa-server-upgrade manually.
>  Timeout exceeded
>  The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
>  
Ok, make sure dirsrv is not running (VERY important), then:

1. edit /etc/dirsrv/slapd-EXAMPLE-COM/dse.ldif
2. set nsslapd-port to 389
3. set nsslapd-security to on
4. set nsslapd-global-backend-lock to off (if you have this attribute at 
all)
5. ipa-server-upgrade

rob

>  
>  
>  On ipaupgrade.log I have:
>  ....
>  2018-05-04T06:54:01Z DEBUG wait_for_open_ports: localhost [389] timeout 300
>  2018-05-04T06:54:01Z DEBUG waiting for port: 389
>  2018-05-04T06:54:01Z DEBUG Failed to connect to port 389 tcp on 127.0.0.1
>  2018-05-04T06:59:01Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> run command ipa-server-upgrade manually.
>  2018-05-04T06:59:01Z DEBUG   File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in
> execute
>       return_value = self.run()
>     File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
> 46, in run
>       server.upgrade()
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1919, in upgrade
>       upgrade_configuration()
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 1586, in upgrade_configuration
>       ds.start(ds_serverid)
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/dsinstance.py",
> line 641, in start
>       super(DsInstance, self).start(*args, **kwargs)
>     File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line
> 401, in start
>       self.service.start(instance_name, capture_output=capture_output, wait=wait)
>     File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line
> 161, in start
>       instance_name, capture_output=capture_output, wait=wait)
>     File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> 300, in start
>       self.wait_for_open_ports(self.service_instance(instance_name))
>     File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line
> 270, in wait_for_open_ports
>       self.api.env.startup_timeout)
>     File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1256, in
> wait_for_open_ports
>       raise socket.timeout("Timeout exceeded")
>  
>  2018-05-04T06:59:01Z DEBUG The ipa-server-upgrade command failed, exception: timeout:
> Timeout exceeded
>  2018-05-04T06:59:01Z ERROR Timeout exceeded
>  2018-05-04T06:59:01Z ERROR The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>  
>  And if I get the ports listening by ns-slapd, only upd port is listening:
>  # netstat -putan | grep 389
>  udp6       0      0 :::389                  :::*                               
> 17469/ns-slapd
>  
>  Thanks & Regards.
>  ______________________________
>  -----Original Message-----
>  From: Rob Crittenden <rcritten(a)redhat.com&gt;
>  Sent: Thursday, May 03, 2018 21:16
>  To: thierry bordaz <tbordaz(a)redhat.com>; SOLER SANGUESA Miguel
> <solerm(a)unicc.org>; FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
>  Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
>  Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from 4.5.0 (RHEL 7.4) to
> 4.5.4 (RHEL 7.5)
>  
>  thierry bordaz wrote:
> > Hi Soler,
> >
> > Thanks for the information.
> > So indexing is hanging because SC cache_init is running, the SC
> > cache_init is hanging because SSSD is not started, SSSD is not started
> > possibly because indexing prevents to get read access to the backend
> > ("Backend is offline" TBC).
> >
> > An option would be to disable SC during upgrade phase ? Rob ?
>  
>  Sure, disabling SC would be a way to get things running again.
>  
>  To do this:
>  
>  1. Be sure dirsrv is not running
>  2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
>  3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
>  4. change nsslapd-pluginEnabled from on to off 5. try the upgrade again
>  
>  Assuming the upgrade is successful you can re-enable it post-install with:
>  
>  # ipa-compat-manage enable
>  
>  rob
>  
> >
> > regards
> > thierry
> >
> > On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
> >> Hello Thierry,
> >>
> >> The version is: slapi-nis-0.56.0-8.el7.x86_64
> >>
> >> And the errors are:
> >> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin -
> >> group &quot;XXXXXX(a)ipa.example.org&quot; does not exist because SSSD is
> offline.
> >> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin -
> >> waiting for SSSD to become online...
> >>
> >> SSSD service is up, but as the ipa services are not (all) running,
> >> the ipa domain is down, logs when started sssd shows:
> >> sssd[be[ipa.example.org]][823]: Backend is offline
> >>
> >> and logs sssd_nss.log & sssd_sudo.log have hundreds of:
> >>    [sss_dp_get_reply] (0x0010): The Data Provider returned an error
> >> [org.freedesktop.sssd.Error.DataProvider.Offline]
> >>
> >> Thanks & Regards.
> >>
> >> -----Original Message-----
> >> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >> Sent: Thursday, May 03, 2018 11:25
> >> To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden
> >> <rcritten(a)redhat.com>; FreeIPA users list
> >> <freeipa-users(a)lists.fedorahosted.org&gt;
> >> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
> >> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>
> >> me again ...
> >>
> >> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during the
> >> hanging period it could also indicate the reason what Schema compat
> >> was waiting for (SSSD ?)
> >>
> >> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
> >>> hello,
> >>>
> >>> Yesterday my ssh console closed the connection, so I had to start
> >>> again the "ipa-server-upgrade", but the result is more or less the
> same:
> >>> # ipa-server-upgrade
> >>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>      [1/10]: stopping directory server
> >>>      [2/10]: saving configuration
> >>>      [3/10]: disabling listeners
> >>>      [4/10]: enabling DS global lock
> >>>      [5/10]: starting directory server
> >>>      [6/10]: updating schema
> >>>      [7/10]: upgrading server
> >>>
> >>> But now, the lines that are repeated on the access log are:
> >>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
> >>>
> base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
> >>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
> nstaskexitcode"
> >>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
> >>> tag=101 nentries=1 etime=1.0000687740
> >>>
> >>> And it is on the same state than the other:
> >>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>> -b
> >>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s base
> SASL/EXTERNAL authentication started SASL username:
> >>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>> SASL SSF: 0
> >>> # extended LDIF
> >>> #
> >>> # LDAPv3
> >>> # base
> >>> <cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
> >>> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
> >>>
> >>> # indextask_l_137445500911864330_4055, index, tasks, config
> >>> dn:
> >>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
> >>> objectClass: top
> >>> objectClass: extensibleObject
> >>> nsindexattribute: l
> >>> nsinstance: userRoot
> >>> cn: indextask_l_137445500911864330_4055
> >>> nstaskcurrentitem: 0
> >>> nstasktotalitems: 1
> >>>
> >>> # search result
> >>> search: 3
> >>> result: 0 Success
> >>>
> >>> # numResponses: 2
> >>> # numEntries: 1
> >>>
> >>> The pstack of the proces: /usr/bin/python2
> >>> /usr/sbin/ipa-server-upgrade # pstack 9805
> >>> #0  0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
> >>> #1  0x00007fb8ff5e0000 in time_sleep () from
> >>> /usr/lib64/python2.7/lib-dynload/timemodule.so
> >>> #2  0x00007fb90730acf0 in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #3  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #4  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #5  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #6  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #7  0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #8  0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #9  0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #21 0x00007fb90730d142 in PyEval_EvalCode () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
> >>> #23 0x00007fb90732773e in PyRun_FileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
> >>> /lib64/libpython2.7.so.1.0
> >>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
> >>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
> >>> #27 0x000000000040066e in _start ()
> >>>
> >>>
> >>> And the pstack from the process: /usr/sbin/ns-slapd -D
> >>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
> >>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
> >>> # pstack 15536
> >>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
> >>> (Thread 0x7f5bc02e8700 (LWP 15540)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
> >>> (Thread 0x7f5bbfae7700 (LWP 15541)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
> >>> (Thread 0x7f5bbf2e6700 (LWP 15542)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
> >>> (Thread 0x7f5bbeae5700 (LWP 15543)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
> >>> (Thread 0x7f5bbe2e4700 (LWP 15544)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
> >>> (Thread 0x7f5bbdae3700 (LWP 15545)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
> >>> (Thread 0x7f5bbd2e2700 (LWP 15546)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
> >>> (Thread 0x7f5bbcae1700 (LWP 15547)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
> >>> (Thread 0x7f5bbc2e0700 (LWP 15548)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
> >>> (Thread 0x7f5bbbadf700 (LWP 15549)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
> >>> (Thread 0x7f5bbb2de700 (LWP 15550)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
> >>> (Thread 0x7f5bbaadd700 (LWP 15551)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
> >>> (Thread 0x7f5bba2dc700 (LWP 15552)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
> >>> (Thread 0x7f5bb9adb700 (LWP 15553)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
> >>> (Thread 0x7f5bb92da700 (LWP 15554)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d33e63 in worker_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
> >>> (Thread 0x7f5bb8ad9700 (LWP 15555)):
> >>> #0  0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
> >>> #1  0x00007f5bd1d9c803 in epoll_dispatch () at
> >>> /lib64/libevent-2.0.so.5
> >>> #2  0x00007f5bd1d883ea in event_base_loop () at
> >>> /lib64/libevent-2.0.so.5
> >>> #3  0x00007f5bd4d34eae in ns_event_fw_loop () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #4  0x00007f5bd4d33ac9 in event_loop_thread_func () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
> >>> (Thread 0x7f5bb2b84700 (LWP 15559)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e41e77 in deadlock_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
> >>> (Thread 0x7f5bb2383700 (LWP 15560)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e45a87 in checkpoint_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
> >>> (Thread 0x7f5bb1b82700 (LWP 15561)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e420c7 in trickle_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
> >>> (Thread 0x7f5bb1381700 (LWP 15562)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e3ce47 in perf_threadmain () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #4  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #5  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
> >>> (Thread 0x7f5bb0b80700 (LWP 15563)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc920213e in cos_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
> >>> (Thread 0x7f5bb037f700 (LWP 15564)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bc6d7e44c in ipa_cldap_worker () at
> >>> /usr/lib64/dirsrv/plugins/libipa_cldap.so
> >>> #2  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #3  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
> >>> (Thread 0x7f5bd5132700 (LWP 15566)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3572033 in _cl5TrimMain () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
> >>> (Thread 0x7f5bafb7e700 (LWP 15567)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
> >>> (Thread 0x7f5baf37d700 (LWP 15568)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bc3587fa4 in protocol_sleep () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #4  0x00007f5bc358b75a in repl5_inc_run () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #5  0x00007f5bc358fbec in prot_thread_main () at
> >>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
> >>> #6  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #7  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #8  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
> >>> (Thread 0x7f5bae759700 (LWP 15569)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
> >>> (Thread 0x7f5badf58700 (LWP 15570)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
> >>> (Thread 0x7f5bad757700 (LWP 15571)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #3  0x00007f5bc22d11ad in roles_cache_wait_on_change () at
> >>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
> >>> (Thread 0x7f5bacd4f700 (LWP 15572)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x0000564591e10b08 in housecleaning ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
> >>> (Thread 0x7f5bac54e700 (LWP 15573)):
> >>> #0  0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
> >>> #2  0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #3  0x00007f5bd4a5ba63 in eq_loop () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
> >>> (Thread 0x7f5babd4d700 (LWP 15574)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
> >>> (Thread 0x7f5bab54c700 (LWP 15575)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
> >>> (Thread 0x7f5baad4b700 (LWP 15576)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
> >>> (Thread 0x7f5baa54a700 (LWP 15577)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
> >>> (Thread 0x7f5ba9d49700 (LWP 15578)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
> >>> (Thread 0x7f5ba9548700 (LWP 15579)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
> >>> (Thread 0x7f5ba8d47700 (LWP 15580)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
> >>> (Thread 0x7f5ba8546700 (LWP 15581)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
> >>> (Thread 0x7f5ba7d45700 (LWP 15582)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
> >>> (Thread 0x7f5ba7544700 (LWP 15583)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
> >>> (Thread 0x7f5ba6d43700 (LWP 15584)):
> >>> #0  0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
> >>> #1  0x00007f5bd2854bc7 in _pr_poll_with_poll () at
> >>> /lib64/libnspr4.so
> >>> #2  0x0000564591e07049 in connection_read_operation ()
> >>> #3  0x0000564591e0765d in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
> >>> (Thread 0x7f5ba6542700 (LWP 15585)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
> >>> (Thread 0x7f5ba5d41700 (LWP 15586)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
> >>> (Thread 0x7f5ba5540700 (LWP 15587)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
> >>> (Thread 0x7f5ba4d3f700 (LWP 15588)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
> >>> (Thread 0x7f5ba453e700 (LWP 15589)):
> >>> #0  0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
> >>> #2  0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
> >>> ()
> >>> #3  0x0000564591e075d7 in connection_threadmain ()
> >>> #4  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #5  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #6  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
> >>> (Thread 0x7f5ba3900700 (LWP 15590)):
> >>> #0  0x00007f5bd189cc03 in select () at /lib64/libc.so.6
> >>> #1  0x00007f5bd4acc599 in DS_Sleep () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc1ea5c6e in backend_set_entry_from () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #3  0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #4  0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd4aace0c in send_ldap_search_entry () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #6  0x00007f5bd4a87ca4 in send_entry.isra.0 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #7  0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #8  0x00007f5bd4a8a0d1 in op_shared_search () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #9  0x00007f5bd4a9b507 in search_internal_callback_pb () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb ()
> >>> at /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
> >>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
> >>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
> >>> (Thread 0x7f5ba30ff700 (LWP 15606)):
> >>> #0  0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
> >>> /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #2  0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #3  0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
> >>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >>> #4  0x00007f5bd4ac5942 in task_index_thread () at
> >>> /usr/lib64/dirsrv/libslapd.so.0
> >>> #5  0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
> >>> #6  0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
> >>> #7  0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
> >>> (Thread 0x7f5bd5135940 (LWP 15536)):
> >>> #0  0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
> >>> #1  0x00007f5bd4d34b2f in ns_thrpool_wait () at
> >>> /usr/lib64/dirsrv/libnunc-stans.so.0
> >>> #2  0x0000564591e0d862 in slapd_daemon ()
> >>> #3  0x0000564591dfe86f in main ()
> >>>
> >>> Thanks & Regards.
> >>> ______________________________
> >>>
> >>> -----Original Message-----
> >>> From: thierry bordaz <tbordaz(a)redhat.com&gt;
> >>> Sent: Thursday, May 03, 2018 9:31
> >>> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
> >>> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
> >>> <solerm(a)unicc.org&gt;
> >>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com&gt;
> >>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>> from
> >>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>
> >>> Hi,
> >>>
> >>> During indexing task we should see in the task status the periodic
> >>> progression of the indexing.
> >>> May be the indexing is hanging somewhere. When the problem occurs
> >>> could you provide a pstack of the dirsrv server ?
> >>>
> >>> best regards
> >>> thierry
> >>>
> >>> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
> >>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>> Hello,
> >>>>>
> >>>>> This is the output of the command (seems that is not complete):
> >>>>>
> >>>>> # ldapsearch -H
> >>>>> ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
> >>>>> -b
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>>> ,c
> >>>>> n
> >>>>> =config
> >>>>> -s base
> >>>>> SASL/EXTERNAL authentication started SASL username:
> >>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
> >>>>> SASL SSF: 0
> >>>>> # extended LDIF
> >>>>> #
> >>>>> # LDAPv3
> >>>>> # base
> >>>>>
> <cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>>> s,
> >>>>> c
> >>>>> n=config>
> >>>>> with scope baseObject
> >>>>> # filter: (objectclass=*)
> >>>>> # requesting: ALL
> >>>>> #
> >>>>>
> >>>>> # indextask_description_137444551994158920_5958, index, tasks,
> >>>>> config
> >>>>> dn:
> >>>>> cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks
> >>>>> ,c
> >>>>> n
> >>>>> =conf
> >>>>>      ig
> >>>>> objectClass: top
> >>>>> objectClass: extensibleObject
> >>>>> nsindexattribute: description
> >>>>> nsinstance: userRoot
> >>>>> cn: indextask_description_137444551994158920_5958
> >>>>> nstaskcurrentitem: 0
> >>>>> nstasktotalitems: 1
> >>>> IPA is looking for nstaskstatus and since that isn't there it is
> >>>> looping forever.
> >>>>
> >>>> I've cc'd a couple of the 389-ds devs to see what they think
> about it.
> >>>>
> >>>> rob
> >>>>
> >>>>> # search result
> >>>>> search: 3
> >>>>> result: 0 Success
> >>>>>
> >>>>> # numResponses: 2
> >>>>> # numEntries: 1
> >>>>>
> >>>>> Thanks & Regards.
> >>>>> ______________________________
> >>>>> -----Original Message-----
> >>>>> From: Rob Crittenden <rcritten(a)redhat.com&gt;
> >>>>> Sent: Tuesday, May 01, 2018 15:18
> >>>>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
> >>>>> from
> >>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>
> >>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> Thank you for your answer, now dirsrv can start, but after
> >>>>>> running the "
> >>>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
> >>>>>>
> >>>>>> # ipa-server-upgrade
> >>>>>>
> >>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>        [1/8]: saving configuration
> >>>>>>
> >>>>>>        [2/8]: disabling listeners
> >>>>>>
> >>>>>>        [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>        [4/8]: starting directory server
> >>>>>>
> >>>>>>        [5/8]: updating schema
> >>>>>>
> >>>>>>        [6/8]: upgrading server
> >>>>>>
> >>>>>> The latest messages on the log is (there are no logs on the
> lasts
> >>>>>> 2h):
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG
> >>>>>> ---------------------------------------------
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying updates
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG dn:
> >>>>>> cn=description,cn=index,cn=userroot,cn=ldbm
> >>>>>> database,cn=plugins,cn=config
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      top
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      nsindex
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      eq
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      sub
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG cn:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      description
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
> >>>>>>
> >>>>>> 2018-05-01T08:13:14Z DEBUG      false
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
> >>>>>> description
> >>>>>>
> >>>>>> 2018-05-01T08:13:19Z DEBUG Task id:
> >>>>>>
> cn=indextask_description_137444551994158920_5958,cn=index,cn=task
> >>>>>> s,
> >>>>>> c
> >>>>>> n=
> >>>>>> config
> >>>>>>
> >>>>>> If I check the ipa services, dirsrv is the only one running:
> >>>>>>
> >>>>>> # ipactl status
> >>>>>>
> >>>>>> Directory Service: RUNNING
> >>>>>>
> >>>>>> krb5kdc Service: STOPPED
> >>>>>>
> >>>>>> kadmin Service: STOPPED
> >>>>>>
> >>>>>> named Service: STOPPED
> >>>>>>
> >>>>>> httpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-custodia Service: STOPPED
> >>>>>>
> >>>>>> ntpd Service: RUNNING
> >>>>>>
> >>>>>> pki-tomcatd Service: STOPPED
> >>>>>>
> >>>>>> smb Service: STOPPED
> >>>>>>
> >>>>>> winbind Service: STOPPED
> >>>>>>
> >>>>>> ipa-otpd Service: STOPPED
> >>>>>>
> >>>>>> ipa-dnskeysyncd Service: STOPPED
> >>>>>>
> >>>>>> ipa: INFO: The ipactl command was successful
> >>>>>>
> >>>>>> Should I stop the ipa-server-upgrade and start ipa services?
> >>>>> It depends on how many entries you have. This is an indexing task
> >>>>> and while 2 hours seems a bit excessive, it depends on how much
> >>>>> work it has to do.
> >>>>>
> >>>>> The 389-ds access and/or error logs may provide details.
> >>>>>
> >>>>> rob
> >>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com&gt;
> >>>>>> Sent: Tuesday, May 01, 2018 9:56
> >>>>>> To: FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org&gt;
> >>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org&gt;
> >>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating
> from
> >>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
> >>>>>>
> >>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users
> wrote:
> >>>>>>
> >>>>>>      >hello,
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >I have an IPA master that updated from 4.5.0 (RHEL 7.4)
> to
> >>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with the
> >>>>>> unique replica I have, but after update dirsrv is not starting.
> >>>>>>
> >>>>>>      >It says it is needed run "ipa-server-upgrade",
> but it also
> >>>>>> fails:
> >>>>>>
> >>>>>>      ># ipactl start
> >>>>>>
> >>>>>>      >Upgrade required: please run ipa-server-upgrade
> command
> >>>>>> Aborting ipactl
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      ># ipa-server-upgrade
> >>>>>>
> >>>>>>      >Upgrading IPA:. Estimated time: 1 minute 30 seconds
> >>>>>>
> >>>>>>      >  [1/8]: saving configuration
> >>>>>>
> >>>>>>      >  [2/8]: disabling listeners
> >>>>>>
> >>>>>>      >  [3/8]: enabling DS global lock
> >>>>>>
> >>>>>>      >  [4/8]: starting directory server
> >>>>>>
> >>>>>>      >  [error] CalledProcessError: Command
> '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMOLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >  [cleanup]: stopping directory server
> >>>>>>
> >>>>>>      >  [cleanup]: restoring configuration
> >>>>>>
> >>>>>>      >IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log
> >>>>>> and run command ipa-server-upgrade manually.
> >>>>>>
> >>>>>>      >Unexpected error - see /var/log/ipaupgrade.log for
> details:
> >>>>>>
> >>>>>>      >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1 The
> >>>>>>
> >>>>>>      >ipa-server-upgrade command failed. See
> >>>>>> /var/log/ipaupgrade.log for more
> >>>>>>
> >>>>>>      >information
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >On the log I can see:
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl
> is-active
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>      >finished, return code=3 2018-04-30T14:36:15Z DEBUG
> >>>>>> stdout=failed ...
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory
> >>>>>> server
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Starting external process
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> 2018-04-30T14:36:15Z DEBUG Process
> >>>>>>
> >>>>>>      >finished, return code=1 2018-04-30T14:36:15Z DEBUG
> stdout=
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG stderr=Job for
> >>>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>
> >>>>>> failed because the control process exited with error code. See
> >>>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>" and
> "journalctl -xe" for
> >>>>>> details.
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG Traceback (most recent call
> last):
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 504, in start_creation
> >>>>>>
> >>>>>>      >    run_step(full_msg, method)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> >>>>>> line 494, in run_step
> >>>>>>
> >>>>>>      >    method()
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
> >>>>>> py
> >>>>>> ",
> >>>>>> line 95, in __start
> >>>>>>
> >>>>>>      >    srv.start(self.serverid, ldapi=True)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py"
> >>>>>> ,
> >>>>>> line 161, in start
> >>>>>>
> >>>>>>      >    instance_name, capture_output=capture_output,
> wait=wait)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
> >>>>>> line 294, in start
> >>>>>>
> >>>>>>      >    skip_output=not capture_output)
> >>>>>>
> >>>>>>      >  File
> >>>>>>
> "/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
> >>>>>> line 542, in run
> >>>>>>
> >>>>>>      >    raise CalledProcessError(p.returncode, arg_string,
> >>>>>> str(output))
> >>>>>>
> >>>>>>      >CalledProcessError: Command '/bin/systemctl start
> >>>>>>
> >>>>>>      &gt;dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >2018-04-30T14:36:15Z DEBUG   [error]
> CalledProcessError:
> >>>>>> Command '/bin/systemctl start
> dirsrv(a)IPA-EXAMPLE-ORG.service
> >>>>>> <mailto:[e-mail]>' returned
> non-zero exit
> >>>>>> status
> >>>>>> 1
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I
> show:
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
> >>>>>> slapd_bootstrap_config
> >>>>>> - The default password storage scheme could not be read or was
> >>>>>> not found in the file
> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif.
> >>>>>> It is mandatory.
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >Checking on internet I show that "dse.ldif"
> could be
> >>>>>> corrupted, so I changed with "dse.ldif.startOK" without
> any
> >>>>>> change and then I changed with "dse.ldif.bak". The
> problem
> >>>>>> persist but the error has
> >>>>>> changed:
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
> >>>>>> config_set_port
> >>>>>> -
> >>>>>>
> >>>>>>      >Non-Secure Port Disabled
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>      >Netscape Portable Runtime error -5975:
> >>>>>>
> >>>>>>      >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
> >>>>>> undefined
> >>>>>> symbol:
> >>>>>>
> >>>>>>      >replication_legacy_plugin_init
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
> >>>>>> symload_report_error
> >>>>>> -
> >>>>>>
> >>>>>>      >Could not load symbol
> "replication_legacy_plugin_init" from
> >>>>>>
> >>>>>>     
> >"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for
> >>>>>> plugin Legacy
> >>>>>>
> >>>>>>      >Replication Plugin
> >>>>>>
> >>>>>>      >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
> >>>>>> load_plugin_entry
> >>>>>> - Unable to load plugin "cn=Legacy Replication
> >>>>>> Plugin,cn=plugins,cn=config"
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >I saw a bug about this problem, but it is still
> opened:
> >>>>>>
> >>>>>>>
> https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529
> >>>>>>> 44
> >>>>>>> 2
> >>>>>>      >
> >>>>>>
> >>>>>>      >Any idea how to fix the issue?
> >>>>>>
> >>>>>>      >
> >>>>>>
> >>>>>>      >If it is not possible to fix it, can I remove the
> replica
> >>>>>> from IPA and install it again with the same name?
> >>>>>>
> >>>>>> A quick fix could be to remove an entry for cn=Legacy
> Replication
> >>>>>> Plugin,cn=plugins,cn=config from
> >>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
> >>>>>>
> >>>>>> when dirsrv is down.
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> / Alexander Bokovoy
> >>>>>>
> >>>>>> Sr. Principal Software Engineer
> >>>>>>
> >>>>>> Security / Identity Management Engineering Red Hat Limited,
> >>>>>> Finland
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> FreeIPA-users mailing list --
> >>>>>> freeipa-users(a)lists.fedorahosted.org
> >>>>>> To unsubscribe send an email to
> >>>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>>
> >
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  
>

[Lachlan Musicman]

On 1 May 2018 at 17:40, SOLER SANGUESA Miguel via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  hello,
>
>
>
>  I have an IPA master that updated from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL
>  7.5). An hour later I tried to do the same with the unique replica I have,
>  but after update dirsrv is not starting.
>
>  It says it is needed run "ipa-server-upgrade", but it also fails:
>
>  *# ipactl start*
>
>  *Upgrade required: please run ipa-server-upgrade command*
>
>  *Aborting ipactl*
>
>
>
>  *# ipa-server-upgrade*
>
>  *Upgrading IPA:. Estimated time: 1 minute 30 seconds*
>
>  *  [1/8]: saving configuration*
>
>  *  [2/8]: disabling listeners*
>
>  *  [3/8]: enabling DS global lock*
>
>  *  [4/8]: starting directory server*
>
>  *  [error] CalledProcessError: Command '/bin/systemctl start
>  dirsrv(a)IPA-EXAMOLE-ORG.service&#39; returned non-zero exit status 1*
>
>  *  [cleanup]: stopping directory server*
>
>  *  [cleanup]: restoring configuration*
>
>  *IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
>  command ipa-server-upgrade manually.*
>
>  *Unexpected error - see /var/log/ipaupgrade.log for details:*
>
>  *CalledProcessError: Command '/bin/systemctl start
>  dirsrv(a)IPA-EXAMPLE-ORG.service&#39; returned non-zero exit status 1*
>
>  *The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
>  more information*
>
>
>
>  On the log I can see:
>
>  2018-04-30T14:36:15Z DEBUG Starting external process
>
>  2018-04-30T14:36:15Z DEBUG args=/bin/systemctl is-active
>  dirsrv(a)IPA-EXAMPLE-ORG.service
>
>  2018-04-30T14:36:15Z DEBUG Process finished, return code=3
>
>  2018-04-30T14:36:15Z DEBUG stdout=failed
>
>  ...
>
>  2018-04-30T14:36:15Z DEBUG   [4/8]: starting directory server
>
>  2018-04-30T14:36:15Z DEBUG Starting external process
>
>  2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>  dirsrv(a)IPA-EXAMPLE-ORG.service
>
>  2018-04-30T14:36:15Z DEBUG Process finished, return code=1
>
>  2018-04-30T14:36:15Z DEBUG stdout=
>
>  2018-04-30T14:36:15Z DEBUG stderr=Job for dirsrv(a)IPA-EXAMPLE-ORG.service
>  failed because the control process exited with error code. See "systemctl
>  status dirsrv(a)IPA-EXAMPLE-ORG.service&quot; and "journalctl -xe" for details.
>
>
>
>  2018-04-30T14:36:15Z DEBUG Traceback (most recent call last):
>
>    File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>  line 504, in start_creation
>
>      run_step(full_msg, method)
>
>    File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>  line 494, in run_step
>
>      method()
>
>    File
> "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py",
>  line 95, in __start
>
>      srv.start(self.serverid, ldapi=True)
>
>    File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
>  line 161, in start
>
>      instance_name, capture_output=capture_output, wait=wait)
>
>    File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
>  line 294, in start
>
>      skip_output=not capture_output)
>
>    File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 542,
>  in run
>
>      raise CalledProcessError(p.returncode, arg_string, str(output))
>
>  CalledProcessError: Command '/bin/systemctl start
>  dirsrv(a)IPA-EXAMPLE-ORG.service&#39; returned non-zero exit status 1
>
>
>
>  2018-04-30T14:36:15Z DEBUG   [error] CalledProcessError: Command
>  '/bin/systemctl start dirsrv(a)IPA-EXAMPLE-ORG.service&#39; returned non-zero
>  exit status 1
>
>
>
>  Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors I show:
>
>  [30/Apr/2018:16:04:52.584220922 +0200] - ERR - slapd_bootstrap_config -
>  The default password storage scheme could not be read or was not found in
>  the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif. It is mandatory.
>
>
>
>  Checking on internet I show that "dse.ldif" could be corrupted, so I
>  changed with "dse.ldif.startOK" without any change and then I changed with
>  "dse.ldif.bak". The problem persist but the error has changed:
>
>  [30/Apr/2018:16:32:13.435210918 +0200] - NOTICE - config_set_port -
>  Non-Secure Port Disabled
>
>  [30/Apr/2018:16:32:13.556581301 +0200] - ERR - symload_report_error -
>  Netscape Portable Runtime error -5975:
> /usr/lib64/dirsrv/plugins/libreplication-plugin.so:
>  undefined symbol: replication_legacy_plugin_init
>
>  [30/Apr/2018:16:32:13.561590553 +0200] - ERR - symload_report_error -
>  Could not load symbol "replication_legacy_plugin_init" from
>  "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin Legacy
>  Replication Plugin
>
>  [30/Apr/2018:16:32:13.564590264 +0200] - ERR - load_plugin_entry - Unable
>  to load plugin "cn=Legacy Replication Plugin,cn=plugins,cn=config"
>
>
>
>  I saw a bug about this problem, but it is still opened:
>
>  https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1529442
>
>
>
>  Any idea how to fix the issue?
>
>
>
>  If it is not possible to fix it, can I remove the replica from IPA and
>  install it again with the same name?
>  

I think I've just bumped into this same problem while updating my dev-ipa
instance.

Once IT have restored the VM from backup, I'll check to see if it's the
same error when I upgrade again, and follow these steps

Is there anything I can do to help diagnose?

Cheers
L.