Detected Disorderly Shutdown last time Directory Server was running, recovering database.

[Carlos Barrabes]

Hello,

we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has 
been running smooth until today when we realized the DS was down and 
wouldn't go up.


After checking all the logs, we got this error message:

      Detected Disorderly Shutdown last time Directory Server was 
running, recovering database.


We have even tried to reboot the entire server but the problem persists. 
Any suggestions on how to fix this will be really appreciated :D


Thanks!

[Noriko Hosoi]

On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
>  Hello,
>
>  we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has 
>  been running smooth until today when we realized the DS was down and 
>  wouldn't go up.
>
>  After checking all the logs, we got this error message:
>
>       Detected Disorderly Shutdown last time Directory Server was 
>  running, recovering database. 
Could you please try this command line?
ps -ef | grep ns-slapd

If you see the ns-slapd process, start up (actually, database recovery) 
is still running.  Depending upon the size of the transaction log, it'd 
take a very long time.

If you don't see the process in the ps output, please start the server 
from gdb:
# gdb /usr/sbin/ns-slapd
(gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0

Once it crashes, get the stacktraces.
(gdb) thread apply all bt

And post the stacktraces?
>  We have even tried to reboot the entire server but the problem 
>  persists. Any suggestions on how to fix this will be really 
>  appreciated :D
>
>
>  Thanks!
>
>  -- 
>  389-users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://lists.fedoraproject.org/admin/lists/[e-mail]... 
>

[Carlos Barrabes]

Hello,

following your advice I ran the debugger and this is what I got:


(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
/tmp/pid -d 0
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
starting up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
cache size 512000B is less than db size 516096B; We recommend to 
increase the entry cache size nsslapd-cachememsize.
[15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry cache 
size 512000B is less than db size 1589248B; We recommend to increase the 
entry cache size nsslapd-cachememsize.
[15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time 
Directory Server was running, recovering database.

Program received signal SIGBUS, Bus error.
0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so


and


(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)):
#0  0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
#1  0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so
#2  0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so
#3  0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so
#4  0x00007fffed3014d5 in dblayer_start () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#5  0x00007fffed356aa1 in ldbm_back_start () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#6  0x00007ffff7b49283 in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#7  0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop 
() from /usr/lib64/dirsrv/libslapd.so.0
#8  0x00005555555653fb in main ()


Thanks!


On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
>  On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
> > Hello,
> >
> > we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has 
> > been running smooth until today when we realized the DS was down and 
> > wouldn't go up.
> >
> > After checking all the logs, we got this error message:
> >
> >      Detected Disorderly Shutdown last time Directory Server was 
> > running, recovering database.
>  Could you please try this command line?
>  ps -ef | grep ns-slapd
>
>  If you see the ns-slapd process, start up (actually, database 
>  recovery) is still running.  Depending upon the size of the 
>  transaction log, it'd take a very long time.
>
>  If you don't see the process in the ps output, please start the server 
>  from gdb:
>  # gdb /usr/sbin/ns-slapd
>  (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
>
>  Once it crashes, get the stacktraces.
>  (gdb) thread apply all bt
>
>  And post the stacktraces?
> >
> > We have even tried to reboot the entire server but the problem 
> > persists. Any suggestions on how to fix this will be really 
> > appreciated :D
> >
> >
> > Thanks!
> >
> > -- 
> > 389-users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >
>
>  -- 
>  389-users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://lists.fedoraproject.org/admin/lists/[e-mail]... 
>

[Noriko Hosoi]

Thank you, Carlos.

Could you repeat running gdb once more after installing debuginfo 
packages as follows?
# yum install 389-ds-base-debuginfo libdb-debuginfo

Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo
enabled=1

Could you also share this output?
# egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif

Thanks.

On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
>  Hello,
>
>  following your advice I ran the debugger and this is what I got:
>
>
>  (gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
>  Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
>  /tmp/pid -d 0
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib64/libthread_db.so.1".
>  [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
>  starting up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
>  cache size 512000B is less than db size 516096B; We recommend to 
>  increase the entry cache size nsslapd-cachememsize.
>  [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry 
>  cache size 512000B is less than db size 1589248B; We recommend to 
>  increase the entry cache size nsslapd-cachememsize.
>  [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time 
>  Directory Server was running, recovering database.
>
>  Program received signal SIGBUS, Bus error.
>  0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
>
>
>  and
>
>
>  (gdb) thread apply all bt
>
>  Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)):
>  #0  0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
>  #1  0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so
>  #2  0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so
>  #3  0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so
>  #4  0x00007fffed3014d5 in dblayer_start () from 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #5  0x00007fffed356aa1 in ldbm_back_start () from 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #6  0x00007ffff7b49283 in plugin_call_func () from 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #7  0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop 
>  () from /usr/lib64/dirsrv/libslapd.so.0
>  #8  0x00005555555653fb in main ()
>
>
>  Thanks!
>
>
>  On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
> > On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
> >> Hello,
> >>
> >> we have a Fedora22 machine running 380 + Dogtag 10.2. Everything has 
> >> been running smooth until today when we realized the DS was down and 
> >> wouldn't go up.
> >>
> >> After checking all the logs, we got this error message:
> >>
> >>      Detected Disorderly Shutdown last time Directory Server was 
> >> running, recovering database.
> > Could you please try this command line?
> > ps -ef | grep ns-slapd
> >
> > If you see the ns-slapd process, start up (actually, database 
> > recovery) is still running.  Depending upon the size of the 
> > transaction log, it'd take a very long time.
> >
> > If you don't see the process in the ps output, please start the 
> > server from gdb:
> > # gdb /usr/sbin/ns-slapd
> > (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
> >
> > Once it crashes, get the stacktraces.
> > (gdb) thread apply all bt
> >
> > And post the stacktraces?
> >>
> >> We have even tried to reboot the entire server but the problem 
> >> persists. Any suggestions on how to fix this will be really 
> >> appreciated :D
> >>
> >>
> >> Thanks!
> >>
> >> -- 
> >> 389-users mailing list
> >> 389-users(a)lists.fedoraproject.org
> >> https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >>
> >
> > -- 
> > 389-users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >
>  -- 
>  389-users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://lists.fedoraproject.org/admin/lists/[e-mail]... 
>

[Carlos Barrabes]

Hello,

this is the output after installing the packages you asked:

(gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
/tmp/pid -d 0
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not 
match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).

warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not match 
"/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not 
match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does not 
match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does not 
match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" 
(CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC mismatch).
warning: the debug information found in 
"/usr/lib/debug//usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" 
does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC 
mismatch).
warning: the debug information found in 
"/usr/lib/debug/usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" does 
not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC mismatch).

[18/Jul/2016:04:55:43 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
starting up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
-- rounding up
[18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
cache size 512000B is less than db size 516096B; We recommend to 
increase the entry cache size nsslapd-cachememsize.
[18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-CA: entry cache 
size 512000B is less than db size 1589248B; We recommend to increase the 
entry cache size nsslapd-cachememsize.
[18/Jul/2016:04:55:43 +0100] - Detected Disorderly Shutdown last time 
Directory Server was running, recovering database.

Program received signal SIGBUS, Bus error.
__env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at 
../../src/env/env_alloc.c:599
599            if (elp->len >= elp_tmp->len)


and:


(gdb) thread apply all bt

Thread 1 (Thread 0x7ffff7fd3800 (LWP 13818)):
#0  __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at 
../../src/env/env_alloc.c:599
#1  0x00007fffeef910ac in __env_detach (env=env@entry=0x555555b37ad0, 
destroy=destroy@entry=1) at ../../src/env/env_region.c:823
#2  0x00007fffeef915da in __env_remove_env 
(env=env@entry=0x555555b37ad0) at ../../src/env/env_region.c:934
#3  0x00007fffeef8afdd in __env_open (dbenv=0x555555b52230, 
db_home=<optimized out>, flags=9987, mode=<optimized out>) at 
../../src/env/env_open.c:201
#4  0x00007fffed3014d5 in dblayer_start () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#5  0x00007fffed356aa1 in ldbm_back_start () from 
/usr/lib64/dirsrv/plugins/libback-ldbm.so
#6  0x00007ffff7b49283 in plugin_call_func () from 
/usr/lib64/dirsrv/libslapd.so.0
#7  0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop 
() from /usr/lib64/dirsrv/libslapd.so.0
#8  0x00005555555653fb in main ()


and finally:

#  egrep nsslapd-db /etc/dirsrv/slapd-pki/dse.ldif
nsslapd-dbcachesize: 500000
nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-pki/db
nsslapd-db-durable-transaction: on
nsslapd-db-checkpoint-interval: 60
nsslapd-db-compactdb-interval: 2592000
nsslapd-db-transaction-batch-val: 0
nsslapd-db-transaction-batch-min-wait: 50
nsslapd-db-transaction-batch-max-wait: 50
nsslapd-db-logbuf-size: 0
nsslapd-db-locks: 10000
nsslapd-db-private-import-mem: on
nsslapd-db-deadlock-policy: 9


Thanks!
Carlos.




On 07/15/2016 07:30 PM, Noriko Hosoi wrote:
>  Thank you, Carlos.
>
>  Could you repeat running gdb once more after installing debuginfo 
>  packages as follows?
>  # yum install 389-ds-base-debuginfo libdb-debuginfo
>
>  Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo
>  enabled=1
>
>  Could you also share this output?
>  # egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif
>
>  Thanks.
>
>  On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
> > Hello,
> >
> > following your advice I ran the debugger and this is what I got:
> >
> >
> > (gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
> > Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
> > /tmp/pid -d 0
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
> > starting up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> > 512000 -- rounding up
> > [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
> > cache size 512000B is less than db size 516096B; We recommend to 
> > increase the entry cache size nsslapd-cachememsize.
> > [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry 
> > cache size 512000B is less than db size 1589248B; We recommend to 
> > increase the entry cache size nsslapd-cachememsize.
> > [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last time 
> > Directory Server was running, recovering database.
> >
> > Program received signal SIGBUS, Bus error.
> > 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
> >
> >
> > and
> >
> >
> > (gdb) thread apply all bt
> >
> > Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)):
> > #0  0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
> > #1  0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so
> > #2  0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so
> > #3  0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so
> > #4  0x00007fffed3014d5 in dblayer_start () from 
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #5  0x00007fffed356aa1 in ldbm_back_start () from 
> > /usr/lib64/dirsrv/plugins/libback-ldbm.so
> > #6  0x00007ffff7b49283 in plugin_call_func () from 
> > /usr/lib64/dirsrv/libslapd.so.0
> > #7  0x00007ffff7b49b84 in 
> > plugin_dependency_startall.isra.10.constprop () from 
> > /usr/lib64/dirsrv/libslapd.so.0
> > #8  0x00005555555653fb in main ()
> >
> >
> > Thanks!
> >
> >
> > On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
> >> On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
> >>> Hello,
> >>>
> >>> we have a Fedora22 machine running 380 + Dogtag 10.2. Everything 
> >>> has been running smooth until today when we realized the DS was 
> >>> down and wouldn't go up.
> >>>
> >>> After checking all the logs, we got this error message:
> >>>
> >>>      Detected Disorderly Shutdown last time Directory Server was 
> >>> running, recovering database.
> >> Could you please try this command line?
> >> ps -ef | grep ns-slapd
> >>
> >> If you see the ns-slapd process, start up (actually, database 
> >> recovery) is still running.  Depending upon the size of the 
> >> transaction log, it'd take a very long time.
> >>
> >> If you don't see the process in the ps output, please start the 
> >> server from gdb:
> >> # gdb /usr/sbin/ns-slapd
> >> (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
> >>
> >> Once it crashes, get the stacktraces.
> >> (gdb) thread apply all bt
> >>
> >> And post the stacktraces?
> >>>
> >>> We have even tried to reboot the entire server but the problem 
> >>> persists. Any suggestions on how to fix this will be really 
> >>> appreciated :D
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> -- 
> >>> 389-users mailing list
> >>> 389-users(a)lists.fedoraproject.org
> >>> https://lists.fedoraproject.org/admin/lists/[e-mail]...
>
> >>>
> >>
> >> -- 
> >> 389-users mailing list
> >> 389-users(a)lists.fedoraproject.org
> >> https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >>
> > -- 
> > 389-users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >
>
>  -- 
>  389-users mailing list
>  389-users(a)lists.fedoraproject.org
>  https://lists.fedoraproject.org/admin/lists/[e-mail]... 
>

[Carlos Barrabes]

Hello,

any update on this issue?

Thanks!
Carlos


On 07/18/2016 11:15 AM, Carlos Barrabes wrote:
>  Hello,
>
>  this is the output after installing the packages you asked:
>
>  (gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
>  Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
>  /tmp/pid -d 0
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not 
>  match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
>
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/libslapd.so.0.0.0.debug" does not 
>  match "/usr/lib64/dirsrv/libslapd.so.0" (CRC mismatch).
>
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib64/libthread_db.so.1".
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libsyntax-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libsyntax-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libbitwise-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libbitwise-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcollation-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libcollation-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpbe-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpwdstorage-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libattr-unique-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libacctpolicy-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacctusability-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libacctusability-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libacl-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libacl-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does 
>  not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/libns-dshttpd.so.0.0.0.debug" does 
>  not match "/usr/lib64/dirsrv/libns-dshttpd.so.0" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libautomember-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libautomember-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libchainingdb-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcos-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libcos-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libcontentsync-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libderef-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libderef-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libdna-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libdna-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libhttp-client-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libback-ldbm.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libback-ldbm.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreplication-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libreplication-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/liblinkedattrs-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libmanagedentries-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libmemberof-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libmemberof-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpam-passthru-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libpassthru-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libpassthru-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libposix-winsync-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libreferint-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libreferint-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libretrocl-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libretrocl-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libroles-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libroles-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/librootdn-access-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libschemareload-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libschemareload-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libstatechange-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libstatechange-plugin.so" 
>  (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libusn-plugin.so.debug" does 
>  not match "/usr/lib64/dirsrv/plugins/libusn-plugin.so" (CRC mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libviews-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libviews-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug//usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC 
>  mismatch).
>  warning: the debug information found in 
>  "/usr/lib/debug/usr/lib64/dirsrv/plugins/libwhoami-plugin.so.debug" 
>  does not match "/usr/lib64/dirsrv/plugins/libwhoami-plugin.so" (CRC 
>  mismatch).
>
>  [18/Jul/2016:04:55:43 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
>  starting up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING -- Minimum cache size is 512000 
>  -- rounding up
>  [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
>  cache size 512000B is less than db size 516096B; We recommend to 
>  increase the entry cache size nsslapd-cachememsize.
>  [18/Jul/2016:04:55:43 +0100] - WARNING: pki-matrix-gabon-CA: entry 
>  cache size 512000B is less than db size 1589248B; We recommend to 
>  increase the entry cache size nsslapd-cachememsize.
>  [18/Jul/2016:04:55:43 +0100] - Detected Disorderly Shutdown last time 
>  Directory Server was running, recovering database.
>
>  Program received signal SIGBUS, Bus error.
>  __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at 
>  ../../src/env/env_alloc.c:599
>  599            if (elp->len >= elp_tmp->len)
>
>
>  and:
>
>
>  (gdb) thread apply all bt
>
>  Thread 1 (Thread 0x7ffff7fd3800 (LWP 13818)):
>  #0  __env_size_insert (head=<optimized out>, elp=0x7fffe9e0d1a8) at 
>  ../../src/env/env_alloc.c:599
>  #1  0x00007fffeef910ac in __env_detach (env=env@entry=0x555555b37ad0, 
>  destroy=destroy@entry=1) at ../../src/env/env_region.c:823
>  #2  0x00007fffeef915da in __env_remove_env 
>  (env=env@entry=0x555555b37ad0) at ../../src/env/env_region.c:934
>  #3  0x00007fffeef8afdd in __env_open (dbenv=0x555555b52230, 
>  db_home=<optimized out>, flags=9987, mode=<optimized out>) at 
>  ../../src/env/env_open.c:201
>  #4  0x00007fffed3014d5 in dblayer_start () from 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #5  0x00007fffed356aa1 in ldbm_back_start () from 
>  /usr/lib64/dirsrv/plugins/libback-ldbm.so
>  #6  0x00007ffff7b49283 in plugin_call_func () from 
>  /usr/lib64/dirsrv/libslapd.so.0
>  #7  0x00007ffff7b49b84 in plugin_dependency_startall.isra.10.constprop 
>  () from /usr/lib64/dirsrv/libslapd.so.0
>  #8  0x00005555555653fb in main ()
>
>
>  and finally:
>
>  #  egrep nsslapd-db /etc/dirsrv/slapd-pki/dse.ldif
>  nsslapd-dbcachesize: 500000
>  nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-pki/db
>  nsslapd-db-durable-transaction: on
>  nsslapd-db-checkpoint-interval: 60
>  nsslapd-db-compactdb-interval: 2592000
>  nsslapd-db-transaction-batch-val: 0
>  nsslapd-db-transaction-batch-min-wait: 50
>  nsslapd-db-transaction-batch-max-wait: 50
>  nsslapd-db-logbuf-size: 0
>  nsslapd-db-locks: 10000
>  nsslapd-db-private-import-mem: on
>  nsslapd-db-deadlock-policy: 9
>
>
>  Thanks!
>  Carlos.
>
>
>
>
>  On 07/15/2016 07:30 PM, Noriko Hosoi wrote:
> > Thank you, Carlos.
> >
> > Could you repeat running gdb once more after installing debuginfo 
> > packages as follows?
> > # yum install 389-ds-base-debuginfo libdb-debuginfo
> >
> > Note: you may need to enable debuginfo in you /etc/yum.repo.d/*.repo
> > enabled=1
> >
> > Could you also share this output?
> > # egrep nsslapd-db /etc/dirsrv/slapd-YOUR_SERVER_INSTANCE/dse.ldif
> >
> > Thanks.
> >
> > On 07/15/2016 02:22 AM, Carlos Barrabes wrote:
> >> Hello,
> >>
> >> following your advice I ran the debugger and this is what I got:
> >>
> >>
> >> (gdb) run -D /etc/dirsrv/slapd-pki -i /tmp/pid -d 0
> >> Starting program: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki -i 
> >> /tmp/pid -d 0
> >> [Thread debugging using libthread_db enabled]
> >> Using host libthread_db library "/lib64/libthread_db.so.1".
> >> [15/Jul/2016:05:03:13 +0100] - 389-Directory/1.3.3.10 B2015.118.1912 
> >> starting up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING -- Minimum cache size is 
> >> 512000 -- rounding up
> >> [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-OCSP: entry 
> >> cache size 512000B is less than db size 516096B; We recommend to 
> >> increase the entry cache size nsslapd-cachememsize.
> >> [15/Jul/2016:05:03:13 +0100] - WARNING: pki-matrix-gabon-CA: entry 
> >> cache size 512000B is less than db size 1589248B; We recommend to 
> >> increase the entry cache size nsslapd-cachememsize.
> >> [15/Jul/2016:05:03:13 +0100] - Detected Disorderly Shutdown last 
> >> time Directory Server was running, recovering database.
> >>
> >> Program received signal SIGBUS, Bus error.
> >> 0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
> >>
> >>
> >> and
> >>
> >>
> >> (gdb) thread apply all bt
> >>
> >> Thread 1 (Thread 0x7ffff7fd3800 (LWP 6597)):
> >> #0  0x00007fffeef86a08 in __env_size_insert () from /lib64/libdb-5.3.so
> >> #1  0x00007fffeef910ac in __env_detach () from /lib64/libdb-5.3.so
> >> #2  0x00007fffeef915da in __env_remove_env () from /lib64/libdb-5.3.so
> >> #3  0x00007fffeef8afdd in __env_open () from /lib64/libdb-5.3.so
> >> #4  0x00007fffed3014d5 in dblayer_start () from 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #5  0x00007fffed356aa1 in ldbm_back_start () from 
> >> /usr/lib64/dirsrv/plugins/libback-ldbm.so
> >> #6  0x00007ffff7b49283 in plugin_call_func () from 
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #7  0x00007ffff7b49b84 in 
> >> plugin_dependency_startall.isra.10.constprop () from 
> >> /usr/lib64/dirsrv/libslapd.so.0
> >> #8  0x00005555555653fb in main ()
> >>
> >>
> >> Thanks!
> >>
> >>
> >> On 07/14/2016 06:22 PM, Noriko Hosoi wrote:
> >>> On 07/14/2016 09:06 AM, Carlos Barrabes wrote:
> >>>> Hello,
> >>>>
> >>>> we have a Fedora22 machine running 380 + Dogtag 10.2. Everything 
> >>>> has been running smooth until today when we realized the DS was 
> >>>> down and wouldn't go up.
> >>>>
> >>>> After checking all the logs, we got this error message:
> >>>>
> >>>>      Detected Disorderly Shutdown last time Directory Server was 
> >>>> running, recovering database.
> >>> Could you please try this command line?
> >>> ps -ef | grep ns-slapd
> >>>
> >>> If you see the ns-slapd process, start up (actually, database 
> >>> recovery) is still running.  Depending upon the size of the 
> >>> transaction log, it'd take a very long time.
> >>>
> >>> If you don't see the process in the ps output, please start the 
> >>> server from gdb:
> >>> # gdb /usr/sbin/ns-slapd
> >>> (gdb) run -D /etc/dirsrv/slapd-SERVER_INSTANCE -i /tmp/pid -d 0
> >>>
> >>> Once it crashes, get the stacktraces.
> >>> (gdb) thread apply all bt
> >>>
> >>> And post the stacktraces?
> >>>>
> >>>> We have even tried to reboot the entire server but the problem 
> >>>> persists. Any suggestions on how to fix this will be really 
> >>>> appreciated :D
> >>>>
> >>>>
> >>>> Thanks!
> >>>>
> >>>> -- 
> >>>> 389-users mailing list
> >>>> 389-users(a)lists.fedoraproject.org
> >>>>
> https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >>>>
> >>>
> >>> -- 
> >>> 389-users mailing list
> >>> 389-users(a)lists.fedoraproject.org
> >>> https://lists.fedoraproject.org/admin/lists/[e-mail]...
>
> >>>
> >> -- 
> >> 389-users mailing list
> >> 389-users(a)lists.fedoraproject.org
> >> https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >>
> >
> > -- 
> > 389-users mailing list
> > 389-users(a)lists.fedoraproject.org
> > https://lists.fedoraproject.org/admin/lists/[e-mail]... 
> >
>