Segfault & Core Dumps

[Dael Maselli]

Hi,

I'm experiencing a lot of segmentation fault on my installations, I have 
the latest release on EPEL on SL 5.4 (389-Directory/1.2.5 B2010.012.2034).

I'm trying to get core dumps but without success. Is there a specific 
configuration that prevents/allows 389-ds to core dump?

The default on linux, you know, is to have core dump soft limit to 0, so 
I changed it for 389 in /etc/sysconfig/dirsrv, adding the line:
  ulimit -c unlimited

and now I get:

# more /proc/`pidof ns-slapd`/limits | grep core
Max core file size        unlimited            unlimited            bytes

Kernel parameters are at default:

# sysctl -a | grep kernel.core
kernel.core_pattern = core
kernel.core_uses_pid = 1

So, I expected a core dump in /proc/`pidof ns-slapd`/cwd/, right?

But when it segfaults no file is created in cwd.

Can you help me, please?

Thank you.

Best regards,
    Dael Maselli.

[Rich Megginson]

Dael Maselli wrote:
> Hi,
>
> I'm experiencing a lot of segmentation fault on my installations, I have 
> the latest release on EPEL on SL 5.4 (389-Directory/1.2.5 B2010.012.2034).

Do you see seg fault messages in /var/log/messages?
> I'm trying to get core dumps but without success. Is there a specific 
> configuration that prevents/allows 389-ds to core dump?
>
> The default on linux, you know, is to have core dump soft limit to 0, so 
> I changed it for 389 in /etc/sysconfig/dirsrv, adding the line:
>   ulimit -c unlimited
>
> and now I get:
>
> # more /proc/`pidof ns-slapd`/limits | grep core
> Max core file size        unlimited            unlimited            bytes
>
> Kernel parameters are at default:
>
> # sysctl -a | grep kernel.core
> kernel.core_pattern = core
> kernel.core_uses_pid = 1
>
> So, I expected a core dump in /proc/`pidof ns-slapd`/cwd/, right?

The directory server dumps core in the log file directory, which by 
default is /var/log/dirsrv/slapd-INSTANCE

Is the crash easily reproducible?
> But when it segfaults no file is created in cwd.
>
> Can you help me, please?
>
> Thank you.
>
> Best regards,
>    Dael Maselli.
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Reinhard Nappert]

Rick,
the server dumps the logs into the configured working directory (nsslapd-workingdir attribute in cn=config), right?

-Reinhard

-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of Rich Megginson
Sent: Tuesday, September 07, 2010 10:56 AM
To: General discussion list for the 389 Directory server project.
Subject: Re: [389-users] Segfault & Core Dumps

Dael Maselli wrote:
> Hi,
>
> I'm experiencing a lot of segmentation fault on my installations, I 
> have the latest release on EPEL on SL 5.4 (389-Directory/1.2.5 B2010.012.2034).

Do you see seg fault messages in /var/log/messages?
> I'm trying to get core dumps but without success. Is there a specific 
> configuration that prevents/allows 389-ds to core dump?
>
> The default on linux, you know, is to have core dump soft limit to 0, 
> so I changed it for 389 in /etc/sysconfig/dirsrv, adding the line:
>   ulimit -c unlimited
>
> and now I get:
>
> # more /proc/`pidof ns-slapd`/limits | grep core
> Max core file size        unlimited            unlimited            bytes
>
> Kernel parameters are at default:
>
> # sysctl -a | grep kernel.core
> kernel.core_pattern = core
> kernel.core_uses_pid = 1
>
> So, I expected a core dump in /proc/`pidof ns-slapd`/cwd/, right?

The directory server dumps core in the log file directory, which by default is /var/log/dirsrv/slapd-INSTANCE

Is the crash easily reproducible?
> But when it segfaults no file is created in cwd.
>
> Can you help me, please?
>
> Thank you.
>
> Best regards,
>    Dael Maselli.
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

--
389 users mailing list
[e-mail]
https://admin.fedoraproject.org/mailman/listinfo/389-users

[Rich Megginson]

Reinhard Nappert wrote:
> Rick,
> the server dumps the logs into the configured working directory (nsslapd-workingdir attribute in cn=config), right?

Right.
> -Reinhard
>
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Rich Megginson
> Sent: Tuesday, September 07, 2010 10:56 AM
> To: General discussion list for the 389 Directory server project.
> Subject: Re: [389-users] Segfault & Core Dumps
>
> Dael Maselli wrote:
> > Hi,
> >
> > I'm experiencing a lot of segmentation fault on my installations, I 
> > have the latest release on EPEL on SL 5.4 (389-Directory/1.2.5 B2010.012.2034).
>
> Do you see seg fault messages in /var/log/messages?
> > I'm trying to get core dumps but without success. Is there a specific 
> > configuration that prevents/allows 389-ds to core dump?
> >
> > The default on linux, you know, is to have core dump soft limit to 0, 
> > so I changed it for 389 in /etc/sysconfig/dirsrv, adding the line:
> >   ulimit -c unlimited
> >
> > and now I get:
> >
> > # more /proc/`pidof ns-slapd`/limits | grep core
> > Max core file size        unlimited            unlimited            bytes
> >
> > Kernel parameters are at default:
> >
> > # sysctl -a | grep kernel.core
> > kernel.core_pattern = core
> > kernel.core_uses_pid = 1
> >
> > So, I expected a core dump in /proc/`pidof ns-slapd`/cwd/, right?
>
> The directory server dumps core in the log file directory, which by default is /var/log/dirsrv/slapd-INSTANCE
>
> Is the crash easily reproducible?
> > But when it segfaults no file is created in cwd.
> >
> > Can you help me, please?
> >
> > Thank you.
> >
> > Best regards,
> >    Dael Maselli.
> > --
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Dael Maselli]

Hi Rich,

On 07/09/10 16.56, Rich Megginson wrote:
> Do you see seg fault messages in /var/log/messages?

Sure: ns-slapd[13737]: segfault at 00000000000000bc rip 0000003abb420375 
rsp 00000000580d85d0 error 4
> The directory server dumps core in the log file directory, which by
> default is /var/log/dirsrv/slapd-INSTANCE

Yes, it is the same as working directory:
# ls -l /proc/`pidof ns-slapd`/cwd
lrwxrwxrwx 1 root root 0 Sep  7 17:12 /proc/18721/cwd -> 
/var/log/dirsrv/slapd-ds1
> Is the crash easily reproducible?

No, it isn't. It seems random, but I can simulate a crash with kill -QUIT.

I tried killing a simple `sleep 10`:

# ulimit -c unlimited

# sleep 10 &
[1] 19726

# kill -QUIT 19726
[1]+  Quit                    (core dumped) sleep 10

# ls -l core.*
-rw------- 1 root root 290816 Aug 31 08:52 core.1008

But if I kill -QUIT ns-slapd no file is created.

Thanks,
    Dael Maselli.

[Ulf Weltman]

On 9/7/2010 8:25 AM, Dael Maselli wrote:
> Hi Rich,
>
> On 07/09/10 16.56, Rich Megginson wrote:
> > Do you see seg fault messages in /var/log/messages?
>
> Sure: ns-slapd[13737]: segfault at 00000000000000bc rip 0000003abb420375
> rsp 00000000580d85d0 error 4
> > The directory server dumps core in the log file directory, which by
> > default is /var/log/dirsrv/slapd-INSTANCE
>
> Yes, it is the same as working directory:
> # ls -l /proc/`pidof ns-slapd`/cwd
> lrwxrwxrwx 1 root root 0 Sep  7 17:12 /proc/18721/cwd ->
> /var/log/dirsrv/slapd-ds1
> > Is the crash easily reproducible?
>
> No, it isn't. It seems random, but I can simulate a crash with kill -QUIT.
>
> I tried killing a simple `sleep 10`:
>
> # ulimit -c unlimited
>
> # sleep 10&
> [1] 19726
>
> # kill -QUIT 19726
> [1]+  Quit                    (core dumped) sleep 10
>
> # ls -l core.*
> -rw------- 1 root root 290816 Aug 31 08:52 core.1008
>
> But if I kill -QUIT ns-slapd no file is created.

ns-slapd typically runs as setuid to a non-root user.  Check what 
fs.suid_dumpable or kernel.suid_dumpable are set to.
> Thanks,
>    Dael Maselli.

[Dael Maselli]

It worked!

I wrote fs.suid_dumpable=1 in /etc/sysctl.conf, `sysctl -p` and 
restarted dirsrv. Now it dumps.

Thank you!

I will report here the backtrace when it occurs.

Regards,
    Dael Maselli.

On 07/09/10 19.44, Ulf Weltman wrote:
> On 9/7/2010 8:25 AM, Dael Maselli wrote:
> > Hi Rich,
> >
> > On 07/09/10 16.56, Rich Megginson wrote:
> > > Do you see seg fault messages in /var/log/messages?
> >
> > Sure: ns-slapd[13737]: segfault at 00000000000000bc rip 0000003abb420375
> > rsp 00000000580d85d0 error 4
> > > The directory server dumps core in the log file directory, which by
> > > default is /var/log/dirsrv/slapd-INSTANCE
> >
> > Yes, it is the same as working directory:
> > # ls -l /proc/`pidof ns-slapd`/cwd
> > lrwxrwxrwx 1 root root 0 Sep 7 17:12 /proc/18721/cwd ->
> > /var/log/dirsrv/slapd-ds1
> > > Is the crash easily reproducible?
> >
> > No, it isn't. It seems random, but I can simulate a crash with kill
> > -QUIT.
> >
> > I tried killing a simple `sleep 10`:
> >
> > # ulimit -c unlimited
> >
> > # sleep 10&
> > [1] 19726
> >
> > # kill -QUIT 19726
> > [1]+ Quit (core dumped) sleep 10
> >
> > # ls -l core.*
> > -rw------- 1 root root 290816 Aug 31 08:52 core.1008
> >
> > But if I kill -QUIT ns-slapd no file is created.
>
> ns-slapd typically runs as setuid to a non-root user. Check what
> fs.suid_dumpable or kernel.suid_dumpable are set to.
> > Thanks,
> > Dael Maselli.
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Rich Megginson]

Dael Maselli wrote:
> It worked!
>
> I wrote fs.suid_dumpable=1 in /etc/sysctl.conf, `sysctl -p` and 
> restarted dirsrv. Now it dumps.
>
> Thank you!
>
> I will report here the backtrace when it occurs.

Great!  Be sure to install the 389-ds-base-debuginfo package to get the 
symbols when generating the backtrace.
> Regards,
>     Dael Maselli.
>
> On 07/09/10 19.44, Ulf Weltman wrote:
> > On 9/7/2010 8:25 AM, Dael Maselli wrote:
> > > Hi Rich,
> > >
> > > On 07/09/10 16.56, Rich Megginson wrote:
> > > > Do you see seg fault messages in /var/log/messages?
> > >
> > > Sure: ns-slapd[13737]: segfault at 00000000000000bc rip 
> > > 0000003abb420375
> > > rsp 00000000580d85d0 error 4
> > > > The directory server dumps core in the log file directory, which by
> > > > default is /var/log/dirsrv/slapd-INSTANCE
> > >
> > > Yes, it is the same as working directory:
> > > # ls -l /proc/`pidof ns-slapd`/cwd
> > > lrwxrwxrwx 1 root root 0 Sep 7 17:12 /proc/18721/cwd ->
> > > /var/log/dirsrv/slapd-ds1
> > > > Is the crash easily reproducible?
> > >
> > > No, it isn't. It seems random, but I can simulate a crash with kill
> > > -QUIT.
> > >
> > > I tried killing a simple `sleep 10`:
> > >
> > > # ulimit -c unlimited
> > >
> > > # sleep 10&
> > > [1] 19726
> > >
> > > # kill -QUIT 19726
> > > [1]+ Quit (core dumped) sleep 10
> > >
> > > # ls -l core.*
> > > -rw------- 1 root root 290816 Aug 31 08:52 core.1008
> > >
> > > But if I kill -QUIT ns-slapd no file is created.
> >
> > ns-slapd typically runs as setuid to a non-root user. Check what
> > fs.suid_dumpable or kernel.suid_dumpable are set to.
> > > Thanks,
> > > Dael Maselli.
> >
> > -- 
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Dael Maselli]

Ok, I backtraced the coredump, the problem was in /usr/lib64/libssl3.so.

I updated from "nss-3.12.6-1.el5_4.x86_64" to "nss-3.12.7-2.el5.x86_64" 
and now all seems to work fine!

Thank you.

Regards,
     Dael Maselli.

On 08/09/10 15.16, Rich Megginson wrote:
> Dael Maselli wrote:
> > It worked!
> >
> > I wrote fs.suid_dumpable=1 in /etc/sysctl.conf, `sysctl -p` and
> > restarted dirsrv. Now it dumps.
> >
> > Thank you!
> >
> > I will report here the backtrace when it occurs.
>
> Great!  Be sure to install the 389-ds-base-debuginfo package to get the
> symbols when generating the backtrace.
> > Regards,
> >      Dael Maselli.
> >
> > On 07/09/10 19.44, Ulf Weltman wrote:
> > > On 9/7/2010 8:25 AM, Dael Maselli wrote:
> > > > Hi Rich,
> > > >
> > > > On 07/09/10 16.56, Rich Megginson wrote:
> > > > > Do you see seg fault messages in /var/log/messages?
> > > >
> > > > Sure: ns-slapd[13737]: segfault at 00000000000000bc rip
> > > > 0000003abb420375
> > > > rsp 00000000580d85d0 error 4
> > > > > The directory server dumps core in the log file directory, which by
> > > > > default is /var/log/dirsrv/slapd-INSTANCE
> > > >
> > > > Yes, it is the same as working directory:
> > > > # ls -l /proc/`pidof ns-slapd`/cwd
> > > > lrwxrwxrwx 1 root root 0 Sep 7 17:12 /proc/18721/cwd ->
> > > > /var/log/dirsrv/slapd-ds1
> > > > > Is the crash easily reproducible?
> > > >
> > > > No, it isn't. It seems random, but I can simulate a crash with kill
> > > > -QUIT.
> > > >
> > > > I tried killing a simple `sleep 10`:
> > > >
> > > > # ulimit -c unlimited
> > > >
> > > > # sleep 10&
> > > > [1] 19726
> > > >
> > > > # kill -QUIT 19726
> > > > [1]+ Quit (core dumped) sleep 10
> > > >
> > > > # ls -l core.*
> > > > -rw------- 1 root root 290816 Aug 31 08:52 core.1008
> > > >
> > > > But if I kill -QUIT ns-slapd no file is created.
> > >
> > > ns-slapd typically runs as setuid to a non-root user. Check what
> > > fs.suid_dumpable or kernel.suid_dumpable are set to.
> > > > Thanks,
> > > > Dael Maselli.
> > >
> > > -- 
> > > 389 users mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/389-users
> >
> > --
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Rich Megginson]

Dael Maselli wrote:
> Ok, I backtraced the coredump, the problem was in /usr/lib64/libssl3.so.
>
> I updated from "nss-3.12.6-1.el5_4.x86_64" to 
> "nss-3.12.7-2.el5.x86_64" and now all seems to work fine!

Thanks, good to know.
> Thank you.
>
> Regards,
>     Dael Maselli.
>
> On 08/09/10 15.16, Rich Megginson wrote:
> > Dael Maselli wrote:
> > > It worked!
> > >
> > > I wrote fs.suid_dumpable=1 in /etc/sysctl.conf, `sysctl -p` and
> > > restarted dirsrv. Now it dumps.
> > >
> > > Thank you!
> > >
> > > I will report here the backtrace when it occurs.
> >
> > Great!  Be sure to install the 389-ds-base-debuginfo package to get the
> > symbols when generating the backtrace.
> > > Regards,
> > >      Dael Maselli.
> > >
> > > On 07/09/10 19.44, Ulf Weltman wrote:
> > > > On 9/7/2010 8:25 AM, Dael Maselli wrote:
> > > > > Hi Rich,
> > > > >
> > > > > On 07/09/10 16.56, Rich Megginson wrote:
> > > > > > Do you see seg fault messages in /var/log/messages?
> > > > >
> > > > > Sure: ns-slapd[13737]: segfault at 00000000000000bc rip
> > > > > 0000003abb420375
> > > > > rsp 00000000580d85d0 error 4
> > > > > > The directory server dumps core in the log file directory, which by
> > > > > > default is /var/log/dirsrv/slapd-INSTANCE
> > > > >
> > > > > Yes, it is the same as working directory:
> > > > > # ls -l /proc/`pidof ns-slapd`/cwd
> > > > > lrwxrwxrwx 1 root root 0 Sep 7 17:12 /proc/18721/cwd ->
> > > > > /var/log/dirsrv/slapd-ds1
> > > > > > Is the crash easily reproducible?
> > > > >
> > > > > No, it isn't. It seems random, but I can simulate a crash with kill
> > > > > -QUIT.
> > > > >
> > > > > I tried killing a simple `sleep 10`:
> > > > >
> > > > > # ulimit -c unlimited
> > > > >
> > > > > # sleep 10&
> > > > > [1] 19726
> > > > >
> > > > > # kill -QUIT 19726
> > > > > [1]+ Quit (core dumped) sleep 10
> > > > >
> > > > > # ls -l core.*
> > > > > -rw------- 1 root root 290816 Aug 31 08:52 core.1008
> > > > >
> > > > > But if I kill -QUIT ns-slapd no file is created.
> > > >
> > > > ns-slapd typically runs as setuid to a non-root user. Check what
> > > > fs.suid_dumpable or kernel.suid_dumpable are set to.
> > > > > Thanks,
> > > > > Dael Maselli.
> > > >
> > > > -- 
> > > > 389 users mailing list
> > > > [e-mail]
> > > > https://admin.fedoraproject.org/mailman/listinfo/389-users
> > >
> > > -- 
> > > 389 users mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/389-users
> >
> > -- 
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users

[Roberto Polli]

On Tuesday 07 September 2010 17:25:05 Dael Maselli wrote:
> .. I can simulate a crash with kill -QUIT.

maybe the sleep command doesn't trap this signal, thus  generating a core file 
like in # man 7 signal.
> But if I kill -QUIT ns-slapd no file is created.

slapd will  trap the QUIT and treat it as a proper EXIT
> ~/tmp/fedora-ds-base-1.1.2# egrep -r SIGQUIT .
> ./lib/base/file.cpp:    signal(SIGQUIT, EXITFUNC);
> ./ldap/servers/slapd/tools/ldclt/ldclt.c:  sigaddset (&(act.sa_mask),

SIGQUIT);
> ./ldap/servers/slapd/tools/ldclt/ldclt.c:  if (sigaction (SIGQUIT, &act,

NULL) < 0)

Moreover just quitting won't create the right core file (the one with the 
boundary condition resulting in segfault).

HTH+Peace,
R.