Startup segfault loop of domain_be on Ubuntu 14.04

[Chris Hartman]

Hi guys,

Ubuntu 14.04 x86_64
SSSD 1.11.5
Active Directory backend

I've been experiencing this problem for a few weeks with no luck in
troubleshooting it. During boot, the sssd_be process will continually spawn
and crash due to a segfault. When it gets stuck in this loop, domain
account login is almost impossible (local accounts still work).

I've attached sanitized logs and my sssd.conf file. Please have a look and
let me know what you think. Thanks!

-Chris

[Jakub Hrozek]

On 09 Jul 2014, at 17:09, Chris Hartman <qrstuv(a)gmail.com&gt; wrote:

>  Hi guys,
>  
>  Ubuntu 14.04 x86_64
>  SSSD 1.11.5
>  Active Directory backend
>   
Hi Chris,

thanks for reporting this issue. I think we have a bug in the dynamic DNS update code.
Here are the last couple of lines before SSSD crashed:
(Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]] [sdap_id_op_connect_step] (0x4000):
reusing cached connection
(Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]] [be_nsupdate_done] (0x0200): nsupdate
child status: 0
(Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]] [nsupdate_msg_create_common] (0x0200):
Creating update message for realm [SANEDOMAIN.LOCAL].
(Wed Jul  9 10:10:42 2014) [sssd[be[sanedomain]]] [server_setup] (0x0400): CONFDB:
/var/lib/sss/db/config.ldb

“server_setup” is the first function that gets executed after SSSD restarts, so the lines
right before that indicate where SSSD was before the crash.

Is it possible to generate a core file? That would help debugging quite a lot. I realise
the core file can contain private information, so feel free to send it to me directly. If
that not acceptable either, generate a backtrace with “bt full” in a gdb session, then you
can send a sanitised backtrace.

I think that the following option (in the domain section) should help you work around the
problem:
dyndns_update = False

Please let us know if disabling the DNS update helped and if you can generate the core
file.

Thanks again for the problem report!

>  I've been experiencing this problem for a few weeks with no luck
> in troubleshooting it. During boot, the sssd_be process will continually spawn and crash
> due to a segfault. When it gets stuck in this loop, domain account login is almost
> impossible (local accounts still work).
>  
>  I've attached sanitized logs and my sssd.conf file. Please have a look and let me
> know what you think. Thanks!
>  
>  -Chris
>  <logs.tar.gz>_______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users

[Chris Hartman]

Hi Jakub,

Is it possible to generate a core file? That would help debugging quite a
>  lot. 
Thanks for your response. I'd be happy to generate a core file for you but
I'm afraid I'm going to need some guidance.

My google-fu has gotten me a little familiar with what commands are
involved, but I'm not exactly sure how I'd go about generating or
retrieving a core dump, especially because the sssd_be process is the one
that's crashing, not the main sssd process.

Would you be able provide assistance?

I think that the following option (in the domain section) should help you
>  work around the problem:
>  dyndns_update = False 

This actually didn't work for me, but this did: dyndns_update_ptr = False

So, apparently, the bug is related the ptr update procedure?

Thanks again!



-Chris


On Wed, Jul 9, 2014 at 5:18 PM, Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:

>
> On 09 Jul 2014, at 17:09, Chris Hartman <qrstuv(a)gmail.com&gt; wrote:
>
> > Hi guys,
> >
> > Ubuntu 14.04 x86_64
> > SSSD 1.11.5
> > Active Directory backend
> >
>
> Hi Chris,
>
> thanks for reporting this issue. I think we have a bug in the dynamic DNS
> update code. Here are the last couple of lines before SSSD crashed:
> (Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]]
> [sdap_id_op_connect_step] (0x4000): reusing cached connection
> (Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]] [be_nsupdate_done]
> (0x0200): nsupdate child status: 0
> (Wed Jul  9 10:10:39 2014) [sssd[be[sanedomain]]]
> [nsupdate_msg_create_common] (0x0200): Creating update message for realm
> [SANEDOMAIN.LOCAL].
> (Wed Jul  9 10:10:42 2014) [sssd[be[sanedomain]]] [server_setup] (0x0400):
> CONFDB: /var/lib/sss/db/config.ldb
>
> “server_setup” is the first function that gets executed after SSSD
> restarts, so the lines right before that indicate where SSSD was before the
> crash.
>
> Is it possible to generate a core file? That would help debugging quite a
> lot. I realise the core file can contain private information, so feel free
> to send it to me directly. If that not acceptable either, generate a
> backtrace with “bt full” in a gdb session, then you can send a sanitised
> backtrace.
>
> I think that the following option (in the domain section) should help you
>  work around the problem:
>  dyndns_update = False 
>
> Please let us know if disabling the DNS update helped and if you can
> generate the core file.
>
> Thanks again for the problem report!
>
> > I've been experiencing this problem for a few weeks with no luck in
> troubleshooting it. During boot, the sssd_be process will continually spawn
> and crash due to a segfault. When it gets stuck in this loop, domain
> account login is almost impossible (local accounts still work).
> >
> > I've attached sanitized logs and my sssd.conf file. Please have a look
> and let me know what you think. Thanks!
> >
> > -Chris
> > <logs.tar.gz>_______________________________________________
> > sssd-users mailing list
> > sssd-users(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>
>

[Lukas Slebodnik]

On (10/07/14 14:30), Chris Hartman wrote:
> Hi Jakub,
>
> Is it possible to generate a core file? That would help debugging quite a
> > lot.
>
> Thanks for your response. I'd be happy to generate a core file for you but
> I'm afraid I'm going to need some guidance.
>
> My google-fu has gotten me a little familiar with what commands are
> involved, but I'm not exactly sure how I'd go about generating or
> retrieving a core dump, especially because the sssd_be process is the one
> that's crashing, not the main sssd process.
>
> Would you be able provide assistance? 
At first, you will need to install package
with debug symbols.
Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04.

I found a howto on debian wiki, which can help you,
https://wiki.debian.org/HowToGetABacktrace#Rebuilding_the_package_you.2BI...

If it is too complicated for you. You can try to file a bug to ubuntu and they
should help you with obtaining coredump(backtrace).

LS

[Jakub Hrozek]

On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
>  On (10/07/14 14:30), Chris Hartman wrote:
>  >Hi Jakub,
>  >
>  >Is it possible to generate a core file? That would help debugging quite a
>  >> lot.
>  >
>  >Thanks for your response. I'd be happy to generate a core file for you but
>  >I'm afraid I'm going to need some guidance.
>  >
>  >My google-fu has gotten me a little familiar with what commands are
>  >involved, but I'm not exactly sure how I'd go about generating or
>  >retrieving a core dump, especially because the sssd_be process is the one
>  >that's crashing, not the main sssd process.
>  >
>  >Would you be able provide assistance?
>  At first, you will need to install package with debug symbols.
>  Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04. 
Maybe Timo (CC) has some tips here?

>  
>  I found a howto on debian wiki, which can help you,
>  https://wiki.debian.org/HowToGetABacktrace#Rebuilding_the_package_you.2BI...
>  
>  If it is too complicated for you. You can try to file a bug to ubuntu and they
>  should help you with obtaining coredump(backtrace).

[Timo Aaltonen]

11.07.2014 11:42, Jakub Hrozek kirjoitti:
>  On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
> > On (10/07/14 14:30), Chris Hartman wrote:
> >> Hi Jakub,
> >>
> >> Is it possible to generate a core file? That would help debugging quite a
> >>> lot.
> >>
> >> Thanks for your response. I'd be happy to generate a core file for you but
> >> I'm afraid I'm going to need some guidance.
> >>
> >> My google-fu has gotten me a little familiar with what commands are
> >> involved, but I'm not exactly sure how I'd go about generating or
> >> retrieving a core dump, especially because the sssd_be process is the one
> >> that's crashing, not the main sssd process.
> >>
> >> Would you be able provide assistance?
> > At first, you will need to install package with debug symbols.
> > Unfortunately, I was not able to find debug symbols for sssd in ubuntu 14.04.
>  
>  Maybe Timo (CC) has some tips here? 
Ubuntu provides ddebs created on package build that you can install, see:

https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages


it'll get in Debian too eventually:

https://wiki.debian.org/AutomaticDebugPackages


-- 
t

[Chris Hartman]

Sorry it's been so long since my original email. Life happens sometimes. At
any rate, I'm able to continue with debugging now.

First off, thanks to Timo and Lukas for their help! I was able to generate
the backtrace and core dump file as Jakub suggested. Since it may contain
sensitive data, I will email him the files directly.

Jakub, look for an email from my momentarily. Thanks for taking the time to
help me out!


-Chris


On Mon, Jul 14, 2014 at 4:07 PM, Timo Aaltonen <tjaalton(a)ubuntu.com&gt; wrote:

>  11.07.2014 11:42, Jakub Hrozek kirjoitti:
>  > On Thu, Jul 10, 2014 at 11:52:18PM +0200, Lukas Slebodnik wrote:
>  >> On (10/07/14 14:30), Chris Hartman wrote:
>  >>> Hi Jakub,
>  >>>
>  >>> Is it possible to generate a core file? That would help debugging
>  quite a
>  >>>> lot.
>  >>>
>  >>> Thanks for your response. I'd be happy to generate a core file for you
>  but
>  >>> I'm afraid I'm going to need some guidance.
>  >>>
>  >>> My google-fu has gotten me a little familiar with what commands are
>  >>> involved, but I'm not exactly sure how I'd go about generating or
>  >>> retrieving a core dump, especially because the sssd_be process is the
>  one
>  >>> that's crashing, not the main sssd process.
>  >>>
>  >>> Would you be able provide assistance?
>  >> At first, you will need to install package with debug symbols.
>  >> Unfortunately, I was not able to find debug symbols for sssd in ubuntu
>  14.04.
>  >
>  > Maybe Timo (CC) has some tips here?
>
>  Ubuntu provides ddebs created on package build that you can install, see:
>
>  https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages
>
>
>  it'll get in Debian too eventually:
>
>  https://wiki.debian.org/AutomaticDebugPackages
>
>
>  --
>  t
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>

[Lukas Slebodnik]

On (07/08/14 14:57), Chris Hartman wrote:
> Sorry it's been so long since my original email. Life happens
> sometimes. At
> any rate, I'm able to continue with debugging now.
>
> First off, thanks to Timo and Lukas for their help! I was able to generate
> the backtrace and core dump file as Jakub suggested. Since it may contain
> sensitive data, I will email him the files directly.
>
> Jakub, look for an email from my momentarily. Thanks for taking the time to
> help me out!
>  
[New LWP 32083]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'.
#0  0xb775d424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb775d424 in __kernel_vsyscall ()
#1  0xb7520d7c in __fdatasync_nocancel () at ../sysdeps/unix/syscall-template.S:81
#2  0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268, length=4) at
../common/transaction.c:548
#3  0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at ../common/transaction.c:603
#4  0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at ../common/transaction.c:1161
#5  0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at ../ldb_tdb/ldb_tdb.c:1141
#6  0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at ../common/ldb.c:467
#7  0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78, req=0x8b38500) at
../common/ldb.c:563
#8  0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380) at
../common/ldb.c:1641

I am not really sure why but crash is in syscall fdatasync.
As you can see, top 9 frames are from libraries libldb and libtdb.
If it is a easy to reproduce could you try to mount tmpfs to the directory
/var/lib/sss/db/. It can be filesystem issue.

tmpfs   /var/lib/sss/db/ tmpfs                  size=300M,mode=0700,noauto 0 0

LS

[Jakub Hrozek]

On Fri, Aug 08, 2014 at 06:49:56PM +0200, Lukas Slebodnik wrote:
>  On (07/08/14 14:57), Chris Hartman wrote:
>  >Sorry it's been so long since my original email. Life happens sometimes. At
>  >any rate, I'm able to continue with debugging now.
>  >
>  >First off, thanks to Timo and Lukas for their help! I was able to generate
>  >the backtrace and core dump file as Jakub suggested. Since it may contain
>  >sensitive data, I will email him the files directly.
>  >
>  >Jakub, look for an email from my momentarily. Thanks for taking the time to
>  >help me out!
>  >
>  
>  [New LWP 32083]
>  [Thread debugging using libthread_db enabled]
>  Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
>  Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'.
>  #0  0xb775d424 in __kernel_vsyscall ()
>  (gdb) bt
>  #0  0xb775d424 in __kernel_vsyscall ()
>  #1  0xb7520d7c in __fdatasync_nocancel () at ../sysdeps/unix/syscall-template.S:81
>  #2  0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268, length=4) at
> ../common/transaction.c:548
>  #3  0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at ../common/transaction.c:603
>  #4  0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at ../common/transaction.c:1161
>  #5  0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at ../ldb_tdb/ldb_tdb.c:1141
>  #6  0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at
> ../common/ldb.c:467
>  #7  0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78, req=0x8b38500) at
> ../common/ldb.c:563
>  #8  0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380) at
> ../common/ldb.c:1641
>  
>  I am not really sure why but crash is in syscall fdatasync.
>  As you can see, top 9 frames are from libraries libldb and libtdb.
>  If it is a easy to reproduce could you try to mount tmpfs to the directory
>  /var/lib/sss/db/. It can be filesystem issue.
>  
>  tmpfs   /var/lib/sss/db/ tmpfs                  size=300M,mode=0700,noauto 0 0
>   
Thanks for looking into this!

Did you notice anything strange about the ldb_message in frame 8? That's
the only thing I can think of that we might influence.

Do you know if Ubuntu uses the latest tdb and ldb versions?

[Chris Hartman]

Hi guys,

If it is a easy to reproduce could you try to mount tmpfs to the directory
>  /var/lib/sss/db/. It can be filesystem issue. 
Mounting with a tmpfs has no effect on the crash.

Do you know if Ubuntu uses the latest tdb and ldb versions?

libtdb1 version in Ubuntu 14.04: 1.2.12-1
libldb1 version in Ubuntu 14.04: 1.1.16-1

According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0
and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17

For full disclosure, I already told this to Jakub in case it matters:

I ran into a little trouble because the sssd_be was the binary that was
>  actually crashing, but it was spawned automatically by the root sssd
>  process. In order to generate the coredump and backtrace, I came up with
>  this inelegant solution:
>
>  service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut
>  -f1 -d" ") -ex generate-core-file
>  
Hope this helps.


-Chris


On Fri, Aug 8, 2014 at 1:16 PM, Jakub Hrozek <jhrozek(a)redhat.com&gt; wrote:

> On Fri, Aug 08, 2014 at 06:49:56PM +0200, Lukas Slebodnik wrote:
> > On (07/08/14 14:57), Chris Hartman wrote:
> > >Sorry it's been so long since my original email. Life happens
> sometimes. At
> > >any rate, I'm able to continue with debugging now.
> > >
> > >First off, thanks to Timo and Lukas for their help! I was able to
> generate
> > >the backtrace and core dump file as Jakub suggested. Since it may
> contain
> > >sensitive data, I will email him the files directly.
> > >
> > >Jakub, look for an email from my momentarily. Thanks for taking the
> time to
> > >help me out!
> > >
> >
> > [New LWP 32083]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library
"/lib/i386-linux-gnu/libthread_db.so.1".
> > Core was generated by `/usr/lib/i386-linux-gnu/sssd/sssd_be'.
> > #0  0xb775d424 in __kernel_vsyscall ()
> > (gdb) bt
> > #0  0xb775d424 in __kernel_vsyscall ()
> > #1  0xb7520d7c in __fdatasync_nocancel () at
> ../sysdeps/unix/syscall-template.S:81
> > #2  0xb6af2be5 in transaction_sync (tdb=0x8b2a138, offset=53268,
> length=4) at ../common/transaction.c:548
> > #3  0xb6af2d9e in _tdb_transaction_cancel (tdb=0x8b2a138) at
> ../common/transaction.c:603
> > #4  0xb6af3c95 in tdb_transaction_commit (tdb=0x8b2a138) at
> ../common/transaction.c:1161
> > #5  0xb6b172d5 in ltdb_end_trans (module=0x8b2b9b0) at
> ../ldb_tdb/ldb_tdb.c:1141
> > #6  0xb770aedf in ldb_transaction_commit (ldb=ldb@entry=0x8b20a78) at
> ../common/ldb.c:467
> > #7  0xb770be49 in ldb_autotransaction_request (ldb=0x8b20a78,
> req=0x8b38500) at ../common/ldb.c:563
> > #8  0xb770ca43 in ldb_modify (ldb=0x8b20a78, message=message@entry=0x8b49380)
> at ../common/ldb.c:1641
> >
> > I am not really sure why but crash is in syscall fdatasync.
> > As you can see, top 9 frames are from libraries libldb and libtdb.
> > If it is a easy to reproduce could you try to mount tmpfs to the
> directory
>
>  /var/lib/sss/db/. It can be filesystem issue. 
> >
> > tmpfs   /var/lib/sss/db/ tmpfs
>  size=300M,mode=0700,noauto 0 0
> >
>
> Thanks for looking into this!
>
> Did you notice anything strange about the ldb_message in frame 8? That's
> the only thing I can think of that we might influence.
>
> Do you know if Ubuntu uses the latest tdb and ldb versions?
> _______________________________________________
> sssd-users mailing list
> sssd-users(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>

[Lukas Slebodnik]

On (08/08/14 14:48), Chris Hartman wrote:
> Hi guys,
>
> If it is a easy to reproduce could you try to mount tmpfs to the directory
> > /var/lib/sss/db/. It can be filesystem issue.
>
> Mounting with a tmpfs has no effect on the crash.
>
> Do you know if Ubuntu uses the latest tdb and ldb versions?
>
> libtdb1 version in Ubuntu 14.04: 1.2.12-1
> libldb1 version in Ubuntu 14.04: 1.1.16-1 
Fedora has almost the same versions.
[user@host ~]$ rpm -q libtdb libldb
libtdb-1.2.12-2.fc20.x86_64
libldb-1.1.16-4.fc20.x86_64

> According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0
> and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
>
> For full disclosure, I already told this to Jakub in case it matters:
>
> I ran into a little trouble because the sssd_be was the binary that was
> > actually crashing, but it was spawned automatically by the root sssd
> > process. In order to generate the coredump and backtrace, I came up with
> > this inelegant solution:
> >
> > service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut
> > -f1 -d" ") -ex generate-core-file 
That's good way how to obtain
coredump. I can be simplified if you have just
one domain in sssd.conf

service sssd start; sleep 3; gdb -p $(pgrep sssd_be) -ex generate-core-file

> >
>
> Hope this helps.
>  
Could you send me more coredumps privately with fuly updated ubuntu 14.04?
Because it really strange that sssd crashed in syscal fdatasync.

LS

[Lukas Slebodnik]

On (08/08/14 14:48), Chris Hartman wrote:
> Hi guys,
>
> If it is a easy to reproduce could you try to mount tmpfs to the directory
> > /var/lib/sss/db/. It can be filesystem issue.
>
> Mounting with a tmpfs has no effect on the crash.
>
> Do you know if Ubuntu uses the latest tdb and ldb versions?
>
> libtdb1 version in Ubuntu 14.04: 1.2.12-1
> libldb1 version in Ubuntu 14.04: 1.1.16-1
>
> According to http://www.samba.org/ftp/tdb/ the latest tbd version is 1.3.0
> and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
>
> For full disclosure, I already told this to Jakub in case it matters:
>
> I ran into a little trouble because the sssd_be was the binary that was
> > actually crashing, but it was spawned automatically by the root sssd
> > process. In order to generate the coredump and backtrace, I came up with
> > this inelegant solution:
> >
> > service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v grep|cut
> > -f1 -d" ") -ex generate-core-file
> >
>
> Hope this helps.
>  
I have another idea which could help us to to find a problem.
We can try to run sssd_be with valgrind.

1) we need to find out arguments for process sssd_be
sh-4.2$ pgrep -af sssd_be
1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     this output will be used in sssd.conf

2) add new option to domain section in sssd.conf
command = valgrind -v --show-reachable=yes
--log-file=/var/log/sssd/valgrind_idm.example.com.log /usr/libexec/sssd/sssd_be --domain
idm.example.com --debug-to-files

    # do not forget to replace output from previous step.


3) It would be good to install debug symbols for libraries libtdb1 and libldb1

4) reproduce problem
    some interesting output can be in log file from valgrind.

LS

[Chris Hartman]

Greetings,

Could you send me more coredumps privately with fuly updated ubuntu 14.04?
>  Because it really strange that sssd crashed in syscal fdatasync.
Lukas, not a problem. Stand by for an email with a 7zip archive of a core
dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed
for this.

I have another idea which could help us to to find a problem.
>  We can try to run sssd_be with valgrind.
>  1) we need to find out arguments for process sssd_be
>  sh-4.2$ pgrep -af sssd_be
>  1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>       this output will be used in sssd.conf
>  2) add new option to domain section in sssd.conf
>  command = valgrind -v --show-reachable=yes
> --log-file=/var/log/sssd/valgrind_idm.example.com.log
>  /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>      # do not forget to replace output from previous step.
>
>  3) It would be good to install debug symbols for libraries libtdb1 and
>  libldb1
>  4) reproduce problem
>      some interesting output can be in log file from valgrind. 
 I did this with interesting(?) results. The sssd_be process never actually
crashed but there is some erroneous output in the valgrind log. Removing
the special command option results in a crash as expected. I've sanitized
and attached the log to this email. Let me know if that reveals anything.


-Chris


On Tue, Aug 12, 2014 at 9:29 AM, Lukas Slebodnik <lslebodn(a)redhat.com&gt;
wrote:

>  On (08/08/14 14:48), Chris Hartman wrote:
>  >Hi guys,
>  >
>  >If it is a easy to reproduce could you try to mount tmpfs to the directory
>  >> /var/lib/sss/db/. It can be filesystem issue.
>  >
>  >Mounting with a tmpfs has no effect on the crash.
>  >
>  >Do you know if Ubuntu uses the latest tdb and ldb versions?
>  >
>  >libtdb1 version in Ubuntu 14.04: 1.2.12-1
>  >libldb1 version in Ubuntu 14.04: 1.1.16-1
>  >
>  >According to http://www.samba.org/ftp/tdb/ the latest tbd version is
>  1.3.0
>  >and http://www.samba.org/ftp/ldb/ says the latest ldb version is 1.1.17
>  >
>  >For full disclosure, I already told this to Jakub in case it matters:
>  >
>  >I ran into a little trouble because the sssd_be was the binary that was
>  >> actually crashing, but it was spawned automatically by the root sssd
>  >> process. In order to generate the coredump and backtrace, I came up with
>  >> this inelegant solution:
>  >>
>  >> service sssd start; sleep 3; gdb -p $(ps ax|grep sssd_be|grep -v
>  grep|cut
>  >> -f1 -d" ") -ex generate-core-file
>  >>
>  >
>  >Hope this helps.
>  >
>  I have another idea which could help us to to find a problem.
>  We can try to run sssd_be with valgrind.
>
>  1) we need to find out arguments for process sssd_be
>  sh-4.2$ pgrep -af sssd_be
>  1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>       this output will be used in sssd.conf
>
>  2) add new option to domain section in sssd.conf
>  command = valgrind -v --show-reachable=yes
>  --log-file=/var/log/sssd/valgrind_idm.example.com.log
>  /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>
>      # do not forget to replace output from previous step.
>
>
>  3) It would be good to install debug symbols for libraries libtdb1 and
>  libldb1
>
>  4) reproduce problem
>      some interesting output can be in log file from valgrind.
>
>  LS
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>

[Lukas Slebodnik]

On (13/08/14 11:12), Chris Hartman wrote:
> Greetings,
>
> Could you send me more coredumps privately with fuly updated ubuntu 14.04?
> > Because it really strange that sssd crashed in syscal fdatasync.
>
> Lukas, not a problem. Stand by for an email with a 7zip archive of a core
> dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed
> for this.
>
> I have another idea which could help us to to find a problem.
> > We can try to run sssd_be with valgrind.
> > 1) we need to find out arguments for process sssd_be
> > sh-4.2$ pgrep -af sssd_be
> > 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
> >      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >      this output will be used in sssd.conf
> > 2) add new option to domain section in sssd.conf
> > command = valgrind -v --show-reachable=yes
> --log-file=/var/log/sssd/valgrind_idm.example.com.log
> > /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
> >     # do not forget to replace output from previous step.
> >
> > 3) It would be good to install debug symbols for libraries libtdb1 and
> > libldb1
> > 4) reproduce problem
> >     some interesting output can be in log file from valgrind.
>
>  I did this with interesting(?) results. The sssd_be process never actually
> crashed but there is some erroneous output in the valgrind log. Removing
> the special command option results in a crash as expected. I've sanitized
> and attached the log to this email. Let me know if that reveals anything.
>  
Thank you very much for log files from valgrind.
There is use after free problem in dynamic dns updates code.

Could you try to disable dynamic dns updates in domain section.
    dyndns_update = false

LS

[Chris Hartman]

Lukas,

Could you try to disable dynamic dns updates in domain section.
>      dyndns_update = false 

I can do that. Would you like valgrind logs? Another coredump? And just to
review, the crash happens only when 'dyndns_update_ptr' is true; the crash
doesn't seem to be affected by the 'dyndns_update' option.


-Chris


On Wed, Aug 13, 2014 at 1:08 PM, Lukas Slebodnik <lslebodn(a)redhat.com&gt;
wrote:

> On (13/08/14 11:12), Chris Hartman wrote:
> >Greetings,
> >
> >Could you send me more coredumps privately with fuly updated ubuntu 14.04?
> >> Because it really strange that sssd crashed in syscal fdatasync.
> >
> >Lukas, not a problem. Stand by for an email with a 7zip archive of a core
> >dump and back trace. The debugging symbols for lib{tdb,ldb}1 were
> installed
> >for this.
> >
> >I have another idea which could help us to to find a problem.
> >> We can try to run sssd_be with valgrind.
> >> 1) we need to find out arguments for process sssd_be
> >> sh-4.2$ pgrep -af sssd_be
> >> 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com
> --debug-to-files
> >>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>      this output will be used in sssd.conf
> >> 2) add new option to domain section in sssd.conf
> >> command = valgrind -v --show-reachable=yes
> --log-file=/var/log/sssd/valgrind_idm.example.com.log
> >> /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
> >>     # do not forget to replace output from previous step.
> >>
> >> 3) It would be good to install debug symbols for libraries libtdb1 and
> >> libldb1
> >> 4) reproduce problem
> >>     some interesting output can be in log file from valgrind.
> >
> > I did this with interesting(?) results. The sssd_be process never
> actually
> >crashed but there is some erroneous output in the valgrind log. Removing
> >the special command option results in a crash as expected. I've sanitized
> >and attached the log to this email. Let me know if that reveals anything.
> >
> Thank you very much for log files from valgrind.
> There is use after free problem in dynamic dns updates code.
>
> Could you try to disable dynamic dns updates in domain section.
>      dyndns_update = false 
>
> LS
> _______________________________________________
> sssd-users mailing list
> sssd-users(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>

[Jakub Hrozek]

On Wed, Aug 13, 2014 at 07:08:33PM +0200, Lukas Slebodnik wrote:
>  On (13/08/14 11:12), Chris Hartman wrote:
>  >Greetings,
>  >
>  >Could you send me more coredumps privately with fuly updated ubuntu 14.04?
>  >> Because it really strange that sssd crashed in syscal fdatasync.
>  >
>  >Lukas, not a problem. Stand by for an email with a 7zip archive of a core
>  >dump and back trace. The debugging symbols for lib{tdb,ldb}1 were installed
>  >for this.
>  >
>  >I have another idea which could help us to to find a problem.
>  >> We can try to run sssd_be with valgrind.
>  >> 1) we need to find out arguments for process sssd_be
>  >> sh-4.2$ pgrep -af sssd_be
>  >> 1191 /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>  >>      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  >>      this output will be used in sssd.conf
>  >> 2) add new option to domain section in sssd.conf
>  >> command = valgrind -v --show-reachable=yes
> --log-file=/var/log/sssd/valgrind_idm.example.com.log
>  >> /usr/libexec/sssd/sssd_be --domain idm.example.com --debug-to-files
>  >>     # do not forget to replace output from previous step.
>  >>
>  >> 3) It would be good to install debug symbols for libraries libtdb1 and
>  >> libldb1
>  >> 4) reproduce problem
>  >>     some interesting output can be in log file from valgrind.
>  >
>  > I did this with interesting(?) results. The sssd_be process never actually
>  >crashed but there is some erroneous output in the valgrind log. Removing
>  >the special command option results in a crash as expected. I've sanitized
>  >and attached the log to this email. Let me know if that reveals anything.
>  >
>  Thank you very much for log files from valgrind.
>  There is use after free problem in dynamic dns updates code.
>  
>  Could you try to disable dynamic dns updates in domain section.
>      dyndns_update = false
>  
>  LS 
Thank you very much for debugging, Chris!

I filed https://fedorahosted.org/sssd/ticket/2405 to track this problem.

[Lukas Slebodnik]

On (13/08/14 13:53), Chris Hartman wrote:
> Lukas,
>
> Could you try to disable dynamic dns updates in domain section.
> >     dyndns_update = false
>
>
> I can do that. Would you like valgrind logs? Another coredump? And just to
> review, the crash happens only when 'dyndns_update_ptr' is true; the crash
> doesn't seem to be affected by the 'dyndns_update' option.
>  
Thank you very much for your time and all log files.

Finally, I am able to reproduce bug locally.

LS

[Lukas Slebodnik]

On (09/07/14 11:09), Chris Hartman wrote:
> Hi guys,
>
> Ubuntu 14.04 x86_64
> SSSD 1.11.5
> Active Directory backend
>
> I've been experiencing this problem for a few weeks with no luck in
> troubleshooting it. During boot, the sssd_be process will continually spawn
> and crash due to a segfault. When it gets stuck in this loop, domain
> account login is almost impossible (local accounts still work).
>
> I've attached sanitized logs and my sssd.conf file. Please have a look and
> let me know what you think. Thanks!
>  
Patch was accepted in the sssd upstream
https://git.fedorahosted.org/cgit/sssd.git/commit/?id=0060992d68ba843d4d9...

Thank you very much for report and help with troubleshooting this bug.

LS