We've had ns-slapd segfault on us recently twice; we don't have a core dump (since the daemon script turns off core dumps, but hopefully we'll have one next time it happens) and I was wondering if anyone had seen this before:
ns-slapd[2725]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2abf04040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000] ns-slapd[2727]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2aab02040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000]
Cheers, Edward
Edward Z. Yang wrote:
We've had ns-slapd segfault on us recently twice; we don't have a core dump (since the daemon script turns off core dumps, but hopefully we'll have one next time it happens) and I was wondering if anyone had seen this before:
ns-slapd[2725]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2abf04040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000] ns-slapd[2727]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2aab02040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000]
I don't think ns-slapd even links against libcrypto. This is very strange.
Cheers, Edward -- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Excerpts from Rich Megginson's message of Mon Sep 20 09:19:50 -0400 2010:
ns-slapd[2725]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2abf04040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000] ns-slapd[2727]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2aab02040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000]
I don't think ns-slapd even links against libcrypto. This is very strange.
We use GSSAPI auth for multimaster replication; perhaps that uses libcrypto?
Edward
Edward Z. Yang wrote:
Excerpts from Rich Megginson's message of Mon Sep 20 09:19:50 -0400 2010:
ns-slapd[2725]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2abf04040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000] ns-slapd[2727]: segfault at 10a3000010af ip 0000003d58c95785 sp 00007ff2aab02040 error 4 in libcrypto.so.0.9.8n[3d58c00000+15b000]
I don't think ns-slapd even links against libcrypto. This is very strange.
We use GSSAPI auth for multimaster replication; perhaps that uses libcrypto?
Edward
What platform? What version of 389-ds-base? Can you install the 389-ds-base-debuginfo package? You will have to enable the directory server to produce a core dump. 1) edit /etc/sysconfig/dirsrv-instancename - add ulimit -c unlimited 2) sysctl -w fs.suid_dumpable=1
Excerpts from Rich Megginson's message of Mon Sep 20 11:37:32 -0400 2010:
What platform? What version of 389-ds-base? Can you install the 389-ds-base-debuginfo package?
Platform: Fedora 11 (yes, I know it's EOL'd, we're migrating this weekend)
[root@pancake-bunny etc]# ns-slapd --version 389 Project 389-Directory/1.2.5 B2010.012.2024
You will have to enable the directory server to produce a core dump.
- edit /etc/sysconfig/dirsrv-instancename - add ulimit -c unlimited
- sysctl -w fs.suid_dumpable=1
Done. We'll send a core dump your way the next time it happens.
Edward
Excerpts from Rich Megginson's message of Mon Sep 20 11:37:32 -0400 2010:
What platform? What version of 389-ds-base? Can you install the 389-ds-base-debuginfo package? You will have to enable the directory server to produce a core dump.
- edit /etc/sysconfig/dirsrv-instancename - add ulimit -c unlimited
- sysctl -w fs.suid_dumpable=1
Hi Rich,
We've seen this error on 1.2.6 (F13) too; unfortunately, we're having a bit of difficulty turning on core dumping.
ns-slapd[10129]: segfault at 74c1 ip 00007f637caff675 sp 00007f636e97acb0 error 4 in libcrypto.so.1.0.0a[7f637ca4b000+171000]
The end of our dirsrv sysconfig is:
ulimit -c unlimited DAEMON_COREFILE_LIMIT=unlimited
and we've managed to get a core dump by sending a manual SEGV to the ns-slapd process. I'm not sure why this particular segfault didn't get a dump. Any insight?
Edward
Edward Z. Yang wrote:
Excerpts from Rich Megginson's message of Mon Sep 20 11:37:32 -0400 2010:
What platform? What version of 389-ds-base? Can you install the 389-ds-base-debuginfo package? You will have to enable the directory server to produce a core dump.
- edit /etc/sysconfig/dirsrv-instancename - add ulimit -c unlimited
- sysctl -w fs.suid_dumpable=1
Hi Rich,
We've seen this error on 1.2.6 (F13) too; unfortunately, we're having a bit of difficulty turning on core dumping.
ns-slapd[10129]: segfault at 74c1 ip 00007f637caff675 sp 00007f636e97acb0 error 4 in libcrypto.so.1.0.0a[7f637ca4b000+171000]
The end of our dirsrv sysconfig is:
ulimit -c unlimited DAEMON_COREFILE_LIMIT=unlimited
and we've managed to get a core dump by sending a manual SEGV to the ns-slapd process. I'm not sure why this particular segfault didn't get a dump. Any insight?
I just don't know - if you have
ulimit -c unlimited
and
sysctl -w fs.suid_dumpable=1
It should allow core dumps in /var/log/dirsrv/slapd-instancename
Edward
After manually attaching GDB, we caught a segfault on one of the dirsrvs. The server's name is old-faithful. Here's the backtrace (with one set of debugging info missing; I can grab that and reload the core dump if you want me to.) File a bug, or do you think it's an upstream bug? The core dump probably contains sensitive info, so let us know if there are specific things you want us to do with it.
(gdb) thread 10 [Switching to thread 10 (Thread 31829)]#0 getrn (lh=0x7fe34c004d20, data=0x7fe36ac45350, rhash=<value optimized out>) at lhash.c:425 425 if (n1->hash != hash) (gdb) bt #0 getrn (lh=0x7fe34c004d20, data=0x7fe36ac45350, rhash=<value optimized out>) at lhash.c:425 #1 0x00007fe36a96d9d6 in lh_insert (lh=0x7fe34c004d20, data=0x7fe36ac45350) at lhash.c:189 #2 0x00007fe36a9700c9 in int_err_set_item (d=0x7fe36ac45350) at err.c:407 #3 0x00007fe36a96fa4c in err_load_strings (lib=0, str=0x7fe36ac45350) at err.c:676 #4 0x00007fe36a970fb1 in ERR_load_ERR_strings () at err.c:662 #5 0x00007fe36a971199 in ERR_load_crypto_strings () at err_all.c:113 #6 0x00007fe36022d857 in openssl_init (cryptoctx=0x7fe358006218) at pkinit_crypto_openssl.c:2350 #7 pkinit_init_plg_crypto (cryptoctx=0x7fe358006218) at pkinit_crypto_openssl.c:277 #8 0x00007fe360222b68 in pkinit_client_plugin_init (context=0x7fe358003830, blob=0x7fe3612b0e40) at pkinit_clnt.c:1362 #9 0x0000003ab92557eb in krb5_init_preauth_context (kcontext=0x7fe358003830) at preauth2.c:146 #10 0x0000003ab925628a in krb5_preauth_request_context_init (context=0x7fe358003830) at preauth2.c:305 #11 0x0000003ab9249ee6 in krb5_get_init_creds (context=0x7fe358003830, creds=0x7fe3612b13c0, client=0x7fe3580021c0, prompter=0, prompter_data=0x0, start_time=0, in_tkt_service=0x0, options=0x7fe358002a50, gak_fct=0x3ab924bac0 <krb5_get_as_key_keytab>, gak_data=0x7fe3580021a0, use_master=0x7fe3612b123c, as_reply=0x0) at get_in_tkt.c:1167 #12 0x0000003ab924bcce in krb5_get_init_creds_keytab (context=0x7fe358003830, creds=0x7fe3612b13c0, client=0x7fe3580021c0, arg_keytab=0x7fe3580021a0, start_time=0, in_tkt_service=0x0, options=0x0) at gic_keytab.c:109 #13 0x0000003c618602f7 in set_krb5_creds (authid=<value optimized out>, username=<value optimized out>, passwd=<value optimized out>, realm=<value optimized out>, vals=0x7fe358002bd0) at ldap/servers/slapd/ldaputil.c:1483 #14 0x0000003c61860a95 in ldap_sasl_set_interact_vals (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=0x7fe358000d40 "", mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:928 #15 slapd_ldap_sasl_interactive_bind (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=0x7fe358000d40 "", mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:1044 #16 0x0000003c61860d17 in slapi_ldap_bind (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=<value optimized out>, mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, timeout=0x0, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:751 #17 0x00007fe3685927db in bind_and_check_pwp (conn=0x1480220) at ldap/servers/plugins/replication/repl5_connection.c:1750 #18 conn_connect (conn=0x1480220) at ldap/servers/plugins/replication/repl5_connection.c:1136 #19 0x00007fe36859aee7 in acquire_replica (prp=0x1480380, prot_oid=<value optimized out>, ruv=0x7fe3612b1d20) at ldap/servers/plugins/replication/repl5_protocol_util.c:168 #20 0x00007fe368595897 in repl5_inc_run (prp=0x1480380) at ldap/servers/plugins/replication/repl5_inc_protocol.c:828 #21 0x00007fe36859a3d5 in prot_thread_main (arg=0x14aedd0) at ldap/servers/plugins/replication/repl5_protocol.c:311 #22 0x0000003abee29593 in _pt_root (arg=0x14ad530) at ../../../mozilla/nsprpub/pr/src/pthreads/ptthread.c:228 #23 0x0000003ab6207761 in start_thread (arg=0x7fe3612b2710) at pthread_create.c:301 #24 0x0000003ab5ae151d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Cheers, Edward
Edward Z. Yang wrote:
After manually attaching GDB, we caught a segfault on one of the dirsrvs. The server's name is old-faithful. Here's the backtrace (with one set of debugging info missing; I can grab that and reload the core dump if you want me to.) File a bug, or do you think it's an upstream bug? The core dump probably contains sensitive info, so let us know if there are specific things you want us to do with it.
Try running with the SHELL (1024) debug error log level. This should give more information about the principal, keytab, etc. that directory server is using.
What is the platform? Are you using a newer version of kerberos?
(gdb) thread 10 [Switching to thread 10 (Thread 31829)]#0 getrn (lh=0x7fe34c004d20, data=0x7fe36ac45350, rhash=<value optimized out>) at lhash.c:425 425 if (n1->hash != hash) (gdb) bt #0 getrn (lh=0x7fe34c004d20, data=0x7fe36ac45350, rhash=<value optimized out>) at lhash.c:425 #1 0x00007fe36a96d9d6 in lh_insert (lh=0x7fe34c004d20, data=0x7fe36ac45350) at lhash.c:189 #2 0x00007fe36a9700c9 in int_err_set_item (d=0x7fe36ac45350) at err.c:407 #3 0x00007fe36a96fa4c in err_load_strings (lib=0, str=0x7fe36ac45350) at err.c:676 #4 0x00007fe36a970fb1 in ERR_load_ERR_strings () at err.c:662 #5 0x00007fe36a971199 in ERR_load_crypto_strings () at err_all.c:113 #6 0x00007fe36022d857 in openssl_init (cryptoctx=0x7fe358006218) at pkinit_crypto_openssl.c:2350 #7 pkinit_init_plg_crypto (cryptoctx=0x7fe358006218) at pkinit_crypto_openssl.c:277 #8 0x00007fe360222b68 in pkinit_client_plugin_init (context=0x7fe358003830, blob=0x7fe3612b0e40) at pkinit_clnt.c:1362 #9 0x0000003ab92557eb in krb5_init_preauth_context (kcontext=0x7fe358003830) at preauth2.c:146 #10 0x0000003ab925628a in krb5_preauth_request_context_init (context=0x7fe358003830) at preauth2.c:305 #11 0x0000003ab9249ee6 in krb5_get_init_creds (context=0x7fe358003830, creds=0x7fe3612b13c0, client=0x7fe3580021c0, prompter=0, prompter_data=0x0, start_time=0, in_tkt_service=0x0, options=0x7fe358002a50, gak_fct=0x3ab924bac0 <krb5_get_as_key_keytab>, gak_data=0x7fe3580021a0, use_master=0x7fe3612b123c, as_reply=0x0) at get_in_tkt.c:1167 #12 0x0000003ab924bcce in krb5_get_init_creds_keytab (context=0x7fe358003830, creds=0x7fe3612b13c0, client=0x7fe3580021c0, arg_keytab=0x7fe3580021a0, start_time=0, in_tkt_service=0x0, options=0x0) at gic_keytab.c:109 #13 0x0000003c618602f7 in set_krb5_creds (authid=<value optimized out>, username=<value optimized out>, passwd=<value optimized out>, realm=<value optimized out>, vals=0x7fe358002bd0) at ldap/servers/slapd/ldaputil.c:1483 #14 0x0000003c61860a95 in ldap_sasl_set_interact_vals (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=0x7fe358000d40 "", mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:928 #15 slapd_ldap_sasl_interactive_bind (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=0x7fe358000d40 "", mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:1044 #16 0x0000003c61860d17 in slapi_ldap_bind (ld=0x7fe358002270, bindid=0x7fe358003700 "uid=ldap/old-faithful.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=<value optimized out>, mech=<value optimized out>, serverctrls=0x0, returnedctrls=0x7fe3612b1ab8, timeout=0x0, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:751 #17 0x00007fe3685927db in bind_and_check_pwp (conn=0x1480220) at ldap/servers/plugins/replication/repl5_connection.c:1750 #18 conn_connect (conn=0x1480220) at ldap/servers/plugins/replication/repl5_connection.c:1136 #19 0x00007fe36859aee7 in acquire_replica (prp=0x1480380, prot_oid=<value optimized out>, ruv=0x7fe3612b1d20) at ldap/servers/plugins/replication/repl5_protocol_util.c:168 #20 0x00007fe368595897 in repl5_inc_run (prp=0x1480380) at ldap/servers/plugins/replication/repl5_inc_protocol.c:828 #21 0x00007fe36859a3d5 in prot_thread_main (arg=0x14aedd0) at ldap/servers/plugins/replication/repl5_protocol.c:311 #22 0x0000003abee29593 in _pt_root (arg=0x14ad530) at ../../../mozilla/nsprpub/pr/src/pthreads/ptthread.c:228 #23 0x0000003ab6207761 in start_thread (arg=0x7fe3612b2710) at pthread_create.c:301 #24 0x0000003ab5ae151d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Cheers, Edward
Excerpts from Rich Megginson's message of Fri Oct 08 18:59:52 -0400 2010:
Try running with the SHELL (1024) debug error log level. This should give more information about the principal, keytab, etc. that directory server is using.
More logs:
[09/Oct/2010:04:29:48 -0400] - Listening on /var/run/dirsrv/slapd-scripts.socket for LDAPI requests [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [better-mousetrap.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - configpluginpath == NULL [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [whole-enchilada.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [cats-whiskers.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:N0KZtwJ] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:N0KZtwJ Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:N0KZtwJ] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:fyHs1On] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:fyHs1On Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:fyHs1On] [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:aIeSCnz] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:aIeSCnz Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:aIeSCnz] [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER
What is the platform? Are you using a newer version of kerberos?
Fedora 13. We have the latest version of Kerberos with one custom patch:
Name : krb5-libs Arch : x86_64 Version : 1.7.1 Release : 14.fc13.scripts.1671 Size : 1.7 M Repo : installed
From repo : scripts
Summary : The shared libraries used by Kerberos 5 URL : http://web.mit.edu/kerberos/www/ License : MIT Description : Kerberos is a network authentication system. The krb5-libs package : contains the shared libraries needed by Kerberos 5. If you are using : Kerberos, you need to install this package.
that modifies src/lib/krb5/os/kuserok.c (which was not in the backtrace).
http://scripts.mit.edu/trac/browser/branches/fc13-dev/server/common/patches/...
Cheers, Edward
Edward Z. Yang wrote:
Excerpts from Rich Megginson's message of Fri Oct 08 18:59:52 -0400 2010:
Try running with the SHELL (1024) debug error log level. This should give more information about the principal, keytab, etc. that directory server is using.
More logs:
[09/Oct/2010:04:29:48 -0400] - Listening on /var/run/dirsrv/slapd-scripts.socket for LDAPI requests [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [better-mousetrap.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - configpluginpath == NULL [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [whole-enchilada.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] slapi_ldap_init_ext - Success: set up conn to [cats-whiskers.mit.edu:389] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - The default credentials cache [FILE:/tmp/krb5cc_485] not found: will create a new one. [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using principal named [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Using keytab named [WRFILE:/etc/dirsrv/keytab] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:N0KZtwJ] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:N0KZtwJ Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:N0KZtwJ] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:fyHs1On] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:fyHs1On Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:fyHs1On] [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Generated new memory ccache [MEMORY:aIeSCnz] [09/Oct/2010:04:29:48 -0400] show_cached_credentials - Ticket cache: MEMORY:aIeSCnz Default principal: ldap/old-faithful.mit.edu@ATHENA.MIT.EDU
[09/Oct/2010:04:29:48 -0400] show_one_credential - Kerberos credential: client [ldap/old-faithful.mit.edu@ATHENA.MIT.EDU] server [krbtgt/ATHENA.MIT.EDU@ATHENA.MIT.EDU] start time [Sat Oct 9 04:30:00 2010] end time [Sun Oct 10 01:45:00 2010] renew time [Sun Oct 10 04:29:49 2010] flags [0x50c00000] [09/Oct/2010:04:29:48 -0400] set_krb5_creds - Set new env for ccache: [KRB5CCNAME=MEMORY:aIeSCnz] [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER [09/Oct/2010:04:29:48 -0400] ldap_sasl_get_val - Using value [(null)] for SASL_CB_USER
What is the platform? Are you using a newer version of kerberos?
Fedora 13. We have the latest version of Kerberos with one custom patch:
Name : krb5-libs Arch : x86_64 Version : 1.7.1 Release : 14.fc13.scripts.1671 Size : 1.7 M Repo : installed From repo : scripts Summary : The shared libraries used by Kerberos 5 URL : http://web.mit.edu/kerberos/www/ License : MIT Description : Kerberos is a network authentication system. The krb5-libs package : contains the shared libraries needed by Kerberos 5. If you are using : Kerberos, you need to install this package.
that modifies src/lib/krb5/os/kuserok.c (which was not in the backtrace).
http://scripts.mit.edu/trac/browser/branches/fc13-dev/server/common/patches/...
Cheers, Edward
Thanks. Based upon this information and the stack traces you provided (Thanks!) it looks like the directory server is freeing something in the krb5_creds creds; that it should not be. The errors look like double free or realloc of already freed memory. I had to rely heavily on the 1.5 and 1.6 kerberos code to make sure I was using krb5_get_init_creds_keytab() and krb5_cc_store_cred() and krb5_free_cred_contents() correctly. It's quite likely that I did not, and the later version of kerberos changed something to "unmask" the problem. Please file a bug at https://bugzilla.redhat.com/enter_bug.cgi?product=389 and please attach your info and stack traces as attachments to the bug.
Here's another related trace. This one isn't a segfault but appears to be libc detecting corruption and bugging out (this explains why dirsrv has crashed and not left a segfault in dmesg).
#0 0x00000033e2433275 in *__GI_raise ( sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00000033e2434a55 in *__GI_abort () at abort.c:88 #2 0x00000033e246fb5b in __libc_message (do_abort=2, fmt=0x7f6dafffd820 "-7f6dc992a000 rw-p 00000000 00:00 0 \n7fffc2e61000-7fffc2e81000 rw-p 00000000 00:00 0", ' ' <repeats 26 times>, "[stack]\n7fffc2e81000-7fffc2e82000 r-xp 00000000 00:00 0", ' ' <repeats 26 times>, "[vdso]\nff"...) at ../sysdeps/unix/sysv/linux/libc_fatal.c:170 #3 0x00000033e2475506 in malloc_printerr (action=3, str=0x33e2535fd8 "double free or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6239 #4 0x00000033e247c280 in _int_realloc (av=0x7f6d9c000020, oldp=0x7f6d9c529430, oldsize=6, nb=2064) at malloc.c:5295 #5 0x00000033e247c8a5 in *__GI___libc_realloc ( oldmem=0x7f6d9c529440, bytes=2048) at malloc.c:3821 #6 0x0000003d58d01e1f in CRYPTO_realloc (str=0x7f6d9c529440, num=1408, file=0x3d58d29dac "stack.c", line=150) at mem.c:355 #7 0x0000003d58c954a0 in sk_insert (st=0x7f6d9c01c110, data=0x7f6da801ee40 "\220", loc=127) at stack.c:149 #8 0x0000003d58c9f49b in EVP_PBE_alg_add (nid=144, cipher=0x3d58f5bba0, md=0x3d58f60220, keygen=0x3d58cdbca0 <PKCS12_PBE_keyivgen>) at evp_pbe.c:129 #9 0x0000003d58cdbf1d in PKCS12_PBE_add () at p12_crpt.c:68 #10 0x0000003d58c9d552 in OpenSSL_add_all_ciphers () at c_allc.c:289 #11 0x0000003d58c9cc7e in OPENSSL_add_all_algorithms_noconf () at c_all.c:83 #12 0x00007f6dbc6e864c in BN_cmp () at bn_lib.c:665 #13 0x00007f6dbc6dd9d8 in BN_cmp () at bn_lib.c:665 #14 0x0000003945065293 in ?? () from /usr/lib64/libkrb5.so.3 #15 0x0000003945065cea in ?? () from /usr/lib64/libkrb5.so.3 #16 0x000000394505c4d1 in krb5_get_init_creds () from /usr/lib64/libkrb5.so.3 #17 0x000000394505d76e in krb5_get_init_creds_keytab () from /usr/lib64/libkrb5.so.3 #18 0x00007f6dc969b317 in set_krb5_creds ( authid=<value optimized out>, username=<value optimized out>, passwd=<value optimized out>, realm=<value optimized out>, vals=0x7f6da8090730) at ldap/servers/slapd/ldaputil.c:1408 #19 0x00007f6dc969bac5 in ldap_sasl_set_interact_vals ( passwd=<value optimized out>, username=<value optimized out>, authid=<value optimized out>, mech=<value optimized out>, ld=<value optimized out>, realm=<value optimized out>) at ldap/servers/slapd/ldaputil.c:853 #20 slapd_ldap_sasl_interactive_bind ( passwd=<value optimized out>, username=<value optimized out>, authid=<value optimized out>, mech=<value optimized out>, ld=<value optimized out>, realm=<value optimized out>) at ldap/servers/slapd/ldaputil.c:969 #21 0x00007f6dc969bd4a in slapi_ldap_bind (ld=0x7f6da8021e90, bindid=0x7f6da808fd30 "uid=ldap/cats-whiskers.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=<value optimized out>, mech=0x7f6dc40b803e "GSSAPI", serverctrls=0x0, returnedctrls=<value optimized out>, timeout=0x0, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:751 #22 0x00007f6dc40875d5 in bind_and_check_pwp ( password=<value optimized out>, binddn=<value optimized out>, conn=<value optimized out>) at ldap/servers/plugins/replication/repl5_connection.c:1676 #23 conn_connect (password=<value optimized out>, binddn=<value optimized out>, conn=<value optimized out>) at ldap/servers/plugins/replication/repl5_connection.c:1134 #24 0x00007f6dc408fa67 in acquire_replica (prp=0xf2ec80, prot_oid=<value optimized out>, ruv=0x7f6dafffef90) at ldap/servers/plugins/replication/repl5_protocol_util.c:168 #25 0x00007f6dc408a5c7 in repl5_inc_run (prp=0xf2ec80) at ldap/servers/plugins/replication/repl5_inc_protocol.c:814 #26 0x00007f6dc408f119 in prot_thread_main ( arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:313 #27 0x00000038a6e29773 in ?? () from /lib64/libnspr4.so #28 0x00000033e300685a in start_thread ( arg=<value optimized out>) at pthread_create.c:297 #29 0x00000033e24de22d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #30 0x0000000000000000 in ?? ()
Cheers, Edward
Another trace, not like the others.
#0 _int_malloc (av=0x7fff0cb91f80, bytes=32671600) at malloc.c:4294 #1 0x000000300ac7ab52 in *__GI___libc_malloc (bytes=15) at malloc.c:3660 #2 0x0000003ce3457481 in krb5_copy_principal () from /usr/lib64/libkrb5.so.3 #3 0x0000003ce3457040 in krb5_copy_creds () from /usr/lib64/libkrb5.so.3 #4 0x0000003ce3447077 in ?? () from /usr/lib64/libkrb5.so.3 #5 0x0000003ce344bfa2 in krb5_cc_store_cred () from /usr/lib64/libkrb5.so.3 #6 0x00007f471fdbc4c1 in set_krb5_creds ( authid=<value optimized out>, username=<value optimized out>, passwd=<value optimized out>, realm=<value optimized out>, vals=0x7f46f4093020) at ldap/servers/slapd/ldaputil.c:1483 #7 0x00007f471fdbcac5 in ldap_sasl_set_interact_vals ( passwd=<value optimized out>, username=<value optimized out>, authid=<value optimized out>, mech=<value optimized out>, ld=<value optimized out>, realm=<value optimized out>) at ldap/servers/slapd/ldaputil.c:853 #8 slapd_ldap_sasl_interactive_bind ( passwd=<value optimized out>, username=<value optimized out>, authid=<value optimized out>, mech=<value optimized out>, ld=<value optimized out>, realm=<value optimized out>) at ldap/servers/slapd/ldaputil.c:969 #9 0x00007f471fdbcd4a in slapi_ldap_bind (ld=0x7f46f4092c20, bindid=0x7f46f40c8f40 "uid=ldap/bees-knees.mit.edu,ou=People,dc=scripts,dc=mit,dc=edu", creds=<value optimized out>, mech=0x7f471a7f003e "GSSAPI", serverctrls=0x0, returnedctrls=<value optimized out>, timeout=0x0, msgidp=0x0) at ldap/servers/slapd/ldaputil.c:751 #10 0x00007f471a7bf5d5 in bind_and_check_pwp ( password=<value optimized out>, binddn=<value optimized out>, conn=<value optimized out>) at ldap/servers/plugins/replication/repl5_connection.c:1676 #11 conn_connect (password=<value optimized out>, binddn=<value optimized out>, conn=<value optimized out>) at ldap/servers/plugins/replication/repl5_connection.c:1134 #12 0x00007f471a7c7a67 in acquire_replica (prp=0x20c3880, prot_oid=<value optimized out>, ruv=0x7f470d579f90) at ldap/servers/plugins/replication/repl5_protocol_util.c:168 #13 0x00007f471a7c25c7 in repl5_inc_run (prp=0x20c3880) at ldap/servers/plugins/replication/repl5_inc_protocol.c:814 #14 0x00007f471a7c7119 in prot_thread_main ( arg=<value optimized out>) at ldap/servers/plugins/replication/repl5_protocol.c:313 #15 0x0000003facc29773 in ?? () from /lib64/libnspr4.so #16 0x000000300b80685a in start_thread ( arg=<value optimized out>) at pthread_create.c:297 #17 0x000000300acde22d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #18 0x0000000000000000 in ?? ()
Cheers, Edward
389-users@lists.fedoraproject.org
-
Edward Z. Yang -
Rich Megginson