3:15 p.m.
Over time, I’ve been having seemingly random sssd quits that I’ve not been able to figure
out. Today, I finally traced it to fluctuations on my Infiniband fabric:
sssd.log
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16
(Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface
idx 4 (ib0) flags 0x1003 (broadcast,multicast,up)
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16
(Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface
idx 4 (ib0) flags 0x11043 (broadcast,multicast,up,running,lower)
This exactly corresponds to the time in /var/log/messages for the unexplained shutdown:
2015-11-03T13:17:59-05:00 node75 sssd[pam]: Shutting down
2015-11-03T13:17:59-05:00 node75 sssd[be[default]]: Shutting down
2015-11-03T13:17:59-05:00 node75 sssd[nss]: Shutting down
Here is sssd_default.log for good measure:
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000):
0x1414770/0x14133d0
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000):
0x1414770/0x13fef90
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_ptask_destructor] (0x0400): Terminating
periodic task [Cleanup of default]
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sdap_handle_release] (0x2000): Trace:
sh[0x14bd850], connected[1], ops[(nil)], ldap[0x1424260], destructor_lock[0],
release_memory[0]
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_connection_callback] (0x4000):
Successfully removed connection callback.
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000):
0x1415970/0x1416430
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The
symlink points to [/var/lib/sss/pipes/private/sbus-dp_default.18702]
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The path
including our pid is [/var/lib/sss/pipes/private/sbus-dp_default.18702]
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): Removed
the symlink
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed
PAM client
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed
NSS client
I can duplicate this by manually taking down the Infiniband link:
[root@node24 ~]# service sssd status
sssd (pid 9132) is running...
[root@node24 ~]# ifdown ib0
[root@node24 ~]# service sssd status
sssd dead but pid file exists
I have also noticed that sssd will not start on boot. As I know that Infiniband tends to
flutter a little bit before the link comes up, I’m thinking this is probably the same
cause.
Can anyone explain this behavior and tell me what I might do to prevent it?
--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Senior Technologist
|| \\ and Health | novosirj(a)rutgers.edu - 973/972.0922 (2x0922)
|| \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
`'