[Freeipa-users] sssd is going down and up and down and up and down and ... until it breaks

[Harald Dunkel]

Hi folks,

Apparently sssd goes down and up again and again. I found this in
/var/log/daemon.log on our git server:

Jul 23 18:02:08 git01 sssd[be[example.de]]: Shutting down
Jul 23 18:02:08 git01 sssd[pam]: Shutting down
Jul 23 18:02:08 git01 sssd[nss]: Shutting down
Jul 23 18:02:09 git01 sssd[pam]: Starting up
Jul 23 18:02:09 git01 sssd[nss]: Starting up
Jul 23 18:02:09 git01 sssd[be[example.de]]: Starting up
Jul 23 18:02:11 git01 sssd[nss]: Starting up
Jul 23 18:02:11 git01 sssd[pam]: Starting up
Jul 23 20:01:33 git01 sssd[nss]: Shutting down
Jul 23 20:01:33 git01 sssd[pam]: Shutting down
Jul 23 20:01:33 git01 sssd[be[example.de]]: Shutting down
Jul 23 20:01:33 git01 sssd[nss]: Starting up
Jul 23 20:01:33 git01 sssd[pam]: Starting up
Jul 23 20:01:33 git01 sssd[be[example.de]]: Starting up
Jul 23 20:01:35 git01 sssd[nss]: Starting up
Jul 23 20:01:35 git01 sssd[pam]: Starting up
Jul 23 20:02:44 git01 sssd[nss]: Shutting down
Jul 23 20:02:44 git01 sssd[nss]: Starting up
Jul 23 20:03:43 git01 sssd[nss]: Shutting down
Jul 23 20:03:43 git01 sssd[pam]: Shutting down
Jul 23 20:03:43 git01 sssd[nss]: Starting up
Jul 23 20:03:43 git01 sssd[pam]: Starting up
Jul 23 20:06:24 git01 sssd[be[example.de]]: Shutting down
Jul 23 20:06:24 git01 sssd[be[example.de]]: Starting up
Jul 23 20:07:34 git01 sssd[be[example.de]]: Shutting down
Jul 23 20:07:37 git01 sssd[pam]: Shutting down
Jul 23 20:07:37 git01 sssd[be[example.de]]: Starting up
Jul 23 20:07:37 git01 sssd[pam]: Starting up
Jul 23 20:07:37 git01 sssd[pam]: Starting up
Jul 23 20:14:39 git01 sssd[pam]: Shutting down
Jul 23 20:14:39 git01 sssd[be[example.de]]: Starting up
Jul 23 20:14:39 git01 sssd[pam]: Starting up
Jul 23 20:18:44 git01 sssd[be[example.de]]: Shutting down
Jul 23 20:18:44 git01 sssd[pam]: Shutting down
Jul 23 20:18:44 git01 sssd[be[example.de]]: Starting up
Jul 23 20:18:44 git01 sssd[pam]: Starting up
Jul 24 04:05:28 git01 sssd[pam]: Shutting down
Jul 24 04:05:28 git01 sssd[pam]: Starting up
Jul 24 05:21:53 git01 sssd[be[example.de]]: Shutting down
Jul 24 05:21:53 git01 sssd[be[example.de]]: Starting up
Jul 24 05:27:50 git01 sssd[pam]: Shutting down
Jul 24 05:27:50 git01 sssd[pam]: Starting up
Jul 24 05:27:50 git01 sssd[be[example.de]]: Starting up
Jul 24 05:27:53 git01 sssd[pam]: Starting up
Jul 24 05:30:31 git01 sssd[pam]: Shutting down
Jul 24 05:30:31 git01 sssd[pam]: Starting up
Jul 24 05:31:59 git01 sssd[nss]: Shutting down
Jul 24 05:31:59 git01 sssd[pam]: Shutting down
Jul 24 05:31:59 git01 sssd[nss]: Starting up
Jul 24 05:31:59 git01 sssd[be[example.de]]: Shutting down
Jul 24 05:31:59 git01 sssd[pam]: Starting up
Jul 24 05:31:59 git01 sssd[be[example.de]]: Starting up
Jul 24 05:32:01 git01 sssd[pam]: Starting up
Jul 24 05:33:24 git01 sssd[pam]: Shutting down
Jul 24 05:33:24 git01 sssd[pam]: Starting up
Jul 24 05:33:24 git01 sssd[be[example.de]]: Starting up
Jul 24 06:01:38 git01 sssd[pam]: Shutting down
Jul 24 06:01:38 git01 sssd[be[example.de]]: Starting up
Jul 24 06:01:38 git01 sssd[pam]: Starting up
Jul 24 06:02:39 git01 sssd[be[example.de]]: Shutting down
Jul 24 06:02:39 git01 sssd[be[example.de]]: Starting up
Jul 24 09:56:52 git01 sssd[pam]: Shutting down
Jul 24 09:56:52 git01 sssd[pam]: Starting up
Jul 24 10:02:42 git01 sssd[nss]: Shutting down
Jul 24 10:02:42 git01 sssd[pam]: Shutting down
Jul 24 10:02:42 git01 sssd[nss]: Starting up
Jul 24 10:02:42 git01 sssd[pam]: Starting up
Jul 24 10:02:42 git01 sssd[nss]: Shutting down
Jul 24 10:02:42 git01 sssd[pam]: Shutting down
Jul 24 10:02:42 git01 sssd[nss]: Starting up
Jul 24 10:02:42 git01 sssd[pam]: Starting up
Jul 24 10:02:42 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:02:42 git01 sssd[be[example.de]]: Starting up
Jul 24 10:06:14 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:06:14 git01 sssd[nss]: Shutting down
Jul 24 10:06:14 git01 sssd[nss]: Starting up
Jul 24 10:06:14 git01 sssd[be[example.de]]: Starting up
Jul 24 10:06:14 git01 sssd[nss]: Starting up
Jul 24 10:15:49 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:15:49 git01 sssd[be[example.de]]: Starting up
Jul 24 10:16:44 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:17:00 git01 sssd[pam]: Shutting down
Jul 24 10:17:00 git01 sssd[pam]: Starting up
Jul 24 10:17:00 git01 sssd[be[example.de]]: Starting up
Jul 24 10:17:00 git01 sssd[pam]: Starting up
Jul 24 10:18:48 git01 sssd[pam]: Shutting down
Jul 24 10:18:48 git01 sssd[pam]: Starting up
Jul 24 10:19:43 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:19:43 git01 sssd[be[example.de]]: Starting up
Jul 24 10:20:32 git01 sssd[pam]: Shutting down
Jul 24 10:20:32 git01 sssd[pam]: Starting up
Jul 24 10:21:12 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:21:15 git01 sssd[be[example.de]]: Starting up
Jul 24 10:27:11 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:27:11 git01 sssd[pam]: Shutting down
Jul 24 10:27:11 git01 sssd[pam]: Starting up
Jul 24 10:27:11 git01 sssd[be[example.de]]: Starting up
Jul 24 10:27:13 git01 sssd[pam]: Starting up
Jul 24 10:30:03 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:30:03 git01 sssd[be[example.de]]: Starting up
Jul 24 10:42:31 git01 sssd[pam]: Shutting down
Jul 24 10:42:31 git01 sssd[pam]: Starting up
Jul 24 10:42:31 git01 sssd[be[example.de]]: Starting up
Jul 24 10:54:45 git01 sssd[pam]: Shutting down
Jul 24 10:54:45 git01 sssd[be[example.de]]: Shutting down
Jul 24 10:54:45 git01 sssd[pam]: Starting up
Jul 24 10:54:45 git01 sssd[be[example.de]]: Starting up
Jul 24 10:54:47 git01 sssd[pam]: Starting up
Jul 24 11:01:37 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:01:37 git01 sssd[be[example.de]]: Starting up
Jul 24 11:04:09 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:04:09 git01 sssd[be[example.de]]: Starting up
Jul 24 11:29:04 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:29:04 git01 sssd[be[example.de]]: Starting up
Jul 24 11:31:29 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:31:29 git01 sssd[be[example.de]]: Starting up
Jul 24 11:37:54 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:37:54 git01 sssd[be[example.de]]: Starting up
Jul 24 11:39:35 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:39:43 git01 sssd[be[example.de]]: Starting up
Jul 24 11:41:13 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:41:25 git01 sssd[be[example.de]]: Starting up
Jul 24 11:41:25 git01 sssd[nss]: Shutting down
Jul 24 11:41:25 git01 sssd[nss]: Starting up
Jul 24 11:41:54 git01 [sssd[krb5_child[33646]]]: Preauthentication failed
Jul 24 11:41:54 git01 [sssd[krb5_child[33646]]]: Preauthentication failed
Jul 24 11:42:27 git01 sssd[nss]: Shutting down
Jul 24 11:42:44 git01 sssd[nss]: Starting up
Jul 24 11:42:46 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:42:46 git01 sssd[be[example.de]]: Starting up
Jul 24 11:46:37 git01 sssd[pam]: Shutting down
Jul 24 11:46:37 git01 sssd[pam]: Starting up
Jul 24 11:48:33 git01 sssd[be[example.de]]: Shutting down
Jul 24 11:48:36 git01 sssd[pam]: Shutting down
Jul 24 11:48:36 git01 sssd[pam]: Starting up
Jul 24 11:48:36 git01 sssd[be[example.de]]: Starting up
Jul 24 11:48:36 git01 sssd[pam]: Starting up
Jul 24 12:01:27 git01 sssd[pam]: Shutting down
Jul 24 12:01:27 git01 sssd[be[example.de]]: Shutting down
Jul 24 12:01:27 git01 sssd[pam]: Starting up
Jul 24 12:01:27 git01 sssd[be[example.de]]: Starting up
Jul 24 12:01:29 git01 sssd[pam]: Starting up
Jul 25 12:01:35 git01 sssd[pam]: Shutting down
Jul 25 12:01:35 git01 sssd[pam]: Starting up
Jul 25 18:03:58 git01 systemd[1]: sssd.service: Main process exited, code=exited,
status=1/FAILURE
Jul 25 18:03:58 git01 systemd[1]: sssd.service: Unit entered failed state.
Jul 25 18:03:58 git01 systemd[1]: sssd.service: Failed with result 'exit-code'.
Jul 25 18:03:58 git01 sssd[be[example.de]]: Shutting down
Jul 25 18:03:58 git01 sssd[nss]: Shutting down
Jul 25 18:03:58 git01 sssd[nss]: Starting up
Jul 25 18:03:58 git01 sssd[nss]: Starting up
Jul 25 18:03:58 git01 sssd[nss]: Starting up
Jul 25 18:03:58 git01 sssd: Exiting the SSSD. Could not restart critical service [nss].
Jul 25 18:03:58 git01 sssd[be[example.de]]: Starting up
Jul 25 18:03:58 git01 sssd[be[example.de]]: Shutting down
Jul 25 18:03:58 git01 sssd[pam]: Shutting down
Jul 25 18:03:58 git01 sssd[pac]: Shutting down
Jul 25 18:15:32 git01 sssd: Starting up
Jul 25 18:15:32 git01 sssd[be[example.de]]: Starting up
Jul 25 18:15:32 git01 sssd[pam]: Starting up
Jul 25 18:15:32 git01 sssd[nss]: Starting up
Jul 25 18:15:32 git01 sssd[pac]: Starting up
Jul 26 10:25:16 git01 sssd[be[example.de]]: Shutting down
Jul 26 10:25:16 git01 sssd[pam]: Shutting down
Jul 26 10:25:16 git01 sssd[nss]: Shutting down
Jul 26 10:25:16 git01 sssd[pac]: Shutting down
Jul 26 10:25:16 git01 sssd: Starting up
Jul 26 10:25:16 git01 sssd[be[example.de]]: Starting up
Jul 26 10:25:17 git01 sssd[pam]: Starting up
Jul 26 10:25:17 git01 sssd[nss]: Starting up
Jul 26 10:25:17 git01 sssd[pac]: Starting up

I wonder why it doesn't keep on running?

Why did it fail completely at 18:03? This triggered an emergency
and had to be resolved manually.

The journal shows

Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Main process exited,
code=exited, status=1/FAILURE
Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Unit entered failed state.
Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Failed with result
'exit-code'.
Jul 25 18:01:52 git01.ac.example.de sssd[be[43718]: Shutting down
Jul 25 18:01:52 git01.ac.example.de sssd[33790]: Shutting down
Jul 25 18:01:56 git01.ac.example.de sssd[100999]: Starting up
Jul 25 18:01:58 git01.ac.example.de sssd[101071]: Starting up
Jul 25 18:02:02 git01.ac.example.de sssd[101163]: Starting up
Jul 25 18:02:45 git01.ac.example.de sssd[73]: Exiting the SSSD. Could not restart critical
service [nss].
Jul 25 18:02:45 git01.ac.example.de sssd[be[101036]: Starting up
Jul 25 18:02:45 git01.ac.example.de sssd[be[101036]: Shutting down
Jul 25 18:02:45 git01.ac.example.de sssd[75226]: Shutting down
Jul 25 18:02:45 git01.ac.example.de sssd[134]: Shutting down
Jul 25 18:03:00 git01.ac.example.de sshd[101666]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101683]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101682]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101684]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101690]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101689]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101691]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101688]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:03:00 git01.ac.example.de sshd[101696]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:04:00 git01.ac.example.de sshd[102433]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:04:00 git01.ac.example.de sshd[102432]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:04:00 git01.ac.example.de sshd[102431]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:04:00 git01.ac.example.de sshd[102437]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103008]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103007]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103009]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103032]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103031]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103030]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:05:00 git01.ac.example.de sshd[103036]: pam_sss(sshd:account): Request to sssd
failed. Connection refused
Jul 25 18:06:28 git01.ac.example.de sshd[103823]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:06:36 git01.ac.example.de sshd[103823]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:08:11 git01.ac.example.de sshd[105080]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:08:24 git01.ac.example.de sshd[105182]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:08:38 git01.ac.example.de sshd[105365]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:10:50 git01.ac.example.de sshd[106920]: pam_sss(sshd:auth): Request to sssd
failed. Connection refused
Jul 25 18:15:32 git01.ac.example.de sssd[110413]: Starting up
Jul 25 18:15:32 git01.ac.example.de sssd[be[110415]: Starting up
Jul 25 18:15:32 git01.ac.example.de sssd[110418]: Starting up
Jul 25 18:15:32 git01.ac.example.de sssd[110417]: Starting up
Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
Jul 25 18:15:32 git01.ac.example.de sssd[110419]: Starting up
Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 2
Jul 25 18:30:42 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1

sssd doesn't write critical errors into its log files by default :-(.
There is no information about this in /var/log/sssd. Of course I have
increased verbosity now.


This was sssd version 1.15.2 on Debian 9. I have upgraded the system
to sssd 1.16.1.


Every helpful comment is highly appreciated.
Harri