[Freeipa-users] Re: sssd is going down and up and down and up and down and ... until it breaks

[John Hearns]

Harri, are you sure about your statement that sssd does not write critical
errors?
There are some very comprehensive logs. You are correct to increase the
verbosity level though.

I would advise heading over to the sssd-users list and asking the same
question.




On Thu, 26 Jul 2018 at 10:49, Harald Dunkel via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Hi folks,
>
>  Apparently sssd goes down and up again and again. I found this in
>  /var/log/daemon.log on our git server:
>
>  Jul 23 18:02:08 git01 sssd[be[example.de]]: Shutting down
>  Jul 23 18:02:08 git01 sssd[pam]: Shutting down
>  Jul 23 18:02:08 git01 sssd[nss]: Shutting down
>  Jul 23 18:02:09 git01 sssd[pam]: Starting up
>  Jul 23 18:02:09 git01 sssd[nss]: Starting up
>  Jul 23 18:02:09 git01 sssd[be[example.de]]: Starting up
>  Jul 23 18:02:11 git01 sssd[nss]: Starting up
>  Jul 23 18:02:11 git01 sssd[pam]: Starting up
>  Jul 23 20:01:33 git01 sssd[nss]: Shutting down
>  Jul 23 20:01:33 git01 sssd[pam]: Shutting down
>  Jul 23 20:01:33 git01 sssd[be[example.de]]: Shutting down
>  Jul 23 20:01:33 git01 sssd[nss]: Starting up
>  Jul 23 20:01:33 git01 sssd[pam]: Starting up
>  Jul 23 20:01:33 git01 sssd[be[example.de]]: Starting up
>  Jul 23 20:01:35 git01 sssd[nss]: Starting up
>  Jul 23 20:01:35 git01 sssd[pam]: Starting up
>  Jul 23 20:02:44 git01 sssd[nss]: Shutting down
>  Jul 23 20:02:44 git01 sssd[nss]: Starting up
>  Jul 23 20:03:43 git01 sssd[nss]: Shutting down
>  Jul 23 20:03:43 git01 sssd[pam]: Shutting down
>  Jul 23 20:03:43 git01 sssd[nss]: Starting up
>  Jul 23 20:03:43 git01 sssd[pam]: Starting up
>  Jul 23 20:06:24 git01 sssd[be[example.de]]: Shutting down
>  Jul 23 20:06:24 git01 sssd[be[example.de]]: Starting up
>  Jul 23 20:07:34 git01 sssd[be[example.de]]: Shutting down
>  Jul 23 20:07:37 git01 sssd[pam]: Shutting down
>  Jul 23 20:07:37 git01 sssd[be[example.de]]: Starting up
>  Jul 23 20:07:37 git01 sssd[pam]: Starting up
>  Jul 23 20:07:37 git01 sssd[pam]: Starting up
>  Jul 23 20:14:39 git01 sssd[pam]: Shutting down
>  Jul 23 20:14:39 git01 sssd[be[example.de]]: Starting up
>  Jul 23 20:14:39 git01 sssd[pam]: Starting up
>  Jul 23 20:18:44 git01 sssd[be[example.de]]: Shutting down
>  Jul 23 20:18:44 git01 sssd[pam]: Shutting down
>  Jul 23 20:18:44 git01 sssd[be[example.de]]: Starting up
>  Jul 23 20:18:44 git01 sssd[pam]: Starting up
>  Jul 24 04:05:28 git01 sssd[pam]: Shutting down
>  Jul 24 04:05:28 git01 sssd[pam]: Starting up
>  Jul 24 05:21:53 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 05:21:53 git01 sssd[be[example.de]]: Starting up
>  Jul 24 05:27:50 git01 sssd[pam]: Shutting down
>  Jul 24 05:27:50 git01 sssd[pam]: Starting up
>  Jul 24 05:27:50 git01 sssd[be[example.de]]: Starting up
>  Jul 24 05:27:53 git01 sssd[pam]: Starting up
>  Jul 24 05:30:31 git01 sssd[pam]: Shutting down
>  Jul 24 05:30:31 git01 sssd[pam]: Starting up
>  Jul 24 05:31:59 git01 sssd[nss]: Shutting down
>  Jul 24 05:31:59 git01 sssd[pam]: Shutting down
>  Jul 24 05:31:59 git01 sssd[nss]: Starting up
>  Jul 24 05:31:59 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 05:31:59 git01 sssd[pam]: Starting up
>  Jul 24 05:31:59 git01 sssd[be[example.de]]: Starting up
>  Jul 24 05:32:01 git01 sssd[pam]: Starting up
>  Jul 24 05:33:24 git01 sssd[pam]: Shutting down
>  Jul 24 05:33:24 git01 sssd[pam]: Starting up
>  Jul 24 05:33:24 git01 sssd[be[example.de]]: Starting up
>  Jul 24 06:01:38 git01 sssd[pam]: Shutting down
>  Jul 24 06:01:38 git01 sssd[be[example.de]]: Starting up
>  Jul 24 06:01:38 git01 sssd[pam]: Starting up
>  Jul 24 06:02:39 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 06:02:39 git01 sssd[be[example.de]]: Starting up
>  Jul 24 09:56:52 git01 sssd[pam]: Shutting down
>  Jul 24 09:56:52 git01 sssd[pam]: Starting up
>  Jul 24 10:02:42 git01 sssd[nss]: Shutting down
>  Jul 24 10:02:42 git01 sssd[pam]: Shutting down
>  Jul 24 10:02:42 git01 sssd[nss]: Starting up
>  Jul 24 10:02:42 git01 sssd[pam]: Starting up
>  Jul 24 10:02:42 git01 sssd[nss]: Shutting down
>  Jul 24 10:02:42 git01 sssd[pam]: Shutting down
>  Jul 24 10:02:42 git01 sssd[nss]: Starting up
>  Jul 24 10:02:42 git01 sssd[pam]: Starting up
>  Jul 24 10:02:42 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:02:42 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:06:14 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:06:14 git01 sssd[nss]: Shutting down
>  Jul 24 10:06:14 git01 sssd[nss]: Starting up
>  Jul 24 10:06:14 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:06:14 git01 sssd[nss]: Starting up
>  Jul 24 10:15:49 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:15:49 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:16:44 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:17:00 git01 sssd[pam]: Shutting down
>  Jul 24 10:17:00 git01 sssd[pam]: Starting up
>  Jul 24 10:17:00 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:17:00 git01 sssd[pam]: Starting up
>  Jul 24 10:18:48 git01 sssd[pam]: Shutting down
>  Jul 24 10:18:48 git01 sssd[pam]: Starting up
>  Jul 24 10:19:43 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:19:43 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:20:32 git01 sssd[pam]: Shutting down
>  Jul 24 10:20:32 git01 sssd[pam]: Starting up
>  Jul 24 10:21:12 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:21:15 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:27:11 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:27:11 git01 sssd[pam]: Shutting down
>  Jul 24 10:27:11 git01 sssd[pam]: Starting up
>  Jul 24 10:27:11 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:27:13 git01 sssd[pam]: Starting up
>  Jul 24 10:30:03 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:30:03 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:42:31 git01 sssd[pam]: Shutting down
>  Jul 24 10:42:31 git01 sssd[pam]: Starting up
>  Jul 24 10:42:31 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:54:45 git01 sssd[pam]: Shutting down
>  Jul 24 10:54:45 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 10:54:45 git01 sssd[pam]: Starting up
>  Jul 24 10:54:45 git01 sssd[be[example.de]]: Starting up
>  Jul 24 10:54:47 git01 sssd[pam]: Starting up
>  Jul 24 11:01:37 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:01:37 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:04:09 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:04:09 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:29:04 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:29:04 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:31:29 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:31:29 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:37:54 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:37:54 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:39:35 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:39:43 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:41:13 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:41:25 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:41:25 git01 sssd[nss]: Shutting down
>  Jul 24 11:41:25 git01 sssd[nss]: Starting up
>  Jul 24 11:41:54 git01 [sssd[krb5_child[33646]]]: Preauthentication failed
>  Jul 24 11:41:54 git01 [sssd[krb5_child[33646]]]: Preauthentication failed
>  Jul 24 11:42:27 git01 sssd[nss]: Shutting down
>  Jul 24 11:42:44 git01 sssd[nss]: Starting up
>  Jul 24 11:42:46 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:42:46 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:46:37 git01 sssd[pam]: Shutting down
>  Jul 24 11:46:37 git01 sssd[pam]: Starting up
>  Jul 24 11:48:33 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 11:48:36 git01 sssd[pam]: Shutting down
>  Jul 24 11:48:36 git01 sssd[pam]: Starting up
>  Jul 24 11:48:36 git01 sssd[be[example.de]]: Starting up
>  Jul 24 11:48:36 git01 sssd[pam]: Starting up
>  Jul 24 12:01:27 git01 sssd[pam]: Shutting down
>  Jul 24 12:01:27 git01 sssd[be[example.de]]: Shutting down
>  Jul 24 12:01:27 git01 sssd[pam]: Starting up
>  Jul 24 12:01:27 git01 sssd[be[example.de]]: Starting up
>  Jul 24 12:01:29 git01 sssd[pam]: Starting up
>  Jul 25 12:01:35 git01 sssd[pam]: Shutting down
>  Jul 25 12:01:35 git01 sssd[pam]: Starting up
>  Jul 25 18:03:58 git01 systemd[1]: sssd.service: Main process exited,
>  code=exited, status=1/FAILURE
>  Jul 25 18:03:58 git01 systemd[1]: sssd.service: Unit entered failed state.
>  Jul 25 18:03:58 git01 systemd[1]: sssd.service: Failed with result
>  'exit-code'.
>  Jul 25 18:03:58 git01 sssd[be[example.de]]: Shutting down
>  Jul 25 18:03:58 git01 sssd[nss]: Shutting down
>  Jul 25 18:03:58 git01 sssd[nss]: Starting up
>  Jul 25 18:03:58 git01 sssd[nss]: Starting up
>  Jul 25 18:03:58 git01 sssd[nss]: Starting up
>  Jul 25 18:03:58 git01 sssd: Exiting the SSSD. Could not restart critical
>  service [nss].
>  Jul 25 18:03:58 git01 sssd[be[example.de]]: Starting up
>  Jul 25 18:03:58 git01 sssd[be[example.de]]: Shutting down
>  Jul 25 18:03:58 git01 sssd[pam]: Shutting down
>  Jul 25 18:03:58 git01 sssd[pac]: Shutting down
>  Jul 25 18:15:32 git01 sssd: Starting up
>  Jul 25 18:15:32 git01 sssd[be[example.de]]: Starting up
>  Jul 25 18:15:32 git01 sssd[pam]: Starting up
>  Jul 25 18:15:32 git01 sssd[nss]: Starting up
>  Jul 25 18:15:32 git01 sssd[pac]: Starting up
>  Jul 26 10:25:16 git01 sssd[be[example.de]]: Shutting down
>  Jul 26 10:25:16 git01 sssd[pam]: Shutting down
>  Jul 26 10:25:16 git01 sssd[nss]: Shutting down
>  Jul 26 10:25:16 git01 sssd[pac]: Shutting down
>  Jul 26 10:25:16 git01 sssd: Starting up
>  Jul 26 10:25:16 git01 sssd[be[example.de]]: Starting up
>  Jul 26 10:25:17 git01 sssd[pam]: Starting up
>  Jul 26 10:25:17 git01 sssd[nss]: Starting up
>  Jul 26 10:25:17 git01 sssd[pac]: Starting up
>
>  I wonder why it doesn't keep on running?
>
>  Why did it fail completely at 18:03? This triggered an emergency
>  and had to be resolved manually.
>
>  The journal shows
>
>  Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Main
>  process exited, code=exited, status=1/FAILURE
>  Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Unit
>  entered failed state.
>  Jul 25 18:02:46 git01.ac.example.de systemd[1]: sssd.service: Failed with
>  result 'exit-code'.
>  Jul 25 18:01:52 git01.ac.example.de sssd[be[43718]: Shutting down
>  Jul 25 18:01:52 git01.ac.example.de sssd[33790]: Shutting down
>  Jul 25 18:01:56 git01.ac.example.de sssd[100999]: Starting up
>  Jul 25 18:01:58 git01.ac.example.de sssd[101071]: Starting up
>  Jul 25 18:02:02 git01.ac.example.de sssd[101163]: Starting up
>  Jul 25 18:02:45 git01.ac.example.de sssd[73]: Exiting the SSSD. Could not
>  restart critical service [nss].
>  Jul 25 18:02:45 git01.ac.example.de sssd[be[101036]: Starting up
>  Jul 25 18:02:45 git01.ac.example.de sssd[be[101036]: Shutting down
>  Jul 25 18:02:45 git01.ac.example.de sssd[75226]: Shutting down
>  Jul 25 18:02:45 git01.ac.example.de sssd[134]: Shutting down
>  Jul 25 18:03:00 git01.ac.example.de sshd[101666]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101683]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101682]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101684]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101690]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101689]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101691]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101688]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:03:00 git01.ac.example.de sshd[101696]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:04:00 git01.ac.example.de sshd[102433]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:04:00 git01.ac.example.de sshd[102432]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:04:00 git01.ac.example.de sshd[102431]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:04:00 git01.ac.example.de sshd[102437]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103008]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103007]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103009]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103032]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103031]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103030]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:05:00 git01.ac.example.de sshd[103036]: pam_sss(sshd:account):
>  Request to sssd failed. Connection refused
>  Jul 25 18:06:28 git01.ac.example.de sshd[103823]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:06:36 git01.ac.example.de sshd[103823]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:08:11 git01.ac.example.de sshd[105080]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:08:24 git01.ac.example.de sshd[105182]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:08:38 git01.ac.example.de sshd[105365]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:10:50 git01.ac.example.de sshd[106920]: pam_sss(sshd:auth):
>  Request to sssd failed. Connection refused
>  Jul 25 18:15:32 git01.ac.example.de sssd[110413]: Starting up
>  Jul 25 18:15:32 git01.ac.example.de sssd[be[110415]: Starting up
>  Jul 25 18:15:32 git01.ac.example.de sssd[110418]: Starting up
>  Jul 25 18:15:32 git01.ac.example.de sssd[110417]: Starting up
>  Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
>  Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
>  Jul 25 18:15:32 git01.ac.example.de sssd[110419]: Starting up
>  Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
>  Jul 25 18:15:32 git01.ac.example.de sssd_be[110415]: GSSAPI client step 2
>  Jul 25 18:30:42 git01.ac.example.de sssd_be[110415]: GSSAPI client step 1
>
>  sssd doesn't write critical errors into its log files by default :-(.
>  There is no information about this in /var/log/sssd. Of course I have
>  increased verbosity now.
>
>
>  This was sssd version 1.15.2 on Debian 9. I have upgraded the system
>  to sssd 1.16.1.
>
>
>  Every helpful comment is highly appreciated.
>  Harri
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>