Interaction with web services is crashing ipa

Monday, 10 June 2019

Seeing a very odd issue.  When we make webservices calls to IPA sssd
crashes.  this started happening within the last few days after onboarding
new members (hosts, not people)  of the domain.  I'm wondering if there's
some kind of database corruption?  The ldap services are all OK.  Here's
the error logs from /var/log/messages:

Jun 10 16:24:51 freeipa1 abrt-hook-ccpp: Process 2823 (sssd_be) of user 0
killed by SIGABRT - dumping core
Jun 10 16:24:51 freeipa1 sssd[be[data.domain.com]]: Starting up
Jun 10 16:24:51 freeipa1 abrt-server: Duplicate: core backtrace
Jun 10 16:24:51 freeipa1 abrt-server: DUP_OF_DIR:
/var/spool/abrt/ccpp-2019-06-07-12:08:45-936
Jun 10 16:24:51 freeipa1 abrt-server: Deleting problem directory
ccpp-2019-06-10-16:24:51-2823 (dup of ccpp-2019-06-07-12:08:45-936)
Jun 10 16:24:51 freeipa1 dbus[874]: [system] Activating service
name='org.freedesktop.problems' (using servicehelper)
Jun 10 16:24:51 freeipa1 dbus[874]: [system] Successfully activated service
'org.freedesktop.problems'
Jun 10 16:24:51 freeipa1 abrt-server: Email address of sender was not
specified. Would you like to do so now? If not, 'user@localhost' is to be
used [y/N]
Jun 10 16:24:51 freeipa1 abrt-server: Email address of receiver was not
specified. Would you like to do so now? If not, 'root@localhost' is to be
used [y/N]
Jun 10 16:24:51 freeipa1 abrt-server: Sending an email...
Jun 10 16:24:51 freeipa1 abrt-server: Sending a notification email to:
root@localhost
Jun 10 16:24:51 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabftFudt
Jun 10 16:24:51 freeipa1 abrt-server: Email was sent to: root@localhost
Jun 10 16:24:51 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabgUy4NG
Jun 10 16:24:51 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabZAy7NU
Jun 10 16:24:52 freeipa1 abrt-hook-ccpp: Process 2845 (sssd_be) of user 0
killed by SIGABRT - ignoring (repeated crash)
Jun 10 16:24:54 freeipa1 sssd[be[data.domain.com]]: Starting up
Jun 10 16:24:54 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabiaqYNk
Jun 10 16:24:54 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabY4XRrH
Jun 10 16:24:55 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytab5Qenw4
Jun 10 16:24:57 freeipa1 abrt-hook-ccpp: Process 2871 (sssd_be) of user 0
killed by SIGABRT - ignoring (repeated crash)
Jun 10 16:25:01 freeipa1 sssd[be[data.domain.com]]: Starting up
Jun 10 16:25:01 freeipa1 systemd: Created slice User Slice of pcp.
Jun 10 16:25:01 freeipa1 systemd: Starting User Slice of pcp.
Jun 10 16:25:01 freeipa1 systemd: Started Session 9 of user pcp.
Jun 10 16:25:01 freeipa1 systemd: Starting Session 9 of user pcp.
Jun 10 16:25:01 freeipa1 systemd: Removed slice User Slice of pcp.
Jun 10 16:25:01 freeipa1 systemd: Stopping User Slice of pcp.
Jun 10 16:25:07 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabDhS9wO
Jun 10 16:25:07 freeipa1 sssd: Keytab successfully retrieved and stored in:
/var/lib/sss/keytabs/domain.com.keytabXv4vAK
Jun 10 16:25:11 freeipa1 abrt-hook-ccpp: Process 2888 (sssd_be) of user 0
killed by SIGABRT - dumping core
Jun 10 16:25:11 freeipa1 sssd: Exiting the SSSD. Could not restart critical
service [data.domain.com].
Jun 10 16:25:11 freeipa1 sssd[pac]: Shutting down
Jun 10 16:25:11 freeipa1 sssd[ssh]: Shutting down
Jun 10 16:25:11 freeipa1 sssd[pam]: Shutting down
Jun 10 16:25:11 freeipa1 sssd[ifp]: Shutting down
Jun 10 16:25:11 freeipa1 sssd: Attempted to unregister path (path[0] = org
path[1] = freedesktop) which isn't registered
Jun 10 16:25:11 freeipa1 sssd: Attempted to unregister path (path[0] = org
path[1] = freedesktop) which isn't registered
Jun 10 16:25:11 freeipa1 sssd: Attempted to unregister path (path[0] = org
path[1] = freedesktop) which isn't registered
Jun 10 16:25:11 freeipa1 sssd[nss]: Shutting down
Jun 10 16:25:11 freeipa1 sssd[sudo]: Shutting down
Jun 10 16:25:11 freeipa1 systemd: sssd.service: main process exited,
code=exited, status=1/FAILURE
Jun 10 16:25:11 freeipa1 systemd: Unit sssd.service entered failed state.
Jun 10 16:25:11 freeipa1 systemd: sssd.service failed.
Jun 10 16:25:11 freeipa1 abrt-server: Duplicate: core backtrace
Jun 10 16:25:11 freeipa1 abrt-server: DUP_OF_DIR:
/var/spool/abrt/ccpp-2019-06-07-12:08:45-936
Jun 10 16:25:11 freeipa1 abrt-server: Deleting problem directory
ccpp-2019-06-10-16:25:11-2888 (dup of ccpp-2019-06-07-12:08:45-936)
Jun 10 16:25:11 freeipa1 abrt-server: Email address of sender was not
specified. Would you like to do so now? If not, 'user@localhost' is to be
used [y/N]
Jun 10 16:25:11 freeipa1 abrt-server: Email address of receiver was not
specified. Would you like to do so now? If not, 'root@localhost' is to be
used [y/N]
Jun 10 16:25:11 freeipa1 abrt-server: Sending an email...
Jun 10 16:25:11 freeipa1 abrt-server: Sending a notification email to:
root@localhost
Jun 10 16:25:11 freeipa1 abrt-server: Email was sent to: root@localhost

Here's the logs from httpd:
[Mon Jun 10 16:36:16.050489 2019] [auth_gssapi:error] [pid 2012] [client
X.X.X.X:47040] NO AUTH DATA Client did not send any authentication headers
[Mon Jun 10 16:36:16.266300 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: group_find(u'',
sizelimit=u'0', pkey_only=u'true'): SUCCESS
[Mon Jun 10 16:36:16.268858 2019] [:warn] [pid 2787] [client X.X.X.X:47042]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:16.286936 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:17.570595 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'applications-openshift',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.758992 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch:
group_show(u'applications-openshift-keytabs', no_members=u'true'):
SUCCESS
[Mon Jun 10 16:36:21.762565 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'approvers-infrastructure',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.763886 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'editors',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.766866 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'infrastructure-admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.768170 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'ipausers',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.769203 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'trust admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:21.769403 2019] [:error] [pid 2009] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: batch(({u'params':
([u'admins'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'applications-openshift'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'applications-openshift-keytabs'], {u'no_members': u'true'}),
u'method':
u'group_show', u'id': 0}, {u'params':
([u'approvers-infrastructure'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'editors'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'infrastructure-admins'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'ipausers'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'trust admins'], {u'no_members': u'true'}), u'method':
u'group_show',
u'id': 0})): SUCCESS
[Mon Jun 10 16:36:21.775349 2019] [:warn] [pid 2787] [client X.X.X.X:47042]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:21.802952 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM:
idoverrideuser_show(u'Default Trust View', u&#39;user(a)domain.com&#39;,
rights=False, all=True): SUCCESS
[Mon Jun 10 16:36:21.869183 2019] [auth_gssapi:error] [pid 2685] [client
X.X.X.X:47054] NO AUTH DATA Client did not send any authentication headers
[Mon Jun 10 16:36:22.050937 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: group_find(u'',
sizelimit=u'0', pkey_only=u'true'): SUCCESS
[Mon Jun 10 16:36:22.053821 2019] [:warn] [pid 2014] [client X.X.X.X:47056]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:22.074007 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.334145 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'applications-openshift',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.555629 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch:
group_show(u'applications-openshift-keytabs', no_members=u'true'):
SUCCESS
[Mon Jun 10 16:36:36.575277 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'approvers-infrastructure',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.576704 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'editors',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.578262 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'infrastructure-admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.579398 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'ipausers',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.580488 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'trust admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.580653 2019] [:error] [pid 2009] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: batch(({u'params':
([u'admins'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'applications-openshift'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'applications-openshift-keytabs'], {u'no_members': u'true'}),
u'method':
u'group_show', u'id': 0}, {u'params':
([u'approvers-infrastructure'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'editors'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'infrastructure-admins'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'ipausers'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'trust admins'], {u'no_members': u'true'}), u'method':
u'group_show',
u'id': 0})): SUCCESS
[Mon Jun 10 16:36:36.584869 2019] [:warn] [pid 2014] [client X.X.X.X:47056]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:36.607421 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM:
idoverrideuser_show(u'Default Trust View', u&#39;user(a)domain.com&#39;,
rights=False, all=True): NotFound
[Mon Jun 10 16:36:36.702126 2019] [auth_gssapi:error] [pid 2012] [client
X.X.X.X:47068] NO AUTH DATA Client did not send any authentication headers
[Mon Jun 10 16:36:36.875245 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: group_find(u'',
sizelimit=u'0', pkey_only=u'true'): SUCCESS
[Mon Jun 10 16:36:36.877648 2019] [:warn] [pid 2013] [client X.X.X.X:47070]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:36.898421 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.900574 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'applications-openshift',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.902432 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch:
group_show(u'applications-openshift-keytabs', no_members=u'true'):
SUCCESS
[Mon Jun 10 16:36:36.903783 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'approvers-infrastructure',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.904848 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'editors',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.906110 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'infrastructure-admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.907088 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'ipausers',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.908196 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'trust admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:36.908404 2019] [:error] [pid 2009] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: batch(({u'params':
([u'admins'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'applications-openshift'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'applications-openshift-keytabs'], {u'no_members': u'true'}),
u'method':
u'group_show', u'id': 0}, {u'params':
([u'approvers-infrastructure'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'editors'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'infrastructure-admins'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'ipausers'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'trust admins'], {u'no_members': u'true'}), u'method':
u'group_show',
u'id': 0})): SUCCESS
[Mon Jun 10 16:36:36.915960 2019] [:warn] [pid 2013] [client X.X.X.X:47070]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:36.938184 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM:
idoverrideuser_show(u'Default Trust View', u&#39;user(a)domain.com&#39;,
rights=False, all=True): NotFound
[Mon Jun 10 16:36:36.998127 2019] [auth_gssapi:error] [pid 2685] [client
X.X.X.X:47074] NO AUTH DATA Client did not send any authentication headers
[Mon Jun 10 16:36:37.174673 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: group_find(u'',
sizelimit=u'0', pkey_only=u'true'): SUCCESS
[Mon Jun 10 16:36:37.180021 2019] [:warn] [pid 2010] [client X.X.X.X:47076]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:37.199364 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.201877 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'applications-openshift',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.204011 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch:
group_show(u'applications-openshift-keytabs', no_members=u'true'):
SUCCESS
[Mon Jun 10 16:36:37.205837 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'approvers-infrastructure',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.206975 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'editors',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.208494 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'infrastructure-admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.209613 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'ipausers',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.211233 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'trust admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.211435 2019] [:error] [pid 2009] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: batch(({u'params':
([u'admins'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'applications-openshift'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'applications-openshift-keytabs'], {u'no_members': u'true'}),
u'method':
u'group_show', u'id': 0}, {u'params':
([u'approvers-infrastructure'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'editors'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'infrastructure-admins'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'ipausers'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'trust admins'], {u'no_members': u'true'}), u'method':
u'group_show',
u'id': 0})): SUCCESS
[Mon Jun 10 16:36:37.214743 2019] [:warn] [pid 2010] [client X.X.X.X:47076]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:37.232248 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM:
idoverrideuser_show(u'Default Trust View', u&#39;user(a)domain.com&#39;,
rights=False, all=True): NotFound
[Mon Jun 10 16:36:37.400208 2019] [auth_gssapi:error] [pid 2012] [client
X.X.X.X:47078] NO AUTH DATA Client did not send any authentication headers
[Mon Jun 10 16:36:37.550208 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: group_find(u'',
sizelimit=u'0', pkey_only=u'true'): SUCCESS
[Mon Jun 10 16:36:37.557728 2019] [:warn] [pid 2011] [client X.X.X.X:47082]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:37.576635 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.579153 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'applications-openshift',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.581039 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch:
group_show(u'applications-openshift-keytabs', no_members=u'true'):
SUCCESS
[Mon Jun 10 16:36:37.582622 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'approvers-infrastructure',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.584955 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'editors',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.586773 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'infrastructure-admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.587765 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'ipausers',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.588728 2019] [:error] [pid 2009] ipa: INFO:
sa-unison-admin(a)DOMAIN.COM: batch: group_show(u'trust admins',
no_members=u'true'): SUCCESS
[Mon Jun 10 16:36:37.588908 2019] [:error] [pid 2009] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM: batch(({u'params':
([u'admins'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'applications-openshift'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'applications-openshift-keytabs'], {u'no_members': u'true'}),
u'method':
u'group_show', u'id': 0}, {u'params':
([u'approvers-infrastructure'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'editors'], {u'no_members': u'true'}), u'method':
u'group_show', u'id':
0}, {u'params': ([u'infrastructure-admins'], {u'no_members':
u'true'}),
u'method': u'group_show', u'id': 0}, {u'params':
([u'ipausers'],
{u'no_members': u'true'}), u'method': u'group_show',
u'id': 0}, {u'params':
([u'trust admins'], {u'no_members': u'true'}), u'method':
u'group_show',
u'id': 0})): SUCCESS
[Mon Jun 10 16:36:37.592774 2019] [:warn] [pid 2011] [client X.X.X.X:47082]
failed to set perms (3140) on file (/var/run/ipa/ccaches/
sa-unison-admin(a)DOMAIN.COM)!, referer: https://freeipa1.DOMAIN.COM/ipa/ui/
[Mon Jun 10 16:36:37.609236 2019] [:error] [pid 2008] ipa: INFO:
[jsonserver_session] sa-unison-admin(a)DOMAIN.COM:
idoverrideuser_show(u'Default Trust View', u&#39;user(a)domain.com&#39;,
rights=False, all=True): NotFound

Here's IPA's info:

red hat release - CentOS Linux release 7.5.1804 (Core)
packages:

python2-ipaserver-4.5.4-10.el7.centos.noarch
sssd-ipa-1.16.0-19.el7.x86_64
python2-ipalib-4.5.4-10.el7.centos.noarch
ipa-common-4.5.4-10.el7.centos.noarch
ipa-client-4.5.4-10.el7.centos.x86_64
python-libipa_hbac-1.16.0-19.el7.x86_64
ipa-server-4.5.4-10.el7.centos.x86_64
ipa-server-dns-4.5.4-10.el7.centos.noarch
python2-ipaclient-4.5.4-10.el7.centos.noarch
ipa-server-trust-ad-4.5.4-10.el7.centos.x86_64
libipa_hbac-1.16.0-19.el7.x86_64
ipa-client-common-4.5.4-10.el7.centos.noarch
ipa-server-common-4.5.4-10.el7.centos.noarch

This has taken out access to our okd cluster so any help would be greatly
appreciated on how to debug this issue.

Thanks
Marc

2024

2023

2022

2021

2020

2019

2018

2017