Error: setroubleshootd dead but subsys locked

[Steven Stromer]

Had a strange, and as yet unexplained, 'event' (I wasn't in front of the 
machine when things went weird) that took place while a system was left 
running a large rsync over ssh. On returning, a majority of the 
directories under /var vanished, and a number of services refused to 
start after a reboot, including auditd, nfsd, system message bus, hpiod, 
hpssd, mysql, syslogd, httpd, sm-client, and setroubleshootd.

In the cases of most of these services, there seemed to be problems 
either with orphaned /var/run/*.pid files, or with orphaned 
/var/lock/subsys/* lock files. Also, many services were reporting 
'subsys locked'. Deleting orphaned files, followed by relabeling the 
filesystem selinux permissions did the trick, with relabeling being the 
key to getting things going again. Debugging was made more challenging 
by the fact that I had no logs to refer to.

Now, almost all seems well, but I can't get setroubleshootd to start 
unless I select 'setroubleshootd_disable_trans'. Without this checked, 
setroubleshootd seems to start, but then fails:

[root@file1 subsys]# rm setroubleshootd
rm: remove regular empty file `setroubleshootd'? y
[root@file1 subsys]# service setroubleshoot status
setroubleshootd is stopped
[root@file1 subsys]# service setroubleshoot start
Starting setroubleshootd:                                  [  OK  ]
[root@file1 subsys]# service setroubleshoot status
setroubleshootd dead but subsys locked


Attempting to run setroubleshoot generates the error:

'attempt to open server connection failed: (2, 'No such file or directory')


Since someone might ask about permissions:

[root@file1 subsys]# ls -laRZ /var/log | grep setroubleshoot
drwxr-xr-x  root  root  system_u:object_r:setroubleshoot_var_log_t 
setroubleshoot
/var/log/setroubleshoot:
drwxr-xr-x  root root system_u:object_r:setroubleshoot_var_log_t .
-rw-r--r--  root root system_u:object_r:setroubleshoot_var_log_t 
setroubleshootd.log
-rw-r--r--  root root system_u:object_r:setroubleshoot_var_log_t 
setroubleshootd.log.1
-rw-r--r--  root root system_u:object_r:setroubleshoot_var_log_t 
setroubleshootd.log.2


Can anyone explain why setroubleshootd_disable_trans should need to be 
selected? Also, since this entire event seems to have close ties to 
selinux, would anyone have an idea what might have happened to this system?


Thanks for any ideas; it's been a long day...

Steven Stromer