Splunk Policy

[Robert Gabriel]

Greetz,

So I have cobbled together a basic policy for Splunk residing

in /opt/splunkdashboards/.

I followed Dan's blog to do the basics.

So I've added all the AVC messages to the splunkdashboards.te and restarted

Splunk with run_init...

Now, no more AVC messages but after a few seconds Splunk crashes.

Nothing in the debug log.

There is a crash log, seems to be a different thread each time crashing.

If I use the browser UI to work with Splunk, it does a few tasks then
something about

"Helper process is in an unknown state due to previous failure"

and then bang!

Seems to be thread permissions?

I'm lost, nothing in the log and no more AVC messages, where to from here?

I have tried so hard so far,  I don't want to be a coward now and hit "
setenforce 0".

I must learn how to do this.

I'm unsure as to mailing list etiquette, do I post all the policy files,
Splunk log etc.?

Please advise.

Any help appreciated, thank you.

[Schincke, Keith D. (JSC-IT)[DB Consulting Group, Inc.]]

Did you have any errors recorded in your splunkd.log file?

Keith Schincke CAP, LPIC-1, RHCA, RHCSS
Team Lead IT Security System Administration, ITAMS
Building 46, Room 110A
email to: [e-mail]
281-244-0183 Office           832-205-1534 Mobile
281-244-5708 Fax

ITAMS - Information Technology And Multimedia Services Contract
"One Team, One Vision >> Partnered For Innovative Solutions"

From: [e-mail] [mailto:[e-mail]] On Behalf Of Robert Gabriel
Sent: Wednesday, August 28, 2013 11:53 AM
To: [e-mail]
Subject: Splunk Policy

Greetz,
So I have cobbled together a basic policy for Splunk residing
in /opt/splunkdashboards/.
I followed Dan's blog to do the basics.
So I've added all the AVC messages to the splunkdashboards.te and restarted
Splunk with run_init...
Now, no more AVC messages but after a few seconds Splunk crashes.
Nothing in the debug log.
There is a crash log, seems to be a different thread each time crashing.
If I use the browser UI to work with Splunk, it does a few tasks then something about

"Helper process is in an unknown state due to previous failure"

and then bang!
Seems to be thread permissions?
I'm lost, nothing in the log and no more AVC messages, where to from here?
I have tried so hard so far,  I don't want to be a coward now and hit "setenforce 0".
I must learn how to do this.

I'm unsure as to mailing list etiquette, do I post all the policy files, Splunk log etc.?
Please advise.

Any help appreciated, thank you.

[Robert Gabriel]

On 28 August 2013 19:04, Schincke, Keith D. (JSC-IT)[DB Consulting Group,
Inc.]  wrote:
> Did you have any errors recorded in your splunkd.log file?****
>
> Keith Schincke CAP, LPIC-1, RHCA, RHCSS****
>
> Team Lead IT Security System Administration, ITAMS ****
>
> Building 46, Room 110A ****
>
> email to: [e-mail]****
>
> 281-244-0183 Office           832-205-1534 Mobile****
>
> 281-244-5708 Fax              ****
>
> ITAMS - Information Technology And Multimedia Services Contract ****
>
> "One Team, One Vision >> Partnered For Innovative Solutions"****
>
> *From:* [e-mail] [mailto:
> [e-mail]] *On Behalf Of *Robert Gabriel
> *Sent:* Wednesday, August 28, 2013 11:53 AM
> *To:* [e-mail]
> *Subject:* Splunk Policy****
>
> Greetz,****
>
> So I have cobbled together a basic policy for Splunk residing****
>
> in /opt/splunkdashboards/.****
>
> I followed Dan's blog to do the basics.****
>
> So I've added all the AVC messages to the splunkdashboards.te and restarted
>
> Splunk with run_init...****
>
> Now, no more AVC messages but after a few seconds Splunk crashes.****
>
> Nothing in the debug log.****
>
> There is a crash log, seems to be a different thread each time crashing.**
> **
>
> If I use the browser UI to work with Splunk, it does a few tasks then
> something about****
>
> "Helper process is in an unknown state due to previous failure"
>
> and then bang!****
>
> Seems to be thread permissions?****
>
> I'm lost, nothing in the log and no more AVC messages, where to from here?
>
> I have tried so hard so far,  I don't want to be a coward now and hit
> "setenforce 0".****
>
> I must learn how to do this.****
>
> I'm unsure as to mailing list etiquette, do I post all the policy files,
> Splunk log etc.?****
>
> Please advise.****
>
> Any help appreciated, thank you.****

I did look, no ERROR or WARN.

I'm quite familiar with Splunk, been working with it for the past 2.5
years, so I kind of have a feel for it's behaviour.

I've checked something now:

[root@pluto splunkdashboards]# aureport --start today --anomaly

Anomaly Report
=========================================
# date time type exe term host auid event
=========================================
1. 08/28/2013 18:02:01 ANOM_ABEND splunkd ? ? 500 822

/var/log/audit/audit.log:
type=ANOM_ABEND msg=audit(1377705721.554:822): auid=500 uid=501
gid=501 ses=1 subj=system_u:system_r:splunkdashboards_t:s0 pid=14464
comm="splunkd" sig=6

/opt/splunkdashboards/var/log/splunk/crash-2013-08-28-16:27:15.log:
[build 149561] 2013-08-28 16:27:15
Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 9075 running under UID 501.
 Crashing thread: DispatchReaper
 Registers:
    RIP:  [0x00002AD7447898A5] gsignal + 53 (/lib64/libc.so.6)
    RDI:  [0x0000000000002373]
    RSI:  [0x0000000000002380]
    RBP:  [0x00002AD749462278]
    RSP:  [0x00002AD7491FF188]
    RAX:  [0x0000000000000000]
    RBX:  [0x000000000196FC38]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0x0000000000000001]
    R9:  [0x206E61206E692073]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x00002AD74581E0C0]
    R13:  [0x00002AD7491FF3A0]
    R14:  [0x00002AD7491FF3E0]
    R15:  [0x00002AD74F8311E8]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]

OS: Linux
 Arch: x86-64

Backtrace:
  [0x00002AD7447898A5] gsignal + 53 (/lib64/libc.so.6)
  [0x00002AD74478B085] abort + 373 (/lib64/libc.so.6)
  [0x00000000012EB4B8] _ZN9__gnu_cxx27__verbose_terminate_handlerEv +
200 (splunkd)
  [0x00000000012EB186] _ZN10__cxxabiv111__terminateEPFvvE + 6 (splunkd)
  [0x00000000012EB1B3] ? (splunkd)
  [0x00000000012EB2B3] ? (splunkd)
  [0x0000000000D7294F] _ZN20ScopedHelperProcLockC1Ev + 271 (splunkd)
  [0x0000000000D763C8]
_ZN20ExternalProcessGroup12terminateAllERK20ConditionWaitTimeout + 56
(splunkd)
  [0x0000000000E9BF1C] _ZN15DispatchProcess9terminateEv + 156 (splunkd)
  [0x0000000000EB6359] _ZN15DispatchProcessD0Ev + 57 (splunkd)
  [0x0000000000EB79E6]
_ZN15DispatchManager24reapAllInactiveProcessesEv + 374 (splunkd)
  [0x0000000000EEB2C5] _ZN20BulletinBoardUpdater4tickEv + 261 (splunkd)
  [0x0000000000DA5553] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval
+ 227 (splunkd)
  [0x0000000000D3A318] _ZN9EventLoop3runEv + 216 (splunkd)
  [0x0000000000EE97B4] _ZN14DispatchReaper4mainEv + 2852 (splunkd)
  [0x0000000000DA2F32] _ZN6Thread8callMainEPv + 66 (splunkd)
  [0x00002AD742F72851] ? (/lib64/libpthread.so.0)
  [0x00002AD74483F90D] clone + 109 (/lib64/libc.so.6)
 Linux / pluto.gdf.gsoc.co.za / 2.6.32-358.11.1.el6.centos.plus.x86_64
/ #1 SMP Wed Jun 12 19:12:17 UTC 2013 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but
also could be old):
    2013-08-28 15:47:13.867 +0200 splunkd started (build 149561)
    terminate called after throwing an instance of 'ProcessRunnerException'
      what():  Helper process is in an unknown state due to previous failure
    2013-08-28 15:49:26.583 +0200 splunkd started (build 149561)
    2013-08-28 15:50:39.141 +0200 Interrupt signal received
    2013-08-28 15:50:50.566 +0200 splunkd started (build 149561)
    terminate called after throwing an instance of 'ProcessRunnerException'
      what():  Helper process is in an unknown state due to previous failure
    2013-08-28 15:51:43.309 +0200 splunkd started (build 149561)
    terminate called after throwing an instance of 'ProcessRunnerException'
      what():  Helper process is in an unknown state due to previous failure

/etc/redhat-release: CentOS release 6.4 (Final)
 glibc version: 2.12
 glibc release: stable
Threads running: 42
argv: [splunkd -h 192.168.122.2 -p 8089 restart]
terminating...

[Dominick Grift]

On Wed, 2013-08-28 at 18:53 +0200, Robert Gabriel wrote:
> Please advise.
>
> Any help appreciated, thank you.

There are various things you may have overlooked:

Some things may be silently denied, thus not showing up in the audit.log
by default

To expose these, follow this procedure:

semodule -DB
reproduce issue
look for avc, user_avc and selinux_err messages in audit.log, and
in /var/log/messages
semodule -B

Make sure you arent overlooking selinux messages. Sometimes SELinux logs
to /var/log/messages but most of the time to /var/log/audit/audit.log

But if you use ausearch to parse the audit.log then use "-m
avc,user_avc,selinux_err", so that it looks for all kinds of selinux
related messages  rather than only regular "avc denials"

When writing policy , one usually needs to do various rounds of testing
because not all issues may surface the time round of testing

Heres the procedure i usually follow ( in that order ):

1. test in permissive mode
2. test in permissive mode with semodule -DB
3. test in enforcing mode with semodule -DB
4. test in enforcing mode

Those are just some suggestions, but since i have little information
about your issues it is hard to determine if this will help

Some questions:

1. does it work in permissive mode?
2. does or do the processes run in the expected context(s)
3. can you enclose your source policy module for review?
> --
> selinux mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/selinux

[Robert Gabriel]

On 28 August 2013 19:16, Dominick Grift  wrote:
> On Wed, 2013-08-28 at 18:53 +0200, Robert Gabriel wrote:
> > Please advise.
> >
> > Any help appreciated, thank you.
>
> There are various things you may have overlooked:
>
> Some things may be silently denied, thus not showing up in the audit.log
> by default
>
> To expose these, follow this procedure:
>
> semodule -DB
> reproduce issue
> look for avc, user_avc and selinux_err messages in audit.log, and
> in /var/log/messages
> semodule -B
>
> Make sure you arent overlooking selinux messages. Sometimes SELinux logs
> to /var/log/messages but most of the time to /var/log/audit/audit.log
>
> But if you use ausearch to parse the audit.log then use "-m
> avc,user_avc,selinux_err", so that it looks for all kinds of selinux
> related messages  rather than only regular "avc denials"
>
> When writing policy , one usually needs to do various rounds of testing
> because not all issues may surface the time round of testing
>
> Heres the procedure i usually follow ( in that order ):
> test in permissive mode
> test in permissive mode with semodule -DB
> test in enforcing mode with semodule -DB
> test in enforcing mode
>
> Principles of Information Security
> Those are just some suggestions, but since i have little information
> about your issues it is hard to determine if this will help
>
> Some questions:
> does it work in permissive mode?
> does or do the processes run in the expected context(s)
> can you enclose your source policy module for review?
>
> Thank you!

I will look into the above tomorrow at the office, I wasn't aware of
looking at other messages.

1. Yes, it runs fine in permissive mode.

2. Yes, the processes are in the expected context.

3. Yes, I can. Pardon my ignorance, but you will then need *.fc, *.te,
*.if, *.sh files yes?

Thank you again, I can't wait to carry on getting to this tomorrow, I'll
advise ASAP.

[Robert Gabriel]

On 28 August 2013 19:16, Dominick Grift  wrote:
> On Wed, 2013-08-28 at 18:53 +0200, Robert Gabriel wrote:
> > Please advise.
> >
> > Any help appreciated, thank you.
>
> There are various things you may have overlooked:
>
> Some things may be silently denied, thus not showing up in the audit.log
> by default
>
> To expose these, follow this procedure:
>
> semodule -DB
> reproduce issue
> look for avc, user_avc and selinux_err messages in audit.log, and
> in /var/log/messages
> semodule -B
>
> Make sure you arent overlooking selinux messages. Sometimes SELinux logs
> to /var/log/messages but most of the time to /var/log/audit/audit.log
>
> But if you use ausearch to parse the audit.log then use "-m
> avc,user_avc,selinux_err", so that it looks for all kinds of selinux
> related messages  rather than only regular "avc denials"
>
> When writing policy , one usually needs to do various rounds of testing
> because not all issues may surface the time round of testing
>
> Heres the procedure i usually follow ( in that order ):
> test in permissive mode
> test in permissive mode with semodule -DB
> test in enforcing mode with semodule -DB
> test in enforcing mode

Dominick,

You are the man!

I'm not sure what happened, but as you explained, yes there were several
other messages in said logs.

I followed your methodology and saw another AVC denied message, added that
and saw other Splunk related, but not denies.

Several restarts and a reboot and Splunk is still up.

THANK YOU THANK YOU THANK YOU!