audit system not working
I am on f26 x86_64, fully updated.
I wanted to find which process keeps my server very busy for about 10 minutes
every few days so decided to use the audit facility to log every launched program.
After some searching I found that I could do
$ sudo auditctl -a always,task
then later I can see what happened with
$ sudo ausearch -i -sc execve|less
and finally remove the rule with
$ sudo auditctl -d always,task
No records were reported by ausearch and no records were added to
/var/log/audit/audit.log.
I then noted that this log file is old, the last entry is from 17/Jan
$ sudo ls -l /var/log/audit/audit.log
-rw-------. 1 root root 6789409 Jan 16 14:59 /var/log/audit/audit.log
$ sudo tail -n 1 /var/log/audit/audit.log
type=DAEMON_END msg=audit(1516075173.204:8779): op=terminate auid=0 pid=1 subj=
res=success
I then checked another machine and it was similar.
However /var/log/messages regularly includes audit messages.
I saw that the audit packages were updated around that time the logging stopped:
$ sudo grep audit /var/log/dnf.log
2018-01-16T03:33:00Z DEBUG ---> Package audit.x86_64 2.8.1-1.fc26 will be upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit.x86_64 2.8.2-1.fc26 will be an upgrade
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs.x86_64 2.8.1-1.fc26 will be
upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs.x86_64 2.8.2-1.fc26 will be an
upgrade
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-python3.x86_64 2.8.1-1.fc26 will be
upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-python3.x86_64 2.8.2-1.fc26 will be
an upgrade
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-python.x86_64 2.8.1-1.fc26 will be
upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-python.x86_64 2.8.2-1.fc26 will be
an upgrade
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-devel.x86_64 2.8.1-1.fc26 will be
upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs-devel.x86_64 2.8.2-1.fc26 will be an
upgrade
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs.i686 2.8.1-1.fc26 will be upgraded
2018-01-16T03:33:00Z DEBUG ---> Package audit-libs.i686 2.8.2-1.fc26 will be an
upgrade
...
checking the service status I see:
$ systemctl status auditd
* auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; disabled; vendor preset:
enabled)
Active: inactive (dead)
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Was it disabled intentionally?
I thought starting the service will do the trick - but no. The system became non
responsive and after
a minute I could not even ping it. I switched to a text console and rebooted (CtlAltDel)
which took
some time but did eventually reboot.
Feb 13 11:16:02 e7 systemd-journald[521]: Journal stopped
Feb 13 22:16:50 e7 kernel: microcode: microcode updated early to revision 0x1c, date =
2015-02-26
...
The log file (audit.log) was full of repetitions of
type=SYSCALL msg=audit(1518480718.009:8330567): arch=c000003e syscall=232 success=yes
exit=1 a0=b a1=56069a5e0660 a2=40 a3=e95f items=0 ppid=1 pid=26480 auid=4294967295 uid=0
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295
comm="auditd" exe="/usr/sbin/auditd" key=(null)
type=PROCTITLE msg=audit(1518480718.009:8330567): proctitle="/sbin/auditd"
type=SYSCALL msg=audit(1518480718.009:8330568): arch=c000003e syscall=45 success=yes
exit=47 a0=3 a1=56069a5e3850 a2=231c a3=40 items=0 ppid=1 pid=26480 auid=4294967295 uid=0
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295
comm="auditd" exe="/usr/sbin/auditd" key=(null)
type=SOCKADDR msg=audit(1518480718.009:8330568): saddr=100000000000000000000000
type=SYSCALL msg=audit(1518480718.009:8330585): arch=c000003e syscall=20 success=no
exit=-11 a0=7 a1=7ffcb38e90c0 a2=2 a3=56069a5e3860 items=0 ppid=1 pid=26480
auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none)
ses=4294967295 comm="auditd" exe="/usr/sbin/auditd" key=(null)
type=PROCTITLE msg=audit(1518480718.009:8330585): proctitle="/sbin/auditd"
... many repeats of the last two lines...
The system logged many messages like:
kernel: kauditd_printk_skb: 31527 callbacks suppressed
systemd-journald[521]: Missed 1146 kernel messages
kernel: Out of memory: Kill process 1847 (/usr/sbin/httpd) score 1 or sacrifice child
The system was clearly in trouble.
I now wonder if the audit system was replaced by another facility which conflicts with
this service.
Maybe the installed packages are leftovers from an old upgrade?
I see a kernel audit thread, running since the last reboot.
$ ps aux|grep audit
root 78 0.3 0.0 0 0 ? S 11:27 0:21 [kauditd]
My questions are:
1) do I need to remove or install any audit packages?
2) how do I set up the audit system properly?
3) How do I log every started program if 'auditctl -a' is not correct?
Maybe 'auditctl -a' is correct, but I need to look at 'messages' and
*not* start the service?
TIA
--
Eyal Lebedinsky (fedora(a)eyal.emu.id.au)