is there a better solution than "killall -9 journalctl"?

[Adam Williamson]

On Mon, 2014-07-14 at 12:37 -0700, Bob Arendt wrote:
> >
> >    ok, i just rebooted and, within a minute, a journalctl was invoked
> > that almost immediately sucked up 99% CPU. so i checked its status via
> > PID and got:
> >
> > $ systemctl status 4230
> > ● abrtd.service - ABRT Automated Bug Reporting Tool
> >     Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
> >     Active: active (running) since Mon 2014-07-14 15:14:51 EDT; 8min ago
> >   Main PID: 801 (abrtd)
> >     CGroup: /system.slice/abrtd.service
> >             ├─ 801 /usr/sbin/abrtd -d -s
> >             ├─4016 abrt-server -s
> >             ├─4017 /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/ccpp-2014-07-14-15:20:51-2330
> >             ├─4070 abrt-server -s
> >             ├─4071 /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/ccpp-2014-07-14-15:21:23-4057
> >             ├─4197 /bin/sh -c if grep '^TracerPid:[[:space:]]*[123456789]' proc_pid_status >/dev/null 2>&1; then             # We see 'TracerPid: <nonzero...
> >             ├─4205 abrt-server -s
> >             ├─4206 /usr/libexec/abrt-handle-event -i -e post-create -- /var/tmp/abrt/ccpp-2014-07-14-15:22:32-4202
> >             ├─4219 /bin/sh -c if grep '^TracerPid:[[:space:]]*[123456789]' proc_pid_status >/dev/null 2>&1; then             # We see 'TracerPid: <nonzero...
> >             ├─4229 /bin/sh -c if grep '^TracerPid:[[:space:]]*[123456789]' proc_pid_status >/dev/null 2>&1; then             # We see 'TracerPid: <nonzero...
> >             ├─4230 journalctl _UID=0 -b
> >             ├─4231 grep -F -e packagekitd
> >             └─4232 tail -99

I don't think this part ^^^^^^^

> >
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/ccpp-2014-05-12-10:31:33-2547/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Unlocked '/var/tmp/abrt/oops-2014-04-06-05:46:42-25982-1/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/ccpp-2014-05-12-10:31:33-2547/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: '/var/tmp/abrt/ccpp-2014-05-12-10:31:33-2547' is not a problem directory
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/oops-2014-06-19-05:49:38-1919-0/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Unlocked '/var/tmp/abrt/oops-2014-04-06-05:46:42-25982-1/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/oops-2014-06-19-05:49:38-1919-0/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Unlocked '/var/tmp/abrt/oops-2014-04-06-05:46:42-25982-1/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/oops-2014-06-19-05:49:38-1919-0/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Unlocked '/var/tmp/abrt/oops-2014-04-06-05:46:42-25982-1/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: '/var/tmp/abrt/oops-2014-04-06-05:46:42-25982-1' is not a problem directory
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Unlocked '/var/tmp/abrt/oops-2014-06-20-06:21:16-848-0/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Missing file: time
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4205]: Unlocked '/var/tmp/abrt/oops-2014-06-19-05:49:38-1919-0/.lock' (no or corrupted 'time' file)
> > Jul 14 15:23:14 localhost.localdomain abrt-server[4070]: Missing file: time
> >
> > ... etc etc ...
> >
> >    thoughts?

Is related to this part ^^^^^^^^

at all.

> > rday
> >
> 
> Looks like https://bugzilla.redhat.com/show_bug.cgi?id=1048279
>  From Comment#2:
> The problem here is that some tmp-files-cleaner has cleared contents of your dump directories and 
> left the empty directories in the dump location. Removing of the empty directories from 
> '/var/tmp/abrt' should help in this case.
> 
> .. more details in the bug report.  Still flagged as open.

I believe this bug is the cause of the *second* chunk of stuff Robert
posted - the old crashes with 'missing files' - but not the cause of the
problem we're actually concerned about here, the journalctl stuff.

If you look at the systemctl status output, it's operating on a very
recent crash. The abrt service started at 15:14, which is presumably
when the system booted. There seem to be three abrt-handle-event
processes running, all operating on crashes with very recent time/dates:

/var/tmp/abrt/ccpp-2014-07-14-15:20:51-2330
/var/tmp/abrt/ccpp-2014-07-14-15:21:23-4057
/var/tmp/abrt/ccpp-2014-07-14-15:22:32-4202

it seems like three crashes happened in the space of two minutes. The
journalctl call is likely part of the crash handling process.

I guess what might be happening is something is crashing very frequently
on Robert's system, so frequently that abrt winds up calling journalctl
almost constantly. Can you tell from the contents of /var/tmp/abrt
or /var/spool/abrt , or just the gnome-abrt tool, what it is that's
crashing so much, Robert?

I might go poking through the abrt code in a minute to see why it's
calling journalctl. It does seem a somewhat inefficient approach.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net