On Thu, 2011-01-13 at 14:46 +0100, Jiri Moskovcak wrote:
On 01/13/2011 01:25 PM, Denys Vlasenko wrote:
On Fri, 2011-01-07 at 15:29 +0100, Jiri Moskovcak wrote:
= ABRT 2.0 Features (what we need) = == need to be done features == plugins free daemon -> add a generic log scanner
I would like to work on this one. Here's my "How to implement log scanning?" brainstorm.
Proposal (1): daemon watches log files, whenever one of them changes, it runs a program which parses the log file. It can give the offset in the log from where parsing should start. Table of files to watch and helpers to run should be somewhere in config file:
# Specialized log watcher programs /var/log/messages abrt-scan-oops -f $FNAME -o $OFF /var/log/Xorg.0.log abrt-scan-xserver -f $FNAME -o $OFF # generic log watcher program /some/other.log abrt-scan-log -f $FNAME -o $OFF -start-regex='ALERT' -end-regex='END' -maxlines=50 ...
I see one problem with this setup. We need to consider a case when watched logfile gets flooded (growing continuously and quickly). This means daemon will be starting the parser all the time, and will have hard time determining where exactly parser stopped parsing last time: say, /var/log/messages was 100M long. We started parser. Parser finished. We look at /var/log/messages and it's 110M long now. How can we be sure that parser read until the very end of 110th meg, that it did not hit EOF at, say, 109M? IOW: where to restart next parsing?
parsers can return some status message saying where they stopped
btw, what happens now if we parse /var/log/messages, find the oops,
but the oops is not fully written there - the line ---- end ----- is not there?
We drop it if it's too short. If it isn't, we submit truncated one.
Seems like this problem can be solved if we keep an open fd to the log file in question, and use that fd to read the file. This way, we always know where we stopped.
- smells like leaking fd, which selinux won't like...
Not if the fd isn't in daemon but in log watcher process.
This leads me to proposal (2): every log file is watched by a separate "log watching daemon". Basically, we just start:
# Specialized log watcher programs abrt-scan-oops -f /var/log/messages& abrt-scan-xserver -f /var/log/Xorg.0.log& # generic log watcher program abrt-scan-log -f /some/other.log -start-regex='ALERT' -end-regex='END' -maxlines=50& ...
in background, and every one of these watches and processes one log file.
- here we would also need some way of adding/removing the scanner which
is the same problem as adding/removing the log files to scan..
Unix already has this mechanism: "killall PROG" and "PROG ARGS &"
- one solution might be to turn all scanners into a separate services
(with their own init files) but now when the trend is to lower the number of services started on boot this doesn't sound as a good idea..
Yes. I decided to make them children of abrtd for now, to not have many more services.
Which proposal do you like more? Do you have other ideas or see other problems to watch for?
-don't have a strong opinion here, but I like the (1) a bit more, because there is only one long running process involved...
I decided to try a mixed approach: log watcher is a separate process, but it is started/stopped by abrtd:
$ abrt-dump-oops Usage: abrt-dump-oops [-vsrdow] FILE
Extract oops from syslog/dmesg file
-v, --verbose Be verbose -s Log to syslog -r Parse kernel's message buffer before parsing FILE -d Create ABRT dump for every oops found -o Print found oopses on standard output -w Do not exit, watch the file for new oopses
abrtd spawns "abrt-dump-oops -drw [-v[v[v]]] [-s] /var/log/messages".
By the way, it also replaces dumpoops.
Sending the patch in a minute.