Tracking a file in real-time

[FS]

Hello guys/gals,

I have a need to track changes to a file being made in real-time. So the
entries in this file are MAC addresses of clients that can get
added/deleted, sometimes multiple entries per second. What I want to do is
to see all the changes were from time A to time B.

I thought about tail -f, but that only lists the entries being added. Doing
diff would require copying the file and running diff against it every second
or so, and then again it wouldn't necessarily list all the changes, just the
one between the last changed file and the current one.

I hope I'm explaining this clearly, but basically the question is, is there
a utility in Linux/Unix world which can track any changes being made to a
file in real-time and list/track/put-in-a-file those changes being made?

Many thanks for any help you can provide.

Regards,
Faisal

[Curtis Doty]

8:50am FS said:

>  I have a need to track changes to a file being made in real-time. So
> the
>  entries in this file are MAC addresses of clients that can get
>  added/deleted, sometimes multiple entries per second. What I want to do is
>  to see all the changes were from time A to time B.
>
>  I thought about tail -f, but that only lists the entries being added. Doing
>  diff would require copying the file and running diff against it every second
>  or so, and then again it wouldn't necessarily list all the changes, just the
>  one between the last changed file and the current one.
>
>  I hope I'm explaining this clearly, but basically the question is, is there
>  a utility in Linux/Unix world which can track any changes being made to a
>  file in real-time and list/track/put-in-a-file those changes being made?
>  
Since linux 2.6.13, the in-kernel callback API for this is called 
inotify(7). RTFM to learn about its innards or how to use it in C.

If you just want a command-line tool, take a peek at inotifywait in the 
inotify-tools package.

../C

[Garry Williams]

On Thursday 11 October 2007 08:50:31 FS wrote:
>  I have a need to track changes to a file being made in real-time.
Curtis mentioned inotify(7).  Here's a small sample program using this
interface.

-- 
Garry T. Williams --- [phone]

[Tony Nelson]

At 7:55 AM -0400 10/13/07, Garry T. Williams wrote:
> On Thursday 11 October 2007 08:50:31 FS wrote:
> > I have a need to track changes to a file being made in real-time.
>
> Curtis mentioned inotify(7).  Here's a small sample program using this
> interface. 
AFAICT, inotify cannot solve the OP's problem.  He already knows the file.
He needs to know /where/ and /what/ changed in the file.  inotify can
optimize the actual solution so it won't need polling.
-- 
____________________________________________________________________
TonyN.:'                       <mailto:[e-mail]>
      '                              <http://www.georgeanelson.com/>

[Garry Williams]

On Saturday 13 October 2007 13:22:44 Tony Nelson wrote:
>  At 7:55 AM -0400 10/13/07, Garry T. Williams wrote:
>  >On Thursday 11 October 2007 08:50:31 FS wrote:
>  >> I have a need to track changes to a file being made in real-time.
>  >
>  >Curtis mentioned inotify(7).  Here's a small sample program using this
>  >interface.
>  
>  AFAICT, inotify cannot solve the OP's problem.  He already knows the file.
>  He needs to know /where/ and /what/ changed in the file.  inotify can
>  optimize the actual solution so it won't need polling. 
Yes.  Inotify handles the when (you're called back whenever a file is
modified).  The application can then determine the what.

I don't know about the where.  Not sure I understand what you mean by
that.

Of course, if we knew more of the detailed requirements and data
structures, we could design the "diff" part of the application too.  I
assume the OP is capable of handling that part.

Finally, it may not be possible to get called back on each atomic
change.  That is, a monitoring application might get a call that was
the result of multiple changes.  (I don't know the design of inotify
well enough to say for sure.  I haven't experimented with the
IN_MODIFY event to be able to even guess, either.)  In that case, if
each individual change must be tracked, then you're probably left with
the "changers" need to be modified to accomplish the task.  That might
be the best solution anyway.  Designing a robust daemon task is one
thing.  Guaranteeing that it never misses a change is another thing.

-- 
Garry T. Williams --- [phone]

[Tony Nelson]

At 2:19 PM -0400 10/13/07, Garry T. Williams wrote:
> On Saturday 13 October 2007 13:22:44 Tony Nelson wrote:
> > At 7:55 AM -0400 10/13/07, Garry T. Williams wrote:
> > >On Thursday 11 October 2007 08:50:31 FS wrote:
> > >> I have a need to track changes to a file being made in real-time.
> > >
> > >Curtis mentioned inotify(7).  Here's a small sample program using this
> > >interface.
> >
> > AFAICT, inotify cannot solve the OP's problem.  He already knows the file.
> > He needs to know /where/ and /what/ changed in the file.  inotify can
> > optimize the actual solution so it won't need polling.
>
> Yes.  Inotify handles the when (you're called back whenever a file is
> modified).  The application can then determine the what.
>
> I don't know about the where.  Not sure I understand what you mean by
> that.
>
> Of course, if we knew more of the detailed requirements and data
> structures, we could design the "diff" part of the application too.  I
> assume the OP is capable of handling that part. 
I think you assume wrong.  Here is the OP, with a fairly detailed
description of the problem to be solved:

At 8:50 AM -0400 10/11/07, FS wrote:
> Hello guys/gals,
>
> I have a need to track changes to a file being made in real-time. So the
> entries in this file are MAC addresses of clients that can get
> added/deleted, sometimes multiple entries per second. What I want to do is
> to see all the changes were from time A to time B.
>
> I thought about tail -f, but that only lists the entries being added.
> Doing diff would require copying the file and running diff against it
> every second or so, and then again it wouldn't necessarily list all the
> changes, just the one between the last changed file and the current one. 
Copy the file, compare the copy against the previous copy.  Note that the
file may have changed during the copy, so the copy may contain partial
changes.


> I hope I'm explaining this clearly, but basically the question is,
> is
> there a utility in Linux/Unix world which can track any changes being made
> to a file in real-time and list/track/put-in-a-file those changes being
> made?
>
> Many thanks for any help you can provide. 
Perhaps inotify can resolve the OP's objection to "running diff against it
every second or so", but it is an optimization to apply after the problem
is solved.

Probably diff'ing is not the correct approach at all, and it would be
helpful to know what is writing that file, and whether it can be persuaded
to write a log file.
-- 
____________________________________________________________________
TonyN.:'                       <mailto:[e-mail]>
      '                              <http://www.georgeanelson.com/>

[Les Mikesell]

Tony Nelson wrote:
B.
> >
> > I thought about tail -f, but that only lists the entries being added.
> > Doing diff would require copying the file and running diff against it
> > every second or so, and then again it wouldn't necessarily list all the
> > changes, just the one between the last changed file and the current one.
>  
>  Copy the file, compare the copy against the previous copy.  Note that the
>  file may have changed during the copy, so the copy may contain partial
>  changes.
>  
>  
> > I hope I'm explaining this clearly, but basically the question is, is
> > there a utility in Linux/Unix world which can track any changes being made
> > to a file in real-time and list/track/put-in-a-file those changes being
> > made?
> >
> > Many thanks for any help you can provide.
>  
>  Perhaps inotify can resolve the OP's objection to "running diff against it
>  every second or so", but it is an optimization to apply after the problem
>  is solved. 
Stat()'s are pretty cheap, so checking if the modification time has 
changed every second might be reasonable and only doing the

>  
>  Probably diff'ing is not the correct approach at all, and it would be
>  helpful to know what is writing that file, and whether it can be persuaded
>  to write a log file. 
As a fairly horrible hack you could strace the process and track the 
lseeks and writes, or use the -e write= to get a copy of the writes to 
the specified file descriptor.

-- 
   Les Mikesell
    lesmikesell(a)gmail.com