[PATCH] task #1104: handle condor log rotations in dbomatic

[Mo Morsi]

---
 src/dbomatic/dbomatic |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
index 1cd1229..3ab9a4f 100755
--- a/src/dbomatic/dbomatic
+++ b/src/dbomatic/dbomatic
@@ -248,6 +248,14 @@ class CondorEventLog < Nokogiri::XML::SAX::Document
 end
 
 def parse_log_file(log_file, parser)
+  # if the log file has been rotated out (condor does this internally)
+  if log_file.pos > File.size(CONDOR_EVENT_LOG_FILE)
+    # since the original file descriptor is pointing at the log
+    # just rotated out, we need to close and reopen
+    log_file.close
+    log_file = File.open(CONDOR_EVENT_LOG_FILE)
+  end
+
   while s = log_file.gets
     parser << s
   end
-- 
1.7.2.3

[Matthew Farrellee]

On 04/08/2011 12:31 PM, Mohammed Morsi wrote:
>  ---
>    src/dbomatic/dbomatic |    8 ++++++++
>    1 files changed, 8 insertions(+), 0 deletions(-)
>
>  diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
>  index 1cd1229..3ab9a4f 100755
>  --- a/src/dbomatic/dbomatic
>  +++ b/src/dbomatic/dbomatic
>  @@ -248,6 +248,14 @@ class CondorEventLog<  Nokogiri::XML::SAX::Document
>    end
>
>    def parse_log_file(log_file, parser)
>  +  # if the log file has been rotated out (condor does this internally)
>  +  if log_file.pos>  File.size(CONDOR_EVENT_LOG_FILE)
>  +    # since the original file descriptor is pointing at the log
>  +    # just rotated out, we need to close and reopen
>  +    log_file.close
>  +    log_file = File.open(CONDOR_EVENT_LOG_FILE)
>  +  end
>  +
>      while s = log_file.gets
>        parser<<  s
>      end 
Three things -

0) You may want to do something with ino or ctime for rotation 
detection. It will be far more robust.

1) Be sure to read the rest of the events in the current log file before 
opening the new one.

2) You'll want to also make sure that rotation happens infrequently 
enough (compared to your polling rate) that you do not miss a rotation.

Best,


matt

[Mo Morsi]

  On 04/10/2011 11:53 AM, Matthew Farrellee wrote:
>  On 04/08/2011 12:31 PM, Mohammed Morsi wrote:
> > ---
> >   src/dbomatic/dbomatic |    8 ++++++++
> >   1 files changed, 8 insertions(+), 0 deletions(-)
> >
> > diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
> > index 1cd1229..3ab9a4f 100755
> > --- a/src/dbomatic/dbomatic
> > +++ b/src/dbomatic/dbomatic
> > @@ -248,6 +248,14 @@ class CondorEventLog<  Nokogiri::XML::SAX::Document
> >   end
> >
> >   def parse_log_file(log_file, parser)
> > +  # if the log file has been rotated out (condor does this internally)
> > +  if log_file.pos>  File.size(CONDOR_EVENT_LOG_FILE)
> > +    # since the original file descriptor is pointing at the log
> > +    # just rotated out, we need to close and reopen
> > +    log_file.close
> > +    log_file = File.open(CONDOR_EVENT_LOG_FILE)
> > +  end
> > +
> >     while s = log_file.gets
> >       parser<<  s
> >     end
>
>  Three things -
>
>  0) You may want to do something with ino or ctime for rotation 
>  detection. It will be far more robust.
>
>  1) Be sure to read the rest of the events in the current log file 
>  before opening the new one.
>
>  2) You'll want to also make sure that rotation happens infrequently 
>  enough (compared to your polling rate) that you do not miss a rotation.
>
>  Best,
>
>
>  matt 
Ya sorry about the multiple revisions to the list, Matt Wagner and I 
were debugging till pretty late Friday night and I didn't have the means 
to reproduce the issue locally (working on setting that up now so that I 
can debug this on my own).

Inotify would work to address the three issues, eg whenever the file is 
modified it would be picked up so that we don't miss any of the events 
in the old log or the rotation itself, were it not for condor doing some 
funky things regarding the logs.

Won't go into all the details here but Condor manages its own log 
rotation on its own internally and how it does so doesn't play well with 
the inotify watchers (they seem to rename the original log file, but 
still keep the file handle to it after the rename, truncating it w/ 
another write before it starts getting filled up again)

I'm sure we can figure out the right handlers to catch all the events, 
am working on it now, its just taking a bit longer due to this 
unexpected complexity.

   -Mo

[Matthew Farrellee]

On 04/10/2011 02:49 PM, Mo Morsi wrote:
>  On 04/10/2011 11:53 AM, Matthew Farrellee wrote:
> > On 04/08/2011 12:31 PM, Mohammed Morsi wrote:
> >> ---
> >> src/dbomatic/dbomatic | 8 ++++++++
> >> 1 files changed, 8 insertions(+), 0 deletions(-)
> >>
> >> diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic
> >> index 1cd1229..3ab9a4f 100755
> >> --- a/src/dbomatic/dbomatic
> >> +++ b/src/dbomatic/dbomatic
> >> @@ -248,6 +248,14 @@ class CondorEventLog< Nokogiri::XML::SAX::Document
> >> end
> >>
> >> def parse_log_file(log_file, parser)
> >> + # if the log file has been rotated out (condor does this internally)
> >> + if log_file.pos> File.size(CONDOR_EVENT_LOG_FILE)
> >> + # since the original file descriptor is pointing at the log
> >> + # just rotated out, we need to close and reopen
> >> + log_file.close
> >> + log_file = File.open(CONDOR_EVENT_LOG_FILE)
> >> + end
> >> +
> >> while s = log_file.gets
> >> parser<< s
> >> end
> >
> > Three things -
> >
> > 0) You may want to do something with ino or ctime for rotation
> > detection. It will be far more robust.
> >
> > 1) Be sure to read the rest of the events in the current log file
> > before opening the new one.
> >
> > 2) You'll want to also make sure that rotation happens infrequently
> > enough (compared to your polling rate) that you do not miss a rotation.
> >
> > Best,
> >
> >
> > matt
>
>  Ya sorry about the multiple revisions to the list, Matt Wagner and I
>  were debugging till pretty late Friday night and I didn't have the means
>  to reproduce the issue locally (working on setting that up now so that I
>  can debug this on my own).
>
>  Inotify would work to address the three issues, eg whenever the file is
>  modified it would be picked up so that we don't miss any of the events
>  in the old log or the rotation itself, were it not for condor doing some
>  funky things regarding the logs.
>
>  Won't go into all the details here but Condor manages its own log
>  rotation on its own internally and how it does so doesn't play well with
>  the inotify watchers (they seem to rename the original log file, but
>  still keep the file handle to it after the rename, truncating it w/
>  another write before it starts getting filled up again)
>
>  I'm sure we can figure out the right handlers to catch all the events,
>  am working on it now, its just taking a bit longer due to this
>  unexpected complexity.
>
>  -Mo 
Condor uses file locking and rename for atomic rotation. How can it be 
changed to interact better with inotify?

Best,


matt

[Mo Morsi]

>  Condor uses file locking and rename for atomic rotation. How can it
> be
>  changed to interact better with inotify?
>  
>  Best,
>  
>  
>  matt 

I think we're all set, I updated inotify in dbomatic to watch the condor
log directory instead of the EventLog file itself so even if the log is
rotated out we can still catch that and parse the old and the new files.

Appreciate the assistance though,
  -Mo