[PATCH aggregator] initial dbomatic implementation

[Mo Morsi]

  This patch contains the rough first pass of dbomatic's implementation.
  There are various things TBD such as event error handling and making sure
  all the correct events are collected with all their metadata but what
  is present should be a rough estimate as to the parsing strategy.

  To get condor to yield the neccessary info for dbomatic to parse, add the
  following to /var/lib/condor/condor_config.local

    EVENT_LOG=$(LOG)/EventLog
    EVENT_LOG_USE_XML=True
   
EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,GlobalJobId,Cmd,JobStartDate,JobCurrentStartDate,JobFinishedHookDone

  Also be sure to set CONDOR_HOST appropriately (I merely set it to 'localhost' in
my case).
  A condor restart is required (eg sudo service condor restart) after these changes.
---
 src/app/models/instance_event.rb                   |   30 +++++++
 .../20100810221250_create_instance_events.rb       |   34 ++++++++
 src/dbomatic/dbomatic.rb                           |   90 ++++++++++++++++++++
 3 files changed, 154 insertions(+), 0 deletions(-)
 create mode 100644 src/app/models/instance_event.rb
 create mode 100644 src/db/migrate/20100810221250_create_instance_events.rb
 create mode 100644 src/dbomatic/dbomatic.rb

diff --git a/src/app/models/instance_event.rb b/src/app/models/instance_event.rb
new file mode 100644
index 0000000..cc1eaa0
--- /dev/null
+++ b/src/app/models/instance_event.rb
@@ -0,0 +1,30 @@
+# Copyright (C) 2010 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.  A copy of the GNU General Public License is
+# also available at http://www.gnu.org/copyleft/gpl.html.
+
+# Filters added to this controller apply to all controllers in the application.
+# Likewise, all the methods added will be available for all controllers.
+
+class InstanceEvent < ActiveRecord::Base
+  belongs_to :instance
+
+  validates_presence_of :instance_id
+  validates_presence_of :event_type
+
+  #validates_inclusion_of :event_type,
+  #   :in => TYPES
+
+end
diff --git a/src/db/migrate/20100810221250_create_instance_events.rb
b/src/db/migrate/20100810221250_create_instance_events.rb
new file mode 100644
index 0000000..22e5191
--- /dev/null
+++ b/src/db/migrate/20100810221250_create_instance_events.rb
@@ -0,0 +1,34 @@
+# Copyright (C) 2010 Red Hat, Inc.
+# Written by Mohammed Morsi <mmorsi(a)redhat.com&gt;
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.  A copy of the GNU General Public License is
+# also available at http://www.gnu.org/copyleft/gpl.html.
+
+class CreateInstanceEvents < ActiveRecord::Migration
+  def self.up
+    create_table :instance_events do |t|
+      t.integer    :instance_id, :null => false
+      t.string     :event_type,  :null => false
+      t.datetime   :event_time
+      t.string     :status
+      t.string     :message
+      t.timestamps
+    end
+  end
+
+  def self.down
+    drop_table :instance_events
+  end
+end
diff --git a/src/dbomatic/dbomatic.rb b/src/dbomatic/dbomatic.rb
new file mode 100644
index 0000000..8b80c81
--- /dev/null
+++ b/src/dbomatic/dbomatic.rb
@@ -0,0 +1,90 @@
+# Copyright (C) 2010 Red Hat, Inc.
+# Written by Mohammed Morsi <mmorsi(a)redhat.com&gt;
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+# MA  02110-1301, USA.  A copy of the GNU General Public License is
+# also available at http://www.gnu.org/copyleft/gpl.html.
+
+$: << File.join(File.dirname(__FILE__), "../dutils")
+require 'dutils'
+require 'nokogiri'
+
+# Handle the event log's xml
+class CondorEventLog < Nokogiri::XML::SAX::Document
+  attr_accessor :tag, :event_type, :event_cmd, :event_time
+
+  # Store the name of the event log attribute we're looking at
+  def start_element(element, attributes)
+    @tag = attributes[1] if element == "a"
+  end
+
+  # Store the value of the event log attribute we're looking at
+  def characters(string)
+    unless string.strip == ""
+      if @tag == "MyType"
+        @event_type = string
+      elsif @tag == "Cmd"
+        @event_cmd = string
+      elsif @tag == "EventTime"
+        @event_time = string
+      end
+    end
+  end
+
+  # Create a new entry for events which we have all the neccessary data for
+  def end_element(element)
+    if element == "c" && !(a)event_cmd.nil?
+      inst = Instance.find(:first, :conditions => ['condor_job_id = ?',
@event_cmd])
+      #puts "Instance event #{inst.name} #{@event_type} #{@event_time}"
+      InstanceEvent.create! :instance => inst,
+                            :event_type => @event_type,
+                            :event_time => @event_time
+      @tag = @event_type = @event_cmd = @event_time = nil
+    end
+  end
+end
+parser = Nokogiri::XML::SAX::PushParser.new(CondorEventLog.new)
+
+# XXX bit of a hack, condor event log doesn't seem to have a top level element
+# enclosing everything else in the doc (as standards conforming xml must).
+# Create one for parsing purposes.
+parser << "<events>"
+
+# last time the event log was modified
+event_log_timestamp = nil
+
+# last position we've read in the log
+event_log_position = 0
+
+# set true to terminate dbomatic
+terminate = false
+until terminate
+  log_file = File.open("/var/log/condor/EventLog")
+
+  # Condor seems to open / close the event log for every
+  # entry. Simply poll for new data in the log
+  unless log_file.mtime == event_log_timestamp
+    event_log_timestamp = log_file.mtime
+    log_file.pos = event_log_position
+    while c = log_file.getc
+      parser << c.chr
+    end
+    event_log_position = log_file.pos
+  end
+
+  sleep 1
+end
+
+parser << "</events>"
+parser.finish
-- 
1.7.2

[Chris Lalancette]

On 08/11/10 - 09:26:43AM, Mohammed Morsi wrote:
>    This patch contains the rough first pass of dbomatic's
> implementation.
>    There are various things TBD such as event error handling and making sure
>    all the correct events are collected with all their metadata but what
>    is present should be a rough estimate as to the parsing strategy.
>  
>    To get condor to yield the neccessary info for dbomatic to parse, add the
>    following to /var/lib/condor/condor_config.local
>  
>      EVENT_LOG=$(LOG)/EventLog
>      EVENT_LOG_USE_XML=True
>     
> EVENT_LOG_JOB_AD_INFORMATION_ATTRS=Owner,GlobalJobId,Cmd,JobStartDate,JobCurrentStartDate,JobFinishedHookDone
>  
>    Also be sure to set CONDOR_HOST appropriately (I merely set it to 'localhost'
> in my case). 
This change to CONDOR_HOST shouldn't be necessary with the latest
condor_config.local I have up on my webpage.

<snip>

>  diff --git a/src/dbomatic/dbomatic.rb b/src/dbomatic/dbomatic.rb
>  new file mode 100644
>  index 0000000..8b80c81
>  --- /dev/null
>  +++ b/src/dbomatic/dbomatic.rb
>  @@ -0,0 +1,90 @@
>  +# Copyright (C) 2010 Red Hat, Inc.
>  +# Written by Mohammed Morsi <mmorsi(a)redhat.com&gt;
>  +#
>  +# This program is free software; you can redistribute it and/or modify
>  +# it under the terms of the GNU General Public License as published by
>  +# the Free Software Foundation; version 2 of the License.
>  +#
>  +# This program is distributed in the hope that it will be useful,
>  +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>  +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>  +# GNU General Public License for more details.
>  +#
>  +# You should have received a copy of the GNU General Public License
>  +# along with this program; if not, write to the Free Software
>  +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
>  +# MA  02110-1301, USA.  A copy of the GNU General Public License is
>  +# also available at http://www.gnu.org/copyleft/gpl.html.
>  +
>  +$: << File.join(File.dirname(__FILE__), "../dutils")
>  +require 'dutils'
>  +require 'nokogiri'
>  +
>  +# Handle the event log's xml
>  +class CondorEventLog < Nokogiri::XML::SAX::Document
>  +  attr_accessor :tag, :event_type, :event_cmd, :event_time
>  +
>  +  # Store the name of the event log attribute we're looking at
>  +  def start_element(element, attributes)
>  +    @tag = attributes[1] if element == "a"
>  +  end
>  +
>  +  # Store the value of the event log attribute we're looking at
>  +  def characters(string)
>  +    unless string.strip == ""
>  +      if @tag == "MyType"
>  +        @event_type = string
>  +      elsif @tag == "Cmd"
>  +        @event_cmd = string
>  +      elsif @tag == "EventTime"
>  +        @event_time = string
>  +      end
>  +    end
>  +  end
>  +
>  +  # Create a new entry for events which we have all the neccessary data for
>  +  def end_element(element)
>  +    if element == "c" && !(a)event_cmd.nil?
>  +      inst = Instance.find(:first, :conditions => ['condor_job_id = ?',
> @event_cmd])
>  +      #puts "Instance event #{inst.name} #{@event_type} #{@event_time}"
>  +      InstanceEvent.create! :instance => inst,
>  +                            :event_type => @event_type,
>  +                            :event_time => @event_time
>  +      @tag = @event_type = @event_cmd = @event_time = nil 
We may want to add additional fields in the future, but this is a good start.

>  +    end
>  +  end
>  +end
>  +parser = Nokogiri::XML::SAX::PushParser.new(CondorEventLog.new)
>  +
>  +# XXX bit of a hack, condor event log doesn't seem to have a top level element
>  +# enclosing everything else in the doc (as standards conforming xml must).
>  +# Create one for parsing purposes.
>  +parser << "<events>"
>  +
>  +# last time the event log was modified
>  +event_log_timestamp = nil
>  +
>  +# last position we've read in the log
>  +event_log_position = 0 
This is a problem for dbomatic restarts.  That is, if you get a few events,
and then restart dbomatic, you'll go back to the beginning of the event log
and re-add those same events to the table.  We are going to have to track this
on persistent storage somehow.

>  +
>  +# set true to terminate dbomatic
>  +terminate = false
>  +until terminate
>  +  log_file = File.open("/var/log/condor/EventLog")
>  +
>  +  # Condor seems to open / close the event log for every
>  +  # entry. Simply poll for new data in the log
>  +  unless log_file.mtime == event_log_timestamp
>  +    event_log_timestamp = log_file.mtime
>  +    log_file.pos = event_log_position
>  +    while c = log_file.getc
>  +      parser << c.chr
>  +    end 
This is probably better done by getting line-by-line (instead of character
by character), no?

>  +    event_log_position = log_file.pos
>  +  end
>  +
>  +  sleep 1 
The sleep is fine for now, but I would prefer if we used something like
inotify to get notified of changes to the file.  That's a future optimization,
though.

>  +end
>  +
>  +parser << "</events>"
>  +parser.finish 
So, this is a really good start.  It looks like it's the framework we need
for doing additional work here.  The biggest questions that come to mind have
to do more with the types of events that condor is generating, and whether
we can properly calculate our QoS data from it.  In that vein, could you
schedule some actual guests on, say, EC2, and then see if the information we
are gathering can answer:

1)  How long it took between us submitting the request, and the request to
be started?  (i.e. gone through matchmaking and the deltacloud GAHP).
2)  How long it took between the request being started and the guest going to
running? (this should be the time the backend cloud took to get the guest up
and running).

And other metrics you can think of.

-- 
Chris Lalancette

[Mo Morsi]

  Sorry for the delay, was busy with the deltacloud appliance.

On 08/13/2010 03:28 PM, Chris Lalancette wrote:
>  <snip>
> > +    end
> > +  end
> > +end
> > +parser = Nokogiri::XML::SAX::PushParser.new(CondorEventLog.new)
> > +
> > +# XXX bit of a hack, condor event log doesn't seem to have a top level element
> > +# enclosing everything else in the doc (as standards conforming xml must).
> > +# Create one for parsing purposes.
> > +parser<<  "<events>"
> > +
> > +# last time the event log was modified
> > +event_log_timestamp = nil
> > +
> > +# last position we've read in the log
> > +event_log_position = 0
>  This is a problem for dbomatic restarts.  That is, if you get a few events,
>  and then restart dbomatic, you'll go back to the beginning of the event log
>  and re-add those same events to the table.  We are going to have to track this
>  on persistent storage somehow. 
Added file to persistantly track this across
dbomatic restarts. Its is 
currently located under /var/run/dbomatic/.

> > +
> > +# set true to terminate dbomatic
> > +terminate = false
> > +until terminate
> > +  log_file = File.open("/var/log/condor/EventLog")
> > +
> > +  # Condor seems to open / close the event log for every
> > +  # entry. Simply poll for new data in the log
> > +  unless log_file.mtime == event_log_timestamp
> > +    event_log_timestamp = log_file.mtime
> > +    log_file.pos = event_log_position
> > +    while c = log_file.getc
> > +      parser<<  c.chr
> > +    end
>  This is probably better done by getting line-by-line (instead of character
>  by character), no?
>  
Changed.

> > +    event_log_position = log_file.pos
> > +  end
> > +
> > +  sleep 1
>  The sleep is fine for now, but I would prefer if we used something like
>  inotify to get notified of changes to the file.  That's a future optimization,
>  though. 
Ah yes, I was looking for inotify but forgot the name. I updated the 
process to track file changes that way.

> > +end
> > +
> > +parser<<  "</events>"
> > +parser.finish
>  So, this is a really good start.  It looks like it's the framework we need
>  for doing additional work here.  The biggest questions that come to mind have
>  to do more with the types of events that condor is generating, and whether
>  we can properly calculate our QoS data from it.  In that vein, could you
>  schedule some actual guests on, say, EC2, and then see if the information we
>  are gathering can answer:
>
>  1)  How long it took between us submitting the request, and the request to
>  be started?  (i.e. gone through matchmaking and the deltacloud GAHP).
>  2)  How long it took between the request being started and the guest going to
>  running? (this should be the time the backend cloud took to get the guest up
>  and running).
>
>  And other metrics you can think of.
>  
I'm still having trouble getting condor fully integrated with the 
aggregator and core components, which is limiting the amount of log info 
I'm collecting locally. We tried debugging this last week but didn't 
fully finish, perhaps sometime when you and Ian are next available we 
can resolve that issue.

Also as far as the dbomatic process itself, how much parametrization 
does it need? The only thing that I can see as being variable is the log 
file location, but even then that's not a major use case. Also would it 
be smart to assume this is going to be started via the init/services 
interface and should be daemonized?

   -Mo

[Chris Lalancette]

On 08/24/10 - 12:28:17AM, Mohammed Morsi wrote:
>  >So, this is a really good start.  It looks like it's the
> framework we need
>  >for doing additional work here.  The biggest questions that come to mind have
>  >to do more with the types of events that condor is generating, and whether
>  >we can properly calculate our QoS data from it.  In that vein, could you
>  >schedule some actual guests on, say, EC2, and then see if the information we
>  >are gathering can answer:
>  >
>  >1)  How long it took between us submitting the request, and the request to
>  >be started?  (i.e. gone through matchmaking and the deltacloud GAHP).
>  >2)  How long it took between the request being started and the guest going to
>  >running? (this should be the time the backend cloud took to get the guest up
>  >and running).
>  >
>  >And other metrics you can think of.
>  >
>  I'm still having trouble getting condor fully integrated with the
>  aggregator and core components, which is limiting the amount of log
>  info I'm collecting locally. We tried debugging this last week but
>  didn't fully finish, perhaps sometime when you and Ian are next
>  available we can resolve that issue. 
Yes, let's take a look at that this week and see if we can get something
more concrete started for you.  We need to answer the above questions to see
how much (if any) modifications we need to make to condor to get the
information we need.

>  
>  Also as far as the dbomatic process itself, how much parametrization
>  does it need? The only thing that I can see as being variable is the
>  log file location, but even then that's not a major use case. Also
>  would it be smart to assume this is going to be started via the
>  init/services interface and should be daemonized? 
As far as parameterization is concerned, I wouldn't worry too much about this
now.  As you said, there isn't really a whole lot to parameterize at the
moment and I think we can do this in the future as needed.

I think we should assume that this will be daemonized on the startup of
the appliance, so we should probably also write an init script for it and
get the whole thing packaged up in the RPM.

-- 
Chris Lalancette