This patch tries to remove Linux-specific dependency with rb-inotify and substitute it with fssm gem which allows to run on multiple platforms.
I would really apprecite if whoever has dbomatic/dbomatic running to test it for me. Thank you! It will make my life easier. Appreciate your time guys!
From: Ladislav Martincik lmartinc@redhat.com
- This patch is simply removing 'rb-inotify' gem dependent on Linux and substituting it with fssm which is multiplatform. --- src/config/environment.rb | 2 +- src/dbomatic/dbomatic | 31 ++++++++++++++++--------------- 2 files changed, 17 insertions(+), 16 deletions(-)
diff --git a/src/config/environment.rb b/src/config/environment.rb index 423f224..be6f77c 100644 --- a/src/config/environment.rb +++ b/src/config/environment.rb @@ -53,7 +53,7 @@ Rails::Initializer.run do |config| config.gem "compass-960-plugin", :lib => "ninesixty" config.gem "simple-navigation" config.gem "typhoeus" - config.gem "rb-inotify" + config.gem "fssm"
config.active_record.observers = :instance_observer, :task_observer # Only load the plugins named here, in the order given. By default, all plugins diff --git a/src/dbomatic/dbomatic b/src/dbomatic/dbomatic index 8a3e968..36bea6a 100755 --- a/src/dbomatic/dbomatic +++ b/src/dbomatic/dbomatic @@ -22,7 +22,7 @@ $: << File.join(File.dirname(__FILE__), "../dutils") require 'rubygems' require 'dutils' require 'nokogiri' -require 'rb-inotify' +require 'fssm' require 'optparse'
help = false @@ -269,7 +269,7 @@ begin # Create one for parsing purposes. parser << "<events>"
- notifier = INotify::Notifier.new + notifier = FSSM::Monitor.new log_file = nil
if File.exists? CONDOR_EVENT_LOG_FILE @@ -284,24 +284,25 @@ begin logger.info "done" end
- # Setup inotify watch for condor event log - notifier.watch(CONDOR_EVENT_LOG_FILE, :modify){ |event| - parse_log_file log_file, parser - } + # Setup file monitor for condor event log + notifier.path condor_event_log_dir, File.basename(CONDOR_EVENT_LOG_FILE) do + update { parse_log_file log_file, parser } + end
# if log file doesn't exist wait until it does else - notifier.watch(condor_event_log_dir, :create){ |event| - if event.name == "EventLog" - log_file = File.open(CONDOR_EVENT_LOG_FILE) - parse_log_file log_file, parser - - # Setup inotify watch for condor event log - notifier.watch(CONDOR_EVENT_LOG_FILE, :modify){ |event| + notifier.path condor_event_log_dir do + create do |base, relative| + if relative.name == "EventLog" + log_file = File.open(CONDOR_EVENT_LOG_FILE) parse_log_file log_file, parser - } + + notifier.path condor_event_log_dir, File.basename(CONDOR_EVENT_LOG_FILE) do + update { parse_log_file log_file, parser } + end + end end - } + end end
while true
On 11/01/10 - 01:07:43PM, lmartinc@redhat.com wrote:
From: Ladislav Martincik lmartinc@redhat.com
- This patch is simply removing 'rb-inotify' gem dependent on Linux and
substituting it with fssm which is multiplatform.
I tested this, and it doesn't seem to work. At least, I am not getting updated events into the database even though condor is transitioning things.
In general, I'm happy to switch over to fssm for support beyond linux, but I will NACK this patch until we have a proper Fedora RPM in review for it. Please add fssm to the list of unpackaged dependencies at http://deltacloud.org/page/Packaging_list
Thanks,
On a related note, what is blocking us from doing this some other way than watching the log for change and parsing its contents?
Does Condor lack proper interprocess communication tools?
Thomas
On 11/01/2010 01:07 PM, lmartinc@redhat.com wrote:
This patch tries to remove Linux-specific dependency with rb-inotify and substitute it with fssm gem which allows to run on multiple platforms.
I would really apprecite if whoever has dbomatic/dbomatic running to test it for me. Thank you! It will make my life easier. Appreciate your time guys! _______________________________________________ deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
On 11/01/10 - 01:54:05PM, Tomas Sedovic wrote:
On a related note, what is blocking us from doing this some other way than watching the log for change and parsing its contents?
Does Condor lack proper interprocess communication tools?
It was the fastest way to get it working. There are some other things we can do with this (most notably, use QMF), but watching the file for changes is actually a pretty good way to do it, in my opinion.
Hi Chris,
just curious, why do you think that file observation is good solution?
-- Ladislav
On Nov 1, 2010, at 2:05 PM, Chris Lalancette wrote:
On 11/01/10 - 01:54:05PM, Tomas Sedovic wrote:
On a related note, what is blocking us from doing this some other way than watching the log for change and parsing its contents?
Does Condor lack proper interprocess communication tools?
It was the fastest way to get it working. There are some other things we can do with this (most notably, use QMF), but watching the file for changes is actually a pretty good way to do it, in my opinion.
-- Chris Lalancette _______________________________________________ deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
On 11/01/10 - 02:32:02PM, Ladislav Martincik wrote:
Hi Chris,
just curious, why do you think that file observation is good solution?
I guess the question is, why do you think it is not? If you were to do something like pipes, that would just be a file anyway, so you wouldn't really get much efficiency gains.
Now, I agree, we eventually want to switch to QMF; that will allow us to do it over the network, meaning we can have condor running on a different machine from the aggregator. But at the moment I don't really see a compelling need to move to anything but QMF.
On Mon, 2010-11-01 at 10:07 -0400, Chris Lalancette wrote:
On 11/01/10 - 02:32:02PM, Ladislav Martincik wrote:
Hi Chris,
just curious, why do you think that file observation is good solution?
I guess the question is, why do you think it is not? If you were to do something like pipes, that would just be a file anyway, so you wouldn't really get much efficiency gains.
Now, I agree, we eventually want to switch to QMF; that will allow us to do it over the network, meaning we can have condor running on a different machine from the aggregator. But at the moment I don't really see a compelling need to move to anything but QMF.
I agree, there's nothing wrong with the file observation method we are using now. The format is good (XML) and we can use eg inotify or other mechanism (kqueue on bsd) to be notified of new data.
AFAIK QMF is still not prime time for condor, though I should poke in on that front again now as it's been some time.
Ian
On Nov 1, 2010, at 7:52 PM, Ian Main wrote:
On Mon, 2010-11-01 at 10:07 -0400, Chris Lalancette wrote:
On 11/01/10 - 02:32:02PM, Ladislav Martincik wrote:
Hi Chris,
just curious, why do you think that file observation is good solution?
I guess the question is, why do you think it is not? If you were to do something like pipes, that would just be a file anyway, so you wouldn't really get much efficiency gains.
Ok. Let's assume that file storage is a good solution. How do u assure that: 1) When the file gets big it will not blow your memory consumption 2) What if somebody deletes the file by accident or change its content (backup, logrotate) 3) What if you miss some messages? Fault tolerant.
Now, I agree, we eventually want to switch to QMF; that will allow us to do it over the network, meaning we can have condor running on a different machine from the aggregator. But at the moment I don't really see a compelling need to move to anything but QMF.
I think the discussion should about Condor being the background job processor. IMHO choosing simple DelayedJob gem/plugin would make things much more simpler and faster to implement from beginning.
Having sufficient message queue would make things even much better.
I agree, there's nothing wrong with the file observation method we are using now. The format is good (XML) and we can use eg inotify or other mechanism (kqueue on bsd) to be notified of new data.
AFAIK QMF is still not prime time for condor, though I should poke in on that front again now as it's been some time.
Ian
My main goal for sub. of rb-inotify has been the fact that I do most of the dev. work on my MacBook and having to comment all the time rb-inotify to be able to run server is just painful.
-- Ladislav
On 11/02/10 - 10:27:11AM, Ladislav Martincik wrote:
On Nov 1, 2010, at 7:52 PM, Ian Main wrote:
On Mon, 2010-11-01 at 10:07 -0400, Chris Lalancette wrote:
On 11/01/10 - 02:32:02PM, Ladislav Martincik wrote:
Hi Chris,
just curious, why do you think that file observation is good solution?
I guess the question is, why do you think it is not? If you were to do something like pipes, that would just be a file anyway, so you wouldn't really get much efficiency gains.
Ok. Let's assume that file storage is a good solution. How do u assure that:
- When the file gets big it will not blow your memory consumption
Could be a problem, in theory. Disk storage is so unbelievably cheap, though, that I'm not inclined to worry too much about it. When we switch to QMF, this will go away.
- What if somebody deletes the file by accident or change its content (backup, logrotate)
We could handle this (though we don't today). Inotify can also tell you when your file goes away, so we could try to handle that intelligently. That being said, you have similar problems with almost anything; what happens if your network socket closes unexpectedly?
- What if you miss some messages? Fault tolerant.
We already handle this. We store the place in the file that we last parsed from, and on startup we look for messages we missed. Again, QMF will also handle this.
Now, I agree, we eventually want to switch to QMF; that will allow us to do it over the network, meaning we can have condor running on a different machine from the aggregator. But at the moment I don't really see a compelling need to move to anything but QMF.
I think the discussion should about Condor being the background job processor. IMHO choosing simple DelayedJob gem/plugin would make things much more simpler and faster to implement from beginning.
Having sufficient message queue would make things even much better.
I know you weren't here when we made this decision, but suffice it to say that removing condor and going with something else is absolutely not an option. Condor is known to scale well (to hundreds of thousands of jobs), and that is what we want to target for our scheduling.
I agree, there's nothing wrong with the file observation method we are using now. The format is good (XML) and we can use eg inotify or other mechanism (kqueue on bsd) to be notified of new data.
AFAIK QMF is still not prime time for condor, though I should poke in on that front again now as it's been some time.
Ian
My main goal for sub. of rb-inotify has been the fact that I do most of the dev. work on my MacBook and having to comment all the time rb-inotify to be able to run server is just painful.
Right. I'm fine with replacing rb-inotify with fssm, I just want to make sure we have the packages and it works before we do that.
deltacloud-devel@lists.fedorahosted.org