[PATCH aggregator] BZ #643150 - [Image Builder Service]: Dies too easily on errors

[Jason Guiditta]

https://bugzilla.redhat.com/show_bug.cgi?id=643150

For any attempted build that throws an error, set the status
of that build to failed, empty the current build list, and
start fresh without the problematic build in queue.
---
 src/image_builder_service/image_builder_service |   29 +++++++++++++++++-----
 1 files changed, 22 insertions(+), 7 deletions(-)

diff --git a/src/image_builder_service/image_builder_service
b/src/image_builder_service/image_builder_service
index 0ffb74d..db1e888 100755
--- a/src/image_builder_service/image_builder_service
+++ b/src/image_builder_service/image_builder_service
@@ -199,13 +199,28 @@ class ImageBuilderService
     loop do
       check_for_queued
       @activebuilds.delete_if do |ab|
-        cur_status = @console.check_status(ab[:build])
-        update_agg(ab, cur_status) unless cur_status.eql?(ab[:status])
-        @log.debug "========================================"
-        @log.debug "Status for target " + ab[:build].target + ": " +
cur_status
-        @log.debug "Builds in array: " + check_build_num.to_s
-        @log.debug "========================================"
-        Image::INACTIVE_STATES.include?(cur_status)
+        begin
+          cur_status = @console.check_status(ab[:build])
+          update_agg(ab, cur_status) unless cur_status.eql?(ab[:status])
+          @log.debug "========================================"
+          @log.debug "Status for target " + ab[:build].target + ": "
+ cur_status
+          @log.debug "Builds in array: " + check_build_num.to_s
+          @log.debug "========================================"
+          Image::INACTIVE_STATES.include?(cur_status)
+        rescue => e
+          @log.fatal "#{e.backtrace.shift}: #{e.message}"
+          e.backtrace.each do |step|
+            @log.fatal "\tfrom #{step}"
+          end
+          @log.info "Something very bad happened. Setting image (id: #{ab[:ar_id]})
to failed"
+          image = Image.find(:first, :conditions => { :target =>
ab[:build].target.to_s,
+                                                      :template_id => ab[:ar_id].to_i
})
+          unless image.nil?
+            image.status = "failed"
+            image.save!
+          end
+          @activebuilds = []
+        end
       end
       sleep @timeout
 
-- 
1.7.2.3

[Chris Lalancette]

On 10/21/10 - 10:49:46PM, Jason Guiditta wrote:
>  https://bugzilla.redhat.com/show_bug.cgi?id=643150
>  
>  For any attempted build that throws an error, set the status
>  of that build to failed, empty the current build list, and
>  start fresh without the problematic build in queue.
>  ---
>   src/image_builder_service/image_builder_service |   29 +++++++++++++++++-----
>   1 files changed, 22 insertions(+), 7 deletions(-) 
I am going to NACK this change for now.  While this is something of an
improvement, the UI validation that we have now should prevent most of the
problems that I've seen.  Hopefully we can rewrite this thing to handle this
more gracefully, but not at present.

-- 
Chris Lalancette