#671 - BZ 691562 fix

[Matt Wagner]

Hi all,

I spent a bit of time tracking down the remnants of #671 / BZ 651562, in
which Apache throws proxy timeout errors because the backend does not
respond in time as a result of API calls timing out.

Most of these errors were fixed a bit ago in deltacloud-client, but it
seems that the versions yum and gem know about don't have this fix, so
the issue was seemingly not fixed. I spent a bit getting deltacloud
0.3.0 running, and was unable to reproduce any errors with Apache
timeouts after attempting to simulate unreachable providers and other
cases where a timeout may occur. The timeouts in deltacloud-client are
slightly shorter than Apache's timeout value.

I'm attaching a patch that cleans up error handling in Conductor
slightly. There were a few spots (e.g., adding a new ProviderAccount
when the provider was unreachable) where the timeout exception was
reaching the user unhandled. This exception now triggers as a
flash[:error] message when raised, in lieu of a hard error.

You will need the 0.3.0 version of deltacloud to test this. (Some older
versions may work, but best not to chance it.) You can build this from
their source, although I suspect some of you already the 0.3.0 RC
installed.

-- Matt

[PATCH] BZ 651562 - Graceful handling of timeout errors

[Matt Wagner]

deltacloud-client 0.3.0 will raise timeouts when connections are
excessively slow; this makes Conductor a bit smarter about catching
this exception and displaying a flash message to the user.
---
 src/app/controllers/application_controller.rb |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/src/app/controllers/application_controller.rb
b/src/app/controllers/application_controller.rb
index 6bda765..7d82f26 100644
--- a/src/app/controllers/application_controller.rb
+++ b/src/app/controllers/application_controller.rb
@@ -36,6 +36,7 @@ class ApplicationController < ActionController::Base
   rescue_from ActionError, :with => :handle_action_error
   rescue_from PartialSuccessError, :with => :handle_partial_success_error
   rescue_from ActiveRecord::RecordNotFound, :with =>
:handle_active_record_not_found_error
+  rescue_from RestClient::RequestTimeout, :with => :handle_request_timeout_error
 
   helper_method :check_privilege
 
@@ -85,6 +86,12 @@ class ApplicationController < ActionController::Base
     end
   end
 
+  # A Timeout is not necessarily fatal; return to request.referrer rather than an error
page
+  def handle_request_timeout_error(error)
+    flash[:error] = "The Provider timed out. Please try again."
+    redirect_to request.referrer
+  end
+
   def html_error_page(title, msg)
     if request.xhr?
       render :template => 'layouts/popup-error', :layout =>
'popup',
-- 
1.7.4.2

[wes hayutin]

woot!!!

On Tue, 2011-04-12 at 11:58 -0400, Matt Wagner wrote:
>  Hi all,
>  
>  I spent a bit of time tracking down the remnants of #671 / BZ 651562, in
>  which Apache throws proxy timeout errors because the backend does not
>  respond in time as a result of API calls timing out.
>  
>  Most of these errors were fixed a bit ago in deltacloud-client, but it
>  seems that the versions yum and gem know about don't have this fix, so
>  the issue was seemingly not fixed. I spent a bit getting deltacloud
>  0.3.0 running, and was unable to reproduce any errors with Apache
>  timeouts after attempting to simulate unreachable providers and other
>  cases where a timeout may occur. The timeouts in deltacloud-client are
>  slightly shorter than Apache's timeout value.
>  
>  I'm attaching a patch that cleans up error handling in Conductor
>  slightly. There were a few spots (e.g., adding a new ProviderAccount
>  when the provider was unreachable) where the timeout exception was
>  reaching the user unhandled. This exception now triggers as a
>  flash[:error] message when raised, in lieu of a hard error.
>  
>  You will need the 0.3.0 version of deltacloud to test this. (Some older
>  versions may work, but best not to chance it.) You can build this from
>  their source, although I suspect some of you already the 0.3.0 RC
>  installed.
>  
>  -- Matt
>  _______________________________________________
>  aeolus-devel mailing list
>  aeolus-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/aeolus-devel

[Hugh O. Brock]

On Thu, Jun 30, 2011 at 10:30:20AM -0400, wes hayutin wrote:
>  woot!!!
>  
>  On Tue, 2011-04-12 at 11:58 -0400, Matt Wagner wrote:
>  > Hi all,
>  > 
>  > I spent a bit of time tracking down the remnants of #671 / BZ 651562, in
>  > which Apache throws proxy timeout errors because the backend does not
>  > respond in time as a result of API calls timing out.
>  > 
>  > Most of these errors were fixed a bit ago in deltacloud-client, but it
>  > seems that the versions yum and gem know about don't have this fix, so
>  > the issue was seemingly not fixed. I spent a bit getting deltacloud
>  > 0.3.0 running, and was unable to reproduce any errors with Apache
>  > timeouts after attempting to simulate unreachable providers and other
>  > cases where a timeout may occur. The timeouts in deltacloud-client are
>  > slightly shorter than Apache's timeout value.
>  > 
>  > I'm attaching a patch that cleans up error handling in Conductor
>  > slightly. There were a few spots (e.g., adding a new ProviderAccount
>  > when the provider was unreachable) where the timeout exception was
>  > reaching the user unhandled. This exception now triggers as a
>  > flash[:error] message when raised, in lieu of a hard error.
>  > 
>  > You will need the 0.3.0 version of deltacloud to test this. (Some older
>  > versions may work, but best not to chance it.) You can build this from
>  > their source, although I suspect some of you already the 0.3.0 RC
>  > installed.
>  >  
Matt, are these correct versions now installed by configure?

Thanks,
--H

-- 
== Hugh Brock, hbrock(a)redhat.com                                   ==
== Engineering Manager, Cloud BU                                   ==
== Aeolus Project: Manage virtual infrastructure across clouds.    ==
== http://aeolusproject.org                                        ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey

[Matt Wagner]

On Fri, Jul 01, 2011 at 10:04:16AM -0400, Hugh Brock wrote:
>  On Thu, Jun 30, 2011 at 10:30:20AM -0400, wes hayutin wrote:
>  > woot!!!
>  > 
>  > On Tue, 2011-04-12 at 11:58 -0400, Matt Wagner wrote:
>  > > Hi all,
>  > > 
>  > > I spent a bit of time tracking down the remnants of #671 / BZ 651562, in
>  > > which Apache throws proxy timeout errors because the backend does not
>  > > respond in time as a result of API calls timing out.
>  > > 
>  > > Most of these errors were fixed a bit ago in deltacloud-client, but it
>  > > seems that the versions yum and gem know about don't have this fix, so
>  > > the issue was seemingly not fixed. I spent a bit getting deltacloud
>  > > 0.3.0 running, and was unable to reproduce any errors with Apache
>  > > timeouts after attempting to simulate unreachable providers and other
>  > > cases where a timeout may occur. The timeouts in deltacloud-client are
>  > > slightly shorter than Apache's timeout value.
>  > > 
>  > > I'm attaching a patch that cleans up error handling in Conductor
>  > > slightly. There were a few spots (e.g., adding a new ProviderAccount
>  > > when the provider was unreachable) where the timeout exception was
>  > > reaching the user unhandled. This exception now triggers as a
>  > > flash[:error] message when raised, in lieu of a hard error.
>  > > 
>  > > You will need the 0.3.0 version of deltacloud to test this. (Some older
>  > > versions may work, but best not to chance it.) You can build this from
>  > > their source, although I suspect some of you already the 0.3.0 RC
>  > > installed.
>  > > 
>  
>  Matt, are these correct versions now installed by configure? 
On a fresh RHEL 6.1 install of aeolus-all, I get:
 deltacloud-core.noarch = 0.3.9999-1308927004.el6
 rubygem-deltacloud-client.noarch = 0.1.0-2.el6_0

Fedora seems to have rubygem-deltacloud-client.noarch = 0.3.1-1.fc14, so
I think we just need to get the client updated for RHEL.

-- Matt

[Hugh O. Brock]

On Fri, Jul 01, 2011 at 10:18:42AM -0400, Matt Wagner wrote:
>  On Fri, Jul 01, 2011 at 10:04:16AM -0400, Hugh Brock wrote:
>  > On Thu, Jun 30, 2011 at 10:30:20AM -0400, wes hayutin wrote:
>  > > woot!!!
>  > > 
>  > > On Tue, 2011-04-12 at 11:58 -0400, Matt Wagner wrote:
>  > > > Hi all,
>  > > > 
>  > > > I spent a bit of time tracking down the remnants of #671 / BZ 651562, in
>  > > > which Apache throws proxy timeout errors because the backend does not
>  > > > respond in time as a result of API calls timing out.
>  > > > 
>  > > > Most of these errors were fixed a bit ago in deltacloud-client, but it
>  > > > seems that the versions yum and gem know about don't have this fix,
> so
>  > > > the issue was seemingly not fixed. I spent a bit getting deltacloud
>  > > > 0.3.0 running, and was unable to reproduce any errors with Apache
>  > > > timeouts after attempting to simulate unreachable providers and other
>  > > > cases where a timeout may occur. The timeouts in deltacloud-client are
>  > > > slightly shorter than Apache's timeout value.
>  > > > 
>  > > > I'm attaching a patch that cleans up error handling in Conductor
>  > > > slightly. There were a few spots (e.g., adding a new ProviderAccount
>  > > > when the provider was unreachable) where the timeout exception was
>  > > > reaching the user unhandled. This exception now triggers as a
>  > > > flash[:error] message when raised, in lieu of a hard error.
>  > > > 
>  > > > You will need the 0.3.0 version of deltacloud to test this. (Some older
>  > > > versions may work, but best not to chance it.) You can build this from
>  > > > their source, although I suspect some of you already the 0.3.0 RC
>  > > > installed.
>  > > > 
>  > 
>  > Matt, are these correct versions now installed by configure?
>  
>  On a fresh RHEL 6.1 install of aeolus-all, I get:
>   deltacloud-core.noarch = 0.3.9999-1308927004.el6
>   rubygem-deltacloud-client.noarch = 0.1.0-2.el6_0
>  
>  Fedora seems to have rubygem-deltacloud-client.noarch = 0.3.1-1.fc14, so
>  I think we just need to get the client updated for RHEL.
>   
OK, let's do that ASAP. Thanks!

--Hugh


-- 
== Hugh Brock, hbrock(a)redhat.com                                   ==
== Engineering Manager, Cloud BU                                   ==
== Aeolus Project: Manage virtual infrastructure across clouds.    ==
== http://aeolusproject.org                                        ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey

[Mo Morsi]

>  Matt, are these correct versions now installed by configure?
>  
>  Thanks,
>  --H
>   
Configure doesn't actually install anything anymore, it just verifies
that the components are present. We can easily re-enable the automatic
installation if components are missing if that is required.

  -Mo

[Hugh O. Brock]

On Fri, Jul 01, 2011 at 11:36:09AM -0400, Mo Morsi wrote:
>  
>  > Matt, are these correct versions now installed by configure?
>  > 
>  > Thanks,
>  > --H
>  > 
>  
>  Configure doesn't actually install anything anymore, it just verifies
>  that the components are present. We can easily re-enable the automatic
>  installation if components are missing if that is required. 
Oh, sorry, I guess I should have just said "installed". I don't
particularly care what installs them, as long as Wes is going to
automatically get the correct versions via some appropriate mechanism.

--H

-- 
== Hugh Brock, hbrock(a)redhat.com                                   ==
== Engineering Manager, Cloud BU                                   ==
== Aeolus Project: Manage virtual infrastructure across clouds.    ==
== http://aeolusproject.org                                        ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey

[Chris Lalancette]

On 07/01/11 - 11:36:09AM, Mo Morsi wrote:
>  
>  > Matt, are these correct versions now installed by configure?
>  > 
>  > Thanks,
>  > --H
>  > 
>  
>  Configure doesn't actually install anything anymore, it just verifies
>  that the components are present. We can easily re-enable the automatic
>  installation if components are missing if that is required. 
Please no :).

-- 
Chris Lalancette