[Maniphest] [Created] T19: Some users seeing HTTP 503 errors from the blockerbugs production instance

[tflink (Tim Flink)]

tflink created this task.
tflink assigned this task to mkrizek.
tflink added a subscriber: tflink.
tflink added a project: blockerbugs

TASK DESCRIPTION
  Earlier today, we got a report of the production app throwing 503 errors but the errors went away shortly after. At first glance, it appears that the user in question was logged in at the time and at approximately the same time, the following error showed up on blockerbugs02:

Nov 20 15:30:58 blockerbugs02 app          2013-11-20 15:30:58 ERROR   Exception on /admin/ [GET]#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1687, in wsgi_app#012    response = self.full_dispatch_request()#012  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1360, in full_dispatch_request#012    rv = self.handle_user_exception(e)#012  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1358, in full_dispatch_request#012    rv = self.dispatch_request()#012  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1344, in dispatch_request#012    return self.view_functions[rule.endpoint](**req.view_args)#012  File "/usr/lib/python2.6/site-packages/flask_admin/base.py", line 49, in inner#012    abort = self._handle_view(f.__name__, **kwargs)#012  File "/usr/lib/python2.6/site-packages/blockerbugs/controllers/admin/__init__.py", line 43, in _handle_view#012    return check_admin_rights()#012  File 
 "/usr/lib/python2.6/site-packages/blockerbugs/controllers/users.py", line 52, in check_admin_rights#012    return fas.login()#012  File "/usr/lib/python2.6/site-packages/flask_fas_openid.py", line 129, in login#012    if 'next' in args.values:#012NameError: global name 'args' is not defined

I suspect that something may be mishandled with login when the load balancer sends traffic to the node on which the user did not login initially, but this is just a suspicion - the issue still needs triage.

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek, tflink
Cc: qa-devel, tflink

[Maniphest] [Commented On] T19: Some users seeing HTTP 503 errors from the blockerbugs production instance

[tflink (Tim Flink)]

mkrizek added a comment.

I am pretty sure that the tb is caused by old version of python-fedora-flask on stg and prod that is missing this: https://github.com/fedora-infra/python-fedora/commit/dba632f599358ebf6399435...

As a workaround, we'd need to call fas.login(return_url=*insert_value_here*) instead of fas.login() (which is only at one place: check_admin_rights():blockerbugs/util/login.py).

This should only happen in admin interface though, so I am not sure if the tb is related to those 503 errors the user was seeing.

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek
Cc: qa-devel, tflink

[Maniphest] [Commented On] T19: Some users seeing HTTP 503 errors from the blockerbugs production instance

[tflink (Tim Flink)]

tflink added a comment.

We hotfixed a newer version of python-fedora-flask on to the production and staging hosts, so that should fix the tb that I pasted earlier.

I propose that we change this bug to be about those tbs and either file a new bug for the 503s or ask kparal to let us know if he sees it again.

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek, tflink
Cc: qa-devel, tflink

[Maniphest] [Triaged] T19: out of date python_flask_openid on production and staging instances is causing tracebacks

[tflink (Tim Flink)]

tflink changed the title from "Some users seeing HTTP 503 errors from the blockerbugs production instance" to "out of date python_flask_openid on production and staging instances is causing tracebacks".
tflink triaged this task as "High" priority.

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek, tflink
Cc: qa-devel, tflink

[Maniphest] [Changed CC] T19: out of date python_flask_openid on production and staging instances is causing tracebacks

[tflink (Tim Flink)]

kparal added a subscriber: kparal.
kparal added a comment.
> ask kparal to let us know if he sees it again

I will.
Btw, is it possible to easily quote a previous comment in Phab?

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek, kparal
Cc: qa-devel, tflink, kparal

[Maniphest] [Closed] T19: out of date python_flask_openid on production and staging instances is causing tracebacks

[tflink (Tim Flink)]

tflink closed this task as "Resolved".
tflink added a comment.

ask kparal to let us know if he sees it again

I will.

Thanks, hopefully it was fixed with the hotfixed python-fedora-flask. Closing the ticket since that hotfix was deployed.

Btw, is it possible to easily quote a previous comment in Phab?

Not that I'm aware of, no.

TASK DETAIL
  https://phab.qadevel.cloud.fedoraproject.org/T19

To: mkrizek, tflink
Cc: qa-devel, tflink, kparal