FBR: Increase the openshift route timeout for greenwave

[Clement Verna]

Hi all,

After doing some investigation on bodhi requests to greenwave I found
that the requests are hitting a default 30 seconds timeout from the
openshift route
(https://stackoverflow.com/questions/47812807/openshift-service-proxy-timeout).

I would like to increase this timeout to 330 seconds, greenwave will
timeout after 300 seconds.

For this the following command needs to be run :

oc project greenwave
oc annotate route greenwave-web --overwrite
haproxy.router.openshift.io/timeout=330s
oc annotate route web-pretty --overwrite
haproxy.router.openshift.io/timeout=330s

I ll be looking at how we can add this into ansible.

+1s ?

[Pierre-Yves Chibon]

On Fri, Mar 15, 2019 at 12:11:45PM +0100, Clement Verna wrote:
> Hi all,
>
> After doing some investigation on bodhi requests to greenwave I found
> that the requests are hitting a default 30 seconds timeout from the
> openshift route
> (https://stackoverflow.com/questions/47812807/openshift-service-proxy-timeout).
>
> I would like to increase this timeout to 330 seconds, greenwave will
> timeout after 300 seconds.
>
> For this the following command needs to be run :
>
> oc project greenwave
> oc annotate route greenwave-web --overwrite
> haproxy.router.openshift.io/timeout=330s
> oc annotate route web-pretty --overwrite
> haproxy.router.openshift.io/timeout=330s
>
> I ll be looking at how we can add this into ansible.
>
> +1s ?

+1 for me

Thanks,
Pierre

[Stephen John Smoogen]

Is there a reason why it takes greenwave 5 minutes to come back with an answer?

On Fri, 15 Mar 2019 at 07:21, Pierre-Yves Chibon  wrote:
> On Fri, Mar 15, 2019 at 12:11:45PM +0100, Clement Verna wrote:
> > Hi all,
> >
> > After doing some investigation on bodhi requests to greenwave I found
> > that the requests are hitting a default 30 seconds timeout from the
> > openshift route
> > (https://stackoverflow.com/questions/47812807/openshift-service-proxy-timeout).
> >
> > I would like to increase this timeout to 330 seconds, greenwave will
> > timeout after 300 seconds.
> >
> > For this the following command needs to be run :
> >
> > oc project greenwave
> > oc annotate route greenwave-web --overwrite
> > haproxy.router.openshift.io/timeout=330s
> > oc annotate route web-pretty --overwrite
> > haproxy.router.openshift.io/timeout=330s
> >
> > I ll be looking at how we can add this into ansible.
> >
> > +1s ?
>
> +1 for me
>
> Thanks,
> Pierre
> _______________________________________________
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Stephen John Smoogen]

On Fri, 15 Mar 2019 at 07:40, Stephen John Smoogen  wrote:
> Is there a reason why it takes greenwave 5 minutes to come back with an answer?

Better question. Is there a reason why it takes more than 30 seconds
for greenwave to come back with an answer?
> On Fri, 15 Mar 2019 at 07:21, Pierre-Yves Chibon  wrote:
> > On Fri, Mar 15, 2019 at 12:11:45PM +0100, Clement Verna wrote:
> > > Hi all,
> > >
> > > After doing some investigation on bodhi requests to greenwave I found
> > > that the requests are hitting a default 30 seconds timeout from the
> > > openshift route
> > > (https://stackoverflow.com/questions/47812807/openshift-service-proxy-timeout).
> > >
> > > I would like to increase this timeout to 330 seconds, greenwave will
> > > timeout after 300 seconds.
> > >
> > > For this the following command needs to be run :
> > >
> > > oc project greenwave
> > > oc annotate route greenwave-web --overwrite
> > > haproxy.router.openshift.io/timeout=330s
> > > oc annotate route web-pretty --overwrite
> > > haproxy.router.openshift.io/timeout=330s
> > >
> > > I ll be looking at how we can add this into ansible.
> > >
> > > +1s ?
> >
> > +1 for me
> >
> > Thanks,
> > Pierre
> > _______________________________________________
> > infrastructure mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> --
> Stephen J Smoogen.

[Pierre-Yves Chibon]

On Fri, Mar 15, 2019 at 07:41:35AM -0400, Stephen John Smoogen wrote:
> On Fri, 15 Mar 2019 at 07:40, Stephen John Smoogen  wrote:
> > Is there a reason why it takes greenwave 5 minutes to come back with an answer?
>
> Better question. Is there a reason why it takes more than 30 seconds
> for greenwave to come back with an answer?

For updates containing a large number of builds (the threshold being undefined
at this time), it takes a while to greenwave to retrieve all the test results
from resultsdb.
This is why some requests may take a while.

Pierre

[Stephen John Smoogen]

On Fri, 15 Mar 2019 at 07:46, Pierre-Yves Chibon  wrote:
> On Fri, Mar 15, 2019 at 07:41:35AM -0400, Stephen John Smoogen wrote:
> > On Fri, 15 Mar 2019 at 07:40, Stephen John Smoogen  wrote:
> > > Is there a reason why it takes greenwave 5 minutes to come back with an answer?
> >
> > Better question. Is there a reason why it takes more than 30 seconds
> > for greenwave to come back with an answer?
>
> For updates containing a large number of builds (the threshold being undefined
> at this time), it takes a while to greenwave to retrieve all the test results
> from resultsdb.
> This is why some requests may take a while.

OK in that case +1 as it is a resource contention outside of what can
be fixed quickly.
> Pierre
> _______________________________________________
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...

[Randy Barlow]

On Fri, 2019-03-15 at 12:11 +0100, Clement Verna wrote:
> I would like to increase this timeout to 330 seconds, greenwave will
> timeout after 300 seconds.

I don't think we should do this. Two problems immediately come to mind.
Bodhi still gets timeout errors from Greenwave, because Bodhi will not
wait forever for a response. Now instead of 500 error e-mails from
Greenwave, I get just as many timeout error e-mails. Secondly, there
are currently 1,717 updates in the testing status in Bodhi. This means
that Bodhi's cron job will make that many queries each time it runs.

We do want to limit how much time Greenwave can take to respond,
otherwise we will just pile up cron jobs. This change effectively made
Bodhi's cron job slower, but did not lower the error rate.