Freeze Break Request: update koji and oz_timeout on buildvm-s390x-02

[Kevin Fenzi]

Greetings.

image composes using oz take a long time on our s390x image builder,
(due to them being in another datacenter). They take so long, they timeout.

See: https://pagure.io/koji/pull-request/837 from sinny and
https://pagure.io/koji/pull-request/841 which makes this value
configurable in kojid.conf

I would like to:

1. Update koji on buildvm-s390x-02 to the version I just built in
rawhide/f28 that has a backport of the patch that makes this value
configurable.

2. Apply the following ansible patch to set the timeout only on the
buildvm-s390x-02 builder:

diff --git a/roles/koji_builder/templates/kojid.conf
b/roles/koji_builder/templates/kojid.conf
index 832d80c..9f10d9c 100644
--- a/roles/koji_builder/templates/kojid.conf
+++ b/roles/koji_builder/templates/kojid.conf
@@ -20,7 +20,12 @@ maxjobs=25
 keepalive=False

rpmbuild_timeout=172800
-
+{% if ansible_hostname.startswith('buildvm-s390x-02') %}
+; Set oz timeout higher on x390x image builder to allow it to finish.
+; Install timeout(seconds) for image build
+; if it's unset, use the number in /etc/oz/oz.cfg, supported since
oz-0.16.0
+oz_install_timeout=14400
+{% endif %}
 use_createrepo_c=True

{% if host in groups['buildvm-s390x'] %}

3. Run the playbook on it to update the config and restart kojid.

+1s?

kevin

[Ricky Elrod]

+1. Do we know 14400 will be enough?

-re

On Fri, Mar 16, 2018 at 3:29 PM, Kevin Fenzi  wrote:
> Greetings.
>
> image composes using oz take a long time on our s390x image builder,
> (due to them being in another datacenter). They take so long, they timeout.
>
> See: https://pagure.io/koji/pull-request/837 from sinny and
> https://pagure.io/koji/pull-request/841 which makes this value
> configurable in kojid.conf
>
> I would like to:
> Update koji on buildvm-s390x-02 to the version I just built in
>
> rawhide/f28 that has a backport of the patch that makes this value
> configurable.
> Apply the following ansible patch to set the timeout only on the
>
> buildvm-s390x-02 builder:
>
> diff --git a/roles/koji_builder/templates/kojid.conf
> b/roles/koji_builder/templates/kojid.conf
> index 832d80c..9f10d9c 100644
> --- a/roles/koji_builder/templates/kojid.conf
> +++ b/roles/koji_builder/templates/kojid.conf
> @@ -20,7 +20,12 @@ maxjobs=25
>  keepalive=False
>
> rpmbuild_timeout=172800
>
> +{% if ansible_hostname.startswith('buildvm-s390x-02') %}
> +; Set oz timeout higher on x390x image builder to allow it to finish.
> +; Install timeout(seconds) for image build
> +; if it's unset, use the number in /etc/oz/oz.cfg, supported since
> oz-0.16.0
> +oz_install_timeout=14400
> +{% endif %}
>  use_createrepo_c=True
>
> {% if host in groups['buildvm-s390x'] %}
> Run the playbook on it to update the config and restart kojid.
>
> +1s?
>
> kevin
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Kevin Fenzi]

On 03/16/2018 12:42 PM, Ricky Elrod wrote:
> +1. Do we know 14400 will be enough?

We don't, but I figured we could start there and adjust it if it wasn't
enough still.

kevin
--
> -re
>
> On Fri, Mar 16, 2018 at 3:29 PM, Kevin Fenzi  wrote:
> > Greetings.
> >
> > image composes using oz take a long time on our s390x image builder,
> > (due to them being in another datacenter). They take so long, they timeout.
> >
> > See: https://pagure.io/koji/pull-request/837 from sinny and
> > https://pagure.io/koji/pull-request/841 which makes this value
> > configurable in kojid.conf
> >
> > I would like to:
> > Update koji on buildvm-s390x-02 to the version I just built in
> >
> > rawhide/f28 that has a backport of the patch that makes this value
> > configurable.
> > Apply the following ansible patch to set the timeout only on the
> >
> > buildvm-s390x-02 builder:
> >
> > diff --git a/roles/koji_builder/templates/kojid.conf
> > b/roles/koji_builder/templates/kojid.conf
> > index 832d80c..9f10d9c 100644
> > --- a/roles/koji_builder/templates/kojid.conf
> > +++ b/roles/koji_builder/templates/kojid.conf
> > @@ -20,7 +20,12 @@ maxjobs=25
> >  keepalive=False
> >
> > rpmbuild_timeout=172800
> >
> > +{% if ansible_hostname.startswith('buildvm-s390x-02') %}
> > +; Set oz timeout higher on x390x image builder to allow it to finish.
> > +; Install timeout(seconds) for image build
> > +; if it's unset, use the number in /etc/oz/oz.cfg, supported since
> > oz-0.16.0
> > +oz_install_timeout=14400
> > +{% endif %}
> >  use_createrepo_c=True
> >
> > {% if host in groups['buildvm-s390x'] %}
> > Run the playbook on it to update the config and restart kojid.
> >
> > +1s?
> >
> > kevin
> >
> > infrastructure mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Patrick Uiterwijk]

+1 to try

On Fri, Mar 16, 2018 at 8:51 PM, Kevin Fenzi  wrote:
> On 03/16/2018 12:42 PM, Ricky Elrod wrote:
> > +1. Do we know 14400 will be enough?
>
> We don't, but I figured we could start there and adjust it if it wasn't
> enough still.
> kevin
> > -re
> >
> > On Fri, Mar 16, 2018 at 3:29 PM, Kevin Fenzi  wrote:
> > > Greetings.
> > >
> > > image composes using oz take a long time on our s390x image builder,
> > > (due to them being in another datacenter). They take so long, they timeout.
> > >
> > > See: https://pagure.io/koji/pull-request/837 from sinny and
> > > https://pagure.io/koji/pull-request/841 which makes this value
> > > configurable in kojid.conf
> > >
> > > I would like to:
> > > Update koji on buildvm-s390x-02 to the version I just built in
> > >
> > > rawhide/f28 that has a backport of the patch that makes this value
> > > configurable.
> > > Apply the following ansible patch to set the timeout only on the
> > >
> > > buildvm-s390x-02 builder:
> > >
> > > diff --git a/roles/koji_builder/templates/kojid.conf
> > > b/roles/koji_builder/templates/kojid.conf
> > > index 832d80c..9f10d9c 100644
> > > --- a/roles/koji_builder/templates/kojid.conf
> > > +++ b/roles/koji_builder/templates/kojid.conf
> > > @@ -20,7 +20,12 @@ maxjobs=25
> > >  keepalive=False
> > >
> > > rpmbuild_timeout=172800
> > >
> > > +{% if ansible_hostname.startswith('buildvm-s390x-02') %}
> > > +; Set oz timeout higher on x390x image builder to allow it to finish.
> > > +; Install timeout(seconds) for image build
> > > +; if it's unset, use the number in /etc/oz/oz.cfg, supported since
> > > oz-0.16.0
> > > +oz_install_timeout=14400
> > > +{% endif %}
> > >  use_createrepo_c=True
> > >
> > > {% if host in groups['buildvm-s390x'] %}
> > > Run the playbook on it to update the config and restart kojid.
> > >
> > > +1s?
> > >
> > > kevin
> > >
> > > infrastructure mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> >
> > infrastructure mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Dusty Mabe]

On 03/16/2018 03:29 PM, Kevin Fenzi wrote:
> Greetings.
>
> image composes using oz take a long time on our s390x image builder,
> (due to them being in another datacenter). They take so long, they timeout.
>
> See: https://pagure.io/koji/pull-request/837 from sinny and
> https://pagure.io/koji/pull-request/841 which makes this value
> configurable in kojid.conf
>
> I would like to:
> Update koji on buildvm-s390x-02 to the version I just built in
>
> rawhide/f28 that has a backport of the patch that makes this value
> configurable.
> Apply the following ansible patch to set the timeout only on the
>
> buildvm-s390x-02 builder:

Thanks Kevin. We are now hitting another issue on s390x during anaconda.

https://koji.fedoraproject.org/koji/taskinfo?taskID=25761596
```
10:58:45,692 ERR anaconda:anaconda: startup_utils: Waiting for modules to be started timed out.
10:58:45,804 ERR anaconda:anaconda: stdout: Anaconda DBus modules failed to start on time.
```

Sinny, can you investigate this new issue?

Dusty

[Sinny Kumari]

On Sat, Mar 17, 2018 at 8:48 PM, Dusty Mabe  wrote:
> On 03/16/2018 03:29 PM, Kevin Fenzi wrote:
> > Greetings.
> >
> > image composes using oz take a long time on our s390x image builder,
> > (due to them being in another datacenter). They take so long, they timeout.
> >
> > See: https://pagure.io/koji/pull-request/837 from sinny and
> > https://pagure.io/koji/pull-request/841 which makes this value
> > configurable in kojid.conf
> >
> > I would like to:
> > Update koji on buildvm-s390x-02 to the version I just built in
> >
> > rawhide/f28 that has a backport of the patch that makes this value
> > configurable.
> > Apply the following ansible patch to set the timeout only on the
> >
> > buildvm-s390x-02 builder:
>
> Thanks Kevin. We are now hitting another issue on s390x during anaconda.
>
> https://koji.fedoraproject.org/koji/taskinfo?taskID=25761596
> 10:58:45,692 ERR anaconda:anaconda: startup_utils: Waiting for modules to be started timed out.
> 10:58:45,804 ERR anaconda:anaconda: stdout: Anaconda DBus modules failed to start on time.
>
> Sinny, can you investigate this new issue?

Last two F28 S390x cloud-base compose from 20180317 [1] and
20180318[2] looks good.
It might be a temporary issue, will look into it if error occurs again.

[1] https://kojipkgs.fedoraproject.org/compose/branched/Fedora-28-20180317.n.0/c...
[2] https://kojipkgs.fedoraproject.org/compose/branched/Fedora-28-20180318.n.0/c...
> Dusty
> _______________________________________________
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Sinny Kumari]

On Sat, Mar 17, 2018 at 12:59 AM, Kevin Fenzi  wrote:
> Greetings.
>
> image composes using oz take a long time on our s390x image builder,
> (due to them being in another datacenter). They take so long, they timeout.
>
> See: https://pagure.io/koji/pull-request/837 from sinny and
> https://pagure.io/koji/pull-request/841 which makes this value
> configurable in kojid.conf

Thanks a lot Kevin for this fix!
> I would like to:
> Update koji on buildvm-s390x-02 to the version I just built in
>
> rawhide/f28 that has a backport of the patch that makes this value
> configurable.
> Apply the following ansible patch to set the timeout only on the
>
> buildvm-s390x-02 builder:
>
> diff --git a/roles/koji_builder/templates/kojid.conf
> b/roles/koji_builder/templates/kojid.conf
> index 832d80c..9f10d9c 100644
> --- a/roles/koji_builder/templates/kojid.conf
> +++ b/roles/koji_builder/templates/kojid.conf
> @@ -20,7 +20,12 @@ maxjobs=25
>  keepalive=False
>
> rpmbuild_timeout=172800
>
> +{% if ansible_hostname.startswith('buildvm-s390x-02') %}
> +; Set oz timeout higher on x390x image builder to allow it to finish.
> +; Install timeout(seconds) for image build
> +; if it's unset, use the number in /etc/oz/oz.cfg, supported since
> oz-0.16.0
> +oz_install_timeout=14400
> +{% endif %}
>  use_createrepo_c=True
>
> {% if host in groups['buildvm-s390x'] %}
> Run the playbook on it to update the config and restart kojid.
>
> +1s?
>
> kevin
>
> infrastructure mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Kevin Fenzi]

On 03/18/2018 08:33 PM, Sinny Kumari wrote:
> On Sat, Mar 17, 2018 at 12:59 AM, Kevin Fenzi  wrote:
> > Greetings.
> >
> > image composes using oz take a long time on our s390x image builder,
> > (due to them being in another datacenter). They take so long, they timeout.
> >
> > See: https://pagure.io/koji/pull-request/837 from sinny and
> > https://pagure.io/koji/pull-request/841 which makes this value
> > configurable in kojid.conf
>
> Thanks a lot Kevin for this fix!

No problem. :)

kevin