Proposal to increase number of concurrent MBS builds

[Mikolaj Izdebski]

I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to
value 100 after beta freeze is over.

The above MBS setting controls how many parallel component builds MBS
can run at the same time. Current value is only 20. That value was
defined 3 years ago, in March 2017 when there were much fewer modules
in Fedora. Our Koji has capacity to run many more builds. Currently we
have 158 Koji builders in default channel that are able to run RPM
builds. Their total capacity is 566. Therefore it should be safe to
increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since
Koji builds submitted by MBS have lower priority than default for
non-modular component builds submitted by packagers, so
packager-submitted builds will take precedence over builds submitted
by MBS.

--
Mikolaj Izdebski

[Stephen John Smoogen]

On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to
>  value 100 after beta freeze is over.
>
>  The above MBS setting controls how many parallel component builds MBS
>  can run at the same time. Current value is only 20. That value was
>  defined 3 years ago, in March 2017 when there were much fewer modules
>  in Fedora. Our Koji has capacity to run many more builds. Currently we
>  have 158 Koji builders in default channel that are able to run RPM
>  builds. Their total capacity is 566. Therefore it should be safe to
>  increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since
>  Koji builds submitted by MBS have lower priority than default for
>  non-modular component builds submitted by packagers, so
>  packager-submitted builds will take precedence over builds submitted
>  by MBS.
>  
Does the koji numbers above take into account architectures? While
have 158 builders they are spread out over a lot of architectures with
~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and
I guess ~70 x86_64 builders. So while we could build a lot more x86_64
builds at once, could we completely swamp the other arches and
actually cause problems for even completing the x86_64 ones?

If the architecture would be a problem, should we look at the
smaller/slowest number of builders and work out what its capacity is
and not go much above that?

>  --
>  Mikolaj Izdebski
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.

[Mikolaj Izdebski]

On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  >
>  > I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to
>  > value 100 after beta freeze is over.
>  >
>  > The above MBS setting controls how many parallel component builds MBS
>  > can run at the same time. Current value is only 20. That value was
>  > defined 3 years ago, in March 2017 when there were much fewer modules
>  > in Fedora. Our Koji has capacity to run many more builds. Currently we
>  > have 158 Koji builders in default channel that are able to run RPM
>  > builds. Their total capacity is 566. Therefore it should be safe to
>  > increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since
>  > Koji builds submitted by MBS have lower priority than default for
>  > non-modular component builds submitted by packagers, so
>  > packager-submitted builds will take precedence over builds submitted
>  > by MBS.
>  >
>
>  Does the koji numbers above take into account architectures? While
>  have 158 builders they are spread out over a lot of architectures with
>  ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and
>  I guess ~70 x86_64 builders. So while we could build a lot more x86_64
>  builds at once, could we completely swamp the other arches and
>  actually cause problems for even completing the x86_64 ones?
>
>  If the architecture would be a problem, should we look at the
>  smaller/slowest number of builders and work out what its capacity is
>  and not go much above that? 
That should not be a problem. Koji can handle thousands of builds
submitted at the same time. This happens, for example, during mass
rebuilds.

--
Mikolaj Izdebski

[Stephen John Smoogen]

On Mon, 11 Mar 2019 at 06:55, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  >
>  > On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  > >
>  > > I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to
>  > > value 100 after beta freeze is over.
>  > >
>  > > The above MBS setting controls how many parallel component builds MBS
>  > > can run at the same time. Current value is only 20. That value was
>  > > defined 3 years ago, in March 2017 when there were much fewer modules
>  > > in Fedora. Our Koji has capacity to run many more builds. Currently we
>  > > have 158 Koji builders in default channel that are able to run RPM
>  > > builds. Their total capacity is 566. Therefore it should be safe to
>  > > increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since
>  > > Koji builds submitted by MBS have lower priority than default for
>  > > non-modular component builds submitted by packagers, so
>  > > packager-submitted builds will take precedence over builds submitted
>  > > by MBS.
>  > >
>  >
>  > Does the koji numbers above take into account architectures? While
>  > have 158 builders they are spread out over a lot of architectures with
>  > ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and
>  > I guess ~70 x86_64 builders. So while we could build a lot more x86_64
>  > builds at once, could we completely swamp the other arches and
>  > actually cause problems for even completing the x86_64 ones?
>  >
>  > If the architecture would be a problem, should we look at the
>  > smaller/slowest number of builders and work out what its capacity is
>  > and not go much above that?
>
>  That should not be a problem. Koji can handle thousands of builds
>  submitted at the same time. This happens, for example, during mass
>  rebuilds.
>  
Yeah.. sorry I didn't explain things clearly. Koji can handle it but
does it make the developer experience worse across the board? We could
do a mass rebuild every day and koji wouldn't break a sweat.. but
developers actually wanting to get stuff out of Fedora would hate it
because the queues for builds would be so long. My questions should
have stated that.



-- 
Stephen J Smoogen.

[Mikolaj Izdebski]

On Mon, Mar 11, 2019 at 12:41 PM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Mon, 11 Mar 2019 at 06:55, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt; wrote:
>  >
>  > On Mon, Mar 11, 2019 at 10:43 AM Stephen John Smoogen <smooge(a)gmail.com&gt;
> wrote:
>  > >
>  > > On Mon, 11 Mar 2019 at 04:52, Mikolaj Izdebski <mizdebsk(a)redhat.com&gt;
> wrote:
>  > > >
>  > > > I would like to propose increasing NUM_CONCURRENT_BUILDS for MBS to
>  > > > value 100 after beta freeze is over.
>  > > >
>  > > > The above MBS setting controls how many parallel component builds MBS
>  > > > can run at the same time. Current value is only 20. That value was
>  > > > defined 3 years ago, in March 2017 when there were much fewer modules
>  > > > in Fedora. Our Koji has capacity to run many more builds. Currently we
>  > > > have 158 Koji builders in default channel that are able to run RPM
>  > > > builds. Their total capacity is 566. Therefore it should be safe to
>  > > > increase MBS NUM_CONCURRENT_BUILDS to at least 100. Especially since
>  > > > Koji builds submitted by MBS have lower priority than default for
>  > > > non-modular component builds submitted by packagers, so
>  > > > packager-submitted builds will take precedence over builds submitted
>  > > > by MBS.
>  > > >
>  > >
>  > > Does the koji numbers above take into account architectures? While
>  > > have 158 builders they are spread out over a lot of architectures with
>  > > ~12 s390x builders, ~30 ppc64le, ~20 arm and ~20 aarch64 builders and
>  > > I guess ~70 x86_64 builders. So while we could build a lot more x86_64
>  > > builds at once, could we completely swamp the other arches and
>  > > actually cause problems for even completing the x86_64 ones?
>  > >
>  > > If the architecture would be a problem, should we look at the
>  > > smaller/slowest number of builders and work out what its capacity is
>  > > and not go much above that?
>  >
>  > That should not be a problem. Koji can handle thousands of builds
>  > submitted at the same time. This happens, for example, during mass
>  > rebuilds.
>  >
>
>  Yeah.. sorry I didn't explain things clearly. Koji can handle it but
>  does it make the developer experience worse across the board? We could
>  do a mass rebuild every day and koji wouldn't break a sweat.. but
>  developers actually wanting to get stuff out of Fedora would hate it
>  because the queues for builds would be so long. My questions should
>  have stated that. 
As I mentioned in the first email, builds submitted by MBS have lower
priority than default. If all Koji builders are loaded with modular
builds then developer may need to wait a bit before one of them
completes. But once any builder has free capacity, it will pick up
developer-submitted builds before MBS-submitted ones.

--
Mikolaj Izdebski

[Kevin Fenzi]

+1 to the change here.

If it gives us any problems we can adjust it later...

kevin

[Stephen John Smoogen]

+1 also. thank you for answering my questions.

On Mon, 11 Mar 2019 at 14:31, Kevin Fenzi <kevin(a)scrye.com&gt; wrote:
>  +1 to the change here.
>
>  If it gives us any problems we can adjust it later...
>
>  kevin
>
>
>  _______________________________________________
>  infrastructure mailing list -- infrastructure(a)lists.fedoraproject.org
>  To unsubscribe send an email to infrastructure-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...


-- 
Stephen J Smoogen.