"Action Items" From FUDCon?

[Michael Schwendt]

On Mon, 14 Jan 2008 10:22:21 -0600, Matt Domsch wrote:

> * Mock has gotten faster recently - enough so that we could probably
>   throw lots more builds at our build systems. 

Can Koji deal with "lots more builds" in appropriate ways? Would it kill
(and later requeue) a low-priority build-job in favour of a more important
build-job?

[...]

It's the huge packages (e.g. openoffice, kernel or several MB of C++)
which occupy a build-host for a long time because compiling them takes
much more time than setting up a mock buildroot. When such build-jobs are
active, they slow down other builds on the same host significantly. Koji
tries to spread the load, but for "lots more builds", such as scratch and
debug builds, it would better use special spare servers if any of those
jobs ran for a long or unexpected period. Unless it can kill+requeue less
important builds automatically.

As Plague has gotten faster recently, too, and no longer chokes on
downloading build results, the build server load becomes more obvious.
Together with mock's caching of initial buildroots, the many small
packages now build in roughly one minute when a build-host is not too
busy. Examples in footnote [1]. But as soon as a builder runs a "heavy"
koji build-job in parallel, it can be seen how that slows down the
additional plague build-job:

  http://buildsys.fedoraproject.org/build-status/job.psp?uid=37880
   - xenbuilder2 took 5 minutes for i386/x86_64
   - ppc2 took 19 minutes (!)

  http://buildsys.fedoraproject.org/build-status/job.psp?uid=37878
   - hammer2 took 1 minute for i386 (!)
   - ppc2 took 3 minutes
   - xenbuilder2 took 10 minutes for x86_64

  http://buildsys.fedoraproject.org/build-status/job.psp?uid=37876
   - ppc2 took 4 minutes
   - xenbuilder2 took 10-11 minutes for i386/x86_64

The second example is most impressive. Plague doesn't choose hammer2 most
of the time, because with the current config it first occupies two of the
four slots on xenbuilder2 before using hammer2. It doesn't know either
when koji already uses a build-host. But I assume that koji builds are
slowed down similarly.

So, better not try to keep the build-hosts busy when that would increase
the average time of response for our packagers. The more free the builders
are, the faster a packager receives results for a build, even for a
helpful scratch-build of a package that compiles for 10-15 minutes till
it ends with an error.

-- 
[1]
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37870
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37875
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37874
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37865
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37859
http://buildsys.fedoraproject.org/build-status/job.psp?uid=37853