Koschei - continuous rebuilds for packages

[Michael Simacek]

Hello,

Recently I've been working on Koschei - a new continuous rebuilding tool for
Fedora and I'd like to announce it publicly and get some feedback.

How will it work?
It will have a set of tracked packages for which it will watch dependency
updates and will (scratch) rebuild packages in Koji after the dependencies
change (are updated, downgraded...). It will use Hawkey library to resolve
packages' dependencies in order to match resolution happening during the Koji
build as closely as possible. That would also enable marking packages with
unresolved build dependencies as unbuildable without needing actual rebuild in
Koji. The packages will be scheduled for rebuild based on their current priority
value, which will be increased with each depenendency update by a value
inversely proportional to the distance between the package and the dependency in
the dependency chain. The priority will also slowly increase over time and will
be reset back when a new rebuild is scheduled.

What will be the benefits?
Currently, with each release of Fedora there are lot of packages that fail to
build from source because their dependencies were updated to an incompatible
version. The breakage is usually detected during the mass rebuild and fixing of
the FTBFS bugs is a tedious process, because it's hard to determine the exact
cause of the failure. Koschei could narrow down the set of packages that may be
the cause to just one or two which could reduce the time spent on fixing such
bugs.

What about Koji load?
It will be limiting the number of rebuilds in order to not overload Koji
builders - by limiting maximum number of running builds and monitoring Koji
current load. Maybe it would also be possible to get a separate Koji channel.
We'll try our best to keep the overall Koji performance unaffected. Any
suggestions regarding this part are welcome.

What's the current status?
The core part is already implemented - dependency resolution, fedmsg listening,
update detection, build submitting, collecting the results, parsing buildroot
differences from mock root logs. The source repository is available at [2]

What packages will be there?
Currently, the packages are added manually by me. Lately it will depend on the
demand and how much it will be affecting Koji. Ideally, any Fedora user could
add packages himself in the web interface.

What is it's output?
Currently there's a simple reporter generating static html report of all builds
of all packages (example at [1]). The current format is of course not very
practical because the number of packges and builds will both increase a lot, so
I'd like to implement a dynamic web interface that will provide a way to
categorize packages into user defined groups and display overview of packages in
given group. In the overview only last Koschei build and last successful build
if the current one fails will be displayed. Then there will be detailed
per-package view showing all package's recent scratch builds, statistics,
buildroot differences, likely failure reason etc.

What is the plan for it?
Get feedback from people. Try to get it deployed in Fedora infrastructure and
see the how does it perform. If it turns out that Koji performance is lowered
we'll stop it and reconsider how to avoid such a condition.


Michael Simacek

[1] http://msimacek.fedorapeople.org/koschei/
[2] https://github.com/msimacek/koschei