On Jun 10, 2020, a new Copr release landed production. Related client tooling
updates are here:
Here is the list of visible changes, and new features:
- Increased build task throughput. The VM spawner is more flexible now.
First, we don't allocate that many workers if they are not actually
needed, but more importantly the upper limit for concurrently running
workers has risen to 140 (including all architectures), previously we
had ~70. We'll see how things go from the backend perspective, but it is
expected that we'll go even higher (waiting for a new HW in the new
Fedora lab, cheaper VM instances in the future, etc.). So the numbers will
likely change; this is to make clear the current state of things.
- Copr project "runtime" dependencies were implemented. Newly you can
specify set of repositories your project depends on. Such repositories
will be installed together with the copr project repo file (e.g., by
'dnf copr enable YOU/YOUR_PROEJCT'). Those repositories can be other
project in Copr or some 3rd party repository. This is very similar to
build-time dependencies implemented long time ago. Take a look at
- There's now more fair build scheduler. Previously, no matter whether the
"background" attribute was set or not for the build - once builder was
allocated for a concrete copr user - copr never terminated the builder
as long as the user kept filling the build queue with new tasks (and it
blocked the quota for others). Newly, there's a limit of at most
eight consecutive builds or 30 minutes for one user (sandbox) on one
builder and the builder is immediately terminated - which gives a chance
to assign new builders to others' tasks (which have a higher priority at
- Copr-cli supports batch build delete feature:
$ copr-cli delete build_id [build_id ...]
Please use this instead of requesting removal of each build_id
separately - it will be much faster because you will save the copr
backend useless createrepo_c runs after each request.
- We also implemented a priority queue algorithm for Action tasks
(non-build tasks, e.g., forking, build/project deleting, etc.). This
should solve several issues when the action queue gets temporarily too
large to process immediately; namely the delayed "initial createrepo"
action problem that previously caused an ugly build failures in freshly
created copr projects.
- Cancel build feature was fixed, and the "cancel" request should reliably
release all occupied builder machines (allowing them to be re-used, or
- Users are not allowed to disable chroot in a project that has some
running builds against the disabled chroot. The running builds need to
be canceled first. This change as done to avoid inconsistencies between
frontend/backend, and wasted builder quota on useless tasks.
- More space for /var/cache/mock directory, so repeated builds on the same
machine won't face the ENOSPC error on mock caches. Considering that
each builder can only do at most eight builds (then terminated), this
problem should be finally fixed.
Is there any way to expose a Koji side tag as an external repo for
Copr? This would be useful to start a rebuild for rawhide before the
side tag is merged without having to set up a temporary Copr repo just
to rebuild that side tag.
We are facing some problems with dist-git importing. The import queue got stuck
several times before - but during the weekend it happened many times and very
Previously I thought it was caused by rather large SRPMs, but I'm not sure now,
we are tracking the bug here: https://pagure.io/copr/copr/issue/1393
Please resubmit your build if you see some failure related to copr-dist-git.
Sorry for inconvenience.
On Sunday, May 31, 2020 7:20:20 AM CEST you wrote:
> This COPR just creates too much tasks for building and other waiting longer
> for available builders.
> Navigate to https://copr.fedorainfracloud.org/admin/legal-flag/
> Contact on owner is: iucar <i.ucar86(a)gmail.com>
We are aware that the build scheduling is not fair (next release will
solve this once and forever, patches are already merged), but even now
builds in one project can not delay builds in other projects.. We have
50 x86_64 builder machines, and one project can only waste 8 builders at
the same time. Can you elaborate on the problems you had? There probably
was much more request from different projects?
Btw., with the next release we'll have much more flexible build spawner,
and in "peak" times it will allow us to have more builders, so please