Hello!
On Jun 10, 2020, a new Copr release landed production. Related client tooling updates are here:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-390cce74d6 https://bodhi.fedoraproject.org/updates/FEDORA-2020-f90873700f https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-3b8c2778f1 https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-feca133c66 https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8449e5d605
Here is the list of visible changes, and new features:
- Increased build task throughput. The VM spawner is more flexible now. First, we don't allocate that many workers if they are not actually needed, but more importantly the upper limit for concurrently running workers has risen to 140 (including all architectures), previously we had ~70. We'll see how things go from the backend perspective, but it is expected that we'll go even higher (waiting for a new HW in the new Fedora lab, cheaper VM instances in the future, etc.). So the numbers will likely change; this is to make clear the current state of things.
- Copr project "runtime" dependencies were implemented. Newly you can specify set of repositories your project depends on. Such repositories will be installed together with the copr project repo file (e.g., by 'dnf copr enable YOU/YOUR_PROEJCT'). Those repositories can be other project in Copr or some 3rd party repository. This is very similar to build-time dependencies implemented long time ago. Take a look at blog post:
https://fedora-copr.github.io/posts/runtime-dependencies
- There's now more fair build scheduler. Previously, no matter whether the "background" attribute was set or not for the build - once builder was allocated for a concrete copr user - copr never terminated the builder as long as the user kept filling the build queue with new tasks (and it blocked the quota for others). Newly, there's a limit of at most eight consecutive builds or 30 minutes for one user (sandbox) on one builder and the builder is immediately terminated - which gives a chance to assign new builders to others' tasks (which have a higher priority at that point).
- Copr-cli supports batch build delete feature:
$ copr-cli delete build_id [build_id ...]
Please use this instead of requesting removal of each build_id separately - it will be much faster because you will save the copr backend useless createrepo_c runs after each request.
- We also implemented a priority queue algorithm for Action tasks (non-build tasks, e.g., forking, build/project deleting, etc.). This should solve several issues when the action queue gets temporarily too large to process immediately; namely the delayed "initial createrepo" action problem that previously caused an ugly build failures in freshly created copr projects.
- Cancel build feature was fixed, and the "cancel" request should reliably release all occupied builder machines (allowing them to be re-used, or terminated).
- Users are not allowed to disable chroot in a project that has some running builds against the disabled chroot. The running builds need to be canceled first. This change as done to avoid inconsistencies between frontend/backend, and wasted builder quota on useless tasks.
- More space for /var/cache/mock directory, so repeated builds on the same machine won't face the ENOSPC error on mock caches. Considering that each builder can only do at most eight builds (then terminated), this problem should be finally fixed.
Happy building! Pavel
On Wed, 10 Jun 2020 at 21:42, Pavel Raiskup praiskup@redhat.com wrote:
- Copr project "runtime" dependencies were implemented.
Great!
Copr-cli supports batch build delete feature:
$ copr-cli delete build_id [build_id ...]
Hooray! Just delete or delete-build?
On Wednesday, June 10, 2020 10:31:32 PM CEST Iñaki Ucar wrote:
Copr-cli supports batch build delete feature:
$ copr-cli delete build_id [build_id ...]
Hooray! Just delete or delete-build?
Yes, typo, I meant "delete-build" command.
Pavel
On 10. 06. 20 21:41, Pavel Raiskup wrote:
Hello!
On Jun 10, 2020, a new Copr release landed production.
Thank You!
Here is the list of visible changes, and new features:
- Increased build task throughput. The VM spawner is more flexible now. First, we don't allocate that many workers if they are not actually needed, but more importantly the upper limit for concurrently running workers has risen to 140 (including all architectures), previously we had ~70. We'll see how things go from the backend perspective, but it is expected that we'll go even higher (waiting for a new HW in the new Fedora lab, cheaper VM instances in the future, etc.). So the numbers will likely change; this is to make clear the current state of things.
Happy to see this in action.
Copr project "runtime" dependencies were implemented. Newly you can specify set of repositories your project depends on. Such repositories will be installed together with the copr project repo file (e.g., by 'dnf copr enable YOU/YOUR_PROEJCT'). Those repositories can be other project in Copr or some 3rd party repository. This is very similar to build-time dependencies implemented long time ago. Take a look at blog post:
This sounds really interesting and helpful. Thanks.
- There's now more fair build scheduler. Previously, no matter whether the "background" attribute was set or not for the build - once builder was allocated for a concrete copr user - copr never terminated the builder as long as the user kept filling the build queue with new tasks (and it blocked the quota for others). Newly, there's a limit of at most eight consecutive builds or 30 minutes for one user (sandbox) on one builder and the builder is immediately terminated - which gives a chance to assign new builders to others' tasks (which have a higher priority at that point).
I've been personalyl affected by the old behavior, happy to see this.
Copr-cli supports batch build delete feature:
$ copr-cli delete build_id [build_id ...]
Please use this instead of requesting removal of each build_id separately - it will be much faster because you will save the copr backend useless createrepo_c runs after each request.
Yet again, \o/
- Cancel build feature was fixed, and the "cancel" request should reliably release all occupied builder machines (allowing them to be re-used, or terminated).
Awesome as well.
Thanks for this release. I've also seen some UX changes wehn viewing builds, the builder-live.log.gz is now 1 click closer. This seems slow, but in bulk, is really awesome.
Keep up the good work. I am so glad we have such innovations!
...snip...
- There's now more fair build scheduler. Previously, no matter whether the "background" attribute was set or not for the build - once builder was allocated for a concrete copr user - copr never terminated the builder as long as the user kept filling the build queue with new tasks (and it blocked the quota for others). Newly, there's a limit of at most eight consecutive builds or 30 minutes for one user (sandbox) on one builder and the builder is immediately terminated - which gives a chance to assign new builders to others' tasks (which have a higher priority at that point).
I am confused by this scheduler part. Does this mean that any build that takes longer than 30 minutes will be cancelled? What about software packages that are larger and require more build time?
-Ian
On Tuesday, June 16, 2020 8:48:32 PM CEST Ian McInerney wrote:
...snip...
- There's now more fair build scheduler. Previously, no matter whether the "background" attribute was set or not for the build - once builder was allocated for a concrete copr user - copr never terminated the builder as long as the user kept filling the build queue with new tasks (and it blocked the quota for others). Newly, there's a limit of at most eight consecutive builds or 30 minutes for one user (sandbox) on one builder and the builder is immediately terminated - which gives a chance to assign new builders to others' tasks (which have a higher priority at that point).
I am confused by this scheduler part. Does this mean that any build that takes longer than 30 minutes will be cancelled? What about software packages that are larger and require more build time?
This deserved better spelling => any builder that was assigned to one user for _more than 30 minutes_ is - after the build _is finished_ - terminated. The build timeout is a different configuration option (currently around 24h, and the timeout concept should change soon so it is more flexible, https://pagure.io/copr/copr/issue/1303 ).
Pavel
-Ian
copr-devel@lists.fedorahosted.org
-
Ian McInerney -
Iñaki Ucar -
Miro Hrončok -
Pavel Raiskup