Highlights from the latest Copr release 2020-06-10

[Pavel Raiskup]

Hello!

On Jun 10, 2020, a new Copr release landed production.  Related client tooling
updates are here:

  https://bodhi.fedoraproject.org/updates/FEDORA-2020-390cce74d6
  https://bodhi.fedoraproject.org/updates/FEDORA-2020-f90873700f
  https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-3b8c2778f1
  https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-feca133c66
  https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8449e5d605

Here is the list of visible changes, and new features:

- Increased build task throughput.  The VM spawner is more flexible now.
  First, we don't allocate that many workers if they are not actually
  needed, but more importantly the upper limit for concurrently running
  workers has risen to 140 (including all architectures), previously we
  had ~70.  We'll see how things go from the backend perspective, but it is
  expected that we'll go even higher (waiting for a new HW in the new
  Fedora lab, cheaper VM instances in the future, etc.).  So the numbers will
  likely change; this is to make clear the current state of things.

- Copr project "runtime" dependencies were implemented.  Newly you can
  specify set of repositories your project depends on. Such repositories
  will be installed together with the copr project repo file (e.g., by
  'dnf copr enable YOU/YOUR_PROEJCT').  Those repositories can be other
  project in Copr or some 3rd party repository.  This is very similar to
  build-time dependencies implemented long time ago.  Take a look at
  blog post:

    https://fedora-copr.github.io/posts/runtime-dependencies

- There's now more fair build scheduler.  Previously, no matter whether the
  "background" attribute was set or not for the build - once builder was
  allocated for a concrete copr user - copr never terminated the builder
  as long as the user kept filling the build queue with new tasks (and it
  blocked the quota for others).  Newly, there's a limit of at most
  eight consecutive builds or 30 minutes for one user (sandbox) on one
  builder and the builder is immediately terminated - which gives a chance
  to assign new builders to others' tasks (which have a higher priority at
  that point).

- Copr-cli supports batch build delete feature:

    $ copr-cli delete build_id [build_id ...]

  Please use this instead of requesting removal of each build_id
  separately - it will be much faster because you will save the copr
  backend useless createrepo_c runs after each request.

- We also implemented a priority queue algorithm for Action tasks
  (non-build tasks, e.g., forking, build/project deleting, etc.).  This
  should solve several issues when the action queue gets temporarily too
  large to process immediately; namely the delayed "initial createrepo"
  action problem that previously caused an ugly build failures in freshly
  created copr projects.

- Cancel build feature was fixed, and the "cancel" request should reliably
  release all occupied builder machines (allowing them to be re-used, or
  terminated).

- Users are not allowed to disable chroot in a project that has some
  running builds against the disabled chroot.  The running builds need to
  be canceled first.  This change as done to avoid inconsistencies between
  frontend/backend, and wasted builder quota on useless tasks.

- More space for /var/cache/mock directory, so repeated builds on the same
  machine won't face the ENOSPC error on mock caches.  Considering that
  each builder can only do at most eight builds (then terminated), this
  problem should be finally fixed.

Happy building!
Pavel

[Iñaki Ucar]

On Wed, 10 Jun 2020 at 21:42, Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:
>  - Copr project "runtime" dependencies were implemented. 
Great!

>  - Copr-cli supports batch build delete feature:
>
>      $ copr-cli delete build_id [build_id ...] 
Hooray! Just delete or delete-build?

-- 
Iñaki Úcar

[Miro Hrončok]

On 10. 06. 20 21:41, Pavel Raiskup wrote:
>  Hello!
>  
>  On Jun 10, 2020, a new Copr release landed production. 
Thank You!

>  Here is the list of visible changes, and new features:
>  
>  - Increased build task throughput.  The VM spawner is more flexible now.
>     First, we don't allocate that many workers if they are not actually
>     needed, but more importantly the upper limit for concurrently running
>     workers has risen to 140 (including all architectures), previously we
>     had ~70.  We'll see how things go from the backend perspective, but it is
>     expected that we'll go even higher (waiting for a new HW in the new
>     Fedora lab, cheaper VM instances in the future, etc.).  So the numbers will
>     likely change; this is to make clear the current state of things. 
Happy to see this in action.

>  - Copr project "runtime" dependencies were implemented. 
> Newly you can
>     specify set of repositories your project depends on. Such repositories
>     will be installed together with the copr project repo file (e.g., by
>     'dnf copr enable YOU/YOUR_PROEJCT').  Those repositories can be other
>     project in Copr or some 3rd party repository.  This is very similar to
>     build-time dependencies implemented long time ago.  Take a look at
>     blog post:
>  
>       https://fedora-copr.github.io/posts/runtime-dependencies 
This sounds really interesting and helpful. Thanks.

>  - There's now more fair build scheduler.  Previously, no matter
> whether the
>     "background" attribute was set or not for the build - once builder was
>     allocated for a concrete copr user - copr never terminated the builder
>     as long as the user kept filling the build queue with new tasks (and it
>     blocked the quota for others).  Newly, there's a limit of at most
>     eight consecutive builds or 30 minutes for one user (sandbox) on one
>     builder and the builder is immediately terminated - which gives a chance
>     to assign new builders to others' tasks (which have a higher priority at
>     that point). 
I've been personalyl affected by the old behavior, happy to see this.

>  - Copr-cli supports batch build delete feature:
>  
>       $ copr-cli delete build_id [build_id ...]
>  
>     Please use this instead of requesting removal of each build_id
>     separately - it will be much faster because you will save the copr
>     backend useless createrepo_c runs after each request. 
Yet again, \o/

>  - Cancel build feature was fixed, and the "cancel" request
> should reliably
>     release all occupied builder machines (allowing them to be re-used, or
>     terminated). 
Awesome as well.


Thanks for this release. I've also seen some UX changes wehn viewing builds, the 
builder-live.log.gz is now 1 click closer. This seems slow, but in bulk, is 
really awesome.

Keep up the good work. I am so glad we have such innovations!

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 11. 06. 20 1:58, Miro Hrončok wrote:
>  This seems slow, but in bulk, is really awesome. 
This seems **small*, but in bulk, is really awesome.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Pavel Raiskup]

On Wednesday, June 10, 2020 10:31:32 PM CEST Iñaki Ucar wrote:
>  > - Copr-cli supports batch build delete feature:
>  >
>  >     $ copr-cli delete build_id [build_id ...]
>
>  Hooray! Just delete or delete-build? 
Yes, typo, I meant "delete-build" command.

Pavel

[Ian McInerney]

...snip...
>  - There's now more fair build scheduler.  Previously, no matter whether the
>    "background" attribute was set or not for the build - once builder was
>    allocated for a concrete copr user - copr never terminated the builder
>    as long as the user kept filling the build queue with new tasks (and it
>    blocked the quota for others).  Newly, there's a limit of at most
>    eight consecutive builds or 30 minutes for one user (sandbox) on one
>    builder and the builder is immediately terminated - which gives a chance
>    to assign new builders to others' tasks (which have a higher priority at
>    that point).
>  
I am confused by this scheduler part. Does this mean that any build that
takes longer than 30 minutes will be cancelled? What about software
packages that are larger and require more build time?

-Ian

[Pavel Raiskup]

On Tuesday, June 16, 2020 8:48:32 PM CEST Ian McInerney wrote:
>  ...snip...
>  >
>  > - There's now more fair build scheduler.  Previously, no matter whether the
>  >   "background" attribute was set or not for the build - once builder was
>  >   allocated for a concrete copr user - copr never terminated the builder
>  >   as long as the user kept filling the build queue with new tasks (and it
>  >   blocked the quota for others).  Newly, there's a limit of at most
>  >   eight consecutive builds or 30 minutes for one user (sandbox) on one
>  >   builder and the builder is immediately terminated - which gives a chance
>  >   to assign new builders to others' tasks (which have a higher priority at
>  >   that point).
>  >
>
>  I am confused by this scheduler part. Does this mean that any build that
>  takes longer than 30 minutes will be cancelled? What about software
>  packages that are larger and require more build time? 
This deserved better spelling => any builder that was assigned to one user for
_more than 30 minutes_ is - after the build _is finished_ - terminated.  The
build timeout is a different configuration option (currently around 24h, and the
timeout concept should change soon so it is more flexible,
https://pagure.io/copr/copr/issue/1303 ).

Pavel

>  -Ian
>