https://fedoraproject.org/wiki/Changes/MemoryConstraintsMacros
== Summary == Introduce macros, similar to openSUSE's [https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints memory-constraints]), for optionally limiting build parallelism for build-time memory-bound packages
== Owner == * Name: [[User:salimma|Michel Alexandre Salim]] * Email: michel AT michel-slm DOT name
== Detailed Description == Some source packages have a memory usage per build thread higher than the RAM:CPU ratio available in some of our builders. Further, this ratio can be different for different build server on different architectures.
At the moment, such packages ([https://src.fedoraproject.org/rpms/ceph/blob/d7454e4e0a98208dc569553b901a497... ceph], [https://src.fedoraproject.org/rpms/chromium/blob/baaf27b384295d6288ef367dd10... chromium], [https://src.fedoraproject.org/rpms/mcrouter/blob/a0f7ecad2ccc51c4214646b0823... mcrouter]) have to implement their own logic for determining the safe amount of parallelism, and redefine `_smp_build_ncpus`.
When this proposal is implemented, they can instead declaratively specify the amount of RAM needed per build thread, e.g.
%limit_build -m 8192
for declaring a build thread should be allocated 8GB of RAM.
Since Koji supports [https://docs.pagure.org/koji/release_notes/release_notes_1.18/#system-change... setting default values for macros], there will be a macro for the default memory limit (name TBD) that, if set, will be used to cap `_smp_build_ncpus` unless overridden by `%limit_build -m`. 0 I'm proposing to tentatively call the macro package `build-constraints-rpm-macros` to allow the possibility of adding macros for related needs e.g. [https://pagure.io/copr/copr/issue/1678 build timeouts] to the same package.
== Benefit to Fedora == This change simplifies maintaining specs for software that are memory-bounded rather than CPU-bounded on our build servers
It could potentially improve build reliability for these packages, by reducing the number of jobs failing because of OOM errors, and reduce the need for package maintainers to debug these failures.
By keeping the user-facing API aligned with what openSUSE is using, we open up the possibility to collaborate with them and with the upstream RPM project to get such macros upstreamed into RPM itself (see [https://github.com/rpm-software-management/rpm/pull/821 previous attempt]). **note** that is not in scope for this Change.
== Scope == * Proposal owners: ** Introduce new macros ** Update known packages to use the new macros, replacing their custom `_smp_build_ncpus` overrides
* Other developers: ** The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros
* Release engineering: [https://pagure.io/releng/issue/10188 #10188] No mass rebuild needed. Affected packages should be rebuilt using the new macro
* Policies and guidelines: Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages
* Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A
== Upgrade/compatibility impact == No impact, affects package building only. Also, the use of the new macros are optional.
== How To Test == 1. Install `build-constraints-rpm-macros` 2. Modify spec to set `%limit_build -n AMOUNT_IN_MB` in `%build` 3. Rebuild in koji and make sure it passes on all supported architectures
== User Experience ==
== Dependencies == This can optionally be added as dependencies of `redhat-rpm-config` and `epel-rpm-macros`, depending on how many packages need this
== Contingency Plan ==
* Contingency mechanism: (What to do? Who will do it?) Revert changed packages to their previous way of capping the number of build jobs
* Contingency deadline: beta * Blocks release? No
== Documentation == Previous discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
openSUSE implementation: https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints
devel-announce@lists.fedoraproject.org
-
Ben Cotton