Re: RFC: declaring estimated per-builder RAM usage in spec file
On 3/30/21 12:43 PM, Nico Kadel-Garcia wrote:
On Tue, Mar 30, 2021 at 4:46 AM Panu Matilainen
<pmatilai(a)redhat.com> wrote:
>
> On 3/29/21 8:17 PM, Michel Alexandre Salim wrote:
>> Hi again,
>>
>> On Fri, 2021-03-26 at 17:29 -0700, Michel Alexandre Salim wrote:
>>>
>> [snip]
>>> Right now I'm just overriding _smp_build_ncpus to 1, but there is a
>>> more elegant solution I'd like to propose:
>>>
>>> What if one can declaratively set the required RAM per build job --
>>> either with a single macro, or maybe two if the LTO usecase requires
>>> even more RAM. e.g. to declare each core might take up to 8 GB:
>>>
>>> %global _smp_build_ram_per_cpu 8192
>>>
>>> then in case this is run on our aarch64 builder with 40GB RAM,
>>> dynamically take the minimum of the existing _smp_build_ncpus (which
>>> AIUI is determined by the number of cores on the machine) and (amount
>>> of RAM / _smp_build_ram_per_cpu), in this case capping the actual
>>> number passed to -j to 5.
>>>
>>> Is there interest in having this be available? I could imagine it
>>> might
>>> be useful for other resource-intensive package builds e.g. for
>>> Chromium.
>>>
>>
>> Thanks to Dennis, Dan and Miroslav for the helpful pointers!
>>
>> Short-term I'll look into adopting something similar to the Ceph
>> script, but medium-term - maybe we can get the scriptlet into redhat-
>> rpm-config and then eventually have it in RPM itself once Panu's patch
>> is reworked?
>>
>> (Having it in redhat-rpm-config or some other RPM is probably needed
>> anyway, for older supported releases, so ideally we use the same macros
>> and only override them in redhat-rpm-config if they were undefined).
>>
>> cc:ing Panu for context on the RPM PR status.
>
> There's no progress on that front, other than occasionally thinking
> about it.
>
> It might not be a bad idea to be able to put an actual BuildRequire on
> the amount of memory (and other similar - disk space also comes to mind)
> into specs.
Let's *not*. The amount of space saved in storage by putting socks and
underwear in their own separate drawers os often overwhelmed by the
space wasted in giving those drawers sides and a way to slide in and
out. In other words, it creates unnecessary overhead in managing build
environment resources. It's also difficult to predict in advance,
especially for the worst offenders, such as builds whose upstream
components are often pulled in dynamically: "pip install", "cpan",
"mvn", "gradle", "rubygem", and my recent favorite for
unpredictable
and only erratically evailble dependencies, "go". While well designed
SRPMs and .spec files to rely only on other RPMs for their
dependencies, many in the field do not. And even when the dependencies
are available, even a small change in "test" procedures can massively
expand RAM and disk use during compilation.
In other words, the memory requirements are likely to diverge,
*massively* and unpredictably. It's safer and simpler to simply
allocate memory generously for build environments.
The point is not that everybody should add such requirements to their
packages, but to have that *option* for those monster packages out there.
- Panu -