On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok <mhroncok@redhat.com> wrote:
To submit more scratch builds we would need larger builder capacity.
This doesn't necessarily mean more or better hardware.
Better Koji configuration would help a lot.
We have some very powerful builders with up to 224 processors, but
their capacity is set to 2.
This means that the builder stops accepting new tasks once load gets
to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org
Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org
224 CPUs, load average: 2.04, 2.07, 2.05
memory: 251Gi total, 7.3Gi used, 242Gi available
Yet, at the time of writing the builder is marked as not ready (!!)
for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
I believe there are 2 builders with 224 CPU's. They are both 'prototype' aarch64 systems we got on loan from a vendor and are tempermental. They do not have good disk IO and do not have the capability of increasing the disk IO. This means that you have a very very fast cpu and slow as treacle disk io. The network on them has also been 'fun' where if the net access got too high, it would require a complete power cycle with the plugs pulled out because the BMC is not licensed.
Most of the 'big' builders I remember are similar in hardware.. They look great on paper, but if you try to give them a large capacity you end up with even slower builds because they are loaners or spares which are good for specific workloads and not general usage.