On Thu, Jun 18, 2020 at 12:30:58PM -0400, Josh Boyer wrote:
On Thu, Jun 18, 2020 at 9:26 AM Neal Gompa <ngompa13(a)gmail.com>
> However, I am concerned that as ELN develops further, we are
> be even more starved for build resources than we have been previously.
> Modules are huge build chains that essentially fill up the builders.
It's a concern, but IMHO not a major one.
ELN builds run at a higher priority than normal builds.
Module are currently the same prirority as normal builds, but we could
adjust them. Or adjust just the ELN ones.
There's a number of things we could do if it becomes a problem.
> Outside of the improved AArch64 hardware, I'm personally
> any improvements in our build capacity to help support the higher
> demands for the build system. To note, we'd have this problem without
There are, and thanks for a place to note them:
The x86_64 buildvm's now in the new datacenter are on newer hardware
thats likely much faster than the old ones.
The new datacenter has a new netapp cluster (which we aren't sharing
with other workloads).
Mainframe admins adjusted things on the s390x z/vm builders to be faster
and indeed they seem much improved.
The new datacenter we have postgresql 12.2 for the koji database.
Our last mass rebuild finished in a few days. How much faster do we need
to be? ;)
> modules if we had Koschei configured to auto-rebuild and submit
> rebuilds on dependency drift so that packagers didn't have to do that
> grunt work manually, so it's a matter of we literally do not have
> enough resources to support more automation. I've mentioned this
> before in other threads, but to reiterate: it is my belief that Fedora
> does not have enough build capacity to support building a modularized
> distribution. Even when we were doing modularized builds in the Fedora
> Rust SIG, it was common for module build jobs to stall out waiting for
> resources, and thus get stuck midway through. This also starved
> regular builds of resources to get things done.
My understanding is that this was discussed with Fedora
infrastructure. I have no insight into that further. We probably
It was, and I don't think it's going to be that much of a problem.
I could of course be wrong (I have been before and will be again!)
aren't going to get additional builders for ELN or Fedora in
anytime soon, so allocation of existing resources is probably
something to consider. I'll offer that there are actually more Fedora
build hosts than there are RHEL 8 build hosts, by comparison.
Of course Fedora builds a ton more things than RHEL... but yeah.
Personally, I have long wanted burst-to-cloud or the ability for
others to donate hosts to the Fedora build system without having to
physically ship hardware. Koji is somewhat limited in that regard.
Maybe developing a shim layer and some security best practices to
allow that would help.
Yeah, there have been several projects to allow this to happen, and if
pressed for resources we could look into them again.
> > Hopefully that provides some context and helps FESCo and the wider
> > community understand where Red Hat is headed with modularity on the
> > Enterprise side.
> It absolutely does, and I hope you continue to engage with us on this!
> Let's make everything better together!
That's the idea :)
100% agreed. :)