On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com> wrote:
Hello Fedora Community!
I am a long-time Fedora Community member, and may be familiar to many
through previous FESCo or devel list discussions and passionate
debates. However I write to you today with a different community hat
on, as a lead Architect for Red Hat Enterprise Linux. The RHEL
organization has been following the modularity discussions within
Fedora, particularly around ELN, and often the question of what plans
we have for modularity in RHEL 9 has come up. Our Fedora Project Lead
and a number of FESCo members have reached out and asked if we can
provide some perspective here, and I am both happy and excited to have
Thank you for taking the opportunity to talk to us from the Red Hat
Enterprise Linux perspective. I greatly appreciate that and I hope
others respond kindly to this outreach with constructive feedback.
As the Fedora Council has pointed out , we certainly acknowledge
there are improvements to be made and have a team already working on
them. They recently outlined their plans in conjunction with our
Product Management team in a Fedora Council call as well . We’re
continuing to invest time and effort in this packaging solution and
are confident that the team can deliver against their plan. It is
somewhat of a new experience for all of us when Red Hat is direct with
our product intentions, but we discussed the larger gaps we see with
usage in RHEL and are putting our efforts towards solving those gaps
with this plan.
Modularity is important to RHEL and those efforts are already
underway. We will be leveraging modularity in RHEL 9 where it most
makes sense. This is primarily centered around our Application
Streams concept, which has been well received by our customer base.
Providing a consistent but improved experience is the base
requirement, which allows us to have continuity from RHEL 8 to RHEL 9
and lowers the hurdle for our customers when upgrading from one major
version to another.
Personally, as a user of the Application Streams stuff in my
custom-built EL containers, it's very nice, and it works well for
providing the flexibility I've needed while having a fully supportable
stack of software. It's a bit less fun on regular servers and VM
environments, but I think this can improve.
It is always good to push the boundaries and search for better ideas
and improvements, and that is part of what makes Fedora great. We are
doing this in the context of the RHEL 9 release as well, so our near
term timeline and requirements mean we are working on evolving
modularity, not a revolution or a replacement. We are excited by ELN,
as it presents a possible space to allow those that want to continue
to iterate on modules a place to do so without necessarily impacting
the broader Fedora distribution in its entirety. It is my personal
hope that we can use that opportunity to improve modules and
modularity in the open source, Fedora-first way we’d prefer. Our near
term effort to improve the existing modularity implementation ahead of
RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
than in closed product development. Longer term, we are open to
contributing to a better replacement that meets many of the same
goals. This is what makes our distribution ecosystem work well, and
not having that upstream lessens the value we all get from such
experimentation in the open.
Something that has been bothering me a bit is that there's a lot of
mixed messaging around ELN, even from Red Hatters.
Don't get me wrong: I *absolutely* want ELN to exist, and I like that
we're doing it. I *want* RHEL development happening in Fedora.
But I'm confused about the purpose of ELN. Is it intended to be the
development playground for Red Hatters? Or is it a community
initiative to support Fedora and Red Hat to come together on
developing RHEL? Or is it just a fake-RHEL built on Fedora to minimize
the burden of forking Fedora for making RHEL later? I had personally
hoped that ELN would be an opportunity to allow the Fedora community
and Red Hat to work together on building the future RHEL more
directly, but I am unsure of what it does or what it is for now.
As for iterating on modularity in Fedora through ELN, I think this is
a good idea. I want to see the implementation of modularity fleshed
out and the packager experience improved, and I think the only way to
do that is to actually use it and inflict all the pain on the people
who need it by not permitting weird hacky workarounds to make module
builds work. If something is broken, the standard code path has to be
fixed, and that's how I expect this will work. I'm already aware that
CentOS rebuilds of RHEL are not necessarily straightforward because
many hacky shortcuts are taken to build RHEL content and CentOS does
not have those in place.
However, I am concerned that as ELN develops further, we are likely to
be even more starved for build resources than we have been previously.
Modules are huge build chains that essentially fill up the builders.
Outside of the improved AArch64 hardware, I'm personally unaware of
any improvements in our build capacity to help support the higher
demands for the build system. To note, we'd have this problem without
modules if we had Koschei configured to auto-rebuild and submit
rebuilds on dependency drift so that packagers didn't have to do that
grunt work manually, so it's a matter of we literally do not have
enough resources to support more automation. I've mentioned this
before in other threads, but to reiterate: it is my belief that Fedora
does not have enough build capacity to support building a modularized
distribution. Even when we were doing modularized builds in the Fedora
Rust SIG, it was common for module build jobs to stall out waiting for
resources, and thus get stuck midway through. This also starved
regular builds of resources to get things done.
Hopefully that provides some context and helps FESCo and the wider
community understand where Red Hat is headed with modularity on the
It absolutely does, and I hope you continue to engage with us on this!
Let's make everything better together!
真実はいつも一つ！/ Always, there's only one truth!