On Thu, Jun 18, 2020 at 9:26 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com> wrote:
> Hello Fedora Community!
> I am a long-time Fedora Community member, and may be familiar to many
> through previous FESCo or devel list discussions and passionate
> debates. However I write to you today with a different community hat
> on, as a lead Architect for Red Hat Enterprise Linux. The RHEL
> organization has been following the modularity discussions within
> Fedora, particularly around ELN, and often the question of what plans
> we have for modularity in RHEL 9 has come up. Our Fedora Project Lead
> and a number of FESCo members have reached out and asked if we can
> provide some perspective here, and I am both happy and excited to have
> that opportunity.
Thank you for taking the opportunity to talk to us from the Red Hat
Enterprise Linux perspective. I greatly appreciate that and I hope
others respond kindly to this outreach with constructive feedback.
> As the Fedora Council has pointed out , we certainly acknowledge
> there are improvements to be made and have a team already working on
> them. They recently outlined their plans in conjunction with our
> Product Management team in a Fedora Council call as well . We’re
> continuing to invest time and effort in this packaging solution and
> are confident that the team can deliver against their plan. It is
> somewhat of a new experience for all of us when Red Hat is direct with
> our product intentions, but we discussed the larger gaps we see with
> usage in RHEL and are putting our efforts towards solving those gaps
> with this plan.
> Modularity is important to RHEL and those efforts are already
> underway. We will be leveraging modularity in RHEL 9 where it most
> makes sense. This is primarily centered around our Application
> Streams concept, which has been well received by our customer base.
> Providing a consistent but improved experience is the base
> requirement, which allows us to have continuity from RHEL 8 to RHEL 9
> and lowers the hurdle for our customers when upgrading from one major
> version to another.
Personally, as a user of the Application Streams stuff in my
custom-built EL containers, it's very nice, and it works well for
providing the flexibility I've needed while having a fully supportable
stack of software. It's a bit less fun on regular servers and VM
environments, but I think this can improve.
> It is always good to push the boundaries and search for better ideas
> and improvements, and that is part of what makes Fedora great. We are
> doing this in the context of the RHEL 9 release as well, so our near
> term timeline and requirements mean we are working on evolving
> modularity, not a revolution or a replacement. We are excited by ELN,
> as it presents a possible space to allow those that want to continue
> to iterate on modules a place to do so without necessarily impacting
> the broader Fedora distribution in its entirety. It is my personal
> hope that we can use that opportunity to improve modules and
> modularity in the open source, Fedora-first way we’d prefer. Our near
> term effort to improve the existing modularity implementation ahead of
> RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
> than in closed product development. Longer term, we are open to
> contributing to a better replacement that meets many of the same
> goals. This is what makes our distribution ecosystem work well, and
> not having that upstream lessens the value we all get from such
> experimentation in the open.
Something that has been bothering me a bit is that there's a lot of
mixed messaging around ELN, even from Red Hatters.
Indeed. It's new :)
Don't get me wrong: I *absolutely* want ELN to exist, and I like
we're doing it. I *want* RHEL development happening in Fedora.
But I'm confused about the purpose of ELN. Is it intended to be the
development playground for Red Hatters? Or is it a community
initiative to support Fedora and Red Hat to come together on
developing RHEL? Or is it just a fake-RHEL built on Fedora to minimize
the burden of forking Fedora for making RHEL later? I had personally
hoped that ELN would be an opportunity to allow the Fedora community
and Red Hat to work together on building the future RHEL more
directly, but I am unsure of what it does or what it is for now.
I can tell you my opinion, but because ELN is a new concept/idea it is
only my interpretation.
I see ELN as an opportunity born from the Fedora community to
experiment with the distribution aimed at a different userbase than
what we'd traditionally see with typical Fedora usage. It is not that
different from Editions in many ways, except Editions were crafted as
a focusing mechanism leveraging the same Fedora packages and defaults.
ELN allows experimentation with different defaults or baselines.
Matthew has talked about a Playground concept before, and ELN
incorporates some of those ideas as well.
Clear as mud, yes? New ideas often make things murky, particularly as
they are being worked through. I see this as a natural occurrence in
With my official Red Hat hat on, I do *not* see ELN as a fake-RHEL to
minimize forking Fedora. If we wanted to do that, we could. ELN has
the potential to become a development space for major RHEL releases,
but that potential only exists as much as we let it. There are many
times where Enterprise and non-Enterprise requirements are in direct
conflict, and ELN could be the place to iterate on them. If people
aren't comfortable developing and experimenting with that deviation in
ELN, then it makes developing RHEL in Fedora harder but it doesn't
make ELN useless or remove options for open RHEL development entirely.
It's an opportunity. It's up to both sides to use it.
As for iterating on modularity in Fedora through ELN, I think this
a good idea. I want to see the implementation of modularity fleshed
out and the packager experience improved, and I think the only way to
do that is to actually use it and inflict all the pain on the people
who need it by not permitting weird hacky workarounds to make module
builds work. If something is broken, the standard code path has to be
fixed, and that's how I expect this will work. I'm already aware that
CentOS rebuilds of RHEL are not necessarily straightforward because
many hacky shortcuts are taken to build RHEL content and CentOS does
not have those in place.
I would say that we shouldn't conflate building a distribution with
modules included and *rebuilding* a distribution with modules
included. CentOS rebuilds are probably not something I would consider
in the context of Fedora, ELN, and modules. Probably best to discuss
that on the CentOS lists.
Put another way, what you phrase as hacky shortcuts and struggles are
not the same issues we'd necessarily see when trying to build a
distribution that way for the first time. There is certainly an
overlap of issues, but we should constrain ourselves a bit.
However, I am concerned that as ELN develops further, we are likely
be even more starved for build resources than we have been previously.
Modules are huge build chains that essentially fill up the builders.
Outside of the improved AArch64 hardware, I'm personally unaware of
any improvements in our build capacity to help support the higher
demands for the build system. To note, we'd have this problem without
modules if we had Koschei configured to auto-rebuild and submit
rebuilds on dependency drift so that packagers didn't have to do that
grunt work manually, so it's a matter of we literally do not have
enough resources to support more automation. I've mentioned this
before in other threads, but to reiterate: it is my belief that Fedora
does not have enough build capacity to support building a modularized
distribution. Even when we were doing modularized builds in the Fedora
Rust SIG, it was common for module build jobs to stall out waiting for
resources, and thus get stuck midway through. This also starved
regular builds of resources to get things done.
My understanding is that this was discussed with Fedora
infrastructure. I have no insight into that further. We probably
aren't going to get additional builders for ELN or Fedora in general
anytime soon, so allocation of existing resources is probably
something to consider. I'll offer that there are actually more Fedora
build hosts than there are RHEL 8 build hosts, by comparison.
Personally, I have long wanted burst-to-cloud or the ability for
others to donate hosts to the Fedora build system without having to
physically ship hardware. Koji is somewhat limited in that regard.
Maybe developing a shim layer and some security best practices to
allow that would help.
> Hopefully that provides some context and helps FESCo and the
> community understand where Red Hat is headed with modularity on the
> Enterprise side.
It absolutely does, and I hope you continue to engage with us on this!
Let's make everything better together!
That's the idea :)