A Big Idea for a New Decade [was: Minutes for CentOS Board of Directors 2019-12-18 Meeting]
On Fri, Jan 03, 2020 at 03:13:53PM -0500, Karsten Wade wrote:
As a centralizing effort, the Board agreed to revisiting the goals document created five years ago, and to undergo an effort to refresh those goals in the light of the project’s evolution. The Board will be inviting various stakeholders into these discussions as we undergo a public revision of the goals at the start of 2020.
Wow, five years goes fast! I'd like to raise an idea for discussion and thought, and possible further discussion between the Fedora Council and the CentOS board. [I'm posting this in both groups.]
Around the same time the CentOS goals were being created, Fedora was working on plans for "Fedora.next", which included among other things Fedora Server. I won't rehash all that here, but in short: while that succeeded at some of its goals, it never really had the user adoption we'd hoped for. Of course, some of that is simply a matter of lifecycle: while a fast-moving server OS is useful in some very real cases, it's decidedly niche.
When Fedora launched that plan, CentOS wasn't really in the "family", and after, we just kind of existed in parallel, with some very loose collaboration in areas like EPEL.
Of course, things are different now. Therefore, I propose that in this next decade, we make that collaboration stronger. Together, we have a very nice answer to the fundamental problem of unified operating systems. That is, some parts move too fast and other parts move too slowly. Between Fedora and CentOS, we have answers to both! And, with the in-progress dist-git merge, we even have both _in the same place_.
Specifically, I'd like to suggest three things.
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so. Let's simplify things!
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
And finally, let's really take advantage of a merged dist-git repository, and the capabilities that Modularity gives us. SIGs building things for CentOS should be able to directly reference Fedora branches where faster-moving software is useful — and people building things in Fedora should be able to pull in branches from Stream where a slower-moving version would make sense. And we could even make additional shared branches where necessary, using modularity to make optional streams which could be used in CentOS SIGs, EPEL, or a Fedora OS.
What do you think?
I'll admit that I've never quite understood the market for Fedora Server (I am running Fedora on a server at home, but it's really a KDE desktop masquerading as a server). I am definitely in favor of working toward having CentOS Stream stand in for Fedora Server. I think it better fits the use case we're targeting and reduces some of the confusion.
For the other points, I agree in general, but I wonder how it would work in practice. If the CentOS Llama SIG and the Fedora Llama SIG merge, they'll still have to follow Fedora's rules for their Fedora output and CentOS's rules for their CentOS output (assuming they decide it makes sense to have output in both places. If they say "llama herding is too fast-moving to need a long-term OS", for example, they can ditch the CentOS output and then it doesn't matter.) This shouldn't be taken as me being opposed, because I do think we should take advantage of the synergy where it exists, but I also wonder how much of it actually exists beyond the basic infrastructure level.
On Mon, Jan 6, 2020 at 4:13 PM Matthew Miller mattdm@mattdm.org wrote:
On Fri, Jan 03, 2020 at 03:13:53PM -0500, Karsten Wade wrote:
As a centralizing effort, the Board agreed to revisiting the goals document created five years ago, and to undergo an effort to refresh those goals in the light of the project’s evolution. The Board will be inviting various stakeholders into these discussions as we undergo a public revision of the goals at the start of 2020.
Wow, five years goes fast! I'd like to raise an idea for discussion and thought, and possible further discussion between the Fedora Council and the CentOS board. [I'm posting this in both groups.]
Around the same time the CentOS goals were being created, Fedora was working on plans for "Fedora.next", which included among other things Fedora Server. I won't rehash all that here, but in short: while that succeeded at some of its goals, it never really had the user adoption we'd hoped for. Of course, some of that is simply a matter of lifecycle: while a fast-moving server OS is useful in some very real cases, it's decidedly niche.
When Fedora launched that plan, CentOS wasn't really in the "family", and after, we just kind of existed in parallel, with some very loose collaboration in areas like EPEL.
Of course, things are different now. Therefore, I propose that in this next decade, we make that collaboration stronger. Together, we have a very nice answer to the fundamental problem of unified operating systems. That is, some parts move too fast and other parts move too slowly. Between Fedora and CentOS, we have answers to both! And, with the in-progress dist-git merge, we even have both _in the same place_.
Specifically, I'd like to suggest three things.
First, let's replace Fedora Server as a user-focused artifact with CentOS Stream. There's still a need for a Fedora Server as an upstream, but most users of Fedora as a server are making custom builds (or using the basic Cloud image), not using Fedora Server per se. So, I think we should stop marketing that to users, and steer people with server use cases outside of the fast-moving niche to CentOS Stream. The current story of CentOS and Fedora is pretty confusing to users, and Stream only makes it more so. Let's simplify things!
Second, where there's overlap, let's look at merging Fedora and CentOS SIGs into joint bodies. We have groups like the cloud image SIGs which are basically doing the exact same thing in two different ways. Same thing with containers. We also have huge overlaps in documentation and community user support and all sorts of non-technical project areas. I'm not saying these all need to be smooshed together, but where it makes sense let's work together. This, too, brings simplification to the "what's with CentOS vs. Fedora" story: rather than making contributors pick, we can welcome all with open arms.
And finally, let's really take advantage of a merged dist-git repository, and the capabilities that Modularity gives us. SIGs building things for CentOS should be able to directly reference Fedora branches where faster-moving software is useful — and people building things in Fedora should be able to pull in branches from Stream where a slower-moving version would make sense. And we could even make additional shared branches where necessary, using modularity to make optional streams which could be used in CentOS SIGs, EPEL, or a Fedora OS.
What do you think?
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
1. CentOS Koji is pretty closed to a couple of folks in CentOS and Red Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing to users. The CBS is a worthless substitute since it's purely for addons. I've been around long enough to remember when Fedora was like this, and back then, it wasn't really considered "okay" for Fedora to be like that (the intent was always there to fix it, and we eventually did!). Today, however, it is considered "okay", and that's a problem.
2. The CentOS community is simply not mature enough to take on the role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes and does not give back much, if at all. I've experienced variations of this issue when people (even from Red Hat!) have done finger-pointing to *me* with my packages in EPEL without any acknowledgement that they could have helped. It's depressing that even some Red Hatters consider EPEL not worth contributing to, but still worth being upset about.
3. Today, the CentOS Project is (sadly) a community project in a very superficial sense. Aside from the Virt SIG's Xen subgroup, all SIGs are basically Red Hat product teams, with varying degrees of allowances for community contributions. The community processes within CentOS are mostly nonfunctional because they've had no reason to develop it. There has been very little community development for CentOS. Even with Fermilab and CERN shutting down Scientific Linux, I haven't seen anything from those people coming "into the fold". There's been no outreach or collaboration with other CentOS based derivatives, commercial or otherwise. The private discussions I've had on the matter seem to indicate that there's still ambivalence, if not outright antipathy and hostility toward the broader ecosystem of CentOS derivatives.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
On Tue, Jan 7, 2020 at 7:11 AM Neal Gompa ngompa13@gmail.com wrote: ...
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I don't think Matthew suggested that we'd stop development of server technologies in Fedora. I think he was saying that we'd drop the "Fedora Server Edition" as a user-targeted deliverable. This is an important distinction. We'd instead focus on server technology development atop the container and Fedora Cloud Edition use-cases.
On Tue, Jan 7, 2020 at 8:48 AM Stephen Gallagher sgallagh@redhat.com wrote:
On Tue, Jan 7, 2020 at 7:11 AM Neal Gompa ngompa13@gmail.com wrote: ...
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities for people to iterate on server-oriented changes before they go into rhel-rawhide and push into CentOS Stream. You also essentially kneecap any motivation for other things related to server environments to iterate faster (such as language stacks that are heavily used for web service software) because you've eliminated the major ability for that to ship to users and contributors. It also further accelerates a trend that I think we need to reverse where people consider Fedora unacceptable for server roles. If anything, Fedora is a lot better at being used for servers then it was five years ago. I've personally *stopped* using CentOS for servers because Fedora has gotten so good at it. Upgrades are a breeze and stuff generally works. When it doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I don't think Matthew suggested that we'd stop development of server technologies in Fedora. I think he was saying that we'd drop the "Fedora Server Edition" as a user-targeted deliverable. This is an important distinction. We'd instead focus on server technology development atop the container and Fedora Cloud Edition use-cases.
Today's Cloud Edition is mostly a further stripped down Server Edition variant with different words in the branding. It's our deliverable for clouds like AWS and cloud-like VPSes like DigitalOcean.
However, Fedora Server Edition is what people use to run their *own* servers with Fedora, and that also includes things like VMs in more traditional hypervisor setups (oVirt/RHV, VMware, XCP-ng/XenServer, etc.).
The key difference between Server and Cloud Edition is the usage of Anaconda to install Fedora. I'm pretty sure we still want to be continually validating *that* works, especially since it's a different codepath in Anaconda from the desktop flavors.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Tue, Jan 07, 2020 at 08:59:17AM -0500, Neal Gompa wrote:
However, Fedora Server Edition is what people use to run their *own* servers with Fedora, and that also includes things like VMs in more traditional hypervisor setups (oVirt/RHV, VMware, XCP-ng/XenServer, etc.).
That's certainly not universal. People *do* use the cloud images for deployment to on-prem virtualization setups. They're generally faster to deploy than full Anaconda installs.
Also, in oVirt/RHV, the preconfigured images (the ovirt-image-repository Glance provider) are based on the generic cloud images produced by Fedora and CentOS.
On Tue, Jan 07, 2020 at 07:09:57AM -0500, Neal Gompa wrote:
What do you think?
First, sorry that this email probably doesn't thread will with either one or the other target mailing lists... I had to pick one to thread with, and other slightly loses. The downside of the initial email not being sent to both MLs at once. :(
Yeah, I decided to send the message to both lists roughly simultaneously rather than cross-post because I figured only a few of us are subscribed to both. I see you went the other way, so we'll see how that goes in this subthread. :)
In concept, I certainly see value in bringing CentOS and Fedora closer. However, I think eliminating Fedora variants in favor of CentOS Stream ones (aka rhel-rawhide based variants) is probably premature. There are several issues with doing this as it currently stands:
To be clear, I don't think this should be done in a rush. I hope to convince poeple that it's a good idea in principle and then work to get there.
- CentOS Koji is pretty closed to a couple of folks in CentOS and Red
Hat. This also includes blocking the download of artifacts from Koji. For reasons I don't understand, CentOS is still gated by the RHEL QE folks, and thus RPMs built in Koji cannot be downloaded until they are finally published. This ruins any testing/development before releasing
I think this is a temporary state while opening up previously-internal development to CentOS Stream. There are plenty of security and partner-privacy related issues that need to be solved and I don't blame Red Hat (or CentOS!) for taking that slowly.
- The CentOS community is simply not mature enough to take on the
role Fedora does in any meaningful capacity. We already know that this is a problem even with Fedora EPEL, and I strongly believe the issue would be massively magnified if those transitioned to the CentOS Project. The CentOS community is largely a user community that takes
This is an area where both communities can learn from our separate experiences, expertise, and strengths. Well, this and also your point 3 below.
What I'm more concerned about is that if you eliminate Fedora from any meaningful server based development, you strip all the opportunities
Wait wait wait -- I'm not suggesting that. We *definitely* still need a voice for server development in Fedora. I'm just not sure a user-focused Edition is the right thing for it. Although I agree that Fedora has advanced leaps and bounds in terms of being suitable for server use in the Fedora.next era (and therefore a better upstream for RHEL and CentOS!), none of the user-focused feature ideas of Fedora Server (like Roles) really stuck.
doesn't, it's fixable! That last part is key. With CentOS, it's not, because it has to bounce back into Red Hat first. And Red Hat doesn't really care about issues discovered by CentOS users, and there are no "CentOS developers".
CentOS Stream is about changing that -- the caring about issues discovered by users, at least, and possibly to some degree development, although not "major release" development.
So, what this long email is actually saying is that I think it's an interesting idea to bring the two projects together, but eliminating aspects of Fedora in favor of CentOS is premature because CentOS has not actually developed as a community project. Maybe it's worth revisiting after six years of actual community development?
I guess I'd like to get us working together sooner rather than trying to reconcile after another six months or a year, where decisions made for perfectly good reasons could end up making things harder later.
council-discuss@lists.fedoraproject.org
-
Ben Cotton -
Lars Seipel -
Matthew Miller -
Matthew Miller -
Neal Gompa -
Stephen Gallagher