RFC: Proposal for a more agile "Fedora.next" (draft of my Flock talk)
Phil Knirsch
pknirsch at redhat.com
Wed Jul 24 13:45:45 UTC 2013
On 07/22/2013 03:38 PM, Matthew Miller wrote:
> This is a draft of the proposal I'm presenting at Flock, "An Architecture
> for a More Agile Fedora" (<http://sched.co/19ugKGM>). It represents a big
> change to how we as the Fedora project put together our distribution. I've
> gone through several drafts of this and talked to a few people, and I'd
> really value all of your feedback to refine it further.
>
> Preliminary slides are available at <http://mattdm.org/fedora/next/>; I will
> update them as this discussion progresses. (So, page numbers might change.)
> What follows is an annotated text version of the presentation — the
> annotations basically being what I might say as each slide displays.
>
> If you have a short attention span, you might want flip through the slides
> first and come back for explanation where it's unclear. And the explanation
> here might be unclear too — please ask questions.
>
>
> An Architecture for a More Agile Fedora
> presented by Matthew Miller ( ← with input from many people)
> ===
>
> Problem Statement (What's this all about?)
> ---
>
> Let's Start with What's Good
> ---
> * Fedora is awesome
> * Great tech, great people
> * Amazing, high standards
> * Best enthusiast distro
> * Solid base for RHEL
> * Pretty decent cloud image
>
> I've been involved to some small degree with Fedora since the beginning.
> This is a project and a Linux distribution which I love, and which I know
> everyone reading this also cares about deeply. We don't want to break what
> is successful and works.
>
>
> But...
> ---
> * We need to be more than that
> * Not widely used by RHEL users
> * Not so great for building on...
> * Including Red Hat's own stuff
> * We're not seen as relevant...
> * Let alone exciting
>
> Whenever I go to a tech meetup or talk to someone from a new startup
> company, their developers are inevitably using a different (usually
> proprietary) desktop OS, plus a non-Fedora distribution on their code. We're
> being left behind and left out. It doesn't matter how theoretically great we
> are if we end up with no users.
>
>
> The Idea
> ---
> Focus core distro as platform, include layers of modular enabling
> technologies, and provide room for special interest groups to create
> solutions within the Fedora circle.
>
>
> Starting with Ring One
> ---
> * We draw a line around a small "base design"
> * E.g.: Everything in @standard, plus the toolchains to build it
> * Line could go elsewhere; that's an implementation detail
> * Most strict change management
> * Packaging standards as current Fedora
>
> The example line is somewhat arbitrary here — I'll talk a little more about
> practicalities at the end of the talk. The idea of using @standard as a
> starting place is just that. That's the group that's one step up from a
> minimal install; common things usually installed. That collection of
> packages is arbitrary and has grown arbitrarily, but it makes a place to
> begin. We'll probably end up with a set of packages that isn't exactly tied
> to any existing group.
>
> Anyway, this is the core OS. It's bigger than a minimal install, and is a
> complete but lightweight Linux distribution. It has a very clear charter:
> use the latest in open source platform-level software integrated into a
> solid foundation for the higher rings.
>
>
> The Base OS
> ---
> * Ring One is base functionality and behavior that everyone can expect of any Fedora system
> * Making a distinction helps focus engineering resources
> * And it makes a foundation that higher levels know they can trust
>
>
> Ring 1 : A Picture
> ---
>
> *********
> ** **
> ** **
> * Ring 1 *
> * *
> * Fedora Core *
> ** **
> ** **
> *********
>
>
> Wait, did you just say Fedora _Core_?
>
> Yes I did.
>
> But, this is not a return to the old Core + Extras, because the line is
> drawn based on _what_ and _how_ rather than on who works for what company.
>
> There were two big problems with the long-ago approach.
>
> First, the "core" was developed internally and not as a community open
> source project. If a package was in that part of the distribution, there was
> no real route for participation. Bad.
>
> Second, the quality standards for Fedora Extras, the collection of packages
> built around the core, were much, much higher. Upside-down!
>
> So, we won't repeat those mistakes. The new "Fedora Core" is a community
> project just like Fedora today. And we focus the most strict standards on
> the inner rings.
>
>
> Ring Zero: Fedora Minimal
> ---
> * Start from the current @core, and aim to shrink
> * Intentional focus on minimalism
> * JEOS — Just Enough Operating System
> * Even stronger rules than Ring 1
> * Not self-hosting
>
> This is pretty self-explanatory. There is a lot of demand and many use cases
> for a "Just Enough Fedora" operating system. Ring 0 exists to provide that
> in a serious way which Fedora has never really covered before.
>
>
> Rings 1 + 0: Together
> ---
>
> (There is a diagram here, but I'm leaving it out of the text. It's
> a circle. With another circle.)
>
>
> Part Two... First, _Why?_
> ---
> * Back to the problem statement: we need to be useful
> * People building on Fedora need a reliable base...
> * ... but need more flexibility in the areas relevant to their code
> * So let's design something that gives that
>
>
> ***DON'T PANIC***
>
> This is a vision for the future. The Fedora Core idea can get started now,
> and the ideas beyond that are for development over a longer term. That
> doesn't mean never, and there are some specific things at the higher level
> to get started on now, but I'm not suggesting to change everything all
> crazily. We're not throwing out what have. The idea is to explore several
> different approaches which will better position Fedora as the distro of
> choice for future development.
>
> So... Rings 2 + 1 + 0
> ---
>
> (There is another diagram here. It's more circles inside of circles.)
>
>
> Ring Two: Environments and Stacks
> ---
> * Environments are where you run the code you care about
> * Stacks are modular collections of _software which is used by other
> software_
>
> In one of the earlier drafts, I had these as individual, separate rings.
> That turns out to be kind of distracting, because depending on your point
> of view either could easily be seen as inside or outside the other. So, I
> put them together in this level.
>
> Environments
> ---
> * Desktop environments, ideally with app containers
> * Platform as a Service, decoupled: integrated OpenShift Gears
> * Core Image (oVirt node, cloud guest)
>
> Stacks
> ---
> * Languages, databases, libraries
> * Maybe X/Wayland
> * The kind of things which could be OpenShift Cartridges or Software
> Collections
>
>
> (For Example)
> ---
>
> And there's another diagram here. In the Ring 2 circle, we have examples
> like Perl; Ruby 1.8.x, 1.9.x, 2.0; Python 2 & 3; Java, Wildfly; some
> databases, desktop environments, OpenShift, and then something called Fedora
> Commons, which I'll talk about in a bit.
>
>
> What's a Ring?
> ---
> * Rings give SIGs a way to replace the default expectations with their own
> * Separate policies may apply to each ring
> * But also, also, we need community, policies, and infrastructure to make
> them _part of Fedora_
>
> So, there's nothing really magical about the circles. The main idea is that
> we need to make room for more exploration within Fedora beyond the
> traditional idea of how a Linux distribution is put together, and this is a
> model for describing how and where that can work.
>
> We still need global Fedora policies, but they'll apply differently to each
> ring. In Ring 2, SIGs will be given a lot of room to develop their own
> rules, but in Ring 1, not so much. Different environments and stacks within
> Ring 2 may have different individual policies from each other, but there
> will also be general policies for Ring 2 which are different from those for
> Ring 1, and of course global policies that apply to the whole thing (e.g.,
> no proprietary code).
>
> Possible Examples
> ---
> * Possible example: bundled libs okay, but must be documented
> * Can override versions of software at lower tier
> * Can overlap with software from other stacks
> * Not necessarily even RPM
> * Different lifecycle (but shared release cadence)
>
> Big stuff here.
>
> Obviously, no-bundled-libs is a crucial part of the packaging guidelines
> today. As a sysadmin, I know why it's important. This is not just a noble
> goal, but also something that pragmatically makes systems better. But, it's
> also keeping us from having software that people really use in Fedora. Chef
> and Hadoop are two big examples. This hurts us more than it helps the world.
> So, in some areas, we need a different approach.
>
> Overriding other parts of the distribution is crucial, and I'll talk a
> little bit more about different approaches and what they mean in a little
> bit.
>
> Some approaches may even move beyond RPM packaging. This can mean focusing
> on "language native" package formats like Jars, Gems, and Eggs; or it can
> mean moving to a Git-based distribution model for some parts of the distro.
>
> And, different lifecycle: a SIG may decide that Ruby 1.9.x support should
> continue for some number of years and maintain that stack against different
> Fedora Core releases. We do this already in some ways, but having a clear
> split makes it feel more natural and will make developers feel more
> confident.
>
>
> SIG-centric
> ---
> * SIGs will form "bubbles" within this ring, possibly varying from packaging
> guidelines in order to meet needs.
> * Change management will also be SIG-focused.
>
> Fedora already largely works this way. We just need more of it. If we can be
> more flexible about what it means to participate in Fedora, we can increase
> the intersection with upstream communities.
>
>
> Fedora Commons
> ---
> * The collection of packages outside Core following traditional policies
> and practices
> * We're not throwing that away
> * Many people continue to be interested in this model
> * Shared by other parts of Ring 2 where possible
>
> This is a special bubble within Ring 2. In fact, on Day 0, it still is _all
> of_ Ring 2. The name here is mean to evoke Creative Commons; we can discuss
> other possibilities too, of course.
>
>
> Many Ways!
> ---
> * Software Collections
> * Stacks 2.0
> * OpenShift Cartridges
> * Fedora Formulas
> * Traditional Packaging
>
> There are many different approaches people are working on right now. I don't
> know which will ultimately be best. We want to enable the experimentation,
> so we can find what succeeds. Some of these things aren't beautiful, but if
> we can invite them in anyway, they can iterate towards perfection.
>
> Exploring Different Approaches
> ---
> * Packaged vs. DVCS
> * Builds on Core vs. Overrides Core
> * Replaces vs. Coexists
> * Appliance vs. Shared
>
> It may be useful as we think about the different approaches to think about
> the different philosophies they take. For example, there are two different
> ways to "override" something. One is to actually replace it — for example,
> your Fedora Formula may replace /usr/bin/python with Python 3. The other is
> to install multiple versions in parallel — using multi-versioned RPMs, or
> Software Collections for whole stacks. In the future, a namespace approach
> could even make /usr/bin/python be Python 3 for some processes but not for
> the core OS.
>
> Additionally, there's a natural distinction between things which build on
> Fedora Core and may override Fedora Commons and those things which may
> override both. As we explore these different approaches, it may be useful to
> be intentional about which approach each SIG is using.
>
> Ring Three: Applications
> ---
> * User-level installable
> * Shift focus away from RPMs
> * Linux Apps... or Git with OpenShift
>
> And, finally, the outside ring. This is end-user software. Exactly how this
> will work is kind of further off, but it's important to think about in the
> complete picture. We want to make this a nicer experience for Fedora users,
> and we want to make it easier for application developers to offer their
> stuff to Fedora.
>
>
> Part Three: Concrete Steps
> ---
> * What do we do to make this happen?
> * What can be done now?
> * What needs to be developed further?
>
> To make this not just all talk....
>
> Three Next Things
> ---
> * Draw the "base design" for Fedora Core
> * Make space for exploring Ring 2 innovation
> * Emphasize upstream relationships (example: RubyGems.org)
>
> Much of this I hope to have further developed in the next few weeks before
> Flock. I'll talk a little bit about these three now, though.
>
> First, I'm working on creating an example spin which would show a basic
> suggestion for Fedora Core, both in a default install and an
> everything-installed (including build deps) configuration. Right now, if we
> take @standard + @core, that gives 430 packages from 330 SRPMs... but
> recursively following the build deps explodes that to 3915 packages from
> 1800 SRPMs. That's smaller than the 13.7K SRPMs in all of Fedora, but bigger
> than would be ideal for the Core. And it's probably not really the package
> set we want. So, I'd like to look at that further, and I'll post more about
> that here as I do.
>
> Second, we can make some changes to allow Software Collections to be built
> for Fedora. There's more we can do with OpenShift. And we can build up the
> Fedora Formulas idea. Basically, let's do all that.
>
> And third, by increasing our engagement upstream, we can reduce our own
> work. For example, right now RubyGems.org doesn't do any validation of
> licenses, basic review for malware, or gem signing. If we knew that this
> basic diligence was happening upstream, we could extend our circle of trust.
> We've long had the mantra of "upstream! upstream! upstream!" for code and
> patches — we can do the same thing for packaging, for the same reasons and
> for similar benefits. (But to do that, we need to work with upstream
> packaging formats rather than demanding RPM — because experience shows that
> that doesn't work.)
>
> Conclusion
> ---
>
> * Refocus Core to provide a better platform for building on
> * Make room for innovation at the "Ring 2" level
> * Empower SIGs to create solutions that fit
> * Won't break what we have
> * And we can start right now
>
> So there we have it. Comments and discussion, please!
>
Hi Matthew.
First of all, let me thank you for doing the first step toward
initiating a very lively discussion about the future of Fedora and the
direction we as a community decide to take it towards.
A lot of great points have been raised throughout the discussion
already, and different opinions on whether your proposal in general or
in specifics is a good idea or not and how people feel about them and
offered suggestions or supplemental ideas or alternatives to your ideas.
Now i don't want to make a long post myself, but a few things i'd like
to quickly mention:
- I personally really like the direction you're presenting here. It is
definitely a clear deviation from what we've done in the past, but my
believe is that this is the right approach towards it. Integrating 1000
packages 10 years ago was task that was manageable. Nowadays we're
beyond 13000 packages, and the only real difference between packages of
the different aspects of our distribution is the critical path packages
we define via the release requirement spins.
- It works really well with something i wanted to initiate more
thoroughly over the next few years, which is to attract more and a much
diverse set of developers again to Fedora. Looking at how high the bar
was set for package approval (with good reason, this is NOT intended to
bash against the FPC) to Fedora i do understand why many developers felt
Fedora to be a less and less easy to use OS to develop on and contribute
to. Offering other paths via the Copr build system or the developer
assistant is a first step, but there is much more we'll need to do to
become developer friendly again. Especially for developers that don't
live in the RPM world as well, which Fedora never was good at even from
the beginning (and again due to a philosophy that Fedora had and has as
an underpinning of the whole OS).
- As quite a few have already mentioned and you are most certainly just
as aware of yourself is that this will disrupt the way we work and
handle Fedora and it won't be a "cure all problems in Fedora, bring
world peace and end hunger for everyone". And the move towards this new
model (if we decide to do it) will as always with any bigger project and
change offer new issues and problems, especially ones we don't even see
right now and which might be nasty to deal with.
But overall i'd rather take the chance to do something about the way we
"do" Fedora than to just waive the issues we have or hide them under the
carpet. And if solving the issues would be easy or important enough then
the community would have stepped up and done something about it. So
maybe that disruption is necessary as a way to shuffle the cards anew,
weed out old cruft, and get things straightened out. And last but not
least this won't be a quick thing, i expect this transition to take
several years to be honest. Anything less is not really realistic for an
OS of the size and complexity that Fedora has now reached.
If there is anything where i can start contributing let me know, i'd
really love to be involved.
Thanks & regards, Phil
PS: Name suggestion for Ring 0: Fedora <3, as it's the heart of
everything and everyone. ;) (</joke>)
--
Philipp Knirsch | Tel.: +49-711-96437-470
Manager Core Services | Fax.: +49-711-96437-111
Red Hat GmbH | Email: Phil Knirsch <pknirsch at redhat.com>
Wankelstrasse 5 | Web: http://www.redhat.com/
D-70563 Stuttgart, Germany
More information about the devel
mailing list