Latest thoughts on user level package management

[Stuart Campbell]

> On Jun 4, 2015, at 3:41 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> In relation to the Fedora Modularization objective approved in
> https://fedorahosted.org/council/ticket/26, I've been thinking further
> about the ideas in
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
> and how it might be possible to evolve them into a maintainable
> multi-tiered ecosystem of components
> 
> This isn't quite the same concept as Fedora.next's "rings", and the
> word "levels" on its own is boring, so I'm going to take inspiration
> from Fedora's logo and use "Aleph"
> (http://en.wikipedia.org/wiki/Aleph_number), which also nicely
> captures the fact that we expect there to be increasingly more
> software in each tier (because the barriers to entry will be lower).
> 
> For each tier, there'd be different expectations on the degree of
> scrutiny applied to affected components and how closely they track
> upstream, with the outermost tiers being a Do-It-Yourself adventure
> where all we're saying is "We've checked, and it's legal for you to
> use this, and the folks publishing it aren't obviously malicious".
> Each tier would also have different constraints on how the components
> were published, and the typically available means of consuming them.
> 
> This is the first time I've written this down, so I expect it to
> require a lot of modification to become a workable proposal, and it
> may not be fixable at all. If folks at the WG meeting this evening
> agree there's at least the seed of a good idea here, I'll move it over
> to the wiki :)
> 
> The 6 proposed levels:
> 
> * Aleph 0: Essential components
> * Aleph 1: Integrated components
> * Aleph 2: Redistributed components
> * Aleph 3: Experimental components
> * Aleph 4: Developer components
> * Aleph 5: Upstream components
> 
> Aleph's 3-5 would be the domain of Environments & Stacks, Aleph 0
> would be the domain of the Base WG and Edition WG's, Aleph 1 would
> include everything else specifically taken into account for release
> management purposes (and perhaps a bit more), while Aleph 2 would
> primarily be the domain of individual package maintainers.
> 
> Tiers using RPM as the publication format may contain distro specific
> patches, tiers using other formats (i.e. Aleph 4 & 5) would always
> provide vanilla upstream packages (as a result, I consider the fact
> the proposal switches technologies at Aleph 4 to be a positive feature
> rather than as a problem to be resolved)
> 
> = Aleph 0: Essential components =
> 
> Publication format: RPM
> Build system: koji
> Consumption formats: RPM, ISO, AMI, OSTree, base Docker images
> 
> All the RPMs that go into Fedora Cloud, Fedora Atomic Host, and the
> base installs for Fedora Server and Fedora Workstation are Aleph 0
> components. Any component proposed for inclusion at this tier *must*
> be available as a policy compliant RPM.
> 
> Some essential components could eventually be made available as
> xdg-app sandboxes or layered Docker images. Whether they're classed as
> "essential" or not would still be determined by whether or not they
> were a default component in one of the Fedora Editions.
> 
> If essential components aren't closely tracking their upstream
> counterparts, we should want to know why. We'd expect essential
> components to be smoothly handed over to new maintainers rather than
> getting orphaned without a migration plan.
> 
> = Aleph 1: Integrated components =
> 
> Publication format: RPM
> Build system: koji
> Consumption formats: RPM, ISO (, layered Docker image?, xdg-app?)
> 
> All the RPMs that go into Labs and Spins would be Aleph 1 components.
> There might be other components that would fit here as well if the
> Fedora starts providing them in a pre-integrated form, like a layered
> Docker image, or an xdg-app. The upstream packages that feed into
> softwarecollections.org should likely also be Aleph 1
> 
> If integrated components aren't closely tracking their upstream
> counterparts, we should want to know why. We'd expect integrated
> components to be smoothly handed over to new maintainers rather than
> getting orphaned without a migration plan.
> 
> = Aleph 2: Redistributed components =
> 
> Publication format: RPM
> Build system: koji
> Consumption formats: RPM(, layered Docker image?, xdg-app?)
> 
> Everything else in the main Fedora repos, as well as everything in EPEL.
> 
> The key differences from Aleph 0 and 1 is that these components may
> not closely track their upstream counterparts, and they may be
> orphaned and retired without a specific migration plan in place.
> 
> = Aleph 3: Experimental components =
> 
> Publication format: RPM
> Build system: COPR
> Consumption formats: RPM
> 
> Fedora Playground, COPR repos in general.
> 
> = Aleph 4: Developer components =
> 
> Publication format: nix???
> Build system: ????
> Consumption formats: nix???
> 
> This level would be developer components in a language and distro
> independent format. nix is the current frontrunner in my mind, mostly
> due to the good discussions between myself, Randy Barlow (Pulp) and
> Domen Kožar (NixOS), and the fact that Nix already uses the model of a
> central system-wide package store, with user and environment specific
> views into that store (this is a much nicer architecture for auditing
> purposes, since you can audit the package store directly, rather than
> having to check each environment).
> 
> The intended use of these components would be local development and
> data analysis, as well as incorporation into layered Docker image and
> xdg-app sandbox builds.
> 
> Generation of these components from their corresponding upstream
> components should be fully automated (letting us easily track upstream
> closely after the initial approval process), so the fact Domen has
> been working on automated PyPI -> Nix conversions
> (https://twitter.com/iElectric/status/605686824443494400) does play a
> part in my inclination towards Nix here :)
> 
> = Aleph 5: Upstream components =
> 
> Publication format: language dependent
> Build system: ????
> Consumption formats: language dependent
> 
> For ecosystems that are designed to accommodate redistribution, this
> level would be for republishing approved upstream components in a
> language dependent format. The key aspects would be the licensing
> review to check it's actually permissible for us to redistribute it in
> accordance with Fedora's policies, and that the publishers of the
> software don't appear to be obviously malicious.
> 
> For language ecosystems where redistribution isn't well supported, it
> would be possible to skip this tier and go straight to Aleph 3 or 4.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> env-and-stacks mailing list
> env-and-stacks at lists.fedoraproject.org
> https://lists.fedoraproject.org/mailman/listinfo/env-and-stacks

I think this looks very sensible and well thought out.  Would the idea be to replace the concept of 'rings'? 

Cheers
Stu

Dr Stuart Campbell