Latest thoughts on user level package management

[Nick Coghlan]

In relation to the Fedora Modularization objective approved in
https://fedorahosted.org/council/ticket/26, I've been thinking further
about the ideas in
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
and how it might be possible to evolve them into a maintainable
multi-tiered ecosystem of components

This isn't quite the same concept as Fedora.next's "rings", and the
word "levels" on its own is boring, so I'm going to take inspiration
from Fedora's logo and use "Aleph"
(http://en.wikipedia.org/wiki/Aleph_number), which also nicely
captures the fact that we expect there to be increasingly more
software in each tier (because the barriers to entry will be lower).

For each tier, there'd be different expectations on the degree of
scrutiny applied to affected components and how closely they track
upstream, with the outermost tiers being a Do-It-Yourself adventure
where all we're saying is "We've checked, and it's legal for you to
use this, and the folks publishing it aren't obviously malicious".
Each tier would also have different constraints on how the components
were published, and the typically available means of consuming them.

This is the first time I've written this down, so I expect it to
require a lot of modification to become a workable proposal, and it
may not be fixable at all. If folks at the WG meeting this evening
agree there's at least the seed of a good idea here, I'll move it over
to the wiki :)

The 6 proposed levels:

* Aleph 0: Essential components
* Aleph 1: Integrated components
* Aleph 2: Redistributed components
* Aleph 3: Experimental components
* Aleph 4: Developer components
* Aleph 5: Upstream components

Aleph's 3-5 would be the domain of Environments & Stacks, Aleph 0
would be the domain of the Base WG and Edition WG's, Aleph 1 would
include everything else specifically taken into account for release
management purposes (and perhaps a bit more), while Aleph 2 would
primarily be the domain of individual package maintainers.

Tiers using RPM as the publication format may contain distro specific
patches, tiers using other formats (i.e. Aleph 4 & 5) would always
provide vanilla upstream packages (as a result, I consider the fact
the proposal switches technologies at Aleph 4 to be a positive feature
rather than as a problem to be resolved)

= Aleph 0: Essential components =

Publication format: RPM
Build system: koji
Consumption formats: RPM, ISO, AMI, OSTree, base Docker images

All the RPMs that go into Fedora Cloud, Fedora Atomic Host, and the
base installs for Fedora Server and Fedora Workstation are Aleph 0
components. Any component proposed for inclusion at this tier *must*
be available as a policy compliant RPM.

Some essential components could eventually be made available as
xdg-app sandboxes or layered Docker images. Whether they're classed as
"essential" or not would still be determined by whether or not they
were a default component in one of the Fedora Editions.

If essential components aren't closely tracking their upstream
counterparts, we should want to know why. We'd expect essential
components to be smoothly handed over to new maintainers rather than
getting orphaned without a migration plan.

= Aleph 1: Integrated components =

Publication format: RPM
Build system: koji
Consumption formats: RPM, ISO (, layered Docker image?, xdg-app?)

All the RPMs that go into Labs and Spins would be Aleph 1 components.
There might be other components that would fit here as well if the
Fedora starts providing them in a pre-integrated form, like a layered
Docker image, or an xdg-app. The upstream packages that feed into
softwarecollections.org should likely also be Aleph 1

If integrated components aren't closely tracking their upstream
counterparts, we should want to know why. We'd expect integrated
components to be smoothly handed over to new maintainers rather than
getting orphaned without a migration plan.

= Aleph 2: Redistributed components =

Publication format: RPM
Build system: koji
Consumption formats: RPM(, layered Docker image?, xdg-app?)

Everything else in the main Fedora repos, as well as everything in EPEL.

The key differences from Aleph 0 and 1 is that these components may
not closely track their upstream counterparts, and they may be
orphaned and retired without a specific migration plan in place.

= Aleph 3: Experimental components =

Publication format: RPM
Build system: COPR
Consumption formats: RPM

Fedora Playground, COPR repos in general.

= Aleph 4: Developer components =

Publication format: nix???
Build system: ????
Consumption formats: nix???

This level would be developer components in a language and distro
independent format. nix is the current frontrunner in my mind, mostly
due to the good discussions between myself, Randy Barlow (Pulp) and
Domen Kožar (NixOS), and the fact that Nix already uses the model of a
central system-wide package store, with user and environment specific
views into that store (this is a much nicer architecture for auditing
purposes, since you can audit the package store directly, rather than
having to check each environment).

The intended use of these components would be local development and
data analysis, as well as incorporation into layered Docker image and
xdg-app sandbox builds.

Generation of these components from their corresponding upstream
components should be fully automated (letting us easily track upstream
closely after the initial approval process), so the fact Domen has
been working on automated PyPI -> Nix conversions
(https://twitter.com/iElectric/status/605686824443494400) does play a
part in my inclination towards Nix here :)

= Aleph 5: Upstream components =

Publication format: language dependent
Build system: ????
Consumption formats: language dependent

For ecosystems that are designed to accommodate redistribution, this
level would be for republishing approved upstream components in a
language dependent format. The key aspects would be the licensing
review to check it's actually permissible for us to redistribute it in
accordance with Fedora's policies, and that the publishers of the
software don't appear to be obviously malicious.

For language ecosystems where redistribution isn't well supported, it
would be possible to skip this tier and go straight to Aleph 3 or 4.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia