[Proposal] Ring-based Packaging Policies

[Hans de Goede]

Hi,

On 12-02-15 19:32, Stephen Gallagher wrote:
> (Logistical note: please keep all replies to this thread on
> devel at lists.fedoraproject.org)
>
> tl;dr Shall we consider requiring a lesser package review for packages
> that are not present on Product or Spin install media?
>
> == Premise ==
>
> So, some time ago, we started talking about dividing up the Fedora
> package set into a series of "rings". One of the driving purposes behind
> this idea was to re-evaluate our policies for packaging in each of these
> rings. A fair amount of time has passed and no formal discussions have
> taken place (that I have seen, at any rate), so I'm going to provide
> here a simple "starter" proposal that we can work from and see where it
> leads.
>
> To begin, I'll try to identify some of the major advantages and
> disadvantages in our current policies. (In no particular order...)
>
> === Advantages ===
> * Consistency: every shared library, language-specific module and
> application should have packaging consistent with others in its
> category. This makes it easier for comaintainers and provenpackagers to
> pick it up and work on it.
>
> * The no-bundled-libraries policy means that when a library module
> requires an update, it means that only one package needs to be modified
> in order to enhance all applications on the system that consumes it.
> This is a significant time-saver when it comes to dealing with
> (increasingly common) security vulnerabilities.
>
> * Package review "front-loads" the task of educating the packagers. It
> guarantees that packages enter the collection by meeting a very high
> standard. This does a great deal to accomplish the "consistency"
> mentioned above.
>
> * Legal review and the FPCA ensures that Fedora remains true to its
> "Freedom" Foundation (as well as protecting Fedora contributors from the
> perils of the international legal system).
>
> === Disadvantages ===
> * Very strict policies often leads to potential packagers giving up.
>
> * Package reviews for less-interesting packages (such as those for less
> popular SIGs) often remain un-reviewed for weeks, months or even years.
>
> * The package policies are only ever reviewed during the initial
> creation of the package. Once that initial (high) hurdle is cleared, the
> packager essentially has free reign to do whatever they want with their
> package. This sometimes means that as time passes, the spec files
> "bit-rot" or otherwise start accumulating other inconsistencies. (A
> common example is the package whose upstream starts bundling a library
> without the packager noticing).
>
> * Many upstream projects do not concern themselves with being "in" any
> particular distribution (with the notable example being the
> Debian/Ubuntu flavors which have amassed a sufficient apparent userbase
> that they sometimes get special treatment). For a variety of reasons,
> this often leads directly to bundling the vast majority of their
> dependencies. This is done for many reasons, but the two most common are
> supportability and portability; it's impossible for many upstreams to
> actually QA their package with every possible distro library. Instead,
> if they ship everything they depend on, they can guarantee *that*
> specific combination. This moves the responsibility from the
> distribution to the upstream package to maintain their bundled
> libraries.
>
> * With many languages, there is no possibility of installing multiple
> versions of the same library on the same system, so if an application
> requires a newer or older version of the library than is in Fedora to
> run, it fails. For those languages that *do* allow this, it still adds a
> significant maintenance burden on the packagers to keep multiple
> versions of the same package up-to-date (which gets particularly hard
> when upstream abandons a version that something else in Fedora depends
> on...)
>
>
> == Analysis ==
> First, I will make an assumption based on the "First" Foundation: We as
> the Fedora Project want people to have access to the software that they
> desire. We may disagree on how that is to be achieved, but I believe the
> goal is the same.
>
> Second, I will call attention to the fact that different Fedora users
> have very different needs from the software. For example, those running
> Fedora Server and Fedora Cloud are likely far more concerned with Fedora
> as a *deployment* platform than they are as a *development* platform.
> Folks running Fedora Workstation or the KDE spin are likely to be
> somewhat more interested in the development side of things (though not
> exclusively).
>
> Packaging software for development and packaging it for real-world
> deployment can be very different. I'm not going to attempt to identify
> all the ways that this can be the case in this email. Others have done
> so with great verbosity in the past and I will not attempt to re-hash
> them.
>
> Thirdly, I will note (again) that many upstream projects have moved away
> from a distro-centric model. This is in part because unlike in the past,
> there are a great many distributions out there now, all with individual
> packaging requirements to be inside those distros. Taken in concert with
> the behaviors of many of the language repositories (like Python, Ruby
> and Node.js), it is simply *easier* and *faster* to just have your
> package carry whatever it needs with it.
>
> As I also mentioned above, the inflexibility of Fedora in carrying
> multiple versions of the same packages means that in many cases, we are
> *incapable* of providing a popular or interesting package in our repos.
> This also serves to drive potential users to other distributions.
>
> == Proposal ==
> With these things in mind, I'd like to propose that we amend the
> packaging policy by splitting it into two forms:
>
> === Core Packages ===
> Any package that is provided on a release-blocking medium (which at
> present includes Fedora Atomic, Fedora Cloud, Fedora Server, Fedora
> Workstation, the KDE Spin and several ARM images) must comply exactly
> with the packaging guidelines as they are written today. These packages
> must receive a full package review *when they are added to the install
> media*. Any package present on the media if this proposal is adopted is
> exempted from this requirement. Any package to newly appear on the
> install media after this time *should* (I hate that word...) receive a
> new package review, even if it is already present in Fedora.
>
> === Ring Packages ===
> Any new package that is *not* going to be part of the install media set
> is required to pass a lighter review and is permitted to carry bundled
> libraries, with caveats to be listed below.
>
> ==== Ring Package Review Requirements ====
> * A reviewer *MUST* establish suitability for the Fedora Project. This
> means verifying that the package contains only permissible content
> (legally-speaking).
>
> * The package *MUST* conform to the FHS and other filesystem conventions
> used by Fedora (such as %{python3_sitelib}), except when granted an
> exception by the FPC.
>
> * The package *MAY* contain bundled libraries or other projects, but if
> it does so, it *MUST* contain a "Provides: bundled(pkg) = version" for
> each such bundling. This is done so that we can use the meta-data to
> identify which packages may be vulnerable in the event of a security
> issue.
>
>
> == Conclusion ==
> So that is my proposal to consider for Fedora 23+ (it's too late in the
> Fedora 22 cycle to consider this, but I'd prefer starting the F23
> discussion right away). Comments and suggestions are strongly
> encouraged.

+1 I think this is a good idea, esp. the bundling libraries thing has
been a big hurdle to get some packages into Fedora. I think we may want
to encourage some unbundling though, in many cases some libs (e.g. zlib)
can be trivially unbundeld, where others are modified / a different
major version then what we may have in core, and there bundling is
something we will have to live with.

But these are just details, I like the general concept. We may want to
do something where core packages are in a different koji tag then
ring packages to ensure a strict self-contained ness also wrt Buildrequires
of the core, we could still generate one repo for the outside world, so as
to not put any extra strain on mirrors, but the repo's generated to do builds
from would be separate for core / ring builds. This will require some work
from rel-eng, and manually tagging all core packages as being core (once),
then once the tagging is done we will notice when any new deps are necessary
for core packages because their build will fail, and we can have a
(lightweight!) procedure for moving ring packages into the core, and vice
versa of course.

Regards,

Hans