= On-demand Side Tags =
== Summary ==
Allow on-demand side tags, and allow packagers to a) tag whatever rpms
as build dependencies into those side tags, b) build packages they
want in those side tags, and then c) tag all or some of those new
packages back into appropriate “main” tag (e.g.
This new process will allow packagers to build rpms using older or
newer versions of build dependencies than the buildroot in a given
Fedora release, providing a subset of Modularity features in an
additive way (no influence on the way dependent packages are built or
installed). It may also coexist with Modularity in its current form.
As a policy change, default modules are disallowed and modules are only opt-in.
== Owners ==
* Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
* Email: zbyszek at in.waw.pl
* Name: [[User:Churchyard| Miro Hrončok]]
== Detailed Description ==
=== Motivation ===
Modularity in its design and current implementation provides a number
of features that can be divided into two: for people ''building rpms
in modules'', and for people ''installing rpms from modules''.
have been a number of issues, RFEs, and discussions about modularity,
but they are mostly about the second, user-facing part. Rpms installed
from modules interact with non-modular rpms, and of course every
aspect of this interaction is visible to users. To make Modularity a
success, we would not only need to make this interaction well-behaved,
we would also need to explain it to users in a satisfactory way.
The first part, module build internals, is not visible to users. Some
packagers don’t like the syntax, which is opaque and best-suited for
programmatic generation and consumption. Constant rebuilding of
modules also creates noticeable load on the infrastructure, making
everything else slower. If modularity became much more popular, and a
significant number of packagers was using it, we are convinced many
discussions would start about this first part too.
Despite some problems, there is clear demand for the features that
Modularity brings. In particular, we need some solution for the "too
fast too slow" problem, where some packages cannot update fast enough
to maintain compatibility with the rest of the distribution.
=== New approach: on-demand side tags ===
The new process proposed here provides an alternative to both parts of
The second part of Modularity is simply removed: '''users only get
normal rpms'''. After rpms have been built, some or all of them are
merged into the normal tags and delivered to users as usual. This is
pretty much the same as the usual rebuild of specific language or
graphic stacks in side tags that we do already. Dependencies between
rpms must be expressed using
The first part is replaced by a simpler scheme: '''we don’t define any
specific syntax or repositories to build packages'''. The packager has
to issue a sequence of 'fedpkg tag-build', 'fedpkg build', and 'koji
wait-repo' commands as appropriate (or 'fedpkg chain-build' or …). The
packager ''may'' link an older rpm into the side tag, allowing
building against old versions of packages. This means that if the
packager is not a committer to the package, they may only use a
version which was previously built in koji. The packager selects which
rpms should be merged back into the main tag. Old packages obviously
cannot be merged back.
To minimize disruption for users and current packagers, modules are
required to be opt-in only. There are no default modules and
non-modular rpms cannot depend for build or installation on modular
rpms. This way there is less burden to other packagers and Modularity
can evolve more easily.
=== Implementation ===
'''Ability to create side tags “on demand”, i.e. with an automated API
call''' must be implemented. Koji and other infra tools would need to
be able to deal with a large number of tags. Modularity already
creates a large number of tags, so this should not be a significant
Packager tools like fedpkg must grow '''new commands to create side
Both the workflow in this proposal, and modules as they currently are
would be allowed to coexist. After we have some experience with both
approaches, one or both could be retired.
Since this proposal removes the possibility to have default modules,
'''modules need to be reset'''. The details are not specified yet.
approach would be to do the equivalent of dnf module reset all during
upgrade to F32, similarly to the proposed solution for
This proposal assumes that '''fedpkg will be enhanced to do extended
chain builds smartly'''. This is not a precondition: even without that
packagers may script something that is appropriate for them. This is
one of the main advantages of the proposal: we put the “smarts” in
local tooling which can be enhanced and modified without breaking what
other packagers do. E.g. a solution to rebuild all 500 rust packages
with new versions may be significantly different than building a
compat version of postgresql, and that is OK.
=== Buildroot only packages ===
They are possible, because not all packages are merged back. The same
is currently possible with side tags, and with non-rawhide builds for
which packagers may never create a bodhi update, so there is no
fundamental change. Whether or not we need to disallow packages which
are purposefully buildroot-only by a policy is left out of this
proposal on purpose. Technically, it is possible.
== Benefit to Fedora ==
=== Costs, for packages converted from modules to this scheme ===
Transitional costs: according to this proposal, modules may still be
used. Nevertheless, if the contents of a module are to be used to
build normal rpms, that module will need to switch to the new process
or simply demodularized and built in the traditional way.
In the case where this happens, packaging would need to be
“translated” from module .yaml to normal spec files. This should
mostly be about adjusting metadata, i.e. package versions and
We would lose one of the main advertised features of Modularity: the
ability to build the same module in different versions. This might not
be a huge loss, for two reasons. First, great majority of modules are
single-version. Second, if desired it is possible to provide compat
packages with different names. This is a bit of annoying work, but we
have the process very well understood. Compat packages have the
additional advantage that they may be made co-installable (vide
python26, python2, python35, python36, python3, python38 packages
currently in F31). Compat packages are exempt from package review. We
may consider additional simplifications to the process to make compat
=== Benefits ===
Vast simplification for other packagers, i.e. those who would like to
depend on rpms which are currently modularized, and for users, who do
not have separate modular repos, just a single repo like before.
Reduced load on the build infrastructure.
Easier to find rpm sources because there again is one canonical repo.
Third parties building on top of Fedora can use our modules to build
their own rpms without trouble.
dnf does not need to handle complicated logic. It should be faster if
less repos are used.
Rpms are built against an up-to-date buildroot, with only the minimal
compat package inserted, instead of some
older-than-all-current-releases Fedora release.
Normal package-level package replacement and obsoletion mechanisms are used.
On-demand side tags will remove the need to open releng tickets to
have them manually created.
== Scope ==
* Proposal owners: detailed documentation how to do things in the new
scheme, client-side tooling improvements.
* Other developers: demodularize specific packages if wanted or necessary.
* Release engineering: implement the server side of side tags on
demand [RELENG TICKET WILL BE FILED LATER]
* Policies and guidelines: the docs would need to be added to packager
documentation. Pull requests will be submitted by proposal owners.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This change should be fully additive to normal rpm packaging, so there
should be no upgrade/compatibility issues.
Some modules will need to be demodularized. This will require packager
work. Once this happens, rpms built this way will be fully available
to build normal rpms and modules, so the upgrade impact should be
limited to the modules being rewritten.
== How To Test a.k.a. examples of the new workflow ==
=== Version bump in a language stack ===
It is determined that a new set of rust rpm versions should be built.
A member of the rust-sig bumps the versions in dist-git for all
relevant packages, creates a side tag, and does the huge chain-build.
Once finished, this side tag is merged into rawhide-build. If desired,
it may also be merged into f31-updates-candidate and/or
f30-updates-candidate and bodhi updates submitted.
rust-sig already has tooling to generate the appropriate build-order
for modularity’s .yaml file. The same code can be reused to generate
the order for the chain build.
=== Three streams of a database package ===
A security patch needs to be applied to three versions of postgresql.
There are postgresql, postgresql82, postgresql81 dist-git repos. The
patch is added in dist-git for each of the packages, and they are
rebuilt as appropriate. Since we want to build each package with
different compiler versions, the packager uses a bash loop to schedule
separate builds of each of the three packages in each of the supported
(Once we have automatic creation of bodhi updates from koji builds,
the packager will not have to do anything more. Right now they still
need to create one big update that will then be split into separate
updates for each Fedora release.)
A small advantage compared to current modules is that packages are not
built against an already obsolete Fedora base, so users get the
benefit of toolchain improvements.
=== Package needs some old dep to build ===
Poor packager needs gcc-9.0, at the time where gcc-9.1 is the default
in the build root, to build koo. They bump the version of koo in
dist-git, create a side tag, tag old gcc-9.0-12.fc31 build into the
tag, build koo, merge the built package back into f32-build and
=== Package needs some old dep to install ===
koo-gui is only compatible with python3-qt5-5.12 when python3-qt5-5.13
has been released, and cannot be built with the new version. koo
maintainers may either a) arrange for python-qt5_5_12 compat package
to be created and then use it in the build root and as installation
dep, b) install old python3-qt5-5.12 package in build root, and bundle
some contents of that package into its output rpms, and add
=== FAQ for “other developers” ===
==== I maintain several modular streams of my package additionally to
the default version in Fedora. Can I keep doing this? ====
Yes, you can keep any alternate version in modular branches. When
appropriate, consider compat packages that can often be
co-installable, unlike modular streams.
==== I maintain my packages as modular only. Can I keep doing this? ====
Yes, as long as those modules are “leaves”, in the sense that no
non-modular rpms depend on them.
==== I would like to use an rpm which is currently only available as
module as a build dependency ====
The owner of the module should be asked to demodularize the module
(and either convert it to fully traditional rpms, or build some rpms
using the new side-tag process). Those rpms can then be used as build-
and installation-time dependencies.
If the owner of the module is not interested in providing non-modular
rpms, other packages may add such rpms to the distribution (and again
build them in one of the two ways).
==== Which rpms can be used as build dependencies in the side tag? ====
In principle, any rpm which is available in koji (including the
recursive dependencies, so it can actually be installed). In addition,
the packager may build rpms and use them in the side tag build root.
In practice, we expect that packagers will use one of the Fedora
releases (with updates), as the base.
== User Experience ==
Users have a clearer picture where their rpms are coming from. If they
opt-in into a module, they get some additional content. Without that,
they only get rpms from the normal Fedora repository. They are able to
rebuild rpms locally if there are no version conflicts. In case of
version conflicts, they might need to install older versions of rpms
from koji to satisfy the build (which might sound bad, but really
isn’t much different from current state with normal rpms.)
== Dependencies ==
On-demand side tags. Without this, this would be completely
infeasible. For limited and initial testing, side tags could be
== Contingency Plan ==
In the implementation phase, this should not impact other rpms in any
way (since it is change really happens on the build farm).
Once deployed, and rpms are built using this approach, they would
still be installable, even if the new process is abandonded. So there
is no flag day for revert, but some packages would effectively become
FTBFS. Any rpms which can only be built in this new scheme (e.g.
because they require older packages that the current ones in the
default build root), would have to be either fixed to build in the
normal build root, or would have to be modularized.
If this approach is abandoned, other approaches including the proposal
may be implemented, but that is only possible in the longer term.
* Contingency mechanism:
1. disable side-tag creation in koji (releng). Hopefully this would be
just a single switch somewhere. Any code that is added to koji can be
removed later (koji developers).
2. undo any documentation changes
3. adjust packages as described above (change owners, package owners).
This is very hard to say how big the scope of this would be, because
it is unclear how many packages would be involed. I think we can
assume that similarly as with Modularity, it would be <10 in the first
Fedora release where this is available.
* Contingency deadline: TBD
* Blocks release? Yes
== Documentation ==
Will be submitted by Change owners.
He / Him / His
Fedora Program Manager