Fedora 32 System-Wide Change proposal: Modules in Non-Modular Buildroot

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot

Enable module default streams in the buildroot repository for modular
and non-modular RPMs.

== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.

== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG

== Detailed Description ==
As a major part of the Modularity design, we have a concept of default
module streams. These streams are built as modules, but the RPM
artifacts they deliver are intended to be used just like non-modular
RPMS. The aspirational goal is that a user of the system who never
executes a module-specific command (such as `dnf module install
nodejs:8`) should experience no meaningful changes in behavior from
how they would interact with a completely non-modular system. In
practice, this may mean that the informational output of package
managers may indicate that modules are being enabled and used, but a
user that does not have a specific reason to interact with the
selection of a module stream should have that managed on their behalf.

Similarly, the experience for package maintainers of non-modular
packages should be unaffected by an RPM dependency moving from the
non-modular repository into a default module stream. Up to the
present, this has not been the case; no module stream content has been
available in the non-modular buildroot for other packages to consume.
Koji builds of non-modular RPMs have had only the other non-modular
RPMs from that release available to their buildroots. In contrast,
building on local systems has access to both the non-modular RPMs and
the RPMs from any of the default module streams. With this Change,
Koji builds will have the same behavior and be able to depend on
content provided by default module streams. It also enables the same
behavior for Modular builds: the `platform` stream will now include
the contents of the default module streams for each release and do not
need to be explicitly specified in the modulemd `buildrequires`.

Note: This Change does not address the other major Modularity issue we
are facing around distribution upgrades with differing default
streams. When discussing this Change, please keep that topic separate.

== Benefit to Fedora ==

This will simplify the lives of package maintainers in Fedora in two
primary ways. I'll use a hypothetical example of the Node.js
interpreter and a JSApp package which is capable of running on Node.js
10 or 12 (but requires newer features than are provided by Node.js 8).
Additionally, the JSApp package requires the same versions of Node.js
at build-time.

* Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F31 lifetime.
* Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F32 lifetime.

On Fedora 29 through 31, the Node.js package maintainer needs to build
the `nodejs:10` package both as a module and as a non-modular RPM in
the distribution so that the JSApp package can be built. With this
Change, the Node.js package maintainer in Fedora 32+ will only need to
build the various Node.js streams and make one of them the default
stream. The packages from it will then be added to the buildroot for
non-modular packages. This will also make the packaging process
somewhat more efficient, as the maintainer needs only to manage the
module stream and the MBS will build it for all configured platforms.

Similarly, from the perspective of dependent maintainers, there will
no longer be anxiety about needing to move their package to a module
if one or more of their dependencies drops their non-modular version
in favor of a default stream. Their builds will continue to work as
they do today.

== Scope ==
* Proposal owners:
# Update Packaging Guidelines with
[https://pagure.io/modularity/issue/146#comment-600328 requirements]
for module default streams
# Create a Pungi configuration to generate the buildroot from the
default module streams.
# Include `default_modules_scm_url` in the platform virtual module specification
# Configure Koji tags for inheriting the new modular-defaults
buildroot into the standard buildroot

* Other developers:

Packagers of default module streams will be required to conform to the
[https://pagure.io/modularity/issue/146#comment-600328 policy]
regarding visibility of stream artifacts. Any default module stream
that is not in compliance by one week before Beta Freeze will cease to
be a default stream.

* Release engineering:
# https://pagure.io/releng/issue/8879 - Create pungi config for
Rawhide/F32 ursa prime buildroot
# https://pagure.io/releng/issue/8880 - Include
`default_modules_scm_url` in platform 31 virtual module
# https://pagure.io/releng/issue/8881 - Configure Koji tags for
inheriting f32-modular-buildroot

* Policies and guidelines:
The Modularity Packaging Guidelines will need to be updated to
indicate the strict requirements on default streams.
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
This change is on the build-system side of things and should not
impact the upgrade process directly.

== How To Test ==
# Build a modular stream
# Make that stream a default stream (or a buildroot override)
# Build a non-modular RPM that requires an artifact RPM from the modular stream.

== User Experience ==
This should not change the end-user experience.

== Dependencies ==
Nothing known that isn't listed in the scope.

== Contingency Plan ==
* Contingency mechanism: Disable the buildroot inheritance in Koji to
revert to the current behavior.
* Blocks release? Ambiguous: lack of complete implementation may
indirectly cause blocking issues.
* Blocks product? No

== Documentation ==


== Release Notes ==
None needed, the Change is not user-facing.

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis