Hello packagers,
I'm glad to announce that now it's possible to move a package back from a module to a nonmodular repository.
Motivation ==========
In the past there was a problem that once a package was added into a module, there was to way to return it back.
Let's say you have a curl:experimental module stream which delivers a future version of curl and which unfortunately needs a not-yet released version of OpenSSL. So you add two components into the module:
filter: rpms: - openssl-devel - openssl-perl components: rpms: openssl: rational: Run-time dependency ref: experimental buildorder: 0 curl: rational: API ref: experimental buildorder: 1
and release Fedora 35 with it:
name: curl stream: experimental version: 1 artifacts: rpms: - curl-0:9999-0.module_42.x86_64 - openssl-libs-1:3.0.1-0.1.module_42.x86_64
Users who want the experimental curl, will enable the stream with "dnf module switch-to curl:experimental" and curl-0:9999-0.module_42.x86_64 with the patched openssl-libs-1:3.0.0-1.module_42.x86_64 will get installed.
Time flows, OpenSSL releases a new 3.0.1 version with the missing feature, Fedora will upgrade the nonmodular openssl to 1:3.0.1-1, and you, as curl:experimental maintainer, want to get rid of the bundled, now redundant openssl.
So you remove it from your module:
components: rpms: curl: rational: API ref: experimental buildorder: 0
pushes it to an updates-testing-modular repository:
name: curl stream: experimental version: 2 artifacts: rpms: - curl-0:9999-0.module_42.x86_64
and after "dnf upgrade", you will find out that your machine is not using the new nonmodular openssl-libs-1:3.0.1-1.fc35 but still your old modular openssl-libs-1:3.0.1-0.1.module_42 package.
How is it possible? What has gone wrong?
The problem ===========
The reason lies in a "modular filtering" performed by DNF. When DNF loads repository metadata, it will see two module builds:
- curl:experimental:1 from fedora-modular repository with these packages: - curl-0:9999-0.module_42.x86_64 - openssl-libs-1:3.0.1-0.1.module_42.x86_64
- curl:experimental:2 from updates-testing-modular with this package: - curl-0:9999-0.module_42.x86_64
DNF will enumerate packages of all the versions of the module, and adds both curl, and openssl-libs to the modular filter. As a result, the nonmodular openssl-libs won't be visible, and instead the two curl and openssl-libs modular packages become visible to an RPM dependency solver.
Simply put, DNF does process old module versions. Why does it do? Because you may want to downgrade a broken package to an older version.
The solution ============
There were two approaches proposed: One was ignore the non-latest modules, another was mark removed packages explicitly. DNF maintainer decided for the latter with an explanation that the former would affect already released modules.
Therefore the process of demodularization is following:
demodularized: rpms: - openssl-libs components: rpms: curl: rational: API ref: experimental buildorder: 0
A new explicit field "demodularized" was introduced. It lists names of the binary packages which are not part of the module stream any longer. This list then appears in the repository:
name: curl stream: experimental version: 2 demodularized: rpms: - openssl-libs artifacts: rpms: - curl-0:9999-0.module_42.x86_64
and DNF will ignore the listed modular package of this stream. It means that "dnf upgrade" will make these packages available to the RPM solver:
- curl-0:9999-0.module_42.x86_64 modular - openssl-libs-1:3.0.1-1.x86_64 nonmodular - openssl-libs-1:3.0.1-0.1.module_42.x86_64 formerly modular
The solver will identify 1:3.0.1-1 as the highest NEVRA and install that. It means that the module's maintainer needs to coordinate the demodularization with the nonmodular maintainer because standard NEVRA ordering will be used. To provide a smooth transition, the nonmodular package should be built in a higher NEVRA before undergoing the demodularization.
DNF also reports the demodularized packages in "dnf module info ..." output.
It's important to mention that only the latest version of the module stream is consulted for the demodularized list. That allows you to reintroduce the package in any future module version simply by removing it from the list. It also means that you need to carry the demodularized list in all future module versions as long as there is a historical version with that artifact in the distribution. It's also good to know that the demodularization is scoped to the stream it was defined in. If there was another active module stream carrying the same-named package, then the demodularization would not take any effect because it would become overridden by the another stream.
Since when ==========
This demodularization feature was implemented in libmodulemd-2.13.0 and libdnf-0.64.0. Both of them are in stable updates of Fedora 33. But be ware that because of a different bug #2004853 it will cause harm to an older package manager (as can found e.g. in a Fedora 34 GA installation media) and you should avoid it there for that reason. So it's safe since Fedora 35.
The new demodularization list is supported in both modulemd-packager-v3 and modulemd-v2 YAML formats.
Thanks ======
I'd like to thank to Jaroslav Mráček for implementing this feature into DNF.
References ==========
- modulemd-packager-v3 specification https://github.com/fedora-modularity/libmodulemd/blob/main/yaml_specs/modulemd_packager_v3.yaml. - Example https://src.fedoraproject.org/modules/perl-CGI/raw/bdaa48c654422e4c3cb92324c36b284b07ee21d3/f/perl-CGI.yaml.
-- Petr