Good news, module maintainers.
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
Skip to "WHAT IS CHANGING" section to see the main message.
HISTORY =======
Up to now, you wrote a YAML document in modulemd-v2 format https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml_specs/modulemd_stream_v2.yaml:
document: modulemd version: 2 data: name: perl-DBI stream: '1.643' license: module: - MIT dependencies: - buildrequires: platform: [] perl: [] requires: platform: [] perl: [] buildopts: rpms: macros: | %this_is_my_module 1 components: rpms: perl-DBI: rationale: API ref: f34
You built it and an output were multiple module builds, one for each combination of a Fedora release and a perl stream. E.g. this matrix:
buildrequires: platform: [f35, f34] perl: [5.30, 5.32] requires: platform: [f35, f34] perl: [5.30, 5.32]
was expaned into 4 unique builds by MBS, Module Build Service. Observe the last, "random" value, context:
perl-DBI:1.643:3420210211145152:7f057cb7 perl-DBI:1.643:3420210211145152:a450835c perl-DBI:1.643:3520210211145152:2c956793 perl-DBI:1.643:3520210211145152:e12b6f3a
These output modules used the same modulemd-v2 format. It was deemed that having the same input and output format is a benefit.
WHY ===
But it turned out that DNF had difficulties to upgrade from an older version of a module to a newer one. The problem was that DNF could not identify which build upgrades which one. E.g. having a Fedora 35 system with these builds in a repository:
1 perl-DBI:1.643:3520210211145152:2c956793 * this one is installed 2 perl-DBI:1.643:3520210211145152:e12b6f3a
3 perl-DBI:1.643:3520210212105947:ab395794 4 perl-DBI:1.643:3520210212105947:c39b3af3
DNF did not know whether to upgrade to the 3rd build or the 4th one. Both of them have the highest version. But which one to use? Context value is no clue here because it differs from the older builds. This happens when a module is changing its build-requires. (Actually first we observed this on RHEL where the platform changes with each minor RHEL release.) Run-time dependencies are also of no help because they could also change. (We also observed this on RHEL. Original design used run-time dependencies for the selection.)
Therefore DNF maintainers declared that this problem is undecidable and requested a change in Modularity.
Modularity team together with DNF maintainers identified that a culprit is the random nature of the context and decided to sacrifice a stream expansion for making the context static and predictable. It was designed that the context value will be fully in hands of the module maintainers who will be able freely add, remove, and maintain a module upgrade path by the means of the context:
Context Context Context A B ↓ ↓ Version 3520210211145152 3520210211145152 requires: perl:5.30 requires: perl:5.32 C (perl:5.34 added) ↓ ↓ ↓ Version 3520210212105947 3520210212105947 3520210212105947 (a new dep. requires: perl:5.30 requires: perl:5.32 requires: perl:5.34 added) requires: nginx:1.20 requires: nginx:1.20 requires: nginx:1.20 (perl:5.30 ended) ↓ ↓ Version 3520210930210943 3520210930210943 (nginx not requires: perl:5.32 requires: perl:5.34 req. anymore) ↓ ↓
WHAT IS CHANGING ================
Since there was no place for the contexts in the old module format, a new, input-only format "modulemd-packager", version 3, was designed and implemented in MBS and other tools. At the opportunity of the format change, some other pet peeves of the old format were solved.
Changes in the format include:
New document type and version:
document: modulemd-packager version: 3
A "module" middle-field removed from the "license" section:
license: - MIT
A "dependencies" section replaced with a "configurations" section:
configurations: - context: A platform: f34 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] - context: B platform: f34 buildrequires: perl: ['5.32'] requires: perl: ['5.32'] - context: C platform: f35 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] - context: D platform: f35 buildrequires: perl: ['5.32'] requires: perl: ['5.32']
The context value is a string and its alphabet and length is limited. All the contexts must be unique among a module stream. The plaform field is a scalar now. However, the stream values of the "buildrequires" and "requires" dependencies are a list. Although the specification requires a single value.
A "buildopts" section was moved from /data to /data/configurations/*/context section. Therefore it's specific to a context and should be repeated if needed any time: configurations: - context: A platform: f34 buildrequires: perl: ['5.30'] requires: perl: ['5.30'] buildopts: rpms: macros: | %this_is_my_module A - context: B platform: f34 buildrequires: perl: ['5.32'] requires: perl: ['5.32'] buildopts: rpms: macros: | %this_is_my_module B
And that's all.
The complete specification of modulemd-packager-v3 format is at https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml_specs/modulemd_packager_v3.yaml.
WHEN ====
Modularity team started to work on the new format a year a ago, the implementation was deployed to Fedora infrastructure few months ago and I've been waiting with the announcement until fixing the most serious bugs. That, I hope, happened last week.
In other words, you can start using the new format right now.
DO I HAVE TO MIGRATE MY MODULES? ================================
No, you don't have to. But Modularity people would like to see moving all Fedora content there.
The old modulemd-v2 format is still supported by MBS.
DNF handles outputs of both of them (the output format is distinguished with "static_context: true" field), but it behaves differently. With the old format, DNF can produce weird warnings or errors. Especially when your module is required by another module.
Will the old format be supporterd forever? I don't know.
Does have the new format some drawbacks? Yes, it has: It's longer than the previous one. You need to update it for each new Fedora release. You cannot run-require an unspecified (= any) module stream (perl:[]). (Though a DNF maintainer says it should not be a problem.)
HOW TO MIGRATE ==============
Read the "WHAT IS CHANGING" section of this e-mail message. Follow the specification if in doubts.
When migrating your modules, you will have to come up with the context value. To preserve a compatibility with the old builds and to preserve the upgrade path, I strongly recommend reusing the old context values. Use Koji https://admin.fedoraproject.org/pkgdb/package/MODULE/, or MBS https://mbs.fedoraproject.org/module-build-service/2/module-builds/?name=MODULE&stream=STREAM&state=5, or "dnf module info MODULE:STREAM" command to locate the lastest version of the module, and pick a context matching the desired dependencies as depicted here:
# dnf -q module info perl-DBI:1.643 Name : perl-DBI Stream : 1.643 Version : 3520210722203952 Context : e12b6f3a ------+ […] | Requires : perl:[5.32] ----+ | platform:[f35] --+ | | | | | The new YAML file: | | | | | | data: | | | configurations: | | | - context: 'e12b6f3a' <----|-|-+ platform: f35 <----+ | buildrequires: | perl: ['5.32'] | requires: | perl: ['5.32'] <------+ buildopts:
You can validate your YAML files with "modulemd-validator --type modulemd-packager-v3 FILE" command.
For a real example look at any perl* module. I've already migrated them. For instance this commit https://src.fedoraproject.org/modules/perl-DBI/c/2200d21f0fab5295226d915c24e49063d226e777?branch=1.643.
FEEDBACK ========
General help on modularity devel@lists.fedoraproject.org. Modularity design issues https://pagure.io/modularity/issues. Fedora MBS instance issues https://pagure.io/fedora-infrastructure/issues. MBS implementation bugs https://pagure.io/fm-orchestrator/issues.
DISCLAIMER ==========
The examples in this document are simplified and sometimes redacted to better illustrate the discussed topic.
I joined Modularity team way after designing this change. Therefore I cannot guarantee that all data provided here are accurate.
I'd like to thank all the people who participated in this change and who helped me to understand it. Especially Martin Curlej, Jaroslav Mracek, Daniel Mach, and Mike McLean.
-- Petr
On Fri, Sep 10, 2021 at 8:26 AM Petr Pisar ppisar@redhat.com wrote:
Good news, module maintainers.
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
So this certainly solves a lot of issues, but it's still frustrating to see that we *need* to declare the platform and that we couldn't inherit the platform from the build environment. This means that at branching time, we need to make a commit to change the requested platform.
That being said, does the platform value influence how the final modulemd is generated? That is, if I made a system that accepted this YAML file as input but *ignored* the platform value (and the platform was forced by other means), would a module build still produce working modules?
-- 真実はいつも一つ！/ Always, there's only one truth!
V Fri, Sep 10, 2021 at 08:50:44AM -0400, Neal Gompa napsal(a):
On Fri, Sep 10, 2021 at 8:26 AM Petr Pisar ppisar@redhat.com wrote:
Good news, module maintainers.
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
So this certainly solves a lot of issues, but it's still frustrating to see that we *need* to declare the platform and that we couldn't inherit the platform from the build environment. This means that at branching time, we need to make a commit to change the requested platform.
I confirm it is so and I fully agree with you that it's unpleasant.
The problem is that if you want to have the context invariant, then you need to store a mapping between a context and its build-dependencies somewhere. And naturally as new Fedora releases are added, and end-of-live releases removed, the mapping must be updated because platform is one of the dependencies.
That being said, does the platform value influence how the final modulemd is generated? That is, if I made a system that accepted this YAML file as input but *ignored* the platform value
It would end up like this https://pagure.io/fm-orchestrator/issue/1714: The build service would mix contexts for different platforms and the module wouldn't probably build at all like in the referred issue.
(and the platform was forced by other means)
For example? The information has to be somewhere stored, somehow retrieved, and delivered to a build and compose systems.
In non-modular world the information is stored in dist-git branch name, delivered to Koji as a build target, and stored in Koji build as tag. The information is updated with each Fedora release. The process of updating is called branching and package retirement.
I was thinking about possible solutions:
One was omitting platform from the context. You would not specify any platform in the module input format.
- context: A
You would submit the module for building. MBS would expand the build for all platforms (= currently supported Fedoras). The result would be four module builds:
Build Dependencies ------------------------------------------- perl:5.32:362021:A platform:36:0:0 perl:5.32:352021:A platform:35:0:0 perl:5.32:342021:A platform:34:0:0 perl:5.32:332021:A platform:33:0:0
But somewhere would have to be stored for which platform was the build done. We can guess it from the version prefix (36..., 35...) now. (ELN has no prefix.)
Then when you build perl-DBI:1.643 which build- and run-requires perl:5.32:
- context: A buildrequires: perl: [5.32] requires: perl: [5.32]
the same way without platforms, MBS would expand a matrix of the platforms and perls:
perl-DBI:1.643:362022:A platform:36:0:0 perl:5.32:362021:A perl-DBI:1.643:352022:A platform:35:0:0 perl:5.32:352021:A perl-DBI:1.643:342022:A platform:34:0:0 perl:5.32:342021:A perl-DBI:1.643:332022:A platform:33:0:0 perl:5.32:332021:A
That would work. But how would a packager specify that he does not want to build for all plaforms? E.g. he adds a module to Fedora 36 and is not interested in the older ones. Or e.g. he stops maintaining the module and the last supported is Fedora 35.
Maybe we could add a new field into the input file:
- platforms: [f33, f34, f35] - context: A buildrequires: perl: [5.32] requires: perl: [5.32]
and default to all platforms if not specified.
When I shared this idea I was told that a context must change with a platform. That everything (MBS?) assumes it. I believe I was also told another, reasonable reason. But I cannot recall now. I'm sorry I cannot provide better explanation now.
However, it's true that it does matter much whether the platform is encoded into the version or into the context (perl:5.32:362021:A vs. perl:5.32:2021:36A). If we hadn't it encoded anywhere, we could not store the builds into Koji, and MBS:
perl:5.32:2021:A platform:36:0:0 perl:5.32:2021:A platform:35:0:0 perl:5.32:2021:A platform:34:0:0 perl:5.32:2021:A platform:33:0:0
And it's true that we would only special-case platform. It wouldn't be fair to other modules. There must a stroger reason.
-- Petr
V Fri, Sep 10, 2021 at 02:26:20PM +0200, Petr Pisar napsal(a):
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
[...]
WHAT IS CHANGING
Since there was no place for the contexts in the old module format, a new, input-only format "modulemd-packager", version 3, was designed and implemented in MBS and other tools. At the opportunity of the format change, some other pet peeves of the old format were solved.
Changes in the format include:
New document type and version:
document: modulemd-packager version: 3
[...]
DNF handles outputs of both of them (the output format is distinguished with "static_context: true" field), but it behaves differently. With the old format, DNF can produce weird warnings or errors. Especially when your module is required by another module.
Will the old format be supporterd forever? I don't know.
Does have the new format some drawbacks?
A new drawback came out https://bugzilla.redhat.com/show_bug.cgi?id=2004853:
Fedora 33 G.A. libmodulemd-2.9.4-3.fc33 does not yet support the new static_context field. Because DNF strictly validates modules found in repositories, libmodulemd will reject modules with the unknown field:
Module yaml error: Unexpected key in data: static_context [line 9 col 3]
and will ignore the module. That implies that DNF will handle packages of those module builds as non-modular before checking an RPM transaction:
Installing: autoconf noarch 2.69-34.fc33 fedora 666 k Installing dependencies: perl-B x86_64 1.80-471.module_f33+12592+71aff0e2 updates-modular 189 k
But then the transaction will be rejected because a strayed modular package is found:
Running transaction check No available modular metadata for modular package 'perl-B-1.80-471.module_f33+12592+71aff0e2.x86_64', it cannot be installed on the system
In short, Fedora 33 users who have not updated their system since 2020-12-16, won't be able to install packages which were (also) built in a module with the new format. This affects e.g. fresh off-line installations.
A workaround is simple: update your system:
# dnf upgrade
or explicitly upgrade libmodulemd:
# dnf --disable-repo 'updates*modular' upgrade libmodulemd
before installing any packages.
Since we cannot change G.A. installation media (ISO and qcow2 images), there is no remedy possible other than not pushing these updates to Fedora 33 repositories.
You have two options: Either postpone the migration to the new format until Fedora 33 reaches its end of life (December, 2021?), or do not build the module for Fedora 33 (i.e. remove a context section with platform: f33).
(That already happened with perl and perl-bootstrap modules. I will rebuild them and repush to stable, unless relengs untag them.)
(That already happened with swig:4.0. Unpushing it from updates-testing should help.)
I'm deeply sorry for this faux pas.
-- Petr
On Tue, Sep 21, 2021 at 11:50 AM Petr Pisar ppisar@redhat.com wrote:
V Fri, Sep 10, 2021 at 02:26:20PM +0200, Petr Pisar napsal(a):
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
[...]
WHAT IS CHANGING
Since there was no place for the contexts in the old module format, a new, input-only format "modulemd-packager", version 3, was designed and implemented in MBS and other tools. At the opportunity of the format change, some other pet peeves of the old format were solved.
Changes in the format include:
New document type and version:
document: modulemd-packager version: 3[...]
DNF handles outputs of both of them (the output format is distinguished with "static_context: true" field), but it behaves differently. With the old format, DNF can produce weird warnings or errors. Especially when your module is required by another module.
Will the old format be supporterd forever? I don't know.
Does have the new format some drawbacks?
A new drawback came out https://bugzilla.redhat.com/show_bug.cgi?id=2004853:
Fedora 33 G.A. libmodulemd-2.9.4-3.fc33 does not yet support the new static_context field. Because DNF strictly validates modules found in repositories, libmodulemd will reject modules with the unknown field:
Module yaml error: Unexpected key in data: static_context [line 9 col 3]and will ignore the module. That implies that DNF will handle packages of those module builds as non-modular before checking an RPM transaction:
I really wonder how DNF is getting into this situation, because we created the API with a `strict=False` option specifically for DNF.[1] If they were reading the documents with that set appropriately, it would be ignoring just the `static_context` attribute rather than the entire document.
Though as I'm writing this, I did notice we have a copy-paste mistake[2] in the documentation where we didn't set the value of strict to False when talking about the DNF case, so that's perhaps part of the reason. Sorry about that!
[1] https://fedora-modularity.github.io/libmodulemd/latest/modulemd-2.0-Modulemd... [2] https://github.com/fedora-modularity/libmodulemd/pull/577
On 9/10/21 6:26 AM, Petr Pisar wrote:
Good news, module maintainers.
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
Question: is this necessary for all modules to migrate to this format, or is it primarily useful for modules that depend on other modules?
V Sat, Oct 02, 2021 at 12:29:01PM -0600, Orion Poplawski napsal(a):
On 9/10/21 6:26 AM, Petr Pisar wrote:
Good news, module maintainers.
I'm relieved to announce an availability of the new module packaging format, modulemd-packager, version 3.
Question: is this necessary for all modules to migrate to this format, or is it primarily useful for modules that depend on other modules?
I cannot provide you with any better answer than I've already stated in my original message: It is primarily useful for modules which depends on other modules. It is not necessary for all modules right now, because both MBS and DNF support both of the formats. However, I won't speculate how long they will support them.
-- Petr
-
Neal Gompa -
Orion Poplawski -
Petr Pisar -
Stephen Gallagher