Modularity: New modulemd-packager format for building modules

[Petr Pisar]

Good news, module maintainers.

I'm relieved to announce an availability of the new module packaging format,
modulemd-packager, version 3.

Skip to "WHAT IS CHANGING" section to see the main message.


HISTORY
=======

Up to now, you wrote a YAML document in modulemd-v2 format
<https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml...;:

document: modulemd
version: 2
data:
    name: perl-DBI
    stream: '1.643'
    license:
        module:
            - MIT
    dependencies:
        - buildrequires:
            platform: []
            perl: []
          requires:
            platform: []
            perl: []
    buildopts:
        rpms:
            macros: |
                %this_is_my_module 1
    components:
        rpms:
            perl-DBI:
                rationale: API
                ref: f34

You built it and an output were multiple module builds, one for each
combination of a Fedora release and a perl stream. E.g. this matrix:

    buildrequires:
      platform: [f35, f34]
      perl: [5.30, 5.32]
    requires:
      platform: [f35, f34]
      perl: [5.30, 5.32]

was expaned into 4 unique builds by MBS, Module Build Service. Observe the
last, "random" value, context:

    perl-DBI:1.643:3420210211145152:7f057cb7
    perl-DBI:1.643:3420210211145152:a450835c
    perl-DBI:1.643:3520210211145152:2c956793
    perl-DBI:1.643:3520210211145152:e12b6f3a

These output modules used the same modulemd-v2 format. It was deemed that
having the same input and output format is a benefit.


WHY
===

But it turned out that DNF had difficulties to upgrade from an older version of
a module to a newer one. The problem was that DNF could not identify which
build upgrades which one. E.g. having a Fedora 35 system with these builds in
a repository:

    1  perl-DBI:1.643:3520210211145152:2c956793 * this one is installed
    2  perl-DBI:1.643:3520210211145152:e12b6f3a

    3  perl-DBI:1.643:3520210212105947:ab395794
    4  perl-DBI:1.643:3520210212105947:c39b3af3

DNF did not know whether to upgrade to the 3rd build or the 4th one. Both of
them have the highest version. But which one to use? Context value is no clue
here because it differs from the older builds. This happens when a module is
changing its build-requires. (Actually first we observed this on RHEL where
the platform changes with each minor RHEL release.) Run-time dependencies are
also of no help because they could also change. (We also observed this on
RHEL. Original design used run-time dependencies for the selection.)

Therefore DNF maintainers declared that this problem is undecidable and
requested a change in Modularity.

Modularity team together with DNF maintainers identified that a culprit is the
random nature of the context and decided to sacrifice a stream expansion for
making the context static and predictable. It was designed that the context
value will be fully in hands of the module maintainers who will be able freely
add, remove, and maintain a module upgrade path by the means of the context:

            Context                 Context                 Context
            A                       B                       
            ↓                       ↓                       
Version     3520210211145152        3520210211145152
            requires: perl:5.30     requires: perl:5.32     C (perl:5.34 added)
            ↓                       ↓                       ↓
Version     3520210212105947        3520210212105947        3520210212105947
(a new dep. requires: perl:5.30     requires: perl:5.32     requires: perl:5.34
 added)     requires: nginx:1.20    requires: nginx:1.20    requires: nginx:1.20
            (perl:5.30 ended)       ↓                       ↓
Version                             3520210930210943        3520210930210943
(nginx not                          requires: perl:5.32     requires: perl:5.34
 req. anymore)                      ↓                       ↓


WHAT IS CHANGING
================

Since there was no place for the contexts in the old module format, a new,
input-only format "modulemd-packager", version 3, was designed and implemented
in MBS and other tools. At the opportunity of the format change, some other
pet peeves of the old format were solved.

Changes in the format include:

New document type and version:

    document: modulemd-packager
    version: 3

A "module" middle-field removed from the "license" section:

    license:
        - MIT

A "dependencies" section replaced with a "configurations" section:

    configurations:
        - context: A
          platform: f34
          buildrequires:
              perl: ['5.30']
          requires:
              perl: ['5.30']
        - context: B
          platform: f34
          buildrequires:
              perl: ['5.32']
          requires:
              perl: ['5.32']
        - context: C
          platform: f35
          buildrequires:
              perl: ['5.30']
          requires:
              perl: ['5.30']
        - context: D
          platform: f35
          buildrequires:
              perl: ['5.32']
          requires:
              perl: ['5.32']

The context value is a string and its alphabet and length is limited. All the
contexts must be unique among a module stream. The plaform field is a scalar
now. However, the stream values of the "buildrequires" and "requires"
dependencies are a list. Although the specification requires a single value.

A "buildopts" section was moved from /data to /data/configurations/*/context
section. Therefore it's specific to a context and should be repeated if needed
any time:
    configurations:
        - context: A
          platform: f34
          buildrequires:
              perl: ['5.30']
          requires:
              perl: ['5.30']
          buildopts:
              rpms:
                  macros: |
                      %this_is_my_module A
        - context: B
          platform: f34
          buildrequires:
              perl: ['5.32']
          requires:
              perl: ['5.32']
          buildopts:
              rpms:
                  macros: |
                      %this_is_my_module B


And that's all.

The complete specification of modulemd-packager-v3 format is at
<https://raw.githubusercontent.com/fedora-modularity/libmodulemd/main/yaml...;.


WHEN
====

Modularity team started to work on the new format a year a ago, the
implementation was deployed to Fedora infrastructure few months ago and
I've been waiting with the announcement until fixing the most serious bugs.
That, I hope, happened last week.

In other words, you can start using the new format right now.


DO I HAVE TO MIGRATE MY MODULES?
================================

No, you don't have to. But Modularity people would like to see moving all
Fedora content there.

The old modulemd-v2 format is still supported by MBS.

DNF handles outputs of both of them (the output format is distinguished with
"static_context: true" field), but it behaves differently. With the old
format, DNF can produce weird warnings or errors. Especially when your module
is required by another module.

Will the old format be supporterd forever? I don't know.

Does have the new format some drawbacks? Yes, it has: It's longer than the
previous one. You need to update it for each new Fedora release. You cannot
run-require an unspecified (= any) module stream (perl:[]). (Though a DNF
maintainer says it should not be a problem.)


HOW TO MIGRATE
==============

Read the "WHAT IS CHANGING" section of this e-mail message. Follow the
specification if in doubts.

When migrating your modules, you will have to come up with the context value.
To preserve a compatibility with the old builds and to preserve the upgrade
path, I strongly recommend reusing the old context values. Use Koji
<https://admin.fedoraproject.org/pkgdb/package/MODULE/>;, or MBS
<https://mbs.fedoraproject.org/module-build-service/2/module-builds/?name=...;,
or "dnf module info MODULE:STREAM" command to locate the lastest version of
the module, and pick a context matching the desired dependencies as depicted here:

# dnf -q module info perl-DBI:1.643
Name             : perl-DBI
Stream           : 1.643
Version          : 3520210722203952
Context          : e12b6f3a        ------+
[…]                                      |
Requires         : perl:[5.32]     ----+ |
                   platform:[f35]  --+ | |
                                     | | |
The new YAML file:                   | | |
                                     | | |
data:                                | | |
    configurations:                  | | |
        - context: 'e12b6f3a'   <----|-|-+
          platform: f35         <----+ |
          buildrequires:               |
              perl: ['5.32']           |
          requires:                    |
              perl: ['5.32']    <------+
          buildopts:

You can validate your YAML files with
"modulemd-validator --type modulemd-packager-v3 FILE" command.

For a real example look at any perl* module. I've already migrated them. For
instance this commit
<https://src.fedoraproject.org/modules/perl-DBI/c/2200d21f0fab5295226d915c...;.


FEEDBACK
========

General help on modularity <devel(a)lists.fedoraproject.org&gt;.
Modularity design issues <https://pagure.io/modularity/issues>;.
Fedora MBS instance issues <https://pagure.io/fedora-infrastructure/issues>;.
MBS implementation bugs <https://pagure.io/fm-orchestrator/issues>;.


DISCLAIMER
==========

The examples in this document are simplified and sometimes redacted to
better illustrate the discussed topic.

I joined Modularity team way after designing this change. Therefore I cannot
guarantee that all data provided here are accurate.

I'd like to thank all the people who participated in this change and who
helped me to understand it. Especially Martin Curlej, Jaroslav Mracek,
Daniel Mach, and Mike McLean.

-- Petr

[Neal Gompa]

On Fri, Sep 10, 2021 at 8:26 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  Good news, module maintainers.
>
>  I'm relieved to announce an availability of the new module packaging format,
>  modulemd-packager, version 3.
>  
So this certainly solves a lot of issues, but it's still frustrating
to see that we *need* to declare the platform and that we couldn't
inherit the platform from the build environment. This means that at
branching time, we need to make a commit to change the requested
platform.

That being said, does the platform value influence how the final
modulemd is generated? That is, if I made a system that accepted this
YAML file as input but *ignored* the platform value (and the platform
was forced by other means), would a module build still produce working modules?




--
真実はいつも一つ！/ Always, there's only one truth!

[Petr Pisar]

V Fri, Sep 10, 2021 at 08:50:44AM -0400, Neal Gompa napsal(a):
>  On Fri, Sep 10, 2021 at 8:26 AM Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > Good news, module maintainers.
>  >
>  > I'm relieved to announce an availability of the new module packaging format,
>  > modulemd-packager, version 3.
>  >
>  
>  So this certainly solves a lot of issues, but it's still frustrating
>  to see that we *need* to declare the platform and that we couldn't
>  inherit the platform from the build environment. This means that at
>  branching time, we need to make a commit to change the requested
>  platform.
>  
I confirm it is so and I fully agree with you that it's unpleasant.

The problem is that if you want to have the context invariant, then you need
to store a mapping between a context and its build-dependencies somewhere. And
naturally as new Fedora releases are added, and end-of-live releases removed,
the mapping must be updated because platform is one of the dependencies.

>  That being said, does the platform value influence how the final
>  modulemd is generated? That is, if I made a system that accepted this
>  YAML file as input but *ignored* the platform value 
It would end up like this <https://pagure.io/fm-orchestrator/issue/1714>;: The
build service would mix contexts for different platforms and the module
wouldn't probably build at all like in the referred issue.

>  (and the platform was forced by other means) 
For example? The information has to be somewhere stored, somehow retrieved,
and delivered to a build and compose systems.

In non-modular world the information is stored in dist-git branch name,
delivered to Koji as a build target, and stored in Koji build as tag. The
information is updated with each Fedora release. The process of updating is
called branching and package retirement.

I was thinking about possible solutions:

One was omitting platform from the context. You would not specify any platform
in the module input format.
    
    - context: A

You would submit the module for building. MBS would expand the build for all
platforms (= currently supported Fedoras). The result would be four module
builds:

    Build                       Dependencies
    -------------------------------------------
    perl:5.32:362021:A          platform:36:0:0
    perl:5.32:352021:A          platform:35:0:0
    perl:5.32:342021:A          platform:34:0:0
    perl:5.32:332021:A          platform:33:0:0

But somewhere would have to be stored for which platform was the build done.
We can guess it from the version prefix (36..., 35...) now. (ELN has no
prefix.)

Then when you build perl-DBI:1.643 which build- and run-requires perl:5.32:

    - context: A
      buildrequires:
        perl: [5.32]
      requires:
        perl: [5.32]

the same way without platforms, MBS would expand a matrix of the platforms and
perls:

    perl-DBI:1.643:362022:A     platform:36:0:0 perl:5.32:362021:A
    perl-DBI:1.643:352022:A     platform:35:0:0 perl:5.32:352021:A
    perl-DBI:1.643:342022:A     platform:34:0:0 perl:5.32:342021:A
    perl-DBI:1.643:332022:A     platform:33:0:0 perl:5.32:332021:A

That would work. But how would a packager specify that he does not want to
build for all plaforms? E.g. he adds a module to Fedora 36 and is not
interested in the older ones. Or e.g. he stops maintaining the module and the
last supported is Fedora 35.

Maybe we could add a new field into the input file:

    - platforms: [f33, f34, f35]
    - context: A
      buildrequires:
        perl: [5.32]
      requires:
        perl: [5.32]

and default to all platforms if not specified.

When I shared this idea I was told that a context must change with a platform.
That everything (MBS?) assumes it. I believe I was also told another,
reasonable reason. But I cannot recall now. I'm sorry I cannot provide better
explanation now.

However, it's true that it does matter much whether the platform is encoded
into the version or into the context (perl:5.32:362021:A vs.
perl:5.32:2021:36A). If we hadn't it encoded anywhere, we could not store the
builds into Koji, and MBS:

    perl:5.32:2021:A          platform:36:0:0
    perl:5.32:2021:A          platform:35:0:0
    perl:5.32:2021:A          platform:34:0:0
    perl:5.32:2021:A          platform:33:0:0

And it's true that we would only special-case platform. It wouldn't be fair to
other modules. There must a stroger reason.

-- Petr

[Petr Pisar]

V Fri, Sep 10, 2021 at 02:26:20PM +0200, Petr Pisar napsal(a):
>  I'm relieved to announce an availability of the new module
> packaging format,
>  modulemd-packager, version 3.
>   
[...]
>  WHAT IS CHANGING
>  ================
>  
>  Since there was no place for the contexts in the old module format, a new,
>  input-only format "modulemd-packager", version 3, was designed and implemented
>  in MBS and other tools. At the opportunity of the format change, some other
>  pet peeves of the old format were solved.
>  
>  Changes in the format include:
>  
>  New document type and version:
>  
>      document: modulemd-packager
>      version: 3
>   
[...]
>  DNF handles outputs of both of them (the output format is
> distinguished with
>  "static_context: true" field), but it behaves differently. With the old
>  format, DNF can produce weird warnings or errors. Especially when your module
>  is required by another module.
>  
>  Will the old format be supporterd forever? I don't know.
>  
>  Does have the new format some drawbacks? 
A new drawback came out <https://bugzilla.redhat.com/show_bug.cgi?id=2004853>;:

Fedora 33 G.A. libmodulemd-2.9.4-3.fc33 does not yet support the new
static_context field. Because DNF strictly validates modules found in
repositories, libmodulemd will reject modules with the unknown field:

    Module yaml error: Unexpected key in data: static_context [line 9 col 3]

and will ignore the module. That implies that DNF will handle packages of
those module builds as non-modular before checking an RPM transaction:

	Installing:
	 autoconf                noarch  2.69-34.fc33                              fedora        
  666 k
	Installing dependencies:
	 perl-B                  x86_64  1.80-471.module_f33+12592+71aff0e2       
updates-modular  189 k

But then the transaction will be rejected because a strayed modular package is
found:

    Running transaction check
    No available modular metadata for modular package
'perl-B-1.80-471.module_f33+12592+71aff0e2.x86_64', it cannot be installed on the
system

In short, Fedora 33 users who have not updated their system since 2020-12-16,
won't be able to install packages which were (also) built in a module with the
new format. This affects e.g. fresh off-line installations.

A workaround is simple: update your system:

    # dnf upgrade

or explicitly upgrade libmodulemd:

    # dnf --disable-repo 'updates*modular' upgrade libmodulemd

before installing any packages.

Since we cannot change G.A. installation media (ISO and qcow2 images), there
is no remedy possible other than not pushing these updates to Fedora 33
repositories.

You have two options: Either postpone the migration to the new format until
Fedora 33 reaches its end of life (December, 2021?), or do not build the
module for Fedora 33 (i.e. remove a context section with platform: f33).

(That already happened with perl and perl-bootstrap modules. I will rebuild
them and repush to stable, unless relengs untag them.)

(That already happened with swig:4.0. Unpushing it from updates-testing should
help.)

I'm deeply sorry for this faux pas.

-- Petr

[Stephen Gallagher]

On Tue, Sep 21, 2021 at 11:50 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  V Fri, Sep 10, 2021 at 02:26:20PM +0200, Petr Pisar napsal(a):
>  > I'm relieved to announce an availability of the new module packaging format,
>  > modulemd-packager, version 3.
>  >
>  [...]
>  > WHAT IS CHANGING
>  > ================
>  >
>  > Since there was no place for the contexts in the old module format, a new,
>  > input-only format "modulemd-packager", version 3, was designed and
> implemented
>  > in MBS and other tools. At the opportunity of the format change, some other
>  > pet peeves of the old format were solved.
>  >
>  > Changes in the format include:
>  >
>  > New document type and version:
>  >
>  >     document: modulemd-packager
>  >     version: 3
>  >
>  [...]
>  > DNF handles outputs of both of them (the output format is distinguished with
>  > "static_context: true" field), but it behaves differently. With the old
>  > format, DNF can produce weird warnings or errors. Especially when your module
>  > is required by another module.
>  >
>  > Will the old format be supporterd forever? I don't know.
>  >
>  > Does have the new format some drawbacks?
>
>  A new drawback came out <https://bugzilla.redhat.com/show_bug.cgi?id=2004853>;:
>
>  Fedora 33 G.A. libmodulemd-2.9.4-3.fc33 does not yet support the new
>  static_context field. Because DNF strictly validates modules found in
>  repositories, libmodulemd will reject modules with the unknown field:
>
>      Module yaml error: Unexpected key in data: static_context [line 9 col 3]
>
>  and will ignore the module. That implies that DNF will handle packages of
>  those module builds as non-modular before checking an RPM transaction:
>  
I really wonder how DNF is getting into this situation, because we
created the API with a `strict=False` option specifically for DNF.[1]
If they were reading the documents with that set appropriately, it
would be ignoring just the `static_context` attribute rather than the
entire document.

Though as I'm writing this, I did notice we have a copy-paste
mistake[2] in the documentation where we didn't set the value of
strict to False when talking about the DNF case, so that's perhaps
part of the reason. Sorry about that!

[1]
https://fedora-modularity.github.io/libmodulemd/latest/modulemd-2.0-Modul...
[2] https://github.com/fedora-modularity/libmodulemd/pull/577

[Orion Poplawski]

On 9/10/21 6:26 AM, Petr Pisar wrote:
>  Good news, module maintainers.
>  
>  I'm relieved to announce an availability of the new module packaging format,
>  modulemd-packager, version 3.
>   
Question: is this necessary for all modules to migrate to this format, 
or is it primarily useful for modules that depend on other modules?


-- 
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems          720-772-5637
NWRA, Boulder/CoRA Office             FAX: [phone]
3380 Mitchell Lane                       orion(a)nwra.com
Boulder, CO 80301                 https://www.nwra.com/

[Petr Pisar]

V Sat, Oct 02, 2021 at 12:29:01PM -0600, Orion Poplawski napsal(a):
>  On 9/10/21 6:26 AM, Petr Pisar wrote:
>  > Good news, module maintainers.
>  > 
>  > I'm relieved to announce an availability of the new module packaging format,
>  > modulemd-packager, version 3.
>  > 
>  
>  Question: is this necessary for all modules to migrate to this format, or is
>  it primarily useful for modules that depend on other modules?
>   
I cannot provide you with any better answer than I've already stated in my
original message: It is primarily useful for modules which depends on other
modules. It is not necessary for all modules right now, because both MBS and
DNF support both of the formats. However, I won't speculate how long they will
support them.

-- Petr