Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
There are few aproaches, like compat packages, or full namespacing (Python). Yet modularity had some unique features, especially retaining nonmangled package names and other RPM dependencies.
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
Comments are welcome.
-- Petr
On Tue, Jun 13, 2023 at 12:33 PM Petr Pisar ppisar@redhat.com wrote:
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
There are few aproaches, like compat packages, or full namespacing (Python). Yet modularity had some unique features, especially retaining nonmangled package names and other RPM dependencies.
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
Comments are welcome.
-- Petr
(First the negative... don't worry, it's not all negative) I wonder why we need this at all. I believe that modularity has failed to gain a foothold in Fedora, just like scl before it, because Fedora just doesn't need this. One of the best things about Fedora is the dependency-convergence and curated nature of all the packaged versions, so that everything works out of the box. The use cases for alternative versions are edge cases that the vast majority of users do not need or care about. And the ones that most users do care about (making sure their application works) are usually satisfied with compat packaging, when needed.
So much (over-?)engineering has gone into such niche use cases, and Fedora is arguably no better off for all that effort. Nowadays, it's so easy to spin up a VM or container using older versions of things, if you need them, or you can use snaps or flatpaks, or language-specific version management tools like mvn or rvm, for the developer use cases. What is the real benefit to Fedora to continue to push down the path of alternative, mutually-exclusive streams that affect the entire system? My experience struggling with modularity as a casual packager, and as a user, has left a bad taste in my mouth for any similar attempts, and I'm highly skeptical of the need of such a thing (and bracing for the impact when it all comes crashing down on my favorite OS due to unintended consequences).
(Now the positive) That said, I like the simplicity of your approach, using metapackages and native RPM tooling to handle the situation. However niche the use case is, having such a simple approach means that it's merely a matter of documenting some guidelines for packagers to follow if they have need of that situation. I still think that the need for that will be rare, and wonder how many of these kinds of metapackages would actually get created in Fedora. I suspect that it would be very few. But the important bit is that if they do get created, they are unlikely to affect me unless I want to use them. This is very different from modularity, which seemed to have a downstream impact on everybody, especially when you were forced to use modularity if your dependencies were... which I think was enough to oust a lot of casual packagers, because they had to understand a whole new dimension of packaging guidelines and tooling. Your approach requires none of that.
The one downside to your approach, that I think was an early selling point of modularity, is that yours kind of requires that the alternative streams are built for every Fedora version. I think modularity marketed itself as being able to have a module whose version lifetimes were longer than OS release cycles (which could reduce packager workload for streams that were slow-moving and didn't change across Fedora releases). I don't know how much that actually happened in practice, though, so I don't know if anybody will care about that difference in your approach.
Is this thread on alternatives to alternatives also relevant: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... ?
Modules provide a mechanism for system level switches between alternative versions as does Petr's proposal (i.e. sudo required). As others have remarked, there is also a need for a similar capability at the user level (without sudo; e.g. environment modules) or even at the shell level.
It would be nice if there were a compendium of all similar options for Fedora.
On Tue, Jun 13, 2023 at 12:21 PM Christopher ctubbsii@fedoraproject.org wrote:
On Tue, Jun 13, 2023 at 12:33 PM Petr Pisar ppisar@redhat.com wrote:
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
best regards
Brad
V Tue, Jun 13, 2023 at 01:06:33PM -0700, Brad Smith napsal(a):
Is this thread on alternatives to alternatives also relevant: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... ?
It's relevant in the meaning that it enables to choose an implementation from multiple alternatives after building the packages, the choice is made on an end system.
However, all the aproaches discussed in the thread only deal with execve(2), or even only with execvpe(3). Anything what happens before is out of their scope.
My proposal, as well as modularity, has a larger goal. And that is make the altarnatives look-alike also on RPM level.
Modules provide a mechanism for system level switches between alternative versions as does Petr's proposal (i.e. sudo required). As others have remarked, there is also a need for a similar capability at the user level (without sudo; e.g. environment modules) or even at the shell level.
Yes. Different people require different distance of control. That's the reason why all of them exist and have their users.
It would be nice if there were a compendium of all similar options for Fedora.
If the compendium means a documentation, then I doubt a Fedora specific, comprehensive documenation would be meaningful. It's like documenting various web browsers a user could use. Moreover, I feel the Packaging Guidelines https://docs.fedoraproject.org/en-US/packaging-guidelines/Alternatives/ do not want a comprehensive list of options there. They rather document the best approach.
If the compendium means that all the frameworks should be packaged in Fedora, I would let it on organic growth. If somebody needs one, he can package and maintain it.
-- Petr
V Tue, Jun 13, 2023 at 03:20:06PM -0400, Christopher napsal(a):
(First the negative... don't worry, it's not all negative) I wonder why we need this at all. I believe that modularity has failed to gain a foothold in Fedora, just like scl before it, because Fedora just doesn't need this. One of the best things about Fedora is the dependency-convergence and curated nature of all the packaged versions, so that everything works out of the box. The use cases for alternative versions are edge cases that the vast majority of users do not need or care about. And the ones that most users do care about (making sure their application works) are usually satisfied with compat packaging, when needed.
So much (over-?)engineering has gone into such niche use cases, and Fedora is arguably no better off for all that effort. Nowadays, it's so easy to spin up a VM or container using older versions of things, if you need them, or you can use snaps or flatpaks, or language-specific version management tools like mvn or rvm, for the developer use cases. What is the real benefit to Fedora to continue to push down the path of alternative, mutually-exclusive streams that affect the entire system? My experience struggling with modularity as a casual packager, and as a user, has left a bad taste in my mouth for any similar attempts, and I'm highly skeptical of the need of such a thing (and bracing for the impact when it all comes crashing down on my favorite OS due to unintended consequences).
I agree that for Fedora it's a niche problem and any attempt to introduce alternative versions never flew long there.
My explanation for it is that what Fedora offers and what Fedora users want is a self-inforced relation. If Fedora choose a short release cycle with single-version, well-integrated code base, then naturally any attempt to bring alternative versions which partion the code is frown. There are different distributions with a different life cycle, e.g. rolling release, that incorporates incompatible versions as an inevitable phenomenon. Those distributions of course have a different vision and different user as well as packager base. This is not a critique of Fedora. It only illustrates that different distributions have different priorities.
You can take my proposal as a mental exercise to explore what is possible and what isn't with the current state of packaging in Fedora. I'm not going to hide that the proposal more targets to long-lifecycle derivates of Fedora and I wanted to get a broader audience, especially from Fedora as an upstream.
(Now the positive) That said, I like the simplicity of your approach, using metapackages and native RPM tooling to handle the situation. However niche the use case is, having such a simple approach means that it's merely a matter of documenting some guidelines for packagers to follow if they have need of that situation. I still think that the need for that will be rare, and wonder how many of these kinds of metapackages would actually get created in Fedora. I suspect that it would be very few. But the important bit is that if they do get created, they are unlikely to affect me unless I want to use them. This is very different from modularity, which seemed to have a downstream impact on everybody, especially when you were forced to use modularity if your dependencies were... which I think was enough to oust a lot of casual packagers, because they had to understand a whole new dimension of packaging guidelines and tooling. Your approach requires none of that.
The one downside to your approach, that I think was an early selling point of modularity, is that yours kind of requires that the alternative streams are built for every Fedora version. I think modularity marketed itself as being able to have a module whose version lifetimes were longer than OS release cycles (which could reduce packager workload for streams that were slow-moving and didn't change across Fedora releases). I don't know how much that actually happened in practice, though, so I don't know if anybody will care about that difference in your approach.
Thanks for the review.
Modularity indeed helped to maintain a single source across all Fedora releases. However, behind the scene it spawned separate builds and updates for each Fedora release. (That's something that current JDK proposal tackles https://fedoraproject.org/wiki/MoveFedoraJDKsToBecomePortableJDKs.) Very contrary, one had to rebuild a module for each new release. Otherwise, the module disapeared from Rawhide.
My proposal does not bring this feature. I think packagers can leverage packages.cfg file in dist-git repositories to configure multiple builds with a single "fedpkg build" command (as it was used in EPEL8 for EPEL8-next builds).
-- Petr
Hi,
Le 13/06/2023 à 18:32, Petr Pisar a écrit :
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/.
First thanks for your work on this
I also thinking a lot about this, and this is a nightmare
I must say that I'm terribly sad we have to reinvent the wheel.
I'm still a big fan of SCL and I still build them for Fedora to be able to install multiple versions simultaneously
# LANG=C dnf list php?? Installed Packages php74.x86_64 7.4-3.fc37.remi php80.x86_64 8.0-3.fc37.remi php81.x86_64 8.1-4.fc37.remi php82.x86_64 8.2-5.fc37.remi php83.x86_64 8.3-1.fc37.remi
Complexity is for packagers and users.
I'm also a big fan of modules. It just works well.
# dnf module list php Remi's Modular repository - Fedora 37 - x86_64 Name Stream Profiles Summary
php remi-7.4 common [d], devel, minimal PHP php remi-8.0 common [d], devel, minimal PHP php remi-8.1 [e] common [d], devel, minimal PHP php remi-8.2 common [d], devel, minimal PHP
Complexity is only for build system.
I will probably never understand hostility from the community against this (bad feeling because of initial broken implementation ?)
Perhaps Fedora, with its very short life, don't really needs alternative versions, while a long life distro obviously needs it.
Your approach will move complexity back to packagers.
I'm also very concerned by the "vendor" approach
It seems possible using some stream-<module>-<vendor>-<stream> (or <stream> being vendor-number, as with module)
It seems also need to manage dependency with a stack or with a specific stream.
Ex: An application will require the default stack (build and run) An extension of the stack will require the specific stream (build and run)
What about CI ?
An application will be build will one stream (default) but we probably need to test it with all available streams.
And perhaps have to use (because not compatible with 3) Requires: (stream-stack-default or stream-stack-2)
What about reviews ?
If I want to create stream-stack-3, with 100 packages ?
For now, there is a review exception for compatibility packages for "older" version, but not for newer version.
Only my initial comments
Regards, Remi
V Wed, Jun 14, 2023 at 10:19:27AM +0200, Remi Collet napsal(a):
Your approach will move complexity back to packagers.
Yes. That's because the problem has a complexity and the complexity needs to live somewhere. With modularity it was in MBS and in modular filtering part of DNF. If you remove modularity, the complexity has to emerge again somewhere. With my approach the somwehere is new metapackages, new dependencies, and new build targets.
I'm also very concerned by the "vendor" approach
It seems possible using some stream-<module>-<vendor>-<stream> (or <stream> being vendor-number, as with module)
I don't understand it. Do you mean that anybody can add his own stream, or sqaut a namespace? Yes, he can. It was possible even in modularity. Here it is worse because those identifiers are free-style package names. There are no separate fields or entities designated for <module> and <stream> anymore.
It seems also need to manage dependency with a stack or with a specific stream.
Ex: An application will require the default stack (build and run)
If the application ignores nondefault stacks, no action is required. That's why the proposal is so obsesed with default streams. I think this will be the most prevaling case. E.g. automake is written in Perl and an automake maintainer does not care whether it works with alternative Perls or not. He does not have to. The same attitude was used in modularity.
An extension of the stack will require the specific stream (build and run)
The extension can add build- and run-time dependenices on packages from that stream. However, that will effectively make the extension part of the stream. In that case a maintainer of the extension should including it into the stream. That, by a the proposed recommendations, would change an RPM name of the extension. I guess that Fedora would insist on this name change as Fedora's packaging guidelines now insist on nonmodular packages to only depend on nonmodular content.
On the other hand, I can imagine that packages (especially top-level applications) which are not a logical part of any specific stream but depends on that nondefault stream, will retain its original name and simply users will have to switch the stream with "--allowerasing". It's the fragmentation of well-integrated Fedora Christopher (?) wrote in this thread. In his eyes there is no need for applications like that.
What about CI ?
CI would have to adapt to switch the streams. As in case of modularity. As far as I know Fedora does not supports modules in CI. But the switching can happen as an implicit part of CI environment installation. If CI populated the enviromemnt with "dnf --root=... install <THE_TESTED_PACKAGE> <PACKAGES_USED_BY_CI>, then DNF would pick the fitting streams automatically. Or if there were no solution, DNF would report a dependency resolution error.
An application will be build will one stream (default) but we probably need to test it with all available streams.
That would be difficult to automate. Packagers could configure CI to repeat tests for listed streams. Modularity was better for automation because streams were a formal entity which could be queried and set. In the my approach there is no such capability because simply there is no modular entity. That's definitely a regression.
Howver, in reality the automation even in RHEL was not so hot. As far as I know a list of stream combinations to test was always defined statically.
And perhaps have to use (because not compatible with 3) Requires: (stream-stack-default or stream-stack-2)
Rich dependencices are one option. Another option is to leverage ordering. Because the packages from streams provide the same base name with different versions:
php74 Provides: php = 7.4 php80 Provides: php = 8.0 php82 Provides: php = 8.2
then you can write "Requires: php >= 8.0" to exclude php-7.4.
Now I'm not sure whether versioning stream-stack from your question wouldn't break default streams. I forgot whether DNF prefers a package with highest provider or a package with highest version and whether the versioning is stronger than weak dependencies or not.
What about reviews ?
If I want to create stream-stack-3, with 100 packages ?
For now, there is a review exception for compatibility packages for "older" version, but not for newer version.
Good catch. Fedora would have to change rules to recognize that stack3 is an upgrade of stack2. Or those could live in stream branches of dist-git where they hide in repository under the base name.
Frankly I haven't yet thought on these policies Fedora asserts. So far I thinking on feasibily from DNF and Koji point of view.
-- Petr
Le 14/06/2023 à 14:14, Petr Pisar a écrit :
V Wed, Jun 14, 2023 at 10:19:27AM +0200, Remi Collet napsal(a):
Your approach will move complexity back to packagers.
Yes. That's because the problem has a complexity and the complexity needs to live somewhere. With modularity it was in MBS and in modular filtering part of DNF.
For memory, all RPMs (even modular ones) are build with rpmbuild ;) mbs is only the "common" way, but not the "simple" way to build modular packages ;)
An extension of the stack will require the specific stream (build and run)
The extension can add build- and run-time dependenices on packages from that stream. However, that will effectively make the extension part of the stream. In that case a maintainer of the extension should including it into the stream. That, by a the proposed recommendations, would change an RPM name of the extension. I guess that Fedora would insist on this name change as Fedora's packaging guidelines now insist on nonmodular packages to only depend on nonmodular content.
Yes, make sense I was thinking of all C extensions (php-pecl-* and some other) which have a dependency on php(zend-abi) which is related to version, and possibly build options (nts, zts, debug...)
On the other hand, I can imagine that packages (especially top-level applications) which are not a logical part of any specific stream but depends on that nondefault stream, will retain its original name and simply users will have to switch the stream with "--allowerasing". It's the fragmentation of well-integrated Fedora Christopher (?) wrote in this thread. In his eyes there is no need for applications like that.
Yes, for now we use dependency on php(language) Usually we only require for minimal version (ex: php(language) >= 8.0) but sometime range dependency make sense (ex: roundcubemail which explicitly have a test for maximal version)
BTW, my preferred solution: keep building real modular packages ;) don't reinvent the wheel. In all cases, Fedora community won't like it.
Remi
On Wed, 2023-06-14 at 10:19 +0200, Remi Collet wrote:
Hi,
Le 13/06/2023 à 18:32, Petr Pisar a écrit :
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/.
First thanks for your work on this
I also thinking a lot about this, and this is a nightmare
I must say that I'm terribly sad we have to reinvent the wheel.
I'm still a big fan of SCL and I still build them for Fedora to be able to install multiple versions simultaneously
# LANG=C dnf list php?? Installed Packages php74.x86_64 7.4-3.fc37.remi php80.x86_64 8.0-3.fc37.remi php81.x86_64 8.1-4.fc37.remi php82.x86_64 8.2-5.fc37.remi php83.x86_64 8.3-1.fc37.remi
Complexity is for packagers and users.
I'm also a big fan of modules. It just works well.
# dnf module list php Remi's Modular repository - Fedora 37 - x86_64 Name Stream Profiles Summary
php remi-7.4 common [d], devel, minimal PHP php remi-8.0 common [d], devel, minimal PHP php remi-8.1 [e] common [d], devel, minimal PHP php remi-8.2 common [d], devel, minimal PHP
Complexity is only for build system.
I will probably never understand hostility from the community against this (bad feeling because of initial broken implementation ?)
In my point of view, modularity goal is have a king SCL (softwarecollections.org), but better, in which allows users have updated versions of some software, because is impossible keep same version of all packages for 10 years, which is the lifetime of RHEL 7, for example. These modules are just an exception thats allow users to run one killing application like Gimp or whatever.
But the problem was, everyone started building modules of everything , and that is not the goal of modularity (IMO), do not make sense to me have lastest commit of libpng or whatever since these libs are alerady stable and common user don't need updates. This brings (IMO) a big fragmentation of we can test and how to test. Modules should always be stable things that we haven't already in our environment, and an opt- out in exceptions and not an opt-in.
In my point of view, the main goal of modules is for long term distributions like RHEL(s) where for example on Centos 8 Stream, we could have ImageImagick 7 as module, and users have ImageMagick updated which is already needed by another software and run it in an "old" distribution (at the moment they can't).
Perhaps Fedora, with its very short life, don't really needs alternative versions, while a long life distro obviously needs it.
Your approach will move complexity back to packagers.
I'm also very concerned by the "vendor" approach
It seems possible using some stream-<module>-<vendor>-<stream> (or <stream> being vendor-number, as with module)
It seems also need to manage dependency with a stack or with a specific stream.
Ex: An application will require the default stack (build and run) An extension of the stack will require the specific stream (build and run)
What about CI ?
An application will be build will one stream (default) but we probably need to test it with all available streams.
And perhaps have to use (because not compatible with 3) Requires: (stream-stack-default or stream-stack-2)
What about reviews ?
If I want to create stream-stack-3, with 100 packages ?
For now, there is a review exception for compatibility packages for "older" version, but not for newer version.
Only my initial comments
Regards, Remi
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi Petře,
Petr Pisar ppisar@redhat.com writes:
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
There are few aproaches, like compat packages, or full namespacing (Python). Yet modularity had some unique features, especially retaining nonmangled package names and other RPM dependencies.
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
I like the proposal, it gives us nearly all the benefits of modularity without the complexity introduced by modular repos and MBS.
I have one question about this part:
The only drawback is one have to decide before GA which software will have an alternative content and create meta-packages for the default streams. Otherwise, users installing from GA media and upgrading later could get installed a nondefault stream.
Why is that?
Wouldn't the process work as follows: - you create your packages, the metapackage and add them to fedora-release - users upgrading from GA will get the new fedora-release with the proper Suggests: - users installing from a respin will have the proper Suggests: from the beginning
The only way that I see how this could go sideways is the scenario where you switch what the default stream is after GA. But I thought that was forbidden by policy anyway?
Thanks for this proposal!
Dan
V Wed, Jun 14, 2023 at 10:29:31AM +0200, Dan Čermák napsal(a):
Petr Pisar ppisar@redhat.com writes:
The only drawback is one have to decide before GA which software will have an alternative content and create meta-packages for the default streams. Otherwise, users installing from GA media and upgrading later could get installed a nondefault stream.
Why is that?
My wording was not completely correct. I'm sorry. The problem is what people mean with "upgrade".
"dnf upgrade" is fine.
"dnf upgrade application" is not fine. "dnf install application" is also not fine. I will start with explaining simpler "dnf install application":
- A user has installed fedora-release which never heard about stack. - There is an application in a repository which requires stack. - The user installs the application and stack is pulled in as a dependency.
Now the same procedure when the first step is before adding stack2 into the repository, but the installation happens after that:
- A user has installed fedora-release which never heard about stack. - The distributor adds stack2, an alternative to stack (Provides: stack = 2), and updates fedora-release to suggest stream-stack-default. - The user installs the application. As a result stack2 is pulled in. The reason is that stack2 provides "stack = 2" which is higher than "stack = 1" provided by stack package.
Now the user end up with the same application but stack2 instead of stack.
A situation with "dnf upgrade application" would be fine if application depended on stack from the very beginning. But what if the dependency on stack was added to application in between:
- A user has installed fedora-release which never heard about stack. - User has installed application. No stack was pulled in because there was no need for it. - The distributor adds stack2, an alternative to stack (Provides: stack = 2), and updats fedora-release to suggest stream-stack-default. The distributor also updates application to depend on stack. - The user updates the application. As a result stack2 instead of stack is installed. The reason is the same very same as in "dnf install application" case -- DNF could choose beetween stack and stack2, so it choose the one with a higher version.
Frankly, I would consider automatic selection of the highest alternative to be a welcome feature, not a bug. At the end, the user only requested application, so he does not care about a version of stack. If the stack version mattered, the user should have specified it explicitly. E.g. "dnf install application stack".
But things then get complicated when users split transactions into multiple steps. E.g.:
# dnf install stack # dnf install application
would end with application and stack, but:
# dnf install application # dnf install stack
would end with application, stack2 and then an error "stack conflicts with stack2".
DNF operations are not commutative. And people expect to have their installation procedures reproducible.
You might also ask why would somebody invoke "dnf upgrade application" instead of "dnf upgrade". Well, some people read security alerts, evaluate a risk, and then explicitly update selected packages. That's how they end up with "dnf upgrade application".
-- Petr
In general, I like the proposal, while it probably has some gaps, e.g. how the code will be organized in dist git.
And another specific concern is: "I recommend abusing a distribution release package (e.g. fedora-release) which is in a default installation:". I think it would probably make sense to separate this into independent package to ease the management (and possibly to opt out from this feature? Not sure about this)
Vít
Dne 13. 06. 23 v 18:32 Petr Pisar napsal(a):
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
There are few aproaches, like compat packages, or full namespacing (Python). Yet modularity had some unique features, especially retaining nonmangled package names and other RPM dependencies.
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
Comments are welcome.
-- Petr
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Petr Pisar wrote:
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
So the linked approach emulates Modularity with its main issues (in particular, the risk of dependency version conflicts ("RPM hell") between packages, due to the mutual exclusiveness of the different versions of a "stack") by massive abuse of the Conflicts RPM tag (which is frowned upon for exactly the aforementioned reason) and without an opt-out (because you want to drop these packages into the main repository rather than the optional modular one that is now going to be disabled by default for a good reason). I fail to see the improvement from that proposal.
If you want to build packages with this (selectable mutually exclusive versions) approach, please keep using the normal Modularity and accept that users now have to explicitly opt in to those modules. And in particular, please accept that non-modular packages are not allowed to require one of those modules (or in particular, a specific version thereof). Attempting to work around those requirements (designed to protect users from version conflicts they cannot resolve) is a very bad idea.
If you really need a package to depend on a specific version of a library, that library version should be packaged as a parallel-installable compatibility library as per the packaging guidelines.
Kevin Kofler
V Thu, Jun 15, 2023 at 01:33:38AM +0200, Kevin Kofler via devel napsal(a):
Petr Pisar wrote:
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/. The linked approach achieves it at the expense of dedicated build targets and an inability to introduce completely new modules (as opposite to new streams of existing modules) after releasing an installation media.
So the linked approach emulates Modularity with its main issues (in particular, the risk of dependency version conflicts ("RPM hell") between packages, due to the mutual exclusiveness of the different versions of a "stack") by massive abuse of the Conflicts RPM tag (which is frowned upon for exactly the aforementioned reason) and without an opt-out (because you want to drop these packages into the main repository rather than the optional modular one that is now going to be disabled by default for a good reason). I fail to see the improvement from that proposal.
You got it right. It is to emulate Modularity features without Modularity.
If you want to build packages with this (selectable mutually exclusive versions) approach, please keep using the normal Modularity
I prefer using the normal Modularity. However, there is a strong pressure to remove it from Fedora. See https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular and https://fedoraproject.org/wiki/Changes/RetireModularity.
and accept that users now have to explicitly opt in to those modules.
That's exactly what killed the current modularity. Nobody felt a presure to fix the tooling because Modularity was off by default.
And in particular, please accept that non-modular packages are not allowed to require one of those modules (or in particular, a specific version thereof). Attempting to work around those requirements (designed to protect users from version conflicts they cannot resolve) is a very bad idea.
I understand the horror that you can have two packages which cannot be installed at the same time. The same as you cannot start httpd and nginx services at the same time. But now the conflict is visible on RPM level.
I think it's a matter of the policy. Not of the underlying technology. Technically you could build and push into repositories a nonmodular package which depended on modular packeges even in the old Modularity. It were only people who did not do it and policies which were set against it.
If you really need a package to depend on a specific version of a library, that library version should be packaged as a parallel-installable compatibility library as per the packaging guidelines.
Of course. My proposal does not forbids it. If users of the parallel-installable packages are fine with rewriting all their RPM dependcies and file locations in their applications, then parallel-installable packages are a perfect solution. It's simply a different problem with a different solution.
-- Petr
On Thu, 15 Jun 2023 at 03:04, Petr Pisar ppisar@redhat.com wrote:
V Thu, Jun 15, 2023 at 01:33:38AM +0200, Kevin Kofler via devel napsal(a):
Petr Pisar wrote:
and accept that users now have to explicitly opt in to those modules.
That's exactly what killed the current modularity. Nobody felt a presure to fix the tooling because Modularity was off by default.
I don't think anyone felt pressure to fix it before it was off by default. Fixes to problems with Fedora MBS were always 'coming real soon' for multiple releases before this decision was made. There have been multiple outages or slowdowns in the Fedora Build System over the years due to MBS issues which only got 'fixed' by herculean last minute efforts.
The truth is that this has been a vicious loop from the beginning. There was a general feeling with many developers inside and outside of Red Hat that no matter what problems they saw, they were going to have to like it or lump it. People who were proponents from the start got burned out either by the lack of fixes to the tooling or community pushback. People working on it felt like they were having to put parts of it in before it was ready but also left to handle the fallout from things not working well.
Reviewing the past 3+ years, I would say that modularity would only work well if the entire 'build system' and work flow around it was built with that in mind. The Fedora system had nearly 20 years of established work flows, tooling and ideas when modularity was added and it never worked well. The CentOS Stream build system was designed from scratch but with many of the tools Fedora already used, and MBS worked better but still has had many issues of square peg in round hole. Remi's modularity seems to have worked best because of at least two things: 1. He designed his entire build workflow to work with his modularity system. He redesigned how he was going to package and how he was going to deal with modules with tools which fit his workflow. 2. He controls what goes into his build system, when it goes into, and how packages look. 3. When it breaks, he is the only person affected by any breakage (aka there aren't 4000 developers trying to build other things at the same time).
In many ways I think that the first 2 items are the core to making modularity work anywhere.
Petr Pisar wrote:
I understand the horror that you can have two packages which cannot be installed at the same time. The same as you cannot start httpd and nginx services at the same time. But now the conflict is visible on RPM level.
You can actually, if you set them to different ports.
But the issue here is that you can have 2 completely unrelated packages clashing over the version of some transitive dependency. Imagine not being able to install a KDE application on GNOME or vice-versa because of the KDE and GNOME stacks requiring a conflicting version of some "plumbing-layer" library. That is exactly the kind of scenario I and others want to avoid.
Of course. My proposal does not forbids it. If users of the parallel-installable packages are fine with rewriting all their RPM dependcies and file locations in their applications, then parallel-installable packages are a perfect solution. It's simply a different problem with a different solution.
It is a different solution indeed, but I disagree about it solving a different problem. It solves the *same* problem in a different, better (from the end user's perspective – I know it is more work for the packager) way.
Kevin Kofler
On Sat, Jun 17, 2023 at 9:12 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Petr Pisar wrote:
I understand the horror that you can have two packages which cannot be installed at the same time. The same as you cannot start httpd and nginx services at the same time. But now the conflict is visible on RPM level.
You can actually, if you set them to different ports.
But the issue here is that you can have 2 completely unrelated packages clashing over the version of some transitive dependency. Imagine not being able to install a KDE application on GNOME or vice-versa because of the KDE and GNOME stacks requiring a conflicting version of some "plumbing-layer" library. That is exactly the kind of scenario I and others want to avoid.
Of course. My proposal does not forbids it. If users of the parallel-installable packages are fine with rewriting all their RPM dependcies and file locations in their applications, then parallel-installable packages are a perfect solution. It's simply a different problem with a different solution.
It is a different solution indeed, but I disagree about it solving a different problem. It solves the *same* problem in a different, better (from the end user's perspective – I know it is more work for the packager) way.
I question the "more work for the packager" viewpoint. I'm not sure that's necessarily true, especially for casual packagers. From my perspective, any Fedora guidelines that I can pull directly from Fedora packaging documentation and merely have to modify a SPEC file, is *far* less work for me than understanding a whole separate MBS system and related tooling. Even if it's technically more "work" (time consuming or tedious), the work is much easier when it's building on existing knowledge. It's really about comprehensibility, and the degree to which you can capitalize on existing knowledge, that contributes to how much work is needed to use one solution vs. another.
Kevin Kofler
On 13/06/2023 18:32, Petr Pisar wrote:
Comments are welcome.
Trying to reinvent Fedora Modularity without modules with the same problems as before.
Instead of using modules, we should use parallel-installable packages.
On Tue, Jun 13, 2023 at 06:32:45PM +0200, Petr Pisar wrote:
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/.
Thank you for the detailed proposal that tries to explore all the corner cases. This looks like it should work. It'd be nice to update "postgresql demodularization pr" [1] to this scheme to test it on a real live package.
[1] https://src.fedoraproject.org/rpms/postgresql/pull-request/59
Zbyszek
Il 20/06/23 17:31, Zbigniew Jędrzejewski-Szmek ha scritto:
[1] https://src.fedoraproject.org/rpms/postgresql/pull-request/59
Zbyszek
Isn't the package name required to match the name of specfile?
Mattia
V Tue, Jun 20, 2023 at 03:47:13PM +0000, Mattia Verga via devel napsal(a):
Il 20/06/23 17:31, Zbigniew Jędrzejewski-Szmek ha scritto:
[1] https://src.fedoraproject.org/rpms/postgresql/pull-request/59
Zbyszek
Isn't the package name required to match the name of specfile?
It is.
But you can work around it by keeping the main package nonexisten (i.e. moving all files from "%files" section to "%files -n MY_FUNNY_NAME" section and removing "%files" section). Source package would retain main package name, but there wouldn't be any binary package of that name.
But I'm not sure whether relengs won't protest that they have multiple source packages with the same name in a sources repository.
-- Petr
V Tue, Jun 20, 2023 at 03:31:27PM +0000, Zbigniew Jędrzejewski-Szmek napsal(a):
On Tue, Jun 13, 2023 at 06:32:45PM +0200, Petr Pisar wrote:
I spent some time thinking how to approximate the nice features with current state of RPM, Koji, and DNF and come up with this approach https://ppisar.fedorapeople.org/postmodular/.
Thank you for the detailed proposal that tries to explore all the corner cases. This looks like it should work. It'd be nice to update "postgresql demodularization pr" [1] to this scheme to test it on a real live package.
[1] https://src.fedoraproject.org/rpms/postgresql/pull-request/59
I can do it, but:
Current policy requires building Rawhide builds from rawhide branch. (Hence I assumed that alternative streams will have separarate dist-git reposirories. Somebody already correctly pointed that dedicated repositories with obligate reviews would be too bureaucratic.)
My proposal requires metapackages which should have a dedicated component. The reason is that the metapackage of a default stream needs to be installed in @build group, i.e. before building the first content package and for building the content package. (I could cram the metapackage into the same spec file with postgresql by bootstrapping, but I guess from process point of view it's uggly.) Also the metapackage needs to be required from fedora-release.
Currently you cannot build my proposal in Koji, because current Koji configures mock not to pass --allowerasing option to "dnf builddep" command. And current Mock actually disabled the option for DNF5 https://github.com/rpm-software-management/mock/pull/1027 because DNF5 did not support it until very recently.
And finally, I found a misfeature in DNF where "dnf install" intentionally ignores RPM Provides https://github.com/rpm-software-management/dnf5/issues/620. If DNF maintainers do not change their mind, my proposal will be buildable after fixing https://github.com/rpm-software-management/dnf5/issues/626, but an installation with a base, no-stream name won't work. Someone invoking "dnf install" would always have to use the real package name with a stream suffix/prefix. (That bug does not affect transitive dependencies.)
And from practical point of view, postgresql is not a good toy package. Most of the time it fails to build.
-- Petr
Le 13/06/2023 à 18:32, Petr Pisar a écrit :
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
Another way/proposal
Keep "modularity", but drop MBS
1/ create a stream package which defines few needed stuff
mostly - %dist => .module+name+stream.distro - %modularitylabel
and possibly the .yaml template
2/ modify createrepo
so all the packages with modularitylabel=name:stream:* are be part of name:stream module
Done.
And we have something which works and have been heavily tested
Yes this is a 1 level only modularity.
P.S. §1 is what MBS does somehow magically and packager know what to build, in which order, something that MBS also try to magically compute
V Wed, Jun 21, 2023 at 11:52:36AM +0200, Remi Collet napsal(a):
Le 13/06/2023 à 18:32, Petr Pisar a écrit :
Hello,
as it seems that module build infrastructure isn't getting any better, as modular YUM repositories are going to be deconfigured https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular, there is a time to look at different ways how to package alternative content.
Another way/proposal
Keep "modularity", but drop MBS
1/ create a stream package which defines few needed stuff
mostly
- %dist => .module+name+stream.distro
- %modularitylabel
That stream package would have to be registered as an optional member of @build-srpm group and because the package is specific to a stream, the package would have to be kept away from nonmodular build tag. Packagers would probaly tag the stream package into their side tags dedicated for their streams.
and possibly the .yaml template
To enforce reprodusibility and auditability, Koji to has a rule that all input is first commited into dist-git and then a Koji task loads it from there and a packager has no way to influence it. A module-build task in Koji would have to be changed to implement a logic for handling the YAML templates. With MBS it was MBS service which implemented the logic and then imported the output as a module build into Koji.
The same goes for the stream package. I have no idea how MBS builds module-build-macros in Koji. A task info reads "Src: cli-build/....src.rpm". Probably a privileged operation which does not involve dist-git.
2/ modify createrepo
so all the packages with modularitylabel=name:stream:* are be part of name:stream module
Or modify createrepo_c to export the modularitylabel into primary.xml. The YAML files, if ever needed, would be synthesized by DNF.
My largest problem with this approach is a manual management of the tags on the modular builds. Koji does not have a way how to be asked for all builds belonging to a stream. MBS worked around it by using ephemeral tags for building each stream and then maintaining a registry of these tags and their builds for a later use (e.g. for updating the stream).
Done.
And we have something which works and have been heavily tested
Yes this is a 1 level only modularity.
I worry that even this user-side-only modularity is unwelcome https://fedoraproject.org/wiki/Changes/No_default_fedora-repos-modular.
-- Petr