Hello fellows,
After last publication on LWN about Fedora Modularity mess, I think it is time to describe the idea I was proposing internally with few other folks (Adam Samalik, Brian Exelbierd) back in the RH times.
Before I actually go deep, I'll try to answer main questions to myself (so that you can understand why I am proposing this particular thing).
1. Do we want to package multiple streams only for "leaf" software or any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could: * Need to create multiple repos for different "streams" * Need to maintain epel7/epel8/f30/f31/master branches * Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is manual work * However, those are supposed to be (according to the guidelines) parallel-installable (and not be conflicting in any way) Doing git merge / git cherry-pick and maintaining many git remotes requires some advanced git knowledge and a time. And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats). We do have some kind of a solution for multiple releases building from one branch (package.cfg), however this work has been never finished, thus there are many problems with this approach.
2. Do we want to support buildtime-only packages? I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
3. Can we have different lifecycles for the software in Fedora? Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
4. Do we want to have some kind of "stream expansion" where software builds against all Pythons, Perls and whatnot? I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
5. Are we still trying to be a Linux distribution or we are just letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
===
How I would imagine having multiple streams: * rpms/nodejs has multiple branches - 10, 11, 12 * in all of them, nodejs.spec exists as-is without "mangled" name (just with different versions and such) * each has package.cfg or some alternative which specifies what is the EOL and/or for which releases to build * when builds are submitted, Koji automatically adds suffix to a main package and subpackages containing branch name * during the rpmbuild, extra provides are added (for the unversioned names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
Notes: * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10` * If that requires some conflicting dependency to be switched, it will be switched automatically * Packages produced from nodejs.src have Provides: stream(nodejs) stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time * If desired, packages can depend on a specific stream via Requires: stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv) * In the very similar way, user can lock themselves to a specific stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well) * DNF can be teached about these special things so that you can automatically swap conflicting dependencies, but lock yourself to some streams of a package * All standard Conflicts/Requires/Obsoletes/… will simply work * With side tags on demand it is now much easier * Mass rebuild scripts will be teached about stream branches and will build packages properly * After branching, there will be a script which untags all packages which should not be supported in the new release * fedpkg will be teached about stream branches and will show which branches build where, so that maintainers can easily maintain them * Packaging guidelines should be adopted to accept conflicting packages and tooling should be improved to show the conflicts and how to resolve them * fedora-release will have Suggests: stream(nodejs:10) so that when user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12) * Some macros can be added to generate virtual package with "default stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release * Going further, I would extend Koji with fedora plugin which would deal with the build submission instead of having this logic on clients so that all people get same result
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
1. Users should have alternate streams of software available. I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
2. Those alternate streams should be able to have different lifecycles. nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
3. Packaging an individual stream for multiple outputs should be easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
===
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
As a meta-goal, we should break up "Modularity" into a number of separate components, some build-side, some client-side. Modularity suffers greatly from trying to encode everything into one document. This greatly raises the complexity of the task and makes it hard to consider alternative proposals for various parts.
Let's consider them separately:
1. what to build against what branches
You said:
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
Right now we use dist-git branches for this. Having a package.cfg might sound easier , but the caveat is that building against different branches requires tweaks sometimes. So we immediately hit a choice: do we want a single branch and the %ifdef mess, or multiple branches and the cherry picking troubles.
Right now, big disadvantage of multiple branches is that %changelogs and Release bumps introduce trivial conflicts. I think we should explore the proposals (incl. your and Neal's) to make this better. Fixing this would be good also for normal packaging, because it'd remove one or two more steps that need to be done every time.
We should explore both directions, i.e. package.conf and easier branches.
We should experiment with better/easier control of what to build where. This would benefit normal packaging too. Right now our tooling is just hard to use, with multiple commands in different tools needed to start builds, schedule updates, control koji status, etc. We need changes for "normal" packages as much as for "streams".
2. how to deliver rpms to users and how to let them control installed versions
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
Yes. The crucial part is that the rpms that are delivered are just normal rpms. All the complications are kept on the build-side and the client side doesn't need to care if this magic happened, the same result could have been achieved with manually mangled spec file.
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically ...
- All standard Conflicts/Requires/Obsoletes/… will simply work
Yes. More of "just normal packages".
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
... and here I'm not sure. I'm not convinced we need to try to manage all this like that. We already have packages which Conflict, or specify conflicting Requires, and the user gives a dnf command, and a certain transaction is prepared for them. The transaction will occasionally specify a list of rpms to remove. If the user doesn't like this, they can cancel and give a different command.
Specifically, some streams might conflict, other might not, etc. We could have a macro that helps to declare this in one line, but let the maintainers of the rpms declare this as needed.
3. [your 2.] Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. [...]
- After branching, there will be a script which untags all packages
which should not be supported in the new release
Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
A different proposal: we add a new rpm flag (maybe as a special Requires: line) that specifies that a package is only supported as a build-time dependency. DNF could then gain a trivial patch that it would ask the user "Do you want to install those unsupported packages? If you do, please do not file bugs, because ...". The advantages: a) packages are available for local builds and downstream rebuilds b) no problem with mock and koji being different c) we avoid any perception of "welding down the engine hood".
This would just acknowledge the truth that there are many packages in Fedora that are not supported beyond an occasional version bump and rebuild, giving better transparency for the users in general.
4.
- Can we have different lifecycles for the software in Fedora?
a.k.a. Should we allow more in-release upgrades?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
We can (and do) encode this information directly in dist-git. No need to provide a meta-mechanism. Even if package.conf becomes a thing, this would contain this information.
I think we should allow "rolling" versions more widely, like we do with firefox, the kernel, spam blockers, and probably some other things. As long some piece software is most backwards compatible, I think we'd be better of abandoning the strict guidelines we have now. This depends on each specific package though.
If this happens, it should be as a FPC guideline change, discussed and specified independently of the tooling changes. If we detach it from the technical side, answer the questions when and if at all we allow this will be clearer.
5.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot?
I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
Right now this would be prohibitively expensive. But maybe this could become much easier if points 1-2 and 4 were better answered. So I think we should shelve this question for now.
6. How to deal with packages which need outdated BuildRequires
This is another part of Modularity, and you proposal glosses over this. But you said:
- With side tags on demand it is now much easier
... and I fully agree here. There was https://fedoraproject.org/wiki/Changes/OnDemandSideTags which plays with this idea. I think side-tags could/should be the basis of a solution.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
I don't have an answer for this.
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
I think all this sounds reasonable. The main part is that it does not require massive changes in the koji or dnf, and that it integrates in a backwards-compatible way with what we have now.
Let's split the problem into chunks and discuss solutions.
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
Zbyszek
On Tue, Nov 26, 2019 at 05:35:27PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
As a meta-goal, we should break up "Modularity" into a number of separate components, some build-side, some client-side. Modularity suffers greatly from trying to encode everything into one document. This greatly raises the complexity of the task and makes it hard to consider alternative proposals for various parts.
Let's consider them separately:
- what to build against what branches
You said:
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
Right now we use dist-git branches for this. Having a package.cfg might sound easier , but the caveat is that building against different branches requires tweaks sometimes. So we immediately hit a choice: do we want a single branch and the %ifdef mess, or multiple branches and the cherry picking troubles.
Yeah, this is a difficult thing to answer, since there's people in both camps right now happy with either end, and lots in between.
Right now, big disadvantage of multiple branches is that %changelogs and Release bumps introduce trivial conflicts. I think we should explore the proposals (incl. your and Neal's) to make this better. Fixing this would be good also for normal packaging, because it'd remove one or two more steps that need to be done every time.
+1
We should explore both directions, i.e. package.conf and easier branches.
We should experiment with better/easier control of what to build where. This would benefit normal packaging too. Right now our tooling is just hard to use, with multiple commands in different tools needed to start builds, schedule updates, control koji status, etc. We need changes for "normal" packages as much as for "streams".
I wonder if we couldn't explore using tags for this. ie, if you tag a commit in a specific way it tells the system what to build, what to run ci on, etc.
...snip...
A different proposal: we add a new rpm flag (maybe as a special Requires: line) that specifies that a package is only supported as a build-time dependency. DNF could then gain a trivial patch that it would ask the user "Do you want to install those unsupported packages? If you do, please do not file bugs, because ...". The advantages: a) packages are available for local builds and downstream rebuilds b) no problem with mock and koji being different c) we avoid any perception of "welding down the engine hood".
This would just acknowledge the truth that there are many packages in Fedora that are not supported beyond an occasional version bump and rebuild, giving better transparency for the users in general.
Well, if we are doing that, couldn't we just not add bugzilla components for them? But perhaps we do want the bugs, even if the package isn't 'maintained' wouldn't it still be good to know about bugs?
...snip...
I think we should allow "rolling" versions more widely, like we do with firefox, the kernel, spam blockers, and probably some other things. As long some piece software is most backwards compatible, I think we'd be better of abandoning the strict guidelines we have now. This depends on each specific package though.
Yes, I think this very much depends on the package.
kevin
On Sat, Nov 30, 2019 at 9:28 PM Kevin Fenzi kevin@scrye.com wrote:
On Tue, Nov 26, 2019 at 05:35:27PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
As a meta-goal, we should break up "Modularity" into a number of separate components, some build-side, some client-side. Modularity suffers greatly from trying to encode everything into one document. This greatly raises the complexity of the task and makes it hard to consider alternative proposals for various parts.
Let's consider them separately:
- what to build against what branches
You said:
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
Right now we use dist-git branches for this. Having a package.cfg might sound easier , but the caveat is that building against different branches requires tweaks sometimes. So we immediately hit a choice: do we want a single branch and the %ifdef mess, or multiple branches and the cherry picking troubles.
Yeah, this is a difficult thing to answer, since there's people in both camps right now happy with either end, and lots in between.
Right now, big disadvantage of multiple branches is that %changelogs and Release bumps introduce trivial conflicts. I think we should explore the proposals (incl. your and Neal's) to make this better. Fixing this would be good also for normal packaging, because it'd remove one or two more steps that need to be done every time.
+1
We should explore both directions, i.e. package.conf and easier branches.
We should experiment with better/easier control of what to build where. This would benefit normal packaging too. Right now our tooling is just hard to use, with multiple commands in different tools needed to start builds, schedule updates, control koji status, etc. We need changes for "normal" packages as much as for "streams".
I wonder if we couldn't explore using tags for this. ie, if you tag a commit in a specific way it tells the system what to build, what to run ci on, etc.
IMO we should be building and testing all commits automatically unless maintainer put some comment like "[skip ci]" in the commit message.
...snip...
A different proposal: we add a new rpm flag (maybe as a special Requires: line) that specifies that a package is only supported as a build-time dependency. DNF could then gain a trivial patch that it would ask the user "Do you want to install those unsupported packages? If you do, please do not file bugs, because ...". The advantages: a) packages are available for local builds and downstream rebuilds b) no problem with mock and koji being different c) we avoid any perception of "welding down the engine hood".
This would just acknowledge the truth that there are many packages in Fedora that are not supported beyond an occasional version bump and rebuild, giving better transparency for the users in general.
Well, if we are doing that, couldn't we just not add bugzilla components for them? But perhaps we do want the bugs, even if the package isn't 'maintained' wouldn't it still be good to know about bugs?
I think we still should create BZ (or have it opt-out and have issues on pagure?) because it is useful to get bugreports about some CVE and whatnot.
...snip...
I think we should allow "rolling" versions more widely, like we do with firefox, the kernel, spam blockers, and probably some other things. As long some piece software is most backwards compatible, I think we'd be better of abandoning the strict guidelines we have now. This depends on each specific package though.
Yes, I think this very much depends on the package.
kevin _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Nov 26, 2019 at 6:35 PM Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl wrote:
As a meta-goal, we should break up "Modularity" into a number of separate components, some build-side, some client-side. Modularity suffers greatly from trying to encode everything into one document. This greatly raises the complexity of the task and makes it hard to consider alternative proposals for various parts.
Let's consider them separately:
- what to build against what branches
You said:
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
Right now we use dist-git branches for this. Having a package.cfg might sound easier , but the caveat is that building against different branches requires tweaks sometimes. So we immediately hit a choice: do we want a single branch and the %ifdef mess, or multiple branches and the cherry picking troubles.
Yeah, but if there is clear map between branches & builds stored somewhere, it could be quite convenient for both. If you say "I want to have separate branches", then you still have f31/f32 while you should be able to say "I have 2.x branch and it is going to be built in all releases". That should be easily discoverable.
Right now, big disadvantage of multiple branches is that %changelogs and Release bumps introduce trivial conflicts. I think we should explore the proposals (incl. your and Neal's) to make this better. Fixing this would be good also for normal packaging, because it'd remove one or two more steps that need to be done every time.
Yeah, at the current job I'm exploring something similar so when I get some time, I'll try to write proposal.
We should explore both directions, i.e. package.conf and easier branches.
+1
We should experiment with better/easier control of what to build where. This would benefit normal packaging too. Right now our tooling is just hard to use, with multiple commands in different tools needed to start builds, schedule updates, control koji status, etc. We need changes for "normal" packages as much as for "streams".
+1
- how to deliver rpms to users and how to let them control installed versions
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
Yes. The crucial part is that the rpms that are delivered are just normal rpms. All the complications are kept on the build-side and the client side doesn't need to care if this magic happened, the same result could have been achieved with manually mangled spec file.
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically ...
- All standard Conflicts/Requires/Obsoletes/… will simply work
Yes. More of "just normal packages".
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
... and here I'm not sure. I'm not convinced we need to try to manage all this like that. We already have packages which Conflict, or specify conflicting Requires, and the user gives a dnf command, and a certain transaction is prepared for them. The transaction will occasionally specify a list of rpms to remove. If the user doesn't like this, they can cancel and give a different command.
This would imply that we have --allowerasing set by default, but we don't... And probably we should not since it is dangerous (a bit?).
Specifically, some streams might conflict, other might not, etc. We could have a macro that helps to declare this in one line, but let the maintainers of the rpms declare this as needed.
- [your 2.] Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. [...]
- After branching, there will be a script which untags all packages
which should not be supported in the new release
Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
A different proposal: we add a new rpm flag (maybe as a special Requires: line) that specifies that a package is only supported as a build-time dependency. DNF could then gain a trivial patch that it would ask the user "Do you want to install those unsupported packages? If you do, please do not file bugs, because ...". The advantages: a) packages are available for local builds and downstream rebuilds b) no problem with mock and koji being different c) we avoid any perception of "welding down the engine hood".
This would just acknowledge the truth that there are many packages in Fedora that are not supported beyond an occasional version bump and rebuild, giving better transparency for the users in general.
+1
- Can we have different lifecycles for the software in Fedora?
a.k.a. Should we allow more in-release upgrades?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
We can (and do) encode this information directly in dist-git. No need to provide a meta-mechanism. Even if package.conf becomes a thing, this would contain this information.
I think we should allow "rolling" versions more widely, like we do with firefox, the kernel, spam blockers, and probably some other things. As long some piece software is most backwards compatible, I think we'd be better of abandoning the strict guidelines we have now. This depends on each specific package though.
Yeah, the problem I hit with Rust packages that they are not backwards-compatible. I, for sure, can create compat package but I would like to have way of telling user "if you have installed this package, don't expect working upgradepath or anything like that".
If this happens, it should be as a FPC guideline change, discussed and specified independently of the tooling changes. If we detach it from the technical side, answer the questions when and if at all we allow this will be clearer.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot?
I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
Right now this would be prohibitively expensive. But maybe this could become much easier if points 1-2 and 4 were better answered. So I think we should shelve this question for now.
- How to deal with packages which need outdated BuildRequires
This is another part of Modularity, and you proposal glosses over this. But you said:
- With side tags on demand it is now much easier
... and I fully agree here. There was https://fedoraproject.org/wiki/Changes/OnDemandSideTags which plays with this idea. I think side-tags could/should be the basis of a solution.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
I don't have an answer for this.
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
I think all this sounds reasonable. The main part is that it does not require massive changes in the koji or dnf, and that it integrates in a backwards-compatible way with what we have now.
Let's split the problem into chunks and discuss solutions.
+1, sadly we can't submit hackfest for devconf.cz, but let's probably meet there with some ideas prepared and discuss them?
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
Zbyszek
Le mardi 26 novembre 2019 à 16:58 +0100, Igor Gnatenko a écrit :
Hi, Igor
And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats).
I suppose you're talking about filename here, because we certainly have packages that provide the same thing today. Anyway, this looks a lot less complex to fix than all the complexity modularity has already created.
I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
Then make the module build infra generate a garbage collector rpm per stream. That's the kind of repetitive thing it's supposed to handle.
Anyway, this kind of package needs to exist and be supported as long as the resulting repo contains leaf packages built from it. Otherwise, you're breaking the security audit chain.
Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems?
In my experience, solving specific problems should come first. You won't get any adoption elsewhere without showing some practical benefit Fedora-side, and chasing all-encompasing mythical beasts is a great way to burn time and money without ever delivering anything.
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
Why the heck would you want that? Packages should not depend on stream anymore than they depend on a release as a whole. They should depend on something specifically inside a stream. And that's just a Requires: something with stream(streamname)
- All standard Conflicts/Requires/Obsoletes/… will simply work
Yes, please make it back a single unified rpm-level dependency graph instead of separate module universes that don't behave with one another or with the main distro stream.
That would be totally AWESOME
Regards,
On Tue, Nov 26, 2019 at 9:23 PM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le mardi 26 novembre 2019 à 16:58 +0100, Igor Gnatenko a écrit :
Hi, Igor
And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats).
I suppose you're talking about filename here, because we certainly have packages that provide the same thing today. Anyway, this looks a lot less complex to fix than all the complexity modularity has already created.
No, I am talking about package name here. Koji always takes latest build for buildroot per name. So if you build foo-1-1 and then foo-2-1, only the latter one will be in buildroot. Also tools like pungi (compose tool) always takes latest version and nothing else.
I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
Then make the module build infra generate a garbage collector rpm per stream. That's the kind of repetitive thing it's supposed to handle.
Do you have some idea how to do that?
Anyway, this kind of package needs to exist and be supported as long as the resulting repo contains leaf packages built from it. Otherwise, you're breaking the security audit chain.
No, I don't. They still exist in Koji and if needed, it can be "reproduced". It simply keeps packages which were used in buildroot of any build. Definitely it is more complicated than this, but that's how it works in short.
Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems?
In my experience, solving specific problems should come first. You won't get any adoption elsewhere without showing some practical benefit Fedora-side, and chasing all-encompasing mythical beasts is a great way to burn time and money without ever delivering anything.
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
Why the heck would you want that? Packages should not depend on stream anymore than they depend on a release as a whole. They should depend on something specifically inside a stream. And that's just a Requires: something with stream(streamname)
yeah, sure. I just wanted to point out that we *can* do that, not that we *need* to actually do it.
- All standard Conflicts/Requires/Obsoletes/… will simply work
Yes, please make it back a single unified rpm-level dependency graph instead of separate module universes that don't behave with one another or with the main distro stream.
That would be totally AWESOME
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 07:23, Igor Gnatenko a écrit :
On Tue, Nov 26, 2019 at 9:23 PM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le mardi 26 novembre 2019 à 16:58 +0100, Igor Gnatenko a écrit :
And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats).
I suppose you're talking about filename here, because we certainly have packages that provide the same thing today. Anyway, this looks a lot less complex to fix than all the complexity modularity has already created.
No, I am talking about package name here. Koji always takes latest build for buildroot per name. So if you build foo-1-1 and then foo-2-1, only the latter one will be in buildroot. Also tools like pungi (compose tool) always takes latest version and nothing else.
Actually, in that case, it would be better to have a foo-1 and foo-2 package names. That's how the Go macros do it: 1. major version ends up in the package name name=foo-<major> 2. if for some stupid reason there is a need for a specific code state within a major version, name=compat-foo-<major>-<state-ide>
I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
Then make the module build infra generate a garbage collector rpm per stream. That's the kind of repetitive thing it's supposed to handle.
Do you have some idea how to do that?
Make the module multibuild thinguie write a generated package list on each run; save the list with the module descriptor in a git repo; on next run, add everything not re-generated to a modulename-obsoletes package.
Anyway, this kind of package needs to exist and be supported as long as the resulting repo contains leaf packages built from it. Otherwise, you're breaking the security audit chain.
No, I don't. They still exist in Koji
That's academic. Downstream users do not get a koji copy, they get a copy of produced repos at a particular date. Please don't start shoving into deep koji corners materials that should end up in package repos. As long as things exist as Fedora packages, all their build chain must exist in Fedora package repos (with eventual minor version updates)
Having all build elements in the package repos themselves is the CORE feature that makes the “share” community dimension of Fedora work. Anyone can take the packages and do whatever he wants with them (audit, rebuild, modify), in any rpm build infra (koji, copr, mock, rpmbuild, obs) without depending on the Fedora build infra of the year.
Regards,
On Mon, Dec 2, 2019 at 8:28 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 07:23, Igor Gnatenko a écrit :
On Tue, Nov 26, 2019 at 9:23 PM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le mardi 26 novembre 2019 à 16:58 +0100, Igor Gnatenko a écrit :
And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats).
I suppose you're talking about filename here, because we certainly have packages that provide the same thing today. Anyway, this looks a lot less complex to fix than all the complexity modularity has already created.
No, I am talking about package name here. Koji always takes latest build for buildroot per name. So if you build foo-1-1 and then foo-2-1, only the latter one will be in buildroot. Also tools like pungi (compose tool) always takes latest version and nothing else.
Actually, in that case, it would be better to have a foo-1 and foo-2 package names. That's how the Go macros do it:
- major version ends up in the package name name=foo-<major>
- if for some stupid reason there is a need for a specific code state
within a major version, name=compat-foo-<major>-<state-ide>
I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
Then make the module build infra generate a garbage collector rpm per stream. That's the kind of repetitive thing it's supposed to handle.
Do you have some idea how to do that?
Make the module multibuild thinguie write a generated package list on each run; save the list with the module descriptor in a git repo; on next run, add everything not re-generated to a modulename-obsoletes package.
I meant more how to do it technically, in Koji :)
Anyway, this kind of package needs to exist and be supported as long as the resulting repo contains leaf packages built from it. Otherwise, you're breaking the security audit chain.
No, I don't. They still exist in Koji
That's academic. Downstream users do not get a koji copy, they get a copy of produced repos at a particular date. Please don't start shoving into deep koji corners materials that should end up in package repos. As long as things exist as Fedora packages, all their build chain must exist in Fedora package repos (with eventual minor version updates)
Having all build elements in the package repos themselves is the CORE feature that makes the “share” community dimension of Fedora work. Anyone can take the packages and do whatever he wants with them (audit, rebuild, modify), in any rpm build infra (koji, copr, mock, rpmbuild, obs) without depending on the Fedora build infra of the year.
It would be nice, however please return back to the reality. Many of packages are FTBFS (e.g. some BRs do not exist anymore (due to the dependency update)). So this does not work in practice. It would be cool to get to the point where Fedora repos are self-hosted, but we are quite far from this.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 08:38, Igor Gnatenko a écrit :
On Mon, Dec 2, 2019 at 8:28 AM Nicolas Mailhot via devel
Having all build elements in the package repos themselves is the CORE feature that makes the “share” community dimension of Fedora work. Anyone can take the packages and do whatever he wants with them (audit, rebuild, modify), in any rpm build infra (koji, copr, mock, rpmbuild, obs) without depending on the Fedora build infra of the year.
It would be nice, however please return back to the reality. Many of packages are FTBFS (e.g. some BRs do not exist anymore (due to the dependency update)). So this does not work in practice.
That’s just a combination of bad tooling, and therefore bad enforcement, and people promising castles in the cloud that would not require any human effort.
The only reason people work on the project is that by and large, the FTBS part is marginal. Make it a normal state, and it will grow, and the distribution will die.
Regards,
On Mon, Dec 2, 2019 at 9:13 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 08:38, Igor Gnatenko a écrit :
On Mon, Dec 2, 2019 at 8:28 AM Nicolas Mailhot via devel
Having all build elements in the package repos themselves is the CORE feature that makes the “share” community dimension of Fedora work. Anyone can take the packages and do whatever he wants with them (audit, rebuild, modify), in any rpm build infra (koji, copr, mock, rpmbuild, obs) without depending on the Fedora build infra of the year.
It would be nice, however please return back to the reality. Many of packages are FTBFS (e.g. some BRs do not exist anymore (due to the dependency update)). So this does not work in practice.
That’s just a combination of bad tooling, and therefore bad enforcement, and people promising castles in the cSure, buloud that would not require any human effort.
The only reason people work on the project is that by and large, the FTBS part is marginal. Make it a normal state, and it will grow, and the distribution will die.
Thanks to Miro, this part is now enforced, though some individuals still do not understand importance of this :/
I am not saying this should not improve. I am just saying that this is where we are and we have had this problem before. And we never had this goal (to make distribution self-contained), as a project.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Hey Igor,
On Tue, 26 Nov 2019 at 17:00, Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
Hello fellows,
After last publication on LWN about Fedora Modularity mess, I think it is time to describe the idea I was proposing internally with few other folks (Adam Samalik, Brian Exelbierd) back in the RH times.
Before I actually go deep, I'll try to answer main questions to myself (so that you can understand why I am proposing this particular thing).
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way) Doing git merge / git cherry-pick and maintaining many git remotes requires some advanced git knowledge and a time. And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats). We do have some kind of a solution for multiple releases building from one branch (package.cfg), however this work has been never finished, thus there are many problems with this approach.
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
- Can we have different lifecycles for the software in Fedora?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot? I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
===
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
Notes:
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
- All standard Conflicts/Requires/Obsoletes/… will simply work
- With side tags on demand it is now much easier
- Mass rebuild scripts will be teached about stream branches and will
build packages properly
- After branching, there will be a script which untags all packages
which should not be supported in the new release
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
===
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
I agree that the approach is you described is good. From you have written, it seems to me that Stream could be an explicit rpm property. Then the names wouldn't be mangled just extended: ENSVRA.
Anyway, i would just go with your approach for start. That way, one can avoid bothering rpm team unless necessary.
This is really the only sensible implementation of modularity (i.e. providing multiple version of software) and the reason is that it provides the functionality required while leveraging already existing binary format to provide the new content: RPM, which makes matters on user-side much easier.
It would be good to make this change as it will send Fedora (and all the other distros that use modularity) in the right direction again. Persistently going the wrong way and pretending it is the right one is not smart.
clime
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Hey Michal,
On Sat, Nov 30, 2019 at 5:08 PM clime clime@fedoraproject.org wrote:
Hey Igor,
On Tue, 26 Nov 2019 at 17:00, Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
Hello fellows,
After last publication on LWN about Fedora Modularity mess, I think it is time to describe the idea I was proposing internally with few other folks (Adam Samalik, Brian Exelbierd) back in the RH times.
Before I actually go deep, I'll try to answer main questions to myself (so that you can understand why I am proposing this particular thing).
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way) Doing git merge / git cherry-pick and maintaining many git remotes requires some advanced git knowledge and a time. And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats). We do have some kind of a solution for multiple releases building from one branch (package.cfg), however this work has been never finished, thus there are many problems with this approach.
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
- Can we have different lifecycles for the software in Fedora?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot? I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
===
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
Notes:
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
- All standard Conflicts/Requires/Obsoletes/… will simply work
- With side tags on demand it is now much easier
- Mass rebuild scripts will be teached about stream branches and will
build packages properly
- After branching, there will be a script which untags all packages
which should not be supported in the new release
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
===
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
I agree that the approach is you described is good. From you have written, it seems to me that Stream could be an explicit rpm property. Then the names wouldn't be mangled just extended: ENSVRA.
Anyway, i would just go with your approach for start. That way, one can avoid bothering rpm team unless necessary.
This is really the only sensible implementation of modularity (i.e. providing multiple version of software) and the reason is that it provides the functionality required while leveraging already existing binary format to provide the new content: RPM, which makes matters on user-side much easier.
It would be good to make this change as it will send Fedora (and all the other distros that use modularity) in the right direction again. Persistently going the wrong way and pretending it is the right one is not smart.
+1
clime
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Nov 26, 2019 at 10:59 AM Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
Hello fellows,
After last publication on LWN about Fedora Modularity mess, I think it is time to describe the idea I was proposing internally with few other folks (Adam Samalik, Brian Exelbierd) back in the RH times.
Before I actually go deep, I'll try to answer main questions to myself (so that you can understand why I am proposing this particular thing).
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way) Doing git merge / git cherry-pick and maintaining many git remotes requires some advanced git knowledge and a time. And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats). We do have some kind of a solution for multiple releases building from one branch (package.cfg), however this work has been never finished, thus there are many problems with this approach.
I think we need to recognize that we've done some poor optimization for the majority of packager workflows. Even if we consider modules, the vast majority of components will never be modularized. Moreover, we already know that the overwhelming majority of specs are managed identically across branches.
In these cases, we can make fedpkg better for handling this. There's no reason that people can't have a single master branch and only branch for distro releases as needed.
I'm trying this workflow myself with libeconf[1], but unfortunately fedpkg doesn't handle this model well right now. I proposed a particular feature request for this[2], but there might be other ways to make this work. The current approach for package.cfg is terribly broken, though.
As for the name mangling, I think this is a relatively minor issue for supporting multiple versions of software. There *are* things we could do to minimize this, but it's not a good point to focus on right now.
[1]: https://src.fedoraproject.org/rpms/libeconf [2]: https://pagure.io/fedpkg/issue/352
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream.
I think we should categorically disallow build-time-only/buildroot-only packages. They don't make sense in a Fedora context and make it impossible to share resources. What's worse with this model as it is currently implemented is that nobody can figure out whether there are buildroot-only packages involved, and multiple maintainers of packages basically can't happen.
Re-introducing those buildroot-only packages back into the main package collection gives people an opportunity to depend on them freely and co-maintain them. Now, we can't force anyone to co-maintain packages, but as we see with the large number of orphanings that have been happening lately, if people actually want or need packages, they tend to take them on too.
I think we can make it easier to support shipping these things to stable releases too. The ability to use side tags to map to Bodhi update submissions is going to be a good step forward there.
Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
I'm not sure I want to have a concept of "not fully supported" (do we have a concept of supported, really?). But at least for the RHEL context, it makes sense to consider it. An RPM tag for setting that data could be added. I think it probably makes sense to have an RPM tag where we could declare supportability. Something less dumb than the RPMTAG_MODULARITYLABEL tag that would be more generically useful...
Short term, we could "cheat" by adding Provides, but I think a tag would be better since we can easily pass a definition to rpmbuild for a package build. Adding tags to RPM is relatively easy, but we should convince the RPM developers to do it.
As for the whole Obsoletes thing, I think you could have some kind of dynamically generated package that is set up to self-destruct and include those Obsoletes. Templating a variation of the fedora-obsolete-packages package that could be updated and generate all the needed Obsoletes should be possible. Ordinarily, these Obsoletes would be part of the package it replaces, but at the Rust ecosystem scale, it makes sense to have a self-destructing package that does that centrally. How we'd specifically implement this? I don't know.
- Can we have different lifecycles for the software in Fedora?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
Software shipped in *any* default enabled repositories that other packages can depend on *must* be available for the life of a Fedora release.
I think we could be more flexible for software shipped in repositories not enabled by default. But today, fedora-modular is enabled by default and is now supported as a buildroot input. So that's completely toast.
For EPEL, we need to have RHEL-like flexibility for supportability. Ideally, we could compose the supportability ranges for components (i.e. packages) through the RPM tag or Provides...
Regardless, if we consider the existing modularity system, we can't actually set SLAs right now anyway, so this is a problem we need to solve for any solution.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot? I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
I think we should be able to do this, but Fedora probably doesn't have near the build system resources that openSUSE has to pull it off.
To give an idea of the disparity here, there's somewhere around ~100 build machines that can take on ~8-10 builds at a time for x86 builds. There's 7 POWER machines that can take on ~35 builds each for ppc64le. There's 30 s390x LPARs that can run ~5-10 builds each. There's ~30 ARM machines that can take on ~5-10 builds at a time for armv7hl/aarch64 builds.
We compete favorably for POWER and match up for s390x and ARM, but we are massively outclassed on x86 builds. Even if you consider that the large chunk of noarch packages we have to build can largely be built on any architecture, packagers get penalized for it building on ARM machines (because they're awful, unfortunately, and they're *never* going to get better because the ARM ecosystem doesn't really care about performance in the same way that all the other supported architectures do). Even if we were forcibly scheduling noarch builds on x86, we'd still wipe out our resources easily with stream expansion.
We already have problems with existing module builds wiping out all the available build resources by pushing hundreds of builds without any reasonable scheduling, ordering, or prioritization. We need to deal with the resourcing and scheduling issues before we can consider doing this on a broad scale.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
The ideal situation would be that we'd develop technology that all distributions would like to use.
But, of course, that requires something that the RPM distribution communities aren't used to doing: collaborating at the early stage. We did it with Rust, but that was an unusual event. It doesn't help that the two major RPM distributions are commercial rivals and the two companies aren't used to collaborating with each other on these kinds of things.
Fortunately, there *are* some links among the distributions on the community side. But the modularity project started from within Red Hat, and I suspect there wasn't any talking to SUSE in the early stages of the development. There wasn't much community solicitation to Fedora in the beginning either, as we know...
I hope that we can recover from this and develop a solution that all distributions would like to use.
===
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
We could do this with dynamically configured dep generators injected in to the buildroot or such along with other tweaks throughout the build process.
Notes:
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
- All standard Conflicts/Requires/Obsoletes/… will simply work
- With side tags on demand it is now much easier
- Mass rebuild scripts will be teached about stream branches and will
build packages properly
- After branching, there will be a script which untags all packages
which should not be supported in the new release
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
We already allow this, so I think we're fine. DNF needs some better representation and advice for what to do in these scenarios, but that's fixable. :)
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
I'm reluctant to stuff things into fedora-release. I think it'd work equally well if we had a fedora-modular-release to define these. I've used a variation of this technique in Mageia (mageia-repos-pkgprefs) and OpenMandriva (openmandriva-repos-pkgprefs), so I know it would work.
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
Makes sense to me. It also means it's easy to change things midway through (which would be necessary in RHEL land)
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
I'm not sure what you mean here...
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
===
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
I'm curious what the original proposal was, but these are very solid ideas...
Hi Neal,
On Sat, Nov 30, 2019 at 11:58 PM Neal Gompa ngompa13@gmail.com wrote:
On Tue, Nov 26, 2019 at 10:59 AM Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
Hello fellows,
After last publication on LWN about Fedora Modularity mess, I think it is time to describe the idea I was proposing internally with few other folks (Adam Samalik, Brian Exelbierd) back in the RH times.
Before I actually go deep, I'll try to answer main questions to myself (so that you can understand why I am proposing this particular thing).
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way) Doing git merge / git cherry-pick and maintaining many git remotes requires some advanced git knowledge and a time. And we can't actually have multiple versions of a package with same name (without "mangled" names) in a repo due to the way how our buildsystem works (and not only buildsystem, with some caveats). We do have some kind of a solution for multiple releases building from one branch (package.cfg), however this work has been never finished, thus there are many problems with this approach.
I think we need to recognize that we've done some poor optimization for the majority of packager workflows. Even if we consider modules, the vast majority of components will never be modularized. Moreover, we already know that the overwhelming majority of specs are managed identically across branches.
Yeah, indeed. I think this (optimization of packager workflows) was never an explicit goal of people working on Modularity.
In these cases, we can make fedpkg better for handling this. There's no reason that people can't have a single master branch and only branch for distro releases as needed.
I'm trying this workflow myself with libeconf[1], but unfortunately fedpkg doesn't handle this model well right now. I proposed a particular feature request for this[2], but there might be other ways to make this work. The current approach for package.cfg is terribly broken, though.
It definitely needs more thinking and probably requires more concrete proposal how it should work overall. Not just "if package.cfg is there, parse it and build for targets specified in there". I'm also not entirely sure if we should encode this information in the git repo as a file because it is not possible to query this information (e.g. to find out which branches build where) and do checks (e.g. that you don't build 1.x and 2.x in the f31 and you choose not to mangle names).
As for the name mangling, I think this is a relatively minor issue for supporting multiple versions of software. There *are* things we could do to minimize this, but it's not a good point to focus on right now.
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages". I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream.
I think we should categorically disallow build-time-only/buildroot-only packages. They don't make sense in a Fedora context and make it impossible to share resources. What's worse with this model as it is currently implemented is that nobody can figure out whether there are buildroot-only packages involved, and multiple maintainers of packages basically can't happen.
Well, if we make easier builds for multiple targets (without pushing 5 times, creating 10 overrides and waiting for repos) and relax update policy for such packages, then I definitely don't mind having them in the main repo.
Re-introducing those buildroot-only packages back into the main package collection gives people an opportunity to depend on them freely and co-maintain them. Now, we can't force anyone to co-maintain packages, but as we see with the large number of orphanings that have been happening lately, if people actually want or need packages, they tend to take them on too.
That's because those packages are required in the runtime, while I was talking about case like with Rust packages which are used only in build time and linked statically.
I think we can make it easier to support shipping these things to stable releases too. The ability to use side tags to map to Bodhi update submissions is going to be a good step forward there.
Sadly, Bodhi maintainers so far said that they don't plan to enable submitting updates from side tags in non-rawhide :/
Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages. And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths? I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED." Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
I'm not sure I want to have a concept of "not fully supported" (do we have a concept of supported, really?). But at least for the RHEL context, it makes sense to consider it. An RPM tag for setting that data could be added. I think it probably makes sense to have an RPM tag where we could declare supportability. Something less dumb than the RPMTAG_MODULARITYLABEL tag that would be more generically useful...
We do have some update policy which would prevent me from updating packages and I simply could not maintain old versions and fix CVEs. Neither it makes any sense since I will need new version anyway.
Short term, we could "cheat" by adding Provides, but I think a tag would be better since we can easily pass a definition to rpmbuild for a package build. Adding tags to RPM is relatively easy, but we should convince the RPM developers to do it.
As for the whole Obsoletes thing, I think you could have some kind of dynamically generated package that is set up to self-destruct and include those Obsoletes. Templating a variation of the fedora-obsolete-packages package that could be updated and generate all the needed Obsoletes should be possible. Ordinarily, these Obsoletes would be part of the package it replaces, but at the Rust ecosystem scale, it makes sense to have a self-destructing package that does that centrally. How we'd specifically implement this? I don't know.
Well, if you have Provides: unsupported(), then you can hook DNF to set SOLVER_ALLOWUNINSTALL + SOLVER_FORCEBEST for such packages, so they won't stay on older version and if they conflict, they would be removed without user intervention of --best --allowerasing. What do you think?
That's probably what we should have done with python2 packages instead of adding thousands of lines into the fedora-obsolete-packages :)
- Can we have different lifecycles for the software in Fedora?
Right now we have to keep all versions of software which was there at GA point. And from the updates repo. We never remove packages entirely. That said, if package was there at GA time, it will have to be "supported" until the end of that Fedora release. I don't think we can (should?) do much in this regard. If we make packages to build from "stream" branch, we can put an information to that branch that this package should be built for all distributions (Fedora, EPEL) until this date. Or even store this info somewhere else like PDC (yes, I know we want to kill it). fedpkg build will take this into account and submit proper builds. And we can design some API in infra which would tell until when some particular stream of a package is supported. Much like it is done with Fedora releases & GNOME Software.
Software shipped in *any* default enabled repositories that other packages can depend on *must* be available for the life of a Fedora release.
I think we could be more flexible for software shipped in repositories not enabled by default. But today, fedora-modular is enabled by default and is now supported as a buildroot input. So that's completely toast.
For EPEL, we need to have RHEL-like flexibility for supportability. Ideally, we could compose the supportability ranges for components (i.e. packages) through the RPM tag or Provides...
Can we actually create multiple repos and ship them? I don't think our tooling is ready for this. That would be quite some manual work (pungi, mirrormanager?).
Regardless, if we consider the existing modularity system, we can't actually set SLAs right now anyway, so this is a problem we need to solve for any solution.
- Do we want to have some kind of "stream expansion" where software
builds against all Pythons, Perls and whatnot? I think this should be conscious choice of a distribution, and in specific cases, maintainer. Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it. Being able to build combinations of software is definitely nice, but I don't think this should be standard practice. openSUSE does that with ruby and python (they build all modules automatically for all versions) and I like this. But having all packages built against all is just combinatoric explosion. Given how many updates have feedback in Bodhi, I'm pretty sure 99% of combination won't be tested (or even installed?) ever.
I think we should be able to do this, but Fedora probably doesn't have near the build system resources that openSUSE has to pull it off.
To give an idea of the disparity here, there's somewhere around ~100 build machines that can take on ~8-10 builds at a time for x86 builds. There's 7 POWER machines that can take on ~35 builds each for ppc64le. There's 30 s390x LPARs that can run ~5-10 builds each. There's ~30 ARM machines that can take on ~5-10 builds at a time for armv7hl/aarch64 builds.
We compete favorably for POWER and match up for s390x and ARM, but we are massively outclassed on x86 builds. Even if you consider that the large chunk of noarch packages we have to build can largely be built on any architecture, packagers get penalized for it building on ARM machines (because they're awful, unfortunately, and they're *never* going to get better because the ARM ecosystem doesn't really care about performance in the same way that all the other supported architectures do). Even if we were forcibly scheduling noarch builds on x86, we'd still wipe out our resources easily with stream expansion.
We already have problems with existing module builds wiping out all the available build resources by pushing hundreds of builds without any reasonable scheduling, ordering, or prioritization. We need to deal with the resourcing and scheduling issues before we can consider doing this on a broad scale.
So far it did not kill Koji, right? :) Though the question was whether we *want*, not whether we have enough resources to do so.
- Are we still trying to be a Linux distribution or we are just
letting people to do whatever they want in our infrastructure? This question bothers me from time to time and I don't have answer. For example, Modularity is very flexible and I very often find people saying that you can expand this and that, but in Fedora we limit its usefulness. Do we actually need to develop something like this (knowing in advance that probably nobody outside of Fedora/RHEL will be using this software / technology)? Are we trying to create technologies which would be very extensible and used in other distributions instead of solving some specific problems? If former, why don't we talk to others about things in advance and not getting other people to work on these cool things?
The ideal situation would be that we'd develop technology that all distributions would like to use.
But, of course, that requires something that the RPM distribution communities aren't used to doing: collaborating at the early stage. We did it with Rust, but that was an unusual event. It doesn't help that the two major RPM distributions are commercial rivals and the two companies aren't used to collaborating with each other on these kinds of things.
Correct me, but I don't think that build infrastructure is the place where Red Hat and SUSE are "fighting". Kernel, live patching, container tools, virt tools, long-term support and things like that makes money. So I don't think anybody needs to waste people's time on such things.
Fortunately, there *are* some links among the distributions on the community side. But the modularity project started from within Red Hat, and I suspect there wasn't any talking to SUSE in the early stages of the development. There wasn't much community solicitation to Fedora in the beginning either, as we know...
Yeah, and that is pretty sad.
I hope that we can recover from this and develop a solution that all distributions would like to use.
+1
===
How I would imagine having multiple streams:
- rpms/nodejs has multiple branches - 10, 11, 12
- in all of them, nodejs.spec exists as-is without "mangled" name
(just with different versions and such)
- each has package.cfg or some alternative which specifies what is the
EOL and/or for which releases to build
- when builds are submitted, Koji automatically adds suffix to a main
package and subpackages containing branch name
- during the rpmbuild, extra provides are added (for the unversioned
names and indication of a stream) and conflicts (possibly depending on some macro which user can disable) automatically
We could do this with dynamically configured dep generators injected in to the buildroot or such along with other tweaks throughout the build process.
Yeah, I actually have done that :)
Notes:
- If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can
describe this in standard way (Requires: (nodejs >= 9 with nodejs <= 11)) and one will be picked up automatically
- If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 nodejs10`
- If that requires some conflicting dependency to be switched, it will
be switched automatically
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
- If desired, packages can depend on a specific stream via Requires:
stream(nodejs:10) without actually depending on nodejs (this requires some small piece of code in libsolv)
- In the very similar way, user can lock themselves to a specific
stream by writing some conf file which DNF would read (if point above is implemented, about just tens of lines of the code in libdnf, or even libsolv can be teached about that as well)
- DNF can be teached about these special things so that you can
automatically swap conflicting dependencies, but lock yourself to some streams of a package
- All standard Conflicts/Requires/Obsoletes/… will simply work
- With side tags on demand it is now much easier
- Mass rebuild scripts will be teached about stream branches and will
build packages properly
- After branching, there will be a script which untags all packages
which should not be supported in the new release
- fedpkg will be teached about stream branches and will show which
branches build where, so that maintainers can easily maintain them
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
We already allow this, so I think we're fine. DNF needs some better representation and advice for what to do in these scenarios, but that's fixable. :)
+1
- fedora-release will have Suggests: stream(nodejs:10) so that when
user types dnf install nodejs, for example nodejs10 would be picked up (instead of nodejs12)
I'm reluctant to stuff things into fedora-release. I think it'd work equally well if we had a fedora-modular-release to define these. I've used a variation of this technique in Mageia (mageia-repos-pkgprefs) and OpenMandriva (openmandriva-repos-pkgprefs), so I know it would work.
Well, that's implementation detail. Just wanted to point out that we *can* do that.
- Some macros can be added to generate virtual package with "default
stream", so that there will be actual nodejs package depending on nodejs12 if that would be default in given Fedora release
Makes sense to me. It also means it's easy to change things midway through (which would be necessary in RHEL land)
- Going further, I would extend Koji with fedora plugin which would
deal with the build submission instead of having this logic on clients so that all people get same result
I'm not sure what you mean here...
So instead of "fedpkg build" looking at package.cfg and deciding for which targets to build, it would just call to Koji and that would spawn multiple builds. The problem with former approach is that everyone has to have latest version of fedpkg and you can't force people to do so.
===
Now let me quote Matthew Miller with the list of requirements and answer how we achieve them.
- Users should have alternate streams of software available.
I think this is achieved with what I have said above. We will ship nodejs9, nodejs10 at the same time.
- Those alternate streams should be able to have different lifecycles.
nodejs 9 and 10 can automatically build for all distros until EOL (which would be stored somewhere). EOL-date aware tooling (fedpkg) would not build for excluded versions of distros and build for distros where EOL of distro is further than EOL of component.
- Packaging an individual stream for multiple outputs should be
easier than before. As I wrote above, we will have one nodejs rpm repo with multiple branches (aka streams) and no name mangling needed. fedpkg build will spawn builds for all releases matching criteria (EOL date, excluded distributions).
===
This is not the exact approach we have tried back in the days, but it is very similar to that. Unfortunately I did not save any documents from those times but Adam or Brian probably still can find them (since they are RH internal).
I'm curious what the original proposal was, but these are very solid ideas...
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 07:47, Igor Gnatenko a écrit :
Hi Neal,
On Sat, Nov 30, 2019 at 11:58 PM Neal Gompa ngompa13@gmail.com wrote:
I think we need to recognize that we've done some poor optimization for the majority of packager workflows. Even if we consider modules, the vast majority of components will never be modularized. Moreover, we already know that the overwhelming majority of specs are managed identically across branches.
Yeah, indeed. I think this (optimization of packager workflows) was never an explicit goal of people working on Modularity.
And for that reason alone, it should have been sent directly back to the drawing board.
Creating new tooling, without trying to optimize the workflow of the intended users of the tooling, was always doomed to have no user adoption community-side.
Making tooling for fairies, because you can redefine at will what the fairies want, and no actual fairy will ever materialize to contradict you, is easier design-side. But, it’s 100% a gamble that can (and most often will) misfire.
Regards,
On Mon, Dec 2, 2019 at 8:58 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 07:47, Igor Gnatenko a écrit :
Hi Neal,
On Sat, Nov 30, 2019 at 11:58 PM Neal Gompa ngompa13@gmail.com wrote:
I think we need to recognize that we've done some poor optimization for the majority of packager workflows. Even if we consider modules, the vast majority of components will never be modularized. Moreover, we already know that the overwhelming majority of specs are managed identically across branches.
Yeah, indeed. I think this (optimization of packager workflows) was never an explicit goal of people working on Modularity.
And for that reason alone, it should have been sent directly back to the drawing board.
It really depends on a goal. If the goal is to provide simple parallel availability, then there is nothing to complain about. Modularity simply has (had?) different goal.
Creating new tooling, without trying to optimize the workflow of the intended users of the tooling, was always doomed to have no user adoption community-side.
As Neal mentioned, it was developed inside Red Hat for the Red Hat needs (and arguably Fedora needs too).
Making tooling for fairies, because you can redefine at will what the fairies want, and no actual fairy will ever materialize to contradict you, is easier design-side. But, it’s 100% a gamble that can (and most often will) misfire.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 10:49, Igor Gnatenko a écrit :
On Mon, Dec 2, 2019 at 8:58 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 07:47, Igor Gnatenko a écrit :
Hi Neal,
On Sat, Nov 30, 2019 at 11:58 PM Neal Gompa ngompa13@gmail.com wrote:
I think we need to recognize that we've done some poor optimization for the majority of packager workflows. Even if we consider modules, the vast majority of components will never be modularized. Moreover, we already know that the overwhelming majority of specs are managed identically across branches.
Yeah, indeed. I think this (optimization of packager workflows) was never an explicit goal of people working on Modularity.
And for that reason alone, it should have been sent directly back to the drawing board.
It really depends on a goal.
It does not. Unless the tooling gets fully autonomous, it won’t work without human operators (tooling users ie packagers), so making operator life easier (or at least not worse) is a requisite for adoption.
The RH side can force employees to use new tooling because it’s the company choice. The Fedora community side has to make the new tooling worth the contributor’s time.
Regards,
Igor Gnatenko wrote:
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
Your proposal addresses these, but skips the same requirement the current Modularity also fails to address by design, i.e.:
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way)
In particular, your proposal suggests:
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
which explicitly excludes the parallel installability. So I do not see how it addresses the main drawback Modularity has compared to compatibility packages.
(Note: The sentence as stated also means that the package Conflicts with itself, which will probably also badly confuse some tools, but that is a technical detail that should be fixable. It is the underlying concept of Conflicting with all other versions that is the real issue.)
Your proposal is essentially a proposal to automate the creation of versioned (with suffixed Name) compatibility packages, but it excludes the most essential part of the compatibility package pattern, the one that makes it suited for this use case unlike the current Modularity. So I fail to see how it addresses in any way the issues we are having with the current Modularity.
You suggest to change the packaging guidelines to match the technical limitations of your proposed technology:
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
but the guideline that compatibility packages should not conflict exists for a strong reason, and removing the guideline will not make the issues that lead to its existence magically go away.
Non-default versions of non-leaf packages MUST be parallel installable with the default version for the distribution to be consistent and for users to be allowed to freely choose their applications without arbitrary conflicts.
Otherwise (i.e., if parallel installability is not implemented), what you write:
Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it.
is just not true. If the 2 different versions of Perl cannot coexist, building Bugzilla against only one version is not possible without making Bugzilla incompatible with anything built against the other version.
I agree that we should only build each package against one version of Perl (the distribution default wherever possible, otherwise the most suitable version), but that requires that the users can actually install more than one version at the same time.
But going back to your questions and answers:
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages".
This is getting dangerously close to the strawman antipattern: you are modifying the question before answering it. That said, you then add:
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work
which limits the scope to build-time-only packages again. So why did you attempt to modify it above?
I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
It is a common misconception that any package removed from Fedora belongs into fedora-obsolete-packages. In most cases, you should actually NOT add the package there. Just stop shipping it. If the user has an obsolete package installed, it should be kept by default. Only if it actually causes file conflicts with current software, it makes sense to Obsolete the package at RPM level. (Arguably also for dependency conflicts, but those can actually be resolved with the DNF --allowremove flag, so it is not that clear cut. I would argue for keeping the packages by default and letting the users remove obsolete packages with broken dependencies manually.) We should not remove software that users may still be using from users' systems for no good reason. Especially not if it causes extra work for you as the maintainer.
And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths?
Because the user may want to use that dependency to either rebuild your package (which is definitely a valid use case in a Free Software distribution), or to compile some other software (something we definitely also want to support), or to develop some new software (which is explicitly one of the target user bases of Fedora Workstation), or even for something entirely different (which might not be possible for a Rust library, but think, e.g., of some LaTeX package that your package needs to compile its documentation, but that can also be used to write documents entirely unrelated to software).
If you are spending your time to get the dependency into Fedora because your package needs it, you should also take the little extra time it takes to support it properly.
I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED."
And I think telling that to the user is absolutely unfair and against the spirit of Fedora.
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
You do not really agree with me because I do not see any valid reason for making a distinction between build-time and runtime dependencies there. If your package depends on something, either you or somebody else must maintain the something in Fedora and it must be available for all users, no matter how exactly the dependency is used. And all the more so if we are not even talking about a build-time-only tool, but about statically-linked Rust code which is actually used at runtime, just not in the form of the package.
Kevin Kofler
Hi Kevin,
On Mon, Dec 2, 2019 at 4:43 AM Kevin Kofler kevin.kofler@chello.at wrote:
Igor Gnatenko wrote:
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
Your proposal addresses these, but skips the same requirement the current Modularity also fails to address by design, i.e.:
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way)
In particular, your proposal suggests:
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
which explicitly excludes the parallel installability. So I do not see how it addresses the main drawback Modularity has compared to compatibility packages.
Yes, however maintainer of nodejs does not want to rename binaries and patch sources to be parallel-installable. And today, we would block adding such "compat" package into the repositories. I agree it would be nice to have them parallel-installable, but I don't think we need to force it onto all packages.
(Note: The sentence as stated also means that the package Conflicts with itself, which will probably also badly confuse some tools, but that is a technical detail that should be fixable. It is the underlying concept of Conflicting with all other versions that is the real issue.)
Your proposal is essentially a proposal to automate the creation of versioned (with suffixed Name) compatibility packages, but it excludes the most essential part of the compatibility package pattern, the one that makes it suited for this use case unlike the current Modularity. So I fail to see how it addresses in any way the issues we are having with the current Modularity.
We have different problems with modularity, like:
* Whole new tooling which has its own inputs/outputs (and many times it actually does not work) * Overcomplication of dependency solving (even after so many years, DNF still can't handle it properly) * Different workflow from standard packages which hurts discoverability and maintainability (basically it creates small distros within one distro)
and many more.
You suggest to change the packaging guidelines to match the technical limitations of your proposed technology:
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
but the guideline that compatibility packages should not conflict exists for a strong reason, and removing the guideline will not make the issues that lead to its existence magically go away.
Non-default versions of non-leaf packages MUST be parallel installable with the default version for the distribution to be consistent and for users to be allowed to freely choose their applications without arbitrary conflicts.
I believe that exactly one of the reasons why Modularity has been created. It requires patching of source code and renaming binaries and sometimes it is not trivial amount of work. And at the end of the day, most of the people probably don't need installed multiple versions of a package at the same time (e.g. nodejs).
Otherwise (i.e., if parallel installability is not implemented), what you write:
Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I think maintainer should choose one version of perl and let bugzilla use it.
is just not true. If the 2 different versions of Perl cannot coexist, building Bugzilla against only one version is not possible without making Bugzilla incompatible with anything built against the other version.
Sure, we just have to accept this until perl is made parallel-installable. Which may or may not happen, but if I have choice between "only one version of perl and no bugzilla" or "two conflicting versions of perl and bugzilla using non-default version of perl", I will choose latter. Unfortunately reality is not perfect.
I agree that we should only build each package against one version of Perl (the distribution default wherever possible, otherwise the most suitable version), but that requires that the users can actually install more than one version at the same time.
But going back to your questions and answers:
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages".
This is getting dangerously close to the strawman antipattern: you are modifying the question before answering it. That said, you then add:
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work
which limits the scope to build-time-only packages again. So why did you attempt to modify it above?
Sorry, it took me quite a while to think about this and write it, so there might be some conflicting points and I might have put something into the places where I should not have done so.
I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
It is a common misconception that any package removed from Fedora belongs into fedora-obsolete-packages. In most cases, you should actually NOT add the package there. Just stop shipping it. If the user has an obsolete package installed, it should be kept by default. Only if it actually causes file conflicts with current software, it makes sense to Obsolete the package at RPM level. (Arguably also for dependency conflicts, but those can actually be resolved with the DNF --allowremove flag, so it is not that clear cut. I would argue for keeping the packages by default and letting the users remove obsolete packages with broken dependencies manually.) We should not remove software that users may still be using from users' systems for no good reason. Especially not if it causes extra work for you as the maintainer.
Sure, but let's take specific case. I have had rust-smallvec+std-devel-0.6.12-1 which depend on same EVR of rust-smallvec-devel. I have updated rust-smallvec-devel to 1.0.0 which does not have +std subpackage anymore. On upgrade that causes conflicts which can be resolved using --allowerasing --best, but our current guidelines say that proper obsoletes must be added somewhere so that upgrade works without manual intervention.
And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths?
Because the user may want to use that dependency to either rebuild your package (which is definitely a valid use case in a Free Software distribution), or to compile some other software (something we definitely also want to support), or to develop some new software (which is explicitly one of the target user bases of Fedora Workstation), or even for something entirely different (which might not be possible for a Rust library, but think, e.g., of some LaTeX package that your package needs to compile its documentation, but that can also be used to write documents entirely unrelated to software).
I believe I never said to not ship such packages (please correct me if I did), we perfectly can ship them to the users, but we need to make them aware that maintainer won't be providing proper upgradepath and might not fix CVEs in the time manner.
If you are spending your time to get the dependency into Fedora because your package needs it, you should also take the little extra time it takes to support it properly.
I simply can't. If somebody gets a package into repo just because he wants to build foo, it is probably not very reasonable to ask them to "properly" maintain it. Otherwise people probably won't upgrade (or package) foo in Fedora and do it in COPR/OBS instead. Which will be definitely even less compliant with our guidelines.
I think we get to the point where we need to find balance between "more packages, but less supported" vs "less packages, but with good quality".
I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED."
And I think telling that to the user is absolutely unfair and against the spirit of Fedora.
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
You do not really agree with me because I do not see any valid reason for making a distinction between build-time and runtime dependencies there. If your package depends on something, either you or somebody else must maintain the something in Fedora and it must be available for all users, no matter how exactly the dependency is used. And all the more so if we are not even talking about a build-time-only tool, but about statically-linked Rust code which is actually used at runtime, just not in the form of the package.
I wanted to point out that you were the most loud about disallowing not shipping packages and that I agree with you that we need to prohibit that, though with some caveats. Sorry if it sounded other way.
Kevin Kofler
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 08:17, Igor Gnatenko a écrit :
I simply can't. If somebody gets a package into repo just because he wants to build foo, it is probably not very reasonable to ask them to "properly" maintain it.
It is very reasonable to expect anyone building in Fedora, reusing the work of countless other Fedora contributors, to produce things in a form others can build and contribute upon.
That’s free software 101.
Building communities takes time and energy and has no immediate benefit. But, long-term, it's the most efficient way to do things (the “free software development model” Red Hat suits love to congratulate themselves on; it’s a development model, not just a bunch of licenses and deployment formats).
Promoting Kleenex packaging, is ultimately self-defeating for Fedora. It does not create the community synergy, that makes working within a distribution worthwhile.
The only people it makes happy are the devs that loathed any community or distribution engagement in the first place.
Regards,
On Mon, Dec 2, 2019 at 8:48 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 08:17, Igor Gnatenko a écrit :
I simply can't. If somebody gets a package into repo just because he wants to build foo, it is probably not very reasonable to ask them to "properly" maintain it.
It is very reasonable to expect anyone building in Fedora, reusing the work of countless other Fedora contributors, to produce things in a form others can build and contribute upon.
Yes, it is reasonable to reuse the work. But it is not reasonable to expect the person who added something into Fedora is willing to fix all bugs and properly maintain it. The whole point I'm trying to make is that all packages are "community supported", however some of them are more supported and some of them can be / are supported less.
That’s free software 101.
Building communities takes time and energy and has no immediate benefit. But, long-term, it's the most efficient way to do things (the “free software development model” Red Hat suits love to congratulate themselves on; it’s a development model, not just a bunch of licenses and deployment formats).
I am not sure why you mention RH here.
Promoting Kleenex packaging, is ultimately self-defeating for Fedora. It does not create the community synergy, that makes working within a distribution worthwhile.
The only people it makes happy are the devs that loathed any community or distribution engagement in the first place.
Regards,
-- Nicolas Mailhot _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Le 2019-12-02 10:45, Igor Gnatenko a écrit :
On Mon, Dec 2, 2019 at 8:48 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le 2019-12-02 08:17, Igor Gnatenko a écrit :
Building communities takes time and energy and has no immediate benefit. But, long-term, it's the most efficient way to do things (the “free software development model” Red Hat suits love to congratulate themselves on; it’s a development model, not just a bunch of licenses and deployment formats).
I am not sure why you mention RH here.
I mention RH because so far, it’s been one of the few companies which has understood the difference between licensing and development model. That is both the reason why it sponsors the Fedora project as it exists today, and (IMHO) the reason it has consistently out-competed companies dabbling in the “open source” space, without understanding the model dimension (allowing RH to continue sponsoring Fedora).
All the proposals that try to renege on the dev model part (the “share” in Fedora) are doomed to long-term failure, both at the Fedora and RH level (and the RH level translates into no funding for the project, so it’s a second-level Fedora failure).
And, people commonly misunderstand dev as meaning upstream source code only. It is not. Dev includes all the integration work done packaging-side.
Regards,
On Monday, December 2, 2019 12:17:20 AM MST Igor Gnatenko wrote:
Sure, we just have to accept this until perl is made parallel-installable. Which may or may not happen, but if I have choice between "only one version of perl and no bugzilla" or "two conflicting versions of perl and bugzilla using non-default version of perl", I will choose latter. Unfortunately reality is not perfect.
Doesn't that defeat the purpose of using a distro? At that point, you might as well go grab some container nonsense, instead of using a real system, if you care more for just "make it run", rather than "make it run properly on a stable, supported Perl".
On 2019-12-02, John M. Harris Jr johnmh@splentity.com wrote:
On Monday, December 2, 2019 12:17:20 AM MST Igor Gnatenko wrote:
Sure, we just have to accept this until perl is made parallel-installable. Which may or may not happen, but if I have choice between "only one version of perl and no bugzilla" or "two conflicting versions of perl and bugzilla using non-default version of perl", I will choose latter. Unfortunately reality is not perfect.
Doesn't that defeat the purpose of using a distro? At that point, you might as well go grab some container nonsense, instead of using a real system, if you care more for just "make it run", rather than "make it run properly on a stable, supported Perl".
You assume that a non-default Perl is not stable and supported. While it's true that non-default streams have probably fewer users and thus are less tested, that does not mean it is an intention.
-- Petr
Petr Pisar wrote:
On 2019-12-02, John M. Harris Jr johnmh@splentity.com wrote:
On Monday, December 2, 2019 12:17:20 AM MST Igor Gnatenko wrote:
Sure, we just have to accept this until perl is made parallel-installable. Which may or may not happen, but if I have choice between "only one version of perl and no bugzilla" or "two conflicting versions of perl and bugzilla using non-default version of perl", I will choose latter. Unfortunately reality is not perfect.
Doesn't that defeat the purpose of using a distro? At that point, you might as well go grab some container nonsense, instead of using a real system, if you care more for just "make it run", rather than "make it run properly on a stable, supported Perl".
You assume that a non-default Perl is not stable and supported. While it's true that non-default streams have probably fewer users and thus are less tested, that does not mean it is an intention.
A non-default Perl packaged in a way that conflicts with the default one is by definition not a "stable, supported Perl", because it is incompatible with many packages in the distribution.
Kevin Kofler
Igor Gnatenko wrote:
Yes, however maintainer of nodejs does not want to rename binaries and patch sources to be parallel-installable.
And that is exactly the problem.
And today, we would block adding such "compat" package into the repositories.
Which is exactly how it should be.
I agree it would be nice to have them parallel-installable, but I don't think we need to force it onto all packages.
But there is actually no way around it because every other approach inevitably leads to conflicts.
We have different problems with modularity, like:
- Whole new tooling which has its own inputs/outputs (and many times
it actually does not work)
- Overcomplication of dependency solving (even after so many years,
DNF still can't handle it properly)
- Different workflow from standard packages which hurts
discoverability and maintainability (basically it creates small distros within one distro)
and many more.
This is true. However, your approach may also introduce this kind of practical problems. E.g., if you keep the self-conflicting packages.
(For the record, the logically correct way to handle it would be: Provides: stream(nodejs) = 10 Conflicts: stream(nodejs) < 10 Conflicts: stream(nodejs) > 10 but I disagree with this Conflicts approach to begin with.)
Non-default versions of non-leaf packages MUST be parallel installable with the default version for the distribution to be consistent and for users to be allowed to freely choose their applications without arbitrary conflicts.
I believe that exactly one of the reasons why Modularity has been created. It requires patching of source code and renaming binaries and sometimes it is not trivial amount of work.
And this is exactly why I am against Modularity, at least for non-leaf packages. (And your proposal does not fix this.)
And at the end of the day, most of the people probably don't need installed multiple versions of a package at the same time (e.g. nodejs).
They do if they want to use 2 (or more) applications that happen to depend on different versions of nodejs (something that is entirely beyond the user's control) on the same Fedora installation. I think it is entirely unacceptable to tell users that they cannot install application A and application B on the same Fedora n installation (even though it might have worked just fine on Fedora n-1) because A depends on libfoo 2 (or foo-interpreter 2, this is actually no different) whereas B is stuck on libfoo 1. (It might have worked just fine on previous releases before the libfoo 2 stream was added.) The purpose of a distribution is to make software from different upstreams work together.
is just not true. If the 2 different versions of Perl cannot coexist, building Bugzilla against only one version is not possible without making Bugzilla incompatible with anything built against the other version.
Sure, we just have to accept this until perl is made parallel-installable. Which may or may not happen, but if I have choice between "only one version of perl and no bugzilla" or "two conflicting versions of perl and bugzilla using non-default version of perl", I will choose latter. Unfortunately reality is not perfect.
I do not see why it would be so hard to provide compatibility Perl versions in non-conflicting compatibility packages. It has already be done in the past. It is just a matter of versioning (suffixing) the binaries and the directories (e.g., /usr/bin/perl5.28, /usr/share/perl5.28/, etc.). The default version may or may not adopt some of this version-suffixing too. The only thing that matters is that the non-default versions use it.
Sure, but let's take specific case. I have had rust-smallvec+std-devel-0.6.12-1 which depend on same EVR of rust-smallvec-devel. I have updated rust-smallvec-devel to 1.0.0 which does not have +std subpackage anymore. On upgrade that causes conflicts which can be resolved using --allowerasing --best, but our current guidelines say that proper obsoletes must be added somewhere so that upgrade works without manual intervention.
In this case, you probably want to add the Obsoletes to rust-smallvec-devel rather than fedora-obsolete-packages. That avoids all the bureaucracy and is just one line in the specfile that you need to update anyway.
That said, you may actually need or want to ship parallel-installable rust-smallvec+std-0.6.12-devel, rust-smallvec-0.6.12-devel, and rust-smallvec-1.0.0-devel instead (the actual package contents are inherently parallel-installable anyway!), in which case you don't have to Obsolete anything, just orphan and stop updating the 0.6.12 ones if you don't need them any longer.
I believe I never said to not ship such packages (please correct me if I did), we perfectly can ship them to the users, but we need to make them aware that maintainer won't be providing proper upgradepath and might not fix CVEs in the time manner.
There is no strict guarantee that bugs will be fixed in a timely manner for ANY package, so there is no point in specifying that explicitly for individual packages. CVEs SHOULD be fixed ASAP, but that also goes for your Rust libraries, where in addition you are supposed to rebuild your statically linked executables ASAP after upgrading the library. If that is too much work for you, then you should not be packaging statically-linked software. (And we really need a dynamic linking solution for Go and Rust. The current approach where libraries are packaged as installed source code is just not a reasonable approach in a binary distribution.)
For upgrade path for dropped packages, well, either add an Obsoletes somewhere if it makes sense (as in the example above) or just ignore it.
If you are spending your time to get the dependency into Fedora because your package needs it, you should also take the little extra time it takes to support it properly.
I simply can't.
I don't see how it makes so much of a difference in effort.
Your proposal is essentially a proposal to automate the creation of versioned (with suffixed Name) compatibility packages, but it excludes the most essential part of the compatibility package pattern, the one that makes it suited for this use case unlike the current Modularity. So I fail to see how it addresses in any way the issues we are having with the current Modularity.
Oh yeah, this is not only the thing I'm proposing. People want to have "stream branch" and build from it to all Fedora and EPEL and I thought that it was clear however it was not. Basically part of the proposal is to have let's say branches by major version which builds into all releases.
The part about submitting the build once and having Koji build for all releases is fine. It has already been proposed by others, and I see no issue with it (as long as the package is still actually built on each release, just without the maintainer having to do it explicitly – build once, run everywhere is not going to work reliably).
It is the automatic suffixing that I have a problem with. I want to see a separate dist-git module created for the suffixed package instead, where the specfile is also made conflict-free by the maintainer. In other words, the good old compatibility package pattern that just works. This part cannot be reasonably automated, and the part of your proposal that proposes automating this destroys the most useful part of the pattern.
For the Rust libraries, the automatic suffixing might actually work without Conflicts, so it could be done there, but not in the general case. Unless maybe you work with %if conditionals somehow (like the ones we used to keep the F8 kdelibs.spec in sync with the F9 kdelibs3.spec and the F8 kdelibs4.spec with the F9 kdelibs.spec). Automatically adding Conflicts to automatically suffixed packages is what I am strongly opposed to. The automation only makes sense if it can be done without RPM Conflicts and without file conflicts.
Kevin Kofler
On Mon, Dec 2, 2019, 04:43 Kevin Kofler kevin.kofler@chello.at wrote:
Igor Gnatenko wrote:
- Do we want to package multiple streams only for "leaf" software or
any kind of it? I believe that we need both, and we do support both. However, it might not look as nice as it could:
- Need to create multiple repos for different "streams"
- Need to maintain epel7/epel8/f30/f31/master branches
- Package names have to be "mangled" (e.g. mozjs38, mozjs60) and it is
manual work
Your proposal addresses these, but skips the same requirement the current Modularity also fails to address by design, i.e.:
- However, those are supposed to be (according to the guidelines)
parallel-installable (and not be conflicting in any way)
In particular, your proposal suggests:
- Packages produced from nodejs.src have Provides: stream(nodejs)
stream(nodejs:9) and Conflicts: stream(nodejs) so that it is explicitly not possible to install 9 and 10 at the same time
which explicitly excludes the parallel installability. So I do not see how it addresses the main drawback Modularity has compared to compatibility packages.
(Note: The sentence as stated also means that the package Conflicts with itself, which will probably also badly confuse some tools, but that is a technical detail that should be fixable. It is the underlying concept of Conflicting with all other versions that is the real issue.)
Your proposal is essentially a proposal to automate the creation of versioned (with suffixed Name) compatibility packages, but it excludes the most essential part of the compatibility package pattern, the one that makes it suited for this use case unlike the current Modularity. So I fail to see how it addresses in any way the issues we are having with the current Modularity.
Oh yeah, this is not only the thing I'm proposing. People want to have "stream branch" and build from it to all Fedora and EPEL and I thought that it was clear however it was not. Basically part of the proposal is to have let's say branches by major version which builds into all releases.
You suggest to change the packaging guidelines to match the technical
limitations of your proposed technology:
- Packaging guidelines should be adopted to accept conflicting
packages and tooling should be improved to show the conflicts and how to resolve them
but the guideline that compatibility packages should not conflict exists for a strong reason, and removing the guideline will not make the issues that lead to its existence magically go away.
Non-default versions of non-leaf packages MUST be parallel installable with the default version for the distribution to be consistent and for users to be allowed to freely choose their applications without arbitrary conflicts.
Otherwise (i.e., if parallel installability is not implemented), what you write:
Using some examples from previous threads, why does bugzilla have to be built against 2 different versions of perl and users could choose? I
think
maintainer should choose one version of perl and let bugzilla use it.
is just not true. If the 2 different versions of Perl cannot coexist, building Bugzilla against only one version is not possible without making Bugzilla incompatible with anything built against the other version.
I agree that we should only build each package against one version of Perl (the distribution default wherever possible, otherwise the most suitable version), but that requires that the users can actually install more than one version at the same time.
But going back to your questions and answers:
- Do we want to support buildtime-only packages?
I would rather generalize this category as "less-supported packages".
This is getting dangerously close to the strawman antipattern: you are modifying the question before answering it. That said, you then add:
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work
which limits the scope to build-time-only packages again. So why did you attempt to modify it above?
I maintain 800+ Rust packages and very often I need to update them to an incompatible version. In Rawhide I just do it, update all dependent packages to use new version, and if I can't do that for some reason, create "compat" package. Obviously, all patches are sent to the upstream. Upstreams are removing features, I need to deal with Obsoletes but I simply can't continuously add new Obsoletes into the fedora-obsolete-packages.
It is a common misconception that any package removed from Fedora belongs into fedora-obsolete-packages. In most cases, you should actually NOT add the package there. Just stop shipping it. If the user has an obsolete package installed, it should be kept by default. Only if it actually causes file conflicts with current software, it makes sense to Obsolete the package at RPM level. (Arguably also for dependency conflicts, but those can actually be resolved with the DNF --allowremove flag, so it is not that clear cut. I would argue for keeping the packages by default and letting the users remove obsolete packages with broken dependencies manually.) We should not remove software that users may still be using from users' systems for no good reason. Especially not if it causes extra work for you as the maintainer.
And what for if they are used only during build of other, more important, packages? Why do I have to spend time with upgradepaths?
Because the user may want to use that dependency to either rebuild your package (which is definitely a valid use case in a Free Software distribution), or to compile some other software (something we definitely also want to support), or to develop some new software (which is explicitly one of the target user bases of Fedora Workstation), or even for something entirely different (which might not be possible for a Rust library, but think, e.g., of some LaTeX package that your package needs to compile its documentation, but that can also be used to write documents entirely unrelated to software).
If you are spending your time to get the dependency into Fedora because your package needs it, you should also take the little extra time it takes to support it properly.
I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED."
And I think telling that to the user is absolutely unfair and against the spirit of Fedora.
Obviously, for packages which are used in runtime need a proper support as we do today for all packages to share work (that's the place where I agree with Kevin Kofler.)
You do not really agree with me because I do not see any valid reason for making a distinction between build-time and runtime dependencies there. If your package depends on something, either you or somebody else must maintain the something in Fedora and it must be available for all users, no matter how exactly the dependency is used. And all the more so if we are not even talking about a build-time-only tool, but about statically-linked Rust code which is actually used at runtime, just not in the form of the package.
Kevin Kofler
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 12/1/19 10:37 PM, Kevin Kofler wrote:
I definitely want some mechanism which will tell to user that "THIS PACKAGE IS NOT FULLY SUPPORTED."
And I think telling that to the user is absolutely unfair and against the spirit of Fedora.
The dilemma is, how to allow the useful stuff to remain, while preventing the accumulation of non-functional abandonware, especially since one person's trash might be someone else's treasure. It's not good when cruft accumulates, as it gives Fedora a bad name and wastes people's time when they stumble into unsupported packages that stop working, like in this case:
https://bugzilla.redhat.com/show_bug.cgi?id=1490074
An 'unsupported' warning of some sort could be actually a good thing. Perhaps such warning could include a count of unresolved Bugzilla cases against the package in question.
-
clime -
Igor Gnatenko -
John M. Harris Jr -
Kevin Fenzi -
Kevin Kofler -
Neal Gompa -
Nicolas Mailhot -
Petr Pisar -
Przemek Klosowski -
Zbigniew Jędrzejewski-Szmek