3:46 p.m.
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
== Detailed Description ==
As a major part of the Modularity design, we have a concept of default
module streams. These streams are built as modules, but the RPM
artifacts they deliver are intended to be used just like non-modular
RPMS. The aspirational goal is that a user of the system who never
executes a module-specific command (such as `dnf module install
nodejs:8`) should experience no meaningful changes in behavior from
how they would interact with a completely non-modular system. In
practice, this may mean that the informational output of package
managers may indicate that modules are being enabled and used, but a
user that does not have a specific reason to interact with the
selection of a module stream should have that managed on their behalf.
Similarly, the experience for package maintainers of non-modular
packages should be unaffected by an RPM dependency moving from the
non-modular repository into a default module stream. Up to the
present, this has not been the case; no module stream content has been
available in the non-modular buildroot for other packages to consume.
Koji builds of non-modular RPMs have had only the other non-modular
RPMs from that release available to their buildroots. In contrast,
building on local systems has access to both the non-modular RPMs and
the RPMs from any of the default module streams. With this Change,
Koji builds will have the same behavior and be able to depend on
content provided by default module streams. It also enables the same
behavior for Modular builds: the `platform` stream will now include
the contents of the default module streams for each release and do not
need to be explicitly specified in the modulemd `buildrequires`.
Note: This Change does not address the other major Modularity issue we
are facing around distribution upgrades with differing default
streams. When discussing this Change, please keep that topic separate.
== Benefit to Fedora ==
This will simplify the lives of package maintainers in Fedora in two
primary ways. I'll use a hypothetical example of the Node.js
interpreter and a JSApp package which is capable of running on Node.js
10 or 12 (but requires newer features than are provided by Node.js 8).
Additionally, the JSApp package requires the same versions of Node.js
at build-time.
* Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F31 lifetime.
* Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F32 lifetime.
On Fedora 29 through 31, the Node.js package maintainer needs to build
the `nodejs:10` package both as a module and as a non-modular RPM in
the distribution so that the JSApp package can be built. With this
Change, the Node.js package maintainer in Fedora 32+ will only need to
build the various Node.js streams and make one of them the default
stream. The packages from it will then be added to the buildroot for
non-modular packages. This will also make the packaging process
somewhat more efficient, as the maintainer needs only to manage the
module stream and the MBS will build it for all configured platforms.
Similarly, from the perspective of dependent maintainers, there will
no longer be anxiety about needing to move their package to a module
if one or more of their dependencies drops their non-modular version
in favor of a default stream. Their builds will continue to work as
they do today.
== Scope ==
* Proposal owners:
# Update Packaging Guidelines with
[https://pagure.io/modularity/issue/146#comment-600328 requirements]
for module default streams
# Create a Pungi configuration to generate the buildroot from the
default module streams.
# Include `default_modules_scm_url` in the platform virtual module specification
# Configure Koji tags for inheriting the new modular-defaults
buildroot into the standard buildroot
* Other developers:
Packagers of default module streams will be required to conform to the
[https://pagure.io/modularity/issue/146#comment-600328 policy]
regarding visibility of stream artifacts. Any default module stream
that is not in compliance by one week before Beta Freeze will cease to
be a default stream.
* Release engineering:
# https://pagure.io/releng/issue/8879 - Create pungi config for
Rawhide/F32 ursa prime buildroot
# https://pagure.io/releng/issue/8880 - Include
`default_modules_scm_url` in platform 31 virtual module
# https://pagure.io/releng/issue/8881 - Configure Koji tags for
inheriting f32-modular-buildroot
* Policies and guidelines:
The Modularity Packaging Guidelines will need to be updated to
indicate the strict requirements on default streams.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This change is on the build-system side of things and should not
impact the upgrade process directly.
== How To Test ==
# Build a modular stream
# Make that stream a default stream (or a buildroot override)
# Build a non-modular RPM that requires an artifact RPM from the modular stream.
== User Experience ==
This should not change the end-user experience.
== Dependencies ==
Nothing known that isn't listed in the scope.
== Contingency Plan ==
* Contingency mechanism: Disable the buildroot inheritance in Koji to
revert to the current behavior.
* Blocks release? Ambiguous: lack of complete implementation may
indirectly cause blocking issues.
* Blocks product? No
== Documentation ==
== Release Notes ==
None needed, the Change is not user-facing.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
5:46 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 09. 10. 19 22:46, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
== Detailed Description ==
As a major part of the Modularity design, we have a concept of default
module streams. These streams are built as modules, but the RPM
artifacts they deliver are intended to be used just like non-modular
RPMS. The aspirational goal is that a user of the system who never
executes a module-specific command (such as `dnf module install
nodejs:8`) should experience no meaningful changes in behavior from
how they would interact with a completely non-modular system. In
practice, this may mean that the informational output of package
managers may indicate that modules are being enabled and used, but a
user that does not have a specific reason to interact with the
selection of a module stream should have that managed on their behalf.
If this is the goal of default modular streams, wouldn't it be in fact much
easier to keep the default versions as urisne packages?
Similarly, the experience for package maintainers of non-modular
packages should be unaffected by an RPM dependency moving from the
non-modular repository into a default module stream. Up to the
present, this has not been the case; no module stream content has been
available in the non-modular buildroot for other packages to consume.
Koji builds of non-modular RPMs have had only the other non-modular
RPMs from that release available to their buildroots. In contrast,
building on local systems has access to both the non-modular RPMs and
the RPMs from any of the default module streams. With this Change,
Koji builds will have the same behavior and be able to depend on
content provided by default module streams. It also enables the same
behavior for Modular builds: the `platform` stream will now include
the contents of the default module streams for each release and do not
need to be explicitly specified in the modulemd `buildrequires`.
What I miss in the description is:
1. How does this thing actually work? is there an additional repository composed
from the default streams available in Koji only?
2. How are conflicts between packages from the default streams and ursine
package be handled?
3. What is the local experience if the packager is using mock. What if they are
using rpmbuild directly?
4. How are we handling default streams with dependencies on non-default streams
of other modules?
...
== Scope ==
* Proposal owners:
# Update Packaging Guidelines with
[https://pagure.io/modularity/issue/146#comment-600328 requirements]
for module default streams
# Create a Pungi configuration to generate the buildroot from the
default module streams.
# Include `default_modules_scm_url` in the platform virtual module specification
# Configure Koji tags for inheriting the new modular-defaults
buildroot into the standard buildroot
* Other developers:
Packagers of default module streams will be required to conform to the
[https://pagure.io/modularity/issue/146#comment-600328 policy]
regarding visibility of stream artifacts. Any default module stream
that is not in compliance by one week before Beta Freeze will cease to
be a default stream.
What are the packagers of ursine packages that are shadowed by their modular
counterparts supposed to do here (assuming such shadowing happens)?
What are the packagers of modular packages that are shadowed by their ursine
counterparts supposed to do here (assuming such shadowing happens)?
...
== How To Test ==
# Build a modular stream
# Make that stream a default stream (or a buildroot override)
# Build a non-modular RPM that requires an artifact RPM from the modular stream.
How can I test this as an ursine package maintainer assuming I have build
dependencies that were ursine packages but now will be form modules?
...
== Contingency Plan ==
* Contingency mechanism: Disable the buildroot inheritance in Koji to
revert to the current behavior.
Assuming ursine packges will be retired from Fedora, what is the contingency
there? Consider this scenario:
1. maintainers of the ook module welcome this change and finally they retire ook
from ursine Fedora, shipping only the default ook:12 modular stream.
2. maintainers of ook-cool and ook-free happily build against modular ook:12
3. a previously unknown fundamental flaw in design triggers the contingency plan
4. ook-cool and ook-free FTBFS, maintainer of ook do no longer wish to maintain
ook as ursine package, because the entire depndncy tree to make that happen was
gone/updated/downgraded in the meantime
Who will revert ook to the previous working state? Is it the change owner?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6:44 p.m.
On Wed, 9 Oct 2019 at 18:46, Miro Hrončok <mhroncok(a)redhat.com> wrote:
> What I miss in the description is:
> 1. How does this thing actually work? is there an
additional repository composed
> from the default streams available in Koji only?
> 2. How are conflicts between packages from the default
streams and ursine
> package be handled?
> 3. What is the local experience if the packager is using
mock. What if they are
> using rpmbuild directly?
So from doing a lot of builds with mock, then the packager should be
ok because mock pulls in modules and you can define in mock which
module streams you want if you needed something differently. For
rpmbuild it is a bit harder because you may have used one module on
your local system and built with another.. However in someways this is
similar to rpm where I might have installed an F31 package on my F30
system to test something and then found out my local rpmbuild didn't
work.
In many ways I found working with mock easier than working with any of
the other system tools when dealing with modules. The file is very
well commented and it was clear what I needed to do to make it work.
So I can't answer anything else.. but I think the local experience for
mock users will be smoother than elsewhere.
--
Stephen J Smoogen.
6:55 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 10. 10. 19 1:44, Stephen John Smoogen wrote:
On Wed, 9 Oct 2019 at 18:46, Miro Hrončok <mhroncok(a)redhat.com>
wrote:
>
> What I miss in the description is:
>
> 1. How does this thing actually work? is there an additional repository composed
> from the default streams available in Koji only?
>
> 2. How are conflicts between packages from the default streams and ursine
> package be handled?
>
> 3. What is the local experience if the packager is using mock. What if they are
> using rpmbuild directly?
>
So from doing a lot of builds with mock, then the packager should be
ok because mock pulls in modules and you can define in mock which
module streams you want if you needed something differently. For
rpmbuild it is a bit harder because you may have used one module on
your local system and built with another.. However in someways this is
similar to rpm where I might have installed an F31 package on my F30
system to test something and then found out my local rpmbuild didn't
work.
In many ways I found working with mock easier than working with any of
the other system tools when dealing with modules. The file is very
well commented and it was clear what I needed to do to make it work.
So I can't answer anything else.. but I think the local experience for
mock users will be smoother than elsewhere.
What I really care about here is that the mock experience more or less equals
the Koji experience. I.e. I don't want the packagers to (need to) enable modules
in mock when in fact in Koji they are not enabled, but some other magic is
happening.
Having a description about what this change actually does to enable modules in
the buildroot will hopefully steer this discussion more to the specifics of how
to enable the same thing in mock (by default).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
9:15 a.m.
On Wed, 9 Oct 2019 at 19:56, Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 10. 10. 19 1:44, Stephen John Smoogen wrote:
> On Wed, 9 Oct 2019 at 18:46, Miro Hrončok <mhroncok(a)redhat.com> wrote:
>>
>
>> What I miss in the description is:
>>
>> 1. How does this thing actually work? is there an additional repository
composed
>> from the default streams available in Koji only?
>>
>> 2. How are conflicts between packages from the default streams and ursine
>> package be handled?
>>
>> 3. What is the local experience if the packager is using mock. What if they are
>> using rpmbuild directly?
>>
>
> So from doing a lot of builds with mock, then the packager should be
> ok because mock pulls in modules and you can define in mock which
> module streams you want if you needed something differently. For
> rpmbuild it is a bit harder because you may have used one module on
> your local system and built with another.. However in someways this is
> similar to rpm where I might have installed an F31 package on my F30
> system to test something and then found out my local rpmbuild didn't
> work.
>
> In many ways I found working with mock easier than working with any of
> the other system tools when dealing with modules. The file is very
> well commented and it was clear what I needed to do to make it work.
> So I can't answer anything else.. but I think the local experience for
> mock users will be smoother than elsewhere.
What I really care about here is that the mock experience more or less equals
the Koji experience. I.e. I don't want the packagers to (need to) enable modules
in mock when in fact in Koji they are not enabled, but some other magic is
happening.
Having a description about what this change actually does to enable modules in
the buildroot will hopefully steer this discussion more to the specifics of how
to enable the same thing in mock (by default).
So for all my builds.. it works by default in mock already. [I am not
going to say that it works for everyone.. but when I rebuilt a couple
thousand packages earlier with mockchain.. mock did the right thing
without me futzing.] It is in koji that it fails because koji has no
idea about modules and does not pass any of the extra data mock can
use to make decisions for itself. [I believe Koji doesn't want mock to
make decisions because then the build is not record-ably
reproducible.] This is more about making the decisions that mock would
do for itself be something koji would be able to record and force.
[AKA where mock would use dnf to pull in the default module X, koji
instead determines all the packages to put into its own repo that it
points mock on the builder to. This is meant to make sure the build is
auditable and repeatable.]
--
Stephen J Smoogen.
10:01 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 10. 10. 19 16:15, Stephen John Smoogen wrote:
On Wed, 9 Oct 2019 at 19:56, Miro Hrončok <mhroncok(a)redhat.com>
wrote:
>
> On 10. 10. 19 1:44, Stephen John Smoogen wrote:
>> On Wed, 9 Oct 2019 at 18:46, Miro Hrončok <mhroncok(a)redhat.com> wrote:
>>>
>>
>>> What I miss in the description is:
>>>
>>> 1. How does this thing actually work? is there an additional repository
composed
>>> from the default streams available in Koji only?
>>>
>>> 2. How are conflicts between packages from the default streams and ursine
>>> package be handled?
>>>
>>> 3. What is the local experience if the packager is using mock. What if they
are
>>> using rpmbuild directly?
>>>
>>
>> So from doing a lot of builds with mock, then the packager should be
>> ok because mock pulls in modules and you can define in mock which
>> module streams you want if you needed something differently. For
>> rpmbuild it is a bit harder because you may have used one module on
>> your local system and built with another.. However in someways this is
>> similar to rpm where I might have installed an F31 package on my F30
>> system to test something and then found out my local rpmbuild didn't
>> work.
>>
>> In many ways I found working with mock easier than working with any of
>> the other system tools when dealing with modules. The file is very
>> well commented and it was clear what I needed to do to make it work.
>> So I can't answer anything else.. but I think the local experience for
>> mock users will be smoother than elsewhere.
>
> What I really care about here is that the mock experience more or less equals
> the Koji experience. I.e. I don't want the packagers to (need to) enable modules
> in mock when in fact in Koji they are not enabled, but some other magic is
> happening.
>
> Having a description about what this change actually does to enable modules in
> the buildroot will hopefully steer this discussion more to the specifics of how
> to enable the same thing in mock (by default).
So for all my builds.. it works by default in mock already. [I am not
going to say that it works for everyone.. but when I rebuilt a couple
thousand packages earlier with mockchain.. mock did the right thing
without me futzing.] It is in koji that it fails because koji has no
idea about modules and does not pass any of the extra data mock can
use to make decisions for itself. [I believe Koji doesn't want mock to
make decisions because then the build is not record-ably
reproducible.] This is more about making the decisions that mock would
do for itself be something koji would be able to record and force.
[AKA where mock would use dnf to pull in the default module X, koji
instead determines all the packages to put into its own repo that it
points mock on the builder to. This is meant to make sure the build is
auditable and repeatable.]
AFAIK mock has currently modular repos disabled by default for Fedora configs.
So it might work for EPEL8, or it might work if you enable them, or it might
work because no modules were used at all.
It does not "work by default" for Fedora modules.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, Oct 10, 2019 at 12:46:00AM +0200, Miro Hrončok wrote:
On 09. 10. 19 22:46, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
> Enable module default streams in the buildroot repository for modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
>
> == Detailed Description ==
> As a major part of the Modularity design, we have a concept of default
> module streams. These streams are built as modules, but the RPM
> artifacts they deliver are intended to be used just like non-modular
> RPMS. The aspirational goal is that a user of the system who never
> executes a module-specific command (such as `dnf module install
> nodejs:8`) should experience no meaningful changes in behavior from
> how they would interact with a completely non-modular system. In
> practice, this may mean that the informational output of package
> managers may indicate that modules are being enabled and used, but a
> user that does not have a specific reason to interact with the
> selection of a module stream should have that managed on their behalf.
If this is the goal of default modular streams, wouldn't it be in fact much
easier to keep the default versions as urisne packages?
This question has been asked a few times now and I think it's a good one and
deserves to be answered.
Pierre
10:46 a.m.
On Thu, Oct 10, 2019 at 12:52 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 09. 10. 19 22:46, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
> Enable module default streams in the buildroot repository for modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
>
> == Detailed Description ==
> As a major part of the Modularity design, we have a concept of default
> module streams. These streams are built as modules, but the RPM
> artifacts they deliver are intended to be used just like non-modular
> RPMS. The aspirational goal is that a user of the system who never
> executes a module-specific command (such as `dnf module install
> nodejs:8`) should experience no meaningful changes in behavior from
> how they would interact with a completely non-modular system. In
> practice, this may mean that the informational output of package
> managers may indicate that modules are being enabled and used, but a
> user that does not have a specific reason to interact with the
> selection of a module stream should have that managed on their behalf.
If this is the goal of default modular streams, wouldn't it be in fact much
easier to keep the default versions as urisne packages?
That means, you have 2 different workflows: for default version and
for an additional one.
When branching happens, instead of just updating one file which points
to the new default, you would have to create new stream, retire the
one which is about to become default and update package in
non-modular.
> > Similarly, the experience for package maintainers of non-modular
> > packages should be unaffected by an RPM dependency moving from the
> > non-modular repository into a default module stream. Up to the
> > present, this has not been the case; no module stream content has been
> > available in the non-modular buildroot for other packages to consume.
> > Koji builds of non-modular RPMs have had only the other non-modular
> > RPMs from that release available to their buildroots. In contrast,
> > building on local systems has access to both the non-modular RPMs and
> > the RPMs from any of the default module streams. With this Change,
> > Koji builds will have the same behavior and be able to depend on
> > content provided by default module streams. It also enables the same
> > behavior for Modular builds: the `platform` stream will now include
> > the contents of the default module streams for each release and do not
> > need to be explicitly specified in the modulemd `buildrequires`.
>
> What I miss in the description is:
>
> 1. How does this thing actually work? is there an additional repository composed
> from the default streams available in Koji only?
>
> 2. How are conflicts between packages from the default streams and ursine
> package be handled?
>
> 3. What is the local experience if the packager is using mock. What if they are
> using rpmbuild directly?
>
> 4. How are we handling default streams with dependencies on non-default streams
> of other modules?
>
> > ...
> > == Scope ==
> > * Proposal owners:
> > # Update Packaging Guidelines with
> > [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> > for module default streams
> > # Create a Pungi configuration to generate the buildroot from the
> > default module streams.
> > # Include `default_modules_scm_url` in the platform virtual module
specification
> > # Configure Koji tags for inheriting the new modular-defaults
> > buildroot into the standard buildroot
> >
> > * Other developers:
> >
> > Packagers of default module streams will be required to conform to the
> > [https://pagure.io/modularity/issue/146#comment-600328 policy]
> > regarding visibility of stream artifacts. Any default module stream
> > that is not in compliance by one week before Beta Freeze will cease to
> > be a default stream.
>
> What are the packagers of ursine packages that are shadowed by their modular
> counterparts supposed to do here (assuming such shadowing happens)?
>
> What are the packagers of modular packages that are shadowed by their ursine
> counterparts supposed to do here (assuming such shadowing happens)?
>
> > ...
> > == How To Test ==
> > # Build a modular stream
> > # Make that stream a default stream (or a buildroot override)
> > # Build a non-modular RPM that requires an artifact RPM from the modular
stream.
>
> How can I test this as an ursine package maintainer assuming I have build
> dependencies that were ursine packages but now will be form modules?
>
> > ...
> > == Contingency Plan ==
> > * Contingency mechanism: Disable the buildroot inheritance in Koji to
> > revert to the current behavior.
>
> Assuming ursine packges will be retired from Fedora, what is the contingency
> there? Consider this scenario:
>
> 1. maintainers of the ook module welcome this change and finally they retire ook
> from ursine Fedora, shipping only the default ook:12 modular stream.
> 2. maintainers of ook-cool and ook-free happily build against modular ook:12
> 3. a previously unknown fundamental flaw in design triggers the contingency plan
> 4. ook-cool and ook-free FTBFS, maintainer of ook do no longer wish to maintain
> ook as ursine package, because the entire depndncy tree to make that happen was
> gone/updated/downgraded in the meantime
>
> Who will revert ook to the previous working state? Is it the change owner?
>
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
10:59 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 10. 10. 19 17:46, Igor Gnatenko wrote:
On Thu, Oct 10, 2019 at 12:52 AM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 09. 10. 19 22:46, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>
>> Enable module default streams in the buildroot repository for modular
>> and non-modular RPMs.
>>
>> == Summary ==
>> This Change (colloquially referred to as "Ursa Prime") enables the
>> Koji build-system to include the RPM artifacts provided by module
>> default streams in the buildroot when building non-modular (or
>> "traditional") RPMs.
>>
>> == Owner ==
>> * Name: [[User:Sgallagh| Stephen Gallagher]]
>> * Email: sgallagh(a)redhat.com
>> * Responsible WG: Modularity WG
>>
>> == Detailed Description ==
>> As a major part of the Modularity design, we have a concept of default
>> module streams. These streams are built as modules, but the RPM
>> artifacts they deliver are intended to be used just like non-modular
>> RPMS. The aspirational goal is that a user of the system who never
>> executes a module-specific command (such as `dnf module install
>> nodejs:8`) should experience no meaningful changes in behavior from
>> how they would interact with a completely non-modular system. In
>> practice, this may mean that the informational output of package
>> managers may indicate that modules are being enabled and used, but a
>> user that does not have a specific reason to interact with the
>> selection of a module stream should have that managed on their behalf.
>
> If this is the goal of default modular streams, wouldn't it be in fact much
> easier to keep the default versions as urisne packages?
That means, you have 2 different workflows: for default version and
for an additional one.
When branching happens, instead of just updating one file which points
to the new default, you would have to create new stream, retire the
one which is about to become default and update package in
non-modular.
Yes. It puts additional (arguably fairly trivial) work for the modular
maintainers. Other than that, it makes lives of every other maiantainar, of the
dnf maintainers and of Fedora users easier.
I call it a good deal.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5:44 a.m.
On Thursday, October 10, 2019 5:59:16 PM CEST Miro Hrončok wrote:
On 10. 10. 19 17:46, Igor Gnatenko wrote:
> On Thu, Oct 10, 2019 at 12:52 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>>
>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>
>>> Enable module default streams in the buildroot repository for modular
>>> and non-modular RPMs.
>>>
>>> == Summary ==
>>> This Change (colloquially referred to as "Ursa Prime") enables
the
>>> Koji build-system to include the RPM artifacts provided by module
>>> default streams in the buildroot when building non-modular (or
>>> "traditional") RPMs.
>>>
>>> == Owner ==
>>> * Name: [[User:Sgallagh| Stephen Gallagher]]
>>> * Email: sgallagh(a)redhat.com
>>> * Responsible WG: Modularity WG
>>>
>>> == Detailed Description ==
>>> As a major part of the Modularity design, we have a concept of default
>>> module streams. These streams are built as modules, but the RPM
>>> artifacts they deliver are intended to be used just like non-modular
>>> RPMS. The aspirational goal is that a user of the system who never
>>> executes a module-specific command (such as `dnf module install
>>> nodejs:8`) should experience no meaningful changes in behavior from
>>> how they would interact with a completely non-modular system. In
>>> practice, this may mean that the informational output of package
>>> managers may indicate that modules are being enabled and used, but a
>>> user that does not have a specific reason to interact with the
>>> selection of a module stream should have that managed on their behalf.
>>
>> If this is the goal of default modular streams, wouldn't it be in fact much
>> easier to keep the default versions as urisne packages?
>
> That means, you have 2 different workflows: for default version and
> for an additional one.
> When branching happens, instead of just updating one file which points
> to the new default, you would have to create new stream, retire the
> one which is about to become default and update package in
> non-modular.
Yes. It puts additional (arguably fairly trivial) work for the modular
maintainers. Other than that, it makes lives of every other maiantainar, of the
dnf maintainers and of Fedora users easier.
I call it a good deal.
+1, that's pattern we decided to do in case of PostgreSQL.
Pavel
11:09 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 10. 10. 19 0:46, Miro Hrončok wrote:> What I miss in the description is:
1. How does this thing actually work? is there an additional repository composed
from the default streams available in Koji only?
2. How are conflicts between packages from the default streams and ursine
package be handled?
3. What is the local experience if the packager is using mock. What if they are
using rpmbuild directly?
4. How are we handling default streams with dependencies on non-default streams
of other modules?
I realize that the discussion got side tracked because of my other higher level
concerns, but I'd still like to know about this.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:31 a.m.
Sorry for the long overdue reply here. Answers to your questions are inline.
On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
...
What I miss in the description is:
1. How does this thing actually work? is there an additional repository composed
from the default streams available in Koji only?
An additional repository is composed from the default streams
(including the module metadata) and is merged with the standard
buildroot using Koji's new "bare" mode. This buildroot will be
available for non-Koji purposes (such as mock).
2. How are conflicts between packages from the default streams and
ursine
package be handled?
As long as the module metadata is present, an artifact with a given
name from an enabled or default module will effectively "hide" the
non-modular version.
Unless you mean "conflicts" in some other sense, in which case I think
RPM-level conflicts should handle it.
3. What is the local experience if the packager is using mock. What
if they are
using rpmbuild directly?
For mock, if it's pointed at the Koji-hosted buildroot repository, it
will work exactly as Koji does (with no local changes required). If
it's pointed at the public mirrors, it will need to have the Modular
repos enabled and it will have whatever is in the standard buildroot
plus default streams available there. One purpose of this Change is to
reconcile those two cases so they are the same. Right now, the Koji
buildroot will be slightly different from the one mock would see
because FESCo asked us to only allow two modules in the Koji buildroot
for testing. Prior to this Change, the Koji buildroot would have only
the non-modular content.
4. How are we handling default streams with dependencies on
non-default streams
of other modules?
Throughout these discussions, it has been made clear that the best
option for now will be to disallow default streams from depending on
non-default streams in Fedora. This *can* (but may not) change in the
future once we have worked out the explicit-vs-implicit enablement and
upgrade questions. But for now, it seems reasonable to put that policy
constraint on it.
12:21 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 13. 11. 19 18:31, Stephen Gallagher wrote:
Sorry for the long overdue reply here. Answers to your questions are
inline.
Thank you.
On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
...
> What I miss in the description is:
>
> 1. How does this thing actually work? is there an additional repository composed
> from the default streams available in Koji only?
An additional repository is composed from the default streams
(including the module metadata) and is merged with the standard
buildroot using Koji's new "bare" mode. This buildroot will be
available for non-Koji purposes (such as mock).
Will this buildroot repository be mirrored or will it be more like the current
"local" repository defined in mock configs?
For reference, this is currently in the rawhide mock config:
[local]
name=local
baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
cost=2000
enabled=0
skip_if_unavailable=False
My concern is that using the "local" repository for mock builds is quite slow
and IIRC puts Koji under unnecessary load.
> 2. How are conflicts between packages from the default streams
and ursine
> package be handled?
As long as the module metadata is present, an artifact with a given
name from an enabled or default module will effectively "hide" the
non-modular version.
I have a hard time to understand this. Could you please dive into the details?
"As long"? What does that mean? When they are not present and when they are
present?
"an enabled or default module"? Who enables a module in the buildroot? Is it
releng? The packager? Somebody else? Is it an explicit or implicit action?
Suppose we put maven:3.5 into the buildroot (I know we don't do that, but that
was the idea origianlly). How do I as a a maintainer of an ursine maven-foo
package BuildRequire the ursine maven package?
Unless you mean "conflicts" in some other sense, in which
case I think
RPM-level conflicts should handle it.
You understood "conflicts" exactly as I've meant it here, sorry for not
being
more explicit.
> 3. What is the local experience if the packager is using mock.
What if they are
> using rpmbuild directly?
For mock, if it's pointed at the Koji-hosted buildroot repository, it
will work exactly as Koji does (with no local changes required).
Makes sense, however, the information I miss is: Will it be pointed to the
Koji-hosted buildroot repository by default or not? My point was that this
should be part of the proposal before it was approved.
If
it's pointed at the public mirrors, it will need to have the Modular
repos enabled and it will have whatever is in the standard buildroot
plus default streams available there.
Will it be pointed to the public mirrors with the Modular repos enabled?
It previously was like that for a while and it was reverted as a wrong behavior.
One purpose of this Change is to
reconcile those two cases so they are the same.
If the cases are the same, isn't it actually what was originally proposed as
Ursa Major? I am seriously confused here, because I understood this change as it
is explicitly doing a different thing, a nonmodular repository composed of the
modular content. So that our downstreams can consume it "traditionally" without
adapting their tooling.
What you are describing here sounds like we are doing Ursa Major, renamed to
Ursa Prime, because Ursa Prime was a no-go. Because I don't assume ill intent, I
am absolutely sure that there is something I'm not getting here, but it it very
hard to figure out what.
Could you please, in great detail describe the differences between the two and
how this approved change addresses all the reasons that the previous attempt was
forbidden?
Right now, the Koji
buildroot will be slightly different from the one mock would see
because FESCo asked us to only allow two modules in the Koji buildroot
for testing.
Why would mock "see" different things based on restrictions FESCo asked to do?
My primary concern is that by default, mock sees what Koji sees.
By this change and by what was approved by FESCo (to clarify: I voted against
this), we are adding new things to what Koji sees. So I am really not sure what
actually happens to what mock sees:
1. Do we add only the 2 FESCo approved modules to what mock sees?
2. Do we add all default modular streams to what mock sees?
3. Do we keep mock to only see ursine content?
My opinion:
1. is ideal, but was not approved nor documented in the change proposal
2. is not ideal and was not approved nor documented in the change proposal
3. is not ideal, but is status quo, hence requires no approval/change
Prior to this Change, the Koji buildroot would have only
the non-modular content.
Correct.
> 4. How are we handling default streams with dependencies on
non-default streams
> of other modules?
Throughout these discussions, it has been made clear that the best
option for now will be to disallow default streams from depending on
non-default streams in Fedora. This *can* (but may not) change in the
future once we have worked out the explicit-vs-implicit enablement and
upgrade questions. But for now, it seems reasonable to put that policy
constraint on it.
I agree. Let's make that happen **before** we actually start doing Ursa
Prime/Major/...?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
12:26 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 13. 11. 19 19:21, Miro Hrončok wrote:
What you are describing here sounds like we are doing Ursa Major,
renamed to
Ursa Prime, because Ursa Prime was a no-go.
"because Ursa Major was a no-go"
Too many terms, sorry.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:15 p.m.
On Wed, Nov 13, 2019 at 1:24 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 13. 11. 19 18:31, Stephen Gallagher wrote:
> Sorry for the long overdue reply here. Answers to your questions are inline.
Thank you.
> On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
> ...
>> What I miss in the description is:
>>
>> 1. How does this thing actually work? is there an additional repository
composed
>> from the default streams available in Koji only?
>
> An additional repository is composed from the default streams
> (including the module metadata) and is merged with the standard
> buildroot using Koji's new "bare" mode. This buildroot will be
> available for non-Koji purposes (such as mock).
Will this buildroot repository be mirrored or will it be more like the current
"local" repository defined in mock configs?
For reference, this is currently in the rawhide mock config:
[local]
name=local
baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
cost=2000
enabled=0
skip_if_unavailable=False
My concern is that using the "local" repository for mock builds is quite slow
and IIRC puts Koji under unnecessary load.
The plan for the moment is to have it be the "local" repository. Given
that it will at least start out fairly small (and not need deltarpms),
we were going to see how well it held up. The longer-term plan will
probably be to mirror it.
>> 2. How are conflicts between packages from the default
streams and ursine
>> package be handled?
>
> As long as the module metadata is present, an artifact with a given
> name from an enabled or default module will effectively "hide" the
> non-modular version.
I have a hard time to understand this. Could you please dive into the details?
"As long"? What does that mean? When they are not present and when they are
present?
Sorry, that was an English colloquialism. Basically I meant "unless we
screw up and don't sync the metadata to the repository". So as long as
a module stream providing an artifact with a given name is enabled or
default, non-modular RPMs of the same name will be invisible to the
solver.
"an enabled or default module"? Who enables a module in the
buildroot? Is it
releng? The packager? Somebody else? Is it an explicit or implicit action?
"enabled" could happen as part of a mock config. That would be an
explicit action on the part of the person creating the mock config.
For the purposes of the discussion of Ursa Prime, you can ignore
"enabled". I was trying to be thorough and ended up being confusing
instead.
Suppose we put maven:3.5 into the buildroot (I know we don't do
that, but that
was the idea origianlly). How do I as a a maintainer of an ursine maven-foo
package BuildRequire the ursine maven package?
You don't.
Making a stream into the default stream is effectively making it
equivalent to a non-modular package for most purposes. It already
behaves this way for client installations; the content from a default
stream is the only version DNF "sees" if it happens to override a
non-modular package. On the client system, you'd need to `dnf module
disable maven` to be able to pull down the non-modular package.
Ursa Prime is basically a Change to have the Koji build environment
behave the same way as the local environment would.
This is why we've been discussing the need for the artifacts of
default streams to behave *exactly* as if they were non-modular once
delivered. Because they effectively are. They were built differently
(though still in our infrastructure), but once built the requirement
is that they must then behave like the non-modular distro would have.
(Side-note: We retired the term "ursine" and replaced it with
"non-modular" because it was confusing for non-native English
speakers.)
> Unless you mean "conflicts" in some other sense, in
which case I think
> RPM-level conflicts should handle it.
You understood "conflicts" exactly as I've meant it here, sorry for not
being
more explicit.
No problem, just wanted to make sure I was answering the correct question.
>> 3. What is the local experience if the packager is using
mock. What if they are
>> using rpmbuild directly?
>
> For mock, if it's pointed at the Koji-hosted buildroot repository, it
> will work exactly as Koji does (with no local changes required).
Makes sense, however, the information I miss is: Will it be pointed to the
Koji-hosted buildroot repository by default or not? My point was that this
should be part of the proposal before it was approved.
Yes, and you're right I will need to update the Change.
> If
> it's pointed at the public mirrors, it will need to have the Modular
> repos enabled and it will have whatever is in the standard buildroot
> plus default streams available there.
Will it be pointed to the public mirrors with the Modular repos enabled?
It previously was like that for a while and it was reverted as a wrong behavior.
It wasn't reverted as "wrong behavior", it was reverted because libdnf
at the time had a serious bug that mock was hitting. We never switched
it back on, but it should work properly now.
What we probably want to do is point it at the public mirrors, but
have mirrormanager just point at Koji for now while we're proving it
out.
> One purpose of this Change is to
> reconcile those two cases so they are the same.
If the cases are the same, isn't it actually what was originally proposed as
Ursa Major? I am seriously confused here, because I understood this change as it
is explicitly doing a different thing, a nonmodular repository composed of the
modular content. So that our downstreams can consume it "traditionally"
without
adapting their tooling.
What you're describing is Ursa Major. It was a plan to generate a
non-modular repository with content from modules. FESCo rejected this
outright because it required manual configuration to keep it working.
We replaced it with Ursa Prime, which was built around the idea that
we could track default streams and that would ensure things stayed in
sync.
In the repo created by Ursa Prime, it *does* carry modular metadata
and relies on libdnf inside the mock environment to retrieve the
correct packages.
What you are describing here sounds like we are doing Ursa Major,
renamed to
Ursa Prime, because Ursa Prime was a no-go. Because I don't assume ill intent, I
am absolutely sure that there is something I'm not getting here, but it it very
hard to figure out what.
Could you please, in great detail describe the differences between
the two and
how this approved change addresses all the reasons that the previous attempt was
forbidden?
Well, the irony here is that throughout this conversation, we've
basically settled on policy that would have made Ursa Major work just
fine in Fedora...
The big problem with Ursa Major is that we knew that it had a clear
design flaw: it could not work in cases where two default streams
depended on differing streams of another module. Ursa Major worked
entirely by Koji tagging; it tagged certain builds into the buildroot
directly. If you have a situation where two different module streams
are both default and both carry a package with the same name, both
would be tagged in and libdnf would end up picking whichever had the
higher NVR (because it didn't have any of the module metadata to
decide which one it actually needed).
The secondary problem was that (partly because of the first problem),
the tagging rules had to be maintained manually because we had no
meaningful way to determine the presence of conflicts. We couldn't
just script it to pull in all the defaults and call it a day, because
of the known conflict problem.
FESCo rejected Ursa Prime largely because infra asserted that
maintaining that tagging list would be prohibitively difficult as
Modularity became more ubiquitous. However, now that we've basically
decided on "default streams may only depend on default streams", the
original Ursa Major would have worked fine and been scriptable to just
use the default streams.
Ursa Prime is a completely different approach. Instead of trying to
tag packages so that Koji's internal buildroot-generator includes
their contents, we instead do a createrepo for the artifacts of the
module default streams and then configure Koji to include that
repository as an "external repo". This modular-buildroot external repo
includes all of the modular metadata (unlike the internal Koji
buildroot approach of Ursa Major) and thus mock can look up and
install whatever packages are in the default streams or non-modular
repository (so long as they aren't hidden by the default module
version). That's what I meant above about how Ursa Prime makes it
behave more like mock on the local system would work if you included
the module repos and changed nothing else. They'll both use libdnf's
logic to pull the appropriate packages.
> Right now, the Koji
> buildroot will be slightly different from the one mock would see
> because FESCo asked us to only allow two modules in the Koji buildroot
> for testing.
Why would mock "see" different things based on restrictions FESCo asked to do?
My primary concern is that by default, mock sees what Koji sees.
By this change and by what was approved by FESCo (to clarify: I voted against
this), we are adding new things to what Koji sees. So I am really not sure what
actually happens to what mock sees:
1. Do we add only the 2 FESCo approved modules to what mock sees?
2. Do we add all default modular streams to what mock sees?
3. Do we keep mock to only see ursine content?
My opinion:
1. is ideal, but was not approved nor documented in the change proposal
2. is not ideal and was not approved nor documented in the change proposal
3. is not ideal, but is status quo, hence requires no approval/change
> Prior to this Change, the Koji buildroot would have only
> the non-modular content.
Correct.
>> 4. How are we handling default streams with dependencies on non-default streams
>> of other modules?
>
> Throughout these discussions, it has been made clear that the best
> option for now will be to disallow default streams from depending on
> non-default streams in Fedora. This *can* (but may not) change in the
> future once we have worked out the explicit-vs-implicit enablement and
> upgrade questions. But for now, it seems reasonable to put that policy
> constraint on it.
I agree. Let's make that happen **before** we actually start doing Ursa
Prime/Major/...?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
4:42 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 13. 11. 19 22:15, Stephen Gallagher wrote:
On Wed, Nov 13, 2019 at 1:24 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 13. 11. 19 18:31, Stephen Gallagher wrote:
>> Sorry for the long overdue reply here. Answers to your questions are inline.
>
> Thank you.
>
>> On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>> ...
>>> What I miss in the description is:
>>>
>>> 1. How does this thing actually work? is there an additional repository
composed
>>> from the default streams available in Koji only?
>>
>> An additional repository is composed from the default streams
>> (including the module metadata) and is merged with the standard
>> buildroot using Koji's new "bare" mode. This buildroot will be
>> available for non-Koji purposes (such as mock).
>
> Will this buildroot repository be mirrored or will it be more like the current
> "local" repository defined in mock configs?
>
> My concern is that using the "local" repository for mock builds is quite
slow
> and IIRC puts Koji under unnecessary load.
>
The plan for the moment is to have it be the "local" repository. Given
that it will at least start out fairly small (and not need deltarpms),
we were going to see how well it held up. The longer-term plan will
probably be to mirror it.
OK. Since currently, nothing really depends on the modules added there, it makes
very little difference. I'm glad that there is a plan for the future.
>>> 2. How are conflicts between packages from the default
streams and ursine
>>> package be handled?
>>
>> As long as the module metadata is present, an artifact with a given
>> name from an enabled or default module will effectively "hide" the
>> non-modular version.
>
> I have a hard time to understand this. Could you please dive into the details?
>
> "As long"? What does that mean? When they are not present and when they are
present?
>
Sorry, that was an English colloquialism. Basically I meant "unless we
screw up and don't sync the metadata to the repository". So as long as
a module stream providing an artifact with a given name is enabled or
default, non-modular RPMs of the same name will be invisible to the
solver.
So the buildroot enabled modular repository contains only the 2 approved modules
including their modular metadata...
> "an enabled or default module"? Who enables a module in
the buildroot? Is it
> releng? The packager? Somebody else? Is it an explicit or implicit action?
>
"enabled" could happen as part of a mock config. That would be an
explicit action on the part of the person creating the mock config.
For the purposes of the discussion of Ursa Prime, you can ignore
"enabled". I was trying to be thorough and ended up being confusing
instead.
...and in Koji mock config, both are enabled and/or default?
> Suppose we put maven:3.5 into the buildroot (I know we don't
do that, but that
> was the idea origianlly). How do I as a a maintainer of an ursine maven-foo
> package BuildRequire the ursine maven package?
>
You don't.
Making a stream into the default stream is effectively making it
equivalent to a non-modular package for most purposes. It already
behaves this way for client installations; the content from a default
stream is the only version DNF "sees" if it happens to override a
non-modular package. On the client system, you'd need to `dnf module
disable maven` to be able to pull down the non-modular package.
I wonder what is the plan to handle this.
I see only two proper ways out of here:
1) We only allow default modular streams if they only contain packages that are
retired from non-modular Fedora. Any request fro a default stream must be
coordinated with all the relevant package maintainers and those packages must be
retired from Fedora.
2) No default modular streams. (Yes, that again.)
In my opinion, if we allow the non-modular packages to stay when there is a
default modular streams that "overrides" them, while effectively, our
maintainers cannot use the non-modular packages in the buildroot and our users
can only install the non-modular packages if they know how to disable a module,
it creates a very confusing situation.
Ursa Prime is basically a Change to have the Koji build environment
behave the same way as the local environment would.
This is why we've been discussing the need for the artifacts of
default streams to behave *exactly* as if they were non-modular once
delivered. Because they effectively are. They were built differently
(though still in our infrastructure), but once built the requirement
is that they must then behave like the non-modular distro would have.
I hope we agree that they currently don't behave exactly like non-modular.
(Side-note: We retired the term "ursine" and replaced it
with
"non-modular" because it was confusing for non-native English
speakers.)
Excellent choice. It was indeed rather confusing. Thank you.
Side-side-note: In my opinion, "Ursa Major/Prime" is equally confusing.
>>> 3. What is the local experience if the packager is using
mock. What if they are
>>> using rpmbuild directly?
>>
>> For mock, if it's pointed at the Koji-hosted buildroot repository, it
>> will work exactly as Koji does (with no local changes required).
>
> Makes sense, however, the information I miss is: Will it be pointed to the
> Koji-hosted buildroot repository by default or not? My point was that this
> should be part of the proposal before it was approved.
>
Yes, and you're right I will need to update the Change.
>> If
>> it's pointed at the public mirrors, it will need to have the Modular
>> repos enabled and it will have whatever is in the standard buildroot
>> plus default streams available there.
>
> Will it be pointed to the public mirrors with the Modular repos enabled?
> It previously was like that for a while and it was reverted as a wrong behavior.
>
It wasn't reverted as "wrong behavior", it was reverted because libdnf
at the time had a serious bug that mock was hitting. We never switched
it back on, but it should work properly now.
I disagree. IIRC it was disabled because it didn't match the Koji behavior.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
By "wrong behavior" I haven't meant "wrong forever", I meant
"wrong at the moment".
At this moment, *not having* the 2 approved modules in default mock config is
equally "wrong behavior".
What we probably want to do is point it at the public mirrors, but
have mirrormanager just point at Koji for now while we're proving it
out.
What I think we should do is to always try to match Koji as much as possible.
As long as Koji uses special repo, mock should use the same special repo or its
mirror. Not the entire modular repository. Not sure what repo exactly you meant
here by "public mirrors".
>> One purpose of this Change is to
>> reconcile those two cases so they are the same.
They were the same until this change. No modules in Koji, no modules in mock.
> ...
Well, the irony here is that throughout this conversation, we've
basically settled on policy that would have made Ursa Major work just
fine in Fedora...
The big problem with Ursa Major is that we knew that it had a clear
design flaw: it could not work in cases where two default streams
depended on differing streams of another module. Ursa Major worked
entirely by Koji tagging; it tagged certain builds into the buildroot
directly. If you have a situation where two different module streams
are both default and both carry a package with the same name, both
would be tagged in and libdnf would end up picking whichever had the
higher NVR (because it didn't have any of the module metadata to
decide which one it actually needed).
The secondary problem was that (partly because of the first problem),
the tagging rules had to be maintained manually because we had no
meaningful way to determine the presence of conflicts. We couldn't
just script it to pull in all the defaults and call it a day, because
of the known conflict problem.
FESCo rejected Ursa Prime largely because infra asserted that
maintaining that tagging list would be prohibitively difficult as
Modularity became more ubiquitous. However, now that we've basically
decided on "default streams may only depend on default streams", the
original Ursa Major would have worked fine and been scriptable to just
use the default streams.
Ursa Prime is a completely different approach. Instead of trying to
tag packages so that Koji's internal buildroot-generator includes
their contents, we instead do a createrepo for the artifacts of the
module default streams and then configure Koji to include that
repository as an "external repo". This modular-buildroot external repo
includes all of the modular metadata (unlike the internal Koji
buildroot approach of Ursa Major) and thus mock can look up and
install whatever packages are in the default streams or non-modular
repository (so long as they aren't hidden by the default module
version). That's what I meant above about how Ursa Prime makes it
behave more like mock on the local system would work if you included
the module repos and changed nothing else. They'll both use libdnf's
logic to pull the appropriate packages.
Thanks for explaining that difference.
>> Right now, the Koji
>> buildroot will be slightly different from the one mock would see
>> because FESCo asked us to only allow two modules in the Koji buildroot
>> for testing.
>
> Why would mock "see" different things based on restrictions FESCo asked to
do?
> My primary concern is that by default, mock sees what Koji sees.
>
> By this change and by what was approved by FESCo (to clarify: I voted against
> this), we are adding new things to what Koji sees. So I am really not sure what
> actually happens to what mock sees:
>
> 1. Do we add only the 2 FESCo approved modules to what mock sees?
> 2. Do we add all default modular streams to what mock sees?
> 3. Do we keep mock to only see ursine content?
>
> My opinion:
>
> 1. is ideal, but was not approved nor documented in the change proposal
> 2. is not ideal and was not approved nor documented in the change proposal
> 3. is not ideal, but is status quo, hence requires no approval/change
Not sure if this was left intentionally out, or missed in my e-mail, so I'm
explicitly stating that this remained unanswered.
>>> 4. How are we handling default streams with dependencies
on non-default streams
>>> of other modules?
>>
>> Throughout these discussions, it has been made clear that the best
>> option for now will be to disallow default streams from depending on
>> non-default streams in Fedora. This *can* (but may not) change in the
>> future once we have worked out the explicit-vs-implicit enablement and
>> upgrade questions. But for now, it seems reasonable to put that policy
>> constraint on it.
>
> I agree. Let's make that happen **before** we actually start doing Ursa
> Prime/Major/...?
Same here.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5:46 p.m.
On Wed, Nov 13, 2019 at 5:46 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 13. 11. 19 22:15, Stephen Gallagher wrote:
...
So the buildroot enabled modular repository contains only the 2
approved modules
including their modular metadata...
>> "an enabled or default module"? Who enables a module in the buildroot?
Is it
>> releng? The packager? Somebody else? Is it an explicit or implicit action?
>>
>
> "enabled" could happen as part of a mock config. That would be an
> explicit action on the part of the person creating the mock config.
> For the purposes of the discussion of Ursa Prime, you can ignore
> "enabled". I was trying to be thorough and ended up being confusing
> instead.
...and in Koji mock config, both are enabled and/or default?
They will be marked as default in the modular-buildroot, yes.
>> Suppose we put maven:3.5 into the buildroot (I know we don't do that, but
that
>> was the idea origianlly). How do I as a a maintainer of an ursine maven-foo
>> package BuildRequire the ursine maven package?
>>
>
> You don't.
>
> Making a stream into the default stream is effectively making it
> equivalent to a non-modular package for most purposes. It already
> behaves this way for client installations; the content from a default
> stream is the only version DNF "sees" if it happens to override a
> non-modular package. On the client system, you'd need to `dnf module
> disable maven` to be able to pull down the non-modular package.
I wonder what is the plan to handle this.
I see only two proper ways out of here:
1) We only allow default modular streams if they only contain packages that are
retired from non-modular Fedora. Any request fro a default stream must be
coordinated with all the relevant package maintainers and those packages must be
retired from Fedora.
2) No default modular streams. (Yes, that again.)
In my opinion, if we allow the non-modular packages to stay when there is a
default modular streams that "overrides" them, while effectively, our
maintainers cannot use the non-modular packages in the buildroot and our users
can only install the non-modular packages if they know how to disable a module,
it creates a very confusing situation.
You are correct. 1) is actually the policy[1] in Fedora today. There
are a couple modules that are in violation of the policy, which is the
cause of many of the issues we've been dealing with. The origin
Specifically: "[I]f the module stream masks part of the Traditional
RPM repos (e.g it replaces an existing RPM or it introduces a
non-trivial set of conflicts) it may not be made a default stream
without the express permission of FESCo. Unfortunately, the maven and
ant module streams were made default before this policy (and the
stronger one describing the api requirements) was in place. If this
had been submitted to FESCo today, it would have been refused.
[1] https://docs.fedoraproject.org/en-US/modularity/making-modules/managing-d...
> Ursa Prime is basically a Change to have the Koji build
environment
> behave the same way as the local environment would.
>
> This is why we've been discussing the need for the artifacts of
> default streams to behave *exactly* as if they were non-modular once
> delivered. Because they effectively are. They were built differently
> (though still in our infrastructure), but once built the requirement
> is that they must then behave like the non-modular distro would have.
I hope we agree that they currently don't behave exactly like non-modular.
Correct, we've beaten that horse already and identified the key pieces
that need to be fixed.
> (Side-note: We retired the term "ursine" and replaced
it with
> "non-modular" because it was confusing for non-native English
> speakers.)
Excellent choice. It was indeed rather confusing. Thank you.
Side-side-note: In my opinion, "Ursa Major/Prime" is equally confusing.
You're not wrong. But we needed some way to refer to "The project for
enabling the use of module artifacts in the non-modular buildroot"
that didn't take fifteen seconds to say every time. These are
essentially code-names, though. An end-user should never hear about
them and once it is in place and actually works, packagers should not
have to hear about it either.
>>>> 3. What is the local experience if the packager is
using mock. What if they are
>>>> using rpmbuild directly?
>>>
>>> For mock, if it's pointed at the Koji-hosted buildroot repository, it
>>> will work exactly as Koji does (with no local changes required).
>>
>> Makes sense, however, the information I miss is: Will it be pointed to the
>> Koji-hosted buildroot repository by default or not? My point was that this
>> should be part of the proposal before it was approved.
>>
>
> Yes, and you're right I will need to update the Change.
>
>>> If
>>> it's pointed at the public mirrors, it will need to have the Modular
>>> repos enabled and it will have whatever is in the standard buildroot
>>> plus default streams available there.
>>
>> Will it be pointed to the public mirrors with the Modular repos enabled?
>> It previously was like that for a while and it was reverted as a wrong
behavior.
>>
>
> It wasn't reverted as "wrong behavior", it was reverted because
libdnf
> at the time had a serious bug that mock was hitting. We never switched
> it back on, but it should work properly now.
I disagree. IIRC it was disabled because it didn't match the Koji behavior.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
By "wrong behavior" I haven't meant "wrong forever", I meant
"wrong at the moment".
At this moment, *not having* the 2 approved modules in default mock config is
equally "wrong behavior".
OK, I misunderstood what you were saying. I think we're on the same page now.
> What we probably want to do is point it at the public mirrors,
but
> have mirrormanager just point at Koji for now while we're proving it
> out.
What I think we should do is to always try to match Koji as much as possible.
As long as Koji uses special repo, mock should use the same special repo or its
mirror. Not the entire modular repository. Not sure what repo exactly you meant
here by "public mirrors".
This would actually differ from how mock works with non-modular RPMs
today. It points at the public mirrors of the Everything repo, not the
Koji repo (unless you make manual edits to the mock config).
This is why the original proposal was for Ursa Prime to just take
whatever was in the defaults for the public Modular repo (plus the
occasional override, when needed). So that it would be as close to
"point mock at the public repos" as possible. I think that's the way
we want this relationship to flow: that Koji's experience should match
the public one.
>>> One purpose of this Change is to
>>> reconcile those two cases so they are the same.
They were the same until this change. No modules in Koji, no modules in mock.
I mean that it was meant to ensure that if you added the Modular repos
to mock, you would get the same *effective* experience as you would
when you do a build in Koji. (As close to identical as you get today
when pointed at the Everything repo rather than the Koji buildroot
(aka "local") repo)
>> ...
> Well, the irony here is that throughout this conversation, we've
> basically settled on policy that would have made Ursa Major work just
> fine in Fedora...
>
> The big problem with Ursa Major is that we knew that it had a clear
> design flaw: it could not work in cases where two default streams
> depended on differing streams of another module. Ursa Major worked
> entirely by Koji tagging; it tagged certain builds into the buildroot
> directly. If you have a situation where two different module streams
> are both default and both carry a package with the same name, both
> would be tagged in and libdnf would end up picking whichever had the
> higher NVR (because it didn't have any of the module metadata to
> decide which one it actually needed).
>
> The secondary problem was that (partly because of the first problem),
> the tagging rules had to be maintained manually because we had no
> meaningful way to determine the presence of conflicts. We couldn't
> just script it to pull in all the defaults and call it a day, because
> of the known conflict problem.
>
> FESCo rejected Ursa Prime largely because infra asserted that
> maintaining that tagging list would be prohibitively difficult as
> Modularity became more ubiquitous. However, now that we've basically
> decided on "default streams may only depend on default streams", the
> original Ursa Major would have worked fine and been scriptable to just
> use the default streams.
>
> Ursa Prime is a completely different approach. Instead of trying to
> tag packages so that Koji's internal buildroot-generator includes
> their contents, we instead do a createrepo for the artifacts of the
> module default streams and then configure Koji to include that
> repository as an "external repo". This modular-buildroot external repo
> includes all of the modular metadata (unlike the internal Koji
> buildroot approach of Ursa Major) and thus mock can look up and
> install whatever packages are in the default streams or non-modular
> repository (so long as they aren't hidden by the default module
> version). That's what I meant above about how Ursa Prime makes it
> behave more like mock on the local system would work if you included
> the module repos and changed nothing else. They'll both use libdnf's
> logic to pull the appropriate packages.
Thanks for explaining that difference.
Was it sufficiently clear? It's a complicated topic and I want to make
sure I'm explaining it sufficiently.
>>> Right now, the Koji
>>> buildroot will be slightly different from the one mock would see
>>> because FESCo asked us to only allow two modules in the Koji buildroot
>>> for testing.
>>
>> Why would mock "see" different things based on restrictions FESCo
asked to do?
>> My primary concern is that by default, mock sees what Koji sees.
>>
Mock will only have access to either the koji modular-buildroot repo
or the public Modular repo. In general, we probably want to prefer
that people use the latter, but during the time period where we're
only allowing a subset of the default streams into the buildroot, we
have to point it at the koji modular-buildroot repo (or else it would
be picking up all the default streams).
>> By this change and by what was approved by FESCo (to
clarify: I voted against
>> this), we are adding new things to what Koji sees. So I am really not sure what
>> actually happens to what mock sees:
>>
>> 1. Do we add only the 2 FESCo approved modules to what mock sees?
If we point mock at the koji modular-buildroot, then mock will only
see what Koji sees.
>> 2. Do we add all default modular streams to what mock sees?
If we point mock at the public, mirrored Modular repo, it will get all
the default streams.
>> 3. Do we keep mock to only see ursine content?
>>
>> My opinion:
>>
>> 1. is ideal, but was not approved nor documented in the change proposal
>> 2. is not ideal and was not approved nor documented in the change proposal
>> 3. is not ideal, but is status quo, hence requires no approval/change
Not sure if this was left intentionally out, or missed in my e-mail, so I'm
explicitly stating that this remained unanswered.
I missed this somehow.
I'd say that we probably want to coordinate shipping a mock config
update that points at the koji modular buildroot at the same time that
we go live with Ursa Prime so that the Koji and mock experience
remains the same.
>>>> 4. How are we handling default streams with
dependencies on non-default streams
>>>> of other modules?
>>>
>>> Throughout these discussions, it has been made clear that the best
>>> option for now will be to disallow default streams from depending on
>>> non-default streams in Fedora. This *can* (but may not) change in the
>>> future once we have worked out the explicit-vs-implicit enablement and
>>> upgrade questions. But for now, it seems reasonable to put that policy
>>> constraint on it.
>>
>> I agree. Let's make that happen **before** we actually start doing Ursa
>> Prime/Major/...?
Same here.
I didn't interpret that as a question, since I'd pretty much agreed
already. But sure.
2:49 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Wed, Nov 13, 2019 at 04:15:57PM -0500, Stephen Gallagher wrote:
On Wed, Nov 13, 2019 at 1:24 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 13. 11. 19 18:31, Stephen Gallagher wrote:
> > Sorry for the long overdue reply here. Answers to your questions are inline.
>
> Thank you.
>
> > On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
> > ...
> >> What I miss in the description is:
> >>
> >> 1. How does this thing actually work? is there an additional repository
composed
> >> from the default streams available in Koji only?
> >
> > An additional repository is composed from the default streams
> > (including the module metadata) and is merged with the standard
> > buildroot using Koji's new "bare" mode. This buildroot will be
> > available for non-Koji purposes (such as mock).
>
> Will this buildroot repository be mirrored or will it be more like the current
> "local" repository defined in mock configs?
>
> For reference, this is currently in the rawhide mock config:
>
> [local]
> name=local
> baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/
> cost=2000
> enabled=0
> skip_if_unavailable=False
>
> My concern is that using the "local" repository for mock builds is quite
slow
> and IIRC puts Koji under unnecessary load.
>
The plan for the moment is to have it be the "local" repository. Given
that it will at least start out fairly small (and not need deltarpms),
we were going to see how well it held up. The longer-term plan will
probably be to mirror it.
Just a quick note, if you foresee some changes needed on MirrorManager to detect
and mirror this new repo you may want to reach to the MirrorManager maintainer
early enough so he can plan for them.
Pierre
10:44 a.m.
https://pagure.io/releng/issue/8880 - Include default_modules_scm_url
in platform 31 virtual module
platform *f32*?
Other than that, it would be nice to have more specific rules *when*
and *how* modules are checked to conform to the guidelines…
On Wed, Oct 9, 2019 at 10:51 PM Ben Cotton <bcotton(a)redhat.com> wrote:
>
> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
> Enable module default streams in the buildroot repository for modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
>
> == Detailed Description ==
> As a major part of the Modularity design, we have a concept of default
> module streams. These streams are built as modules, but the RPM
> artifacts they deliver are intended to be used just like non-modular
> RPMS. The aspirational goal is that a user of the system who never
> executes a module-specific command (such as `dnf module install
> nodejs:8`) should experience no meaningful changes in behavior from
> how they would interact with a completely non-modular system. In
> practice, this may mean that the informational output of package
> managers may indicate that modules are being enabled and used, but a
> user that does not have a specific reason to interact with the
> selection of a module stream should have that managed on their behalf.
>
> Similarly, the experience for package maintainers of non-modular
> packages should be unaffected by an RPM dependency moving from the
> non-modular repository into a default module stream. Up to the
> present, this has not been the case; no module stream content has been
> available in the non-modular buildroot for other packages to consume.
> Koji builds of non-modular RPMs have had only the other non-modular
> RPMs from that release available to their buildroots. In contrast,
> building on local systems has access to both the non-modular RPMs and
> the RPMs from any of the default module streams. With this Change,
> Koji builds will have the same behavior and be able to depend on
> content provided by default module streams. It also enables the same
> behavior for Modular builds: the `platform` stream will now include
> the contents of the default module streams for each release and do not
> need to be explicitly specified in the modulemd `buildrequires`.
>
> Note: This Change does not address the other major Modularity issue we
> are facing around distribution upgrades with differing default
> streams. When discussing this Change, please keep that topic separate.
>
> == Benefit to Fedora ==
>
> This will simplify the lives of package maintainers in Fedora in two
> primary ways. I'll use a hypothetical example of the Node.js
> interpreter and a JSApp package which is capable of running on Node.js
> 10 or 12 (but requires newer features than are provided by Node.js 8).
> Additionally, the JSApp package requires the same versions of Node.js
> at build-time.
>
> * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> streams. The `nodejs:10` stream is set as the default stream.
> * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> streams. The `nodejs:10` stream is set as the default stream.
> * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
> `nodejs:12` stream is set as the default stream. The `nodejs:14`
> stream will likely become available during the F31 lifetime.
> * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
> `nodejs:12` stream is set as the default stream. The `nodejs:14`
> stream will likely become available during the F32 lifetime.
>
> On Fedora 29 through 31, the Node.js package maintainer needs to build
> the `nodejs:10` package both as a module and as a non-modular RPM in
> the distribution so that the JSApp package can be built. With this
> Change, the Node.js package maintainer in Fedora 32+ will only need to
> build the various Node.js streams and make one of them the default
> stream. The packages from it will then be added to the buildroot for
> non-modular packages. This will also make the packaging process
> somewhat more efficient, as the maintainer needs only to manage the
> module stream and the MBS will build it for all configured platforms.
>
> Similarly, from the perspective of dependent maintainers, there will
> no longer be anxiety about needing to move their package to a module
> if one or more of their dependencies drops their non-modular version
> in favor of a default stream. Their builds will continue to work as
> they do today.
>
> == Scope ==
> * Proposal owners:
> # Update Packaging Guidelines with
> [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> for module default streams
> # Create a Pungi configuration to generate the buildroot from the
> default module streams.
> # Include `default_modules_scm_url` in the platform virtual module specification
> # Configure Koji tags for inheriting the new modular-defaults
> buildroot into the standard buildroot
>
> * Other developers:
>
> Packagers of default module streams will be required to conform to the
> [https://pagure.io/modularity/issue/146#comment-600328 policy]
> regarding visibility of stream artifacts. Any default module stream
> that is not in compliance by one week before Beta Freeze will cease to
> be a default stream.
>
> * Release engineering:
> # https://pagure.io/releng/issue/8879 - Create pungi config for
> Rawhide/F32 ursa prime buildroot
> # https://pagure.io/releng/issue/8880 - Include
> `default_modules_scm_url` in platform 31 virtual module
> # https://pagure.io/releng/issue/8881 - Configure Koji tags for
> inheriting f32-modular-buildroot
>
> * Policies and guidelines:
> The Modularity Packaging Guidelines will need to be updated to
> indicate the strict requirements on default streams.
> * Trademark approval: N/A (not needed for this Change)
>
> == Upgrade/compatibility impact ==
> This change is on the build-system side of things and should not
> impact the upgrade process directly.
>
> == How To Test ==
> # Build a modular stream
> # Make that stream a default stream (or a buildroot override)
> # Build a non-modular RPM that requires an artifact RPM from the modular stream.
>
> == User Experience ==
> This should not change the end-user experience.
>
> == Dependencies ==
> Nothing known that isn't listed in the scope.
>
> == Contingency Plan ==
> * Contingency mechanism: Disable the buildroot inheritance in Koji to
> revert to the current behavior.
> * Blocks release? Ambiguous: lack of complete implementation may
> indirectly cause blocking issues.
> * Blocks product? No
>
> == Documentation ==
>
>
> == Release Notes ==
> None needed, the Change is not user-facing.
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedora...
12:59 a.m.
On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
== Detailed Description ==
As a major part of the Modularity design, we have a concept of default
module streams. These streams are built as modules, but the RPM
artifacts they deliver are intended to be used just like non-modular
RPMS. The aspirational goal is that a user of the system who never
executes a module-specific command (such as `dnf module install
nodejs:8`) should experience no meaningful changes in behavior from
how they would interact with a completely non-modular system. In
practice, this may mean that the informational output of package
managers may indicate that modules are being enabled and used, but a
user that does not have a specific reason to interact with the
selection of a module stream should have that managed on their behalf.
Similarly, the experience for package maintainers of non-modular
packages should be unaffected by an RPM dependency moving from the
non-modular repository into a default module stream. Up to the
present, this has not been the case; no module stream content has been
available in the non-modular buildroot for other packages to consume.
Koji builds of non-modular RPMs have had only the other non-modular
RPMs from that release available to their buildroots. In contrast,
building on local systems has access to both the non-modular RPMs and
the RPMs from any of the default module streams. With this Change,
Koji builds will have the same behavior and be able to depend on
content provided by default module streams. It also enables the same
behavior for Modular builds: the `platform` stream will now include
the contents of the default module streams for each release and do not
need to be explicitly specified in the modulemd `buildrequires`.
Note: This Change does not address the other major Modularity issue we
are facing around distribution upgrades with differing default
streams. When discussing this Change, please keep that topic separate.
== Benefit to Fedora ==
This will simplify the lives of package maintainers in Fedora in two
primary ways. I'll use a hypothetical example of the Node.js
interpreter and a JSApp package which is capable of running on Node.js
10 or 12 (but requires newer features than are provided by Node.js 8).
Additionally, the JSApp package requires the same versions of Node.js
at build-time.
* Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F31 lifetime.
* Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F32 lifetime.
On Fedora 29 through 31, the Node.js package maintainer needs to build
the `nodejs:10` package both as a module and as a non-modular RPM in
the distribution so that the JSApp package can be built. With this
Change, the Node.js package maintainer in Fedora 32+ will only need to
build the various Node.js streams and make one of them the default
stream. The packages from it will then be added to the buildroot for
non-modular packages. This will also make the packaging process
somewhat more efficient, as the maintainer needs only to manage the
module stream and the MBS will build it for all configured platforms.
Similarly, from the perspective of dependent maintainers, there will
no longer be anxiety about needing to move their package to a module
if one or more of their dependencies drops their non-modular version
in favor of a default stream. Their builds will continue to work as
they do today.
== Scope ==
* Proposal owners:
# Update Packaging Guidelines with
[https://pagure.io/modularity/issue/146#comment-600328 requirements]
for module default streams
# Create a Pungi configuration to generate the buildroot from the
default module streams.
# Include `default_modules_scm_url` in the platform virtual module
specification
# Configure Koji tags for inheriting the new
modular-defaults
buildroot into the standard buildroot
* Other developers:
Packagers of default module streams will be required to conform to the
[https://pagure.io/modularity/issue/146#comment-600328 policy]
regarding visibility of stream artifacts. Any default module stream
that is not in compliance by one week before Beta Freeze will cease to
be a default stream.
* Release engineering:
# https://pagure.io/releng/issue/8879 - Create pungi config for
Rawhide/F32 ursa prime buildroot
# https://pagure.io/releng/issue/8880 - Include
`default_modules_scm_url` in platform 31 virtual module
# https://pagure.io/releng/issue/8881 - Configure Koji tags for
inheriting f32-modular-buildroot
* Policies and guidelines:
The Modularity Packaging Guidelines will need to be updated to
indicate the strict requirements on default streams.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This change is on the build-system side of things and should not
impact the upgrade process directly.
== How To Test ==
# Build a modular stream
# Make that stream a default stream (or a buildroot override)
# Build a non-modular RPM that requires an artifact RPM from the modular
stream.
== User Experience ==
This should not change the end-user experience.
== Dependencies ==
Nothing known that isn't listed in the scope.
== Contingency Plan ==
* Contingency mechanism: Disable the buildroot inheritance in Koji to
revert to the current behavior.
* Blocks release? Ambiguous: lack of complete implementation may
indirectly cause blocking issues.
* Blocks product? No
== Documentation ==
== Release Notes ==
None needed, the Change is not user-facing.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
That sounds like a really bad idea. Isn't the entire goal for traditional RPMs
to exist separately from modules? This will lead to more packages being
maintained as modules only, and I can only see it getting worse from there.
Are there any actual benefits to this? I can't think of any.
--
John M. Harris, Jr.
Splentity
1:42 a.m.
On Mon, Oct 14, 2019 at 9:00 AM John M. Harris Jr <johnmh(a)splentity.com>
wrote:
On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
> Enable module default streams in the buildroot repository for modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
>
> == Detailed Description ==
> As a major part of the Modularity design, we have a concept of default
> module streams. These streams are built as modules, but the RPM
> artifacts they deliver are intended to be used just like non-modular
> RPMS. The aspirational goal is that a user of the system who never
> executes a module-specific command (such as `dnf module install
> nodejs:8`) should experience no meaningful changes in behavior from
> how they would interact with a completely non-modular system. In
> practice, this may mean that the informational output of package
> managers may indicate that modules are being enabled and used, but a
> user that does not have a specific reason to interact with the
> selection of a module stream should have that managed on their behalf.
>
> Similarly, the experience for package maintainers of non-modular
> packages should be unaffected by an RPM dependency moving from the
> non-modular repository into a default module stream. Up to the
> present, this has not been the case; no module stream content has been
> available in the non-modular buildroot for other packages to consume.
> Koji builds of non-modular RPMs have had only the other non-modular
> RPMs from that release available to their buildroots. In contrast,
> building on local systems has access to both the non-modular RPMs and
> the RPMs from any of the default module streams. With this Change,
> Koji builds will have the same behavior and be able to depend on
> content provided by default module streams. It also enables the same
> behavior for Modular builds: the `platform` stream will now include
> the contents of the default module streams for each release and do not
> need to be explicitly specified in the modulemd `buildrequires`.
>
> Note: This Change does not address the other major Modularity issue we
> are facing around distribution upgrades with differing default
> streams. When discussing this Change, please keep that topic separate.
>
> == Benefit to Fedora ==
>
> This will simplify the lives of package maintainers in Fedora in two
> primary ways. I'll use a hypothetical example of the Node.js
> interpreter and a JSApp package which is capable of running on Node.js
> 10 or 12 (but requires newer features than are provided by Node.js 8).
> Additionally, the JSApp package requires the same versions of Node.js
> at build-time.
>
> * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> streams. The `nodejs:10` stream is set as the default stream.
> * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> streams. The `nodejs:10` stream is set as the default stream.
> * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
> `nodejs:12` stream is set as the default stream. The `nodejs:14`
> stream will likely become available during the F31 lifetime.
> * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
> `nodejs:12` stream is set as the default stream. The `nodejs:14`
> stream will likely become available during the F32 lifetime.
>
> On Fedora 29 through 31, the Node.js package maintainer needs to build
> the `nodejs:10` package both as a module and as a non-modular RPM in
> the distribution so that the JSApp package can be built. With this
> Change, the Node.js package maintainer in Fedora 32+ will only need to
> build the various Node.js streams and make one of them the default
> stream. The packages from it will then be added to the buildroot for
> non-modular packages. This will also make the packaging process
> somewhat more efficient, as the maintainer needs only to manage the
> module stream and the MBS will build it for all configured platforms.
>
> Similarly, from the perspective of dependent maintainers, there will
> no longer be anxiety about needing to move their package to a module
> if one or more of their dependencies drops their non-modular version
> in favor of a default stream. Their builds will continue to work as
> they do today.
>
> == Scope ==
> * Proposal owners:
> # Update Packaging Guidelines with
> [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> for module default streams
> # Create a Pungi configuration to generate the buildroot from the
> default module streams.
> # Include `default_modules_scm_url` in the platform virtual module
> specification
# Configure Koji tags for inheriting the new
> modular-defaults
> buildroot into the standard buildroot
>
> * Other developers:
>
> Packagers of default module streams will be required to conform to the
> [https://pagure.io/modularity/issue/146#comment-600328 policy]
> regarding visibility of stream artifacts. Any default module stream
> that is not in compliance by one week before Beta Freeze will cease to
> be a default stream.
>
> * Release engineering:
> # https://pagure.io/releng/issue/8879 - Create pungi config for
> Rawhide/F32 ursa prime buildroot
> # https://pagure.io/releng/issue/8880 - Include
> `default_modules_scm_url` in platform 31 virtual module
> # https://pagure.io/releng/issue/8881 - Configure Koji tags for
> inheriting f32-modular-buildroot
>
> * Policies and guidelines:
> The Modularity Packaging Guidelines will need to be updated to
> indicate the strict requirements on default streams.
> * Trademark approval: N/A (not needed for this Change)
>
> == Upgrade/compatibility impact ==
> This change is on the build-system side of things and should not
> impact the upgrade process directly.
>
> == How To Test ==
> # Build a modular stream
> # Make that stream a default stream (or a buildroot override)
> # Build a non-modular RPM that requires an artifact RPM from the modular
> stream.
> == User Experience ==
> This should not change the end-user experience.
>
> == Dependencies ==
> Nothing known that isn't listed in the scope.
>
> == Contingency Plan ==
> * Contingency mechanism: Disable the buildroot inheritance in Koji to
> revert to the current behavior.
> * Blocks release? Ambiguous: lack of complete implementation may
> indirectly cause blocking issues.
> * Blocks product? No
>
> == Documentation ==
>
>
> == Release Notes ==
> None needed, the Change is not user-facing.
>
> --
> Ben Cotton
> He / Him / His
> Fedora Program Manager
> Red Hat
> TZ=America/Indiana/Indianapolis
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
> Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
> Archives:
>
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
That sounds like a really bad idea. Isn't the entire goal for traditional
RPMs
to exist separately from modules? This will lead to more packages being
maintained as modules only, and I can only see it getting worse from there.
Are there any actual benefits to this? I can't think of any.
IMHO it's exactly the opposite. E.g. Eclipse is moving to a module because
it requires Maven 3.6 which is available as a module only. If this was
implemented earlier we wouldn't have bothered making Eclipse module. To
continue if this is delayed osgi/swt apps which depend on parts of eclipse
will find it easier to just make their apps modules too and so on.
--
John M. Harris, Jr.
Splentity
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Alexander Kurtakov
Red Hat Eclipse Team
2:12 a.m.
On Sunday, October 13, 2019 11:42:41 PM MST Aleksandar Kurtakov wrote:
On Mon, Oct 14, 2019 at 9:00 AM John M. Harris Jr
<johnmh(a)splentity.com>
wrote:
> On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> >
> > Enable module default streams in the buildroot repository for modular
> > and non-modular RPMs.
> >
> > == Summary ==
> > This Change (colloquially referred to as "Ursa Prime") enables the
> > Koji build-system to include the RPM artifacts provided by module
> > default streams in the buildroot when building non-modular (or
> > "traditional") RPMs.
> >
> > == Owner ==
> > * Name: [[User:Sgallagh| Stephen Gallagher]]
> > * Email: sgallagh(a)redhat.com
> > * Responsible WG: Modularity WG
> >
> > == Detailed Description ==
> > As a major part of the Modularity design, we have a concept of default
> > module streams. These streams are built as modules, but the RPM
> > artifacts they deliver are intended to be used just like non-modular
> > RPMS. The aspirational goal is that a user of the system who never
> > executes a module-specific command (such as `dnf module install
> > nodejs:8`) should experience no meaningful changes in behavior from
> > how they would interact with a completely non-modular system. In
> > practice, this may mean that the informational output of package
> > managers may indicate that modules are being enabled and used, but a
> > user that does not have a specific reason to interact with the
> > selection of a module stream should have that managed on their behalf.
> >
> > Similarly, the experience for package maintainers of non-modular
> > packages should be unaffected by an RPM dependency moving from the
> > non-modular repository into a default module stream. Up to the
> > present, this has not been the case; no module stream content has been
> > available in the non-modular buildroot for other packages to consume.
> > Koji builds of non-modular RPMs have had only the other non-modular
> > RPMs from that release available to their buildroots. In contrast,
> > building on local systems has access to both the non-modular RPMs and
> > the RPMs from any of the default module streams. With this Change,
> > Koji builds will have the same behavior and be able to depend on
> > content provided by default module streams. It also enables the same
> > behavior for Modular builds: the `platform` stream will now include
> > the contents of the default module streams for each release and do not
> > need to be explicitly specified in the modulemd `buildrequires`.
> >
> > Note: This Change does not address the other major Modularity issue we
> > are facing around distribution upgrades with differing default
> > streams. When discussing this Change, please keep that topic separate.
> >
> > == Benefit to Fedora ==
> >
> > This will simplify the lives of package maintainers in Fedora in two
> > primary ways. I'll use a hypothetical example of the Node.js
> > interpreter and a JSApp package which is capable of running on Node.js
> > 10 or 12 (but requires newer features than are provided by Node.js 8).
> > Additionally, the JSApp package requires the same versions of Node.js
> > at build-time.
> >
> > * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > streams. The `nodejs:10` stream is set as the default stream.
> > * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > streams. The `nodejs:10` stream is set as the default stream.
> > * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
> > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > stream will likely become available during the F31 lifetime.
> > * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
> > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > stream will likely become available during the F32 lifetime.
> >
> > On Fedora 29 through 31, the Node.js package maintainer needs to build
> > the `nodejs:10` package both as a module and as a non-modular RPM in
> > the distribution so that the JSApp package can be built. With this
> > Change, the Node.js package maintainer in Fedora 32+ will only need to
> > build the various Node.js streams and make one of them the default
> > stream. The packages from it will then be added to the buildroot for
> > non-modular packages. This will also make the packaging process
> > somewhat more efficient, as the maintainer needs only to manage the
> > module stream and the MBS will build it for all configured platforms.
> >
> > Similarly, from the perspective of dependent maintainers, there will
> > no longer be anxiety about needing to move their package to a module
> > if one or more of their dependencies drops their non-modular version
> > in favor of a default stream. Their builds will continue to work as
> > they do today.
> >
> > == Scope ==
> > * Proposal owners:
> > # Update Packaging Guidelines with
> > [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> > for module default streams
> > # Create a Pungi configuration to generate the buildroot from the
> > default module streams.
> > # Include `default_modules_scm_url` in the platform virtual module
> > specification
>
> # Configure Koji tags for inheriting the new
>
> > modular-defaults
> > buildroot into the standard buildroot
> >
> > * Other developers:
> >
> > Packagers of default module streams will be required to conform to the
> > [https://pagure.io/modularity/issue/146#comment-600328 policy]
> > regarding visibility of stream artifacts. Any default module stream
> > that is not in compliance by one week before Beta Freeze will cease to
> > be a default stream.
> >
> > * Release engineering:
> > # https://pagure.io/releng/issue/8879 - Create pungi config for
> > Rawhide/F32 ursa prime buildroot
> > # https://pagure.io/releng/issue/8880 - Include
> > `default_modules_scm_url` in platform 31 virtual module
> > # https://pagure.io/releng/issue/8881 - Configure Koji tags for
> > inheriting f32-modular-buildroot
> >
> > * Policies and guidelines:
> > The Modularity Packaging Guidelines will need to be updated to
> > indicate the strict requirements on default streams.
> > * Trademark approval: N/A (not needed for this Change)
> >
> > == Upgrade/compatibility impact ==
> > This change is on the build-system side of things and should not
> > impact the upgrade process directly.
> >
> > == How To Test ==
> > # Build a modular stream
> > # Make that stream a default stream (or a buildroot override)
> > # Build a non-modular RPM that requires an artifact RPM from the modular
> > stream.
> >
> > == User Experience ==
> > This should not change the end-user experience.
> >
> > == Dependencies ==
> > Nothing known that isn't listed in the scope.
> >
> > == Contingency Plan ==
> > * Contingency mechanism: Disable the buildroot inheritance in Koji to
> > revert to the current behavior.
> > * Blocks release? Ambiguous: lack of complete implementation may
> > indirectly cause blocking issues.
> > * Blocks product? No
> >
> > == Documentation ==
> >
> >
> > == Release Notes ==
> > None needed, the Change is not user-facing.
> >
> > --
> > Ben Cotton
> > He / Him / His
> > Fedora Program Manager
> > Red Hat
> > TZ=America/Indiana/Indianapolis
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
> > Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
>
> > Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> g
>
> That sounds like a really bad idea. Isn't the entire goal for traditional
> RPMs
> to exist separately from modules? This will lead to more packages being
> maintained as modules only, and I can only see it getting worse from
> there.
>
> Are there any actual benefits to this? I can't think of any.
IMHO it's exactly the opposite. E.g. Eclipse is moving to a module because
it requires Maven 3.6 which is available as a module only. If this was
implemented earlier we wouldn't have bothered making Eclipse module. To
continue if this is delayed osgi/swt apps which depend on parts of eclipse
will find it easier to just make their apps modules too and so on.
> --
> John M. Harris, Jr.
> Splentity
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> g
It seems that you just proved my point.. You decided to make Eclipse a module
because somebody decided to make Maven 3.6 a module, instead of just shipping
the latest stable version of Maven as a traditional package.
--
John M. Harris, Jr.
Splentity
3:42 a.m.
On Mon, Oct 14, 2019 at 10:13 AM John M. Harris Jr <johnmh(a)splentity.com>
wrote:
On Sunday, October 13, 2019 11:42:41 PM MST Aleksandar Kurtakov
wrote:
> On Mon, Oct 14, 2019 at 9:00 AM John M. Harris Jr <johnmh(a)splentity.com>
>
> wrote:
> > On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
> > >
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> > >
> > > Enable module default streams in the buildroot repository for modular
> > > and non-modular RPMs.
> > >
> > > == Summary ==
> > > This Change (colloquially referred to as "Ursa Prime") enables
the
> > > Koji build-system to include the RPM artifacts provided by module
> > > default streams in the buildroot when building non-modular (or
> > > "traditional") RPMs.
> > >
> > > == Owner ==
> > > * Name: [[User:Sgallagh| Stephen Gallagher]]
> > > * Email: sgallagh(a)redhat.com
> > > * Responsible WG: Modularity WG
> > >
> > > == Detailed Description ==
> > > As a major part of the Modularity design, we have a concept of
default
> > > module streams. These streams are built as modules, but the RPM
> > > artifacts they deliver are intended to be used just like non-modular
> > > RPMS. The aspirational goal is that a user of the system who never
> > > executes a module-specific command (such as `dnf module install
> > > nodejs:8`) should experience no meaningful changes in behavior from
> > > how they would interact with a completely non-modular system. In
> > > practice, this may mean that the informational output of package
> > > managers may indicate that modules are being enabled and used, but a
> > > user that does not have a specific reason to interact with the
> > > selection of a module stream should have that managed on their
behalf.
> > >
> > > Similarly, the experience for package maintainers of non-modular
> > > packages should be unaffected by an RPM dependency moving from the
> > > non-modular repository into a default module stream. Up to the
> > > present, this has not been the case; no module stream content has
been
> > > available in the non-modular buildroot for other packages to consume.
> > > Koji builds of non-modular RPMs have had only the other non-modular
> > > RPMs from that release available to their buildroots. In contrast,
> > > building on local systems has access to both the non-modular RPMs and
> > > the RPMs from any of the default module streams. With this Change,
> > > Koji builds will have the same behavior and be able to depend on
> > > content provided by default module streams. It also enables the same
> > > behavior for Modular builds: the `platform` stream will now include
> > > the contents of the default module streams for each release and do
not
> > > need to be explicitly specified in the modulemd `buildrequires`.
> > >
> > > Note: This Change does not address the other major Modularity issue
we
> > > are facing around distribution upgrades with differing default
> > > streams. When discussing this Change, please keep that topic
separate.
> > >
> > > == Benefit to Fedora ==
> > >
> > > This will simplify the lives of package maintainers in Fedora in two
> > > primary ways. I'll use a hypothetical example of the Node.js
> > > interpreter and a JSApp package which is capable of running on
Node.js
> > > 10 or 12 (but requires newer features than are provided by Node.js
8).
> > > Additionally, the JSApp package requires the same versions of Node.js
> > > at build-time.
> > >
> > > * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > > streams. The `nodejs:10` stream is set as the default stream.
> > > * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > > streams. The `nodejs:10` stream is set as the default stream.
> > > * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
> > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > > stream will likely become available during the F31 lifetime.
> > > * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
> > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > > stream will likely become available during the F32 lifetime.
> > >
> > > On Fedora 29 through 31, the Node.js package maintainer needs to
build
> > > the `nodejs:10` package both as a module and as a non-modular RPM in
> > > the distribution so that the JSApp package can be built. With this
> > > Change, the Node.js package maintainer in Fedora 32+ will only need
to
> > > build the various Node.js streams and make one of them the default
> > > stream. The packages from it will then be added to the buildroot for
> > > non-modular packages. This will also make the packaging process
> > > somewhat more efficient, as the maintainer needs only to manage the
> > > module stream and the MBS will build it for all configured platforms.
> > >
> > > Similarly, from the perspective of dependent maintainers, there will
> > > no longer be anxiety about needing to move their package to a module
> > > if one or more of their dependencies drops their non-modular version
> > > in favor of a default stream. Their builds will continue to work as
> > > they do today.
> > >
> > > == Scope ==
> > > * Proposal owners:
> > > # Update Packaging Guidelines with
> > > [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> > > for module default streams
> > > # Create a Pungi configuration to generate the buildroot from the
> > > default module streams.
> > > # Include `default_modules_scm_url` in the platform virtual module
> > > specification
> >
> > # Configure Koji tags for inheriting the new
> >
> > > modular-defaults
> > > buildroot into the standard buildroot
> > >
> > > * Other developers:
> > >
> > > Packagers of default module streams will be required to conform to
the
> > > [https://pagure.io/modularity/issue/146#comment-600328 policy]
> > > regarding visibility of stream artifacts. Any default module stream
> > > that is not in compliance by one week before Beta Freeze will cease
to
> > > be a default stream.
> > >
> > > * Release engineering:
> > > # https://pagure.io/releng/issue/8879 - Create pungi config for
> > > Rawhide/F32 ursa prime buildroot
> > > # https://pagure.io/releng/issue/8880 - Include
> > > `default_modules_scm_url` in platform 31 virtual module
> > > # https://pagure.io/releng/issue/8881 - Configure Koji tags for
> > > inheriting f32-modular-buildroot
> > >
> > > * Policies and guidelines:
> > > The Modularity Packaging Guidelines will need to be updated to
> > > indicate the strict requirements on default streams.
> > > * Trademark approval: N/A (not needed for this Change)
> > >
> > > == Upgrade/compatibility impact ==
> > > This change is on the build-system side of things and should not
> > > impact the upgrade process directly.
> > >
> > > == How To Test ==
> > > # Build a modular stream
> > > # Make that stream a default stream (or a buildroot override)
> > > # Build a non-modular RPM that requires an artifact RPM from the
modular
> > > stream.
> > >
> > > == User Experience ==
> > > This should not change the end-user experience.
> > >
> > > == Dependencies ==
> > > Nothing known that isn't listed in the scope.
> > >
> > > == Contingency Plan ==
> > > * Contingency mechanism: Disable the buildroot inheritance in Koji to
> > > revert to the current behavior.
> > > * Blocks release? Ambiguous: lack of complete implementation may
> > > indirectly cause blocking issues.
> > > * Blocks product? No
> > >
> > > == Documentation ==
> > >
> > >
> > > == Release Notes ==
> > > None needed, the Change is not user-facing.
> > >
> > > --
> > > Ben Cotton
> > > He / Him / His
> > > Fedora Program Manager
> > > Red Hat
> > > TZ=America/Indiana/Indianapolis
> > > _______________________________________________
> > > devel mailing list -- devel(a)lists.fedoraproject.org
> > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
> > > Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List
> >
> > > Archives:
> >
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> > g
> >
> > That sounds like a really bad idea. Isn't the entire goal for
traditional
> > RPMs
> > to exist separately from modules? This will lead to more packages being
> > maintained as modules only, and I can only see it getting worse from
> > there.
> >
> > Are there any actual benefits to this? I can't think of any.
>
> IMHO it's exactly the opposite. E.g. Eclipse is moving to a module
because
> it requires Maven 3.6 which is available as a module only. If this was
> implemented earlier we wouldn't have bothered making Eclipse module. To
> continue if this is delayed osgi/swt apps which depend on parts of
eclipse
> will find it easier to just make their apps modules too and so on.
>
> > --
> > John M. Harris, Jr.
> > Splentity
> >
> > _______________________________________________
> > devel mailing list -- devel(a)lists.fedoraproject.org
> > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> >
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> > g
It seems that you just proved my point.. You decided to make Eclipse a
module
because somebody decided to make Maven 3.6 a module, instead of just
shipping
the latest stable version of Maven as a traditional package.
You seem to totally miss the point - there is no one even trying to ship
Maven as a traditional package so what should we do give up on having
anything built with Maven in the distro? Or someone will jump and do the
needed work? I keep hearing these arguments but I'm yet to see someone
actually stepping in to the work.
This is clear example of "the one who does decides".
--
John M. Harris, Jr.
Splentity
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Alexander Kurtakov
Red Hat Eclipse Team
4:05 a.m.
On Mon, Oct 14, 2019 at 10:43 AM Aleksandar Kurtakov
<akurtako(a)redhat.com> wrote:
On Mon, Oct 14, 2019 at 10:13 AM John M. Harris Jr <johnmh(a)splentity.com> wrote:
>
> On Sunday, October 13, 2019 11:42:41 PM MST Aleksandar Kurtakov wrote:
> > On Mon, Oct 14, 2019 at 9:00 AM John M. Harris Jr <johnmh(a)splentity.com>
> >
> > wrote:
> > > On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
> > > >
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> > > >
> > > > Enable module default streams in the buildroot repository for modular
> > > > and non-modular RPMs.
> > > >
> > > > == Summary ==
> > > > This Change (colloquially referred to as "Ursa Prime")
enables the
> > > > Koji build-system to include the RPM artifacts provided by module
> > > > default streams in the buildroot when building non-modular (or
> > > > "traditional") RPMs.
> > > >
> > > > == Owner ==
> > > > * Name: [[User:Sgallagh| Stephen Gallagher]]
> > > > * Email: sgallagh(a)redhat.com
> > > > * Responsible WG: Modularity WG
> > > >
> > > > == Detailed Description ==
> > > > As a major part of the Modularity design, we have a concept of
default
> > > > module streams. These streams are built as modules, but the RPM
> > > > artifacts they deliver are intended to be used just like non-modular
> > > > RPMS. The aspirational goal is that a user of the system who never
> > > > executes a module-specific command (such as `dnf module install
> > > > nodejs:8`) should experience no meaningful changes in behavior from
> > > > how they would interact with a completely non-modular system. In
> > > > practice, this may mean that the informational output of package
> > > > managers may indicate that modules are being enabled and used, but a
> > > > user that does not have a specific reason to interact with the
> > > > selection of a module stream should have that managed on their
behalf.
> > > >
> > > > Similarly, the experience for package maintainers of non-modular
> > > > packages should be unaffected by an RPM dependency moving from the
> > > > non-modular repository into a default module stream. Up to the
> > > > present, this has not been the case; no module stream content has
been
> > > > available in the non-modular buildroot for other packages to consume.
> > > > Koji builds of non-modular RPMs have had only the other non-modular
> > > > RPMs from that release available to their buildroots. In contrast,
> > > > building on local systems has access to both the non-modular RPMs and
> > > > the RPMs from any of the default module streams. With this Change,
> > > > Koji builds will have the same behavior and be able to depend on
> > > > content provided by default module streams. It also enables the same
> > > > behavior for Modular builds: the `platform` stream will now include
> > > > the contents of the default module streams for each release and do
not
> > > > need to be explicitly specified in the modulemd `buildrequires`.
> > > >
> > > > Note: This Change does not address the other major Modularity issue
we
> > > > are facing around distribution upgrades with differing default
> > > > streams. When discussing this Change, please keep that topic
separate.
> > > >
> > > > == Benefit to Fedora ==
> > > >
> > > > This will simplify the lives of package maintainers in Fedora in two
> > > > primary ways. I'll use a hypothetical example of the Node.js
> > > > interpreter and a JSApp package which is capable of running on
Node.js
> > > > 10 or 12 (but requires newer features than are provided by Node.js
8).
> > > > Additionally, the JSApp package requires the same versions of Node.js
> > > > at build-time.
> > > >
> > > > * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > > > streams. The `nodejs:10` stream is set as the default stream.
> > > > * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
> > > > streams. The `nodejs:10` stream is set as the default stream.
> > > > * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
> > > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > > > stream will likely become available during the F31 lifetime.
> > > > * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
> > > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
> > > > stream will likely become available during the F32 lifetime.
> > > >
> > > > On Fedora 29 through 31, the Node.js package maintainer needs to
build
> > > > the `nodejs:10` package both as a module and as a non-modular RPM in
> > > > the distribution so that the JSApp package can be built. With this
> > > > Change, the Node.js package maintainer in Fedora 32+ will only need
to
> > > > build the various Node.js streams and make one of them the default
> > > > stream. The packages from it will then be added to the buildroot for
> > > > non-modular packages. This will also make the packaging process
> > > > somewhat more efficient, as the maintainer needs only to manage the
> > > > module stream and the MBS will build it for all configured platforms.
> > > >
> > > > Similarly, from the perspective of dependent maintainers, there will
> > > > no longer be anxiety about needing to move their package to a module
> > > > if one or more of their dependencies drops their non-modular version
> > > > in favor of a default stream. Their builds will continue to work as
> > > > they do today.
> > > >
> > > > == Scope ==
> > > > * Proposal owners:
> > > > # Update Packaging Guidelines with
> > > > [https://pagure.io/modularity/issue/146#comment-600328 requirements]
> > > > for module default streams
> > > > # Create a Pungi configuration to generate the buildroot from the
> > > > default module streams.
> > > > # Include `default_modules_scm_url` in the platform virtual module
> > > > specification
> > >
> > > # Configure Koji tags for inheriting the new
> > >
> > > > modular-defaults
> > > > buildroot into the standard buildroot
> > > >
> > > > * Other developers:
> > > >
> > > > Packagers of default module streams will be required to conform to
the
> > > > [https://pagure.io/modularity/issue/146#comment-600328 policy]
> > > > regarding visibility of stream artifacts. Any default module stream
> > > > that is not in compliance by one week before Beta Freeze will cease
to
> > > > be a default stream.
> > > >
> > > > * Release engineering:
> > > > # https://pagure.io/releng/issue/8879 - Create pungi config for
> > > > Rawhide/F32 ursa prime buildroot
> > > > # https://pagure.io/releng/issue/8880 - Include
> > > > `default_modules_scm_url` in platform 31 virtual module
> > > > # https://pagure.io/releng/issue/8881 - Configure Koji tags for
> > > > inheriting f32-modular-buildroot
> > > >
> > > > * Policies and guidelines:
> > > > The Modularity Packaging Guidelines will need to be updated to
> > > > indicate the strict requirements on default streams.
> > > > * Trademark approval: N/A (not needed for this Change)
> > > >
> > > > == Upgrade/compatibility impact ==
> > > > This change is on the build-system side of things and should not
> > > > impact the upgrade process directly.
> > > >
> > > > == How To Test ==
> > > > # Build a modular stream
> > > > # Make that stream a default stream (or a buildroot override)
> > > > # Build a non-modular RPM that requires an artifact RPM from the
modular
> > > > stream.
> > > >
> > > > == User Experience ==
> > > > This should not change the end-user experience.
> > > >
> > > > == Dependencies ==
> > > > Nothing known that isn't listed in the scope.
> > > >
> > > > == Contingency Plan ==
> > > > * Contingency mechanism: Disable the buildroot inheritance in Koji to
> > > > revert to the current behavior.
> > > > * Blocks release? Ambiguous: lack of complete implementation may
> > > > indirectly cause blocking issues.
> > > > * Blocks product? No
> > > >
> > > > == Documentation ==
> > > >
> > > >
> > > > == Release Notes ==
> > > > None needed, the Change is not user-facing.
> > > >
> > > > --
> > > > Ben Cotton
> > > > He / Him / His
> > > > Fedora Program Manager
> > > > Red Hat
> > > > TZ=America/Indiana/Indianapolis
> > > > _______________________________________________
> > > > devel mailing list -- devel(a)lists.fedoraproject.org
> > > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
> > > > Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List
> > >
> > > > Archives:
> > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> > > g
> > >
> > > That sounds like a really bad idea. Isn't the entire goal for
traditional
> > > RPMs
> > > to exist separately from modules? This will lead to more packages being
> > > maintained as modules only, and I can only see it getting worse from
> > > there.
> > >
> > > Are there any actual benefits to this? I can't think of any.
> >
> > IMHO it's exactly the opposite. E.g. Eclipse is moving to a module because
> > it requires Maven 3.6 which is available as a module only. If this was
> > implemented earlier we wouldn't have bothered making Eclipse module. To
> > continue if this is delayed osgi/swt apps which depend on parts of eclipse
> > will find it easier to just make their apps modules too and so on.
> >
> > > --
> > > John M. Harris, Jr.
> > > Splentity
> > >
> > > _______________________________________________
> > > devel mailing list -- devel(a)lists.fedoraproject.org
> > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
> > > g
>
> It seems that you just proved my point.. You decided to make Eclipse a module
> because somebody decided to make Maven 3.6 a module, instead of just shipping
> the latest stable version of Maven as a traditional package.
(snip)
You seem to totally miss the point - there is no one even trying to
ship Maven as a traditional package so what should we do give up on having anything built
with Maven in the distro? Or someone will jump and do the needed work? I keep hearing
these arguments but I'm yet to see someone actually stepping in to the work.
This is clear example of "the one who does decides".
You seem to have missed that, in fact, people *did* show up and
started doing the work to keep the non-modular Java stack in a working
state. That was *the whole point* of starting the Stewardship SIG.
And arguably, the non-modular stack might soon be (or already is?) in
better shape than the modular branches. Only a few major updates are
still missing, among them the update to maven 3.6, which we've just
not started working on yet, since it's a big update that affects
almost all other Java packages.
Fabio
>>
>>
>> --
>> John M. Harris, Jr.
>> Splentity
>>
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
>
>
> --
> Alexander Kurtakov
> Red Hat Eclipse Team
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
4:21 a.m.
On Mon, Oct 14, 2019 at 12:06 PM Fabio Valentini <decathorpe(a)gmail.com>
wrote:
On Mon, Oct 14, 2019 at 10:43 AM Aleksandar Kurtakov
<akurtako(a)redhat.com> wrote:
>
>
>
> On Mon, Oct 14, 2019 at 10:13 AM John M. Harris Jr <johnmh(a)splentity.com>
wrote:
>>
>> On Sunday, October 13, 2019 11:42:41 PM MST Aleksandar Kurtakov wrote:
>> > On Mon, Oct 14, 2019 at 9:00 AM John M. Harris Jr <
johnmh(a)splentity.com>
>> >
>> > wrote:
>> > > On Wednesday, October 9, 2019 1:46:52 PM MST Ben Cotton wrote:
>> > > >
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>> > > >
>> > > > Enable module default streams in the buildroot repository for
modular
>> > > > and non-modular RPMs.
>> > > >
>> > > > == Summary ==
>> > > > This Change (colloquially referred to as "Ursa Prime")
enables the
>> > > > Koji build-system to include the RPM artifacts provided by
module
>> > > > default streams in the buildroot when building non-modular (or
>> > > > "traditional") RPMs.
>> > > >
>> > > > == Owner ==
>> > > > * Name: [[User:Sgallagh| Stephen Gallagher]]
>> > > > * Email: sgallagh(a)redhat.com
>> > > > * Responsible WG: Modularity WG
>> > > >
>> > > > == Detailed Description ==
>> > > > As a major part of the Modularity design, we have a concept of
default
>> > > > module streams. These streams are built as modules, but the RPM
>> > > > artifacts they deliver are intended to be used just like
non-modular
>> > > > RPMS. The aspirational goal is that a user of the system who
never
>> > > > executes a module-specific command (such as `dnf module install
>> > > > nodejs:8`) should experience no meaningful changes in behavior
from
>> > > > how they would interact with a completely non-modular system. In
>> > > > practice, this may mean that the informational output of package
>> > > > managers may indicate that modules are being enabled and used,
but a
>> > > > user that does not have a specific reason to interact with the
>> > > > selection of a module stream should have that managed on their
behalf.
>> > > >
>> > > > Similarly, the experience for package maintainers of non-modular
>> > > > packages should be unaffected by an RPM dependency moving from
the
>> > > > non-modular repository into a default module stream. Up to the
>> > > > present, this has not been the case; no module stream content
has
been
>> > > > available in the non-modular buildroot for other packages to
consume.
>> > > > Koji builds of non-modular RPMs have had only the other
non-modular
>> > > > RPMs from that release available to their buildroots. In
contrast,
>> > > > building on local systems has access to both the non-modular
RPMs
and
>> > > > the RPMs from any of the default module streams. With this
Change,
>> > > > Koji builds will have the same behavior and be able to depend on
>> > > > content provided by default module streams. It also enables the
same
>> > > > behavior for Modular builds: the `platform` stream will now
include
>> > > > the contents of the default module streams for each release and
do not
>> > > > need to be explicitly specified in the modulemd `buildrequires`.
>> > > >
>> > > > Note: This Change does not address the other major Modularity
issue we
>> > > > are facing around distribution upgrades with differing default
>> > > > streams. When discussing this Change, please keep that topic
separate.
>> > > >
>> > > > == Benefit to Fedora ==
>> > > >
>> > > > This will simplify the lives of package maintainers in Fedora in
two
>> > > > primary ways. I'll use a hypothetical example of the Node.js
>> > > > interpreter and a JSApp package which is capable of running on
Node.js
>> > > > 10 or 12 (but requires newer features than are provided by
Node.js 8).
>> > > > Additionally, the JSApp package requires the same versions of
Node.js
>> > > > at build-time.
>> > > >
>> > > > * Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
>> > > > streams. The `nodejs:10` stream is set as the default stream.
>> > > > * Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
>> > > > streams. The `nodejs:10` stream is set as the default stream.
>> > > > * Fedora 31 ships `nodejs:10` and `nodejs:12` module streams.
The
>> > > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
>> > > > stream will likely become available during the F31 lifetime.
>> > > > * Fedora 32 ships `nodejs:10` and `nodejs:12` module streams.
The
>> > > > `nodejs:12` stream is set as the default stream. The `nodejs:14`
>> > > > stream will likely become available during the F32 lifetime.
>> > > >
>> > > > On Fedora 29 through 31, the Node.js package maintainer needs to
build
>> > > > the `nodejs:10` package both as a module and as a non-modular
RPM
in
>> > > > the distribution so that the JSApp package can be built. With
this
>> > > > Change, the Node.js package maintainer in Fedora 32+ will only
need to
>> > > > build the various Node.js streams and make one of them the
default
>> > > > stream. The packages from it will then be added to the buildroot
for
>> > > > non-modular packages. This will also make the packaging process
>> > > > somewhat more efficient, as the maintainer needs only to manage
the
>> > > > module stream and the MBS will build it for all configured
platforms.
>> > > >
>> > > > Similarly, from the perspective of dependent maintainers, there
will
>> > > > no longer be anxiety about needing to move their package to a
module
>> > > > if one or more of their dependencies drops their non-modular
version
>> > > > in favor of a default stream. Their builds will continue to work
as
>> > > > they do today.
>> > > >
>> > > > == Scope ==
>> > > > * Proposal owners:
>> > > > # Update Packaging Guidelines with
>> > > > [https://pagure.io/modularity/issue/146#comment-600328
requirements]
>> > > > for module default streams
>> > > > # Create a Pungi configuration to generate the buildroot from
the
>> > > > default module streams.
>> > > > # Include `default_modules_scm_url` in the platform virtual
module
>> > > > specification
>> > >
>> > > # Configure Koji tags for inheriting the new
>> > >
>> > > > modular-defaults
>> > > > buildroot into the standard buildroot
>> > > >
>> > > > * Other developers:
>> > > >
>> > > > Packagers of default module streams will be required to conform
to the
>> > > > [https://pagure.io/modularity/issue/146#comment-600328 policy]
>> > > > regarding visibility of stream artifacts. Any default module
stream
>> > > > that is not in compliance by one week before Beta Freeze will
cease to
>> > > > be a default stream.
>> > > >
>> > > > * Release engineering:
>> > > > # https://pagure.io/releng/issue/8879 - Create pungi config for
>> > > > Rawhide/F32 ursa prime buildroot
>> > > > # https://pagure.io/releng/issue/8880 - Include
>> > > > `default_modules_scm_url` in platform 31 virtual module
>> > > > # https://pagure.io/releng/issue/8881 - Configure Koji tags for
>> > > > inheriting f32-modular-buildroot
>> > > >
>> > > > * Policies and guidelines:
>> > > > The Modularity Packaging Guidelines will need to be updated to
>> > > > indicate the strict requirements on default streams.
>> > > > * Trademark approval: N/A (not needed for this Change)
>> > > >
>> > > > == Upgrade/compatibility impact ==
>> > > > This change is on the build-system side of things and should not
>> > > > impact the upgrade process directly.
>> > > >
>> > > > == How To Test ==
>> > > > # Build a modular stream
>> > > > # Make that stream a default stream (or a buildroot override)
>> > > > # Build a non-modular RPM that requires an artifact RPM from the
modular
>> > > > stream.
>> > > >
>> > > > == User Experience ==
>> > > > This should not change the end-user experience.
>> > > >
>> > > > == Dependencies ==
>> > > > Nothing known that isn't listed in the scope.
>> > > >
>> > > > == Contingency Plan ==
>> > > > * Contingency mechanism: Disable the buildroot inheritance in
Koji to
>> > > > revert to the current behavior.
>> > > > * Blocks release? Ambiguous: lack of complete implementation may
>> > > > indirectly cause blocking issues.
>> > > > * Blocks product? No
>> > > >
>> > > > == Documentation ==
>> > > >
>> > > >
>> > > > == Release Notes ==
>> > > > None needed, the Change is not user-facing.
>> > > >
>> > > > --
>> > > > Ben Cotton
>> > > > He / Him / His
>> > > > Fedora Program Manager
>> > > > Red Hat
>> > > > TZ=America/Indiana/Indianapolis
>> > > > _______________________________________________
>> > > > devel mailing list -- devel(a)lists.fedoraproject.org
>> > > > To unsubscribe send an email to
devel-leave(a)lists.fedoraproject.org
>> > > > Fedora Code of Conduct:
>> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List
>> > > > Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines List
>> > >
>> > > > Archives:
>> > >
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
>> > > g
>> > >
>> > > That sounds like a really bad idea. Isn't the entire goal for
traditional
>> > > RPMs
>> > > to exist separately from modules? This will lead to more packages
being
>> > > maintained as modules only, and I can only see it getting worse from
>> > > there.
>> > >
>> > > Are there any actual benefits to this? I can't think of any.
>> >
>> > IMHO it's exactly the opposite. E.g. Eclipse is moving to a module
because
>> > it requires Maven 3.6 which is available as a module only. If this was
>> > implemented earlier we wouldn't have bothered making Eclipse module.
To
>> > continue if this is delayed osgi/swt apps which depend on parts of
eclipse
>> > will find it easier to just make their apps modules too and so on.
>> >
>> > > --
>> > > John M. Harris, Jr.
>> > > Splentity
>> > >
>> > > _______________________________________________
>> > > devel mailing list -- devel(a)lists.fedoraproject.org
>> > > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> > > Fedora Code of Conduct:
>> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > > List Archives:
>> > >
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.or
>> > > g
>>
>> It seems that you just proved my point.. You decided to make Eclipse a
module
>> because somebody decided to make Maven 3.6 a module, instead of just
shipping
>> the latest stable version of Maven as a traditional package.
(snip)
> You seem to totally miss the point - there is no one even trying to ship
Maven as a traditional package so what should we do give up on having
anything built with Maven in the distro? Or someone will jump and do the
needed work? I keep hearing these arguments but I'm yet to see someone
actually stepping in to the work.
> This is clear example of "the one who does decides".
You seem to have missed that, in fact, people *did* show up and
started doing the work to keep the non-modular Java stack in a working
state. That was *the whole point* of starting the Stewardship SIG.
And arguably, the non-modular stack might soon be (or already is?)
in
better shape than the modular branches. Only a few major updates are
still missing, among them the update to maven 3.6, which we've just
not started working on yet, since it's a big update that affects
almost all other Java packages.
I haven't missed that part and I wish good luck to everyone working on
Fedora to improve it any way. Maven update is the true measure for me
whether the non-modular Java stack is going to live. Participated in it
myself few times makes everything makes me remember the amount of
coordination and updates through the whole stack to drive it through. With
these memories maintaining servers like jetty/tomcat or ant look quite
simpler task.
Fabio
>>
>>
>> --
>> John M. Harris, Jr.
>> Splentity
>>
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
>
>
> --
> Alexander Kurtakov
> Red Hat Eclipse Team
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Alexander Kurtakov
Red Hat Eclipse Team
5:02 a.m.
Aleksandar Kurtakov wrote:
You seem to totally miss the point - there is no one even trying to
ship
Maven as a traditional package so what should we do give up on having
anything built with Maven in the distro?
If module-only packages finally get banned (as they should have been from
the onset), the Maven maintainers will HAVE TO maintain the non-modular
version if they want to maintain Maven at all.
It is just not acceptable to move packages as central as Maven to module-
only and leave everything depending on them in the cold. This has been
pointed out (by me and others) to the Maven maintainers ever since they
first announced their plan, and was blatantly ignored. So we really need a
policy to enforce social common sense (i.e., module-only = no go).
Kevin Kofler
5:44 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Mon, Oct 14, 2019 at 12:02:46PM +0200, Kevin Kofler wrote:
Aleksandar Kurtakov wrote:
> You seem to totally miss the point - there is no one even trying to ship
> Maven as a traditional package so what should we do give up on having
> anything built with Maven in the distro?
If module-only packages finally get banned (as they should have been from
the onset), the Maven maintainers will HAVE TO maintain the non-modular
version if they want to maintain Maven at all.
You're assuming they will want to.
I agree with Kevin Fenzi's email that the here the input from the people who
have made the choice to go full-modular would be nice.
If we follow your hard rule we may end up choosing between:
- java/maven being available as module
- no java/maven at all in Fedora
(this modulo the heroic efforts of the stewardship SIG - itself modulo the
fact that the SIG defines itself has a temporary solution [1])
You say that modularity is making people leave but this applies to any changes,
including the one you are proposing here.
Pierre
[1] https://fedoraproject.org/wiki/SIGs/Stewardship
5:58 a.m.
On Mon, Oct 14, 2019, 12:47 Pierre-Yves Chibon <pingou(a)pingoured.fr> wrote:
On Mon, Oct 14, 2019 at 12:02:46PM +0200, Kevin Kofler wrote:
> Aleksandar Kurtakov wrote:
> > You seem to totally miss the point - there is no one even trying to
ship
> > Maven as a traditional package so what should we do give up on having
> > anything built with Maven in the distro?
>
> If module-only packages finally get banned (as they should have been
from
> the onset), the Maven maintainers will HAVE TO maintain the non-modular
> version if they want to maintain Maven at all.
You're assuming they will want to.
I agree with Kevin Fenzi's email that the here the input from the people
who
have made the choice to go full-modular would be nice.
If we follow your hard rule we may end up choosing between:
- java/maven being available as module
- no java/maven at all in Fedora
(this modulo the heroic efforts of the stewardship SIG - itself modulo
the
fact that the SIG defines itself has a temporary solution [1])
FWIW, since I don't see the Java SIG rising from the ashes that were left
by it burning to the ground anytime soon, we definitely *will continue* to
maintain critical packages for the foreseeable future. So maven and all
its dependencies are safe. I also have early plans to update it to the 3.6
branch for f32.
I will later update the wiki page to reflect the current reality. The
documents in our pagure project should be up to date though.
Fabio
You say that modularity is making people leave but this applies to
any
changes,
including the one you are proposing here.
Pierre
[1] https://fedoraproject.org/wiki/SIGs/Stewardship
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
5:16 a.m.
Enable module default streams in the buildroot repository for
modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
I have already stated this multiple times in other threads, but just so that
nobody claims that I have not given feedback to this change (which has been
claimed for some other changes in a similar situation), I am going to state
this again:
I oppose this change because I believe Miro's proposal to require all
packages to have a non-modular version (i.e., to ship the default stream as
non-modular packages) is the much better approach to the issue and makes
this change proposal entirely moot.
To be clear, I propose the following:
* All packages MUST have a default version in any given Fedora release.
* The default version MUST be shipped as non-modular (not as a modular
default stream).
* It follows that packages cannot be module-only.
* This applies to ALL packages including build tools. In particular,
packages must not be hidden from users through constructs such as
buildroot-only modules or non-API packages in modules. If a package is in
a module, no matter in what function, there MUST be a default version in
the non-modular repository.
* The fedora-modular repository SHOULD be disabled by default on user
machines, matching the current build system behavior. This also helps
enforcing the above policies.
I believe that the above proposal is essentially identical to Miro's
proposal from the other thread, which I support. The exact wording can be
discussed, but it is important that the essence of the proposal (module-only
= no go) is retained.
Kevin Kofler
6:09 a.m.
On Monday, October 14, 2019 3:16:27 AM MST Kevin Kofler wrote:
> Enable module default streams in the buildroot repository for
modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
I have already stated this multiple times in other threads, but just so that
nobody claims that I have not given feedback to this change (which has
been claimed for some other changes in a similar situation), I am going to
state this again:
I oppose this change because I believe Miro's proposal to require all
packages to have a non-modular version (i.e., to ship the default stream as
non-modular packages) is the much better approach to the issue and makes
this change proposal entirely moot.
To be clear, I propose the following:
* All packages MUST have a default version in any given Fedora release.
* The default version MUST be shipped as non-modular (not as a modular
default stream).
* It follows that packages cannot be module-only.
* This applies to ALL packages including build tools. In particular,
packages must not be hidden from users through constructs such as
buildroot-only modules or non-API packages in modules. If a package is in
a module, no matter in what function, there MUST be a default version in
the non-modular repository.
* The fedora-modular repository SHOULD be disabled by default on user
machines, matching the current build system behavior. This also helps
enforcing the above policies.
I believe that the above proposal is essentially identical to Miro's
proposal from the other thread, which I support. The exact wording can be
discussed, but it is important that the essence of the proposal (module-only
= no go) is retained.
Kevin Kofler
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
That seems to be the best approach to this situation.
--
John M. Harris, Jr.
Splentity
6:23 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14.10.2019 12:16, Kevin Kofler wrote:
To be clear, I propose the following:
* All packages MUST have a default version in any given Fedora release.
* The default version MUST be shipped as non-modular (not as a modular
default stream).
* It follows that packages cannot be module-only.
* This applies to ALL packages including build tools. In particular,
packages must not be hidden from users through constructs such as
buildroot-only modules or non-API packages in modules. If a package is in
a module, no matter in what function, there MUST be a default version in
the non-modular repository.
* The fedora-modular repository SHOULD be disabled by default on user
machines, matching the current build system behavior. This also helps
enforcing the above policies.
I totally agree with this proposals.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
10:52 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 10. 19 12:16, Kevin Kofler wrote:
> Enable module default streams in the buildroot repository for
modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
>
> == Owner ==
> * Name: [[User:Sgallagh| Stephen Gallagher]]
> * Email: sgallagh(a)redhat.com
> * Responsible WG: Modularity WG
I have already stated this multiple times in other threads, but just so that
nobody claims that I have not given feedback to this change (which has been
claimed for some other changes in a similar situation), I am going to state
this again:
I oppose this change because I believe Miro's proposal to require all
packages to have a non-modular version (i.e., to ship the default stream as
non-modular packages) is the much better approach to the issue and makes
this change proposal entirely moot.
...
* The fedora-modular repository SHOULD be disabled by default on user
machines, matching the current build system behavior. This also helps
enforcing the above policies.
As long as we enforce the policy of no default streams, this is actually
unnecessary. Modules will only be used when people do "dnf module ...".
Regardless whether this is a good or a wrong idea, just makign clear that I have
never proposed to disable modular repos by default.
OTOH I propose that disabling modular repos must be considered a legitimate
action and the distro must remain it's core functionality.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
8:44 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Wed, Oct 09, 2019 at 04:46:52PM -0400, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
Thanks for writing this up Stephen, I fully support this.
For some people here it is clear they don't want to develop modules and
that will always be fine. Others see a benefit (whether small or large)
from developing as modules, and that should also be fine and I want
Fedora to allow that. Allowing modules in buildroots prevents the
conflict between packagers who make different choices (e.g. non-modular
Eclipse can't use module-only Maven) so seems like a good compromise.
I find myself a bit reluctant to write this mail because the language
others are using in this thread is fairly ugly for a technical
discussion in an open source project - about "forcing" people to develop
packages in a certain way, "teaching them a lesson" etc. Please calm it
down & have some respect for technical decisions of other developers.
Regards, Joe
10:56 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 10. 19 15:44, Joe Orton wrote:
I find myself a bit reluctant to write this mail because the
language
others are using in this thread is fairly ugly for a technical
discussion in an open source project - about "forcing" people to develop
packages in a certain way, "teaching them a lesson" etc. Please calm it
down & have some respect for technical decisions of other developers.
I completely agree with this message.
I don't agree with where modularity is going in Fedora, but it doesn't mean I
don't like the people involved ;)
On the technical side, iff we decide to ban default modular stream, I don't
think we should force people or teach them a lesson, I think we need to
coordinate a huge effort to make everything work again - together.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4:44 p.m.
Joe Orton wrote:
I find myself a bit reluctant to write this mail because the
language
others are using in this thread is fairly ugly for a technical
discussion in an open source project - about "forcing" people to develop
packages in a certain way, "teaching them a lesson" etc. Please calm it
down & have some respect for technical decisions of other developers.
I am sorry if my wording irritated you. However, I would like to point out
that…
For some people here it is clear they don't want to develop
modules and
that will always be fine. Others see a benefit (whether small or large)
from developing as modules, and that should also be fine and I want
Fedora to allow that. Allowing modules in buildroots prevents the
conflict between packagers who make different choices (e.g. non-modular
Eclipse can't use module-only Maven) so seems like a good compromise.
… this is not a complete solution, because it is taking the end users
entirely out of the equation. Allowing modules in buildroots will have no
impact on the end user.
The net result of this proposed Change for the end user is still the same as
the status quo: They have to use modules whether they want to or not, the
choice is taken away from them. And while the default stream approach tries
to hide Modularity from the users (and with this proposed Change, also from
the packagers of dependent packages), the abstraction is leaky, as
evidenced, e.g., by the libgit2 upgrade blocker.
I think it is not fair to force modules onto all users of the distribution
just because it is the technical preference of a few individual developers.
Because, in the end, it is the developers who choose to make their packages
module-only who are forcing their way onto all users (and other developers,
too). My proposal would only "force" the developers to give the choice back
to the users. So I do not see myself as being the one forcing their way onto
others. I am sorry if my suboptimal wording gave you that unfortunate
impression.
If the user wishes to use a non-default version of a package, sure,
Modularity can help them. But otherwise, modules should not be a requirement
to use the distribution. There is no technical necessity for that.
Requiring packages to have a non-modular version in the non-modular
repository does in no way preclude providing alternate versions in modules,
so I do not see how that proposal would impede developing modules if the
maintainer wishes to do that.
Kevin Kofler
5:10 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Mon, Oct 14, 2019 at 11:44:46PM +0200, Kevin Kofler wrote:
The net result of this proposed Change for the end user is still the
same as
the status quo: They have to use modules whether they want to or not, the
choice is taken away from them. And while the default stream approach tries
to hide Modularity from the users (and with this proposed Change, also from
the packagers of dependent packages), the abstraction is leaky, as
evidenced, e.g., by the libgit2 upgrade blocker.
Yup, it is certainly not perfect and we'll have to work through those
kinds of issues carefully as Stephen & others are doing. I am very
confident in trusting Fedora developers to make sensible choices for our
users, though - including whether we should make packages module-only.
As package maintainers we all make technical decisions which have
significant impact on our users every day - whether that's in the choice
of defaults, choice of build flags, or whatever. Honestly delivering as
modules-vs-non-modules is a completely trivial issue compared to most of
the stuff I spend time on. If "yum install X" still works most people
just don't care about the RPM/dnf/repo mechanics behind that.
If you don't want to ship any of your packages as modules I think that's
great and you should continue doing that. On the other hand, I want to
move a bunch of my packages to module-only because I think I can do a
better job serving Fedora users that way.
Regards, Joe
5:36 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 15. 10. 19 12:10, Joe Orton wrote:
On Mon, Oct 14, 2019 at 11:44:46PM +0200, Kevin Kofler wrote:
> The net result of this proposed Change for the end user is still the same as
> the status quo: They have to use modules whether they want to or not, the
> choice is taken away from them. And while the default stream approach tries
> to hide Modularity from the users (and with this proposed Change, also from
> the packagers of dependent packages), the abstraction is leaky, as
> evidenced, e.g., by the libgit2 upgrade blocker.
Yup, it is certainly not perfect and we'll have to work through those
kinds of issues carefully as Stephen & others are doing. I am very
confident in trusting Fedora developers to make sensible choices for our
users, though - including whether we should make packages module-only.
As package maintainers we all make technical decisions which have
significant impact on our users every day - whether that's in the choice
of defaults, choice of build flags, or whatever. Honestly delivering as
modules-vs-non-modules is a completely trivial issue compared to most of
the stuff I spend time on. If "yum install X" still works most people
just don't care about the RPM/dnf/repo mechanics behind that.
Except it works only half way. The installation works. Later, dependencies are
broken. Upgrades are broken. "yum remove X" does not undo the action
completely.
The main issue is: user just enabled a module without doing it explicitly. The
user needs to know how to handle modules in order to recover.
We are also Fedora developers and we are trying to do the sensible choice here.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
10:19 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
>As package maintainers we all make technical decisions which
have
>significant impact on our users every day - whether that's in the choice
>of defaults, choice of build flags, or whatever. Honestly delivering as
>modules-vs-non-modules is a completely trivial issue compared to most of
>the stuff I spend time on. If "yum install X" still works most people
>just don't care about the RPM/dnf/repo mechanics behind that.
Except it works only half way. The installation works. Later,
dependencies are broken. Upgrades are broken. "yum remove X" does
not undo the action completely.
The main issue is: user just enabled a module without doing it
explicitly. The user needs to know how to handle modules in order to
recover.
I never expect "yum remove X" to be the inverse of "yum install X".
DNF's
magical leaf tracking makes it a bit more so, but not exactly. So, I don't
think we should make that a very high priority concern (although if we can
improve it, so much the better).
Upgrades need to work, though. And they need to work regardless of whether
we ban default modules or not. So, given that, I'm not _really_ seeing big
differences in practice for the user beteen these two proposals, and the one
(no default streams) negates one of the whole intended advantages of the
entire thing.
What am I missing?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
10:35 a.m.
On Tue, Oct 15, 2019 at 11:22 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> >As package maintainers we all make technical decisions which have
> >significant impact on our users every day - whether that's in the choice
> >of defaults, choice of build flags, or whatever. Honestly delivering as
> >modules-vs-non-modules is a completely trivial issue compared to most of
> >the stuff I spend time on. If "yum install X" still works most
people
> >just don't care about the RPM/dnf/repo mechanics behind that.
>
> Except it works only half way. The installation works. Later,
> dependencies are broken. Upgrades are broken. "yum remove X" does
> not undo the action completely.
>
> The main issue is: user just enabled a module without doing it
> explicitly. The user needs to know how to handle modules in order to
> recover.
I never expect "yum remove X" to be the inverse of "yum install X".
DNF's
magical leaf tracking makes it a bit more so, but not exactly. So, I don't
think we should make that a very high priority concern (although if we can
improve it, so much the better).
Upgrades need to work, though. And they need to work regardless of whether
we ban default modules or not. So, given that, I'm not _really_ seeing big
differences in practice for the user beteen these two proposals, and the one
(no default streams) negates one of the whole intended advantages of the
entire thing.
What am I missing?
Upgrades can never work without changing fundamental properties of modules.
There are two traits of modules that break everything:
* Module activation is introduces a hard lock of content source
* There is no concept of module transitions
In the RHEL world, these two qualities are not great, but they are
serviceable (RHEL 9 is going to hurt!). In the Fedora world, they make
having default modules essentially untenable. It could even be argued
that non-default modules are not serviceable either.
There is no way in modularity to say that a module is being replaced
with another. There's also no way to say that a module is going away
and transition accordingly. Moreover, since modules have a strong
dependency on an abstract property that is defined as a consequence of
the system content (the "platform" module), it is not possible to have
orphan modules that are still binary compatible as you upgrade.
Finally, modularization and foregoing non-modular versions of content
has an implicit commitment that you *never* demodularize because there
is no module transition capabilities in the modularity technology.
These flaws boil down to Fedora Modularity being a failure because the
technology does not account for the needs of Fedora as a distribution,
as a project, or as a community.
--
真実はいつも一つ！/ Always, there's only one truth!
11:11 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Tue, 2019-10-15 at 11:35 -0400, Neal Gompa wrote:
On Tue, Oct 15, 2019 at 11:22 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
> On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> > > As package maintainers we all make technical decisions which have
> > > significant impact on our users every day - whether that's in the
choice
> > > of defaults, choice of build flags, or whatever. Honestly delivering as
> > > modules-vs-non-modules is a completely trivial issue compared to most of
> > > the stuff I spend time on. If "yum install X" still works most
people
> > > just don't care about the RPM/dnf/repo mechanics behind that.
> >
> > Except it works only half way. The installation works. Later,
> > dependencies are broken. Upgrades are broken. "yum remove X" does
> > not undo the action completely.
> >
> > The main issue is: user just enabled a module without doing it
> > explicitly. The user needs to know how to handle modules in order to
> > recover.
>
> I never expect "yum remove X" to be the inverse of "yum install
X". DNF's
> magical leaf tracking makes it a bit more so, but not exactly. So, I don't
> think we should make that a very high priority concern (although if we can
> improve it, so much the better).
>
> Upgrades need to work, though. And they need to work regardless of whether
> we ban default modules or not. So, given that, I'm not _really_ seeing big
> differences in practice for the user beteen these two proposals, and the one
> (no default streams) negates one of the whole intended advantages of the
> entire thing.
>
> What am I missing?
>
Upgrades can never work without changing fundamental properties of modules.
There are two traits of modules that break everything:
* Module activation is introduces a hard lock of content source
* There is no concept of module transitions
In the RHEL world, these two qualities are not great, but they are
serviceable (RHEL 9 is going to hurt!). In the Fedora world, they make
having default modules essentially untenable. It could even be argued
that non-default modules are not serviceable either.
There is no way in modularity to say that a module is being replaced
with another. There's also no way to say that a module is going away
and transition accordingly. Moreover, since modules have a strong
dependency on an abstract property that is defined as a consequence of
the system content (the "platform" module), it is not possible to have
orphan modules that are still binary compatible as you upgrade.
Finally, modularization and foregoing non-modular versions of content
has an implicit commitment that you *never* demodularize because there
is no module transition capabilities in the modularity technology.
These flaws boil down to Fedora Modularity being a failure because the
technology does not account for the needs of Fedora as a distribution,
as a project, or as a community.
I think this is actually a very cogent summary of the major problems
we're dealing with in modularity right now, but it's *also* needlessly
negative.
Inventing new stuff is hard. You rarely get it right the first time.
Speaking personally here I think it was a big mistake to do this second
version of Modularity as fast as we did and drop it into RHEL 8 before
it had been in Fedora for two minutes, that was an unforced error; but
even if we hadn't done that, I'm not surprised the second cut at
Modularity isn't perfect. The second cut at RPM wasn't perfect either,
that's why we're on 4.x not 2.x. :)
What's happening right now is the process of us trying something out
and finding out where the problems are. That's what happens when you
invent new stuff, it's harder than just carrying on doing the old
stuff.
So: I'm on board with most of what you say here, but there's no need to
say it means Modularity is "a failure". It means right now it's not
entirely baked and we're realizing the concept needs extending and
perhaps reworking a bit, just like we realized with the *first* cut at
Modularity which we abandoned between a Beta release and a Final
release. This is causing us to have to deal with some icky problems,
but again, that's not *new*. We had to deal with some fairly icky
problems when systemd showed up. We had to deal with some fairly icky
problems when grub2 showed up. It's a process we've dealt with before.
We know how to do it. We just need to hold our noses and fix the icky
problems, and then sit down and think about the design issues that have
become apparent in Modularity v2 through our actually implementing it
and using it (which is what Fedora is for, remember!) and figure out
how to address them in Modularity v3.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:25 a.m.
On Tue, Oct 15, 2019 at 12:13 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Tue, 2019-10-15 at 11:35 -0400, Neal Gompa wrote:
> On Tue, Oct 15, 2019 at 11:22 AM Matthew Miller
> <mattdm(a)fedoraproject.org> wrote:
> > On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> > > > As package maintainers we all make technical decisions which have
> > > > significant impact on our users every day - whether that's in the
choice
> > > > of defaults, choice of build flags, or whatever. Honestly delivering
as
> > > > modules-vs-non-modules is a completely trivial issue compared to most
of
> > > > the stuff I spend time on. If "yum install X" still works
most people
> > > > just don't care about the RPM/dnf/repo mechanics behind that.
> > >
> > > Except it works only half way. The installation works. Later,
> > > dependencies are broken. Upgrades are broken. "yum remove X"
does
> > > not undo the action completely.
> > >
> > > The main issue is: user just enabled a module without doing it
> > > explicitly. The user needs to know how to handle modules in order to
> > > recover.
> >
> > I never expect "yum remove X" to be the inverse of "yum install
X". DNF's
> > magical leaf tracking makes it a bit more so, but not exactly. So, I don't
> > think we should make that a very high priority concern (although if we can
> > improve it, so much the better).
> >
> > Upgrades need to work, though. And they need to work regardless of whether
> > we ban default modules or not. So, given that, I'm not _really_ seeing big
> > differences in practice for the user beteen these two proposals, and the one
> > (no default streams) negates one of the whole intended advantages of the
> > entire thing.
> >
> > What am I missing?
> >
>
> Upgrades can never work without changing fundamental properties of modules.
>
> There are two traits of modules that break everything:
> * Module activation is introduces a hard lock of content source
> * There is no concept of module transitions
>
> In the RHEL world, these two qualities are not great, but they are
> serviceable (RHEL 9 is going to hurt!). In the Fedora world, they make
> having default modules essentially untenable. It could even be argued
> that non-default modules are not serviceable either.
>
> There is no way in modularity to say that a module is being replaced
> with another. There's also no way to say that a module is going away
> and transition accordingly. Moreover, since modules have a strong
> dependency on an abstract property that is defined as a consequence of
> the system content (the "platform" module), it is not possible to have
> orphan modules that are still binary compatible as you upgrade.
> Finally, modularization and foregoing non-modular versions of content
> has an implicit commitment that you *never* demodularize because there
> is no module transition capabilities in the modularity technology.
>
> These flaws boil down to Fedora Modularity being a failure because the
> technology does not account for the needs of Fedora as a distribution,
> as a project, or as a community.
I think this is actually a very cogent summary of the major problems
we're dealing with in modularity right now, but it's *also* needlessly
negative.
Inventing new stuff is hard. You rarely get it right the first time.
Speaking personally here I think it was a big mistake to do this second
version of Modularity as fast as we did and drop it into RHEL 8 before
it had been in Fedora for two minutes, that was an unforced error; but
even if we hadn't done that, I'm not surprised the second cut at
Modularity isn't perfect. The second cut at RPM wasn't perfect either,
that's why we're on 4.x not 2.x. :)
What's happening right now is the process of us trying something out
and finding out where the problems are. That's what happens when you
invent new stuff, it's harder than just carrying on doing the old
stuff.
So: I'm on board with most of what you say here, but there's no need to
say it means Modularity is "a failure". It means right now it's not
entirely baked and we're realizing the concept needs extending and
perhaps reworking a bit, just like we realized with the *first* cut at
Modularity which we abandoned between a Beta release and a Final
release. This is causing us to have to deal with some icky problems,
but again, that's not *new*. We had to deal with some fairly icky
problems when systemd showed up. We had to deal with some fairly icky
problems when grub2 showed up. It's a process we've dealt with before.
We know how to do it. We just need to hold our noses and fix the icky
problems, and then sit down and think about the design issues that have
become apparent in Modularity v2 through our actually implementing it
and using it (which is what Fedora is for, remember!) and figure out
how to address them in Modularity v3.
Are we going to do a Modularity v3? Because people seemed to be
*really* reluctant to go down that path because it would break
compatibility with RHEL. Honestly, my big problem is how much we're
being held hostage by RHEL for this. I would have been happier if we'd
focused on Fedora cases and naturally extended it to RHEL cases
afterward.
There's a very big difference between those examples and this one. I
don't recall us being forced into a window to *ensure* it was in the
sources from deriving into RHEL before.
You are right that Fedora Modularity as a project is not necessarily a
failure. But I stand by my consideration that the current
implementation is a failure. From the build side to the user and
developer UX, it's extremely painful and so far hasn't presented a lot
of upside from a Fedora context.
--
真実はいつも一つ！/ Always, there's only one truth!
11:51 a.m.
On Tue, Oct 15, 2019 at 12:27 PM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Tue, Oct 15, 2019 at 12:13 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
>
> On Tue, 2019-10-15 at 11:35 -0400, Neal Gompa wrote:
> > On Tue, Oct 15, 2019 at 11:22 AM Matthew Miller
> > <mattdm(a)fedoraproject.org> wrote:
> > > On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> > > > > As package maintainers we all make technical decisions which
have
> > > > > significant impact on our users every day - whether that's
in the choice
> > > > > of defaults, choice of build flags, or whatever. Honestly
delivering as
> > > > > modules-vs-non-modules is a completely trivial issue compared to
most of
> > > > > the stuff I spend time on. If "yum install X" still
works most people
> > > > > just don't care about the RPM/dnf/repo mechanics behind
that.
> > > >
> > > > Except it works only half way. The installation works. Later,
> > > > dependencies are broken. Upgrades are broken. "yum remove
X" does
> > > > not undo the action completely.
> > > >
> > > > The main issue is: user just enabled a module without doing it
> > > > explicitly. The user needs to know how to handle modules in order to
> > > > recover.
> > >
> > > I never expect "yum remove X" to be the inverse of "yum
install X". DNF's
> > > magical leaf tracking makes it a bit more so, but not exactly. So, I
don't
> > > think we should make that a very high priority concern (although if we
can
> > > improve it, so much the better).
> > >
> > > Upgrades need to work, though. And they need to work regardless of
whether
> > > we ban default modules or not. So, given that, I'm not _really_ seeing
big
> > > differences in practice for the user beteen these two proposals, and the
one
> > > (no default streams) negates one of the whole intended advantages of the
> > > entire thing.
> > >
> > > What am I missing?
> > >
> >
> > Upgrades can never work without changing fundamental properties of modules.
> >
> > There are two traits of modules that break everything:
> > * Module activation is introduces a hard lock of content source
> > * There is no concept of module transitions
> >
> > In the RHEL world, these two qualities are not great, but they are
> > serviceable (RHEL 9 is going to hurt!). In the Fedora world, they make
> > having default modules essentially untenable. It could even be argued
> > that non-default modules are not serviceable either.
> >
> > There is no way in modularity to say that a module is being replaced
> > with another. There's also no way to say that a module is going away
> > and transition accordingly. Moreover, since modules have a strong
> > dependency on an abstract property that is defined as a consequence of
> > the system content (the "platform" module), it is not possible to
have
> > orphan modules that are still binary compatible as you upgrade.
> > Finally, modularization and foregoing non-modular versions of content
> > has an implicit commitment that you *never* demodularize because there
> > is no module transition capabilities in the modularity technology.
> >
> > These flaws boil down to Fedora Modularity being a failure because the
> > technology does not account for the needs of Fedora as a distribution,
> > as a project, or as a community.
>
> I think this is actually a very cogent summary of the major problems
> we're dealing with in modularity right now, but it's *also* needlessly
> negative.
>
> Inventing new stuff is hard. You rarely get it right the first time.
> Speaking personally here I think it was a big mistake to do this second
> version of Modularity as fast as we did and drop it into RHEL 8 before
> it had been in Fedora for two minutes, that was an unforced error; but
> even if we hadn't done that, I'm not surprised the second cut at
> Modularity isn't perfect. The second cut at RPM wasn't perfect either,
> that's why we're on 4.x not 2.x. :)
>
> What's happening right now is the process of us trying something out
> and finding out where the problems are. That's what happens when you
> invent new stuff, it's harder than just carrying on doing the old
> stuff.
>
> So: I'm on board with most of what you say here, but there's no need to
> say it means Modularity is "a failure". It means right now it's not
> entirely baked and we're realizing the concept needs extending and
> perhaps reworking a bit, just like we realized with the *first* cut at
> Modularity which we abandoned between a Beta release and a Final
> release. This is causing us to have to deal with some icky problems,
> but again, that's not *new*. We had to deal with some fairly icky
> problems when systemd showed up. We had to deal with some fairly icky
> problems when grub2 showed up. It's a process we've dealt with before.
> We know how to do it. We just need to hold our noses and fix the icky
> problems, and then sit down and think about the design issues that have
> become apparent in Modularity v2 through our actually implementing it
> and using it (which is what Fedora is for, remember!) and figure out
> how to address them in Modularity v3.
Are we going to do a Modularity v3? Because people seemed to be
*really* reluctant to go down that path because it would break
compatibility with RHEL. Honestly, my big problem is how much we're
being held hostage by RHEL for this. I would have been happier if we'd
focused on Fedora cases and naturally extended it to RHEL cases
afterward.
RHEL isn't holding Fedora hostage. That's also unnecessarily antagonistic.
I think treating Fedora and RHEL as entirely separate things, and not
part of the same ecosystem is exactly what got us into this in the
first place. Red Hat is trying to address this because we very much
see how the disconnects happen, and how they are problematic for
everyone involved. There is pain felt on the enterprise side as well.
If we take a step back and consider all of the various pieces of our
ecosystem, it makes sense to try and figure out ways to make them as
cohesive as possible. Not the same, but not wildly and
problematically different either.
Fedora, RHEL, and now CentOS Stream are all pieces of that. What you
call "holding hostage" I would call an attempt to proactively
contribute in the various communities we are all a part of. There has
been reluctance to do that in the past and it has cost both sides. It
will be messy and there will be growing pains. I would ask that we
have patience and assume positive intent. It is fine to raise
technical concerns, but they don't need to be portrayed in a manner
that makes it seem like the communities are in direct conflict.
josh
> There's a very big difference between those examples and this one. I
> don't recall us being forced into a window to *ensure* it was in the
> sources from deriving into RHEL before.
>
> You are right that Fedora Modularity as a project is not necessarily a
> failure. But I stand by my consideration that the current
> implementation is a failure. From the build side to the user and
> developer UX, it's extremely painful and so far hasn't presented a lot
> of upside from a Fedora context.
>
>
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
12:10 p.m.
On Tue, Oct 15, 2019 at 12:52 PM Josh Boyer <jwboyer(a)fedoraproject.org> wrote:
On Tue, Oct 15, 2019 at 12:27 PM Neal Gompa <ngompa13(a)gmail.com> wrote:
>
> On Tue, Oct 15, 2019 at 12:13 PM Adam Williamson
> <adamwill(a)fedoraproject.org> wrote:
> >
> > On Tue, 2019-10-15 at 11:35 -0400, Neal Gompa wrote:
> > > On Tue, Oct 15, 2019 at 11:22 AM Matthew Miller
> > > <mattdm(a)fedoraproject.org> wrote:
> > > > On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> > > > > > As package maintainers we all make technical decisions
which have
> > > > > > significant impact on our users every day - whether
that's in the choice
> > > > > > of defaults, choice of build flags, or whatever. Honestly
delivering as
> > > > > > modules-vs-non-modules is a completely trivial issue
compared to most of
> > > > > > the stuff I spend time on. If "yum install X"
still works most people
> > > > > > just don't care about the RPM/dnf/repo mechanics behind
that.
> > > > >
> > > > > Except it works only half way. The installation works. Later,
> > > > > dependencies are broken. Upgrades are broken. "yum remove
X" does
> > > > > not undo the action completely.
> > > > >
> > > > > The main issue is: user just enabled a module without doing it
> > > > > explicitly. The user needs to know how to handle modules in
order to
> > > > > recover.
> > > >
> > > > I never expect "yum remove X" to be the inverse of
"yum install X". DNF's
> > > > magical leaf tracking makes it a bit more so, but not exactly. So, I
don't
> > > > think we should make that a very high priority concern (although if
we can
> > > > improve it, so much the better).
> > > >
> > > > Upgrades need to work, though. And they need to work regardless of
whether
> > > > we ban default modules or not. So, given that, I'm not _really_
seeing big
> > > > differences in practice for the user beteen these two proposals, and
the one
> > > > (no default streams) negates one of the whole intended advantages of
the
> > > > entire thing.
> > > >
> > > > What am I missing?
> > > >
> > >
> > > Upgrades can never work without changing fundamental properties of
modules.
> > >
> > > There are two traits of modules that break everything:
> > > * Module activation is introduces a hard lock of content source
> > > * There is no concept of module transitions
> > >
> > > In the RHEL world, these two qualities are not great, but they are
> > > serviceable (RHEL 9 is going to hurt!). In the Fedora world, they make
> > > having default modules essentially untenable. It could even be argued
> > > that non-default modules are not serviceable either.
> > >
> > > There is no way in modularity to say that a module is being replaced
> > > with another. There's also no way to say that a module is going away
> > > and transition accordingly. Moreover, since modules have a strong
> > > dependency on an abstract property that is defined as a consequence of
> > > the system content (the "platform" module), it is not possible
to have
> > > orphan modules that are still binary compatible as you upgrade.
> > > Finally, modularization and foregoing non-modular versions of content
> > > has an implicit commitment that you *never* demodularize because there
> > > is no module transition capabilities in the modularity technology.
> > >
> > > These flaws boil down to Fedora Modularity being a failure because the
> > > technology does not account for the needs of Fedora as a distribution,
> > > as a project, or as a community.
> >
> > I think this is actually a very cogent summary of the major problems
> > we're dealing with in modularity right now, but it's *also* needlessly
> > negative.
> >
> > Inventing new stuff is hard. You rarely get it right the first time.
> > Speaking personally here I think it was a big mistake to do this second
> > version of Modularity as fast as we did and drop it into RHEL 8 before
> > it had been in Fedora for two minutes, that was an unforced error; but
> > even if we hadn't done that, I'm not surprised the second cut at
> > Modularity isn't perfect. The second cut at RPM wasn't perfect either,
> > that's why we're on 4.x not 2.x. :)
> >
> > What's happening right now is the process of us trying something out
> > and finding out where the problems are. That's what happens when you
> > invent new stuff, it's harder than just carrying on doing the old
> > stuff.
> >
> > So: I'm on board with most of what you say here, but there's no need
to
> > say it means Modularity is "a failure". It means right now it's
not
> > entirely baked and we're realizing the concept needs extending and
> > perhaps reworking a bit, just like we realized with the *first* cut at
> > Modularity which we abandoned between a Beta release and a Final
> > release. This is causing us to have to deal with some icky problems,
> > but again, that's not *new*. We had to deal with some fairly icky
> > problems when systemd showed up. We had to deal with some fairly icky
> > problems when grub2 showed up. It's a process we've dealt with before.
> > We know how to do it. We just need to hold our noses and fix the icky
> > problems, and then sit down and think about the design issues that have
> > become apparent in Modularity v2 through our actually implementing it
> > and using it (which is what Fedora is for, remember!) and figure out
> > how to address them in Modularity v3.
>
> Are we going to do a Modularity v3? Because people seemed to be
> *really* reluctant to go down that path because it would break
> compatibility with RHEL. Honestly, my big problem is how much we're
> being held hostage by RHEL for this. I would have been happier if we'd
> focused on Fedora cases and naturally extended it to RHEL cases
> afterward.
RHEL isn't holding Fedora hostage. That's also unnecessarily antagonistic.
I think treating Fedora and RHEL as entirely separate things, and not
part of the same ecosystem is exactly what got us into this in the
first place. Red Hat is trying to address this because we very much
see how the disconnects happen, and how they are problematic for
everyone involved. There is pain felt on the enterprise side as well.
If we take a step back and consider all of the various pieces of our
ecosystem, it makes sense to try and figure out ways to make them as
cohesive as possible. Not the same, but not wildly and
problematically different either.
Fedora, RHEL, and now CentOS Stream are all pieces of that. What you
call "holding hostage" I would call an attempt to proactively
contribute in the various communities we are all a part of. There has
been reluctance to do that in the past and it has cost both sides. It
will be messy and there will be growing pains. I would ask that we
have patience and assume positive intent. It is fine to raise
technical concerns, but they don't need to be portrayed in a manner
that makes it seem like the communities are in direct conflict.
The hostage situation predates all this new stuff. Perhaps now it's
not a hostage situation, but it definitely felt like it leading up to
RHEL 8's release. I felt like we were railroaded into the design that
we have today, and aside from some minor late-stage adjustments, we
felt really out of the loop for its actual development.
--
真実はいつも一つ！/ Always, there's only one truth!
11:10 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Tue, 2019-10-15 at 12:25 -0400, Neal Gompa wrote:
> So: I'm on board with most of what you say here, but there's no need to
> say it means Modularity is "a failure". It means right now it's not
> entirely baked and we're realizing the concept needs extending and
> perhaps reworking a bit, just like we realized with the *first* cut at
> Modularity which we abandoned between a Beta release and a Final
> release. This is causing us to have to deal with some icky problems,
> but again, that's not *new*. We had to deal with some fairly icky
> problems when systemd showed up. We had to deal with some fairly icky
> problems when grub2 showed up. It's a process we've dealt with before.
> We know how to do it. We just need to hold our noses and fix the icky
> problems, and then sit down and think about the design issues that have
> become apparent in Modularity v2 through our actually implementing it
> and using it (which is what Fedora is for, remember!) and figure out
> how to address them in Modularity v3.
Are we going to do a Modularity v3? Because people seemed to be
*really* reluctant to go down that path because it would break
compatibility with RHEL.
Well, those are just the terms I use to think about it, they have no
official validity. So in a sense it's a pointless semantic question.
Let's put it this way: I'd be surprised if we don't wind up having to
make significant changes/enhancements to the modularity rules and/or
implementation to be able to resolve the issues we're dealing with
right now (Stephen, in some of his replies, seems to agree) and to me
it'd be valid to call that 'modularity v3'. Whether anyone else would
call it that or not doesn't seem too important :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
12:28 p.m.
On Wed, Oct 16, 2019 at 12:11 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
On Tue, 2019-10-15 at 12:25 -0400, Neal Gompa wrote:
>
> > So: I'm on board with most of what you say here, but there's no need
to
> > say it means Modularity is "a failure". It means right now it's
not
> > entirely baked and we're realizing the concept needs extending and
> > perhaps reworking a bit, just like we realized with the *first* cut at
> > Modularity which we abandoned between a Beta release and a Final
> > release. This is causing us to have to deal with some icky problems,
> > but again, that's not *new*. We had to deal with some fairly icky
> > problems when systemd showed up. We had to deal with some fairly icky
> > problems when grub2 showed up. It's a process we've dealt with before.
> > We know how to do it. We just need to hold our noses and fix the icky
> > problems, and then sit down and think about the design issues that have
> > become apparent in Modularity v2 through our actually implementing it
> > and using it (which is what Fedora is for, remember!) and figure out
> > how to address them in Modularity v3.
>
> Are we going to do a Modularity v3? Because people seemed to be
> *really* reluctant to go down that path because it would break
> compatibility with RHEL.
Well, those are just the terms I use to think about it, they have no
official validity. So in a sense it's a pointless semantic question.
Let's put it this way: I'd be surprised if we don't wind up having to
make significant changes/enhancements to the modularity rules and/or
implementation to be able to resolve the issues we're dealing with
right now (Stephen, in some of his replies, seems to agree) and to me
it'd be valid to call that 'modularity v3'. Whether anyone else would
call it that or not doesn't seem too important :)
I agree with everything Adam just said. I'd avoid the term "Modularity
v3" in general, because it sounds like a total redesign (which is what
the v1->v2 switch ended up being). So far, all of the things I've been
proposing have been done with an aim of not causing any additional
breakage and to ensure that an upgrade path exists.
That said, as we've rolled this out, new information has come up that
is requiring us to re-evaluate some of the original design decisions
(the most obvious being the "changing default streams" case). The two
different proposals for how to handle that have been offered up
because they provide an upgrade path to get out of the current
situation without significant manual intervention.
Also, I want to reiterate this point, because it is central: There are
three core issues from which most of the complaints in this thread
ultimately stem:
1) Modular default streams are not available to the non-modular
buildroot, so packages that want to build against them are forced to
either become modules or vanish from Fedora (or use complex build-time
workarounds).
2) Once a stream has been enabled on the system, users are bound to it
- even across upgrades between releases. Since the purpose of default
streams is to eliminate the need for users to understand module
interactions if they want to continue operating the way they did
pre-Modularity, they need to be able to follow stream changes on
upgrades to match the intended defaults for the new system.
3) The policy on default streams was not clearly defined and
communicated. The Modularity WG recently voted to affirm that all
artifacts installable from default streams of a module must obey all
of the rules of packages in non-modular Fedora (primarily Stable
Updates policy and the expectation that they are treated as supported
API for all purposes). This is the "Java/Maven Problem", where the
Java team created a maven module that included stripped-down
dependencies in the default stream, thus owning those package names
with unsupported content.
We have plans in place for how to deal with each of these critical
problems (without resorting to throwing it all away).
1) This will be solved by the new Koji/MBS feature that we've
codenamed "Ursa Prime" (as a replacement for the original "Ursa
Major"
tool that was built for RHEL 8). Unlike its predecessor, it requires
no additional daemon service running to handle things. We will create
a buildroot compose for each release that is created from the set of
packages available from all of the default streams and both Koji (in
infra) and mock (on packager systems) will be able to consume this.
Koji will behave the same as mock, where it will rely on libdnf's
default module stream handling to populate the buildroot, so we won't
have to worry about disparities between the local and infrastructure
packager experiences.
2) There is an entire email thread [1] dedicated to how we are going
to solve this. I won't repeat it here.
3) We need to get the policy I described above written onto -stone
tablets- the Packaging Guidelines and then we need to go and make any
stream that isn't compliant with it a non-default stream.
[1]
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
1:40 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Wed, Oct 16, 2019 at 01:28:37PM -0400, Stephen Gallagher wrote:
1) This will be solved by the new Koji/MBS feature that we've
codenamed "Ursa Prime" (as a replacement for the original "Ursa
Major"
tool that was built for RHEL 8). Unlike its predecessor, it requires
no additional daemon service running to handle things. We will create
a buildroot compose for each release that is created from the set of
packages available from all of the default streams and both Koji (in
infra) and mock (on packager systems) will be able to consume this.
Koji will behave the same as mock, where it will rely on libdnf's
default module stream handling to populate the buildroot, so we won't
have to worry about disparities between the local and infrastructure
packager experiences.
How would this act in the case where a default stream depends on a
non-default stream?
(And how would restricting default streams to only be able to depend on
default streams change things?)
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:30 p.m.
On Wed, Oct 16, 2019 at 2:41 PM Matthew Miller <mattdm(a)fedoraproject.org> wrote:
On Wed, Oct 16, 2019 at 01:28:37PM -0400, Stephen Gallagher wrote:
> 1) This will be solved by the new Koji/MBS feature that we've
> codenamed "Ursa Prime" (as a replacement for the original "Ursa
Major"
> tool that was built for RHEL 8). Unlike its predecessor, it requires
> no additional daemon service running to handle things. We will create
> a buildroot compose for each release that is created from the set of
> packages available from all of the default streams and both Koji (in
> infra) and mock (on packager systems) will be able to consume this.
> Koji will behave the same as mock, where it will rely on libdnf's
> default module stream handling to populate the buildroot, so we won't
> have to worry about disparities between the local and infrastructure
> packager experiences.
How would this act in the case where a default stream depends on a
non-default stream?
The idea is that it would act exactly the same way that dnf on the
local system would act: if you builddep software from a stream that
requires a non-default stream, it would enable that non-default
stream.
(And how would restricting default streams to only be able to depend
on
default streams change things?)
There wouldn't need to be a behavior change; this is all done at the
libdnf layer; it just means that there is less available software in
the buildroot (since we'd necessarily have to exclude anything that
would conflict).
4:30 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Wed, Oct 16, 2019 at 04:30:32PM -0400, Stephen Gallagher wrote:
The idea is that it would act exactly the same way that dnf on the
local system would act: if you builddep software from a stream that
requires a non-default stream, it would enable that non-default
stream.
Ah, I see. Thanks!
> (And how would restricting default streams to only be able to
depend on
> default streams change things?)
There wouldn't need to be a behavior change; this is all done at the
libdnf layer; it just means that there is less available software in
the buildroot (since we'd necessarily have to exclude anything that
would conflict).
I still see "non default stream enabled by default stream" to be surprising
behavior we should avoid. But perhaps I am making too big a hammer of it. :)
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
5:05 p.m.
On Wednesday, October 16, 2019 2:30:21 PM MST Matthew Miller wrote:
On Wed, Oct 16, 2019 at 04:30:32PM -0400, Stephen Gallagher wrote:
> The idea is that it would act exactly the same way that dnf on the
> local system would act: if you builddep software from a stream that
> requires a non-default stream, it would enable that non-default
> stream.
Ah, I see. Thanks!
> > (And how would restricting default streams to only be able to depend on
> > default streams change things?)
>
> There wouldn't need to be a behavior change; this is all done at the
> libdnf layer; it just means that there is less available software in
> the buildroot (since we'd necessarily have to exclude anything that
> would conflict).
I still see "non default stream enabled by default stream" to be surprising
behavior we should avoid. But perhaps I am making too big a hammer of it.
:)
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List
Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List
Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Realistically, I believe that default streams themselves are something we
should avoid, if the cost is low, and it is. There are many users, probably
the vast majority of users, that don't use Modularity. It's great to have the
option available, but to force it upon them is really unfortunate.
Additionally, wouldn't a "default module" pulling in a "non-default
module"
cause mass breakage? I can only see that causing ABI breakage..
--
John M. Harris, Jr.
Splentity
2:32 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Wed, Oct 16, 2019 at 03:05:43PM -0700, John M. Harris Jr wrote:
Realistically, I believe that default streams themselves are
something we
should avoid, if the cost is low, and it is. There are many users,
probably the vast majority of users, that don't use Modularity. It's great
to have the option available, but to force it upon them is really
unfortunate.
I don't think characterizing this as "forced upon users" is a good framing.
It's unnecessarily hostile. Users who happen to get packages which are in a
default stream shouldn't experience any practical difference from having a
non-modular package.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
4:52 p.m.
On Thu, Oct 17, 2019 at 9:33 PM Matthew Miller <mattdm(a)fedoraproject.org> wrote:
On Wed, Oct 16, 2019 at 03:05:43PM -0700, John M. Harris Jr wrote:
> Realistically, I believe that default streams themselves are something we
> should avoid, if the cost is low, and it is. There are many users,
> probably the vast majority of users, that don't use Modularity. It's great
> to have the option available, but to force it upon them is really
> unfortunate.
I don't think characterizing this as "forced upon users" is a good
framing.
It's unnecessarily hostile.
I agree that the language is a tad aggressive, but still:
Users who happen to get packages which are in a
default stream shouldn't experience any practical difference from having a
non-modular package.
And yet, the differences *are undeniably there*, and users are
expected to fix issues this causes on their systems down the road
themselves, even if they didn't consciously "opted in" to anything.
I'd not characterize this as "forcing something upon users", but it's
still getting in through the back door while they're possibly not
looking.
(And that's why the first thing I do for all my new fedora installs is
to disable modular repos, just so no modules get installed
"accidentally".)
Fabio
> --
> Matthew Miller
> <mattdm(a)fedoraproject.org>
> Fedora Project Leader
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
5:06 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 10/17/19 1:32 PM, Matthew Miller wrote:
On Wed, Oct 16, 2019 at 03:05:43PM -0700, John M. Harris Jr wrote:
> Realistically, I believe that default streams themselves are something we
> should avoid, if the cost is low, and it is. There are many users,
> probably the vast majority of users, that don't use Modularity. It's great
> to have the option available, but to force it upon them is really
> unfortunate.
I don't think characterizing this as "forced upon users" is a good
framing.
It's unnecessarily hostile. Users who happen to get packages which are in a
default stream shouldn't experience any practical difference from having a
non-modular package.
I don't like the language either, but it's undeniable that dealing with
modules is different than dealing with normal packages. Enabling a module
excludes those packages in the module from "normal" depsolving. I've just
run
into that with the RHEL8 rhn-tools module which provides koan (and excludes
cobbler). I couldn't install my own cobbler or koan packages (with higher
EVRs) from my copr until I disabled that module - and it was not easy to
figure that out.
I'm also not adverse to learning new ways of doing things. I think modules
can still provide some clear benefits to users and developers eventually, but
let's not pretend there are no differences involved.
--
Orion Poplawski
Manager of NWRA Technical Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane orion(a)nwra.com
Boulder, CO 80301 https://www.nwra.com/
7:11 p.m.
Stephen Gallagher wrote:
3) We need to get the policy I described above written onto -stone
tablets- the Packaging Guidelines and then we need to go and make any
stream that isn't compliant with it a non-default stream.
But then we need a policy that requires a default version (non-modular or at
least a default stream) to be available. Otherwise we end up with packages
that are not installable out of the box because they have no default version
at all.
Matthew Miller wrote:
How would this act in the case where a default stream depends on a
non-default stream?
From a policy standpoint, that non-default stream then ought to be bound by
the same rules as default streams. But allowing a default stream to depend
on a non-default stream paves the way for version conflicts to happen, so I
am not convinced that it is a good idea to begin with.
(And how would restricting default streams to only be able to depend
on
default streams change things?)
It would solve the version conflicts issue, so it makes a lot of sense, but
at that point, why not require the default versions to just be non-modular
instead? The main argument for using default streams was that they can
depend on non-default streams of other modules. So if we disallow this
(which I think makes sense), we may as well disallow default streams
entirely and simplify things for everyone. (We would just need a short-term
workaround for the default streams that exist now. The problem would be gone
in the long run.)
Kevin Kofler
7:41 p.m.
On Wed, Oct 16, 2019 at 8:12 PM Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Stephen Gallagher wrote:
> 3) We need to get the policy I described above written onto -stone
> tablets- the Packaging Guidelines and then we need to go and make any
> stream that isn't compliant with it a non-default stream.
But then we need a policy that requires a default version (non-modular or at
least a default stream) to be available. Otherwise we end up with packages
that are not installable out of the box because they have no default version
at all.
Not necessarily. It may be that we have to content ourselves with some
software always requiring a module enablement to use it. For example,
I maintain a module for Review Board, a Django-based code review tool.
For complicated reasons, it cannot run against Django newer than 1.6
(the Review Board upstream maintains the 1.6 stream of Django for
security patches). If we decide that deps of default streams are bound
to the same rules as default streams (or, alternately, Matthew's
proposal of requiring only depending on default streams), then the
necessary outcome is that Review Board can only be available via
non-default stream (since it depends on django:1.6).
That said, I'm sure that we can come up with ways to make that more
discoverable, but today we have limitations with how DNF is able to do
`search` or `repoquery`... it can only manage this with default or
enabled streams and it doesn't "see" the non-default streams. If we go
this path, I'd want to hear some input (and get some help!) on
improving the discovery experience.
Matthew Miller wrote:
> How would this act in the case where a default stream depends on a
> non-default stream?
From a policy standpoint, that non-default stream then ought to be bound by
the same rules as default streams. But allowing a default stream to depend
on a non-default stream paves the way for version conflicts to happen, so I
am not convinced that it is a good idea to begin with.
With the notable caveat of the example above, I'm willing to accept
this restriction in Fedora. We just need to acknowledge that it *will*
mean that some software we provide is less discoverable. If that's
deemed to be a worthwhile tradeoff, I have no problems with
implementing it.
> (And how would restricting default streams to only be able to
depend on
> default streams change things?)
It would solve the version conflicts issue, so it makes a lot of sense, but
at that point, why not require the default versions to just be non-modular
instead? The main argument for using default streams was that they can
depend on non-default streams of other modules. So if we disallow this
(which I think makes sense), we may as well disallow default streams
entirely and simplify things for everyone. (We would just need a short-term
workaround for the default streams that exist now. The problem would be gone
in the long run.)
There's still a case that you haven't considered, which is things that
work at runtime as a default but cannot *build* against the default
set of packages. For example, we might have packages whose buildsystem
still relies on Python 2 (WAF?) but doesn't require it at runtime.
There might be packages that haven't yet migrated to a new,
backwards-incompatible change to Docbook for generating documentation.
Or packages that only build properly with a newer version of golang
than shipped in that release, or any of a thousand other examples that
are easily solved by build-time-only content and dependencies in
module streams.
Also there are yet other packages (such as in the Go and Rust
ecosystems) that could simply be built once on Rawhide with the latest
compiler and shipped to each of the other releases without needing a
rebuild because they are statically linked. Modules allow this, basic
RPMs not so much.
So even if we eliminate the version conflicts issue by restricting
what comprises a default stream, there are other benefits to module
default streams.
8:15 p.m.
On Wed, Oct 16, 2019 at 8:42 PM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Wed, Oct 16, 2019 at 8:12 PM Kevin Kofler <kevin.kofler(a)chello.at> wrote:
>
> Stephen Gallagher wrote:
> > 3) We need to get the policy I described above written onto -stone
> > tablets- the Packaging Guidelines and then we need to go and make any
> > stream that isn't compliant with it a non-default stream.
>
> But then we need a policy that requires a default version (non-modular or at
> least a default stream) to be available. Otherwise we end up with packages
> that are not installable out of the box because they have no default version
> at all.
>
Not necessarily. It may be that we have to content ourselves with some
software always requiring a module enablement to use it. For example,
I maintain a module for Review Board, a Django-based code review tool.
For complicated reasons, it cannot run against Django newer than 1.6
(the Review Board upstream maintains the 1.6 stream of Django for
security patches). If we decide that deps of default streams are bound
to the same rules as default streams (or, alternately, Matthew's
proposal of requiring only depending on default streams), then the
necessary outcome is that Review Board can only be available via
non-default stream (since it depends on django:1.6).
That said, I'm sure that we can come up with ways to make that more
discoverable, but today we have limitations with how DNF is able to do
`search` or `repoquery`... it can only manage this with default or
enabled streams and it doesn't "see" the non-default streams. If we go
this path, I'd want to hear some input (and get some help!) on
improving the discovery experience.
A big chunk of the problem in fixing this will go away once this is
merged: https://github.com/rpm-software-management/libdnf/pull/814
> Matthew Miller wrote:
> > How would this act in the case where a default stream depends on a
> > non-default stream?
>
> From a policy standpoint, that non-default stream then ought to be bound by
> the same rules as default streams. But allowing a default stream to depend
> on a non-default stream paves the way for version conflicts to happen, so I
> am not convinced that it is a good idea to begin with.
>
With the notable caveat of the example above, I'm willing to accept
this restriction in Fedora. We just need to acknowledge that it *will*
mean that some software we provide is less discoverable. If that's
deemed to be a worthwhile tradeoff, I have no problems with
implementing it.
We need a way for module dependencies to not care of
default/non-default and handle this gracefully, even in transitions.
> > (And how would restricting default streams to only be able
to depend on
> > default streams change things?)
>
> It would solve the version conflicts issue, so it makes a lot of sense, but
> at that point, why not require the default versions to just be non-modular
> instead? The main argument for using default streams was that they can
> depend on non-default streams of other modules. So if we disallow this
> (which I think makes sense), we may as well disallow default streams
> entirely and simplify things for everyone. (We would just need a short-term
> workaround for the default streams that exist now. The problem would be gone
> in the long run.)
>
There's still a case that you haven't considered, which is things that
work at runtime as a default but cannot *build* against the default
set of packages. For example, we might have packages whose buildsystem
still relies on Python 2 (WAF?) but doesn't require it at runtime.
There might be packages that haven't yet migrated to a new,
backwards-incompatible change to Docbook for generating documentation.
Or packages that only build properly with a newer version of golang
than shipped in that release, or any of a thousand other examples that
are easily solved by build-time-only content and dependencies in
module streams.
Also there are yet other packages (such as in the Go and Rust
ecosystems) that could simply be built once on Rawhide with the latest
compiler and shipped to each of the other releases without needing a
rebuild because they are statically linked. Modules allow this, basic
RPMs not so much.
This is actually not true. Every language stack we have does some
degree of dynamic linking, even if it's only to glibc.
The *only* reason we don't allow non-modular RPMs to be built once in
rawhide and shipped to all releases without a rebuild is purely
policy. We can mechanically do that since Koji lets you multi-tag RPMs
into multiple collections.
So even if we eliminate the version conflicts issue by restricting
what comprises a default stream, there are other benefits to module
default streams.
Ehh...
--
真実はいつも一つ！/ Always, there's only one truth!
8:53 p.m.
Stephen Gallagher wrote:
Not necessarily. It may be that we have to content ourselves with
some
software always requiring a module enablement to use it. For example,
I maintain a module for Review Board, a Django-based code review tool.
For complicated reasons, it cannot run against Django newer than 1.6
(the Review Board upstream maintains the 1.6 stream of Django for
security patches). If we decide that deps of default streams are bound
to the same rules as default streams (or, alternately, Matthew's
proposal of requiring only depending on default streams), then the
necessary outcome is that Review Board can only be available via
non-default stream (since it depends on django:1.6).
The user-friendly approach for that is to use a parallel-installable
compatibility package (with a suffixed Name, such as django1.6) instead of a
module.
There's still a case that you haven't considered, which is
things that
work at runtime as a default but cannot *build* against the default
set of packages. For example, we might have packages whose buildsystem
still relies on Python 2 (WAF?) but doesn't require it at runtime.
There too, the correct approach is to provide (continue providing) a python2
compatibility package. (Even the python27 one the Python SIG has introduced
in Rawhide might be enough, actually.) There is no technical reason for it
being in a module.
There might be packages that haven't yet migrated to a new,
backwards-incompatible change to Docbook for generating documentation.
Or packages that only build properly with a newer version of golang
than shipped in that release, or any of a thousand other examples that
are easily solved by build-time-only content and dependencies in
module streams.
The preferred solution there is really to fix the package to build with the
latest version, which is the way we have always handled such FTBFS issues.
But in the worst case, we can just ship the last successful build until the
FTBFS is fixed. I think we are giving way too much importance to FTBFS bugs.
Also there are yet other packages (such as in the Go and Rust
ecosystems) that could simply be built once on Rawhide with the latest
compiler and shipped to each of the other releases without needing a
rebuild because they are statically linked. Modules allow this, basic
RPMs not so much.
As Neal Gompa hinted at, that is not a good idea because the build will be
linked with the Rawhide glibc and, as a result, typically not run on the
older releases.
Kevin Kofler
9:52 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 03:53 +0200, Kevin Kofler wrote:
The user-friendly approach for that is to use a parallel-installable
compatibility package (with a suffixed Name, such as django1.6)
instead of a
module.
I've always liked Gentoo's solution to "too fast too slow" (which has
been around for what seems like more than 10 years, though I'm not sure
the exact date they introduced it, so let's just say "a long time")
with their slots mechanism. Their packages just have another metadata
field that is called slot, and typically corresponds to a portion of
the version that indicates how granular the packager wants to go with
parallel availability (and in what seems like all cases I've noticed,
they also have parallel installability, though I'm not confident enough
to claim they offer that for all packages without verification). For
example, I have Python 2.7 and 3.6 one one of my machines, but it's not
called python2 and python3, both are just called "python":
$ equery l python
dev-lang/python-2.7.15
dev-lang/python-3.6.5
And all Python packages I install don't need python2- or python3-
prefixes, they can just use their normal names and Gentoo does the
right thing based on which Python versions I have installed. It
actually doesn't even install n versions of each Python package, just
one that determines which Pythons you have and just gets installed for
those versions. It's great. rPath's Conary also had some of these
concepts (and had my favorite packaging format that I've used - your
"spec file" is just a Python class. No DSL, and you can take advantage
of inheritance to make pretty simple and clean spec files!).
Why not just stand on the shoulders of giants and add a "slot" field to
the spec file? Then you can have everything be regular RPMs. No
complicated build services to figure out. It's simpler.
I've also wondered why we didn't take a queue from NixOS, which seems
to have also solved these problems a long time ago in an even more
advanced way than Gentoo did. It'd be a bigger change than using slots,
but I think it'd be even more advantageous.
Are we re-implementing something that's been solved by others before
instead of joining them upstream? We aren't solving the "too fast, too
slow" problem no matter what we do, because it was already solved long
ago by others. Why not just benefit from their efforts and adopt their
technology and experiences to Fedora, rather than doing it ourselves?
Isn't that the advantage of open source software?
12:10 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 10:52 -0400, Randy Barlow wrote:
I've always liked Gentoo's solution to "too fast too
slow"
with their slots mechanism.
I realized it would be good if I explained what this is in more detail
for those who aren't familiar.
The slot is another field on the package, and the package manager
ensures that you only have one installed from each slot of a package.
That's basically it.
Conventionally, the slot format is often a leading portion of the
package's version that users might want to be able to install side by
side. Using Python as an example, Gentoo usually would use things like
"2.7" or "3.6" in the "slot" field, i.e., leaving out the .z
component
of the version. This means users could install 2.7, 3.5, 3.6, and 3.7
at the same time, but that 3.7.0 and 3.7.1 would not be offered. Which
granularity of slots are offered depend on the package and the packager
(i.e., the packager makes sure the package can be simultaneously
installed at the granularity of the slot, and of course also volunteers
to maintain the various versions.)
I like this as a user because it's presented in a simple way to me.
Slotted packages don't feel any different than non-slotted packages. I
use the same commands to install them as I do any other package, unless
I want a specific version, in which case I just add some <>= type
statement on my install command to specify which version I want
(otherwise I get the default, which is typically latest available as
long as it doesn't conflict with other packages I have installed.)
I also like this as a packager, because my packaging experience is also
pretty much the same as a non-slotted package, with the obvious caveat
that I also need to make sure that the slots can co-exist. But what I
mean is that the tooling and packaging file format is pretty much the
same. No new things to learn, no other services. It's just another
package.
4:15 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 17. 10. 19 2:41, Stephen Gallagher wrote:
For example, we might have packages whose buildsystem
still relies on Python 2 (WAF?) but doesn't require it at runtime.
We do have them. How does that relate to modularity at all?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
6:41 a.m.
On Thu, Oct 17, 2019 at 5:17 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 17. 10. 19 2:41, Stephen Gallagher wrote:
> For example, we might have packages whose buildsystem
> still relies on Python 2 (WAF?) but doesn't require it at runtime.
We do have them. How does that relate to modularity at all?
That was a representative example. I came up with it at 11pm after a
long day. Don't read too much into the specifics. The point was that
builds may require newer or older software than we have available in
the non-modular buildroot, but don't require them at runtime.
Similarly, the example of "build on Rawhide, run anywhere" was
backwards. I should have said "build on oldest supported Fedora, carry
through".
6:46 a.m.
On Thu, Oct 17, 2019 at 7:42 AM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
Similarly, the example of "build on Rawhide, run anywhere" was
backwards. I should have said "build on oldest supported Fedora, carry
through".
Modules currently fail at this because they have a platform
dependency. And we could *easily* permit it for non-modular packages
if we actually wanted it. I think we've generally said we don't want
to maintain content that can't be built for current releases, so this
is problematic.
Also, if we did this, we should drop the DistTag, because it confuses
people otherwise.
--
真実はいつも一つ！/ Always, there's only one truth!
6:52 a.m.
On Thu, Oct 17, 2019 at 7:48 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Thu, Oct 17, 2019 at 7:42 AM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
>
> Similarly, the example of "build on Rawhide, run anywhere" was
> backwards. I should have said "build on oldest supported Fedora, carry
> through".
Modules currently fail at this because they have a platform
dependency.
I meant to address that in an earlier email when I saw you mention
that. It was wrong then, too.
Most modules *specify* a platform dependency, but they *do not have to*.
Take a look at the maven module, for example:
https://src.fedoraproject.org/modules/maven/blob/3.5/f/maven.yaml
You will notice that it specifies F29 as a build requirement but
specifies '[ ]' for the platform: this means "Build it on F29, run it
anywhere".
And we could *easily* permit it for non-modular packages
if we actually wanted it. I think we've generally said we don't want
to maintain content that can't be built for current releases, so this
is problematic.
We could permit it, sure. Modularity has facilities for implementing
it already, whereas non-modular would need to grow new rules for the
Message Tagging Service or do a lot of manual tagging.
Also, if we did this, we should drop the DistTag, because it
confuses
people otherwise.
Could you explain that?
7:31 a.m.
On Thu, Oct 17, 2019 at 7:53 AM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Thu, Oct 17, 2019 at 7:48 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
>
> On Thu, Oct 17, 2019 at 7:42 AM Stephen Gallagher <sgallagh(a)redhat.com>
wrote:
> >
> > Similarly, the example of "build on Rawhide, run anywhere" was
> > backwards. I should have said "build on oldest supported Fedora, carry
> > through".
>
> Modules currently fail at this because they have a platform
> dependency.
I meant to address that in an earlier email when I saw you mention
that. It was wrong then, too.
Most modules *specify* a platform dependency, but they *do not have to*.
Take a look at the maven module, for example:
https://src.fedoraproject.org/modules/maven/blob/3.5/f/maven.yaml
You will notice that it specifies F29 as a build requirement but
specifies '[ ]' for the platform: this means "Build it on F29, run it
anywhere".
Ah, so we already have the BR/R split for module deps.
> And we could *easily* permit it for non-modular packages
> if we actually wanted it. I think we've generally said we don't want
> to maintain content that can't be built for current releases, so this
> is problematic.
>
We could permit it, sure. Modularity has facilities for implementing
it already, whereas non-modular would need to grow new rules for the
Message Tagging Service or do a lot of manual tagging.
> Also, if we did this, we should drop the DistTag, because it confuses
> people otherwise.
>
Could you explain that?
People regularly look at NEVRAs to identify whether there are
"broken"/"old" packages to clean up, and when you see .fc29 installed
on an otherwise .fc31 system, it looks like something has gone wrong.
If packages are intended to be built once and pushed to multiple
releases, it shouldn't have the DistTag suffix.
--
真実はいつも一つ！/ Always, there's only one truth!
7:36 a.m.
On Thu, Oct 17, 2019 at 8:33 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
People regularly look at NEVRAs to identify whether there are
"broken"/"old" packages to clean up, and when you see .fc29
installed
on an otherwise .fc31 system, it looks like something has gone wrong.
If packages are intended to be built once and pushed to multiple
releases, it shouldn't have the DistTag suffix.
OK, I misunderstood you because DistTag and %{?dist} are actually
distinct things in RPM and you're using the wrong one here :)
That's an interesting thought and we should consider what to do about
it. Mind opening a ticket on https://pagure.io/modularity so we don't
forget?
7:42 a.m.
On Thu, Oct 17, 2019 at 8:37 AM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Thu, Oct 17, 2019 at 8:33 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
> People regularly look at NEVRAs to identify whether there are
> "broken"/"old" packages to clean up, and when you see .fc29
installed
> on an otherwise .fc31 system, it looks like something has gone wrong.
> If packages are intended to be built once and pushed to multiple
> releases, it shouldn't have the DistTag suffix.
OK, I misunderstood you because DistTag and %{?dist} are actually
distinct things in RPM and you're using the wrong one here :)
It's called that in the Packaging Guidelines:
https://docs.fedoraproject.org/en-US/packaging-guidelines/DistTag/
It should probably be named differently now that there's an _actual_ DistTag.
That's an interesting thought and we should consider what to do
about
it. Mind opening a ticket on https://pagure.io/modularity so we don't
forget?
Done: https://pagure.io/modularity/issue/157
--
真実はいつも一つ！/ Always, there's only one truth!
6:52 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 17. 10. 19 13:41, Stephen Gallagher wrote:
On Thu, Oct 17, 2019 at 5:17 AM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 17. 10. 19 2:41, Stephen Gallagher wrote:
>> For example, we might have packages whose buildsystem
>> still relies on Python 2 (WAF?) but doesn't require it at runtime.
>
> We do have them. How does that relate to modularity at all?
>
That was a representative example. I came up with it at 11pm after a
long day. Don't read too much into the specifics. The point was that
builds may require newer or older software than we have available in
the non-modular buildroot, but don't require them at runtime.
Yes, they do. There are many ways to solve the problem, from compat packages
trough various tag-magic to buildroot overrides in custom side tags. We have had
this problem for years and we were able to deal with it quite well.
Similarly, the example of "build on Rawhide, run anywhere"
was
backwards. I should have said "build on oldest supported Fedora, carry
through".
As others have pointed out, this is technically possible without modules and we
don't do it because we are (arguably rightfully) obsessed with rebuilds.
There is no technical problem to tag a .fc29 build into f29, f30, f31 and f32
(except that currently, bodhi will not let you do that, so admin permissions are
probably necessary; however this is already possible for anybody for branched
(prior to bodhi enablement) + rawhide).
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
7:08 a.m.
On Thu, Oct 17, 2019 at 7:53 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
> That was a representative example. I came up with it at 11pm
after a
> long day. Don't read too much into the specifics. The point was that
> builds may require newer or older software than we have available in
> the non-modular buildroot, but don't require them at runtime.
Yes, they do. There are many ways to solve the problem, from compat packages
trough various tag-magic to buildroot overrides in custom side tags. We have had
this problem for years and we were able to deal with it quite well.
Please let's not confuse "we have figured out how to work around it by
making our packagers do a lot of tricky hacks" with "able to deal with
it quite well". This is a major issue for non-expert packagers. I
don't have statistics on how many people give up on a package because
of this issue, but I know *I* have done so (and I'd consider myself
more experienced than most people at packaging).
One of the (often un- or misinformed) major arguments people keep
using against Modularity is "it makes packaging harder!". This is one
place where it makes things *much* easier on the packagers. It's a
clear reduction in complexity.
7:23 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 17. 10. 19 14:08, Stephen Gallagher wrote:
On Thu, Oct 17, 2019 at 7:53 AM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>> That was a representative example. I came up with it at 11pm after a
>> long day. Don't read too much into the specifics. The point was that
>> builds may require newer or older software than we have available in
>> the non-modular buildroot, but don't require them at runtime.
>
> Yes, they do. There are many ways to solve the problem, from compat packages
> trough various tag-magic to buildroot overrides in custom side tags. We have had
> this problem for years and we were able to deal with it quite well.
>
Please let's not confuse "we have figured out how to work around it by
making our packagers do a lot of tricky hacks" with "able to deal with
it quite well". This is a major issue for non-expert packagers. I
don't have statistics on how many people give up on a package because
of this issue, but I know *I* have done so (and I'd consider myself
more experienced than most people at packaging).
One of the (often un- or misinformed) major arguments people keep
using against Modularity is "it makes packaging harder!". This is one
place where it makes things *much* easier on the packagers. It's a
clear reduction in complexity.
The real problem here is: It makes things easier for packagers who opt-in to
modularity at a cost of making things harder for packagers who don't.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:56 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 08:08 -0400, Stephen Gallagher wrote:
One of the (often un- or misinformed) major arguments people keep
using against Modularity is "it makes packaging harder!". This is one
place where it makes things *much* easier on the packagers. It's a
clear reduction in complexity.
I tried to maintain a package I have, rpick, as a module, and I found
it to be difficult. In order to package the module version of rpick, I
had to write a yaml file that listed hashes of every dependency of
rpick, and every dependency of those dependencies, and their
dependencies, and so on. This file:
https://src.fedoraproject.org/modules/rpick/blob/latest/f/rpick.yaml
I also maintain another Rust package, yubibomb, which is just a normal
spec file:
https://src.fedoraproject.org/rpms/rust-yubibomb/blob/master/f/rust-yubib...
I've heard it claimed that there is something specific to Rust here,
but to be honest the explanation of that was over my head. I have also
heard it claimed that Modularity was specifically supposed to help with
Rust packaging, but I haven't personally felt like it was helpful due
to the yaml file.
I'm sure my information is out of touch, but I'm sharing my experience
as someone who doesn't have a lot of time to go learn something new -
my reaction is that it's easier to maintain the "traditional" version.
My other reaction is that other distributions have solved the "too
fast, too slow" problem in simpler ways, as I wrote about elsewhere on
this thread, and I am curious why we didn't adopt their solutions (or
at least use them to guide ourselves). I've not packaged for Nix
before, but I have packaged for rPath and for Gentoo and I found their
solutions to be easy and natural.
12:18 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 12:56 -0400, Randy Barlow wrote:
I
had to write a yaml file that listed hashes of every dependency of
rpick, and every dependency of those dependencies, and their
dependencies, and so on.
By the way, I didn't actually end up doing this, Igor did it for me. I
didn't mean to take credit, I mean more that "I, the packager, had to
do this", not "I, Randy, had to do this". Igor rocks.
12:31 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
----- Original Message -----
From: "Randy Barlow" <bowlofeggs(a)fedoraproject.org>
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Thursday, October 17, 2019 1:18:08 PM
Subject: Re: Fedora 32 System-Wide Change proposal: Modules in Non-Modular Buildroot
On Thu, 2019-10-17 at 12:56 -0400, Randy Barlow wrote:
> I
> had to write a yaml file that listed hashes of every dependency of
> rpick, and every dependency of those dependencies, and their
> dependencies, and so on.
By the way, I didn't actually end up doing this, Igor did it for me. I
didn't mean to take credit, I mean more that "I, the packager, had to
do this", not "I, Randy, had to do this". Igor rocks.
Just for the obvious reply... :)
`ref` behaves just like `git checkout` would. So yeah, you can pass a very
explicit hash and then you won't get any updates until you bump the hash
in the spec again. However, you could pass tags (like you mention in the
comment next to each hash) or branches instead. The latter is what some
other modules do, like the eclipse one:
https://src.fedoraproject.org/modules/eclipse/blob/2019-06/f/eclipse.yaml...
See also:
https://github.com/fedora-modularity/libmodulemd/blob/master/spec.v2.yaml...
In the latter case (in RHEL at least), you have to commit a (potentially
empty) commit to the module and then a rebuild of the module will pull in
the latest contents from the branches you specify for each component RPM.
Perhaps we need to collect a "tips and tricks" section for modularity?
HTH,
Alex
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
1:14 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 08:08 -0400, Stephen Gallagher wrote:
One of the (often un- or misinformed) major arguments people keep
using against Modularity is "it makes packaging harder!".
One thing I've found to be a problem with modularity is that it's easy
to be un- or misinformed. I spent a lot of time reading the docs (and
I'm glad there are docs, I'm not criticizing the docs) and I still
found it difficult to modularize my rpick package.
My experiences as a packager for the other distributions I've worked
with that offer parallel installability (which necessarily means
parallel availability) was that I didn't need to know very much to do
it. It was generally obvious what to do to make my package use slots,
for example.
I think the problem is that modularity is complicated enough that it's
hard to know it well without investment.
Could we think of a solution that is simple so that packagers can more
easily understand how it works? Or better, can we employ a solution
that another distribution has developed?
2:04 p.m.
On Thu, 17 Oct 2019 at 14:15, Randy Barlow <bowlofeggs(a)fedoraproject.org> wrote:
Could we think of a solution that is simple so that packagers can
more
easily understand how it works?
The issue is how many different choices are you allowing and where you
are allowing them to be made. A lot of the gentoo and nixos seem to be
made on the personal system. I choose how I want to build out my box
and it builds itself to do that. If you push the decisions up into the
central build system, then you need to start working with other people
and letting them mess with your decisions. That means various design
choices no longer are simple.. and are less technical. They are
instead policy and bureaucracy in order to grease the interactions and
different opinions to get a compromise or in the end a hard decision
even when no one wants it.
The problem I found with both the documentation and most of our
conversations is that we have been trying to treat this as a
technology problem when it is a social one. A distribution is a social
contract between different packagers to make a 'product' (I am trying
for a word which is less commercial so if you know one.. use it
instead). A module is just a smaller social contract where we need to
make a set of things work together in a way that works both with the
larger product and its own self. Most of the rules in writing a good
spec file are meant to make it easier that someone else can maintain
it later.. that is a social requirement. We might dress it up with
'MUST have %{0fedora}' but it is a social contract. The rpm doesn't
care if I use %0fedora or 1. [Man I have digressed into finding the
source of the Nile somewhere.. ok pull it together smooge]
In the end, we need to work out what the social wants/needs for the
technological marvel we create are before we build it. The larger the
number of people you are working with said marvel, the more important
that social rules/upkeep is. Putting the brakes on the car after it is
going 100mph is a little late.
Or better, can we employ a solution
that another distribution has developed?
Not without using their packaging system, their build system and their
other design choices. Working out slots would mean needing to make
changes into how RPM works and how yum/dnf work. It might also not be
possible because a bit of Gentoo's magic is letting the local system
build all the different slot choices instead of having to build all
the combinatorics that having 3 different glibc and N gcc compilers
would need. To do the magic NixOS does.. we need to eject the FHS and
use a similar system. At that point, we aren't developing Fedora
anymore.. we are developing a clone of NixOS or Gentoo. [And there
would be no magic way to move from a Fedora 33 system to Fedora-Nix-34
or Fedora-Gen-34.. at which point we might as well just call the whole
thing from scratch.] If we are going that far we might as well rewrite
conary in python3 or rust and start from there...
So any solution will have to 'learn' the lessons of these groups but
design and write a solution from scratch to meet them.
--
Stephen J Smoogen.
5:11 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, 2019-10-17 at 15:04 -0400, Stephen John Smoogen wrote:
Not without using their packaging system, their build system and
their
other design choices.
Frankly, this is not a bad caveat. Keep in mind that we also had to
change our build system for modularity.
Working out slots would mean needing to make
changes into how RPM works and how yum/dnf work.
This is also true for modularity.
It might also not be
possible because a bit of Gentoo's magic is letting the local system
build all the different slot choices instead of having to build all
the combinatorics that having 3 different glibc and N gcc compilers
would need.
Conary had use flags without the local system doing the builds. But in
general, I think the slots thing would not require any local builds -
it's just a way to specify cleanly which versions of something can be
available for parallel install and which cannot.
By the way, I think I started using Gentoo in 2004 or so, and back then
they didn't have parallel installability, but they did have parallel
availability, even without slots. They just had various versions of
each package available in the same repository at the same time, and I
could use the package manager to express which one I want.
To do the magic NixOS does.. we need to eject the FHS and
use a similar system. At that point, we aren't developing Fedora
anymore.. we are developing a clone of NixOS or Gentoo.
Well there's more to the distro than the package manager. I think we
could use great tools that other distros have made instead of having to
make our own just because we want to be more separate from them.
[And there
would be no magic way to move from a Fedora 33 system to Fedora-Nix-
34
or Fedora-Gen-34.. at which point we might as well just call the
whole
thing from scratch.] If we are going that far we might as well
rewrite
conary in python3 or rust and start from there...
I agree that upgrading would be hard.
Resurrecting conary for Python 3 would be awesome, I'm sad that it is
just sitting there in Python 2.old not being used - it's really good.
Rust is also great.
10:21 a.m.
Stephen John Smoogen <smooge(a)gmail.com> writes:
On Thu, 17 Oct 2019 at 14:15, Randy Barlow
<bowlofeggs(a)fedoraproject.org> wrote:
> Or better, can we employ a solution that another distribution has
> developed?
Not without using their packaging system, their build system and their
other design choices. Working out slots would mean needing to make
changes into how RPM works and how yum/dnf work. It might also not be
possible because a bit of Gentoo's magic is letting the local system
build all the different slot choices instead of having to build all
the combinatorics that having 3 different glibc and N gcc compilers
would need. To do the magic NixOS does.. we need to eject the FHS and
use a similar system. At that point, we aren't developing Fedora
anymore.. we are developing a clone of NixOS or Gentoo. [And there
would be no magic way to move from a Fedora 33 system to Fedora-Nix-34
or Fedora-Gen-34.. at which point we might as well just call the whole
thing from scratch.] If we are going that far we might as well rewrite
conary in python3 or rust and start from there...
So any solution will have to 'learn' the lessons of these groups but
design and write a solution from scratch to meet them.
While you're right that the solutions from source distros (i.e., NixOS
and Gentoo) would be very hard to adapt, binary distros have also solved
this problem in different ways. I'm most familiar with Debian's
solution (virtual packages[2], provides:, and alternatives [1]) which
to my mind maps much more clearly onto Fedora's setup. Obviously we
can't use their code wholesale without migrating to APT, but as you say,
the goal is to derive inspiration.
Thanks,
--Robbie
1: https://wiki.debian.org/DebianAlternatives
2: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual
10:42 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Fri, 2019-10-18 at 11:21 -0400, Robbie Harwood wrote:
Obviously we
can't use their code wholesale without migrating to APT, but as you
say,
the goal is to derive inspiration.
I honestly think it should be on the table to consider switching to a
different packaging technology than rpm/dnf. Collaborating with other
distributions to solve common problems has benefits. We do that to some
extent with SUSE (which is cool!), but not to the fullest extent one
could imagine.
But yeah as you say here, my original point was more that we could
learn from what others did and integrate similar concepts into RPM/dnf,
especially since what they did was simpler, and as a result of that
simplicity didn't struggle to gain adoption. Those concepts were also
easier to write the code to make them happen. For example, adding a
slot field to the RPM would be pretty easy and I think is obvious that
we wouldn't need to create build services like Ursa Major to make that
work. We would need a change to Koji and dnf to consider the slot as
part of a package's uniqueness, but I think that's a small change in
comparison. It's easier to create and easier to use, which I see as a
win-win.
Thanks for mentioning the Debian solution. I wasn't familiar with that
one. This means we have even more solutions to study for inspiration.
Too fast, too slow is a solved problem and there are existing solutions
to learn from or even adapt. We don't need to make our own.
10:56 a.m.
On Fri, Oct 18, 2019 at 11:43 AM Randy Barlow
<bowlofeggs(a)fedoraproject.org> wrote:
On Fri, 2019-10-18 at 11:21 -0400, Robbie Harwood wrote:
> Obviously we
> can't use their code wholesale without migrating to APT, but as you
> say,
> the goal is to derive inspiration.
But yeah as you say here, my original point was more that we could
learn from what others did and integrate similar concepts into RPM/dnf,
especially since what they did was simpler, and as a result of that
simplicity didn't struggle to gain adoption. Those concepts were also
easier to write the code to make them happen. For example, adding a
slot field to the RPM would be pretty easy and I think is obvious that
we wouldn't need to create build services like Ursa Major to make that
work. We would need a change to Koji and dnf to consider the slot as
part of a package's uniqueness, but I think that's a small change in
comparison. It's easier to create and easier to use, which I see as a
win-win.
Thanks for mentioning the Debian solution. I wasn't familiar with that
one. This means we have even more solutions to study for inspiration.
Too fast, too slow is a solved problem and there are existing solutions
to learn from or even adapt. We don't need to make our own.
RPM + DNF actually already supports Debian's solution.
Philosophically, RH/Fedora systems don't typically use alternatives as
aggressively as Mageia, SUSE, or Debian systems. We do actually
generate virtual names and use them, and we can use them to satisfy
things like this. We already do this with Rust by having parallel
installable stuff with common names that work with ranged
dependencies.
There are other things we could do along these lines, too...
--
真実はいつも一つ！/ Always, there's only one truth!
11:16 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 18.10.19 17:21, Robbie Harwood wrote:
While you're right that the solutions from source distros (i.e.,
NixOS
and Gentoo) would be very hard to adapt, binary distros have also solved
this problem in different ways. I'm most familiar with Debian's
solution (virtual packages[2], provides:, and alternatives [1]) which
to my mind maps much more clearly onto Fedora's setup. Obviously we
can't use their code wholesale without migrating to APT, but as you say,
the goal is to derive inspiration.
Thanks,
--Robbie
1: https://wiki.debian.org/DebianAlternatives
2: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual
I'm not a Fedora packager (yet[1]), so correct me if I'm
misunderstanding things completely, but is there any reason not to adapt
the existing RPM provides: functionality for this?
That is essentially what Archlinux does with their pacman, where an
second version of the same program will typically both Provide: and
Conflict: with the 'default' package. It's extensively used by users to
provide e.g. development versions of stable packages, see e.g.
inkscape-git [2].
- if you install inkscape, you get the default version
- if you install inkscape-git, you get the development version
- when resolving dependencies, pacman knows that inkscape-git provides
the capabilities of (Provides:) but cannot be installed together with
(Conflicts:) of the package named inkscape
- otherwise, they are two separate packages, so you don't have the
problem of implicitly switching streams
Parallel installability is taken care of with compat packages as usual,
though if the packages happen to be parallel installable, there's
probably no reason not to allow that via 'provides:' without 'conflicts:'
What I like about this approach that all this magic only comes into play
when resolving dependencies. In all other circumstances they are
packages like any other, and are treated as such.
Christopher
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1756582
[2] https://aur.archlinux.org/packages/inkscape-git
3:25 p.m.
Christopher Engelhard <ce(a)lcts.de> writes:
On 18.10.19 17:21, Robbie Harwood wrote:
> While you're right that the solutions from source distros (i.e., NixOS
> and Gentoo) would be very hard to adapt, binary distros have also solved
> this problem in different ways. I'm most familiar with Debian's
> solution (virtual packages[2], provides:, and alternatives [1]) which
> to my mind maps much more clearly onto Fedora's setup. Obviously we
> can't use their code wholesale without migrating to APT, but as you say,
> the goal is to derive inspiration.
>
> 1: https://wiki.debian.org/DebianAlternatives
> 2: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual
I'm not a Fedora packager (yet[1]), so correct me if I'm
misunderstanding things completely, but is there any reason not to
adapt the existing RPM provides: functionality for this?
I'm not aware of technical reasons not to do this - as Neal elaborated
on, this is already possible.
That is essentially what Archlinux does with their pacman, where an
second version of the same program will typically both Provide: and
Conflict: with the 'default' package. It's extensively used by users to
provide e.g. development versions of stable packages, see e.g.
inkscape-git [2].
- if you install inkscape, you get the default version
- if you install inkscape-git, you get the development version
- when resolving dependencies, pacman knows that inkscape-git provides
the capabilities of (Provides:) but cannot be installed together with
(Conflicts:) of the package named inkscape
- otherwise, they are two separate packages, so you don't have the
problem of implicitly switching streams
Parallel installability is taken care of with compat packages as usual,
though if the packages happen to be parallel installable, there's
probably no reason not to allow that via 'provides:' without
'conflicts:'
What I like about this approach that all this magic only comes into play
when resolving dependencies. In all other circumstances they are
packages like any other, and are treated as such.
Yup!
Thanks,
--Robbie
4 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Fri, 2019-10-18 at 16:25 -0400, Robbie Harwood wrote:
Christopher Engelhard <ce(a)lcts.de> writes:
> On 18.10.19 17:21, Robbie Harwood wrote:
>
> > While you're right that the solutions from source distros (i.e., NixOS
> > and Gentoo) would be very hard to adapt, binary distros have also solved
> > this problem in different ways. I'm most familiar with Debian's
> > solution (virtual packages[2], provides:, and alternatives [1]) which
> > to my mind maps much more clearly onto Fedora's setup. Obviously we
> > can't use their code wholesale without migrating to APT, but as you say,
> > the goal is to derive inspiration.
> >
> > 1: https://wiki.debian.org/DebianAlternatives
> > 2: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual
>
> I'm not a Fedora packager (yet[1]), so correct me if I'm
> misunderstanding things completely, but is there any reason not to
> adapt the existing RPM provides: functionality for this?
I'm not aware of technical reasons not to do this - as Neal elaborated
on, this is already possible.
I don't think there is one. But it's not a perfect solution either.
I mean, we're talking about *stacks* here, remember. FreeIPA isn't one
package - you can't just have 'freeipa4' and 'freeipa5' both
providing
'freeipa' and call it a day. There's a whole ton of stuff that goes in
there. 'freeipa5' might need different version of ten different other
components compared to 'freeipa4'.
So, you're no longer looking at one pair of packages with the same
'provides', you're looking at a dozen, and someone needs to be keeping
track of which ones go together and what they're all for in the first
place. But with this approach you're not building any infrastructure
for *doing* that. These things are clearly a conceptual lump, but
you're not really providing a way to express or work with that.
Modularity is intended to do that - to provide a framework and some
technical backing for treating alternate groups of packages that
provide different implementations of the same thing *as* groups,
essentially.
I mean, it's not like the Modularity team wasn't aware of these other
approaches, I'm pretty sure they were discussed. It might be reasonable
to go back and check the various docs and blog posts and things the
modularity team put out before just suggesting 'hey what about this
other thing', especially where 'this other thing' is something you
could reasonably imagine at least one of them would probably have heard
or thought of...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
11:14 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Fri, Oct 18, 2019 at 02:00:11PM -0700, Adam Williamson wrote:
On Fri, 2019-10-18 at 16:25 -0400, Robbie Harwood wrote:
> Christopher Engelhard <ce(a)lcts.de> writes:
>
> > On 18.10.19 17:21, Robbie Harwood wrote:
> >
> > > While you're right that the solutions from source distros (i.e.,
NixOS
> > > and Gentoo) would be very hard to adapt, binary distros have also solved
> > > this problem in different ways. I'm most familiar with Debian's
> > > solution (virtual packages[2], provides:, and alternatives [1]) which
> > > to my mind maps much more clearly onto Fedora's setup. Obviously we
> > > can't use their code wholesale without migrating to APT, but as you
say,
> > > the goal is to derive inspiration.
> > >
> > > 1: https://wiki.debian.org/DebianAlternatives
> > > 2:
https://www.debian.org/doc/debian-policy/ch-relationships.html#s-virtual
> >
> > I'm not a Fedora packager (yet[1]), so correct me if I'm
> > misunderstanding things completely, but is there any reason not to
> > adapt the existing RPM provides: functionality for this?
>
> I'm not aware of technical reasons not to do this - as Neal elaborated
> on, this is already possible.
I don't think there is one. But it's not a perfect solution either.
Modularity is intended to do that - to provide a framework and some
technical backing for treating alternate groups of packages that
provide different implementations of the same thing *as* groups,
essentially.
I mean, it's not like the Modularity team wasn't aware of these other
approaches, I'm pretty sure they were discussed. It might be reasonable
to go back and check the various docs and blog posts and things the
modularity team put out before just suggesting 'hey what about this
other thing', especially where 'this other thing' is something you
could reasonably imagine at least one of them would probably have heard
or thought of...
Well, you might think that, but it's not clear that this was the case.
Let's not assume one way or the other.
I mean, we're talking about *stacks* here, remember. FreeIPA
isn't one
package - you can't just have 'freeipa4' and 'freeipa5' both
providing
'freeipa' and call it a day. There's a whole ton of stuff that goes in
there. 'freeipa5' might need different version of ten different other
components compared to 'freeipa4'.
So, you're no longer looking at one pair of packages with the same
'provides', you're looking at a dozen, and someone needs to be keeping
track of which ones go together and what they're all for in the first
place. But with this approach you're not building any infrastructure
for *doing* that.
I would really like this explained. Taking this particular example, to
let dnf install all dependencies properly, freeipa5 must already list
*all* dependencies through rpm Requires/Conflicts/Recommends/Suggests,
including versions. Likewise, freeipa4 must also do that. It doesn't
matter if the rpms are built normally or in a module, listing all deps
is needed in both cases. If the rpm doesn't work with all versions of
dependencies, it might be fine during initial installation, where you
get everything latest by default, but will break if the user does a
partial upgrade.
This means that in one case the user says 'dnf install freeipa5' and they
get the full stack, and in the other case they say 'dnf install freeipa4'
and they get a different (partially overlapping) set of recursive deps.
These things are clearly a conceptual lump, but
you're not really providing a way to express or work with that.
Why not? What is wrong with having rpm Requires?
Zbyszek
4:23 a.m.
On Thu, Oct 17, 2019 at 8:09 AM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Thu, Oct 17, 2019 at 7:53 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
> > That was a representative example. I came up with it at 11pm after a
> > long day. Don't read too much into the specifics. The point was that
> > builds may require newer or older software than we have available in
> > the non-modular buildroot, but don't require them at runtime.
>
> Yes, they do. There are many ways to solve the problem, from compat packages
> trough various tag-magic to buildroot overrides in custom side tags. We have had
> this problem for years and we were able to deal with it quite well.
>
Please let's not confuse "we have figured out how to work around it by
making our packagers do a lot of tricky hacks" with "able to deal with
it quite well". This is a major issue for non-expert packagers. I
don't have statistics on how many people give up on a package because
of this issue, but I know *I* have done so (and I'd consider myself
more experienced than most people at packaging).
One of the (often un- or misinformed) major arguments people keep
using against Modularity is "it makes packaging harder!". This is one
place where it makes things *much* easier on the packagers. It's a
clear reduction in complexity.
As someone who's published over 500 RPMs to various commercial and
free software environments, it's a complete waste of my time, it's
breaking my build my build environments and wasting time I'd rather
spend on building or updating individual components without confusion
about maintaining multiple dependency trees.
4:26 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
----- Original Message -----
From: "Stephen Gallagher" <sgallagh(a)redhat.com>
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Thursday, October 17, 2019 2:41:28 AM
Subject: Re: Fedora 32 System-Wide Change proposal: Modules in Non-Modular Buildroot
> > (And how would restricting default streams to only be able to depend on
> > default streams change things?)
>
> It would solve the version conflicts issue, so it makes a lot of sense, but
> at that point, why not require the default versions to just be non-modular
> instead? The main argument for using default streams was that they can
> depend on non-default streams of other modules. So if we disallow this
> (which I think makes sense), we may as well disallow default streams
> entirely and simplify things for everyone. (We would just need a short-term
> workaround for the default streams that exist now. The problem would be
> gone
> in the long run.)
>
There's still a case that you haven't considered, which is things that
work at runtime as a default but cannot *build* against the default
set of packages. For example, we might have packages whose buildsystem
still relies on Python 2 (WAF?) but doesn't require it at runtime.
There might be packages that haven't yet migrated to a new,
backwards-incompatible change to Docbook for generating documentation.
Or packages that only build properly with a newer version of golang
than shipped in that release, or any of a thousand other examples that
are easily solved by build-time-only content and dependencies in
module streams.
In current context(1.X) package that doesn't build with new version of Go has usually
serious issue(s) on the side of the package/upstream. Fortunately this is really uncommon
in my experience(even for packages/upstreams that fear/think that they will break with
newer version of Go).
Also there are yet other packages (such as in the Go and Rust
ecosystems) that could simply be built once on Rawhide with the latest
compiler and shipped to each of the other releases without needing a
rebuild because they are statically linked. Modules allow this, basic
RPMs not so much.
For Go this is oversimplification and common misconception(go built binaries are not
uncommonly dynamically linked in the "C/ELF" sense(glibc,...) and statically
linked in Go sense). It might work for some selected(maybe even most) Go based packages,
but not always(new glibc,...) and not universally.
JC
>
> So even if we eliminate the version conflicts issue by restricting
> what comprises a default stream, there are other benefits to module
> default streams.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
6:42 a.m.
On Thu, Oct 17, 2019 at 5:27 AM Jakub Cajka <jcajka(a)redhat.com> wrote:
For Go this is oversimplification and common misconception(go built
binaries are not uncommonly dynamically linked in the "C/ELF" sense(glibc,...)
and statically linked in Go sense). It might work for some selected(maybe even most) Go
based packages, but not always(new glibc,...) and not universally.
See my other reply. These were meant to be stand-in examples. Don't
read too much into the specifics; I wrote them when I was tired.
4:09 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 16. 10. 19 19:28, Stephen Gallagher wrote:
1) This will be solved by the new Koji/MBS feature that we've
codenamed "Ursa Prime" (as a replacement for the original "Ursa
Major"
tool that was built for RHEL 8). Unlike its predecessor, it requires
no additional daemon service running to handle things. We will create
a buildroot compose for each release that is created from the set of
packages available from all of the default streams and both Koji (in
infra) and mock (on packager systems) will be able to consume this.
Koji will behave the same as mock, where it will rely on libdnf's
default module stream handling to populate the buildroot, so we won't
have to worry about disparities between the local and infrastructure
packager experiences.
Thank You! Please describe this in the change proposal.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4:44 p.m.
3) We need to get the policy I described above written onto -stone
tablets- the Packaging Guidelines and then we need to go and make
any
stream that isn't compliant with it a non-default stream.
Thank you. If we want to use default streams, then we indeed need a strict
policy on how they are formed and we need to make sure, the
requirements are met here. I dare to say, that we should block on every
default stream that would not meet the requirements.
--
Lukáš Růžička
FEDORA QE, RHCE
Red Hat
<https://www.redhat.com>
Purkyňova 115
612 45 Brno - Královo Pole
lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>
4:10 p.m.
On Tue, Oct 15, 2019 at 10:13 AM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
What's happening right now is the process of us trying something
out
and finding out where the problems are. That's what happens when you
invent new stuff, it's harder than just carrying on doing the old
stuff.
I agree Adam.
I think Neal's summary helps and I appreciate it. I agree it's
negative, but I think there is one additional aspect from my point of
view: There are some proponents of Modularity that keep pitching the
idea to me that everything I do needs to go into a module, like,
yesterday. Maybe we can tone that messaging down a bit around Fedora
for a while.
- Ken
- Ken
12:56 p.m.
Matthew Miller <mattdm(a)fedoraproject.org> writes:
On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
>> As package maintainers we all make technical decisions which have
>> significant impact on our users every day - whether that's in the
>> choice of defaults, choice of build flags, or whatever. Honestly
>> delivering as modules-vs-non-modules is a completely trivial issue
>> compared to most of the stuff I spend time on. If "yum install X"
>> still works most people just don't care about the RPM/dnf/repo
>> mechanics behind that.
>
> Except it works only half way. The installation works. Later,
> dependencies are broken. Upgrades are broken. "yum remove X" does not
> undo the action completely.
>
> The main issue is: user just enabled a module without doing it
> explicitly. The user needs to know how to handle modules in order to
> recover.
I never expect "yum remove X" to be the inverse of "yum install
X". DNF's magical leaf tracking makes it a bit more so, but not
exactly. So, I don't think we should make that a very high priority
concern (although if we can improve it, so much the better).
I don't think it's an unreasonable expectation, especially for those
coming from APT land (Debian, Ubuntu) where `apt install foo` *is* the
inverse operation of `apt remove foo`.
Thanks,
--Robbie
1:12 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Tue, Oct 15, 2019 at 01:56:11PM -0400, Robbie Harwood wrote:
Matthew Miller <mattdm(a)fedoraproject.org> writes:
> On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
>>> As package maintainers we all make technical decisions which have
>>> significant impact on our users every day - whether that's in the
>>> choice of defaults, choice of build flags, or whatever. Honestly
>>> delivering as modules-vs-non-modules is a completely trivial issue
>>> compared to most of the stuff I spend time on. If "yum install
X"
>>> still works most people just don't care about the RPM/dnf/repo
>>> mechanics behind that.
>>
>> Except it works only half way. The installation works. Later,
>> dependencies are broken. Upgrades are broken. "yum remove X" does not
>> undo the action completely.
>>
>> The main issue is: user just enabled a module without doing it
>> explicitly. The user needs to know how to handle modules in order to
>> recover.
>
> I never expect "yum remove X" to be the inverse of "yum install
> X". DNF's magical leaf tracking makes it a bit more so, but not
> exactly. So, I don't think we should make that a very high priority
> concern (although if we can improve it, so much the better).
I don't think it's an unreasonable expectation, especially for those
coming from APT land (Debian, Ubuntu) where `apt install foo` *is* the
inverse operation of `apt remove foo`.
It isn't, you need to supplement it with `apt autoremove` to get rid
of auto-installed dependencies.
We have `dnf history undo X` to invert installation command.
--
Tomasz .. oo o. oo o. .o .o o. o. oo o. ..
Torcz .. .o .o .o .o oo oo .o .. .. oo oo
o.o.o. .o .. o. o. o. o. o. o. oo .. .. o.
3:08 p.m.
Tomasz Torcz <tomek(a)pipebreaker.pl> writes:
On Tue, Oct 15, 2019 at 01:56:11PM -0400, Robbie Harwood wrote:
> Matthew Miller <mattdm(a)fedoraproject.org> writes:
>
> > On Tue, Oct 15, 2019 at 12:36:15PM +0200, Miro Hrončok wrote:
> >>> As package maintainers we all make technical decisions which have
> >>> significant impact on our users every day - whether that's in the
> >>> choice of defaults, choice of build flags, or whatever. Honestly
> >>> delivering as modules-vs-non-modules is a completely trivial issue
> >>> compared to most of the stuff I spend time on. If "yum install
X"
> >>> still works most people just don't care about the RPM/dnf/repo
> >>> mechanics behind that.
> >>
> >> Except it works only half way. The installation works. Later,
> >> dependencies are broken. Upgrades are broken. "yum remove X" does
not
> >> undo the action completely.
> >>
> >> The main issue is: user just enabled a module without doing it
> >> explicitly. The user needs to know how to handle modules in order to
> >> recover.
> >
> > I never expect "yum remove X" to be the inverse of "yum install
> > X". DNF's magical leaf tracking makes it a bit more so, but not
> > exactly. So, I don't think we should make that a very high priority
> > concern (although if we can improve it, so much the better).
>
> I don't think it's an unreasonable expectation, especially for those
> coming from APT land (Debian, Ubuntu) where `apt install foo` *is* the
> inverse operation of `apt remove foo`.
It isn't, you need to supplement it with `apt autoremove` to get rid
of auto-installed dependencies.
We have `dnf history undo X` to invert installation command.
Ah, you're right. It is the case with aptitude, but not with apt or
apt-get. Sorry for the noise.
Thanks,
--Robbie
5:05 p.m.
Matthew Miller wrote:
Upgrades need to work, though. And they need to work regardless of
whether
we ban default modules or not. So, given that, I'm not _really_ seeing big
differences in practice for the user beteen these two proposals, and the
one (no default streams) negates one of the whole intended advantages of
the entire thing.
What am I missing?
I don't see how requiring a non-modular default version negates a central
advantage of Modularity. You can still offer all the versions of your module
and get all the benefits of Modularity. You just cannot force the module
onto all users, because that turned out to be causing many more problems
than it solves.
Kevin Kofler
4:59 p.m.
Joe Orton wrote:
If you don't want to ship any of your packages as modules I think
that's
great and you should continue doing that. On the other hand, I want to
move a bunch of my packages to module-only because I think I can do a
better job serving Fedora users that way.
How so? That is the big question that you are leaving unanswered: In what
way do you think you can do a better job serving Fedora users with module-
only packages than by also delivering a non-modular version (along with the
modular version for those who actually want it)?
Kevin Kofler
10:50 a.m.
On Wed, Oct 9, 2019 at 4:47 PM Ben Cotton <bcotton(a)redhat.com> wrote:
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
== Owner ==
* Name: [[User:Sgallagh| Stephen Gallagher]]
* Email: sgallagh(a)redhat.com
* Responsible WG: Modularity WG
== Detailed Description ==
As a major part of the Modularity design, we have a concept of default
module streams. These streams are built as modules, but the RPM
artifacts they deliver are intended to be used just like non-modular
RPMS. The aspirational goal is that a user of the system who never
executes a module-specific command (such as `dnf module install
nodejs:8`) should experience no meaningful changes in behavior from
how they would interact with a completely non-modular system. In
practice, this may mean that the informational output of package
managers may indicate that modules are being enabled and used, but a
user that does not have a specific reason to interact with the
selection of a module stream should have that managed on their behalf.
Similarly, the experience for package maintainers of non-modular
packages should be unaffected by an RPM dependency moving from the
non-modular repository into a default module stream. Up to the
present, this has not been the case; no module stream content has been
available in the non-modular buildroot for other packages to consume.
Koji builds of non-modular RPMs have had only the other non-modular
RPMs from that release available to their buildroots. In contrast,
building on local systems has access to both the non-modular RPMs and
the RPMs from any of the default module streams. With this Change,
Koji builds will have the same behavior and be able to depend on
content provided by default module streams. It also enables the same
behavior for Modular builds: the `platform` stream will now include
the contents of the default module streams for each release and do not
need to be explicitly specified in the modulemd `buildrequires`.
Note: This Change does not address the other major Modularity issue we
are facing around distribution upgrades with differing default
streams. When discussing this Change, please keep that topic separate.
== Benefit to Fedora ==
This will simplify the lives of package maintainers in Fedora in two
primary ways. I'll use a hypothetical example of the Node.js
interpreter and a JSApp package which is capable of running on Node.js
10 or 12 (but requires newer features than are provided by Node.js 8).
Additionally, the JSApp package requires the same versions of Node.js
at build-time.
* Fedora 29 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 30 ships `nodejs:8`, `nodejs:10` and `nodejs:12` module
streams. The `nodejs:10` stream is set as the default stream.
* Fedora 31 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F31 lifetime.
* Fedora 32 ships `nodejs:10` and `nodejs:12` module streams. The
`nodejs:12` stream is set as the default stream. The `nodejs:14`
stream will likely become available during the F32 lifetime.
On Fedora 29 through 31, the Node.js package maintainer needs to build
the `nodejs:10` package both as a module and as a non-modular RPM in
the distribution so that the JSApp package can be built. With this
Change, the Node.js package maintainer in Fedora 32+ will only need to
build the various Node.js streams and make one of them the default
stream. The packages from it will then be added to the buildroot for
non-modular packages. This will also make the packaging process
somewhat more efficient, as the maintainer needs only to manage the
module stream and the MBS will build it for all configured platforms.
Similarly, from the perspective of dependent maintainers, there will
no longer be anxiety about needing to move their package to a module
if one or more of their dependencies drops their non-modular version
in favor of a default stream. Their builds will continue to work as
they do today.
This might be a dumb question (perhaps I missed the answer somewhere
in the thread), but how do we handle setting overrides for modular
content? Without that, it's going to be hard to do modular update to
fix non-modular builds...
--
真実はいつも一つ！/ Always, there's only one truth!
12:16 p.m.
On Thu, Nov 14, 2019 at 11:52 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
> Similarly, from the perspective of dependent maintainers, there will
> no longer be anxiety about needing to move their package to a module
> if one or more of their dependencies drops their non-modular version
> in favor of a default stream. Their builds will continue to work as
> they do today.
>
This might be a dumb question (perhaps I missed the answer somewhere
in the thread), but how do we handle setting overrides for modular
content? Without that, it's going to be hard to do modular update to
fix non-modular builds...
Sorry, I don't understand what question you're asking here. Could you
try using an example to make it clearer?
11:08 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 09. 10. 19 22:46, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
Enable module default streams in the buildroot repository for modular
and non-modular RPMs.
== Summary ==
This Change (colloquially referred to as "Ursa Prime") enables the
Koji build-system to include the RPM artifacts provided by module
default streams in the buildroot when building non-modular (or
"traditional") RPMs.
I have one more technical concern.
Suppose a packager decides to package the "mycoolapp" software as a non-modular
package. "mycoolapp" is written in Python, it builds again non-modular Python,
currently 3.8, it requires "python(abi) = 3.8" on runtime.
The packager decides to use avocado in %check. Avocado comes from a module, it
requires "python(abi) = 3.8" as well, because the modular package was built with
Python 3.8. Avocado is in the bulidroot, so everything works.
The Python maintainers (that includes me) decide to update to Python 3.9 in
Fedora 33. They request a side tag to do that. They update the python3 to 3.9
and they mass rebuild all non-modular Python packages in it. "mycoolapp" cannot
resolve build dependencies because avocado requires "python(abi) = 3.8".
The Python maintainers need to detect this and figure out what happened.
Then, the Python maintainers need to either:
1. Exclude "mycoolapp" from the rebuild. That is possible until dozens of
other packages require "mycoolapp".
2. Ask the avocado maintainers to rebuild their module in the side tag and ask
releng to add the side-tag-built module into the side-tag buildroot (if it is
even possible).
3. Modify the spec of "mycoolapp" to temporarily disable %check and loose the
avocado dependency.
Or is there some other way?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:36 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
On 09. 10. 19 22:46, Ben Cotton wrote:
>https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
>Enable module default streams in the buildroot repository for modular
>and non-modular RPMs.
>
>== Summary ==
>This Change (colloquially referred to as "Ursa Prime") enables the
>Koji build-system to include the RPM artifacts provided by module
>default streams in the buildroot when building non-modular (or
>"traditional") RPMs.
I have one more technical concern.
Suppose a packager decides to package the "mycoolapp" software as a
non-modular package. "mycoolapp" is written in Python, it builds
again non-modular Python, currently 3.8, it requires "python(abi) =
3.8" on runtime.
Hmm, and what about an even simpler case:
"myswankyapp" is also written in Python, and is packaged as a module.
Python is rebuilt in a side tag, then the module blocks the upgrade.
What is supposed to happen in that case?
Zbyszek
> The packager decides to use avocado in %check. Avocado comes from a
> module, it requires "python(abi) = 3.8" as well, because the modular
> package was built with Python 3.8. Avocado is in the bulidroot, so
> everything works.
>
> The Python maintainers (that includes me) decide to update to Python
> 3.9 in Fedora 33. They request a side tag to do that. They update
> the python3 to 3.9 and they mass rebuild all non-modular Python
> packages in it. "mycoolapp" cannot resolve build dependencies
> because avocado requires "python(abi) = 3.8".
>
> The Python maintainers need to detect this and figure out what happened.
> Then, the Python maintainers need to either:
>
> 1. Exclude "mycoolapp" from the rebuild. That is possible until
> dozens of other packages require "mycoolapp".
>
> 2. Ask the avocado maintainers to rebuild their module in the side
> tag and ask releng to add the side-tag-built module into the
> side-tag buildroot (if it is even possible).
>
> 3. Modify the spec of "mycoolapp" to temporarily disable %check and
> loose the avocado dependency.
>
> Or is there some other way?
>
> --
> Miro Hrončok
> --
> Phone: +420777974800
> IRC: mhroncok
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
11:43 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
> On 09. 10. 19 22:46, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>
>> Enable module default streams in the buildroot repository for modular
>> and non-modular RPMs.
>>
>> == Summary ==
>> This Change (colloquially referred to as "Ursa Prime") enables the
>> Koji build-system to include the RPM artifacts provided by module
>> default streams in the buildroot when building non-modular (or
>> "traditional") RPMs.
>
> I have one more technical concern.
>
> Suppose a packager decides to package the "mycoolapp" software as a
> non-modular package. "mycoolapp" is written in Python, it builds
> again non-modular Python, currently 3.8, it requires "python(abi) =
> 3.8" on runtime.
Hmm, and what about an even simpler case:
"myswankyapp" is also written in Python, and is packaged as a module.
Python is rebuilt in a side tag, then the module blocks the upgrade.
What is supposed to happen in that case?
The module is rebuilt after the side tag is merged. Al least that is what I
think happened with avocado when we upgraded to Python 3.8.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:57 a.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>On 09. 10. 19 22:46, Ben Cotton wrote:
>>>https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>
>>>Enable module default streams in the buildroot repository for modular
>>>and non-modular RPMs.
>>>
>>>== Summary ==
>>>This Change (colloquially referred to as "Ursa Prime") enables the
>>>Koji build-system to include the RPM artifacts provided by module
>>>default streams in the buildroot when building non-modular (or
>>>"traditional") RPMs.
>>
>>I have one more technical concern.
>>
>>Suppose a packager decides to package the "mycoolapp" software as a
>>non-modular package. "mycoolapp" is written in Python, it builds
>>again non-modular Python, currently 3.8, it requires "python(abi) =
>>3.8" on runtime.
>
>Hmm, and what about an even simpler case:
>"myswankyapp" is also written in Python, and is packaged as a module.
>Python is rebuilt in a side tag, then the module blocks the upgrade.
>What is supposed to happen in that case?
The module is rebuilt after the side tag is merged. Al least that is
what I think happened with avocado when we upgraded to Python 3.8.
Automatically? And if the build fails?
Zbyszek
noon
On Thu, Nov 14, 2019 at 12:59 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
> >On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
> >>On 09. 10. 19 22:46, Ben Cotton wrote:
> >>>https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> >>>
> >>>Enable module default streams in the buildroot repository for modular
> >>>and non-modular RPMs.
> >>>
> >>>== Summary ==
> >>>This Change (colloquially referred to as "Ursa Prime") enables
the
> >>>Koji build-system to include the RPM artifacts provided by module
> >>>default streams in the buildroot when building non-modular (or
> >>>"traditional") RPMs.
> >>
> >>I have one more technical concern.
> >>
> >>Suppose a packager decides to package the "mycoolapp" software as
a
> >>non-modular package. "mycoolapp" is written in Python, it builds
> >>again non-modular Python, currently 3.8, it requires "python(abi) =
> >>3.8" on runtime.
> >
> >Hmm, and what about an even simpler case:
> >"myswankyapp" is also written in Python, and is packaged as a module.
> >Python is rebuilt in a side tag, then the module blocks the upgrade.
> >What is supposed to happen in that case?
>
> The module is rebuilt after the side tag is merged. Al least that is
> what I think happened with avocado when we upgraded to Python 3.8.
Automatically? And if the build fails?
It's not automatic. Release Engineering has to kick it by hand by
making a weird empty commit and triggering a build.
--
真実はいつも一つ！/ Always, there's only one truth!
12:11 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On Thu, Nov 14, 2019 at 01:00:52PM -0500, Neal Gompa wrote:
On Thu, Nov 14, 2019 at 12:59 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl> wrote:
>
> On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
> > On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
> > >On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
> > >>On 09. 10. 19 22:46, Ben Cotton wrote:
> >
>>>https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> > >>>
> > >>>Enable module default streams in the buildroot repository for
modular
> > >>>and non-modular RPMs.
> > >>>
> > >>>== Summary ==
> > >>>This Change (colloquially referred to as "Ursa Prime")
enables the
> > >>>Koji build-system to include the RPM artifacts provided by module
> > >>>default streams in the buildroot when building non-modular (or
> > >>>"traditional") RPMs.
> > >>
> > >>I have one more technical concern.
> > >>
> > >>Suppose a packager decides to package the "mycoolapp"
software as a
> > >>non-modular package. "mycoolapp" is written in Python, it
builds
> > >>again non-modular Python, currently 3.8, it requires "python(abi)
=
> > >>3.8" on runtime.
> > >
> > >Hmm, and what about an even simpler case:
> > >"myswankyapp" is also written in Python, and is packaged as a
module.
> > >Python is rebuilt in a side tag, then the module blocks the upgrade.
> > >What is supposed to happen in that case?
> >
> > The module is rebuilt after the side tag is merged. Al least that is
> > what I think happened with avocado when we upgraded to Python 3.8.
>
> Automatically? And if the build fails?
>
It's not automatic. Release Engineering has to kick it by hand by
making a weird empty commit and triggering a build.
And if the build fails?
In case this isn't obvious: right now python-sig will pummel all the
modules until they stop FTBFSing. If modules don't get the same
treatment, we have a problem.
Zbyszek
12:28 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 11. 19 19:11, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Nov 14, 2019 at 01:00:52PM -0500, Neal Gompa wrote:
> On Thu, Nov 14, 2019 at 12:59 PM Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl> wrote:
>>
>> On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
>>> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>>>> On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>>>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>>>>>
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>>>>
>>>>>> Enable module default streams in the buildroot repository for
modular
>>>>>> and non-modular RPMs.
>>>>>>
>>>>>> == Summary ==
>>>>>> This Change (colloquially referred to as "Ursa Prime")
enables the
>>>>>> Koji build-system to include the RPM artifacts provided by
module
>>>>>> default streams in the buildroot when building non-modular (or
>>>>>> "traditional") RPMs.
>>>>>
>>>>> I have one more technical concern.
>>>>>
>>>>> Suppose a packager decides to package the "mycoolapp"
software as a
>>>>> non-modular package. "mycoolapp" is written in Python, it
builds
>>>>> again non-modular Python, currently 3.8, it requires
"python(abi) =
>>>>> 3.8" on runtime.
>>>>
>>>> Hmm, and what about an even simpler case:
>>>> "myswankyapp" is also written in Python, and is packaged as a
module.
>>>> Python is rebuilt in a side tag, then the module blocks the upgrade.
>>>> What is supposed to happen in that case?
>>>
>>> The module is rebuilt after the side tag is merged. Al least that is
>>> what I think happened with avocado when we upgraded to Python 3.8.
>>
>> Automatically? And if the build fails?
>>
>
> It's not automatic. Release Engineering has to kick it by hand by
> making a weird empty commit and triggering a build.
And if the build fails?
In case this isn't obvious: right now python-sig will pummel all the
modules until they stop FTBFSing. If modules don't get the same
treatment, we have a problem.
Currently, modules don't get the same treatment. The reason basically is:
I don't know how to repoquery all modules to see what modular packages still
require Python 3.7. I was trying to get this information but at a certain point,
I've stopped trying.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1:58 p.m.
On Thu, Nov 14, 2019 at 1:33 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 14. 11. 19 19:11, Zbigniew Jędrzejewski-Szmek wrote:
> On Thu, Nov 14, 2019 at 01:00:52PM -0500, Neal Gompa wrote:
>> On Thu, Nov 14, 2019 at 12:59 PM Zbigniew Jędrzejewski-Szmek
>> <zbyszek(a)in.waw.pl> wrote:
>>>
>>> On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
>>>> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>>>>> On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>>>>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>>>>>>
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>>>>>
>>>>>>> Enable module default streams in the buildroot repository
for modular
>>>>>>> and non-modular RPMs.
>>>>>>>
>>>>>>> == Summary ==
>>>>>>> This Change (colloquially referred to as "Ursa
Prime") enables the
>>>>>>> Koji build-system to include the RPM artifacts provided by
module
>>>>>>> default streams in the buildroot when building non-modular
(or
>>>>>>> "traditional") RPMs.
>>>>>>
>>>>>> I have one more technical concern.
>>>>>>
>>>>>> Suppose a packager decides to package the "mycoolapp"
software as a
>>>>>> non-modular package. "mycoolapp" is written in Python,
it builds
>>>>>> again non-modular Python, currently 3.8, it requires
"python(abi) =
>>>>>> 3.8" on runtime.
>>>>>
>>>>> Hmm, and what about an even simpler case:
>>>>> "myswankyapp" is also written in Python, and is packaged
as a module.
>>>>> Python is rebuilt in a side tag, then the module blocks the
upgrade.
>>>>> What is supposed to happen in that case?
>>>>
>>>> The module is rebuilt after the side tag is merged. Al least that is
>>>> what I think happened with avocado when we upgraded to Python 3.8.
>>>
>>> Automatically? And if the build fails?
>>>
>>
>> It's not automatic. Release Engineering has to kick it by hand by
>> making a weird empty commit and triggering a build.
>
> And if the build fails?
>
> In case this isn't obvious: right now python-sig will pummel all the
> modules until they stop FTBFSing. If modules don't get the same
> treatment, we have a problem.
Currently, modules don't get the same treatment. The reason basically is:
I don't know how to repoquery all modules to see what modular packages still
require Python 3.7. I was trying to get this information but at a certain point,
I've stopped trying.
That's a reasonable concern and a problem we do need to solve. Would
you mind adding it to https://pagure.io/modularity/issues ?
(Side note: that tracker has gotten badly backlogged, but Petr Sabata
and I are planning to go through it tomorrow to clean it up and
prioritize things. Don't be frightened by its current state!)
2:12 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 11. 19 20:58, Stephen Gallagher wrote:
On Thu, Nov 14, 2019 at 1:33 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 14. 11. 19 19:11, Zbigniew Jędrzejewski-Szmek wrote:
>> On Thu, Nov 14, 2019 at 01:00:52PM -0500, Neal Gompa wrote:
>>> On Thu, Nov 14, 2019 at 12:59 PM Zbigniew Jędrzejewski-Szmek
>>> <zbyszek(a)in.waw.pl> wrote:
>>>>
>>>> On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
>>>>> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>>>>>> On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>>>>>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>>>>>>>
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>>>>>>
>>>>>>>> Enable module default streams in the buildroot repository
for modular
>>>>>>>> and non-modular RPMs.
>>>>>>>>
>>>>>>>> == Summary ==
>>>>>>>> This Change (colloquially referred to as "Ursa
Prime") enables the
>>>>>>>> Koji build-system to include the RPM artifacts provided
by module
>>>>>>>> default streams in the buildroot when building
non-modular (or
>>>>>>>> "traditional") RPMs.
>>>>>>>
>>>>>>> I have one more technical concern.
>>>>>>>
>>>>>>> Suppose a packager decides to package the
"mycoolapp" software as a
>>>>>>> non-modular package. "mycoolapp" is written in
Python, it builds
>>>>>>> again non-modular Python, currently 3.8, it requires
"python(abi) =
>>>>>>> 3.8" on runtime.
>>>>>>
>>>>>> Hmm, and what about an even simpler case:
>>>>>> "myswankyapp" is also written in Python, and is
packaged as a module.
>>>>>> Python is rebuilt in a side tag, then the module blocks the
upgrade.
>>>>>> What is supposed to happen in that case?
>>>>>
>>>>> The module is rebuilt after the side tag is merged. Al least that is
>>>>> what I think happened with avocado when we upgraded to Python 3.8.
>>>>
>>>> Automatically? And if the build fails?
>>>>
>>>
>>> It's not automatic. Release Engineering has to kick it by hand by
>>> making a weird empty commit and triggering a build.
>>
>> And if the build fails?
>>
>> In case this isn't obvious: right now python-sig will pummel all the
>> modules until they stop FTBFSing. If modules don't get the same
>> treatment, we have a problem.
>
> Currently, modules don't get the same treatment. The reason basically is:
>
> I don't know how to repoquery all modules to see what modular packages still
> require Python 3.7. I was trying to get this information but at a certain point,
> I've stopped trying.
>
That's a reasonable concern and a problem we do need to solve. Would
you mind adding it to https://pagure.io/modularity/issues ?
Done: https://pagure.io/modularity/issue/160
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
12:03 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 11. 19 18:57, Zbigniew Jędrzejewski-Szmek wrote:
On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>> On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>>
>>>> Enable module default streams in the buildroot repository for modular
>>>> and non-modular RPMs.
>>>>
>>>> == Summary ==
>>>> This Change (colloquially referred to as "Ursa Prime") enables
the
>>>> Koji build-system to include the RPM artifacts provided by module
>>>> default streams in the buildroot when building non-modular (or
>>>> "traditional") RPMs.
>>>
>>> I have one more technical concern.
>>>
>>> Suppose a packager decides to package the "mycoolapp" software as
a
>>> non-modular package. "mycoolapp" is written in Python, it builds
>>> again non-modular Python, currently 3.8, it requires "python(abi) =
>>> 3.8" on runtime.
>>
>> Hmm, and what about an even simpler case:
>> "myswankyapp" is also written in Python, and is packaged as a module.
>> Python is rebuilt in a side tag, then the module blocks the upgrade.
>> What is supposed to happen in that case?
>
> The module is rebuilt after the side tag is merged. Al least that is
> what I think happened with avocado when we upgraded to Python 3.8.
Automatically? And if the build fails?
I don't think so. I think somebody actually went and did it. CCing Merlin.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
12:17 p.m.
On Thu, Nov 14, 2019 at 12:09 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 14. 11. 19 18:57, Zbigniew Jędrzejewski-Szmek wrote:
> On Thu, Nov 14, 2019 at 06:43:42PM +0100, Miro Hrončok wrote:
>> On 14. 11. 19 18:36, Zbigniew Jędrzejewski-Szmek wrote:
>>> On Thu, Nov 14, 2019 at 06:08:52PM +0100, Miro Hrončok wrote:
>>>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>>>>
https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>>>
>>>>> Enable module default streams in the buildroot repository for
modular
>>>>> and non-modular RPMs.
>>>>>
>>>>> == Summary ==
>>>>> This Change (colloquially referred to as "Ursa Prime")
enables the
>>>>> Koji build-system to include the RPM artifacts provided by module
>>>>> default streams in the buildroot when building non-modular (or
>>>>> "traditional") RPMs.
>>>>
>>>> I have one more technical concern.
>>>>
>>>> Suppose a packager decides to package the "mycoolapp" software
as a
>>>> non-modular package. "mycoolapp" is written in Python, it
builds
>>>> again non-modular Python, currently 3.8, it requires "python(abi)
=
>>>> 3.8" on runtime.
>>>
>>> Hmm, and what about an even simpler case:
>>> "myswankyapp" is also written in Python, and is packaged as a
module.
>>> Python is rebuilt in a side tag, then the module blocks the upgrade.
>>> What is supposed to happen in that case?
>>
>> The module is rebuilt after the side tag is merged. Al least that is
>> what I think happened with avocado when we upgraded to Python 3.8.
>
> Automatically? And if the build fails?
I don't think so. I think somebody actually went and did it. CCing Merlin.
At least for avocado, yes, I manually rebuilt it.
Merlin
12:39 p.m.
On Thu, Nov 14, 2019 at 12:12 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 09. 10. 19 22:46, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>
> Enable module default streams in the buildroot repository for modular
> and non-modular RPMs.
>
> == Summary ==
> This Change (colloquially referred to as "Ursa Prime") enables the
> Koji build-system to include the RPM artifacts provided by module
> default streams in the buildroot when building non-modular (or
> "traditional") RPMs.
I have one more technical concern.
Suppose a packager decides to package the "mycoolapp" software as a
non-modular
package. "mycoolapp" is written in Python, it builds again non-modular Python,
currently 3.8, it requires "python(abi) = 3.8" on runtime.
The packager decides to use avocado in %check. Avocado comes from a module, it
requires "python(abi) = 3.8" as well, because the modular package was built
with
Python 3.8. Avocado is in the bulidroot, so everything works.
The Python maintainers (that includes me) decide to update to Python 3.9 in
Fedora 33. They request a side tag to do that. They update the python3 to 3.9
and they mass rebuild all non-modular Python packages in it. "mycoolapp"
cannot
resolve build dependencies because avocado requires "python(abi) = 3.8".
The Python maintainers need to detect this and figure out what happened.
Then, the Python maintainers need to either:
1. Exclude "mycoolapp" from the rebuild. That is possible until dozens of
other packages require "mycoolapp".
2. Ask the avocado maintainers to rebuild their module in the side tag and ask
releng to add the side-tag-built module into the side-tag buildroot (if it is
even possible).
3. Modify the spec of "mycoolapp" to temporarily disable %check and loose
the
avocado dependency.
Or is there some other way?
Does Python actually break ABI on minor releases? So a full rebuild is
required for that case?
What we would probably need to do is tag the python39 build into the
modular buildroot for the appropriate platform and then rebuild
avocado while that was in the buildroot.
Another option would be to create a `python3` module stream for each
minor release and make one of them the default stream for the side-tag
so the non-modular versions could be built against that. Then the
avocado module would set its dependencies to be:
```
buildrequires:
platform: [ ]
python3: [ ]
```
(Read: build for all streams of python for all available platforms).
Then whenever the default stream of python3 changes over in Rawhide,
avocado would follow because it was already built for the next
version.
12:42 p.m.
On Thu, Nov 14, 2019 at 1:40 PM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Thu, Nov 14, 2019 at 12:12 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>
> On 09. 10. 19 22:46, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> >
> > Enable module default streams in the buildroot repository for modular
> > and non-modular RPMs.
> >
> > == Summary ==
> > This Change (colloquially referred to as "Ursa Prime") enables the
> > Koji build-system to include the RPM artifacts provided by module
> > default streams in the buildroot when building non-modular (or
> > "traditional") RPMs.
>
> I have one more technical concern.
>
> Suppose a packager decides to package the "mycoolapp" software as a
non-modular
> package. "mycoolapp" is written in Python, it builds again non-modular
Python,
> currently 3.8, it requires "python(abi) = 3.8" on runtime.
>
> The packager decides to use avocado in %check. Avocado comes from a module, it
> requires "python(abi) = 3.8" as well, because the modular package was
built with
> Python 3.8. Avocado is in the bulidroot, so everything works.
>
> The Python maintainers (that includes me) decide to update to Python 3.9 in
> Fedora 33. They request a side tag to do that. They update the python3 to 3.9
> and they mass rebuild all non-modular Python packages in it. "mycoolapp"
cannot
> resolve build dependencies because avocado requires "python(abi) = 3.8".
>
> The Python maintainers need to detect this and figure out what happened.
> Then, the Python maintainers need to either:
>
> 1. Exclude "mycoolapp" from the rebuild. That is possible until dozens
of
> other packages require "mycoolapp".
>
> 2. Ask the avocado maintainers to rebuild their module in the side tag and ask
> releng to add the side-tag-built module into the side-tag buildroot (if it is
> even possible).
>
> 3. Modify the spec of "mycoolapp" to temporarily disable %check and
loose the
> avocado dependency.
>
> Or is there some other way?
>
Does Python actually break ABI on minor releases? So a full rebuild is
required for that case?
It is, yes. The Python ABI is versioned as MAJOR.MINOR.
--
真実はいつも一つ！/ Always, there's only one truth!
12:53 p.m.
On Thu, Nov 14, 2019 at 7:48 PM Stephen Gallagher <sgallagh(a)redhat.com> wrote:
On Thu, Nov 14, 2019 at 12:12 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>
> On 09. 10. 19 22:46, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
> >
> > Enable module default streams in the buildroot repository for modular
> > and non-modular RPMs.
> >
> > == Summary ==
> > This Change (colloquially referred to as "Ursa Prime") enables the
> > Koji build-system to include the RPM artifacts provided by module
> > default streams in the buildroot when building non-modular (or
> > "traditional") RPMs.
>
> I have one more technical concern.
>
> Suppose a packager decides to package the "mycoolapp" software as a
non-modular
> package. "mycoolapp" is written in Python, it builds again non-modular
Python,
> currently 3.8, it requires "python(abi) = 3.8" on runtime.
>
> The packager decides to use avocado in %check. Avocado comes from a module, it
> requires "python(abi) = 3.8" as well, because the modular package was
built with
> Python 3.8. Avocado is in the bulidroot, so everything works.
>
> The Python maintainers (that includes me) decide to update to Python 3.9 in
> Fedora 33. They request a side tag to do that. They update the python3 to 3.9
> and they mass rebuild all non-modular Python packages in it. "mycoolapp"
cannot
> resolve build dependencies because avocado requires "python(abi) = 3.8".
>
> The Python maintainers need to detect this and figure out what happened.
> Then, the Python maintainers need to either:
>
> 1. Exclude "mycoolapp" from the rebuild. That is possible until dozens
of
> other packages require "mycoolapp".
>
> 2. Ask the avocado maintainers to rebuild their module in the side tag and ask
> releng to add the side-tag-built module into the side-tag buildroot (if it is
> even possible).
>
> 3. Modify the spec of "mycoolapp" to temporarily disable %check and
loose the
> avocado dependency.
>
> Or is there some other way?
>
Does Python actually break ABI on minor releases? So a full rebuild is
required for that case?
What we would probably need to do is tag the python39 build into the
modular buildroot for the appropriate platform and then rebuild
avocado while that was in the buildroot.
That would mean, each release - bunch of additional manual work.
Another option would be to create a `python3` module stream for each
minor release and make one of them the default stream for the side-tag
so the non-modular versions could be built against that. Then the
avocado module would set its dependencies to be:
```
buildrequires:
platform: [ ]
python3: [ ]
```
(Read: build for all streams of python for all available platforms).
Then whenever the default stream of python3 changes over in Rawhide,
avocado would follow because it was already built for the next
version.
This implies that Python maintainers want to deal with Modularity
which I believe is not the case.
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
1:08 p.m.
New subject: Fedora 32 System-Wide Change proposal: Modules in Non-Modular
Buildroot
On 14. 11. 19 19:39, Stephen Gallagher wrote:
On Thu, Nov 14, 2019 at 12:12 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> On 09. 10. 19 22:46, Ben Cotton wrote:
>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>
>> Enable module default streams in the buildroot repository for modular
>> and non-modular RPMs.
>>
>> == Summary ==
>> This Change (colloquially referred to as "Ursa Prime") enables the
>> Koji build-system to include the RPM artifacts provided by module
>> default streams in the buildroot when building non-modular (or
>> "traditional") RPMs.
>
> I have one more technical concern.
>
> Suppose a packager decides to package the "mycoolapp" software as a
non-modular
> package. "mycoolapp" is written in Python, it builds again non-modular
Python,
> currently 3.8, it requires "python(abi) = 3.8" on runtime.
>
> The packager decides to use avocado in %check. Avocado comes from a module, it
> requires "python(abi) = 3.8" as well, because the modular package was built
with
> Python 3.8. Avocado is in the bulidroot, so everything works.
>
> The Python maintainers (that includes me) decide to update to Python 3.9 in
> Fedora 33. They request a side tag to do that. They update the python3 to 3.9
> and they mass rebuild all non-modular Python packages in it. "mycoolapp"
cannot
> resolve build dependencies because avocado requires "python(abi) = 3.8".
>
> The Python maintainers need to detect this and figure out what happened.
> Then, the Python maintainers need to either:
>
> 1. Exclude "mycoolapp" from the rebuild. That is possible until dozens
of
> other packages require "mycoolapp".
>
> 2. Ask the avocado maintainers to rebuild their module in the side tag and ask
> releng to add the side-tag-built module into the side-tag buildroot (if it is
> even possible).
>
> 3. Modify the spec of "mycoolapp" to temporarily disable %check and
loose the
> avocado dependency.
>
> Or is there some other way?
>
Does Python actually break ABI on minor releases? So a full rebuild is
required for that case?
Yes, the full rebuild is needed for various reasons. We've talked about this on
IRC, but to make this more transparent to others, I will repeat it here. The
main reasons include that "3.8" is in the paths (such as /usr/lib/python3.8) and
that the bytecode is changed. There are other reasons that may or may not be
relevant to avocade.
What we would probably need to do is tag the python39 build into the
modular buildroot for the appropriate platform and then rebuild
avocado while that was in the buildroot.
This would affect all modules that need to be rebuilt before the python39
side-tag is merged, right?
If I understand that right, it would also make the avocado module
non-installable for regular rawhide buidlroot before the python39 side tag is
merged.
One of the purposes that we do this in the side tag is to avoid disrupting
rawhide while we are mid-upgrade.
Another option would be to create a `python3` module stream for each
minor release and make one of them the default stream for the side-tag
so the non-modular versions could be built against that.
Modular stream of what module?
Then the
avocado module would set its dependencies to be:
```
buildrequires:
platform: [ ]
python3: [ ]
```
(Read: build for all streams of python for all available platforms).
Then whenever the default stream of python3 changes over in Rawhide,
avocado would follow because it was already built for the next
version.
Default stream of python3? python3 is not a module. And even if I wanted it to
be (I don't), it would be a very hard thing to do.
See one failed attempt at https://fedoraproject.org/wiki/Changes/System_Python
See a very nasty hack in RHEL 8.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
3:28 p.m.
On Thu, Nov 14, 2019 at 2:10 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 14. 11. 19 19:39, Stephen Gallagher wrote:
> On Thu, Nov 14, 2019 at 12:12 PM Miro Hrončok <mhroncok(a)redhat.com> wrote:
>>
>> On 09. 10. 19 22:46, Ben Cotton wrote:
>>> https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
>>>
>>> Enable module default streams in the buildroot repository for modular
>>> and non-modular RPMs.
>>>
>>> == Summary ==
>>> This Change (colloquially referred to as "Ursa Prime") enables
the
>>> Koji build-system to include the RPM artifacts provided by module
>>> default streams in the buildroot when building non-modular (or
>>> "traditional") RPMs.
>>
>> I have one more technical concern.
>>
>> Suppose a packager decides to package the "mycoolapp" software as a
non-modular
>> package. "mycoolapp" is written in Python, it builds again non-modular
Python,
>> currently 3.8, it requires "python(abi) = 3.8" on runtime.
>>
>> The packager decides to use avocado in %check. Avocado comes from a module, it
>> requires "python(abi) = 3.8" as well, because the modular package was
built with
>> Python 3.8. Avocado is in the bulidroot, so everything works.
>>
>> The Python maintainers (that includes me) decide to update to Python 3.9 in
>> Fedora 33. They request a side tag to do that. They update the python3 to 3.9
>> and they mass rebuild all non-modular Python packages in it.
"mycoolapp" cannot
>> resolve build dependencies because avocado requires "python(abi) =
3.8".
>>
>> The Python maintainers need to detect this and figure out what happened.
>> Then, the Python maintainers need to either:
>>
>> 1. Exclude "mycoolapp" from the rebuild. That is possible until
dozens of
>> other packages require "mycoolapp".
>>
>> 2. Ask the avocado maintainers to rebuild their module in the side tag and
ask
>> releng to add the side-tag-built module into the side-tag buildroot (if it is
>> even possible).
>>
>> 3. Modify the spec of "mycoolapp" to temporarily disable %check and
loose the
>> avocado dependency.
>>
>> Or is there some other way?
>>
>
> Does Python actually break ABI on minor releases? So a full rebuild is
> required for that case?
Yes, the full rebuild is needed for various reasons. We've talked about this on
IRC, but to make this more transparent to others, I will repeat it here. The
main reasons include that "3.8" is in the paths (such as /usr/lib/python3.8)
and
that the bytecode is changed. There are other reasons that may or may not be
relevant to avocade.
> What we would probably need to do is tag the python39 build into the
> modular buildroot for the appropriate platform and then rebuild
> avocado while that was in the buildroot.
This would affect all modules that need to be rebuilt before the python39
side-tag is merged, right?
If I understand that right, it would also make the avocado module
non-installable for regular rawhide buidlroot before the python39 side tag is
merged.
One of the purposes that we do this in the side tag is to avoid disrupting
rawhide while we are mid-upgrade.
> Another option would be to create a `python3` module stream for each
> minor release and make one of them the default stream for the side-tag
> so the non-modular versions could be built against that.
Modular stream of what module?
> Then the
> avocado module would set its dependencies to be:
> ```
> buildrequires:
> platform: [ ]
> python3: [ ]
> ```
> (Read: build for all streams of python for all available platforms).
> Then whenever the default stream of python3 changes over in Rawhide,
> avocado would follow because it was already built for the next
> version.
Default stream of python3? python3 is not a module. And even if I wanted it to
be (I don't), it would be a very hard thing to do.
So, what I meant here (as I said in another mail) was actually to
create a python3 module with streams for each release version whose
purpose is *not* to carry the actual python content. That should stay
with the non-modular RPMs. It would instead carry just a (new)
python-binaries package that would behave halfway between
`alternatives` and a metapackage, in that it would depend on the
appropriate non-modular packages and drop symlinks for the correct
binary versions into place. Then we could use those module streams as
source for doing stream expansion for modules that depend on Python.
Then it becomes easy to rebuild things like Avocado, since it will see
the new 3.9 stream and add that to the matrix.
Am I explaining that well?
1623
days inactive
1659
days old
109 comments
29 participants
participants (29)
-
Adam Williamson -
Aleksandar Kurtakov -
Alexander Scheel -
Ben Cotton -
Christopher Engelhard -
Fabio Valentini -
Igor Gnatenko -
Jakub Cajka -
Joe Orton -
John M. Harris Jr -
Josh Boyer -
Ken Dreyer -
Kevin Kofler -
Lukas Ruzicka -
Matthew Miller -
Merlin Mathesius -
Miro Hrončok -
Neal Gompa -
Nico Kadel-Garcia -
Orion Poplawski -
Pavel Raiskup -
Pierre-Yves Chibon -
Randy Barlow -
Robbie Harwood -
Stephen Gallagher -
Stephen John Smoogen -
Tomasz Torcz -
Vitaly Zaitsev -
Zbigniew Jędrzejewski-Szmek