RHEL 9 and modularity

[Josh Boyer]

Hello Fedora Community!

I am a long-time Fedora Community member, and may be familiar to many
through previous FESCo or devel list discussions and passionate
debates.  However I write to you today with a different community hat
on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
organization has been following the modularity discussions within
Fedora, particularly around ELN, and often the question of what plans
we have for modularity in RHEL 9 has come up.  Our Fedora Project Lead
and a number of FESCo members have reached out and asked if we can
provide some perspective here, and I am both happy and excited to have
that opportunity.

As the Fedora Council has pointed out [1], we certainly acknowledge
there are improvements to be made and have a team already working on
them.  They recently outlined their plans in conjunction with our
Product Management team in a Fedora Council call as well [2].  We’re
continuing to invest time and effort in this packaging solution and
are confident that the team can deliver against their plan.  It is
somewhat of a new experience for all of us when Red Hat is direct with
our product intentions, but we discussed the larger gaps we see with
usage in RHEL and are putting our efforts towards solving those gaps
with this plan.

Modularity is important to RHEL and those efforts are already
underway.  We will be leveraging modularity in RHEL 9 where it most
makes sense.  This is primarily centered around our Application
Streams concept, which has been well received by our customer base.
Providing a consistent but improved experience is the base
requirement, which allows us to have continuity from RHEL 8 to RHEL 9
and lowers the hurdle for our customers when upgrading from one major
version to another.

It is always good to push the boundaries and search for better ideas
and improvements, and that is part of what makes Fedora great.  We are
doing this in the context of the RHEL 9 release as well, so our near
term timeline and requirements mean we are working on evolving
modularity, not a revolution or a replacement.  We are excited by ELN,
as it presents a possible space to allow those that want to continue
to iterate on modules a place to do so without necessarily impacting
the broader Fedora distribution in its entirety.  It is my personal
hope that we can use that opportunity to improve modules and
modularity in the open source, Fedora-first way we’d prefer.  Our near
term effort to improve the existing modularity implementation ahead of
RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
than in closed product development.  Longer term, we are open to
contributing to a better replacement that meets many of the same
goals.  This is what makes our distribution ecosystem work well, and
not having that upstream lessens the value we all get from such
experimentation in the open.

Hopefully that provides some context and helps FESCo and the wider
community understand where Red Hat is headed with modularity on the
Enterprise side.

josh

[1] https://communityblog.fedoraproject.org/fedora-council-and-the-future-of-...
[2] https://bluejeans.com/s/W_P0D

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>  Hello Fedora Community! 
Hi Josh,

>  I am a long-time Fedora Community member, and may be familiar to
> many
>  through previous FESCo or devel list discussions and passionate
>  debates.  However I write to you today with a different community hat
>  on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  organization has been following the modularity discussions within
>  Fedora, particularly around ELN, and often the question of what plans
>  we have for modularity in RHEL 9 has come up.  Our Fedora Project
>  Lead
>  and a number of FESCo members have reached out and asked if we can
>  provide some perspective here, and I am both happy and excited to
>  have
>  that opportunity.
>  
>  As the Fedora Council has pointed out [1], we certainly acknowledge
>  there are improvements to be made and have a team already working on
>  them.  They recently outlined their plans in conjunction with our
>  Product Management team in a Fedora Council call as well [2].  We’re
>  continuing to invest time and effort in this packaging solution and
>  are confident that the team can deliver against their plan.  It is
>  somewhat of a new experience for all of us when Red Hat is direct
>  with
>  our product intentions, but we discussed the larger gaps we see with
>  usage in RHEL and are putting our efforts towards solving those gaps
>  with this plan.
>  
>  Modularity is important to RHEL and those efforts are already
>  underway.  We will be leveraging modularity in RHEL 9 where it most
>  makes sense.  This is primarily centered around our Application
>  Streams concept, which has been well received by our customer base.
>  Providing a consistent but improved experience is the base
>  requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  and lowers the hurdle for our customers when upgrading from one major
>  version to another. 
It is nice to hear that it is helping to solve problems in RHEL (even
though I've heard many people saying that it is nightmare now). Is
there a list of requirements that you have so that we could potentially
develop something that would be useful to Fedora same as for RHEL 10+?

>  It is always good to push the boundaries and search for better ideas
>  and improvements, and that is part of what makes Fedora great.  We
>  are
>  doing this in the context of the RHEL 9 release as well, so our near
>  term timeline and requirements mean we are working on evolving
>  modularity, not a revolution or a replacement.  We are excited by
>  ELN,
>  as it presents a possible space to allow those that want to continue
>  to iterate on modules a place to do so without necessarily impacting
>  the broader Fedora distribution in its entirety.  It is my personal
>  hope that we can use that opportunity to improve modules and
>  modularity in the open source, Fedora-first way we’d prefer.  Our
>  near
>  term effort to improve the existing modularity implementation ahead
>  of
>  RHEL 9 needs to occur, and we’d like to do that work in Fedora,
>  rather
>  than in closed product development.  Longer term, we are open to
>  contributing to a better replacement that meets many of the same
>  goals.  This is what makes our distribution ecosystem work well, and
>  not having that upstream lessens the value we all get from such
>  experimentation in the open. 
While I support you that we should do it in Fedora, does this
essentially mean that this technology is useful only for RHEL and you
do not plan to develop it *for Fedora*, but rather *for RHEL in
Fedora*?

>  Hopefully that provides some context and helps FESCo and the wider
>  community understand where Red Hat is headed with modularity on the
>  Enterprise side. 
Sadly no. It helps to understand your plans, however it does not help
to understand the reasons behind, whether you can't change UX in the
RHEL 9, or you think that technology is good enough for your use-cases
or any other reasons.

Basically this email just says "We decided for Modularity in RHEL 9 and
we would like to do it in Fedora Infrastructure first".

>  josh
>  
>  [1]   
>  https://communityblog.fedoraproject.org/fedora-council-and-the-future-of-...
>  [2] https://bluejeans.com/s/W_P0D
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:   
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:   
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:   
>  https://lists.fedoraproject.org/archives/list/[e-mail] 
- -- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7rZngACgkQEV1auJxc
> Hh5puBAAoyT6t5tBZ+czWOt4WiS3sE7Zeuc8HsMH8B9VbB5P4GK+lI1u7PIT1UfI
> RiCGEOqtG0lhDXVTb/RoW2aZWKefnPK1irLwJJc3exR/xd4a6OLerO4IFAnJR50P
> ZMM/0dKvoaO74cf/590XC1sOVxesRMKEBf2L0IhFyW5hGkDmLPVRpVIKDlS2c/FN
> aCMOBmVerCOSMlxOa4bZwAEoQC6QeBLrL9dyGP5ygoI2dgiEMteUEShbgnzfYnJS
> FhnCbovftHBMXyVxtKfhXa6RCFtUpQNvh4dT2AKwO8A+EVfGuq9TUdtTe36ckWkA
> 9zDpr2e9VJYBw/3euFbQkS7POK7NREGowubzEfVAr/Ar94qZjyHgmCkmUCxQhQ6u
> LjRsOwmml3S6YH6DF0oF1f9Fe07wgqR0+dwr5ygFKi4CutOQbT1YujpxSvmhJE2g
> at+XX0aiZD3qOPkVVpOWjSJiyCIf+yM1fs5j23eYmWFl2bnBpkkZJqejb9Gc1BiW
> G70FdIk8OqPLzkYbS/RxoZP5iK9lTI2EEMy3ysbgiFnWYk1+blDhWkBSsvLHmV9R
> aWCKb9wD4aZOp87bs1mgHtcwL+n/1ygfF1bbBzVyRdYjofpP6zbWEKSnh1VW0ybn
> ASIVXX06TwpOHNFbqHcdqd1m2B14aLpKAzBXexGJTCIFlscOyTw=
> =S2Dk
> -----END PGP SIGNATURE-----

[Josh Boyer]

On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
<ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA512
>
>  On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>  > Hello Fedora Community!
>
>  Hi Josh,
>
>  > I am a long-time Fedora Community member, and may be familiar to many
>  > through previous FESCo or devel list discussions and passionate
>  > debates.  However I write to you today with a different community hat
>  > on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  > organization has been following the modularity discussions within
>  > Fedora, particularly around ELN, and often the question of what plans
>  > we have for modularity in RHEL 9 has come up.  Our Fedora Project
>  > Lead
>  > and a number of FESCo members have reached out and asked if we can
>  > provide some perspective here, and I am both happy and excited to
>  > have
>  > that opportunity.
>  >
>  > As the Fedora Council has pointed out [1], we certainly acknowledge
>  > there are improvements to be made and have a team already working on
>  > them.  They recently outlined their plans in conjunction with our
>  > Product Management team in a Fedora Council call as well [2].  We’re
>  > continuing to invest time and effort in this packaging solution and
>  > are confident that the team can deliver against their plan.  It is
>  > somewhat of a new experience for all of us when Red Hat is direct
>  > with
>  > our product intentions, but we discussed the larger gaps we see with
>  > usage in RHEL and are putting our efforts towards solving those gaps
>  > with this plan.
>  >
>  > Modularity is important to RHEL and those efforts are already
>  > underway.  We will be leveraging modularity in RHEL 9 where it most
>  > makes sense.  This is primarily centered around our Application
>  > Streams concept, which has been well received by our customer base.
>  > Providing a consistent but improved experience is the base
>  > requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  > and lowers the hurdle for our customers when upgrading from one major
>  > version to another.
>
>  It is nice to hear that it is helping to solve problems in RHEL (even
>  though I've heard many people saying that it is nightmare now). Is
>  there a list of requirements that you have so that we could potentially
>  develop something that would be useful to Fedora same as for RHEL 10+? 
The dnf team is working to gather those internally.  RHEL 10+ is still
~5 years away, and while we're working hard to develop our product
roadmap, that's still far enough off that we haven't put much down in
terms of requirements :)

>  > It is always good to push the boundaries and search for better
> ideas
>  > and improvements, and that is part of what makes Fedora great.  We
>  > are
>  > doing this in the context of the RHEL 9 release as well, so our near
>  > term timeline and requirements mean we are working on evolving
>  > modularity, not a revolution or a replacement.  We are excited by
>  > ELN,
>  > as it presents a possible space to allow those that want to continue
>  > to iterate on modules a place to do so without necessarily impacting
>  > the broader Fedora distribution in its entirety.  It is my personal
>  > hope that we can use that opportunity to improve modules and
>  > modularity in the open source, Fedora-first way we’d prefer.  Our
>  > near
>  > term effort to improve the existing modularity implementation ahead
>  > of
>  > RHEL 9 needs to occur, and we’d like to do that work in Fedora,
>  > rather
>  > than in closed product development.  Longer term, we are open to
>  > contributing to a better replacement that meets many of the same
>  > goals.  This is what makes our distribution ecosystem work well, and
>  > not having that upstream lessens the value we all get from such
>  > experimentation in the open.
>
>  While I support you that we should do it in Fedora, does this
>  essentially mean that this technology is useful only for RHEL and you
>  do not plan to develop it *for Fedora*, but rather *for RHEL in
>  Fedora*? 
I wouldn't say that, nor do I personally think that.  I think there is
value for Fedora as well.  Matthew Miller has often given a common
example of having two streams of software that can build across RHEL,
Fedora, CentOS, etc.  This is value for end user consumption, and
having older software available in Fedora is a usecase modularity can
help address.  However, I would prefer to avoid discussing value to
Fedora in this thread.  There are so many other threads debating that
already :)

>  > Hopefully that provides some context and helps FESCo and the
> wider
>  > community understand where Red Hat is headed with modularity on the
>  > Enterprise side.
>
>  Sadly no. It helps to understand your plans, however it does not help
>  to understand the reasons behind, whether you can't change UX in the
>  RHEL 9, or you think that technology is good enough for your use-cases
>  or any other reasons. 
The base requirement is that the UX remain largely the same.  As I
said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
commonality so that our customers are not forced to learn something
entirely different to adopt RHEL 9.  Improvements in the underlying
functionality are of course welcome and planned, but we are not going
to do something like replace modules with a different artifact type,
or add separate discrete repos per Application Stream, etc.

>  Basically this email just says "We decided for Modularity in
> RHEL 9 and
>  we would like to do it in Fedora Infrastructure first". 
Mostly, yes.  We were told there was ambiguity on whether modularity
would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
get ahead of that.

josh

[Miro Hrončok]

On 18. 06. 20 15:24, Josh Boyer wrote:
> > Basically this email just says "We decided for Modularity in
> RHEL 9 and
> > we would like to do it in Fedora Infrastructure first".
>  Mostly, yes.  We were told there was ambiguity on whether modularity
>  would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  get ahead of that. 
I wonder where did this ambiguity came from. I always considered this as a very 
well known fact, but thanks for saying it explicitly, Josh.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Thu, Jun 18, 2020 at 9:29 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 18. 06. 20 15:24, Josh Boyer wrote:
>  >> Basically this email just says "We decided for Modularity in RHEL 9 and
>  >> we would like to do it in Fedora Infrastructure first".
>  > Mostly, yes.  We were told there was ambiguity on whether modularity
>  > would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  > get ahead of that.
>
>  I wonder where did this ambiguity came from. I always considered this as a very
>  well known fact, but thanks for saying it explicitly, Josh. 
I have found that it doesn't matter how much I think something is well
understood or clear, there is always someone that doesn't know or got
a different variation of the story.

¯\_(ツ)_/¯

In any case, happy to make it explicit.

josh

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, 2020-06-18 at 09:24 -0400, Josh Boyer wrote:
>  On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
>  <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > 
>  > -----BEGIN PGP SIGNED MESSAGE-----
>  > Hash: SHA512
>  > 
>  > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>  > > Hello Fedora Community!
>  > 
>  > Hi Josh,
>  > 
>  > > I am a long-time Fedora Community member, and may be familiar to
>  > > many
>  > > through previous FESCo or devel list discussions and passionate
>  > > debates.  However I write to you today with a different community
>  > > hat
>  > > on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  > > organization has been following the modularity discussions within
>  > > Fedora, particularly around ELN, and often the question of what
>  > > plans
>  > > we have for modularity in RHEL 9 has come up.  Our Fedora Project
>  > > Lead
>  > > and a number of FESCo members have reached out and asked if we
>  > > can
>  > > provide some perspective here, and I am both happy and excited to
>  > > have
>  > > that opportunity.
>  > > 
>  > > As the Fedora Council has pointed out [1], we certainly
>  > > acknowledge
>  > > there are improvements to be made and have a team already working
>  > > on
>  > > them.  They recently outlined their plans in conjunction with our
>  > > Product Management team in a Fedora Council call as well [2]. 
>  > > We’re
>  > > continuing to invest time and effort in this packaging solution
>  > > and
>  > > are confident that the team can deliver against their plan.  It
>  > > is
>  > > somewhat of a new experience for all of us when Red Hat is direct
>  > > with
>  > > our product intentions, but we discussed the larger gaps we see
>  > > with
>  > > usage in RHEL and are putting our efforts towards solving those
>  > > gaps
>  > > with this plan.
>  > > 
>  > > Modularity is important to RHEL and those efforts are already
>  > > underway.  We will be leveraging modularity in RHEL 9 where it
>  > > most
>  > > makes sense.  This is primarily centered around our Application
>  > > Streams concept, which has been well received by our customer
>  > > base.
>  > > Providing a consistent but improved experience is the base
>  > > requirement, which allows us to have continuity from RHEL 8 to
>  > > RHEL 9
>  > > and lowers the hurdle for our customers when upgrading from one
>  > > major
>  > > version to another.
>  > 
>  > It is nice to hear that it is helping to solve problems in RHEL
>  > (even
>  > though I've heard many people saying that it is nightmare now). Is
>  > there a list of requirements that you have so that we could
>  > potentially
>  > develop something that would be useful to Fedora same as for RHEL
>  > 10+?
>  
>  The dnf team is working to gather those internally.  RHEL 10+ is
>  still
>  ~5 years away, and while we're working hard to develop our product
>  roadmap, that's still far enough off that we haven't put much down in
>  terms of requirements :)
>  
>  > > It is always good to push the boundaries and search for better
>  > > ideas
>  > > and improvements, and that is part of what makes Fedora great. 
>  > > We
>  > > are
>  > > doing this in the context of the RHEL 9 release as well, so our
>  > > near
>  > > term timeline and requirements mean we are working on evolving
>  > > modularity, not a revolution or a replacement.  We are excited by
>  > > ELN,
>  > > as it presents a possible space to allow those that want to
>  > > continue
>  > > to iterate on modules a place to do so without necessarily
>  > > impacting
>  > > the broader Fedora distribution in its entirety.  It is my
>  > > personal
>  > > hope that we can use that opportunity to improve modules and
>  > > modularity in the open source, Fedora-first way we’d prefer.  Our
>  > > near
>  > > term effort to improve the existing modularity implementation
>  > > ahead
>  > > of
>  > > RHEL 9 needs to occur, and we’d like to do that work in Fedora,
>  > > rather
>  > > than in closed product development.  Longer term, we are open to
>  > > contributing to a better replacement that meets many of the same
>  > > goals.  This is what makes our distribution ecosystem work well,
>  > > and
>  > > not having that upstream lessens the value we all get from such
>  > > experimentation in the open.
>  > 
>  > While I support you that we should do it in Fedora, does this
>  > essentially mean that this technology is useful only for RHEL and
>  > you
>  > do not plan to develop it *for Fedora*, but rather *for RHEL in
>  > Fedora*?
>  
>  I wouldn't say that, nor do I personally think that.  I think there
>  is
>  value for Fedora as well.  Matthew Miller has often given a common
>  example of having two streams of software that can build across RHEL,
>  Fedora, CentOS, etc.  This is value for end user consumption, and
>  having older software available in Fedora is a usecase modularity can
>  help address.  However, I would prefer to avoid discussing value to
>  Fedora in this thread.  There are so many other threads debating that
>  already :) 
Let's agree to disagree :)

>  > > Hopefully that provides some context and helps FESCo and
> the
>  > > wider
>  > > community understand where Red Hat is headed with modularity on
>  > > the
>  > > Enterprise side.
>  > 
>  > Sadly no. It helps to understand your plans, however it does not
>  > help
>  > to understand the reasons behind, whether you can't change UX in
>  > the
>  > RHEL 9, or you think that technology is good enough for your use-
>  > cases
>  > or any other reasons.
>  
>  The base requirement is that the UX remain largely the same.  As I
>  said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  commonality so that our customers are not forced to learn something
>  entirely different to adopt RHEL 9.  Improvements in the underlying
>  functionality are of course welcome and planned, but we are not going
>  to do something like replace modules with a different artifact type,
>  or add separate discrete repos per Application Stream, etc.
>  
>  > Basically this email just says "We decided for Modularity in RHEL 9
>  > and
>  > we would like to do it in Fedora Infrastructure first".
>  
>  Mostly, yes.  We were told there was ambiguity on whether modularity
>  would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  get ahead of that. 
Sure, I think this is answered quite clearly here. Thanks.

>  josh
>   
- -- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7rbDUACgkQEV1auJxc
> Hh7+ORAAg1jmZZvcCmqeW6YCi44eFnXERea6Z2QTXSVmZYINFVoATQeuzDFElEOF
> ETi72MX7bAw3KwjMJU++vlOKof8WpBHV2fEC1ldUeq2vk+FrCvBKEyiGrXmtkXnu
> hRJ7BzqBBfcPyZzplpXBrXqFapr0vLYwvT9RRRThHWMvLaGneMUF/q4kPgKT+Pm3
> +K8Vo5hDL8fCjw3XdnYNLaAkGGeR4XxJ13D/i+9JmGPd41kv2/eVrui5N/FTRC1L
> a+cJXJSQiBE0iO7oe8kDtKvSmYUTSG0sZMRR15MastdxbRyYYDG6TVL99XYG1PuA
> iGQXIJGIUdb81ttdPD/6gsefk/StSZKDL+CSLfCoQuqEdqBHhjha9W6WQVJxClMk
> iSpVHGZpUF9mtsPqKQ3kV4q77w8M0Qz1GbqpLP2H+iVtzVtqQfRJ3QdD8PZDkuBu
> +e40oTC5AOopo5P8/p/VD1um2G4K7d7GmspsOOGoNOVDUwEV3/KjXbW5CCeSOC1v
> soEH41S8idQygVxI02+BrJTdNfJUrz+o6HaqmY4TfXZoXRe5jBX+9gj0JofFlis0
> cgTuixHAljPT2btC7IP4xrdTr3WdCnZJAjaqTANYZclv/TGZeGTKap92pIV405nu
> 8cxjkd9mCHsL6mUApyIXxaro2yJ5y7ShRmWl5dO9+P1qeJjYqtY=
> =OmeX
> -----END PGP SIGNATURE-----

[John M. Harris Jr]

On Thursday, June 18, 2020 6:24:46 AM MST Josh Boyer wrote:
>  On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
>  <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  
>  >
>  >
>  > -----BEGIN PGP SIGNED MESSAGE-----
>  > Hash: SHA512
>  >
>  >
>  >
>  > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>  > 
>  > > Hello Fedora Community!
>  >
>  >
>  >
>  > Hi Josh,
>  >
>  >
>  >
>  > > I am a long-time Fedora Community member, and may be familiar to many
>  > > through previous FESCo or devel list discussions and passionate
>  > > debates.  However I write to you today with a different community hat
>  > > on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  > > organization has been following the modularity discussions within
>  > > Fedora, particularly around ELN, and often the question of what plans
>  > > we have for modularity in RHEL 9 has come up.  Our Fedora Project
>  > > Lead
>  > > and a number of FESCo members have reached out and asked if we can
>  > > provide some perspective here, and I am both happy and excited to
>  > > have
>  > > that opportunity.
>  > >
>  > >
>  > >
>  > > As the Fedora Council has pointed out [1], we certainly acknowledge
>  > > there are improvements to be made and have a team already working on
>  > > them.  They recently outlined their plans in conjunction with our
>  > > Product Management team in a Fedora Council call as well [2].  We’re
>  > > continuing to invest time and effort in this packaging solution and
>  > > are confident that the team can deliver against their plan.  It is
>  > > somewhat of a new experience for all of us when Red Hat is direct
>  > > with
>  > > our product intentions, but we discussed the larger gaps we see with
>  > > usage in RHEL and are putting our efforts towards solving those gaps
>  > > with this plan.
>  > >
>  > >
>  > >
>  > > Modularity is important to RHEL and those efforts are already
>  > > underway.  We will be leveraging modularity in RHEL 9 where it most
>  > > makes sense.  This is primarily centered around our Application
>  > > Streams concept, which has been well received by our customer base.
>  > > Providing a consistent but improved experience is the base
>  > > requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  > > and lowers the hurdle for our customers when upgrading from one major
>  > > version to another.
>  >
>  >
>  >
>  > It is nice to hear that it is helping to solve problems in RHEL (even
>  > though I've heard many people saying that it is nightmare now). Is
>  > there a list of requirements that you have so that we could potentially
>  > develop something that would be useful to Fedora same as for RHEL 10+?
>  
>  
>  The dnf team is working to gather those internally.  RHEL 10+ is still
>  ~5 years away, and while we're working hard to develop our product
>  roadmap, that's still far enough off that we haven't put much down in
>  terms of requirements :)
>  
>  
>  > > It is always good to push the boundaries and search for better ideas
>  > > and improvements, and that is part of what makes Fedora great.  We
>  > > are
>  > > doing this in the context of the RHEL 9 release as well, so our near
>  > > term timeline and requirements mean we are working on evolving
>  > > modularity, not a revolution or a replacement.  We are excited by
>  > > ELN,
>  > > as it presents a possible space to allow those that want to continue
>  > > to iterate on modules a place to do so without necessarily impacting
>  > > the broader Fedora distribution in its entirety.  It is my personal
>  > > hope that we can use that opportunity to improve modules and
>  > > modularity in the open source, Fedora-first way we’d prefer.  Our
>  > > near
>  > > term effort to improve the existing modularity implementation ahead
>  > > of
>  > > RHEL 9 needs to occur, and we’d like to do that work in Fedora,
>  > > rather
>  > > than in closed product development.  Longer term, we are open to
>  > > contributing to a better replacement that meets many of the same
>  > > goals.  This is what makes our distribution ecosystem work well, and
>  > > not having that upstream lessens the value we all get from such
>  > > experimentation in the open.
>  >
>  >
>  >
>  > While I support you that we should do it in Fedora, does this
>  > essentially mean that this technology is useful only for RHEL and you
>  > do not plan to develop it *for Fedora*, but rather *for RHEL in
>  > Fedora*?
>  
>  
>  I wouldn't say that, nor do I personally think that.  I think there is
>  value for Fedora as well.  Matthew Miller has often given a common
>  example of having two streams of software that can build across RHEL,
>  Fedora, CentOS, etc.  This is value for end user consumption, and
>  having older software available in Fedora is a usecase modularity can
>  help address.  However, I would prefer to avoid discussing value to
>  Fedora in this thread.  There are so many other threads debating that
>  already :)
>  
>  
>  > > Hopefully that provides some context and helps FESCo and the wider
>  > > community understand where Red Hat is headed with modularity on the
>  > > Enterprise side.
>  >
>  >
>  >
>  > Sadly no. It helps to understand your plans, however it does not help
>  > to understand the reasons behind, whether you can't change UX in the
>  > RHEL 9, or you think that technology is good enough for your use-cases
>  > or any other reasons.
>  
>  
>  The base requirement is that the UX remain largely the same.  As I
>  said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  commonality so that our customers are not forced to learn something
>  entirely different to adopt RHEL 9.  Improvements in the underlying
>  functionality are of course welcome and planned, but we are not going
>  to do something like replace modules with a different artifact type,
>  or add separate discrete repos per Application Stream, etc. 
Why is this a concern for RHEL9, where it wasn't for RHEL8? Moving to 
Modularity has certainly hurt RHEL7 migrations for exactly that reason, 
customers are forced to learn something entirely different to adopt RHEL8, and 
figure out undocumented issues with Modularity.

>  
>  > Basically this email just says "We decided for Modularity in RHEL 9 and
>  > we would like to do it in Fedora Infrastructure first".
>  
>  
>  Mostly, yes.  We were told there was ambiguity on whether modularity
>  would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  get ahead of that. 
That's unfortunate, and definitely isn't in line with what I've been told in 
response to emails asking me to move my RHEL6 and RHEL7 systems to RHEL8, 
where I responded that we would wait for the next product. I can see how that 
may be out of line with your views, but I can't say it was really expected in 
this way. Thank you for clarifying. Has a stable Enterprise Linux product been 
considered, like RHEL5,6,7, perhaps moving Modularity to its own product for 
the few that have a use for it?

-- 
John M. Harris, Jr.

[Josh Boyer]

On Thu, Jun 18, 2020 at 1:59 PM John M. Harris Jr <johnmh(a)splentity.com&gt; wrote:
>  On Thursday, June 18, 2020 6:24:46 AM MST Josh Boyer wrote: 
<snip>

>  > The base requirement is that the UX remain largely the same.  As
> I
>  > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  > commonality so that our customers are not forced to learn something
>  > entirely different to adopt RHEL 9.  Improvements in the underlying
>  > functionality are of course welcome and planned, but we are not going
>  > to do something like replace modules with a different artifact type,
>  > or add separate discrete repos per Application Stream, etc.
>
>  Why is this a concern for RHEL9, where it wasn't for RHEL8? Moving to
>  Modularity has certainly hurt RHEL7 migrations for exactly that reason,
>  customers are forced to learn something entirely different to adopt RHEL8, and
>  figure out undocumented issues with Modularity. 
Actually, it was a concern for RHEL 8 in many ways.  The introduction
of default module streams was a direct result of wanting to help
customers that are used to running 'yum install mariadb' still be able
to do that.  The fact that it is packaged in a module is transparent
for that usecase.  Customers wanting to use non-default Application
Streams will indeed need to learn some new commands and concepts, but
they still have some analogs to other technology we use in RHEL 7,
like SCLs, where newer content is accessible in different channels via
different tools.  By having modules be implemented in yum itself,
those concepts become more central and common to the distribution
overall.

As with any new major release, there is always a need to introduce new
features or technology that causes some disruption.  It is often the
only time we can do so in an Enterprise distribution.  We try to
balance that with ease of use and adoption, which are always a top
concern.  If you have issues with RHEL 8 and modularity, I would
encourage you to file a case in the Customer Portal.  Getting that
feedback helps ensure we're addressing customer concerns.

>  > > Basically this email just says "We decided for
> Modularity in RHEL 9 and
>  > > we would like to do it in Fedora Infrastructure first".
>  >
>  >
>  > Mostly, yes.  We were told there was ambiguity on whether modularity
>  > would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  > get ahead of that.
>
>  That's unfortunate, and definitely isn't in line with what I've been told in
>  response to emails asking me to move my RHEL6 and RHEL7 systems to RHEL8,
>  where I responded that we would wait for the next product. I can see how that
>  may be out of line with your views, but I can't say it was really expected in
>  this way. Thank you for clarifying. Has a stable Enterprise Linux product been
>  considered, like RHEL5,6,7, perhaps moving Modularity to its own product for
>  the few that have a use for it? 
If you have problems with the RHEL 8 packages and functionality they
offer, please follow up through the support channels you have as a
RHEL customer.  The distribution itself is quite stable, and in some
cases almost twice as performant as RHEL 7.  I'd like to keep this
thread focused on Fedora, ELN, and modules if we can.

josh

[John M. Harris Jr]

On Thursday, June 18, 2020 12:22:08 PM MST Josh Boyer wrote:
>  On Thu, Jun 18, 2020 at 1:59 PM John M. Harris Jr
> <johnmh(a)splentity.com&gt;
>  wrote:
>  >
>  >
>  > On Thursday, June 18, 2020 6:24:46 AM MST Josh Boyer wrote:
>  
>  <snip>
>  
>  
>  > > The base requirement is that the UX remain largely the same.  As I
>  > > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  > > commonality so that our customers are not forced to learn something
>  > > entirely different to adopt RHEL 9.  Improvements in the underlying
>  > > functionality are of course welcome and planned, but we are not going
>  > > to do something like replace modules with a different artifact type,
>  > > or add separate discrete repos per Application Stream, etc.
>  >
>  >
>  >
>  > Why is this a concern for RHEL9, where it wasn't for RHEL8? Moving to
>  > Modularity has certainly hurt RHEL7 migrations for exactly that reason,
>  > customers are forced to learn something entirely different to adopt RHEL8,
>  > and figure out undocumented issues with Modularity.
>  
>  
>  Actually, it was a concern for RHEL 8 in many ways.  The introduction
>  of default module streams was a direct result of wanting to help
>  customers that are used to running 'yum install mariadb' still be able
>  to do that.  The fact that it is packaged in a module is transparent
>  for that usecase.  Customers wanting to use non-default Application
>  Streams will indeed need to learn some new commands and concepts, but
>  they still have some analogs to other technology we use in RHEL 7,
>  like SCLs, where newer content is accessible in different channels via
>  different tools.  By having modules be implemented in yum itself,
>  those concepts become more central and common to the distribution
>  overall.
>  
>  As with any new major release, there is always a need to introduce new
>  features or technology that causes some disruption.  It is often the
>  only time we can do so in an Enterprise distribution.  We try to
>  balance that with ease of use and adoption, which are always a top
>  concern.  If you have issues with RHEL 8 and modularity, I would
>  encourage you to file a case in the Customer Portal.  Getting that
>  feedback helps ensure we're addressing customer concerns.
>  
>  
>  > > > Basically this email just says "We decided for Modularity in RHEL 9
>  > > > and
>  > > > we would like to do it in Fedora Infrastructure first".
>  > >
>  > >
>  > >
>  > >
>  > > Mostly, yes.  We were told there was ambiguity on whether modularity
>  > > would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
>  > > get ahead of that.
>  >
>  >
>  >
>  > That's unfortunate, and definitely isn't in line with what I've been
> told
>  > in response to emails asking me to move my RHEL6 and RHEL7 systems to
>  > RHEL8, where I responded that we would wait for the next product. I can
>  > see how that may be out of line with your views, but I can't say it was
>  > really expected in this way. Thank you for clarifying. Has a stable
>  > Enterprise Linux product been considered, like RHEL5,6,7, perhaps moving
>  > Modularity to its own product for the few that have a use for it?
>  
>  
>  If you have problems with the RHEL 8 packages and functionality they
>  offer, please follow up through the support channels you have as a
>  RHEL customer.  The distribution itself is quite stable, and in some
>  cases almost twice as performant as RHEL 7.  I'd like to keep this
>  thread focused on Fedora, ELN, and modules if we can. 
The issues I've seen so far affect both Fedora and RHEL, but have gotten a bit 
better in Fedora. For example, a major concern that has been much worse in 
Fedora than RHEL, for obvious reasons:

One month you can do a fresh install, install a package that, as it turns out, 
is a module for some reason.

Then you install a fresh system the next month, install the same package. 
Perform a dnf update on the previous system, and you'll find that you have a 
different version of the package installed, because you're tracking a 
different version of a default stream.

-- 
John M. Harris, Jr.

[Stephen Gallagher]

On Thu, Jun 18, 2020 at 3:34 PM John M. Harris Jr <johnmh(a)splentity.com&gt; wrote:
>  The issues I've seen so far affect both Fedora and RHEL, but have gotten a bit
>  better in Fedora. For example, a major concern that has been much worse in
>  Fedora than RHEL, for obvious reasons:
>
>  One month you can do a fresh install, install a package that, as it turns out,
>  is a module for some reason.
>
>  Then you install a fresh system the next month, install the same package.
>  Perform a dnf update on the previous system, and you'll find that you have a
>  different version of the package installed, because you're tracking a
>  different version of a default stream.
>  
Can you give an example of where you've seen this? Because our
policies in Fedora forbid changing a default stream in a released
Fedora. There were a couple exceptions around Java/Maven and libgit2
in the past due to their default streams being broken and causing
issues, but other than those extreme cases, this should never happen
in a stable release. (On Rawhide, that's a different story and if
that's what you're describing, that's a known issue and is being
tracked as part of the upgrade path work.)

[Vít Ondruch]

Dne 18. 06. 20 v 21:40 Stephen Gallagher napsal(a):
>  On Thu, Jun 18, 2020 at 3:34 PM John M. Harris Jr
> <johnmh(a)splentity.com&gt; wrote:
> > The issues I've seen so far affect both Fedora and RHEL, but have gotten a bit
> > better in Fedora. For example, a major concern that has been much worse in
> > Fedora than RHEL, for obvious reasons:
> >
> > One month you can do a fresh install, install a package that, as it turns out,
> > is a module for some reason.
> >
> > Then you install a fresh system the next month, install the same package.
> > Perform a dnf update on the previous system, and you'll find that you have a
> > different version of the package installed, because you're tracking a
> > different version of a default stream.
> >
>  Can you give an example of where you've seen this? Because our
>  policies in Fedora forbid changing a default stream in a released
>  Fedora. There were a couple exceptions around Java/Maven and libgit2
>  in the past due to their default streams being broken 

Sorry, but I don't remember this as "their default streams being
broken". AFAIR, there were multiple applications trying to use different
version of libgit2 at the same time which is not possible. That is
problem of modules design, not problem of the specific default stream as
you put it.


Vít


>   and causing
>  issues, but other than those extreme cases, this should never happen
>  in a stable release. (On Rawhide, that's a different story and if
>  that's what you're describing, that's a known issue and is being
>  tracked as part of the upgrade path work.)
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Daniel P. Berrangé]

On Fri, Jun 19, 2020 at 11:28:58AM +0200, Vít Ondruch wrote:
>  
>  Dne 18. 06. 20 v 21:40 Stephen Gallagher napsal(a):
>  > On Thu, Jun 18, 2020 at 3:34 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
> wrote:
>  >> The issues I've seen so far affect both Fedora and RHEL, but have gotten a
> bit
>  >> better in Fedora. For example, a major concern that has been much worse in
>  >> Fedora than RHEL, for obvious reasons:
>  >>
>  >> One month you can do a fresh install, install a package that, as it turns out,
>  >> is a module for some reason.
>  >>
>  >> Then you install a fresh system the next month, install the same package.
>  >> Perform a dnf update on the previous system, and you'll find that you have
> a
>  >> different version of the package installed, because you're tracking a
>  >> different version of a default stream.
>  >>
>  > Can you give an example of where you've seen this? Because our
>  > policies in Fedora forbid changing a default stream in a released
>  > Fedora. There were a couple exceptions around Java/Maven and libgit2
>  > in the past due to their default streams being broken
>  
>  
>  Sorry, but I don't remember this as "their default streams being
>  broken". AFAIR, there were multiple applications trying to use different
>  version of libgit2 at the same time which is not possible. That is
>  problem of modules design, not problem of the specific default stream as
>  you put it. 
That clashing libraries problem is the fatal design flaw in modularity
as it exists today.

We've made it possible for each module to ship arbitrarily different
versions of the same libraries, while all wanting to install into
the same directory prefix. Thus two separately maintained modules can
easily become mutually exclusive, which is a big pain for users who
need to use both concurrently. This is essentially the "DLL Hell"
problem.

The single shared /usr hierarchy only works (in the general case) when
you have a single stream where everyone agrees on using the same version
of each package.

I can only see this being solvable if non-default modules streams are
required to be built into a unique /opt prefix instead of /usr.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Alex Scheel]

----- Original Message -----
>  From: "Daniel P. Berrangé" <berrange(a)redhat.com&gt;
>  To: "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
>  Sent: Friday, June 19, 2020 5:58:28 AM
>  Subject: Re: RHEL 9 and modularity
>  
>  On Fri, Jun 19, 2020 at 11:28:58AM +0200, Vít Ondruch wrote:
>  > 
>  > Dne 18. 06. 20 v 21:40 Stephen Gallagher napsal(a):
>  > > On Thu, Jun 18, 2020 at 3:34 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
>  > > wrote:
>  > >> The issues I've seen so far affect both Fedora and RHEL, but have
> gotten
>  > >> a bit
>  > >> better in Fedora. For example, a major concern that has been much worse
>  > >> in
>  > >> Fedora than RHEL, for obvious reasons:
>  > >>
>  > >> One month you can do a fresh install, install a package that, as it
>  > >> turns out,
>  > >> is a module for some reason.
>  > >>
>  > >> Then you install a fresh system the next month, install the same
>  > >> package.
>  > >> Perform a dnf update on the previous system, and you'll find that you
>  > >> have a
>  > >> different version of the package installed, because you're tracking a
>  > >> different version of a default stream.
>  > >>
>  > > Can you give an example of where you've seen this? Because our
>  > > policies in Fedora forbid changing a default stream in a released
>  > > Fedora. There were a couple exceptions around Java/Maven and libgit2
>  > > in the past due to their default streams being broken
>  > 
>  > 
>  > Sorry, but I don't remember this as "their default streams being
>  > broken". AFAIR, there were multiple applications trying to use different
>  > version of libgit2 at the same time which is not possible. That is
>  > problem of modules design, not problem of the specific default stream as
>  > you put it.
>  
>  That clashing libraries problem is the fatal design flaw in modularity
>  as it exists today.
>  
>  We've made it possible for each module to ship arbitrarily different
>  versions of the same libraries, while all wanting to install into
>  the same directory prefix. Thus two separately maintained modules can
>  easily become mutually exclusive, which is a big pain for users who
>  need to use both concurrently. This is essentially the "DLL Hell"
>  problem.
>  
>  The single shared /usr hierarchy only works (in the general case) when
>  you have a single stream where everyone agrees on using the same version
>  of each package.
>  
>  I can only see this being solvable if non-default modules streams are
>  required to be built into a unique /opt prefix instead of /usr. 
Or better, if all the work that has gone into maintaining separately
packaged libraries went into maintaining one version and fixing
associated dependent packages as necessary... Especially for Java and
libgit2.

>  
>  Regards,
>  Daniel
>  --
>  |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange
>  |:|
>  |: https://libvirt.org         -o-            https://fstop138.berrange.com
>  |:|
>  |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange
>  |:|
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

[Dominik 'Rathann' Mierzejewski]

On Friday, 19 June 2020 at 11:58, Daniel P. Berrangé wrote:
[...]
>  I can only see this being solvable if non-default modules streams
> are
>  required to be built into a unique /opt prefix instead of /usr. 
Are you trying to re-invent the SCLs?

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
        -- from "Collected Sayings of Muad'Dib" by the Princess Irulan

[Daniel P. Berrangé]

On Fri, Jun 19, 2020 at 02:32:19PM +0200, Dominik 'Rathann' Mierzejewski wrote:
>  On Friday, 19 June 2020 at 11:58, Daniel P. Berrangé wrote:
>  [...]
>  > I can only see this being solvable if non-default modules streams are
>  > required to be built into a unique /opt prefix instead of /usr.
>  
>  Are you trying to re-invent the SCLs? 
I'm not trying to do anything myself, just pointing out that I believe
modularity is broken by design because it leads to the need to have
mutually incompatible versions of things installed at the same place
at the same time. SCL was one concept that nicely avoided this problem,
by giving users the ability to have multiple versions of a stream
installed in parallel.

Flatpaks and containers are alternative ways to let users deploy 
different versions of software without any clashing with the default
package set provided by the distro.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Martin Jackson]

On 6/19/20 7:46 AM, Daniel P. Berrangé wrote:
>  On Fri, Jun 19, 2020 at 02:32:19PM +0200, Dominik 'Rathann'
> Mierzejewski wrote:
> > On Friday, 19 June 2020 at 11:58, Daniel P. Berrangé wrote:
> > [...]
> >> I can only see this being solvable if non-default modules streams are
> >> required to be built into a unique /opt prefix instead of /usr.
> > Are you trying to re-invent the SCLs?
>  I'm not trying to do anything myself, just pointing out that I believe
>  modularity is broken by design because it leads to the need to have
>  mutually incompatible versions of things installed at the same place
>  at the same time. SCL was one concept that nicely avoided this problem,
>  by giving users the ability to have multiple versions of a stream
>  installed in parallel.
>
>  Flatpaks and containers are alternative ways to let users deploy
>  different versions of software without any clashing with the default
>  package set provided by the distro. 
To be fair, while SCL's do solve this problem, they do so via subtle but 
really important changes to user behavior.  In my $work environment we 
had lots of users that preferred to, for example, compile their own 
versions of Python rather than use the SCL version.  While the 
restrictions SCL imposes make sense if you understand the underlying 
plumbing with linking and so on, many users (even power users) want 
/usr/bin/python in a shebang line.

Modularity attempts to solve the user experience problem by effectively 
shifting this complexity to the admin side of the equation.  So if the 
only thing you care about is getting /usr/bin/something (and you have 
other things that depend on conventional pathing for any of a host of 
reasons), you can get it.

I agree in general that the problems that modularity solves haven't been 
worth the complexities in the ecosystem that they've generated.  There 
are several other ways to install multiple pythons, and make them 
installable in parallel that are easier than SCL (python2 vs. python3 in 
/usr/bin).

I use flatpaks on Fedora (Discord and okular), and I've really enjoyed 
the experience with them.  I'm not sure how well that would translate to 
the server environment though, but that general approach seems to do a 
good job of preserving user experience while isolating potentially 
troublesome conflicts in a way that doesn't mess up the "base system".

Marty

[Stephen Gallagher]

On Fri, Jun 19, 2020 at 9:08 AM Martin Jackson <mhjacks(a)swbell.net&gt; wrote:


>  I use flatpaks on Fedora (Discord and okular), and I've really
> enjoyed
>  the experience with them.  I'm not sure how well that would translate to
>  the server environment though, but that general approach seems to do a
>  good job of preserving user experience while isolating potentially
>  troublesome conflicts in a way that doesn't mess up the "base system".
>  
I love how people hold up "containers" as a solution to these problems
without considering for a moment how exactly the container itself gets
built. If you were to look into the flatpak build system in Fedora,
you'd see that they are using Modularity to construct them.

One of the reasons for Modularity is that we agree that containers are
one "right" way to handle parallel-installability. But we also know
from past experience (SCLs) that having content in unusual locations
like /opt means that applications have to be modified. By using
modules to put the version of software you want into the standard
location and then using a container to isolate it and/or provide
parallel-installability, you also get the assurance of knowing the the
content in your container is just as trusted as your standard RPM
deployments.

[Ken Dreyer]

On Fri, Jun 19, 2020 at 9:16 AM Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:

>  I love how people hold up "containers" as a solution to
> these problems
>  without considering for a moment how exactly the container itself gets
>  built. If you were to look into the flatpak build system in Fedora,
>  you'd see that they are using Modularity to construct them. 
You're right, Flatpak is doing that.

Alternatively, my team and I build containers with multiple repos when
we want to choose versions of software.

"fedpkg container-build --repo-url" takes multiple .repo files. This
works well because it uses the standard behavior that we've relied on
in Yum and DNF for many years. The corner-cases around dependencies
and what-version-overrides-what are well understood across the
developer community. This same .repo method works well across RHEL 7,
RHEL 8, and beyond. The .repo files are extremely flexible and can
come from a variety of systems that are not MBS.

Contrast this with the way that OSBS and ODCS work. The current
implementation is misleading, and the reason I say this is because
several independent teams recently arrived at the same erroneous
conclusions around how developers are supposed to use modules with
OSBS and ODCS: https://github.com/containerbuildsystem/osbs-docs/pull/152

- Ken

[Daniel P. Berrangé]

On Fri, Jun 19, 2020 at 11:16:33AM -0400, Stephen Gallagher wrote:
>  On Fri, Jun 19, 2020 at 9:08 AM Martin Jackson
> <mhjacks(a)swbell.net&gt; wrote:
>  
>  
>  > I use flatpaks on Fedora (Discord and okular), and I've really enjoyed
>  > the experience with them.  I'm not sure how well that would translate to
>  > the server environment though, but that general approach seems to do a
>  > good job of preserving user experience while isolating potentially
>  > troublesome conflicts in a way that doesn't mess up the "base
> system".
>  >
>  
>  I love how people hold up "containers" as a solution to these problems
>  without considering for a moment how exactly the container itself gets
>  built. If you were to look into the flatpak build system in Fedora,
>  you'd see that they are using Modularity to construct them.
>
>  One of the reasons for Modularity is that we agree that containers are
>  one "right" way to handle parallel-installability. But we also know
>  from past experience (SCLs) that having content in unusual locations
>  like /opt means that applications have to be modified. By using
>  modules to put the version of software you want into the standard
>  location and then using a container to isolate it and/or provide
>  parallel-installability, you also get the assurance of knowing the the
>  content in your container is just as trusted as your standard RPM
>  deployments. 
IIUC from the docs, when using Modularity to build Flatpaks, the
prefix is changed to /app instead of /usr, which makes it much
closers to SCL:

   https://docs.fedoraproject.org/en-US/flatpak/concepts/

When I looked a packaging my Entangle application as a Flatpak though,
I ended up picking Flathub instead of Fedora. That felt like a simpler
approach than using modularity, as it eliminates the intermediate RPM
packaging step which isn't really relevant to the end result. It is a
quicker code, build, test cycle if you can build from upstream sources
straight into the flatpak root in one go, especially if testing locally
outside the build system.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Colin Walters]

On Fri, Jun 19, 2020, at 12:44 PM, Daniel P. Berrangé wrote:

>  IIUC from the docs, when using Modularity to build Flatpaks, the
>  prefix is changed to /app instead of /usr, which makes it much
>  closers to SCL:
>  
>     https://docs.fedoraproject.org/en-US/flatpak/concepts/ 
The reason for that has nothing to do with modularity, but actually a core distinction
between flatpak and "containers" by which it's usually understood one is
talking about Docker/OCI.  (Also the confusion gets even bigger here because flatpak can
use Docker/OCI as a transport, but differs in the model of how it uses those images)

flatpak *dynamically links* applications with the runtime (in contrast to the Docker/OCI
model).  That's why flatpak apps are in /app.

(There's nothing truly desktop/flatpak specific about doing this, but trying to
propagate it to the wider container world around e.g. Kubernetes is a large complex topic)

[Petr Pisar]

On Fri, Jun 19, 2020 at 05:44:37PM +0100, Daniel P. Berrangé wrote:
>  On Fri, Jun 19, 2020 at 11:16:33AM -0400, Stephen Gallagher wrote:
>  > On Fri, Jun 19, 2020 at 9:08 AM Martin Jackson <mhjacks(a)swbell.net&gt; wrote:
>  > 
>  > 
>  > > I use flatpaks on Fedora (Discord and okular), and I've really enjoyed
>  > > the experience with them.  I'm not sure how well that would translate to
>  > > the server environment though, but that general approach seems to do a
>  > > good job of preserving user experience while isolating potentially
>  > > troublesome conflicts in a way that doesn't mess up the "base
> system".
>  > >
>  > 
>  > I love how people hold up "containers" as a solution to these problems
>  > without considering for a moment how exactly the container itself gets
>  > built. If you were to look into the flatpak build system in Fedora,
>  > you'd see that they are using Modularity to construct them.
>  >
>  > One of the reasons for Modularity is that we agree that containers are
>  > one "right" way to handle parallel-installability. But we also know
>  > from past experience (SCLs) that having content in unusual locations
>  > like /opt means that applications have to be modified. By using
>  > modules to put the version of software you want into the standard
>  > location and then using a container to isolate it and/or provide
>  > parallel-installability, you also get the assurance of knowing the the
>  > content in your container is just as trusted as your standard RPM
>  > deployments.
>  
>  IIUC from the docs, when using Modularity to build Flatpaks, the
>  prefix is changed to /app instead of /usr, which makes it much
>  closers to SCL:
>   
The relocation to /app brings exactly the same issues as SCL
<https://pagure.io/packaging-committee/issue/848>;,
<https://pagure.io/packaging-committee/pull-request/847>;.

-- Petr

[David Cantrell]

On Fri, Jun 19, 2020 at 08:06:29AM -0500, Martin Jackson wrote:
> On 6/19/20 7:46 AM, Daniel P. Berrangé wrote:
> >On Fri, Jun 19, 2020 at 02:32:19PM +0200, Dominik 'Rathann' Mierzejewski
> wrote:
> >>On Friday, 19 June 2020 at 11:58, Daniel P. Berrangé wrote:
> >>[...]
> >>>I can only see this being solvable if non-default modules streams are
> >>>required to be built into a unique /opt prefix instead of /usr.
> >>Are you trying to re-invent the SCLs?
> >I'm not trying to do anything myself, just pointing out that I believe
> >modularity is broken by design because it leads to the need to have
> >mutually incompatible versions of things installed at the same place
> >at the same time. SCL was one concept that nicely avoided this problem,
> >by giving users the ability to have multiple versions of a stream
> >installed in parallel.
> >
> >Flatpaks and containers are alternative ways to let users deploy
> >different versions of software without any clashing with the default
> >package set provided by the distro.
>
> To be fair, while SCL's do solve this problem, they do so via subtle 
> but really important changes to user behavior.  In my $work 
> environment we had lots of users that preferred to, for example, 
> compile their own versions of Python rather than use the SCL version.  
> While the restrictions SCL imposes make sense if you understand the 
> underlying plumbing with linking and so on, many users (even power 
> users) want /usr/bin/python in a shebang line.
>
> Modularity attempts to solve the user experience problem by 
> effectively shifting this complexity to the admin side of the 
> equation.  So if the only thing you care about is getting 
> /usr/bin/something (and you have other things that depend on 
> conventional pathing for any of a host of reasons), you can get it.
>
> I agree in general that the problems that modularity solves haven't 
> been worth the complexities in the ecosystem that they've generated.  
> There are several other ways to install multiple pythons, and make 
> them installable in parallel that are easier than SCL (python2 vs. 
> python3 in /usr/bin).
>
> I use flatpaks on Fedora (Discord and okular), and I've really enjoyed 
> the experience with them.  I'm not sure how well that would translate 
> to the server environment though, but that general approach seems to 
> do a good job of preserving user experience while isolating 
> potentially troublesome conflicts in a way that doesn't mess up the 
> "base system". 
Both of your points are valid.  Daniel's point of the shared install namespace
but delivering multiple conflicting versions of files will lead to problems.
There is no amount of covering things up that escape that reality.

[And by conflicting files, I have taken it to mean I have libsomething version
2.0 and libsomething version 2.1.  Both deliver libsomething.so.2 (that's what
other programs will list as a DT_NEEDED) but because of life being what it is
in the world of Linux distributions, those libraries actually carry different
symbols even though they advertise the same SONAME.  None of that matters
though because at the dnf level the NVR's of the packages are
libsomething-2.0-1 and libsomething-2.1-1 and dnf is enforcing installation,
upgrade, or downgrade based on that information.]

Like you say, SCLs get around this fact by impacting the user experience.
It's not terrible, but it could be better.  Modularity has, as I see it, not
yet clearly defined this reality.  We could enforce uniform SONAME rules,
install things in separate prefixes, make use of DT_RPATH, or any of a number
of other mechanisms to let files from packages and modules coexist.  And my
example here doesn't even address languages like Python or Perl, but I feel
those offer path management flexibility where additional subdirectories could
be added and not collide with files of the same name from another package.

I find flatpaks really nice from a user perspective.  I have not spent a lot
of time building them.  I went through the docs and built an example one for
myself and it was easy, but I did not get in to a lot of details.

Thanks,

-- 
David Cantrell <dcantrell(a)redhat.com&gt;
Red Hat, Inc. | Boston, MA | EST5EDT

[Miro Hrončok]

On 18. 06. 20 21:22, Josh Boyer wrote:
>  The introduction
>  of default module streams was a direct result of wanting to help
>  customers that are used to running 'yum install mariadb' still be able
>  to do that. 
Hello Josh.

I'd like to ask whether RHEL 9 has decided for default modular streams despite 
their failure in Fedora, whether this decision is final and what was the 
reasoning behind it.

When discussing default modular streams in ELN, we have heard that ELN needs 
default streams because RHEL 9 needs them. I would like to know if this 
information is something that comes from RHEL leadership directly or whether it 
is a personal option of the people who said such things.

Thanks,
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Mon, Jun 22, 2020 at 1:27 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 18. 06. 20 21:22, Josh Boyer wrote:
>  > The introduction
>  > of default module streams was a direct result of wanting to help
>  > customers that are used to running 'yum install mariadb' still be able
>  > to do that.
>
>  Hello Josh.
>
>  I'd like to ask whether RHEL 9 has decided for default modular streams despite
>  their failure in Fedora, whether this decision is final and what was the
>  reasoning behind it. 
That's an interesting question.  I think for the purposes of this
discussion, we should acknowledge that usage of default module streams
in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
adoption in Fedora doesn't necessarily equate to failure in RHEL.
With that context, I'll continue.

>  When discussing default modular streams in ELN, we have heard that
> ELN needs
>  default streams because RHEL 9 needs them. I would like to know if this
>  information is something that comes from RHEL leadership directly or whether it
>  is a personal option of the people who said such things. 
Why does it matter if "RHEL leadership" said it, or if a RHEL package
maintainer said it?  Politely, I find that an odd way to frame that
discussion, devalues individual team autonomy, and ignores what the
conversation points should be.  Let me suggest a different way.

We know within RHEL we have teams that will likely continue using
default streams.  We also know that some teams will not.  Further we
know that somes teams will likely not use modules at all, just as
teams in RHEL 8 did not use modules.  The flexibility to choose the
approach that makes the most sense for that team and their package set
would be what I would hope we try to enable in Fedora and ELN.  It is
fair to ask why a team would want to continue using default streams,
and I can offer guesses but they would be only that.  I would hope
such teams could freely chime in here.  The point is, within RHEL it
is actually their choice to make, balancing the needs of their
customers with the content they are packaging, etc.

It remains unclear to me why Fedora would go out of its way to
disallow usage of default streams in ELN.  I understand they can
present some issues if used incorrectly, or for something that is core
to non-modular content, but the concept of a default stream being
forbidden outright is strange.  Default streams in ELN don't impact
the wider Fedora distribution and removing them eliminates options and
flexibility, forcing their usage to become a downstream-only concept
which is exactly what we're trying to avoid with ELN to begin with.

josh

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 22, 2020 at 03:36:30PM -0400, Josh Boyer wrote:
>  On Mon, Jun 22, 2020 at 1:27 PM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
>  >
>  > On 18. 06. 20 21:22, Josh Boyer wrote:
>  > > The introduction
>  > > of default module streams was a direct result of wanting to help
>  > > customers that are used to running 'yum install mariadb' still be able
>  > > to do that.
>  >
>  > Hello Josh.
>  >
>  > I'd like to ask whether RHEL 9 has decided for default modular streams despite
>  > their failure in Fedora, whether this decision is final and what was the
>  > reasoning behind it. 
Hi Josh, Miro,

I think there has been a misunderstanding. I'm pretty sure Miro's
question is about "default modules" not "default streams"
(i.e. "modules enabled by default" vs. "the stream of a module to use
when a different one is not explicitly selected").
Default streams are not subject of much discussion. There is no plan
to limit them either in Fedora or ELN. But from your reply it is not
clear which of those you have in mind. Let's please clarify what we're
talking about first...

Zbyszek

> That's an interesting question.  I think for the purposes of this
> discussion, we should acknowledge that usage of default module streams
> in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
> adoption in Fedora doesn't necessarily equate to failure in RHEL.
> With that context, I'll continue.
> 
> > When discussing default modular streams in ELN, we have heard that ELN needs
> > default streams because RHEL 9 needs them. I would like to know if this
> > information is something that comes from RHEL leadership directly or whether it
> > is a personal option of the people who said such things.
> 
> Why does it matter if "RHEL leadership" said it, or if a RHEL package
> maintainer said it?  Politely, I find that an odd way to frame that
> discussion, devalues individual team autonomy, and ignores what the
> conversation points should be.  Let me suggest a different way.
> 
> We know within RHEL we have teams that will likely continue using
> default streams.  We also know that some teams will not.  Further we
> know that somes teams will likely not use modules at all, just as
> teams in RHEL 8 did not use modules.  The flexibility to choose the
> approach that makes the most sense for that team and their package set
> would be what I would hope we try to enable in Fedora and ELN.  It is
> fair to ask why a team would want to continue using default streams,
> and I can offer guesses but they would be only that.  I would hope
> such teams could freely chime in here.  The point is, within RHEL it
> is actually their choice to make, balancing the needs of their
> customers with the content they are packaging, etc.
> 
> It remains unclear to me why Fedora would go out of its way to
> disallow usage of default streams in ELN.  I understand they can
> present some issues if used incorrectly, or for something that is core
> to non-modular content, but the concept of a default stream being
> forbidden outright is strange.  Default streams in ELN don't impact
> the wider Fedora distribution and removing them eliminates options and
> flexibility, forcing their usage to become a downstream-only concept
> which is exactly what we're trying to avoid with ELN to begin with.
> 
> josh
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Miro Hrončok]

On 23. 06. 20 8:50, Zbigniew Jędrzejewski-Szmek wrote:
>  Hi Josh, Miro,
>  
>  I think there has been a misunderstanding. I'm pretty sure Miro's
>  question is about "default modules" not "default streams"
>  (i.e. "modules enabled by default" vs. "the stream of a module to use
>  when a different one is not explicitly selected").
>  Default streams are not subject of much discussion. There is no plan
>  to limit them either in Fedora or ELN. But from your reply it is not
>  clear which of those you have in mind. Let's please clarify what we're
>  talking about first... 
Hello Zbyszek. My question is about "default modular streams" i.e. about the 
automatically enabled modular streams filtering non-modular content out of dnf 
transactions. The thing that is prohibited in Fedora.

https://pagure.io/fedora-docs/modularity/blob/master/f/modules/ROOT/pages...

There is no "the stream of a module to use when a different one is not 
explicitly selected" concept in current modularity without also having "modules

enabled by default". I would love to have that instead, but we don't.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Zbigniew Jędrzejewski-Szmek]

On Tue, Jun 23, 2020 at 10:03:31AM +0200, Miro Hrončok wrote:
>  On 23. 06. 20 8:50, Zbigniew Jędrzejewski-Szmek wrote:
>  >Hi Josh, Miro,
>  >
>  >I think there has been a misunderstanding. I'm pretty sure Miro's
>  >question is about "default modules" not "default streams"
>  >(i.e. "modules enabled by default" vs. "the stream of a module to use
>  >when a different one is not explicitly selected").
>  >Default streams are not subject of much discussion. There is no plan
>  >to limit them either in Fedora or ELN. But from your reply it is not
>  >clear which of those you have in mind. Let's please clarify what we're
>  >talking about first...
>  
>  Hello Zbyszek. My question is about "default modular streams" i.e.
>  about the automatically enabled modular streams filtering
>  non-modular content out of dnf transactions. The thing that is
>  prohibited in Fedora.
>  
>
> https://pagure.io/fedora-docs/modularity/blob/master/f/modules/ROOT/pages...
Hmmm, that is not how I understood the terminology. I _think_ we were
using the terms differently in the past. But I see I got this one
wrong, and "default module stream" is the most commonly used name and
has been for a while. So please disregard my objection. I'll reply to
Josh's mail separately.

>  There is no "the stream of a module to use when a different one
> is
>  not explicitly selected" concept in current modularity without also
>  having "modules enabled by default". I would love to have that
>  instead, but we don't. 
I meant this:
$ sudo dnf module enable sway
...
Enabling module streams:
 sway              rolling                                                                
  
Is this ok [y/N]:

I thought 'rolling' is specified as default stream to pick, but it
seems not. It's just the only one, so dnf picks it without asking.
Somehow this got cross-wired in my brain with default module profiles.
Sorry for the noise.

Zbyszek

[Josh Boyer]

On Tue, Jun 23, 2020 at 3:32 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Mon, Jun 22, 2020 at 03:36:30PM -0400, Josh Boyer wrote:
>  > On Mon, Jun 22, 2020 at 1:27 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  > >
>  > > On 18. 06. 20 21:22, Josh Boyer wrote:
>  > > > The introduction
>  > > > of default module streams was a direct result of wanting to help
>  > > > customers that are used to running 'yum install mariadb' still be
> able
>  > > > to do that.
>  > >
>  > > Hello Josh.
>  > >
>  > > I'd like to ask whether RHEL 9 has decided for default modular streams
> despite
>  > > their failure in Fedora, whether this decision is final and what was the
>  > > reasoning behind it.
>
>  Hi Josh, Miro,
>
>  I think there has been a misunderstanding. I'm pretty sure Miro's
>  question is about "default modules" not "default streams"
>  (i.e. "modules enabled by default" vs. "the stream of a module to use
>  when a different one is not explicitly selected").
>  Default streams are not subject of much discussion. There is no plan
>  to limit them either in Fedora or ELN. But from your reply it is not
>  clear which of those you have in mind. Let's please clarify what we're
>  talking about first... 
Could you clarify more then?  What is a "default module"?  One that is
in the non-modular buildroot?

josh

[Zbigniew Jędrzejewski-Szmek]

On Tue, Jun 23, 2020 at 07:12:09AM -0400, Josh Boyer wrote:
>  On Tue, Jun 23, 2020 at 3:32 AM Zbigniew Jędrzejewski-Szmek
>  <zbyszek(a)in.waw.pl&gt; wrote:
>  >
>  > On Mon, Jun 22, 2020 at 03:36:30PM -0400, Josh Boyer wrote:
>  > > On Mon, Jun 22, 2020 at 1:27 PM Miro Hrončok <mhroncok(a)redhat.com&gt;
> wrote:
>  > > >
>  > > > On 18. 06. 20 21:22, Josh Boyer wrote:
>  > > > > The introduction
>  > > > > of default module streams was a direct result of wanting to help
>  > > > > customers that are used to running 'yum install mariadb'
> still be able
>  > > > > to do that.
>  > > >
>  > > > Hello Josh.
>  > > >
>  > > > I'd like to ask whether RHEL 9 has decided for default modular streams
> despite
>  > > > their failure in Fedora, whether this decision is final and what was the
>  > > > reasoning behind it.
>  >
>  > Hi Josh, Miro,
>  >
>  > I think there has been a misunderstanding. I'm pretty sure Miro's
>  > question is about "default modules" not "default streams"
>  > (i.e. "modules enabled by default" vs. "the stream of a module to
> use
>  > when a different one is not explicitly selected").
>  > Default streams are not subject of much discussion. There is no plan
>  > to limit them either in Fedora or ELN. But from your reply it is not
>  > clear which of those you have in mind. Let's please clarify what we're
>  > talking about first...
>  
>  Could you clarify more then?  What is a "default module"?  One that is
>  in the non-modular buildroot? 
Please disregard. I got the terminology wrong.

Zbyszek

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 22, 2020 at 03:36:30PM -0400, Josh Boyer wrote:
>  We know within RHEL we have teams that will likely continue using
>  default streams.  We also know that some teams will not.  Further we
>  know that somes teams will likely not use modules at all, just as
>  teams in RHEL 8 did not use modules.  The flexibility to choose the
>  approach that makes the most sense for that team and their package set
>  would be what I would hope we try to enable in Fedora and ELN.  It is
>  fair to ask why a team would want to continue using default streams,
>  and I can offer guesses but they would be only that.  I would hope
>  such teams could freely chime in here.  The point is, within RHEL it
>  is actually their choice to make, balancing the needs of their
>  customers with the content they are packaging, etc. 
[After clearing up my misunderstanding about terminology in the other subthread:]

The choice of allowing "default module streams" is not a choice that
should be taken lightly on the level of individual contributors /
teams, because it impacts the way other teams build *their* packages
and/or modules. In particular, if "enabled", other maintainers need
to remain aware of modules and need to take the quirks of modularity
into account when working on their packages. And past history shows that
demodularization (i.e. switching of modular content back to non-modular
rpms) is surprisingly complex, so the decision to "enable" cannot be
undone easily. Thus the discussion we're having now about the choices
for ELN.

>  It remains unclear to me why Fedora would go out of its way to
>  disallow usage of default streams in ELN.  I understand they can
>  present some issues if used incorrectly, or for something that is core
>  to non-modular content, but the concept of a default stream being
>  forbidden outright is strange.  Default streams in ELN don't impact
>  the wider Fedora distribution and removing them eliminates options and
>  flexibility, forcing their usage to become a downstream-only concept
>  which is exactly what we're trying to avoid with ELN to begin with. 
Default streams in ELN definitely impact how RH packagers work with
ELN and rawhide. Roughly:
- if we have no default modules in rawhide or eln, work done in rawhide
  translates 1:1 to eln and it seems likely packages can be just automatically
  rebuilt for eln.

- if we have default modules in eln, and work is done first in rawhide,
  then additional work is required to adapt the modules in eln.

  (It *may* be possible to automatize this, but not as easily as with
  singular packages. And considering that non-modularized packages
  need to be handled too, there will be at least two paths.)

- (hypothetically) if we have default modules in eln, and work is done
  in those modules and skipping rawhide (for example by not building the
  packages in rawhide), we have an unpleasant situation where eln and
  rawhide diverge.

Since the first scenario seems the most straightforward and pleasant for
everyone involved, it is imho a good question whether there are technical
reasons to follow the second option.

Zbyszek

[Josh Boyer]

On Tue, Jun 23, 2020 at 5:14 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Mon, Jun 22, 2020 at 03:36:30PM -0400, Josh Boyer wrote:
>  > We know within RHEL we have teams that will likely continue using
>  > default streams.  We also know that some teams will not.  Further we
>  > know that somes teams will likely not use modules at all, just as
>  > teams in RHEL 8 did not use modules.  The flexibility to choose the
>  > approach that makes the most sense for that team and their package set
>  > would be what I would hope we try to enable in Fedora and ELN.  It is
>  > fair to ask why a team would want to continue using default streams,
>  > and I can offer guesses but they would be only that.  I would hope
>  > such teams could freely chime in here.  The point is, within RHEL it
>  > is actually their choice to make, balancing the needs of their
>  > customers with the content they are packaging, etc.
>
>  [After clearing up my misunderstanding about terminology in the other subthread:]
>
>  The choice of allowing "default module streams" is not a choice that
>  should be taken lightly on the level of individual contributors /
>  teams, because it impacts the way other teams build *their* packages
>  and/or modules. In particular, if "enabled", other maintainers need
>  to remain aware of modules and need to take the quirks of modularity
>  into account when working on their packages. And past history shows that
>  demodularization (i.e. switching of modular content back to non-modular
>  rpms) is surprisingly complex, so the decision to "enable" cannot be
>  undone easily. Thus the discussion we're having now about the choices
>  for ELN. 
Agreed.

>  > It remains unclear to me why Fedora would go out of its way to
>  > disallow usage of default streams in ELN.  I understand they can
>  > present some issues if used incorrectly, or for something that is core
>  > to non-modular content, but the concept of a default stream being
>  > forbidden outright is strange.  Default streams in ELN don't impact
>  > the wider Fedora distribution and removing them eliminates options and
>  > flexibility, forcing their usage to become a downstream-only concept
>  > which is exactly what we're trying to avoid with ELN to begin with.
>
>  Default streams in ELN definitely impact how RH packagers work with
>  ELN and rawhide. Roughly:
>  - if we have no default modules in rawhide or eln, work done in rawhide
>    translates 1:1 to eln and it seems likely packages can be just automatically
>    rebuilt for eln.
>
>  - if we have default modules in eln, and work is done first in rawhide,
>    then additional work is required to adapt the modules in eln. 
If Fedora forbids default modules in ELN, then they'll have to do
additional work downstream in RHEL if that is the best choice there.
That is also bad.

>    (It *may* be possible to automatize this, but not as easily as
> with
>    singular packages. And considering that non-modularized packages
>    need to be handled too, there will be at least two paths.)
>
>  - (hypothetically) if we have default modules in eln, and work is done
>    in those modules and skipping rawhide (for example by not building the
>    packages in rawhide), we have an unpleasant situation where eln and
>    rawhide diverge. 
This is a very tenuous strawman.  You could also run into a case where
ELN forbids modules or default module streams and the maintainers
simply choose not to maintain anything in Fedora at all.  That's far
worse than divergence in my opinion.  Fortunately, I think neither are
actually likely and this part of the conversation seems like it's
pointless to debate.

>  Since the first scenario seems the most straightforward and pleasant
> for
>  everyone involved, it is imho a good question whether there are technical
>  reasons to follow the second option. 
And maybe that is the option most will choose.  However, not even
letting ELN have the option seems to preclude that choice entirely.

josh

[Miro Hrončok]

On 23. 06. 20 13:29, Josh Boyer wrote:
> >    (It*may*  be possible to automatize this, but not as easily as
> with
> >    singular packages. And considering that non-modularized packages
> >    need to be handled too, there will be at least two paths.)
> >
> > - (hypothetically) if we have default modules in eln, and work is done
> >    in those modules and skipping rawhide (for example by not building the
> >    packages in rawhide), we have an unpleasant situation where eln and
> >    rawhide diverge.
>  This is a very tenuous strawman.  You could also run into a case where
>  ELN forbids modules or default module streams and the maintainers
>  simply choose not to maintain anything in Fedora at all.  That's far
>  worse than divergence in my opinion. 
When ELN was proposed and discussed, separate eln branches were proposed by 
several Fedora and RHEL maintainers. It was dismissed, and it was said 
repeatedly that rawhide/ELN divergence MUST be avoided. I wonder if that 
requirement has changed.

>  Fortunately, I think neither are
>  actually likely and this part of the conversation seems like it's
>  pointless to debate. 
This has happened in the past when Fedora had default modular streams. Whether 
likely or not to repeat, we have experience with the problem, so the discussion 
is not pointless at all.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Tue, Jun 23, 2020 at 7:36 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 23. 06. 20 13:29, Josh Boyer wrote:
>  >>    (It*may*  be possible to automatize this, but not as easily as with
>  >>    singular packages. And considering that non-modularized packages
>  >>    need to be handled too, there will be at least two paths.)
>  >>
>  >> - (hypothetically) if we have default modules in eln, and work is done
>  >>    in those modules and skipping rawhide (for example by not building the
>  >>    packages in rawhide), we have an unpleasant situation where eln and
>  >>    rawhide diverge.
>  > This is a very tenuous strawman.  You could also run into a case where
>  > ELN forbids modules or default module streams and the maintainers
>  > simply choose not to maintain anything in Fedora at all.  That's far
>  > worse than divergence in my opinion.
>
>  When ELN was proposed and discussed, separate eln branches were proposed by
>  several Fedora and RHEL maintainers. It was dismissed, and it was said
>  repeatedly that rawhide/ELN divergence MUST be avoided. I wonder if that
>  requirement has changed. 
Divergence where?  At the source level?  Why would the existence of a
default module in ELN cause divergence at the source level?  It
wouldn't.  The rawhide sources would be used for the module build in
ELN.

If you mean at the binary level, then I have no idea how anyone
possibly thinks ELN and Rawhide are the same because ELN is built with
entirely different flags, settings, etc.

>  > Fortunately, I think neither are
>  > actually likely and this part of the conversation seems like it's
>  > pointless to debate.
>
>  This has happened in the past when Fedora had default modular streams. Whether
>  likely or not to repeat, we have experience with the problem, so the discussion
>  is not pointless at all. 
You seem to be concerned less about divergence and more about
abandonment of packages in Fedora, at least in ways counter to how the
default distribution is built.  You could come up with some guidelines
on usage of ELN modules that require existing content to be maintained
as it is in Fedora if that's what you want to ensure.  It's onerous
and causes extra work, but allows people to do their work in the open.
However, if you prevent that from happening entirely, you run the risk
of them abandoning the packages entirely which is counter to your goal
(I think).

josh

[Miro Hrončok]

On 23. 06. 20 13:43, Josh Boyer wrote:
>  On Tue, Jun 23, 2020 at 7:36 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 23. 06. 20 13:29, Josh Boyer wrote:
> >>>     (It*may*  be possible to automatize this, but not as easily as with
> >>>     singular packages. And considering that non-modularized packages
> >>>     need to be handled too, there will be at least two paths.)
> >>>
> >>> - (hypothetically) if we have default modules in eln, and work is done
> >>>     in those modules and skipping rawhide (for example by not building the
> >>>     packages in rawhide), we have an unpleasant situation where eln and
> >>>     rawhide diverge.
> >> This is a very tenuous strawman.  You could also run into a case where
> >> ELN forbids modules or default module streams and the maintainers
> >> simply choose not to maintain anything in Fedora at all.  That's far
> >> worse than divergence in my opinion.
> >
> > When ELN was proposed and discussed, separate eln branches were proposed by
> > several Fedora and RHEL maintainers. It was dismissed, and it was said
> > repeatedly that rawhide/ELN divergence MUST be avoided. I wonder if that
> > requirement has changed.
>  
>  Divergence where?  At the source level?  Why would the existence of a
>  default module in ELN cause divergence at the source level?  It
>  wouldn't.  The rawhide sources would be used for the module build in
>  ELN. 
I ma concerned about divergence at source level. Modules in Fedora are built 
from stream branches. Rawhide content is built from the "master" branch. This is

the first time I've heard that rawhide sources would be used for the module 
build in ELN and it certainly makes the entire thing more appealing. I've 
already asked about this in:

https://pagure.io/fesco/issue/2390#comment-659188

>  If you mean at the binary level, then I have no idea how anyone
>  possibly thinks ELN and Rawhide are the same because ELN is built with
>  entirely different flags, settings, etc. 
No, I don't.

> >> Fortunately, I think neither are
> >> actually likely and this part of the conversation seems like it's
> >> pointless to debate.
> >
> > This has happened in the past when Fedora had default modular streams. Whether
> > likely or not to repeat, we have experience with the problem, so the discussion
> > is not pointless at all.
>  
>  You seem to be concerned less about divergence and more about
>  abandonment of packages in Fedora, at least in ways counter to how the
>  default distribution is built.  You could come up with some guidelines
>  on usage of ELN modules that require existing content to be maintained
>  as it is in Fedora if that's what you want to ensure.  It's onerous
>  and causes extra work, but allows people to do their work in the open.
>  However, if you prevent that from happening entirely, you run the risk
>  of them abandoning the packages entirely which is counter to your goal
>  (I think). 
I can totally support that. Thanks.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Tue, Jun 23, 2020 at 7:56 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 23. 06. 20 13:43, Josh Boyer wrote:
>  > On Tue, Jun 23, 2020 at 7:36 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  >>
>  >> On 23. 06. 20 13:29, Josh Boyer wrote:
>  >>>>     (It*may*  be possible to automatize this, but not as easily as with
>  >>>>     singular packages. And considering that non-modularized packages
>  >>>>     need to be handled too, there will be at least two paths.)
>  >>>>
>  >>>> - (hypothetically) if we have default modules in eln, and work is done
>  >>>>     in those modules and skipping rawhide (for example by not building
> the
>  >>>>     packages in rawhide), we have an unpleasant situation where eln and
>  >>>>     rawhide diverge.
>  >>> This is a very tenuous strawman.  You could also run into a case where
>  >>> ELN forbids modules or default module streams and the maintainers
>  >>> simply choose not to maintain anything in Fedora at all.  That's far
>  >>> worse than divergence in my opinion.
>  >>
>  >> When ELN was proposed and discussed, separate eln branches were proposed by
>  >> several Fedora and RHEL maintainers. It was dismissed, and it was said
>  >> repeatedly that rawhide/ELN divergence MUST be avoided. I wonder if that
>  >> requirement has changed.
>  >
>  > Divergence where?  At the source level?  Why would the existence of a
>  > default module in ELN cause divergence at the source level?  It
>  > wouldn't.  The rawhide sources would be used for the module build in
>  > ELN.
>
>  I ma concerned about divergence at source level. Modules in Fedora are built
>  from stream branches. Rawhide content is built from the "master" branch. This
> is
>  the first time I've heard that rawhide sources would be used for the module
>  build in ELN and it certainly makes the entire thing more appealing. I've
>  already asked about this in:
>
>  https://pagure.io/fesco/issue/2390#comment-659188 
Hmm.  I have introduced confusion.  My apologies for that, and let me rephrase.

First, I was talking about the sources themselves, not necessarily the
branches.  If you look at a single package, its source doesn't really
change when it's built as a stand-alone RPM, or a modular RPM.  It's
just a package.  Therefore, there isn't divergence at the source
level.  If the same version of the source is used in rawhide and ELN,
then I don't see divergence.

Secondly, I did not mean to imply there were concrete plans to build
modules from the rawhide branch in Fedora.  I should have said
"could", not "would".  However, if that's an approach that makes
things better in some ways then perhaps it should be looked at in
earnest as a compromise?

josh

>  > If you mean at the binary level, then I have no idea how anyone
>  > possibly thinks ELN and Rawhide are the same because ELN is built with
>  > entirely different flags, settings, etc.
>
>  No, I don't.
>
>  >>> Fortunately, I think neither are
>  >>> actually likely and this part of the conversation seems like it's
>  >>> pointless to debate.
>  >>
>  >> This has happened in the past when Fedora had default modular streams. Whether
>  >> likely or not to repeat, we have experience with the problem, so the discussion
>  >> is not pointless at all.
>  >
>  > You seem to be concerned less about divergence and more about
>  > abandonment of packages in Fedora, at least in ways counter to how the
>  > default distribution is built.  You could come up with some guidelines
>  > on usage of ELN modules that require existing content to be maintained
>  > as it is in Fedora if that's what you want to ensure.  It's onerous
>  > and causes extra work, but allows people to do their work in the open.
>  > However, if you prevent that from happening entirely, you run the risk
>  > of them abandoning the packages entirely which is counter to your goal
>  > (I think).
>
>  I can totally support that. Thanks.
>
>  --
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>

[Vít Ondruch]

Dne 23. 06. 20 v 14:02 Josh Boyer napsal(a):
>  On Tue, Jun 23, 2020 at 7:56 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > On 23. 06. 20 13:43, Josh Boyer wrote:
> >> On Tue, Jun 23, 2020 at 7:36 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
> >>> On 23. 06. 20 13:29, Josh Boyer wrote:
> >>>>>     (It*may*  be possible to automatize this, but not as easily as
> with
> >>>>>     singular packages. And considering that non-modularized packages
> >>>>>     need to be handled too, there will be at least two paths.)
> >>>>>
> >>>>> - (hypothetically) if we have default modules in eln, and work is
> done
> >>>>>     in those modules and skipping rawhide (for example by not
> building the
> >>>>>     packages in rawhide), we have an unpleasant situation where eln
> and
> >>>>>     rawhide diverge.
> >>>> This is a very tenuous strawman.  You could also run into a case where
> >>>> ELN forbids modules or default module streams and the maintainers
> >>>> simply choose not to maintain anything in Fedora at all.  That's far
> >>>> worse than divergence in my opinion.
> >>> When ELN was proposed and discussed, separate eln branches were proposed by
> >>> several Fedora and RHEL maintainers. It was dismissed, and it was said
> >>> repeatedly that rawhide/ELN divergence MUST be avoided. I wonder if that
> >>> requirement has changed.
> >> Divergence where?  At the source level?  Why would the existence of a
> >> default module in ELN cause divergence at the source level?  It
> >> wouldn't.  The rawhide sources would be used for the module build in
> >> ELN.
> > I ma concerned about divergence at source level. Modules in Fedora are built
> > from stream branches. Rawhide content is built from the "master" branch.
> This is
> > the first time I've heard that rawhide sources would be used for the module
> > build in ELN and it certainly makes the entire thing more appealing. I've
> > already asked about this in:
> >
> > https://pagure.io/fesco/issue/2390#comment-659188
>  Hmm.  I have introduced confusion.  My apologies for that, and let me rephrase.
>
>  First, I was talking about the sources themselves, not necessarily the
>  branches.  If you look at a single package, its source doesn't really
>  change when it's built as a stand-alone RPM, or a modular RPM.  It's
>  just a package.  Therefore, there isn't divergence at the source
>  level.  If the same version of the source is used in rawhide and ELN,
>  then I don't see divergence. 

This would be true if module was supposed to support only single Fedora
version. If module supports multiple versions, this cannot be further
from reality. And in the context of ELN, this is the later case.


Vít


>  Secondly, I did not mean to imply there were concrete plans to build
>  modules from the rawhide branch in Fedora.  I should have said
>  "could", not "would".  However, if that's an approach that makes
>  things better in some ways then perhaps it should be looked at in
>  earnest as a compromise?
>
>  josh
>
> >> If you mean at the binary level, then I have no idea how anyone
> >> possibly thinks ELN and Rawhide are the same because ELN is built with
> >> entirely different flags, settings, etc.
> > No, I don't.
> >
> >>>> Fortunately, I think neither are
> >>>> actually likely and this part of the conversation seems like it's
> >>>> pointless to debate.
> >>> This has happened in the past when Fedora had default modular streams.
> Whether
> >>> likely or not to repeat, we have experience with the problem, so the
> discussion
> >>> is not pointless at all.
> >> You seem to be concerned less about divergence and more about
> >> abandonment of packages in Fedora, at least in ways counter to how the
> >> default distribution is built.  You could come up with some guidelines
> >> on usage of ELN modules that require existing content to be maintained
> >> as it is in Fedora if that's what you want to ensure.  It's onerous
> >> and causes extra work, but allows people to do their work in the open.
> >> However, if you prevent that from happening entirely, you run the risk
> >> of them abandoning the packages entirely which is counter to your goal
> >> (I think).
> > I can totally support that. Thanks.
> >
> > --
> > Miro Hrončok
> > --
> > Phone: [phone]
> > IRC: mhroncok
> >
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Miro Hrončok]

On 22. 06. 20 21:36, Josh Boyer wrote:
> > I'd like to ask whether RHEL 9 has decided for default
> modular streams despite
> > their failure in Fedora, whether this decision is final and what was the
> > reasoning behind it.
>  
>  That's an interesting question.  I think for the purposes of this
>  discussion, we should acknowledge that usage of default module streams
>  in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
>  adoption in Fedora doesn't necessarily equate to failure in RHEL.
>  With that context, I'll continue. 
Before we continue with that context, could you please elaborate on this?

Obviously  we can say "usage of default module streams in Fedora and RHEL is 
different" and to some extent this will always be true. However I would  like to 
know why they are *significantly* different to justify saying  the failure in 
Fedora does not necessarily mean RHEL would experience  the same failure.

What makes RHEL so different that the failure  is not relevant to it? Is it the 
stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it 
the less "wild" development  process? Is it something different?

I am not saying it isn't true, I am genuinely interested on the "whys"
behind 
your statement.  Without them, I need to guess and it's obvious that once we 
guess  rationales of the words said by others, we cannot make proper informed 
decisions or a reasonable discussion.

> > When discussing default modular streams in ELN, we have heard
> that ELN needs
> > default streams because RHEL 9 needs them. I would like to know if this
> > information is something that comes from RHEL leadership directly or whether it
> > is a personal option of the people who said such things.
>  
>  Why does it matter if "RHEL leadership" said it, or if a RHEL package
>  maintainer said it?  Politely, I find that an odd way to frame that
>  discussion, devalues individual team autonomy, and ignores what the
>  conversation points should be.  Let me suggest a different way. 
Every  package maintainer's (Fedora and RHEL alike) opinion matters. However, 
when making distro-wide decisions, it makes total sense to evaluate the 
following two statements with different merit, would you not agree?

  a) "Grinch doesn't like modularity, he doesn't want default modular streams
in 
Fedora."
  b) "After months of evaluation, the engineering leadership of Fedora has 
decided that default modular streams are not allowed in Fedora."

Similarly  I'd like to know whether "we need default modular streams in ELN 
because we need them in RHEL 9" is an opinion of a single person, single team, 
whether it is an assumed opinion of the distro as a whole, or the documented 
position of RHEL 9 leadership. Whatever the answer is, it doesn't  make the 
statement more or less valid, it merely helps us to better understand the scope.

Since the "because RHEL 9 needs it" argument has so far been the end point of 
the rationale, and since you've been so kind to share some RHEL 9 plans wrt 
modularity on Fedora devel list, I've decided to ask about this particular topic 
here as well.

We have spend 3+ Fedora releases debating default modular streams in Fedora and 
once we were about to prohibit them, a request has come from ELN to have them. 
Hence, I've assumed there were some RHEL conversations and decisions about 
whether to have default modular streams in RHEL 9 even when they are not allowed 
in Fedora and the request to have them in ELN is the direct result of such 
conversations and/or decisions. To understand the reasoning, I've asked 
clarifying questions.

I am not trying to argue here, I am only asking for more insight. For me, the 
request for default modular streams was surprising. Now I am trying to approach 
the request with an open mind, but to do that, I would like to understand it better.

>  We know within RHEL we have teams that will likely continue using
>  default streams.  We also know that some teams will not.  Further we
>  know that somes teams will likely not use modules at all, just as
>  teams in RHEL 8 did not use modules.  The flexibility to choose the
>  approach that makes the most sense for that team and their package set
>  would be what I would hope we try to enable in Fedora and ELN. 
This  is exactly the flexibility we've already had enabled in Fedora for 3+ 
releases and after a very long and very painful discussion Fedora has decided to 
prohibit it. Hence, I don't understand what are the reasons to have it in RHEL 9 
regardless, which is the reason I ask the questions.

>  It is
>  fair to ask why a team would want to continue using default streams,
>  and I can offer guesses but they would be only that.  I would hope
>  such teams could freely chime in here.  The point is, within RHEL it
>  is actually their choice to make, balancing the needs of their
>  customers with the content they are packaging, etc.
>  It remains unclear to me why Fedora would go out of its way to
>  disallow usage of default streams in ELN.  I understand they can
>  present some issues if used incorrectly, or for something that is core
>  to non-modular content, but the concept of a default stream being
>  forbidden outright is strange.  Default streams in ELN don't impact
>  the wider Fedora distribution and removing them eliminates options and
>  flexibility, forcing their usage to become a downstream-only concept
>  which is exactly what we're trying to avoid with ELN to begin with. 
I haven't asked the questions to argue about this. I just wanted to get things 
clarified, so we can have a discussion at FESCo based on some actual RHEL 
information instead of guesses.

Frankly, I don't know how RHEL decisions are made or who makes them. As a Fedora 
package maintainer (and hence by extension an ELN package maintainer) I just 
want to know what decisions were made wrt default streams in RHEL 9 and why, so 
I can understand the request for default streams in ELN better.

Thank you very much for sharing some RHEL information here.
-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 22. 06. 20 21:36, Josh Boyer wrote:
>  >> I'd like to ask whether RHEL 9 has decided for default modular streams
> despite
>  >> their failure in Fedora, whether this decision is final and what was the
>  >> reasoning behind it.
>  >
>  > That's an interesting question.  I think for the purposes of this
>  > discussion, we should acknowledge that usage of default module streams
>  > in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
>  > adoption in Fedora doesn't necessarily equate to failure in RHEL.
>  > With that context, I'll continue.
>
>  Before we continue with that context, could you please elaborate on this? 
If I must.

>  Obviously  we can say "usage of default module streams in Fedora
> and RHEL is
>  different" and to some extent this will always be true. However I would  like to
>  know why they are *significantly* different to justify saying  the failure in
>  Fedora does not necessarily mean RHEL would experience  the same failure. 
For all the same reasons SCLs and containers are widely used and
adopted in RHEL, but not Fedora.  (And before people say "Fedora has
containers", I know that.  They're fine.  That doesn't mean you see a
massive adoption of Fedora base images on a wide scale.)

>  What makes RHEL so different that the failure  is not relevant to it?
> Is it the
>  stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  the less "wild" development  process? Is it something different? 
Primarily, RHEL:

- Moves much much slower
- Has a base distribution that is extremely stable and does not
version bump often across the "platform" layer of libraries, etc
- Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
- Has a broader downstream ecosystem of ISVs and products that require
stability and continuity

A default module stream in Fedora, which is only around for 13 months,
provides little value when the next Fedora release likely is going to
have a different default or a newer version in 6mo anyway.  Fedora
moves the entire distribution too fast for there to be a lot of usage
there.  RHEL can pin on a default and have that be the default on a
long enough timeline that it actually works.

Are there other ways to accomplish the same in RHEL?  Yes.  Does that
mean failure in Fedora translates to failure everywhere?  No.

>  I am not saying it isn't true, I am genuinely interested on the
> "whys" behind
>  your statement.  Without them, I need to guess and it's obvious that once we
>  guess  rationales of the words said by others, we cannot make proper informed
>  decisions or a reasonable discussion. 
You are smart enough to come up with your own reasons.  The fact that
you phrased them as questions rather than statements highlights your
perspective on the discussion more than any actual merit of what I
just provided above.

>  >> When discussing default modular streams in ELN, we have
> heard that ELN needs
>  >> default streams because RHEL 9 needs them. I would like to know if this
>  >> information is something that comes from RHEL leadership directly or whether it
>  >> is a personal option of the people who said such things.
>  >
>  > Why does it matter if "RHEL leadership" said it, or if a RHEL package
>  > maintainer said it?  Politely, I find that an odd way to frame that
>  > discussion, devalues individual team autonomy, and ignores what the
>  > conversation points should be.  Let me suggest a different way.
>
>  Every  package maintainer's (Fedora and RHEL alike) opinion matters. However,
>  when making distro-wide decisions, it makes total sense to evaluate the
>  following two statements with different merit, would you not agree? 
You mean like evaluating a technology across two different
distributions with different merit? :)

>    a) "Grinch doesn't like modularity, he doesn't want
> default modular streams in
>  Fedora."
>    b) "After months of evaluation, the engineering leadership of Fedora has
>  decided that default modular streams are not allowed in Fedora."
>
>  Similarly  I'd like to know whether "we need default modular streams in ELN
>  because we need them in RHEL 9" is an opinion of a single person, single team,
>  whether it is an assumed opinion of the distro as a whole, or the documented
>  position of RHEL 9 leadership. Whatever the answer is, it doesn't  make the
>  statement more or less valid, it merely helps us to better understand the scope.
>
>  Since the "because RHEL 9 needs it" argument has so far been the end point of
>  the rationale, and since you've been so kind to share some RHEL 9 plans wrt
>  modularity on Fedora devel list, I've decided to ask about this particular topic
>  here as well. 
I see.  I can offer only that we are driving from a position of
continuity across RHEL releases unless there is reason enough to
deviate.  With RHEL major releases being every 3 years, the ability to
change course rapidly(ish) exists but it would be extremely jarring to
customers to do that.  Particularly when many are still transitioning
from RHEL 6 and RHEL 7.

>  We have spend 3+ Fedora releases debating default modular streams in
> Fedora and
>  once we were about to prohibit them, a request has come from ELN to have them.
>  Hence, I've assumed there were some RHEL conversations and decisions about
>  whether to have default modular streams in RHEL 9 even when they are not allowed
>  in Fedora and the request to have them in ELN is the direct result of such
>  conversations and/or decisions. To understand the reasoning, I've asked
>  clarifying questions.
>
>  I am not trying to argue here, I am only asking for more insight. For me, the
>  request for default modular streams was surprising. Now I am trying to approach
>  the request with an open mind, but to do that, I would like to understand it better.
Hopefully that provides more context.  There may be more coming later
as we can offer it, but I suspect it won't be on this list nor will it
be in time for FESCo this week

>  > We know within RHEL we have teams that will likely continue
> using
>  > default streams.  We also know that some teams will not.  Further we
>  > know that somes teams will likely not use modules at all, just as
>  > teams in RHEL 8 did not use modules.  The flexibility to choose the
>  > approach that makes the most sense for that team and their package set
>  > would be what I would hope we try to enable in Fedora and ELN.
>
>  This  is exactly the flexibility we've already had enabled in Fedora for 3+
>  releases and after a very long and very painful discussion Fedora has decided to
>  prohibit it. Hence, I don't understand what are the reasons to have it in RHEL 9
>  regardless, which is the reason I ask the questions.
>
>  > It is
>  > fair to ask why a team would want to continue using default streams,
>  > and I can offer guesses but they would be only that.  I would hope
>  > such teams could freely chime in here.  The point is, within RHEL it
>  > is actually their choice to make, balancing the needs of their
>  > customers with the content they are packaging, etc.
>  > It remains unclear to me why Fedora would go out of its way to
>  > disallow usage of default streams in ELN.  I understand they can
>  > present some issues if used incorrectly, or for something that is core
>  > to non-modular content, but the concept of a default stream being
>  > forbidden outright is strange.  Default streams in ELN don't impact
>  > the wider Fedora distribution and removing them eliminates options and
>  > flexibility, forcing their usage to become a downstream-only concept
>  > which is exactly what we're trying to avoid with ELN to begin with.
>
>  I haven't asked the questions to argue about this. I just wanted to get things
>  clarified, so we can have a discussion at FESCo based on some actual RHEL
>  information instead of guesses.
>
>  Frankly, I don't know how RHEL decisions are made or who makes them. As a Fedora
>  package maintainer (and hence by extension an ELN package maintainer) I just
>  want to know what decisions were made wrt default streams in RHEL 9 and why, so
>  I can understand the request for default streams in ELN better. 
I would ask that you separate the possibility to create a default
stream from the act of actually doing so.  That's been the entire
point of this subthread.  If Fedora and/or ELN want to put guidelines
in place that make it a broader discussion before something can be
made a default stream in ELN, then go for it.  However, forbidding
them entirely just drives work elsewhere and that's counter to
developing RHEL in an open manner.

One of my favorite movie quotes is from Jurassic Park.  There is a
scene where Ian Malcom says "Your scientists were so preoccupied with
whether or not they could, they didn’t stop to think if they should."
Normally I use this when pointing out some errant decisions.  In this
case, I'm actually asking to retain flexibility on whether ELN *could*
have default streams and then further figuring out when they should
based on merit.  If anyone actually wants to influence future RHEL
releases, writing up those kinds of guidelines has far more value than
preventing a technology from being used.  It allows people to use
technology with responsible consideration, rather than making ELN a
place full of giant distro-eating dinosaurs.

josh

[Miro Hrončok]

On 23. 06. 20 14:30, Josh Boyer wrote:
>  On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 22. 06. 20 21:36, Josh Boyer wrote:
> >>> I'd like to ask whether RHEL 9 has decided for default modular streams
> despite
> >>> their failure in Fedora, whether this decision is final and what was the
> >>> reasoning behind it.
> >>
> >> That's an interesting question.  I think for the purposes of this
> >> discussion, we should acknowledge that usage of default module streams
> >> in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
> >> adoption in Fedora doesn't necessarily equate to failure in RHEL.
> >> With that context, I'll continue.
> >
> > Before we continue with that context, could you please elaborate on this?
>  
>  If I must. 
Not at all, feel free to not continue this conversation if it bothers you in any 
way. But I think it is extremely helpful to have the things said here and I 
thank you for doing it.

> > What makes RHEL so different that the failure  is not relevant to
> it? Is it the
> > stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
> > the less "wild" development  process? Is it something different?
>  
>  Primarily, RHEL:
>  
>  - Moves much much slower
>  - Has a base distribution that is extremely stable and does not
>  version bump often across the "platform" layer of libraries, etc
>  - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  - Has a broader downstream ecosystem of ISVs and products that require
>  stability and continuity 
I can see technical challenges with default modular streams in ELN, because even 
thou ELN is RHEL-like, the content is Rawhide-like and hence most of the above 
does not apply.

(Notice I say challenges: I don't say this is a reason why we should not have this.)

>  You are smart enough to come up with your own reasons.  The fact
> that
>  you phrased them as questions rather than statements highlights your
>  perspective on the discussion more than any actual merit of what I
>  just provided above. 
I don't even know what to reply to this. If this is the way to handle the 
conversation, I'd rather end it here. Thanks for your replies, Josh.

AFAIK Stephan and Igor are working on general guidelines for default streams in 
ELN and any further discussion should probably happen after that.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 23. 06. 20 15:42, Miro Hrončok wrote:
>  AFAIK Stephan and Igor 
Sorry, I've meant Stephen.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Josh Boyer]

On Tue, Jun 23, 2020 at 9:42 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 23. 06. 20 14:30, Josh Boyer wrote:
>  > On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  >>
>  >> On 22. 06. 20 21:36, Josh Boyer wrote:
>  >>>> I'd like to ask whether RHEL 9 has decided for default modular
> streams despite
>  >>>> their failure in Fedora, whether this decision is final and what was
> the
>  >>>> reasoning behind it.
>  >>>
>  >>> That's an interesting question.  I think for the purposes of this
>  >>> discussion, we should acknowledge that usage of default module streams
>  >>> in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
>  >>> adoption in Fedora doesn't necessarily equate to failure in RHEL.
>  >>> With that context, I'll continue.
>  >>
>  >> Before we continue with that context, could you please elaborate on this?
>  >
>  > If I must.
>
>  Not at all, feel free to not continue this conversation if it bothers you in any
>  way. But I think it is extremely helpful to have the things said here and I
>  thank you for doing it. 
Bothers is perhaps too strong of a word.  It's simply very time
consuming, but likely worthwhile.

>  >> What makes RHEL so different that the failure  is not
> relevant to it? Is it the
>  >> stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  >> the less "wild" development  process? Is it something different?
>  >
>  > Primarily, RHEL:
>  >
>  > - Moves much much slower
>  > - Has a base distribution that is extremely stable and does not
>  > version bump often across the "platform" layer of libraries, etc
>  > - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  > - Has a broader downstream ecosystem of ISVs and products that require
>  > stability and continuity
>
>  I can see technical challenges with default modular streams in ELN, because even
>  thou ELN is RHEL-like, the content is Rawhide-like and hence most of the above
>  does not apply. 
I think that discounts that all Enterprise distributions have to start
somewhere, so defining a default for a period of time to further that
bootstrap has value.  If one expects ELN to be a continuous stream
that is always updatable from day to day, then changing the default
would present challenges.  I actually have no idea what the long-term
implications of ELN are.

>  (Notice I say challenges: I don't say this is a reason why we
> should not have this.)
>
>  > You are smart enough to come up with your own reasons.  The fact that
>  > you phrased them as questions rather than statements highlights your
>  > perspective on the discussion more than any actual merit of what I
>  > just provided above.
>
>  I don't even know what to reply to this. If this is the way to handle the
>  conversation, I'd rather end it here. Thanks for your replies, Josh. 
My point, possibly too subtly stated, is that I see highly intelligent
people intentionally saying they are only fedora packagers or only
rhel packagers, and that kind of framing seems to start a lot of
conversations off on the wrong foot.  You were clearly able to think
about your own question and come up with reasons yourself.  If more
people did that, it would foster a conversation where we seek
compromise instead of positioning.  E.g. put yourself in the other
person's shoes.

I was saying your reasons were already just as valid as anything I was
going to tell you.

>  AFAIK Stephan and Igor are working on general guidelines for default
> streams in
>  ELN and any further discussion should probably happen after that. 
That is good to hear.

josh

[Pierre-Yves Chibon]

On Tue, Jun 23, 2020 at 08:30:37AM -0400, Josh Boyer wrote:
>  On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
>  >
>  > On 22. 06. 20 21:36, Josh Boyer wrote:
>  > >> I'd like to ask whether RHEL 9 has decided for default modular streams
> despite
>  > >> their failure in Fedora, whether this decision is final and what was the
>  > >> reasoning behind it.
>  > >
>  > > That's an interesting question.  I think for the purposes of this
>  > > discussion, we should acknowledge that usage of default module streams
>  > > in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
>  > > adoption in Fedora doesn't necessarily equate to failure in RHEL.
>  > > With that context, I'll continue.
>  >
>  > Before we continue with that context, could you please elaborate on this?
>  
>  If I must.
>  
>  > Obviously  we can say "usage of default module streams in Fedora and RHEL is
>  > different" and to some extent this will always be true. However I would  like
> to
>  > know why they are *significantly* different to justify saying  the failure in
>  > Fedora does not necessarily mean RHEL would experience  the same failure.
>  
>  For all the same reasons SCLs and containers are widely used and
>  adopted in RHEL, but not Fedora.  (And before people say "Fedora has
>  containers", I know that.  They're fine.  That doesn't mean you see a
>  massive adoption of Fedora base images on a wide scale.)
>  
>  > What makes RHEL so different that the failure  is not relevant to it? Is it the
>  > stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  > the less "wild" development  process? Is it something different?
>  
>  Primarily, RHEL:
>  
>  - Moves much much slower
>  - Has a base distribution that is extremely stable and does not
>  version bump often across the "platform" layer of libraries, etc
>  - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  - Has a broader downstream ecosystem of ISVs and products that require
>  stability and continuity
>  
>  A default module stream in Fedora, which is only around for 13 months,
>  provides little value when the next Fedora release likely is going to
>  have a different default or a newer version in 6mo anyway.  Fedora
>  moves the entire distribution too fast for there to be a lot of usage
>  there.  RHEL can pin on a default and have that be the default on a
>  long enough timeline that it actually works. 
I have most definitely been less involved than Miro on the modularity work, so I
may be wrong (and will happily stand corrected if so) about the following.
If I understood correctly one of the major issue with default streams were
basically upgrades: how do we go from Fn to Fn+1?

RHEL has the advantages that your points #1 and #3 above high-light, however, I
expect that the upgrade question of default stream will still show up between
say RHEL8 and RHEL9.

I seem to recall that RHEL does not officially support upgrades from version N
to N+1, is that correct?

If so, there is right here a key difference between Fedora and RHEL: the life
cycle of a default stream is fixed to the life cycle of the underlying OS
without supported possibilities to move from version N to N+1.
If we were to do that in Fedora (not support N -> N+1), it is likely that the
default stream question would be much less problematic and vice-versa if Red Hat
start supporting upgrades from RHEL8 to RHEL9, the default stream question would
likely become much more challenging there.


Pierre

[Josh Boyer]

On Tue, Jun 23, 2020 at 10:14 AM Pierre-Yves Chibon <pingou(a)pingoured.fr&gt; wrote:
>  On Tue, Jun 23, 2020 at 08:30:37AM -0400, Josh Boyer wrote:
>  > On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  > >
>  > > On 22. 06. 20 21:36, Josh Boyer wrote:
>  > > >> I'd like to ask whether RHEL 9 has decided for default modular
> streams despite
>  > > >> their failure in Fedora, whether this decision is final and what was
> the
>  > > >> reasoning behind it.
>  > > >
>  > > > That's an interesting question.  I think for the purposes of this
>  > > > discussion, we should acknowledge that usage of default module streams
>  > > > in Fedora and usage in RHEL aren't equivalent.  Therefore, failure of
>  > > > adoption in Fedora doesn't necessarily equate to failure in RHEL.
>  > > > With that context, I'll continue.
>  > >
>  > > Before we continue with that context, could you please elaborate on this?
>  >
>  > If I must.
>  >
>  > > Obviously  we can say "usage of default module streams in Fedora and RHEL
> is
>  > > different" and to some extent this will always be true. However I would 
> like to
>  > > know why they are *significantly* different to justify saying  the failure in
>  > > Fedora does not necessarily mean RHEL would experience  the same failure.
>  >
>  > For all the same reasons SCLs and containers are widely used and
>  > adopted in RHEL, but not Fedora.  (And before people say "Fedora has
>  > containers", I know that.  They're fine.  That doesn't mean you see a
>  > massive adoption of Fedora base images on a wide scale.)
>  >
>  > > What makes RHEL so different that the failure  is not relevant to it? Is it
> the
>  > > stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  > > the less "wild" development  process? Is it something different?
>  >
>  > Primarily, RHEL:
>  >
>  > - Moves much much slower
>  > - Has a base distribution that is extremely stable and does not
>  > version bump often across the "platform" layer of libraries, etc
>  > - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  > - Has a broader downstream ecosystem of ISVs and products that require
>  > stability and continuity
>  >
>  > A default module stream in Fedora, which is only around for 13 months,
>  > provides little value when the next Fedora release likely is going to
>  > have a different default or a newer version in 6mo anyway.  Fedora
>  > moves the entire distribution too fast for there to be a lot of usage
>  > there.  RHEL can pin on a default and have that be the default on a
>  > long enough timeline that it actually works.
>
>  I have most definitely been less involved than Miro on the modularity work, so I
>  may be wrong (and will happily stand corrected if so) about the following.
>  If I understood correctly one of the major issue with default streams were
>  basically upgrades: how do we go from Fn to Fn+1?
>
>  RHEL has the advantages that your points #1 and #3 above high-light, however, I
>  expect that the upgrade question of default stream will still show up between
>  say RHEL8 and RHEL9. 
It will, but it is different than how it presents itself in Fedora.

>  I seem to recall that RHEL does not officially support upgrades from
> version N
>  to N+1, is that correct? 
No.  There are some constraints around what is supported but we have
tooling that aids upgrades from RHEL 7 to RHEL 8, for example.  It
analyzes and assists customers in doing that operation.  You can find
more information here:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/...

>  If so, there is right here a key difference between Fedora and RHEL:
> the life
>  cycle of a default stream is fixed to the life cycle of the underlying OS
>  without supported possibilities to move from version N to N+1.
>  If we were to do that in Fedora (not support N -> N+1), it is likely that the
>  default stream question would be much less problematic and vice-versa if Red Hat
>  start supporting upgrades from RHEL8 to RHEL9, the default stream question would
>  likely become much more challenging there. 
Indeed, RHEL 8 -> RHEL 9 will have a challenge in some ways.  However,
a major RHEL release is also very different from a Fedora release, at
least in this regard.  Major RHEL releases are boundary points where
defaults can and will change, which is possible because our lifecycle
is so long.  Within a single RHEL major version though, the defaults
don't change from one minor release to another (e.g. 8.2 -> 8.3).

Fedora actually tries to mix both models within its release framework
today, and struggles with it because the lifecycle is so short from
one release to another.  It is somewhat unique to Fedora, and it's why
it takes 3-4 releases to make a default change.

josh

[Fabio Valentini]

On Tue, Jun 23, 2020 at 2:31 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
>  > What makes RHEL so different that the failure  is not relevant to it? Is it the
>  > stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  > the less "wild" development  process? Is it something different?
>
>  Primarily, RHEL:
>
>  - Moves much much slower
>  - Has a base distribution that is extremely stable and does not
>  version bump often across the "platform" layer of libraries, etc
>  - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  - Has a broader downstream ecosystem of ISVs and products that require
>  stability and continuity
>
>  A default module stream in Fedora, which is only around for 13 months,
>  provides little value when the next Fedora release likely is going to
>  have a different default or a newer version in 6mo anyway.  Fedora
>  moves the entire distribution too fast for there to be a lot of usage
>  there.  RHEL can pin on a default and have that be the default on a
>  long enough timeline that it actually works.
>
>  Are there other ways to accomplish the same in RHEL?  Yes.  Does that
>  mean failure in Fedora translates to failure everywhere?  No. 
This is actually a point I've raised multiple times over the past year or so:

I appreciate that Modularity and default streams actually solve some
problems that exist in RHEL, *because* it has a pretty long lifecycle.
However, this doesn't make much sense in fedora, *because* it has a
short lifespan (~13 months) and development cycle (6 months). In the
fedora case, the decision which version of a particular package to
include for which version of fedora is easily aligned with major
fedora releases, and this goes through either the Change process, or
packagers just apply the relevant Updates Policy.
In RHEL, waiting for the next major release won't work, so Modularity
can actually solve this problem by providing alternative versions for
those who need them.

Now, because ELN is "just" a subset of rawhide rebuilt in a more
RHEL-like environment, it is pretty closely aligned with the
development cycle of fedora, and so - in my opinion - iterating on
default streams and modules would be pretty painful (and pointless)
there, *because* rawhide moves so quickly that decisions on what (and
which versions) to modularize are obsolete after months or weeks, and
have to be revisited after 6 months at most.
For this reason, I'd argue that the decision about 1) which things to
ship as modules, 2) which versions should be shipped by default / as
default streams, and 3) which alternative versions to provide as
modules, should probably happen more closely to the RHEL-next side of
things than to the fedora side of things, and not in ELN. This has the
added benefit of allowing RHEL packagers and fedora packagers to work
*together* on the *default* version of packages, and then RHEL
packagers can provide alternative versions as modules on top of that.

Fabio

[Josh Boyer]

On Tue, Jun 23, 2020 at 11:21 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  On Tue, Jun 23, 2020 at 2:31 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > On Tue, Jun 23, 2020 at 8:01 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  > > What makes RHEL so different that the failure  is not relevant to it? Is it
> the
>  > > stable nature of RHEL content? Is it  the limited scope of RHEL content? Is it
>  > > the less "wild" development  process? Is it something different?
>  >
>  > Primarily, RHEL:
>  >
>  > - Moves much much slower
>  > - Has a base distribution that is extremely stable and does not
>  > version bump often across the "platform" layer of libraries, etc
>  > - Has a lifecycle that is equivalent to 20 Fedora releases (yes, twenty)
>  > - Has a broader downstream ecosystem of ISVs and products that require
>  > stability and continuity
>  >
>  > A default module stream in Fedora, which is only around for 13 months,
>  > provides little value when the next Fedora release likely is going to
>  > have a different default or a newer version in 6mo anyway.  Fedora
>  > moves the entire distribution too fast for there to be a lot of usage
>  > there.  RHEL can pin on a default and have that be the default on a
>  > long enough timeline that it actually works.
>  >
>  > Are there other ways to accomplish the same in RHEL?  Yes.  Does that
>  > mean failure in Fedora translates to failure everywhere?  No.
>
>  This is actually a point I've raised multiple times over the past year or so:
>
>  I appreciate that Modularity and default streams actually solve some
>  problems that exist in RHEL, *because* it has a pretty long lifecycle.
>  However, this doesn't make much sense in fedora, *because* it has a
>  short lifespan (~13 months) and development cycle (6 months). In the
>  fedora case, the decision which version of a particular package to
>  include for which version of fedora is easily aligned with major
>  fedora releases, and this goes through either the Change process, or
>  packagers just apply the relevant Updates Policy.
>  In RHEL, waiting for the next major release won't work, so Modularity
>  can actually solve this problem by providing alternative versions for
>  those who need them.
>
>  Now, because ELN is "just" a subset of rawhide rebuilt in a more
>  RHEL-like environment, it is pretty closely aligned with the
>  development cycle of fedora, and so - in my opinion - iterating on
>  default streams and modules would be pretty painful (and pointless)
>  there, *because* rawhide moves so quickly that decisions on what (and
>  which versions) to modularize are obsolete after months or weeks, and
>  have to be revisited after 6 months at most. 
I'm not sure that follows.  Rawhide is a continuous stream.  Even
Fedora branches and stabilizes their releases in different branches.
Therefore you have an ability to iterate on something in ELN that is
longer than 6mo if it makes sense and with the proper planning.

>  For this reason, I'd argue that the decision about 1) which
> things to
>  ship as modules, 2) which versions should be shipped by default / as
>  default streams, and 3) which alternative versions to provide as
>  modules, should probably happen more closely to the RHEL-next side of
>  things than to the fedora side of things, and not in ELN. This has the 
I don't agree for 1 and 3.  There is value in doing both of those in
ELN now and I don't see why we'd take the unfortunate and expensive
option of trying to do it all internally.  Again, ELN represents a
space to do work in the open and part of that work is figuring out
which versions and modules make sense.

Item 2 may have challenges, but working on it in ELN will help
accelerate learning.  If it fails, it fails fast and we all learn.

>  added benefit of allowing RHEL packagers and fedora packagers to
> work
>  *together* on the *default* version of packages, and then RHEL
>  packagers can provide alternative versions as modules on top of that. 
We could do all the non-default stuff somewhere else.  CentOS Stream,
for example.  I actually think that would be a net negative because it
splits the development ecosystem even further and reduces the feedback
loop with Fedora that we want for major release development.  The more
experimentation and iteration we have to do *not* in Fedora, the less
relevance and influence Fedora has.

josh

[clime]

On Thu, 18 Jun 2020 at 15:25, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
>  <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  >
>  > -----BEGIN PGP SIGNED MESSAGE-----
>  > Hash: SHA512
>  >
>  > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>  > > Hello Fedora Community!
>  >
>  > Hi Josh,
>  >
>  > > I am a long-time Fedora Community member, and may be familiar to many
>  > > through previous FESCo or devel list discussions and passionate
>  > > debates.  However I write to you today with a different community hat
>  > > on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  > > organization has been following the modularity discussions within
>  > > Fedora, particularly around ELN, and often the question of what plans
>  > > we have for modularity in RHEL 9 has come up.  Our Fedora Project
>  > > Lead
>  > > and a number of FESCo members have reached out and asked if we can
>  > > provide some perspective here, and I am both happy and excited to
>  > > have
>  > > that opportunity.
>  > >
>  > > As the Fedora Council has pointed out [1], we certainly acknowledge
>  > > there are improvements to be made and have a team already working on
>  > > them.  They recently outlined their plans in conjunction with our
>  > > Product Management team in a Fedora Council call as well [2].  We’re
>  > > continuing to invest time and effort in this packaging solution and
>  > > are confident that the team can deliver against their plan.  It is
>  > > somewhat of a new experience for all of us when Red Hat is direct
>  > > with
>  > > our product intentions, but we discussed the larger gaps we see with
>  > > usage in RHEL and are putting our efforts towards solving those gaps
>  > > with this plan.
>  > >
>  > > Modularity is important to RHEL and those efforts are already
>  > > underway.  We will be leveraging modularity in RHEL 9 where it most
>  > > makes sense.  This is primarily centered around our Application
>  > > Streams concept, which has been well received by our customer base.
>  > > Providing a consistent but improved experience is the base
>  > > requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  > > and lowers the hurdle for our customers when upgrading from one major
>  > > version to another.
>  >
>  > It is nice to hear that it is helping to solve problems in RHEL (even
>  > though I've heard many people saying that it is nightmare now). Is
>  > there a list of requirements that you have so that we could potentially
>  > develop something that would be useful to Fedora same as for RHEL 10+?
>
>  The dnf team is working to gather those internally.  RHEL 10+ is still
>  ~5 years away, and while we're working hard to develop our product
>  roadmap, that's still far enough off that we haven't put much down in
>  terms of requirements :)
>
>  > > It is always good to push the boundaries and search for better ideas
>  > > and improvements, and that is part of what makes Fedora great.  We
>  > > are
>  > > doing this in the context of the RHEL 9 release as well, so our near
>  > > term timeline and requirements mean we are working on evolving
>  > > modularity, not a revolution or a replacement.  We are excited by
>  > > ELN,
>  > > as it presents a possible space to allow those that want to continue
>  > > to iterate on modules a place to do so without necessarily impacting
>  > > the broader Fedora distribution in its entirety.  It is my personal
>  > > hope that we can use that opportunity to improve modules and
>  > > modularity in the open source, Fedora-first way we’d prefer.  Our
>  > > near
>  > > term effort to improve the existing modularity implementation ahead
>  > > of
>  > > RHEL 9 needs to occur, and we’d like to do that work in Fedora,
>  > > rather
>  > > than in closed product development.  Longer term, we are open to
>  > > contributing to a better replacement that meets many of the same
>  > > goals.  This is what makes our distribution ecosystem work well, and
>  > > not having that upstream lessens the value we all get from such
>  > > experimentation in the open.
>  >
>  > While I support you that we should do it in Fedora, does this
>  > essentially mean that this technology is useful only for RHEL and you
>  > do not plan to develop it *for Fedora*, but rather *for RHEL in
>  > Fedora*?
>
>  I wouldn't say that, nor do I personally think that.  I think there is
>  value for Fedora as well.  Matthew Miller has often given a common
>  example of having two streams of software that can build across RHEL,
>  Fedora, CentOS, etc.  This is value for end user consumption, and
>  having older software available in Fedora is a usecase modularity can
>  help address.  However, I would prefer to avoid discussing value to
>  Fedora in this thread.  There are so many other threads debating that
>  already :)
>
>  > > Hopefully that provides some context and helps FESCo and the wider
>  > > community understand where Red Hat is headed with modularity on the
>  > > Enterprise side.
>  >
>  > Sadly no. It helps to understand your plans, however it does not help
>  > to understand the reasons behind, whether you can't change UX in the
>  > RHEL 9, or you think that technology is good enough for your use-cases
>  > or any other reasons.
>
>  The base requirement is that the UX remain largely the same.  As I
>  said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  commonality so that our customers are not forced to learn something
>  entirely different to adopt RHEL 9.  Improvements in the underlying
>  functionality are of course welcome and planned, but we are not going
>  to do something like replace modules with a different artifact type, 
Hello Josh,

you can change the artifact type while keeping interface the same and
it would be a _HUGE_ win because it would make modularity finally
understandable for mere humans and better maintainable.

Namely, modules should become rpms and therefore obey standard rpm rules.

Thanks for looking at this
clime

> or add separate discrete repos per Application Stream, etc.
>
> > Basically this email just says "We decided for Modularity in RHEL 9 and
> > we would like to do it in Fedora Infrastructure first".
>
> Mostly, yes.  We were told there was ambiguity on whether modularity
> would be used in RHEL 9 or not.  Very clearly it will, so I wanted to
> get ahead of that.
>
> josh
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Josh Boyer]

On Thu, Jun 18, 2020 at 5:51 PM clime <clime(a)fedoraproject.org&gt; wrote:
>  On Thu, 18 Jun 2020 at 15:25, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  >
>  > On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
>  > <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > >
>  > > -----BEGIN PGP SIGNED MESSAGE-----
>  > > Hash: SHA512
>  > >
>  > > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote: 
<snip>

>  > > > Hopefully that provides some context and helps FESCo
> and the wider
>  > > > community understand where Red Hat is headed with modularity on the
>  > > > Enterprise side.
>  > >
>  > > Sadly no. It helps to understand your plans, however it does not help
>  > > to understand the reasons behind, whether you can't change UX in the
>  > > RHEL 9, or you think that technology is good enough for your use-cases
>  > > or any other reasons.
>  >
>  > The base requirement is that the UX remain largely the same.  As I
>  > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  > commonality so that our customers are not forced to learn something
>  > entirely different to adopt RHEL 9.  Improvements in the underlying
>  > functionality are of course welcome and planned, but we are not going
>  > to do something like replace modules with a different artifact type,
>
>  Hello Josh,
>
>  you can change the artifact type while keeping interface the same and
>  it would be a _HUGE_ win because it would make modularity finally
>  understandable for mere humans and better maintainable.
>
>  Namely, modules should become rpms and therefore obey standard rpm rules. 
I'm not sure I entirely understand what you mean, but it sounds like
you have some interesting ideas.

I'm looking forward to seeing what you and the community can build
from them, and how they could be brought into RHEL 10+!  That kind of
collaboration is what makes Fedora great.

josh

[clime]

On Fri, 19 Jun 2020 at 01:59, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 5:51 PM clime <clime(a)fedoraproject.org&gt; wrote:
>  >
>  > On Thu, 18 Jun 2020 at 15:25, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > >
>  > > On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
>  > > <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > > >
>  > > > -----BEGIN PGP SIGNED MESSAGE-----
>  > > > Hash: SHA512
>  > > >
>  > > > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
>
>  <snip>
>
>  > > > > Hopefully that provides some context and helps FESCo and the wider
>  > > > > community understand where Red Hat is headed with modularity on the
>  > > > > Enterprise side.
>  > > >
>  > > > Sadly no. It helps to understand your plans, however it does not help
>  > > > to understand the reasons behind, whether you can't change UX in the
>  > > > RHEL 9, or you think that technology is good enough for your use-cases
>  > > > or any other reasons.
>  > >
>  > > The base requirement is that the UX remain largely the same.  As I
>  > > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  > > commonality so that our customers are not forced to learn something
>  > > entirely different to adopt RHEL 9.  Improvements in the underlying
>  > > functionality are of course welcome and planned, but we are not going
>  > > to do something like replace modules with a different artifact type,
>  >
>  > Hello Josh,
>  >
>  > you can change the artifact type while keeping interface the same and
>  > it would be a _HUGE_ win because it would make modularity finally
>  > understandable for mere humans and better maintainable.
>  >
>  > Namely, modules should become rpms and therefore obey standard rpm rules.
>
>  I'm not sure I entirely understand what you mean, but it sounds like
>  you have some interesting ideas.
>
>  I'm looking forward to seeing what you and the community can build
>  from them, and how they could be brought into RHEL 10+!  That kind of
>  collaboration is what makes Fedora great. 
I know this probably won't change anything because this was mentioned
many times (by me at least) and nothing has changed but still...

Currently, modules are essentially yum sub-repos, they are not really
"modules", instead they are collections of rpms that reinvent rpm-like
relations (obsoletes, requires, build-requires, etc.).

There is no reason for this wheel-reintervention. Modules (the
collections) can be simply squashed into an rpm by automation and this
resulting rpm can go to the modular repo together with other modules.

That way we don't have two types of objects we complex inter-relations
but only one we well-known behavior.

I wonder if this is clear to everyone but nobody really cares or
doesn't really want to say it or I don't know.

Is this clear to everyone? I mean either I am stating an obvious stuff
that nobody really considers worth typing or idk.

Anyway best regards
clime

>
> josh
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Naheem Zaffar]

On Mon, 22 Jun 2020, 02:57 clime, <clime(a)fedoraproject.org&gt; wrote:

>  On Fri, 19 Jun 2020 at 01:59, Josh Boyer <jwboyer(a)redhat.com&gt;
> wrote:
> >  
>  > On Thu, Jun 18, 2020 at 5:51 PM clime
> <clime(a)fedoraproject.org&gt; wrote:
>  >
> >  
>  > > On Thu, 18 Jun 2020 at 15:25, Josh Boyer
> <jwboyer(a)redhat.com&gt; wrote:
>  > >
> >  
>  > > > On Thu, Jun 18, 2020
> at 9:05 AM Igor Raits
>  > > > <ignatenkobrain(a)fedoraproject.org&gt; wrote:
>  > > >
> >  
>  > > > > -----BEGIN PGP
> SIGNED MESSAGE-----
>  > > > > Hash: SHA512
>  > > >
> >  
>  > > > > On Thu,
> 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
> >  
>  > <snip
> >  
> >  
>  > > > > > Hopefully that provides some context
> and helps FESCo and the
>  wider
>  > > > > > community understand where Red Hat is headed with modularity on
>  the
>  > > > > > Enterprise side.
>  > > >
> >  
>  > > > > Sadly no. It
> helps to understand your plans, however it does not
>  help
>  > > > > to understand the reasons behind, whether you can't change UX in
>  the
>  > > > > RHEL 9, or you think that technology is good enough for your
>  use-cases
>  > > > > or any other reasons.
>  > >
> >  
>  > > > The base requirement
> is that the UX remain largely the same.  As I
>  > > > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
>  > > > commonality so that our customers are not forced to learn something
>  > > > entirely different to adopt RHEL 9.  Improvements in the underlying
>  > > > functionality are of course welcome and planned, but we are not going
>  > > > to do something like replace modules with a different artifact type,
>  >
> >  
>  > > Hello Josh,
>  >
> >  
>  > > you can change the artifact type while keeping
> interface the same and
>  > > it would be a _HUGE_ win because it would make modularity finally
>  > > understandable for mere humans and better maintainable.
>  >
> >  
>  > > Namely, modules should become rpms and therefore
> obey standard rpm
>  rules.
> >  
>  > I'm not sure I entirely understand what you mean, but
> it sounds like
>  > you have some interesting ideas.
> >  
>  > I'm looking forward to seeing what you and the
> community can build
>  > from them, and how they could be brought into RHEL 10+!  That kind of
>  > collaboration is what makes Fedora great.
>
>  I know this probably won't change anything because this was mentioned
>  many times (by me at least) and nothing has changed but still...
>
>  Currently, modules are essentially yum sub-repos, they are not really
>  "modules", instead they are collections of rpms that reinvent rpm-like
>  relations (obsoletes, requires, build-requires, etc.).
>
>  There is no reason for this wheel-reintervention. Modules (the
>  collections) can be simply squashed into an rpm by automation and this
>  resulting rpm can go to the modular repo together with other modules.
>
>  That way we don't have two types of objects we complex inter-relations
>  but only one we well-known behavior.
>
>  I wonder if this is clear to everyone but nobody really cares or
>  doesn't really want to say it or I don't know.
>
>  Is this clear to everyone? I mean either I am stating an obvious stuff
>  that nobody really considers worth typing or idk.
>  
How would this work when there are optional rpms in the module?

You do not need to install every rpm in  eg the php module (different
graphics/database backends) for that module to be useful, but every version
of the module will have the rpm as an option which wont work outside a
module of multiple rpms.

>

[clime]

On Mon, 22 Jun 2020 at 04:12, Naheem Zaffar <naheemzaffar(a)gmail.com&gt; wrote:
>  On Mon, 22 Jun 2020, 02:57 clime, <clime(a)fedoraproject.org&gt; wrote:
> >
> > On Fri, 19 Jun 2020 at 01:59, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
> > >
> > > On Thu, Jun 18, 2020 at 5:51 PM clime <clime(a)fedoraproject.org&gt; wrote:
> > > >
> > > > On Thu, 18 Jun 2020 at 15:25, Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
> > > > >
> > > > > On Thu, Jun 18, 2020 at 9:05 AM Igor Raits
> > > > > <ignatenkobrain(a)fedoraproject.org&gt; wrote:
> > > > > >
> > > > > > -----BEGIN PGP SIGNED MESSAGE-----
> > > > > > Hash: SHA512
> > > > > >
> > > > > > On Thu, 2020-06-18 at 08:44 -0400, Josh Boyer wrote:
> > >
> > > <snip>
> > >
> > > > > > > Hopefully that provides some context and helps FESCo and the
> wider
> > > > > > > community understand where Red Hat is headed with modularity
> on the
> > > > > > > Enterprise side.
> > > > > >
> > > > > > Sadly no. It helps to understand your plans, however it does not
> help
> > > > > > to understand the reasons behind, whether you can't change UX
> in the
> > > > > > RHEL 9, or you think that technology is good enough for your
> use-cases
> > > > > > or any other reasons.
> > > > >
> > > > > The base requirement is that the UX remain largely the same.  As I
> > > > > said, from a RHEL perspective, we need RHEL 8 and RHEL 9 to have
> > > > > commonality so that our customers are not forced to learn something
> > > > > entirely different to adopt RHEL 9.  Improvements in the underlying
> > > > > functionality are of course welcome and planned, but we are not going
> > > > > to do something like replace modules with a different artifact type,
> > > >
> > > > Hello Josh,
> > > >
> > > > you can change the artifact type while keeping interface the same and
> > > > it would be a _HUGE_ win because it would make modularity finally
> > > > understandable for mere humans and better maintainable.
> > > >
> > > > Namely, modules should become rpms and therefore obey standard rpm rules.
> > >
> > > I'm not sure I entirely understand what you mean, but it sounds like
> > > you have some interesting ideas.
> > >
> > > I'm looking forward to seeing what you and the community can build
> > > from them, and how they could be brought into RHEL 10+!  That kind of
> > > collaboration is what makes Fedora great.
> >
> > I know this probably won't change anything because this was mentioned
> > many times (by me at least) and nothing has changed but still...
> >
> > Currently, modules are essentially yum sub-repos, they are not really
> > "modules", instead they are collections of rpms that reinvent rpm-like
> > relations (obsoletes, requires, build-requires, etc.).
> >
> > There is no reason for this wheel-reintervention. Modules (the
> > collections) can be simply squashed into an rpm by automation and this
> > resulting rpm can go to the modular repo together with other modules.
> >
> > That way we don't have two types of objects we complex inter-relations
> > but only one we well-known behavior.
> >
> > I wonder if this is clear to everyone but nobody really cares or
> > doesn't really want to say it or I don't know.
> >
> > Is this clear to everyone? I mean either I am stating an obvious stuff
> > that nobody really considers worth typing or idk.
>
>
>  How would this work when there are optional rpms in the module?
>
>  You do not need to install every rpm in  eg the php module (different graphics/database
> backends) for that module to be useful, but every version of the module will have the rpm
> as an option which wont work outside a module of multiple rpms. 
Glad you ask, I wasn't precise...

Well, I didn't mean everything always needs to be squashed, instead,
it would be an optional step in modulemd processing. For some
use-cases (like delicately compiled postgresql server), you can create
a single rpm that contains all - postgresql-server, postgresql,
postgresql-libs compiled in a specific way, optionally with some
postgresql modules pre-included, so it would be let's say time-series
optimized postgresql. Here it makes sense to make a single rpm from it
- you install that and you are all set up for your use-case.

Then there are language stacks where you might want to build things in
a specific order - there nothing really needs to be squashed (or
certain subset can if it makes sense) but you can still use modularity
to easily batch-build certain rpms. If there are runtime optional
deps, they can be described by Recommends/Suggests.

Basically, once a "module" (things that comes from modulemd) is built,
it should be put into normal repos and the "module" boundary should be
forgotten (unless it is a single rpm), i.e. "module" is a built-time
thing, at install-time we just have standard packages with standard
deps.

dnf interface could be kept given that we "Stream" rpm property is
added. This is still a bit rough what I am saying but hopefully it
makes at least a bit of sense...

clime

>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Zbigniew Jędrzejewski-Szmek]

On Mon, Jun 22, 2020 at 04:55:10AM +0200, clime wrote:
>  >> > > Hello Josh,
>  >> > >
>  >> > > you can change the artifact type while keeping interface the same and
>  >> > > it would be a _HUGE_ win because it would make modularity finally
>  >> > > understandable for mere humans and better maintainable.
>  >> > >
>  >> > > Namely, modules should become rpms and therefore obey standard rpm
> rules.
>  >> >
>  >> > I'm not sure I entirely understand what you mean, but it sounds like
>  >> > you have some interesting ideas.
>  >> >
>  >> > I'm looking forward to seeing what you and the community can build
>  >> > from them, and how they could be brought into RHEL 10+!  That kind of
>  >> > collaboration is what makes Fedora great.
>  >>
>  >> I know this probably won't change anything because this was mentioned
>  >> many times (by me at least) and nothing has changed but still...
>  >>
>  >> Currently, modules are essentially yum sub-repos, they are not really
>  >> "modules", instead they are collections of rpms that reinvent
> rpm-like
>  >> relations (obsoletes, requires, build-requires, etc.).
>  >>
>  >> There is no reason for this wheel-reintervention. Modules (the
>  >> collections) can be simply squashed into an rpm by automation and this
>  >> resulting rpm can go to the modular repo together with other modules. 
I agree with this general idea, even if not with the exact implementation
(comments below). In the past this was stated as "divorcing the build ordering
mechanism from the rpm delivery mechanism". The fact that we have two layers
of dependencies make Modularity conceptually hard and destroy the interaction
with the dependency solver. Also, if we disconnect the build and delivery
mechanisms, we can iterate and improve both separately.
 
>  >> That way we don't have two types of objects we complex
> inter-relations
>  >> but only one we well-known behavior.
>  >>
>  >> I wonder if this is clear to everyone but nobody really cares or
>  >> doesn't really want to say it or I don't know.
>  >>
>  >> Is this clear to everyone? I mean either I am stating an obvious stuff
>  >> that nobody really considers worth typing or idk.
>  >
>  >
>  > How would this work when there are optional rpms in the module?
>  >
>  > You do not need to install every rpm in  eg the php module (different
> graphics/database backends) for that module to be useful, but every version of the module
> will have the rpm as an option which wont work outside a module of multiple rpms.
>  
>  Glad you ask, I wasn't precise...
>  
>  Well, I didn't mean everything always needs to be squashed, instead,
>  it would be an optional step in modulemd processing. 
So... if it's only optional, that means that the general case where
squashing is not done needs to be solved anyway. And once you have
solved the general case, what would the point of squashing be?
Thus, I don't find squashing useful.

>  For some
>  use-cases (like delicately compiled postgresql server), you can create
>  a single rpm that contains all - postgresql-server, postgresql,
>  postgresql-libs compiled in a specific way, optionally with some
>  postgresql modules pre-included, so it would be let's say time-series
>  optimized postgresql. Here it makes sense to make a single rpm from it
>  - you install that and you are all set up for your use-case.
>  
>  Then there are language stacks where you might want to build things in
>  a specific order - there nothing really needs to be squashed (or
>  certain subset can if it makes sense) but you can still use modularity
>  to easily batch-build certain rpms. If there are runtime optional
>  deps, they can be described by Recommends/Suggests.
>  
>  Basically, once a "module" (things that comes from modulemd) is built,
>  it should be put into normal repos and the "module" boundary should be
>  forgotten (unless it is a single rpm), i.e. "module" is a built-time
>  thing, at install-time we just have standard packages with standard
>  deps. 
Yep.

The unanswered question is what mechanism would be used make sure that
the rpms from the "module" are all installed. One option would be to
somehow mangle rpm names, another option would be to add some kind of
Provides/Requires, etc. But *some* mechanism is needed, because without
that dnf would often pick other rpms.

In Modularity the solution is that the rpms from the module shadow
rpms with the same name from outside. That's probably the single
feature of Modularity that causes the most problems.

>  dnf interface could be kept given that we "Stream" rpm
> property is
>  added. This is still a bit rough what I am saying but hopefully it
>  makes at least a bit of sense... 
Zbyszek

[clime]

On Mon, 22 Jun 2020 at 08:14, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Mon, Jun 22, 2020 at 04:55:10AM +0200, clime wrote:
>  > >> > > Hello Josh,
>  > >> > >
>  > >> > > you can change the artifact type while keeping interface the same
> and
>  > >> > > it would be a _HUGE_ win because it would make modularity
> finally
>  > >> > > understandable for mere humans and better maintainable.
>  > >> > >
>  > >> > > Namely, modules should become rpms and therefore obey standard
> rpm rules.
>  > >> >
>  > >> > I'm not sure I entirely understand what you mean, but it sounds
> like
>  > >> > you have some interesting ideas.
>  > >> >
>  > >> > I'm looking forward to seeing what you and the community can
> build
>  > >> > from them, and how they could be brought into RHEL 10+!  That kind of
>  > >> > collaboration is what makes Fedora great.
>  > >>
>  > >> I know this probably won't change anything because this was mentioned
>  > >> many times (by me at least) and nothing has changed but still...
>  > >>
>  > >> Currently, modules are essentially yum sub-repos, they are not really
>  > >> "modules", instead they are collections of rpms that reinvent
> rpm-like
>  > >> relations (obsoletes, requires, build-requires, etc.).
>  > >>
>  > >> There is no reason for this wheel-reintervention. Modules (the
>  > >> collections) can be simply squashed into an rpm by automation and this
>  > >> resulting rpm can go to the modular repo together with other modules.
>
>  I agree with this general idea, even if not with the exact implementation
>  (comments below). In the past this was stated as "divorcing the build ordering
>  mechanism from the rpm delivery mechanism". The fact that we have two layers
>  of dependencies make Modularity conceptually hard and destroy the interaction
>  with the dependency solver. Also, if we disconnect the build and delivery
>  mechanisms, we can iterate and improve both separately. 
Indeed, I agree build and delivery mechanisms should be treated independently.

Things would finally get a clear shape.

>
> > >> That way we don't have two types of objects we complex
inter-relations
> > >> but only one we well-known behavior.
> > >>
> > >> I wonder if this is clear to everyone but nobody really cares or
> > >> doesn't really want to say it or I don't know.
> > >>
> > >> Is this clear to everyone? I mean either I am stating an obvious stuff
> > >> that nobody really considers worth typing or idk.
> > >
> > >
> > > How would this work when there are optional rpms in the module?
> > >
> > > You do not need to install every rpm in  eg the php module (different
graphics/database backends) for that module to be useful, but every version of the module
will have the rpm as an option which wont work outside a module of multiple rpms.
> >
> > Glad you ask, I wasn't precise...
> >
> > Well, I didn't mean everything always needs to be squashed, instead,
> > it would be an optional step in modulemd processing.
>
> So... if it's only optional, that means that the general case where
> squashing is not done needs to be solved anyway. And once you have
> solved the general case, what would the point of squashing be?
> Thus, I don't find squashing useful.
>
> > For some
> > use-cases (like delicately compiled postgresql server), you can create
> > a single rpm that contains all - postgresql-server, postgresql,
> > postgresql-libs compiled in a specific way, optionally with some
> > postgresql modules pre-included, so it would be let's say time-series
> > optimized postgresql. Here it makes sense to make a single rpm from it
> > - you install that and you are all set up for your use-case.
> >
> > Then there are language stacks where you might want to build things in
> > a specific order - there nothing really needs to be squashed (or
> > certain subset can if it makes sense) but you can still use modularity
> > to easily batch-build certain rpms. If there are runtime optional
> > deps, they can be described by Recommends/Suggests.
> >
> > Basically, once a "module" (things that comes from modulemd) is
built,
> > it should be put into normal repos and the "module" boundary should
be
> > forgotten (unless it is a single rpm), i.e. "module" is a built-time
> > thing, at install-time we just have standard packages with standard
> > deps.
>
> Yep.
>
> The unanswered question is what mechanism would be used make sure that
> the rpms from the "module" are all installed. One option would be to
> somehow mangle rpm names, another option would be to add some kind of
> Provides/Requires, etc. But *some* mechanism is needed, because without
> that dnf would often pick other rpms.
>
> In Modularity the solution is that the rpms from the module shadow
> rpms with the same name from outside. That's probably the single
> feature of Modularity that causes the most problems.
>
> > dnf interface could be kept given that we "Stream" rpm property is
> > added. This is still a bit rough what I am saying but hopefully it
> > makes at least a bit of sense...
>
> Zbyszek
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[clime]

On Mon, 22 Jun 2020 at 08:14, Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Mon, Jun 22, 2020 at 04:55:10AM +0200, clime wrote:
>  > >> > > Hello Josh,
>  > >> > >
>  > >> > > you can change the artifact type while keeping interface the same
> and
>  > >> > > it would be a _HUGE_ win because it would make modularity
> finally
>  > >> > > understandable for mere humans and better maintainable.
>  > >> > >
>  > >> > > Namely, modules should become rpms and therefore obey standard
> rpm rules.
>  > >> >
>  > >> > I'm not sure I entirely understand what you mean, but it sounds
> like
>  > >> > you have some interesting ideas.
>  > >> >
>  > >> > I'm looking forward to seeing what you and the community can
> build
>  > >> > from them, and how they could be brought into RHEL 10+!  That kind of
>  > >> > collaboration is what makes Fedora great.
>  > >>
>  > >> I know this probably won't change anything because this was mentioned
>  > >> many times (by me at least) and nothing has changed but still...
>  > >>
>  > >> Currently, modules are essentially yum sub-repos, they are not really
>  > >> "modules", instead they are collections of rpms that reinvent
> rpm-like
>  > >> relations (obsoletes, requires, build-requires, etc.).
>  > >>
>  > >> There is no reason for this wheel-reintervention. Modules (the
>  > >> collections) can be simply squashed into an rpm by automation and this
>  > >> resulting rpm can go to the modular repo together with other modules.
>
>  I agree with this general idea, even if not with the exact implementation
>  (comments below). In the past this was stated as "divorcing the build ordering
>  mechanism from the rpm delivery mechanism". The fact that we have two layers
>  of dependencies make Modularity conceptually hard and destroy the interaction
>  with the dependency solver. Also, if we disconnect the build and delivery
>  mechanisms, we can iterate and improve both separately.
>
>  > >> That way we don't have two types of objects we complex inter-relations
>  > >> but only one we well-known behavior.
>  > >>
>  > >> I wonder if this is clear to everyone but nobody really cares or
>  > >> doesn't really want to say it or I don't know.
>  > >>
>  > >> Is this clear to everyone? I mean either I am stating an obvious stuff
>  > >> that nobody really considers worth typing or idk.
>  > >
>  > >
>  > > How would this work when there are optional rpms in the module?
>  > >
>  > > You do not need to install every rpm in  eg the php module (different
> graphics/database backends) for that module to be useful, but every version of the module
> will have the rpm as an option which wont work outside a module of multiple rpms.
>  >
>  > Glad you ask, I wasn't precise...
>  >
>  > Well, I didn't mean everything always needs to be squashed, instead,
>  > it would be an optional step in modulemd processing.
>
>  So... if it's only optional, that means that the general case where
>  squashing is not done needs to be solved anyway. And once you have
>  solved the general case, what would the point of squashing be?
>  Thus, I don't find squashing useful.
>
>  > For some
>  > use-cases (like delicately compiled postgresql server), you can create
>  > a single rpm that contains all - postgresql-server, postgresql,
>  > postgresql-libs compiled in a specific way, optionally with some
>  > postgresql modules pre-included, so it would be let's say time-series
>  > optimized postgresql. Here it makes sense to make a single rpm from it
>  > - you install that and you are all set up for your use-case.
>  >
>  > Then there are language stacks where you might want to build things in
>  > a specific order - there nothing really needs to be squashed (or
>  > certain subset can if it makes sense) but you can still use modularity
>  > to easily batch-build certain rpms. If there are runtime optional
>  > deps, they can be described by Recommends/Suggests.
>  >
>  > Basically, once a "module" (things that comes from modulemd) is built,
>  > it should be put into normal repos and the "module" boundary should be
>  > forgotten (unless it is a single rpm), i.e. "module" is a built-time
>  > thing, at install-time we just have standard packages with standard
>  > deps.
>
>  Yep.
>
>  The unanswered question is what mechanism would be used make sure that
>  the rpms from the "module" are all installed. One option would be to
>  somehow mangle rpm names, another option would be to add some kind of
>  Provides/Requires, etc. But *some* mechanism is needed, because without
>  that dnf would often pick other rpms.
>
>  In Modularity the solution is that the rpms from the module shadow
>  rpms with the same name from outside. That's probably the single
>  feature of Modularity that causes the most problems. 
Yeah, I could notice modularity causes quite a lot of troubles.

The question is whether those troubles can be somehow technically
justified, i.e. by the benefit modularity will eventually bring.

But I personally don't see it. Actually, (big revelation) I have never
understood modularity and I was even there in RH when it was just
starting to be born. Probably, one of the reasons why I didn't
understand it at that point was that there seemed to be no clear
specification floating around. We knew modularity should solve "too
fast, too slow" problem of distributions but it wasn't exactly clear
how. It was a cool buzz word but nobody seemed to know what it means
(at least that was my view of the situation).

I and perhaps others were thinking that it wants to provide parallel
availability+installability of the distribution software but after
some time, it was cleared out this isn't the case and that the goal is
just "parallel availability".

OK, but even this isn't clear to me.

I mean I understand the usefulness of modularity for build-time where
you can create a recipe (modulemd) that will build your packages with
interdependencies in a predictable and automatic manner. I think
that's a cool thing to have.

But what about run-time (or install-time in other words)? That's the
part I don't understand. And I have spent quite some time trying to
understand it but never managed. So it's possible that I am missing
something all these years...in that case, it would be great if
somebody could shed a light on it for me. Here, I would really like to
give modularity a chance.

From what I understand, the use of modularity in run-time is to
provide rpm namespaces. Natural way to do this would be to use
separate repositories ala COPR where rpms are namespaced by repo ID
but I know one of the requirements of modularity was to use a single
repo for those namespaces with an argument that dnf is slow when
working with a large number of repos...to me that reason always seemed
quite artificial...something is slow...ok, then it can be made faster.
I could understand if we were talking about let's say thousands of
modules - there I would believe that initiating a thousand (or
multiplied by few) new downloads of repo files might already have its
price. But okay, if thousands of modules were the plan, then I could
understand this argument.

But now comes even more curious part. So...run-time modularity
provides rpm namespacing if I understand it correctly. Basically
<module>/<stream>/<package_name>. The easy solution for this would be
to put the namespace implicitly into package name like python does it
when there should be multiple pythons available, e.g. currently in
CentOS7/EPEL7, there is python34-requests and python36-requests (I
understand there will be a dot between major and minor at some point
so e.g. python3.6-requests but that's another thing :)). So if we have
different rpm names (because the namespace is already included in the
rpm name itself), then there is no problem to provide multiple
variants of the "same package" (the same thing but intended e.g. for a
different python interpreter) in the same repo.

So I would be willing to accept that this is a hacky solution or just
a workaround (even though I am not sure it is). But even if I accept
that it is just a hack and we need a more proper solution, I still
have an issue in my mind. Let's say we have this two-level namespacing
(<module>/<stream>/) and it enables us to have a package of the same
name twice or more times in the same repo and it enables us to avoid
mangling the rpm names. Great, isn't it? Well...but what if those
different variants of the same package are actually
parallel-installable and a user would benefit from having them
parallel-installable (because it's a dev working with different
versions of the same language at the same time)? We can only install a
package of a certain name once into the system so that's why
modularity enables us to use always just a single stream from all
available streams of a module, i.e. you can only switch between the
individual streams, having multiple of them enabled at the same time
is not possible.

So basically, modularity gives parallel-availability but at the same
time, it disables the option of parallel-installability which could be
achieved through alternatives and some smart packaging for probably
all the language stacks if I understand correctly. I think that's a
too much of a limitation. To avoid it, we would need to keep an rpm DB
per the namespace (<module>/<stream>/) and these various DBs would be
handled by dnf, which would basically mean, rpm command itself
wouldn't know what's all installed on the system - hard to imagine
that people would be alright with it. ...Or we can bring the notion of
the namespaces into rpm itself (that's where my suggestion of "Stream"
rpm attribute comes from but it could also be called just
"Namespace"). But then there is the argument: "Why not just put the
namespace into rpm name itself?" I mean...I wouldn't mind having it as
a separate attribute but the usefulness of it would need to be
discussed.

So I don't really get even after almost five years where modularity is
going or what it wants to achieve. I don't understand its use-case for
any of Fedora, RHEL, and CentOS because disabling
parallel-installability to allow parallel availability is imho not
really an option. But yeah...maybe I am missing some angle. In that
case, please, explain it to me because I would really like to
understand...

clime

>
> > dnf interface could be kept given that we "Stream" rpm property is
> > added. This is still a bit rough what I am saying but hopefully it
> > makes at least a bit of sense...
>
> Zbyszek
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Terry Bowling]

On Tue, Jun 23, 2020 at 2:33 PM clime <clime(a)fedoraproject.org&gt; wrote:

>  So I don't really get even after almost five years where modularity is
>  going or what it wants to achieve. I don't understand its use-case for
>  any of Fedora, RHEL, and CentOS because disabling
>  parallel-installability to allow parallel availability is imho not
>  really an option. But yeah...maybe I am missing some angle. In that
>  case, please, explain it to me because I would really like to
>  understand...
>
>  clime
>  
clime,
I just started a new thread where I try to share some background history
and context.  Hopefully it helps.  It should direct you to all of the other
documentation that explains most of your questions.
https://lists.fedoraproject.org/archives/list/[e-mail].o...

[clime]

On Tue, 23 Jun 2020 at 20:59, Terry Bowling <tbowling(a)redhat.com&gt; wrote:
>  On Tue, Jun 23, 2020 at 2:33 PM clime <clime(a)fedoraproject.org&gt; wrote:
> >
> >
> > So I don't really get even after almost five years where modularity is
> > going or what it wants to achieve. I don't understand its use-case for
> > any of Fedora, RHEL, and CentOS because disabling
> > parallel-installability to allow parallel availability is imho not
> > really an option. But yeah...maybe I am missing some angle. In that
> > case, please, explain it to me because I would really like to
> > understand...
> >
> > clime
>
>
>  clime,
>  I just started a new thread where I try to share some background history and context. 
> Hopefully it helps.  It should direct you to all of the other documentation that explains
> most of your questions.
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
Great, thank you.

> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Zbigniew Jędrzejewski-Szmek]

On Tue, Jun 23, 2020 at 08:31:06PM +0200, clime wrote:
>  > The unanswered question is what mechanism would be used make
> sure that
>  > the rpms from the "module" are all installed. One option would be to
>  > somehow mangle rpm names, another option would be to add some kind of
>  > Provides/Requires, etc. But *some* mechanism is needed, because without
>  > that dnf would often pick other rpms.
>  >
>  > In Modularity the solution is that the rpms from the module shadow
>  > rpms with the same name from outside. That's probably the single
>  > feature of Modularity that causes the most problems.
>  
>  Yeah, I could notice modularity causes quite a lot of troubles.
>  
>  The question is whether those troubles can be somehow technically
>  justified, i.e. by the benefit modularity will eventually bring.
>  
>  But I personally don't see it. Actually, (big revelation) I have never
>  understood modularity and I was even there in RH when it was just
>  starting to be born. Probably, one of the reasons why I didn't
>  understand it at that point was that there seemed to be no clear
>  specification floating around. We knew modularity should solve "too
>  fast, too slow" problem of distributions but it wasn't exactly clear
>  how. It was a cool buzz word but nobody seemed to know what it means
>  (at least that was my view of the situation).
>  
>  I and perhaps others were thinking that it wants to provide parallel
>  availability+installability of the distribution software but after
>  some time, it was cleared out this isn't the case and that the goal is
>  just "parallel availability".
>  
>  OK, but even this isn't clear to me.
>  
>  I mean I understand the usefulness of modularity for build-time where
>  you can create a recipe (modulemd) that will build your packages with
>  interdependencies in a predictable and automatic manner. I think
>  that's a cool thing to have.
>  
>  But what about run-time (or install-time in other words)? That's the
>  part I don't understand. And I have spent quite some time trying to
>  understand it but never managed. So it's possible that I am missing
>  something all these years...in that case, it would be great if
>  somebody could shed a light on it for me. Here, I would really like to
>  give modularity a chance.
>  
>  From what I understand, the use of modularity in run-time is to
>  provide rpm namespaces. Natural way to do this would be to use
>  separate repositories ala COPR where rpms are namespaced by repo ID
>  but I know one of the requirements of modularity was to use a single
>  repo for those namespaces with an argument that dnf is slow when
>  working with a large number of repos...to me that reason always seemed
>  quite artificial...something is slow...ok, then it can be made faster.
>  I could understand if we were talking about let's say thousands of
>  modules - there I would believe that initiating a thousand (or
>  multiplied by few) new downloads of repo files might already have its
>  price. But okay, if thousands of modules were the plan, then I could
>  understand this argument.
>  
>  But now comes even more curious part. So...run-time modularity
>  provides rpm namespacing if I understand it correctly. Basically
>  <module>/<stream>/<package_name>. The easy solution for this would be
>  to put the namespace implicitly into package name like python does it
>  when there should be multiple pythons available, e.g. currently in
>  CentOS7/EPEL7, there is python34-requests and python36-requests (I
>  understand there will be a dot between major and minor at some point
>  so e.g. python3.6-requests but that's another thing :)). So if we have
>  different rpm names (because the namespace is already included in the
>  rpm name itself), then there is no problem to provide multiple
>  variants of the "same package" (the same thing but intended e.g. for a
>  different python interpreter) in the same repo. 
Yes. Putting the "stream identification" into the package name is the
most natural solution, and has been floated various times. It seems that
it would even be possible to make the implementation relatively painless
using some clever macros. And this alone would give 80% of what
modularity offers on the delivery side. I know various people have been
working on prototypes and I hope we have some serious proposal in the end.

>  So I would be willing to accept that this is a hacky solution or
> just
>  a workaround (even though I am not sure it is). But even if I accept
>  that it is just a hack and we need a more proper solution, I still
>  have an issue in my mind. Let's say we have this two-level namespacing
>  (<module>/<stream>/) and it enables us to have a package of the same
>  name twice or more times in the same repo and it enables us to avoid
>  mangling the rpm names. Great, isn't it? Well...but what if those
>  different variants of the same package are actually
>  parallel-installable and a user would benefit from having them
>  parallel-installable (because it's a dev working with different
>  versions of the same language at the same time)? We can only install a
>  package of a certain name once into the system so that's why
>  modularity enables us to use always just a single stream from all
>  available streams of a module, i.e. you can only switch between the
>  individual streams, having multiple of them enabled at the same time
>  is not possible. 
Yep, my personal view is that Modularity makes easy things complex,
and advanced things are labelled "out of scope".

>  So basically, modularity gives parallel-availability but at the same
>  time, it disables the option of parallel-installability which could be
>  achieved through alternatives and some smart packaging for probably
>  all the language stacks if I understand correctly. I think that's a
>  too much of a limitation. To avoid it, we would need to keep an rpm DB
>  per the namespace (<module>/<stream>/) and these various DBs would be
>  handled by dnf, which would basically mean, rpm command itself
>  wouldn't know what's all installed on the system - hard to imagine
>  that people would be alright with it. ...Or we can bring the notion of
>  the namespaces into rpm itself (that's where my suggestion of "Stream"
>  rpm attribute comes from but it could also be called just
>  "Namespace"). But then there is the argument: "Why not just put the
>  namespace into rpm name itself?" I mean...I wouldn't mind having it as
>  a separate attribute but the usefulness of it would need to be
>  discussed. 
I think there's some fear that "name mangling" is not a general
solution, and we'd have cases where names conflict. I think the
concern is realistic, but not a big issue in practice. With some
careful naming guidelines we are able to resolve naming conflicts, and
I'm sure we could extend the guidelines to multiple versions of language
stacks or whatever. We'd probably have slightly longer names or packages,
but that's not the end of the world.

>  So I don't really get even after almost five years where
> modularity is
>  going or what it wants to achieve. I don't understand its use-case for
>  any of Fedora, RHEL, and CentOS because disabling
>  parallel-installability to allow parallel availability is imho not
>  really an option. But yeah...maybe I am missing some angle. In that
>  case, please, explain it to me because I would really like to
>  understand... 
Zbyszek

[Petr Pisar]

On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  I think there's some fear that "name mangling" is not a
> general
>  solution, and we'd have cases where names conflict. I think the
>  concern is realistic, but not a big issue in practice. With some
>  careful naming guidelines we are able to resolve naming conflicts, and
>  I'm sure we could extend the guidelines to multiple versions of language
>  stacks or whatever. We'd probably have slightly longer names or packages,
>  but that's not the end of the world.
>  
Namespacing RPM package names is only a tip of the iceberg. You need to
namespace RPM requires, provides, shared library sonames, shared library
symbols, header files, header file symbols, pkg-config files, manual pages,
cross references in the manual pages, executable names etc. You need to
namespace everything if you aim for a parallel installability. That's the
reason why modularity does not tackle it and instead conflicts the concurent
versions by filtering the packages from the concurent streams.

-- Petr

[Zbigniew Jędrzejewski-Szmek]

On Wed, Jun 24, 2020 at 09:22:39AM +0200, Petr Pisar wrote:
>  On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > I think there's some fear that "name mangling" is not a general
>  > solution, and we'd have cases where names conflict. I think the
>  > concern is realistic, but not a big issue in practice. With some
>  > careful naming guidelines we are able to resolve naming conflicts, and
>  > I'm sure we could extend the guidelines to multiple versions of language
>  > stacks or whatever. We'd probably have slightly longer names or packages,
>  > but that's not the end of the world.
>  >
>  Namespacing RPM package names is only a tip of the iceberg. You need to
>  namespace RPM requires, provides, shared library sonames, shared library
>  symbols, header files, header file symbols, pkg-config files, manual pages,
>  cross references in the manual pages, executable names etc. You need to
>  namespace everything if you aim for a parallel installability. That's the
>  reason why modularity does not tackle it and instead conflicts the concurent
>  versions by filtering the packages from the concurent streams. 
I am very much aware that making things parallel-installable is an
order-of-magnitude harder than just parallalel-available. P-I is certainly
not going to come for free and would require some careful packaging. But
determined packagers can (and do) make it work. Also, some cases could be
easier, for example different minor versions of python (e.g. 3.x and 3.y).
Another example is P-I of different so-versions of libraries: Debian
has been doing this at scale since forever, so it's clearly not that hard.

The point I was trying to make is that we need a solution that works for
the easy case (P-A), but can allow handling the hard case (P-I) too with
additional preparation.

Zbyszek

[Petr Pisar]

On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  Yes. Putting the "stream identification" into the package
> name is the
>  most natural solution, and has been floated various times. 
This already happens. But not in Fedora. In RHEL, modular packages have
Modularitylabel RPM tag that carries the module name and stream.

-- Petr

[clime]

On Wed, 24 Jun 2020 at 09:40, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > Yes. Putting the "stream identification" into the package name is the
>  > most natural solution, and has been floated various times.
>
>  This already happens. But not in Fedora. In RHEL, modular packages have
>  Modularitylabel RPM tag that carries the module name and stream. 
Does "ModularityLabel" actually propagates to rpm package name or is
it just a "hidden" rpm attribute?

>
> -- Petr
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Petr Pisar]

On Wed, Jun 24, 2020 at 10:22:38AM +0200, clime wrote:
>  On Wed, 24 Jun 2020 at 09:40, Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > Yes. Putting the "stream identification" into the package name is
> the
>  > > most natural solution, and has been floated various times.
>  >
>  > This already happens. But not in Fedora. In RHEL, modular packages have
>  > Modularitylabel RPM tag that carries the module name and stream.
>  
>  Does "ModularityLabel" actually propagates to rpm package name or is
>  it just a "hidden" rpm attribute?
>  
It's a RPM tag as well as a package name or a package version are the RPM
tags. I don't understand your question.

-- Petr

[clime]

On Wed, 24 Jun 2020 at 10:35, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  On Wed, Jun 24, 2020 at 10:22:38AM +0200, clime wrote:
>  > On Wed, 24 Jun 2020 at 09:40, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > >
>  > > On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > > Yes. Putting the "stream identification" into the package name
> is the
>  > > > most natural solution, and has been floated various times.
>  > >
>  > > This already happens. But not in Fedora. In RHEL, modular packages have
>  > > Modularitylabel RPM tag that carries the module name and stream.
>  >
>  > Does "ModularityLabel" actually propagates to rpm package name or is
>  > it just a "hidden" rpm attribute?
>  >
>  It's a RPM tag as well as a package name or a package version are the RPM
>  tags. I don't understand your question. 
Well, the original sentence was: "Putting the "stream identification"
into the package *name* is the most natural solution".

And the answer was: "This already happens. But not in Fedora. In RHEL, ..."

But my question was answered, thank you.

>
> -- Petr
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Petr Pisar]

On Wed, Jun 24, 2020 at 11:01:55AM +0200, clime wrote:
>  On Wed, 24 Jun 2020 at 10:35, Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > On Wed, Jun 24, 2020 at 10:22:38AM +0200, clime wrote:
>  > > On Wed, 24 Jun 2020 at 09:40, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  > > >
>  > > > On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > > > > Yes. Putting the "stream identification" into the package
> name is the
>  > > > > most natural solution, and has been floated various times.
>  > > >
>  > > > This already happens. But not in Fedora. In RHEL, modular packages have
>  > > > Modularitylabel RPM tag that carries the module name and stream.
>  > >
>  > > Does "ModularityLabel" actually propagates to rpm package name or is
>  > > it just a "hidden" rpm attribute?
>  > >
>  > It's a RPM tag as well as a package name or a package version are the RPM
>  > tags. I don't understand your question.
>  
>  Well, the original sentence was: "Putting the "stream identification"
>  into the package *name* is the most natural solution".
>  
>  And the answer was: "This already happens. But not in Fedora. In RHEL, ..."
>  
>  But my question was answered, thank you.
>  
I see. I focused on having the stream information on RPM level. Then the
answer is no, the package name does not contain the information.

My idea was that DNF could discriminate the same-name package using the
ModularityLabel tag instead of relying on modulemd documents delivered in the
repository metadata.

-- Petr

[Nicolas Mailhot]

Le mercredi 24 juin 2020 à 11:56 +0200, Petr Pisar a écrit :
>  I see. I focused on having the stream information on RPM level. Then
>  the
>  answer is no, the package name does not contain the information.
>  
>  My idea was that DNF could discriminate the same-name package using
>  the
>  ModularityLabel tag instead of relying on modulemd documents
>  delivered in the
>  repository metadata. 
One problem of having it a tag (which we do not even have in Fedora) is
that it requires rewriting dependency resolution logic at dnf level,
and a Tag does not come with all the dependency manipulation verbs we
have evolved over the years for Provides and Requires.

-- 
Nicolas Mailhot

[Nicolas Mailhot]

Le mercredi 24 juin 2020 à 12:03 +0200, Nicolas Mailhot a écrit :
>  Le mercredi 24 juin 2020 à 11:56 +0200, Petr Pisar a écrit :
>  > I see. I focused on having the stream information on RPM level.
>  > Then
>  > the
>  > answer is no, the package name does not contain the information.
>  > 
>  > My idea was that DNF could discriminate the same-name package using
>  > the
>  > ModularityLabel tag instead of relying on modulemd documents
>  > delivered in the
>  > repository metadata.
>  
>  One problem of having it a tag (which we do not even have in Fedora)
>  is
>  that it requires rewriting dependency resolution logic at dnf level,
>  and a Tag does not come with all the dependency manipulation verbs we
>  have evolved over the years for Provides and Requires. 
That is what killed the group tag and comps groups as generic ways to
declare package grouping BTW.

RPM maintainers were long opposed to metapackages, but in the end
metapackages offer more packager flexibility, and appear as normal
objects in the dependency graph, meaning you can do things with them
you could never achieve with an out-of-graph Comps/Tag group.

-- 
Nicolas Mailhot

[Petr Pisar]

On Wed, Jun 24, 2020 at 12:03:05PM +0200, Nicolas Mailhot via devel wrote:
>  Le mercredi 24 juin 2020 à 11:56 +0200, Petr Pisar a écrit :
>  > I see. I focused on having the stream information on RPM level. Then the
>  > answer is no, the package name does not contain the information.
>  > 
>  > My idea was that DNF could discriminate the same-name package using the
>  > ModularityLabel tag instead of relying on modulemd documents delivered in
>  > the repository metadata.
>  
>  One problem of having it a tag (which we do not even have in Fedora) 
I believe having it in Fedora is only a matter of changing MBS configuration.

>  is that it requires rewriting dependency resolution logic at dnf
> level, 
DNF has a steadfast idea that an upgrade path is only based on a package name.
Without changes in DNF, DNF will either switch a stream just because a package
from a concurrent stream has a higher version, or will complain that it cannot
upgrade to the lastest version. Neither of the options is a desired behavior.
Thus I believe that changing DNF is inevitable.

>  and a Tag does not come with all the dependency manipulation verbs
> we
>  have evolved over the years for Provides and Requires.
>   
ModularityLabel designates an affilation to a stream. That can be reduced to
"Requires: module(name:stream)" or "Requires: module(name) = stream".
I'm not
agaist abusing Requires for that purpose. But it alone won't fix the issue with
enforcing a presence of a stream I drafted above.

-- Petr

[Daniel Mach]

Dne 24. 06. 20 v 11:56 Petr Pisar napsal(a):
>  On Wed, Jun 24, 2020 at 11:01:55AM +0200, clime wrote:
> > On Wed, 24 Jun 2020 at 10:35, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
> >>
> >> On Wed, Jun 24, 2020 at 10:22:38AM +0200, clime wrote:
> >>> On Wed, 24 Jun 2020 at 09:40, Petr Pisar <ppisar(a)redhat.com&gt; wrote:
> >>>>
> >>>> On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
> >>>>> Yes. Putting the "stream identification" into the package
> name is the
> >>>>> most natural solution, and has been floated various times.
> >>>>
> >>>> This already happens. But not in Fedora. In RHEL, modular packages have
> >>>> Modularitylabel RPM tag that carries the module name and stream.
> >>>
> >>> Does "ModularityLabel" actually propagates to rpm package name or
> is
> >>> it just a "hidden" rpm attribute?
> >>>
> >> It's a RPM tag as well as a package name or a package version are the RPM
> >> tags. I don't understand your question.
> >
> > Well, the original sentence was: "Putting the "stream identification"
> > into the package *name* is the most natural solution".
> >
> > And the answer was: "This already happens. But not in Fedora. In RHEL,
> ..."
> >
> > But my question was answered, thank you.
> >
>  I see. I focused on having the stream information on RPM level. Then the
>  answer is no, the package name does not contain the information.
>  
>  My idea was that DNF could discriminate the same-name package using the
>  ModularityLabel tag instead of relying on modulemd documents delivered in the
>  repository metadata.
>   
The "modularitylabel" is not going to help.
It's designed as a boolean flag.
If it holds any value, it indicates that the RPM is part of a module.
If it's empty, then the RPM is non-modular.
If you're looking for any sense of the header, it's probably closest to 
a reference to the module build it comes from.

RPMs are supposed to be re-used among modules/contexts and for that 
reason we cannot hardcode this relation directly to them.

[Petr Pisar]

On Wed, Jun 24, 2020 at 12:48:07PM +0200, Daniel Mach wrote:
>  > My idea was that DNF could discriminate the same-name package
> using the
>  > ModularityLabel tag instead of relying on modulemd documents delivered in the
>  > repository metadata.
>  > 
>  The "modularitylabel" is not going to help.
>  It's designed as a boolean flag.
>  If it holds any value, it indicates that the RPM is part of a module.
>  If it's empty, then the RPM is non-modular.
>  If you're looking for any sense of the header, it's probably closest to a
>  reference to the module build it comes from.
>  
>  RPMs are supposed to be re-used among modules/contexts and for that reason
>  we cannot hardcode this relation directly to them. 
I know all this things, but:

ModularityLabel tag can be interpreted not only as a boolean. It was added to
recognize a modular package for a case when a repository with modulemd
metadata disappears. This is where the interpretatinon as a boolean type comes
from. But here we discuss a new modularity that does not have to align to the
current implementation of modularity.

The ModularityLabel tag carry name:stream:version:context of a module build
where the RPM package was built. I know that a package can be reused in a new
module version and then the ModularityLabel won't match exactly. But that's
not a problem, because nobody cares about a version of the module. E.g.
current DNF implementation puts all modular packages from a single
name:stream:context onto one heap. The version is ignored. The same would
apply to the ModularityLabel tag.

Referring to modulemd data that are not available on RPM level when this
mental excercise is about blending modularity into RPM level defies the
purpose.

-- Petr

[James Cassell]

On Wed, Jun 24, 2020, at 3:37 AM, Petr Pisar wrote:
>  On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek
> wrote:
>  > Yes. Putting the "stream identification" into the package name is the
>  > most natural solution, and has been floated various times.
>  
>  This already happens. But not in Fedora. In RHEL, modular packages have
>  Modularitylabel RPM tag that carries the module name and stream.
>   
It's too bad this isn't exposed in `rpm -qi` output. Is there a way to query the
Modularitylabel RPM tag on an installed system?

V/r,
James Cassell

[Petr Pisar]

On Wed, Jun 24, 2020 at 04:36:06AM -0400, James Cassell wrote:
>  
>  On Wed, Jun 24, 2020, at 3:37 AM, Petr Pisar wrote:
>  > On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > Yes. Putting the "stream identification" into the package name is
> the
>  > > most natural solution, and has been floated various times.
>  > 
>  > This already happens. But not in Fedora. In RHEL, modular packages have
>  > Modularitylabel RPM tag that carries the module name and stream.
>  > 
>  
>  It's too bad this isn't exposed in `rpm -qi` output. Is there a way to query
>  the Modularitylabel RPM tag on an installed system?
>   
rpm -q --qf '%{Modularitylabel}\n' PACKAGE

-- Petr

[Stephen Gallagher]

On Wed, Jun 24, 2020 at 3:38 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > Yes. Putting the "stream identification" into the package name is the
>  > most natural solution, and has been floated various times.
>
>  This already happens. But not in Fedora. In RHEL, modular packages have
>  Modularitylabel RPM tag that carries the module name and stream.
>  
The ModularityLabel RPM tag is also present in Fedora. However, it's
documented as *not* reliable for reverse lookups. It only lists the
NSVCA of the build it first occurred in, which may not be what you're
looking for if the package was unchanged and has been reused for
subsequent builds. (Though I *think* you can rely on the name and
stream being the same right now, since we don't yet have cross-module
component reuse in MBS, but that's coming.)

[Petr Pisar]

On Wed, Jun 24, 2020 at 08:14:39AM -0400, Stephen Gallagher wrote:
>  On Wed, Jun 24, 2020 at 3:38 AM Petr Pisar <ppisar(a)redhat.com&gt;
> wrote:
>  >
>  > On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > Yes. Putting the "stream identification" into the package name is
> the
>  > > most natural solution, and has been floated various times.
>  >
>  > This already happens. But not in Fedora. In RHEL, modular packages have
>  > Modularitylabel RPM tag that carries the module name and stream.
>  >
>  
>  The ModularityLabel RPM tag is also present in Fedora. 
Interesting. I checked the first modular package for the first module "dnf
module list" gives in Fedora 33
(ant-0:1.10.5-3.module_f28+4207+d722d224.noarch), and it does not contain the
tag. I did not checked other ones. Thus I wrote that it was not in Fedora.

Now I see that the package was built on Fedora 28 platform. When I check the
latest build in Koji,  ant-1.10.7-2.module_f31+7074+f8e1675d.noarch, the tag
is there.

My apology. Probably the new tag was enabled sometime in between and nobody
cared to rebuild the module after that (although relengs mass-rebuild modules
on branching) and nobody cared to submit to latest build to stable.

>  However, it's documented as *not* reliable for reverse lookups.
> It only
>  lists the NSVCA of the build it first occurred in, which may not be what
>  you're looking for if the package was unchanged and has been reused for
>  subsequent builds. (Though I *think* you can rely on the name and stream
>  being the same right now, since we don't yet have cross-module component
>  reuse in MBS, but that's coming.) 
Exactly. We do not reuse components across modules. I cannot see how reliable
it could be considering every module can have a different build root resulting
into incompatible builds. So the fact that it "is coming" is news for me.

-- Petr

[Przemek Klosowski]

On 6/24/20 8:56 AM, Petr Pisar wrote:
>  On Wed, Jun 24, 2020 at 08:14:39AM -0400, Stephen Gallagher wrote:
> > On Wed, Jun 24, 2020 at 3:38 AM Petr Pisar<ppisar(a)redhat.com>  wrote:
> >> On Wed, Jun 24, 2020 at 06:51:36AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
> >>> Yes. Putting the "stream identification" into the package name is
> the
> >>> most natural solution, and has been floated various times.
> >> This already happens. But not in Fedora. In RHEL, modular packages have
> >> Modularitylabel RPM tag that carries the module name and stream.
> >>
> > The ModularityLabel RPM tag is also present in Fedora.
>  Interesting. I checked the first modular package for the first module "dnf
>  module list" gives in Fedora 33
>  (ant-0:1.10.5-3.module_f28+4207+d722d224.noarch), and it does not contain the
>  tag. I did not checked other ones. Thus I wrote that it was not in Fedora.
>
>  Now I see that the package was built on Fedora 28 platform. When I check the
>  latest build in Koji,  ant-1.10.7-2.module_f31+7074+f8e1675d.noarch, the tag
>  is there.
>
>  My apology. Probably the new tag was enabled sometime in between and nobody
>  cared to rebuild the module after that (although relengs mass-rebuild modules
>  on branching) and nobody cared to submit to latest build to stable.
>  
FWIW,  other F31 packages with it are:

gimp gimp:2.10:3120191106095052:f636be4b
gimp-libs gimp:2.10:3120191106095052:f636be4b
ninja-build ninja:latest:3120190304180949:f636be4b

[Nicolas Mailhot]

Le mardi 23 juin 2020 à 20:31 +0200, clime a écrit :
>  Or we can bring the notion of
>  the namespaces into rpm itself (that's where my suggestion of
>  "Stream"
>  rpm attribute comes from but it could also be called just
>  "Namespace"). But then there is the argument: "Why not just put the
>  namespace into rpm name itself?" I mean...I wouldn't mind having it
>  as
>  a separate attribute but the usefulness of it would need to be
>  discussed. 
The namespacing information will eventually need to be pushed to the
package level because  it’s not possible to do cool stuff with things
that do not exist at the atomic component level.

However rpm (the tool that creates the deployment format) sucks loads
for managing variables/attributes at scale. RPM tags are completely
inadequate for the level of variabilisation the distribution needs
today (and getting more inadequate BTW — there are new tag constrains
in rpm 4.15+ that did not exist before).

That’s why we have to shove level after level of unrelated complexity
in a single Release tag, that’s the whole changelog/autobumping
discussion (I have other examples but they would not speak to the
general audience on this list).

You can bypass a lot of rpm format limitations by giving up on Tag uses
altogether, using rpm macros/variables instead of tags in the spec
file, and exposing the new metadata in foo() provides, but you will
still hit the brick wall of handling all those new variables at the
spec level.

IMHO Modularity functional objectives were good, implementation was
bad, first by confusing a build framework with a deployment stream
format, and second, by not investing in the actual tool that creates
our deployment format, to make it scale to the levels of complexity
modularity requires at the component creation stage.

Regards,

-- 
Nicolas Mailhot

[Neal Gompa]

On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  Hello Fedora Community!
>
>  I am a long-time Fedora Community member, and may be familiar to many
>  through previous FESCo or devel list discussions and passionate
>  debates.  However I write to you today with a different community hat
>  on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  organization has been following the modularity discussions within
>  Fedora, particularly around ELN, and often the question of what plans
>  we have for modularity in RHEL 9 has come up.  Our Fedora Project Lead
>  and a number of FESCo members have reached out and asked if we can
>  provide some perspective here, and I am both happy and excited to have
>  that opportunity.
>  
Thank you for taking the opportunity to talk to us from the Red Hat
Enterprise Linux perspective. I greatly appreciate that and I hope
others respond kindly to this outreach with constructive feedback.

>  As the Fedora Council has pointed out [1], we certainly acknowledge
>  there are improvements to be made and have a team already working on
>  them.  They recently outlined their plans in conjunction with our
>  Product Management team in a Fedora Council call as well [2].  We’re
>  continuing to invest time and effort in this packaging solution and
>  are confident that the team can deliver against their plan.  It is
>  somewhat of a new experience for all of us when Red Hat is direct with
>  our product intentions, but we discussed the larger gaps we see with
>  usage in RHEL and are putting our efforts towards solving those gaps
>  with this plan.
>
>  Modularity is important to RHEL and those efforts are already
>  underway.  We will be leveraging modularity in RHEL 9 where it most
>  makes sense.  This is primarily centered around our Application
>  Streams concept, which has been well received by our customer base.
>  Providing a consistent but improved experience is the base
>  requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  and lowers the hurdle for our customers when upgrading from one major
>  version to another.
>  
Personally, as a user of the Application Streams stuff in my
custom-built EL containers, it's very nice, and it works well for
providing the flexibility I've needed while having a fully supportable
stack of software. It's a bit less fun on regular servers and VM
environments, but I think this can improve.

>  It is always good to push the boundaries and search for better ideas
>  and improvements, and that is part of what makes Fedora great.  We are
>  doing this in the context of the RHEL 9 release as well, so our near
>  term timeline and requirements mean we are working on evolving
>  modularity, not a revolution or a replacement.  We are excited by ELN,
>  as it presents a possible space to allow those that want to continue
>  to iterate on modules a place to do so without necessarily impacting
>  the broader Fedora distribution in its entirety.  It is my personal
>  hope that we can use that opportunity to improve modules and
>  modularity in the open source, Fedora-first way we’d prefer.  Our near
>  term effort to improve the existing modularity implementation ahead of
>  RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
>  than in closed product development.  Longer term, we are open to
>  contributing to a better replacement that meets many of the same
>  goals.  This is what makes our distribution ecosystem work well, and
>  not having that upstream lessens the value we all get from such
>  experimentation in the open.
>  
Something that has been bothering me a bit is that there's a lot of
mixed messaging around ELN, even from Red Hatters.

Don't get me wrong: I *absolutely* want ELN to exist, and I like that
we're doing it. I *want* RHEL development happening in Fedora.

But I'm confused about the purpose of ELN. Is it intended to be the
development playground for Red Hatters? Or is it a community
initiative to support Fedora and Red Hat to come together on
developing RHEL? Or is it just a fake-RHEL built on Fedora to minimize
the burden of forking Fedora for making RHEL later? I had personally
hoped that ELN would be an opportunity to allow the Fedora community
and Red Hat to work together on building the future RHEL more
directly, but I am unsure of what it does or what it is for now.

As for iterating on modularity in Fedora through ELN, I think this is
a good idea. I want to see the implementation of modularity fleshed
out and the packager experience improved, and I think the only way to
do that is to actually use it and inflict all the pain on the people
who need it by not permitting weird hacky workarounds to make module
builds work. If something is broken, the standard code path has to be
fixed, and that's how I expect this will work. I'm already aware that
CentOS rebuilds of RHEL are not necessarily straightforward because
many hacky shortcuts are taken to build RHEL content and CentOS does
not have those in place.

However, I am concerned that as ELN develops further, we are likely to
be even more starved for build resources than we have been previously.
Modules are huge build chains that essentially fill up the builders.
Outside of the improved AArch64 hardware, I'm personally unaware of
any improvements in our build capacity to help support the higher
demands for the build system. To note, we'd have this problem without
modules if we had Koschei configured to auto-rebuild and submit
rebuilds on dependency drift so that packagers didn't have to do that
grunt work manually, so it's a matter of we literally do not have
enough resources to support more automation. I've mentioned this
before in other threads, but to reiterate: it is my belief that Fedora
does not have enough build capacity to support building a modularized
distribution. Even when we were doing modularized builds in the Fedora
Rust SIG, it was common for module build jobs to stall out waiting for
resources, and thus get stuck midway through. This also starved
regular builds of resources to get things done.

>  Hopefully that provides some context and helps FESCo and the wider
>  community understand where Red Hat is headed with modularity on the
>  Enterprise side.
>  
It absolutely does, and I hope you continue to engage with us on this!
Let's make everything better together!



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen John Smoogen]

On Thu, 18 Jun 2020 at 09:27, Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > 
>  However, I am concerned that as ELN develops further, we are likely
> to
>  be even more starved for build resources than we have been previously.
>  Modules are huge build chains that essentially fill up the builders.
>  Outside of the improved AArch64 hardware, I'm personally unaware of
>  any improvements in our build capacity to help support the higher
>  demands for the build system. To note, we'd have this problem without
>  modules if we had Koschei configured to auto-rebuild and submit
>  rebuilds on dependency drift so that packagers didn't have to do that
>  grunt work manually, so it's a matter of we literally do not have
>  enough resources to support more automation. I've mentioned this
>  before in other threads, but to reiterate: it is my belief that Fedora
>  does not have enough build capacity to support building a modularized
>  distribution. Even when we were doing modularized builds in the Fedora
>  Rust SIG, it was common for module build jobs to stall out waiting for
>  resources, and thus get stuck midway through. This also starved
>  regular builds of resources to get things done.
>  
One of the problems with having more build capacity is that the build
system will need checks to find where we have hidden O(n) or greater
code.  Adding more builders may not make various things faster, and I
am expecting we are going to see some slow downs from the current
'fast koji' people are seeing as we stand up more resources in the new
datacenter in the next couple of months. There is a lot of software
written for specific cases being used in general cases, and there is a
lot of software where it could only be written and tested for 1-4
boxes now being used for scale. There are also a bunch of software
tools which were organically added under time crunches with no review
to fix which work 'good enough' but aren't going to work 'better' if
we added more builders.



-- 
Stephen J Smoogen.

[Josh Boyer]

On Thu, Jun 18, 2020 at 9:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  >
>  > Hello Fedora Community!
>  >
>  > I am a long-time Fedora Community member, and may be familiar to many
>  > through previous FESCo or devel list discussions and passionate
>  > debates.  However I write to you today with a different community hat
>  > on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
>  > organization has been following the modularity discussions within
>  > Fedora, particularly around ELN, and often the question of what plans
>  > we have for modularity in RHEL 9 has come up.  Our Fedora Project Lead
>  > and a number of FESCo members have reached out and asked if we can
>  > provide some perspective here, and I am both happy and excited to have
>  > that opportunity.
>  >
>
>  Thank you for taking the opportunity to talk to us from the Red Hat
>  Enterprise Linux perspective. I greatly appreciate that and I hope
>  others respond kindly to this outreach with constructive feedback.
>
>  > As the Fedora Council has pointed out [1], we certainly acknowledge
>  > there are improvements to be made and have a team already working on
>  > them.  They recently outlined their plans in conjunction with our
>  > Product Management team in a Fedora Council call as well [2].  We’re
>  > continuing to invest time and effort in this packaging solution and
>  > are confident that the team can deliver against their plan.  It is
>  > somewhat of a new experience for all of us when Red Hat is direct with
>  > our product intentions, but we discussed the larger gaps we see with
>  > usage in RHEL and are putting our efforts towards solving those gaps
>  > with this plan.
>  >
>  > Modularity is important to RHEL and those efforts are already
>  > underway.  We will be leveraging modularity in RHEL 9 where it most
>  > makes sense.  This is primarily centered around our Application
>  > Streams concept, which has been well received by our customer base.
>  > Providing a consistent but improved experience is the base
>  > requirement, which allows us to have continuity from RHEL 8 to RHEL 9
>  > and lowers the hurdle for our customers when upgrading from one major
>  > version to another.
>  >
>
>  Personally, as a user of the Application Streams stuff in my
>  custom-built EL containers, it's very nice, and it works well for
>  providing the flexibility I've needed while having a fully supportable
>  stack of software. It's a bit less fun on regular servers and VM
>  environments, but I think this can improve.
>
>  > It is always good to push the boundaries and search for better ideas
>  > and improvements, and that is part of what makes Fedora great.  We are
>  > doing this in the context of the RHEL 9 release as well, so our near
>  > term timeline and requirements mean we are working on evolving
>  > modularity, not a revolution or a replacement.  We are excited by ELN,
>  > as it presents a possible space to allow those that want to continue
>  > to iterate on modules a place to do so without necessarily impacting
>  > the broader Fedora distribution in its entirety.  It is my personal
>  > hope that we can use that opportunity to improve modules and
>  > modularity in the open source, Fedora-first way we’d prefer.  Our near
>  > term effort to improve the existing modularity implementation ahead of
>  > RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
>  > than in closed product development.  Longer term, we are open to
>  > contributing to a better replacement that meets many of the same
>  > goals.  This is what makes our distribution ecosystem work well, and
>  > not having that upstream lessens the value we all get from such
>  > experimentation in the open.
>  >
>
>  Something that has been bothering me a bit is that there's a lot of
>  mixed messaging around ELN, even from Red Hatters. 
Indeed.  It's new :)

>  Don't get me wrong: I *absolutely* want ELN to exist, and I like
> that
>  we're doing it. I *want* RHEL development happening in Fedora.
>
>  But I'm confused about the purpose of ELN. Is it intended to be the
>  development playground for Red Hatters? Or is it a community
>  initiative to support Fedora and Red Hat to come together on
>  developing RHEL? Or is it just a fake-RHEL built on Fedora to minimize
>  the burden of forking Fedora for making RHEL later? I had personally
>  hoped that ELN would be an opportunity to allow the Fedora community
>  and Red Hat to work together on building the future RHEL more
>  directly, but I am unsure of what it does or what it is for now. 
I can tell you my opinion, but because ELN is a new concept/idea it is
only my interpretation.

I see ELN as an opportunity born from the Fedora community to
experiment with the distribution aimed at a different userbase than
what we'd traditionally see with typical Fedora usage.  It is not that
different from Editions in many ways, except Editions were crafted as
a focusing mechanism leveraging the same Fedora packages and defaults.
ELN allows experimentation with different defaults or baselines.
Matthew has talked about a Playground concept before, and ELN
incorporates some of those ideas as well.

Clear as mud, yes?  New ideas often make things murky, particularly as
they are being worked through.  I see this as a natural occurrence in
open source.

With my official Red Hat hat on, I do *not* see ELN as a fake-RHEL to
minimize forking Fedora.  If we wanted to do that, we could.  ELN has
the potential to become a development space for major RHEL releases,
but that potential only exists as much as we let it.  There are many
times where Enterprise and non-Enterprise requirements are in direct
conflict, and ELN could be the place to iterate on them.  If people
aren't comfortable developing and experimenting with that deviation in
ELN, then it makes developing RHEL in Fedora harder but it doesn't
make ELN useless or remove options for open RHEL development entirely.

It's an opportunity.  It's up to both sides to use it.

>  As for iterating on modularity in Fedora through ELN, I think this
> is
>  a good idea. I want to see the implementation of modularity fleshed
>  out and the packager experience improved, and I think the only way to
>  do that is to actually use it and inflict all the pain on the people
>  who need it by not permitting weird hacky workarounds to make module
>  builds work. If something is broken, the standard code path has to be
>  fixed, and that's how I expect this will work. I'm already aware that
>  CentOS rebuilds of RHEL are not necessarily straightforward because
>  many hacky shortcuts are taken to build RHEL content and CentOS does
>  not have those in place. 
I would say that we shouldn't conflate building a distribution with
modules included and *rebuilding* a distribution with modules
included.  CentOS rebuilds are probably not something I would consider
in the context of Fedora, ELN, and modules.  Probably best to discuss
that on the CentOS lists.

Put another way, what you phrase as hacky shortcuts and struggles are
not the same issues we'd necessarily see when trying to build a
distribution that way for the first time.  There is certainly an
overlap of issues, but we should constrain ourselves a bit.

>  However, I am concerned that as ELN develops further, we are likely
> to
>  be even more starved for build resources than we have been previously.
>  Modules are huge build chains that essentially fill up the builders.
>  Outside of the improved AArch64 hardware, I'm personally unaware of
>  any improvements in our build capacity to help support the higher
>  demands for the build system. To note, we'd have this problem without
>  modules if we had Koschei configured to auto-rebuild and submit
>  rebuilds on dependency drift so that packagers didn't have to do that
>  grunt work manually, so it's a matter of we literally do not have
>  enough resources to support more automation. I've mentioned this
>  before in other threads, but to reiterate: it is my belief that Fedora
>  does not have enough build capacity to support building a modularized
>  distribution. Even when we were doing modularized builds in the Fedora
>  Rust SIG, it was common for module build jobs to stall out waiting for
>  resources, and thus get stuck midway through. This also starved
>  regular builds of resources to get things done. 
My understanding is that this was discussed with Fedora
infrastructure.  I have no insight into that further.  We probably
aren't going to get additional builders for ELN or Fedora in general
anytime soon, so allocation of existing resources is probably
something to consider.  I'll offer that there are actually more Fedora
build hosts than there are RHEL 8 build hosts, by comparison.

Personally, I have long wanted burst-to-cloud or the ability for
others to donate hosts to the Fedora build system without having to
physically ship hardware.  Koji is somewhat limited in that regard.
Maybe developing a shim layer and some security best practices to
allow that would help.

>  > Hopefully that provides some context and helps FESCo and the
> wider
>  > community understand where Red Hat is headed with modularity on the
>  > Enterprise side.
>  >
>
>  It absolutely does, and I hope you continue to engage with us on this!
>  Let's make everything better together! 
That's the idea :)

josh

[Kevin Fenzi]

On Thu, Jun 18, 2020 at 12:30:58PM -0400, Josh Boyer wrote:
>  On Thu, Jun 18, 2020 at 9:26 AM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote: 
...snip...
>  > However, I am concerned that as ELN develops further, we are
> likely to
>  > be even more starved for build resources than we have been previously.
>  > Modules are huge build chains that essentially fill up the builders. 
It's a concern, but IMHO not a major one. 

ELN builds run at a higher priority than normal builds. 

Module are currently the same prirority as normal builds, but we could
adjust them. Or adjust just the ELN ones. 

There's a number of things we could do if it becomes a problem. 

>  > Outside of the improved AArch64 hardware, I'm personally
> unaware of
>  > any improvements in our build capacity to help support the higher
>  > demands for the build system. To note, we'd have this problem without 
There are, and thanks for a place to note them: 

The x86_64 buildvm's now in the new datacenter are on newer hardware
thats likely much faster than the old ones. 

The new datacenter has a new netapp cluster (which we aren't sharing
with other workloads). 

Mainframe admins adjusted things on the s390x z/vm builders to be faster
and indeed they seem much improved.

The new datacenter we have postgresql 12.2 for the koji database. 

Our last mass rebuild finished in a few days. How much faster do we need
to be? ;) 

>  > modules if we had Koschei configured to auto-rebuild and submit
>  > rebuilds on dependency drift so that packagers didn't have to do that
>  > grunt work manually, so it's a matter of we literally do not have
>  > enough resources to support more automation. I've mentioned this
>  > before in other threads, but to reiterate: it is my belief that Fedora
>  > does not have enough build capacity to support building a modularized
>  > distribution. Even when we were doing modularized builds in the Fedora
>  > Rust SIG, it was common for module build jobs to stall out waiting for
>  > resources, and thus get stuck midway through. This also starved
>  > regular builds of resources to get things done.
>  
>  My understanding is that this was discussed with Fedora
>  infrastructure.  I have no insight into that further.  We probably 
It was, and I don't think it's going to be that much of a problem.
I could of course be wrong (I have been before and will be again!)

>  aren't going to get additional builders for ELN or Fedora in
> general
>  anytime soon, so allocation of existing resources is probably
>  something to consider.  I'll offer that there are actually more Fedora
>  build hosts than there are RHEL 8 build hosts, by comparison. 
Of course Fedora builds a ton more things than RHEL... but yeah. 

>  Personally, I have long wanted burst-to-cloud or the ability for
>  others to donate hosts to the Fedora build system without having to
>  physically ship hardware.  Koji is somewhat limited in that regard.
>  Maybe developing a shim layer and some security best practices to
>  allow that would help. 
Yeah, there have been several projects to allow this to happen, and if
pressed for resources we could look into them again. 

>  
>  > > Hopefully that provides some context and helps FESCo and the wider
>  > > community understand where Red Hat is headed with modularity on the
>  > > Enterprise side.
>  > >
>  >
>  > It absolutely does, and I hope you continue to engage with us on this!
>  > Let's make everything better together!
>  
>  That's the idea :) 
100% agreed. :) 

kevin

[Ken Dreyer]

On Thu, Jun 18, 2020 at 10:31 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  Personally, I have long wanted burst-to-cloud or the ability for
>  others to donate hosts to the Fedora build system without having to
>  physically ship hardware.  Koji is somewhat limited in that regard.
>  Maybe developing a shim layer and some security best practices to
>  allow that would help. 
I'm interested in this because I think it would make Koji more
flexible, and there are some challenges. I think we would need a
separate Koji daemon to watch the task queues on the hub and bring
additional builders up or down as needed. Maybe an OpenShift operator
could do this. Non-x86_64 arches are complicated as well, because not
all cloud providers have s390x (for example). A service needs to
inspect the Koji buildArch task parameters to determine what arches to
bring up, and that's just for RPMs - we'd need code to do it for the
VM image tasks, containers, etc.

Do you have specific vendors lined up who would donate build hosts? In
the Ceph project, we have something like what you're describing with
libcloud and Jenkins. Our CI build hosts' costs were wildly expensive
compared to our bare metal hosts, and the performance can be
variable/worse. At a certain point, there is a constant baseline load
in the buildsystem, and it makes sense to run as much of that on our
own hosts as we can.

- Ken

[Josh Boyer]

On Thu, Jun 18, 2020 at 2:39 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 10:31 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > Personally, I have long wanted burst-to-cloud or the ability for
>  > others to donate hosts to the Fedora build system without having to
>  > physically ship hardware.  Koji is somewhat limited in that regard.
>  > Maybe developing a shim layer and some security best practices to
>  > allow that would help.
>
>  I'm interested in this because I think it would make Koji more
>  flexible, and there are some challenges. I think we would need a
>  separate Koji daemon to watch the task queues on the hub and bring
>  additional builders up or down as needed. Maybe an OpenShift operator
>  could do this. Non-x86_64 arches are complicated as well, because not
>  all cloud providers have s390x (for example). A service needs to
>  inspect the Koji buildArch task parameters to determine what arches to
>  bring up, and that's just for RPMs - we'd need code to do it for the
>  VM image tasks, containers, etc. 
I don't think burst-to-cloud means we only burst to a single cloud.
That seems like a great way to just lock into that cloud with no
flexibility.  Rather, I would look at it as a hybrid cloud opportunity
and use AWS, or the IBM cloud that offers s390x instances, or Packet
for ARM, etc.

>  Do you have specific vendors lined up who would donate build hosts?
> In
>  the Ceph project, we have something like what you're describing with
>  libcloud and Jenkins. Our CI build hosts' costs were wildly expensive
>  compared to our bare metal hosts, and the performance can be
>  variable/worse. At a certain point, there is a constant baseline load
>  in the buildsystem, and it makes sense to run as much of that on our
>  own hosts as we can. 
I have no vendors lined up.  I would simply like it to be an option.
Consider it a wish list idea and nothing more.  I'm fully aware of how
far that gets me in an open source community, so I expect nothing
because I'm not even scratching my own itch :)

josh

[Ken Dreyer]

On Thu, Jun 18, 2020 at 1:27 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  I don't think burst-to-cloud means we only burst to a single
> cloud.
>  That seems like a great way to just lock into that cloud with no
>  flexibility.  Rather, I would look at it as a hybrid cloud opportunity
>  and use AWS, or the IBM cloud that offers s390x instances, or Packet
>  for ARM, etc. 
Right, I think the challenge here is that multi-cloud is a hard
requirement for a first implementation to be viable. (libcloud really
helped Ceph out here).

- Ken

[Neal Gompa]

On Thu, Jun 18, 2020 at 2:39 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 10:31 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > Personally, I have long wanted burst-to-cloud or the ability for
>  > others to donate hosts to the Fedora build system without having to
>  > physically ship hardware.  Koji is somewhat limited in that regard.
>  > Maybe developing a shim layer and some security best practices to
>  > allow that would help.
>
>  I'm interested in this because I think it would make Koji more
>  flexible, and there are some challenges. I think we would need a
>  separate Koji daemon to watch the task queues on the hub and bring
>  additional builders up or down as needed. Maybe an OpenShift operator
>  could do this. Non-x86_64 arches are complicated as well, because not
>  all cloud providers have s390x (for example). A service needs to
>  inspect the Koji buildArch task parameters to determine what arches to
>  bring up, and that's just for RPMs - we'd need code to do it for the
>  VM image tasks, containers, etc.
>  
I *really* don't want to see Koji tied to Kubernetes. That adds a
whole bigger burden to actually deploying the thing, which I don't
think most folks deploying Koji are equipped to deal with.

>  Do you have specific vendors lined up who would donate build hosts?
> In
>  the Ceph project, we have something like what you're describing with
>  libcloud and Jenkins. Our CI build hosts' costs were wildly expensive
>  compared to our bare metal hosts, and the performance can be
>  variable/worse. At a certain point, there is a constant baseline load
>  in the buildsystem, and it makes sense to run as much of that on our
>  own hosts as we can.
>  
Why does it have to be specific vendors? This is the biggest problem
we have with the mindset around our infrastructure. Why is it that
individuals can't contribute?

Aside from that, vendors, in general, will *not* want to give stuff to
Fedora because Fedora is legally not separate from Red Hat. That makes
things very complicated, especially when trying to extract promises of
equipment being used only for Fedora and not for Red Hat commercial
purposes. I know that if I was in charge of a donation, I wouldn't
donate equipment unless I was _dead certain_ it wouldn't get used for
anything except Fedora stuff.

And finally, cloud infrastructure costs for CI massively exceed the
costs of having fixed infrastructure and scaling that. Burstiness is
overrated, because eventually all traffic rises to fill with the newly
available capacity. Unless you have a *really* good way to control
costs (which wow, you'd be in a realm beyond everyone else), CI
infrastructure in the cloud at Fedora scale would cost a ton more than
just adding hypervisors to the datacenter.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[David Cantrell]

On Thu, Jun 18, 2020 at 08:44:39AM -0400, Josh Boyer wrote:
> Hello Fedora Community!
>
> I am a long-time Fedora Community member, and may be familiar to many
> through previous FESCo or devel list discussions and passionate
> debates.  However I write to you today with a different community hat
> on, as a lead Architect for Red Hat Enterprise Linux.  The RHEL
> organization has been following the modularity discussions within
> Fedora, particularly around ELN, and often the question of what plans
> we have for modularity in RHEL 9 has come up.  Our Fedora Project Lead
> and a number of FESCo members have reached out and asked if we can
> provide some perspective here, and I am both happy and excited to have
> that opportunity.
>
> As the Fedora Council has pointed out [1], we certainly acknowledge
> there are improvements to be made and have a team already working on
> them.  They recently outlined their plans in conjunction with our
> Product Management team in a Fedora Council call as well [2].  We’re
> continuing to invest time and effort in this packaging solution and
> are confident that the team can deliver against their plan.  It is
> somewhat of a new experience for all of us when Red Hat is direct with
> our product intentions, but we discussed the larger gaps we see with
> usage in RHEL and are putting our efforts towards solving those gaps
> with this plan.
>
> Modularity is important to RHEL and those efforts are already
> underway.  We will be leveraging modularity in RHEL 9 where it most
> makes sense.  This is primarily centered around our Application
> Streams concept, which has been well received by our customer base.
> Providing a consistent but improved experience is the base
> requirement, which allows us to have continuity from RHEL 8 to RHEL 9
> and lowers the hurdle for our customers when upgrading from one major
> version to another.
>
> It is always good to push the boundaries and search for better ideas
> and improvements, and that is part of what makes Fedora great.  We are
> doing this in the context of the RHEL 9 release as well, so our near
> term timeline and requirements mean we are working on evolving
> modularity, not a revolution or a replacement.  We are excited by ELN,
> as it presents a possible space to allow those that want to continue
> to iterate on modules a place to do so without necessarily impacting
> the broader Fedora distribution in its entirety.  It is my personal
> hope that we can use that opportunity to improve modules and
> modularity in the open source, Fedora-first way we’d prefer.  Our near
> term effort to improve the existing modularity implementation ahead of
> RHEL 9 needs to occur, and we’d like to do that work in Fedora, rather
> than in closed product development.  Longer term, we are open to
> contributing to a better replacement that meets many of the same
> goals.  This is what makes our distribution ecosystem work well, and
> not having that upstream lessens the value we all get from such
> experimentation in the open.
>
> Hopefully that provides some context and helps FESCo and the wider
> community understand where Red Hat is headed with modularity on the
> Enterprise side. 
Around the idea and concept of modularity... what are the benefits to Fedora,
Fedora developers, and Fedora contributors?  Through the various discussions
on modularity, nothing solid in this regard has been presented.  If I am
Fedora contributor now, what can modularity do for me?

Most of the remainder of this thread talks about the problems with the
implementation as it exists today and problems with other known options.
Putting that aside for now, why should Fedora contributors care about
modularity?

Put another way, what does the developer experience look like for modularity?

Thanks,

-- 
David Cantrell <dcantrell(a)redhat.com&gt;
Red Hat, Inc. | Boston, MA | EST5EDT

[Josh Boyer]

On Fri, Jun 19, 2020 at 2:54 PM David Cantrell <dcantrell(a)redhat.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 08:44:39AM -0400, Josh Boyer wrote:
>  >Hopefully that provides some context and helps FESCo and the wider
>  >community understand where Red Hat is headed with modularity on the
>  >Enterprise side.
>
>  Around the idea and concept of modularity... what are the benefits to Fedora,
>  Fedora developers, and Fedora contributors?  Through the various discussions
>  on modularity, nothing solid in this regard has been presented.  If I am
>  Fedora contributor now, what can modularity do for me?
>
>  Most of the remainder of this thread talks about the problems with the
>  implementation as it exists today and problems with other known options.
>  Putting that aside for now, why should Fedora contributors care about
>  modularity?
>
>  Put another way, what does the developer experience look like for modularity? 
These are good questions, but I feel like there has been about 2+
years of discussion and debate about what Fedora could get out of
modularity.  I'm not sure I have anything to offer that is directly
new or better.  I do think that ELN presents a new opportunity for
those that have already found value in the Fedora community to use and
improve it there in the absence of widespread Fedora adoption.
Perhaps that would allow a place for the ideas to grow and could prove
to be illustrative.

At any rate, my goal isn't to evangelize or convert Fedora to suddenly
embrace modularity wholescale.  I am simply offering clarity on the
plans for RHEL 9 with this technology.

josh

[Dan Čermák]

Josh Boyer <jwboyer(a)redhat.com&gt; writes:

>  On Fri, Jun 19, 2020 at 2:54 PM David Cantrell
> <dcantrell(a)redhat.com&gt; wrote:
> >
> > On Thu, Jun 18, 2020 at 08:44:39AM -0400, Josh Boyer wrote:
> > >Hopefully that provides some context and helps FESCo and the wider
> > >community understand where Red Hat is headed with modularity on the
> > >Enterprise side.
> >
> > Around the idea and concept of modularity... what are the benefits to Fedora,
> > Fedora developers, and Fedora contributors?  Through the various discussions
> > on modularity, nothing solid in this regard has been presented.  If I am
> > Fedora contributor now, what can modularity do for me?
> >
> > Most of the remainder of this thread talks about the problems with the
> > implementation as it exists today and problems with other known options.
> > Putting that aside for now, why should Fedora contributors care about
> > modularity?
> >
> > Put another way, what does the developer experience look like for modularity?
>
>  These are good questions, but I feel like there has been about 2+
>  years of discussion and debate about what Fedora could get out of
>  modularity. 
Well, a short tl;dr; would certainly help, as I must admit that even
after 2+ years of discussion I see very little incentive to modularize
any of my packages.

I see benefits of modularity for CentOS/RHEL, but not so clearly for
Fedora (except for more special cases like sway, where we have the
quickly evolving wlroots library and can thus deliver an up to date sway
even for older Fedora releases).

>  I'm not sure I have anything to offer that is directly
>  new or better.  I do think that ELN presents a new opportunity for
>  those that have already found value in the Fedora community to use and
>  improve it there in the absence of widespread Fedora adoption.
>  Perhaps that would allow a place for the ideas to grow and could prove
>  to be illustrative.
>
>  At any rate, my goal isn't to evangelize or convert Fedora to suddenly
>  embrace modularity wholescale.  I am simply offering clarity on the
>  plans for RHEL 9 with this technology.
>
>  josh
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Neal Gompa]

On Sat, Jun 20, 2020 at 5:31 AM Dan Čermák
<dan.cermak(a)cgc-instruments.com&gt; wrote:
>  Josh Boyer <jwboyer(a)redhat.com&gt; writes:
>
>  > On Fri, Jun 19, 2020 at 2:54 PM David Cantrell <dcantrell(a)redhat.com&gt; wrote:
>  >>
>  >> On Thu, Jun 18, 2020 at 08:44:39AM -0400, Josh Boyer wrote:
>  >> >Hopefully that provides some context and helps FESCo and the wider
>  >> >community understand where Red Hat is headed with modularity on the
>  >> >Enterprise side.
>  >>
>  >> Around the idea and concept of modularity... what are the benefits to Fedora,
>  >> Fedora developers, and Fedora contributors?  Through the various discussions
>  >> on modularity, nothing solid in this regard has been presented.  If I am
>  >> Fedora contributor now, what can modularity do for me?
>  >>
>  >> Most of the remainder of this thread talks about the problems with the
>  >> implementation as it exists today and problems with other known options.
>  >> Putting that aside for now, why should Fedora contributors care about
>  >> modularity?
>  >>
>  >> Put another way, what does the developer experience look like for modularity?
>  >
>  > These are good questions, but I feel like there has been about 2+
>  > years of discussion and debate about what Fedora could get out of
>  > modularity.
>
>  Well, a short tl;dr; would certainly help, as I must admit that even
>  after 2+ years of discussion I see very little incentive to modularize
>  any of my packages.
>
>  I see benefits of modularity for CentOS/RHEL, but not so clearly for
>  Fedora (except for more special cases like sway, where we have the
>  quickly evolving wlroots library and can thus deliver an up to date sway
>  even for older Fedora releases).
>  
TL;DR benefits of modularity for Fedora:

* Automating build chains for producing artifacts
* Straightforward mechanism of producing non-rpm artifacts using our
existing tooling (modules -> flatpaks/containers/etc.)
* Path to provide alternative versions of stacks that don't natively
multiversion (Nodejs, Perl, PHP, etc.)




--
真実はいつも一つ！/ Always, there's only one truth!

[John M. Harris Jr]

On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  TL;DR benefits of modularity for Fedora:
>  
>  * Automating build chains for producing artifacts
>  * Straightforward mechanism of producing non-rpm artifacts using our
>  existing tooling (modules -> flatpaks/containers/etc.) 
Both of these have nothing to do with Modularity, and can be done with 
existing RPMs.

>  * Path to provide alternative versions of stacks that don't
> natively
>  multiversion (Nodejs, Perl, PHP, etc.) 
Modularity doesn't support installing multiple versions of the same software. 
It's one of the key issues with the tech.

-- 
John M. Harris, Jr.

[Neal Gompa]

On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt; wrote:
>  On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  > TL;DR benefits of modularity for Fedora:
>  >
>  > * Automating build chains for producing artifacts
>  > * Straightforward mechanism of producing non-rpm artifacts using our
>  > existing tooling (modules -> flatpaks/containers/etc.)
>
>  Both of these have nothing to do with Modularity, and can be done with
>  existing RPMs.
>  
They have everything to do with Modularity, because that layer is
where that stuff was implemented. Modularity was the result of the
efforts involved with Factory 2.0, which gave us a lot of improvements
in our build infrastructure tooling for the first time since 2007.
Most of that rolled out in 2017, a full ten years after the last
revamp of our infrastructure.

>  > * Path to provide alternative versions of stacks that don't
> natively
>  > multiversion (Nodejs, Perl, PHP, etc.)
>
>  Modularity doesn't support installing multiple versions of the same software.
>  It's one of the key issues with the tech.
>  
Modules can be designed to be parallel installable if the underlying
software natively supports that. For example, Python works that way
now, and thus in RHEL there are parallel versions of Python shipped as
modules. It doesn't change the nature of the software.

But it makes it easier to make multiple complete, yet conflicting,
collections of a language stack.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[John M. Harris Jr]

On Saturday, June 20, 2020 2:40:48 PM MST Neal Gompa wrote:
>  On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr
> <johnmh(a)splentity.com&gt;
>  wrote:
>  >
>  >
>  > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  > 
>  > > TL;DR benefits of modularity for Fedora:
>  > >
>  > >
>  > >
>  > > * Automating build chains for producing artifacts
>  > > * Straightforward mechanism of producing non-rpm artifacts using our
>  > > existing tooling (modules -> flatpaks/containers/etc.)
>  >
>  >
>  >
>  > Both of these have nothing to do with Modularity, and can be done with
>  > existing RPMs.
>  >
>  >
>  
>  
>  They have everything to do with Modularity, because that layer is
>  where that stuff was implemented. Modularity was the result of the
>  efforts involved with Factory 2.0, which gave us a lot of improvements
>  in our build infrastructure tooling for the first time since 2007.
>  Most of that rolled out in 2017, a full ten years after the last
>  revamp of our infrastructure. 
As far as I'm aware, flatpacks can be created without any Modules. Containers 
certainly can, we've been doing that for over a decade now without them.

>  > > * Path to provide alternative versions of stacks that
> don't natively
>  > > multiversion (Nodejs, Perl, PHP, etc.)
>  >
>  >
>  >
>  > Modularity doesn't support installing multiple versions of the same
>  > software. It's one of the key issues with the tech.
>  >
>  >
>  
>  
>  Modules can be designed to be parallel installable if the underlying
>  software natively supports that. For example, Python works that way
>  now, and thus in RHEL there are parallel versions of Python shipped as
>  modules. It doesn't change the nature of the software.
>  
>  But it makes it easier to make multiple complete, yet conflicting,
>  collections of a language stack. 
Where the underlying software already supports it, you don't need Modules to 
do that, just regular old packages. See Python, for example. Modularity is not 
a requirement for that.

-- 
John M. Harris, Jr.

[Neal Gompa]

On Sat, Jun 20, 2020 at 6:51 PM John M. Harris Jr <johnmh(a)splentity.com&gt; wrote:
>  On Saturday, June 20, 2020 2:40:48 PM MST Neal Gompa wrote:
>  > On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
>  > wrote:
>  > >
>  > >
>  > > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  > >
>  > > > TL;DR benefits of modularity for Fedora:
>  > > >
>  > > >
>  > > >
>  > > > * Automating build chains for producing artifacts
>  > > > * Straightforward mechanism of producing non-rpm artifacts using our
>  > > > existing tooling (modules -> flatpaks/containers/etc.)
>  > >
>  > >
>  > >
>  > > Both of these have nothing to do with Modularity, and can be done with
>  > > existing RPMs.
>  > >
>  > >
>  >
>  >
>  > They have everything to do with Modularity, because that layer is
>  > where that stuff was implemented. Modularity was the result of the
>  > efforts involved with Factory 2.0, which gave us a lot of improvements
>  > in our build infrastructure tooling for the first time since 2007.
>  > Most of that rolled out in 2017, a full ten years after the last
>  > revamp of our infrastructure.
>
>  As far as I'm aware, flatpacks can be created without any Modules. Containers
>  certainly can, we've been doing that for over a decade now without them.
>  
Yes, they can, but it's a lot more painful to do so. The Modularity
tooling automates reusing the existing RPM packaging we have and
transforms them into building blocks used to produce a Flatpak
application bundle based on the Fedora runtime that people can use
easily on Fedora or any other distribution.

As for containers, the modularity tooling makes it possible to produce
base containers for various language stacks that look and feel like
native stacks provided by Fedora. You've probably never used any of
the RHEL UBI stuff, or any of the language stack containers built on
UBI. Those containers are *fantastic* to work with. My developer
friends find them a literal *joy* to use compared to the alternatives.
I want *that* in Fedora as well, so people can take advantage of the
fresher base and attract more developer-types to our community.

And Modularity makes that stuff possible in a reasonable way.

>  > > > * Path to provide alternative versions of stacks that
> don't natively
>  > > > multiversion (Nodejs, Perl, PHP, etc.)
>  > >
>  > >
>  > >
>  > > Modularity doesn't support installing multiple versions of the same
>  > > software. It's one of the key issues with the tech.
>  > >
>  > >
>  >
>  >
>  > Modules can be designed to be parallel installable if the underlying
>  > software natively supports that. For example, Python works that way
>  > now, and thus in RHEL there are parallel versions of Python shipped as
>  > modules. It doesn't change the nature of the software.
>  >
>  > But it makes it easier to make multiple complete, yet conflicting,
>  > collections of a language stack.
>
>  Where the underlying software already supports it, you don't need Modules to
>  do that, just regular old packages. See Python, for example. Modularity is not
>  a requirement for that.
>  
It is easy without modules for the interpreter, but it's a royal pain
to do it for *everything* (interpreter, language package manager, and
all associated Python modules). Modules provide a straightforward way
to define what that would look like and build that in a repeatable
manner. A Python stack is roughly ~4K source packages, with all kinds
of "fun" ordering requirements. The Modularity tooling provides a way
to do this sort of thing properly. Without Modularity, we've got
*nothing* to do that in a way that's reasonably automated and easy to
maintain.


-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen John Smoogen]

On Sat, 20 Jun 2020 at 17:42, Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt; wrote:
>  >
>  > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  > > TL;DR benefits of modularity for Fedora:
>  > >
>  > > * Automating build chains for producing artifacts
>  > > * Straightforward mechanism of producing non-rpm artifacts using our
>  > > existing tooling (modules -> flatpaks/containers/etc.)
>  >
>  > Both of these have nothing to do with Modularity, and can be done with
>  > existing RPMs.
>  >
>
>  They have everything to do with Modularity, because that layer is
>  where that stuff was implemented. Modularity was the result of the
>  efforts involved with Factory 2.0, which gave us a lot of improvements
>  in our build infrastructure tooling for the first time since 2007.
>  Most of that rolled out in 2017, a full ten years after the last
>  revamp of our infrastructure.
>  
I think that John and others aren't aware of how a module is built
enough to understand what you meant by
* Automating build chains for producing artifacts
compared to how it is done normally.

In normal times, a packager goes through a list of packages, updates
spec files with new tags and rebuilds them one by one as needed..
sometimes multiple times because of bootstrapping or finding out that
the order you tried was wrong. A made up example from my days of doing
this for a different place (this isn't what is needed anymore but long
ago I had something similar):
bison
flex
gcc [options 1]
bison
flex
gcc [options 2]
glibc
bison
flex
gcc

do them in one order and the apps came out working... do them in the
wrong order and it might not. Rust, Java and other language stacks
have similar loops. A packager may have to coordinate with multiple
people to do this several times.

In a module, you write that all down in the manifest with the order
that you want packages built in and if you need to loop through them
with different options. Then MBS does the builds in an automated
fashion and it is repeatable. To me this is the biggest win here as
for various groups of mass-rebuilds it cuts down errors when order
matters and you have to do multiple ones to get from package set A to
package set A+1.

Making it easier to make flatpacks and stuff also is built into the
tools which came with modularity. The tools which were doing it for
the Fedora buildsystem before this  were 'fragile'.

Yes a packager or system administrator can build these things without
the modularity build system but trying to do it in scale in Fedora
ended up needing the tools which came with MBS.


-- 
Stephen J Smoogen.

[John M. Harris Jr]

On Saturday, June 20, 2020 4:37:06 PM MST Stephen John Smoogen wrote:
>  On Sat, 20 Jun 2020 at 17:42, Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  
>  >
>  >
>  > On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
>  > wrote:
>  
>  > >
>  > >
>  > > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  > > 
>  > > > TL;DR benefits of modularity for Fedora:
>  > > >
>  > > >
>  > > >
>  > > > * Automating build chains for producing artifacts
>  > > > * Straightforward mechanism of producing non-rpm artifacts using our
>  > > > existing tooling (modules -> flatpaks/containers/etc.)
>  > >
>  > >
>  > >
>  > > Both of these have nothing to do with Modularity, and can be done with
>  > > existing RPMs.
>  > >
>  > >
>  >
>  >
>  >
>  > They have everything to do with Modularity, because that layer is
>  > where that stuff was implemented. Modularity was the result of the
>  > efforts involved with Factory 2.0, which gave us a lot of improvements
>  > in our build infrastructure tooling for the first time since 2007.
>  > Most of that rolled out in 2017, a full ten years after the last
>  > revamp of our infrastructure.
>  >
>  >
>  
>  
>  I think that John and others aren't aware of how a module is built
>  enough to understand what you meant by
>  * Automating build chains for producing artifacts
>  compared to how it is done normally.
>  
>  In normal times, a packager goes through a list of packages, updates
>  spec files with new tags and rebuilds them one by one as needed..
>  sometimes multiple times because of bootstrapping or finding out that
>  the order you tried was wrong. A made up example from my days of doing
>  this for a different place (this isn't what is needed anymore but long
>  ago I had something similar):
>  bison
>  flex
>  gcc [options 1]
>  bison
>  flex
>  gcc [options 2]
>  glibc
>  bison
>  flex
>  gcc
>  
>  do them in one order and the apps came out working... do them in the
>  wrong order and it might not. Rust, Java and other language stacks
>  have similar loops. A packager may have to coordinate with multiple
>  people to do this several times.
>  
>  In a module, you write that all down in the manifest with the order
>  that you want packages built in and if you need to loop through them
>  with different options. Then MBS does the builds in an automated
>  fashion and it is repeatable. To me this is the biggest win here as
>  for various groups of mass-rebuilds it cuts down errors when order
>  matters and you have to do multiple ones to get from package set A to
>  package set A+1.
>  
>  Making it easier to make flatpacks and stuff also is built into the
>  tools which came with modularity. The tools which were doing it for
>  the Fedora buildsystem before this  were 'fragile'.
>  
>  Yes a packager or system administrator can build these things without
>  the modularity build system but trying to do it in scale in Fedora
>  ended up needing the tools which came with MBS. 
While the tooling came at the same time, it doesn't necessarily need 
Modularity. See Ken Dreyer's example in the above subthread.

We don't need Modules to do a lot of things that happened to drop around the 
same time, or that were created to work with Modularity.

-- 
John M. Harris, Jr.

[Dan Čermák]

Hi Stephen,

this will probably get buried given the immense amount of replies in
this thread already, but nevertheless, here's my 2cts.

Stephen John Smoogen <smooge(a)gmail.com&gt; writes:

>  On Sat, 20 Jun 2020 at 17:42, Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
> >
> > On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
> wrote:
> > >
> > > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
> > > > TL;DR benefits of modularity for Fedora:
> > > >
> > > > * Automating build chains for producing artifacts
> > > > * Straightforward mechanism of producing non-rpm artifacts using our
> > > > existing tooling (modules -> flatpaks/containers/etc.)
> > >
> > > Both of these have nothing to do with Modularity, and can be done with
> > > existing RPMs.
> > >
> >
> > They have everything to do with Modularity, because that layer is
> > where that stuff was implemented. Modularity was the result of the
> > efforts involved with Factory 2.0, which gave us a lot of improvements
> > in our build infrastructure tooling for the first time since 2007.
> > Most of that rolled out in 2017, a full ten years after the last
> > revamp of our infrastructure.
> >
>
>  I think that John and others aren't aware of how a module is built
>  enough to understand what you meant by
>  * Automating build chains for producing artifacts
>  compared to how it is done normally.
>
>  In normal times, a packager goes through a list of packages, updates
>  spec files with new tags and rebuilds them one by one as needed..
>  sometimes multiple times because of bootstrapping or finding out that
>  the order you tried was wrong. A made up example from my days of doing
>  this for a different place (this isn't what is needed anymore but long
>  ago I had something similar):
>  bison
>  flex
>  gcc [options 1]
>  bison
>  flex
>  gcc [options 2]
>  glibc
>  bison
>  flex
>  gcc
>
>  do them in one order and the apps came out working... do them in the
>  wrong order and it might not. Rust, Java and other language stacks
>  have similar loops. A packager may have to coordinate with multiple
>  people to do this several times.
>
>  In a module, you write that all down in the manifest with the order
>  that you want packages built in and if you need to loop through them
>  with different options. Then MBS does the builds in an automated
>  fashion and it is repeatable. To me this is the biggest win here as
>  for various groups of mass-rebuilds it cuts down errors when order
>  matters and you have to do multiple ones to get from package set A to
>  package set A+1. 
This is actually freaking awesome! The one thing where our build system
is really laking is that we have to manually figure out the build order
and rebuild stuff manually. Other systems like OBS handle this
completely automatically, albeit bootstrapping on OBS has its issues as
well.

Anyway, is there any chance that we can get this functionality split out
of MBS and add it to koji? If we then convince koschei to kick of real
builds in Rawhide, then we'd be in a much better shape build system wise
imho.

>  Making it easier to make flatpacks and stuff also is built into the
>  tools which came with modularity. The tools which were doing it for
>  the Fedora buildsystem before this  were 'fragile'.
>
>  Yes a packager or system administrator can build these things without
>  the modularity build system but trying to do it in scale in Fedora
>  ended up needing the tools which came with MBS.
>
>
>  -- 
>  Stephen J Smoogen.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Stephen John Smoogen]

On Thu, 25 Jun 2020 at 05:29, Dan Čermák <dan.cermak(a)cgc-instruments.com&gt; wrote:
>  Hi Stephen,
>
>  this will probably get buried given the immense amount of replies in
>  this thread already, but nevertheless, here's my 2cts.
>
>  Stephen John Smoogen <smooge(a)gmail.com&gt; writes:
>
>  > On Sat, 20 Jun 2020 at 17:42, Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >>
>  >> On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr <johnmh(a)splentity.com&gt;
> wrote:
>  >> >
>  >> > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
>  >> > > TL;DR benefits of modularity for Fedora:
>  >> > >
>  >> > > * Automating build chains for producing artifacts
>  >> > > * Straightforward mechanism of producing non-rpm artifacts using our
>  >> > > existing tooling (modules -> flatpaks/containers/etc.)
>  >> >
>  >> > Both of these have nothing to do with Modularity, and can be done with
>  >> > existing RPMs.
>  >> >
>  >>
>  >> They have everything to do with Modularity, because that layer is
>  >> where that stuff was implemented. Modularity was the result of the
>  >> efforts involved with Factory 2.0, which gave us a lot of improvements
>  >> in our build infrastructure tooling for the first time since 2007.
>  >> Most of that rolled out in 2017, a full ten years after the last
>  >> revamp of our infrastructure.
>  >>
>  >
>  > I think that John and others aren't aware of how a module is built
>  > enough to understand what you meant by
>  > * Automating build chains for producing artifacts
>  > compared to how it is done normally.
>  >
>  > In normal times, a packager goes through a list of packages, updates
>  > spec files with new tags and rebuilds them one by one as needed..
>  > sometimes multiple times because of bootstrapping or finding out that
>  > the order you tried was wrong. A made up example from my days of doing
>  > this for a different place (this isn't what is needed anymore but long
>  > ago I had something similar):
>  > bison
>  > flex
>  > gcc [options 1]
>  > bison
>  > flex
>  > gcc [options 2]
>  > glibc
>  > bison
>  > flex
>  > gcc
>  >
>  > do them in one order and the apps came out working... do them in the
>  > wrong order and it might not. Rust, Java and other language stacks
>  > have similar loops. A packager may have to coordinate with multiple
>  > people to do this several times.
>  >
>  > In a module, you write that all down in the manifest with the order
>  > that you want packages built in and if you need to loop through them
>  > with different options. Then MBS does the builds in an automated
>  > fashion and it is repeatable. To me this is the biggest win here as
>  > for various groups of mass-rebuilds it cuts down errors when order
>  > matters and you have to do multiple ones to get from package set A to
>  > package set A+1.
>
>  This is actually freaking awesome! The one thing where our build system
>  is really laking is that we have to manually figure out the build order
>  and rebuild stuff manually. Other systems like OBS handle this
>  completely automatically, albeit bootstrapping on OBS has its issues as
>  well.
>
>  Anyway, is there any chance that we can get this functionality split out
>  of MBS and add it to koji? If we then convince koschei to kick of real
>  builds in Rawhide, then we'd be in a much better shape build system wise
>  imho.
>  
Not without packagers en-mass accepting changes to how 'THEIR'
packages are built. That middle gcc needs to have a different RPM
release than the one before it and the one after it. That means that
in our current work flow you could start working on a package and
between your starting on updating a .spec file and committing and
trying to push it.. it automatically got bumped several above because
someone else was doing these builds to get 'their' package built
correctly. This has been the gordian knot of our package
ownership/maintenance.. with various packagers reverting and undoing
things in the middle of someone else because THEIR package got updated
in a way they didn't want. This has been fought over since the first
days of the Fedora.us project and honestly never seems to go away.

The political solution was to assume that in a module you take
responsibility for all the packages in that module even if other
people 'own' their 'non-modular' versions. You can then do what is
needed to the packages and then not publish those special middle
versions of gcc which you needed but would have broken normal gcc for
everyone else.. or some other odd thing. [This again is all made up
examples.. it could be a language which seems to need you to build
each version only from a specific previous version and so you need to
chain an 'infinite' list to bootstrap to a newer version.]

I honestly do not know enough about OpenSuSE Build System to know how
they deal with this automatically. I had assumed it wasn't too
automatic but comes from how detailed Suse spec files are so that you
don't have a package solver 'guess' what was needed. That simplifies
the dependency graph a lot.  Also that packagers do not 'own' things
in the same way they have in Fedora land so the politics is skipped.
-- 
Stephen J Smoogen.

[Vít Ondruch]

Dne 20. 06. 20 v 23:40 Neal Gompa napsal(a):
>  On Sat, Jun 20, 2020 at 5:25 PM John M. Harris Jr
> <johnmh(a)splentity.com&gt; wrote:
> > On Saturday, June 20, 2020 4:42:17 AM MST Neal Gompa wrote:
> >> TL;DR benefits of modularity for Fedora:
> >>
> >> * Automating build chains for producing artifacts
> >> * Straightforward mechanism of producing non-rpm artifacts using our
> >> existing tooling (modules -> flatpaks/containers/etc.)
> > Both of these have nothing to do with Modularity, and can be done with
> > existing RPMs.
> >
>  They have everything to do with Modularity, because that layer is
>  where that stuff was implemented. Modularity was the result of the
>  efforts involved with Factory 2.0, which gave us a lot of improvements
>  in our build infrastructure tooling for the first time since 2007.
>  Most of that rolled out in 2017, a full ten years after the last
>  revamp of our infrastructure.
>
> >> * Path to provide alternative versions of stacks that don't natively
> >> multiversion (Nodejs, Perl, PHP, etc.)
> > Modularity doesn't support installing multiple versions of the same software.
> > It's one of the key issues with the tech.
> >
>  Modules can be designed to be parallel installable if the underlying
>  software natively supports that. 

I think you are freely interchange "modules" and "module streams".
While
modules can indeed parallel installable, the module streams cannot even
if the underlying software natively supports that and even if the RPMs
don't collide.


>  For example, Python works that way
>  now, and thus in RHEL there are parallel versions of Python shipped as
>  modules. It doesn't change the nature of the software. 

Having python27 and python36 modules is fail, because these should be
2.7 and 3.6 streams of python module.


Vít


>  But it makes it easier to make multiple complete, yet conflicting,
>  collections of a language stack.
>
>
>

[Miro Hrončok]

On 24. 06. 20 14:41, Vít Ondruch wrote:
>  Having python27 and python36 modules is fail, because these should
> be
>  2.7 and 3.6 streams of python module. 
Oh. We are so sorry for the failure. Could you please report is as a bug in RHEL 
8 and explain why it is a problem?

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Vít Ondruch]

Dne 24. 06. 20 v 15:47 Miro Hrončok napsal(a):
>  On 24. 06. 20 14:41, Vít Ondruch wrote:
> > Having python27 and python36 modules is fail, because these should be
> > 2.7 and 3.6 streams of python module.
>
>  Oh. We are so sorry for the failure. Could you please report is as a
>  bug in RHEL 8 and explain why it is a problem?
>  
If you have not stripped the context, it would be more obvious.

There is no philosophical (or design?) reason to not have Python 2.7 and
3.6 streams in one module. There are just technical reasons, such as
that you want to install them in parallel which is not possible.


Vít

[Neal Gompa]

On Wed, Jun 24, 2020 at 10:45 AM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 24. 06. 20 v 15:47 Miro Hrončok napsal(a):
>  > On 24. 06. 20 14:41, Vít Ondruch wrote:
>  >> Having python27 and python36 modules is fail, because these should be
>  >> 2.7 and 3.6 streams of python module.
>  >
>  > Oh. We are so sorry for the failure. Could you please report is as a
>  > bug in RHEL 8 and explain why it is a problem?
>  >
>
>  If you have not stripped the context, it would be more obvious.
>
>  There is no philosophical (or design?) reason to not have Python 2.7 and
>  3.6 streams in one module. There are just technical reasons, such as
>  that you want to install them in parallel which is not possible.
>  
My CentOS 8.2 system with Python 2.7, 3.6, and 3.8 all simultaneously
installed disagrees with this statement. Modularity makes no effort to
hide the complexity of parallel installability. It only helps make
clearer how to handle availability of multiple sets of content from a
repository. Parallel installability is up to the module author.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Vít Ondruch]

Dne 24. 06. 20 v 17:04 Neal Gompa napsal(a):
>  On Wed, Jun 24, 2020 at 10:45 AM Vít Ondruch
> <vondruch(a)redhat.com&gt; wrote:
> >
> > Dne 24. 06. 20 v 15:47 Miro Hrončok napsal(a):
> >> On 24. 06. 20 14:41, Vít Ondruch wrote:
> >>> Having python27 and python36 modules is fail, because these should be
> >>> 2.7 and 3.6 streams of python module.
> >> Oh. We are so sorry for the failure. Could you please report is as a
> >> bug in RHEL 8 and explain why it is a problem?
> >>
> > If you have not stripped the context, it would be more obvious.
> >
> > There is no philosophical (or design?) reason to not have Python 2.7 and
> > 3.6 streams in one module. There are just technical reasons, such as
> > that you want to install them in parallel which is not possible.
> >
>  My CentOS 8.2 system with Python 2.7, 3.6, and 3.8 all simultaneously
>  installed disagrees with this statement. Modularity makes no effort to
>  hide the complexity of parallel installability. It only helps make
>  clearer how to handle availability of multiple sets of content from a
>  repository. Parallel installability is up to the module author. 

Is it? Does modularity allow to install multiple streams if they don't
collide? IOW if there was python module with non colliding 2.7 and 3.6
streams, would I be able to simply run:


~~~

$ sudo dnf module enable python 2.7

$ sudo dnf module enable python 3.6

$ python2.7 -V

Python 2.7.5

$ python3.6 -V

Python 3.6.8

~~~


Reading following resources, I don't think it is possible, but I might
be wrong:


https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/

https://docs.fedoraproject.org/en-US/modularity/using-modules-switching-s...


If the above is really possible, I would like to see the documentation
extended to cover such scenario.


Vít

[Nico Kadel-Garcia]

On Wed, Jun 24, 2020 at 10:45 AM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
>  Dne 24. 06. 20 v 15:47 Miro Hrončok napsal(a):
>  > On 24. 06. 20 14:41, Vít Ondruch wrote:
>  >> Having python27 and python36 modules is fail, because these should be
>  >> 2.7 and 3.6 streams of python module.
>  >
>  > Oh. We are so sorry for the failure. Could you please report is as a
>  > bug in RHEL 8 and explain why it is a problem?
>  >
>
>  If you have not stripped the context, it would be more obvious.
>
>  There is no philosophical (or design?) reason to not have Python 2.7 and
>  3.6 streams in one module. There are just technical reasons, such as
>  that you want to install them in parallel which is not possible. 
Not publishing unsupportable and inevitably compatible software which
relies on two entirely distinct versions of the same interpreter is a
pretty compelling set of philosophical and design reasons.

[Petr Pisar]

On Fri, Jun 19, 2020 at 02:53:56PM -0400, David Cantrell wrote:
>  Around the idea and concept of modularity... what are the benefits to
> Fedora,
>  Fedora developers, and Fedora contributors?  Through the various discussions
>  on modularity, nothing solid in this regard has been presented.  If I am
>  Fedora contributor now, what can modularity do for me?
>   
It can define a build order among packages of a module.
It can change RPM macros in a build root (including SRPM build root).
It can use transient build dependencies (a package is built, only used as a build
dependency of a consquitive package, and then dropped and never shipped).
It can build and deliver a package against multiple exclusive dependencies
(e.g. for different Fedora releases).
It can build a package from dist-git sources that do not depend on a Fedora
release.
(Teoretically you can build a module from non-RPM components; I've never tried
it.)

-- Petr

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Mon, 2020-06-22 at 09:19 +0200, Petr Pisar wrote:
>  On Fri, Jun 19, 2020 at 02:53:56PM -0400, David Cantrell wrote:
>  > Around the idea and concept of modularity... what are the benefits
>  > to Fedora,
>  > Fedora developers, and Fedora contributors?  Through the various
>  > discussions
>  > on modularity, nothing solid in this regard has been presented.  If
>  > I am
>  > Fedora contributor now, what can modularity do for me?
>  > 
>  It can define a build order among packages of a module. 
For that you don't actually have to design some special format. The
buildsystem should figure this out on its own. And there are such
buildsystems that do that - Open Build Service.

>  It can change RPM macros in a build root (including SRPM build root).
I think this is not really good idea, but again - you don't have to
invent new format for this. Open Build Service allows you to do that in
a project configuration.

>  It can use transient build dependencies (a package is built, only
>  used as a build
>  dependency of a consquitive package, and then dropped and never
>  shipped). 
Same here.

>  It can build and deliver a package against multiple exclusive
>  dependencies
>  (e.g. for different Fedora releases). 
The same thing.

>  It can build a package from dist-git sources that do not depend on a
>  Fedora
>  release. 
Again, same.

>  (Teoretically you can build a module from non-RPM components;
> I've
>  never tried
>  it.)
>  
>  -- Petr
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:      
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:      
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:      
>  https://lists.fedoraproject.org/archives/list/[e-mail] 
- -- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl7wdP4ACgkQEV1auJxc
> Hh5iiBAAsdv3yWu6y+Un2LIGFcAomRmwijA3C3aajdNXDHWO3mco/MYDukjPVMIl
> FHO7xDg13ZCvxQ+AsKyVjJwRXI8ugyLatLBTO9sXLUw8LBtl36RfH/P2wXYCA6kM
> kB1aWYxxBlBLuZ2cJmyYJHfXEtOFDQHz1gORIX5HyXnL/asXOPwgeh+VjYGnNMPn
> FHSGCelFrW+aRU8E6vxxuqNrqFwZwXbv3ccHEJIihRMeFKoGFqfDRy/KMQFQ7zZm
> NagszNoFpfAhRLEqRYaHg3GR0Kl8DYlcJCNJrXmoBdlu/nlDy2soOOzs7PiiFtuN
> izdZeASITvpcBvR5h/wt52IZCAGP/G/YmSZE73LWteBZhiUBimuAzl1cAEQWfPBc
> /y8/mdMXaGAKtQcWZ8ZZNMXYqpyR2Ahrl13fK2XwFBcyi6/mnCsX27KAlczhdnlS
> siBzxyQW8PBvcbZdWF2XlR4W39RHH+EHdIHeeHdGauc+Oent6xObXVW9HIp/Vvkh
> VH87cdSBPGiJDmZMXBWs1QqViTHzxSlyX8PI3bFxFoYA+7ZwrER75hQnKNN22Or+
> Bfl5i6EHtReatUW1uxtmVODON5hZ8fcxi8v6prjDAGTUMWHHkb6f7tcR1dH42/Sf
> 1OLXojU8gYMuuY/ooace7zto5ByfIJT0ViUFi0eV0qvG5w053D8=
> =iMXG
> -----END PGP SIGNATURE-----

[Petr Pisar]

On Mon, Jun 22, 2020 at 11:08:14AM +0200, Igor Raits wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA512
>  
>  On Mon, 2020-06-22 at 09:19 +0200, Petr Pisar wrote:
>  > On Fri, Jun 19, 2020 at 02:53:56PM -0400, David Cantrell wrote:
>  > > Around the idea and concept of modularity... what are the benefits
>  > > to Fedora,
>  > > Fedora developers, and Fedora contributors?  Through the various
>  > > discussions
>  > > on modularity, nothing solid in this regard has been presented.  If
>  > > I am
>  > > Fedora contributor now, what can modularity do for me?
>  > > 
>  > It can define a build order among packages of a module.
>  
>  For that you don't actually have to design some special format. The
>  buildsystem should figure this out on its own. And there are such
>  buildsystems that do that - Open Build Service.
>  
>  > It can change RPM macros in a build root (including SRPM build root).
>  
>  I think this is not really good idea, but again - you don't have to
>  invent new format for this. Open Build Service allows you to do that in
>  a project configuration.
>  
>  > It can use transient build dependencies (a package is built, only
>  > used as a build
>  > dependency of a consquitive package, and then dropped and never
>  > shipped).
>  
>  Same here.
>  
>  > It can build and deliver a package against multiple exclusive
>  > dependencies
>  > (e.g. for different Fedora releases).
>  
>  The same thing.
>  
>  > It can build a package from dist-git sources that do not depend on a
>  > Fedora
>  > release.
>  
>  Again, same.
>   
Igor, the question was about modularity. Not about Open Build Service.

-- Petr

[Nico Kadel-Garcia]

On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:

Modularity has been an interesting idea on paper, but not worth the
effort. It should not be used for RHEL 9.

>  It is always good to push the boundaries and search for better ideas
>  and improvements, and that is part of what makes Fedora great.  We are
>  doing this in the context of the RHEL 9 release as well, so our near
>  term timeline and requirements mean we are working on evolving
>  modularity, not a revolution or a replacement.  We are excited by ELN, 
Or: you could discard modularity for RHEL 9. My experience with recent
Fedora releases and RHEL 8 has been that it's been destabilizing to
active and to build environments, with no detectable benefit.

Can you, or can anyone else, cite a single issue it has helped with?
Especially one that was not better resolved with other existing
deployment tools, such as /etc/alternatives or version pinning?

[Neal Gompa]

On Sat, Jun 20, 2020 at 7:40 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  On Thu, Jun 18, 2020 at 8:45 AM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>
>  Modularity has been an interesting idea on paper, but not worth the
>  effort. It should not be used for RHEL 9.
>  
This is the wrong place to try to convince Red Hat otherwise. It's
also not going to happen. And frankly, I wouldn't want to.

>  > It is always good to push the boundaries and search for better
> ideas
>  > and improvements, and that is part of what makes Fedora great.  We are
>  > doing this in the context of the RHEL 9 release as well, so our near
>  > term timeline and requirements mean we are working on evolving
>  > modularity, not a revolution or a replacement.  We are excited by ELN,
>
>  Or: you could discard modularity for RHEL 9. My experience with recent
>  Fedora releases and RHEL 8 has been that it's been destabilizing to
>  active and to build environments, with no detectable benefit.
>
>  Can you, or can anyone else, cite a single issue it has helped with?
>  Especially one that was not better resolved with other existing
>  deployment tools, such as /etc/alternatives or version pinning? 
Yes, actually. The Nodejs and PHP language stacks, as they are shipped
in RHEL/CentOS 8, are tremendously useful in the current form.

From my perspective, there are two major advantages:

* The language stacks are maintained and operate consistently. What I
mean by this is that each variant of the PHP and Nodejs language
stacks presents the same, consistent interface. I can build on those
interfaces with one version and trivially move forward to the next one
when I'm ready.

* The module stream locking feature is useful in that I can trust that
updates will flow freely for those modules, and I don't have to fiddle
with any fragile configuration in the package manager to keep things
going as I desire. It gives me the necessary control to shift at *my*
pace while still trusting that the whole stack is supported.

Are there problems with it? Absolutely. It was a tremendous effort for
me to bring up modularity support in my internal infrastructure. But
once I got there, I was able to realize these benefits for my own
pipelines. There are still problems, and I've filed bug reports about
them and communicated to the various developers about them, but it's
on a "trending upward" path for me.

And I won't lie and say I don't want improvements. I want to be able
to leverage it more. I want improved packager ergonomics. I want
content reusability (builds across multiple modules to be reusable for
modular content instead of building for each module if the
configuration is effectively the same). And so on.

But I will *not* argue to drop it in RHEL (or even Fedora) because I
think it is useful in a lot of ways and has potential to really make
our lives better.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Pat Riehecky]

Thanks for providing this information!

I'll confess my primary worry/complaint about Modularity/AppStream in RHEL8 is one of
lifecycle.

The default Stream in RHEL8 does not have the full 10 year lifecycle[1] present in RHEL7. 
Folks doing a standard 'yum install ZZZZ' get the default stream, but may not be
aware it isn't promised to be patched for the full RHEL8 lifecycle.

I like the ability to fetch new and different versions.  However, I find the ability to
audit the module state on my system a bit lacking.  Which modules are installed?  When do
they reach end of life?  Building that information on a RHEL8 box is more of a challenge
than expected.

I'm hopeful that RHEL9's Modularity/AppStream plan will make this reporting easier
and ideally provide a default stream I can use for the full lifecycle of the OS.

Pat

[1] https://access.redhat.com/support/policy/updates/rhel8-app-streams-life-c...