[Modularity] Module streams with two different, non-overlapping, package sets?

[Severin Gehwolf]

Hi,

I'm exploring the Fedora Modularity world and would like to know
whether it's possible to have two different module streams install two
different sets of packages? If that's possible, would installing one
stream remove the packages from the other stream even though the set of
packages don't overlap? I'm aware that it's not a goal of modularity to
support parallel installability. But what if the packages themselves
already allow parallel installability?

Example:

Module name: jdk
Streams: 8, 11

1) "dnf module install jdk:8/default" installs these packages:

java-1.8.0-openjdk
java-1.8.0-openjdk-devel
java-1.8.0-openjdk-headless

2) "dnf module install jdk:11/default" installs these packages:

java-11-openjdk
java-11-openjdk-headless
java-11-openjdk-devel

After 1) *and* 2) would I have both module streams' packages on my
system? Would I have only the set from 2) with 1) removed?

Thanks for your help!

Cheers,
Severin

[Josh Boyer]

On Wed, Jun 20, 2018 at 7:52 AM Severin Gehwolf <sgehwolf(a)redhat.com&gt; wrote:
>  Hi,
>
>  I'm exploring the Fedora Modularity world and would like to know
>  whether it's possible to have two different module streams install two
>  different sets of packages? If that's possible, would installing one
>  stream remove the packages from the other stream even though the set of
>  packages don't overlap? I'm aware that it's not a goal of modularity to
>  support parallel installability. But what if the packages themselves
>  already allow parallel installability?
>
>  Example:
>
>  Module name: jdk
>  Streams: 8, 11
>
>  1) "dnf module install jdk:8/default" installs these packages:
>
>  java-1.8.0-openjdk
>  java-1.8.0-openjdk-devel
>  java-1.8.0-openjdk-headless
>
>  2) "dnf module install jdk:11/default" installs these packages:
>
>  java-11-openjdk
>  java-11-openjdk-headless
>  java-11-openjdk-devel
>
>  After 1) *and* 2) would I have both module streams' packages on my
>  system? Would I have only the set from 2) with 1) removed? 
I think you would only have the set from 2.

Parallel installation can't be done via streams, so it must be done at
the module level.  So you'd have jdk8 and jdk11 modules, each with a
default stream.  Given the default is specified, an installation would
look something like:

"dnf module install jdk8; dnf module install jdk11"

josh

[Severin Gehwolf]

On Wed, 2018-06-20 at 07:59 -0400, Josh Boyer wrote:
>  On Wed, Jun 20, 2018 at 7:52 AM Severin Gehwolf
> <sgehwolf(a)redhat.com&gt; wrote:
>  > 
>  > Hi,
>  > 
>  > I'm exploring the Fedora Modularity world and would like to know
>  > whether it's possible to have two different module streams install two
>  > different sets of packages? If that's possible, would installing one
>  > stream remove the packages from the other stream even though the set of
>  > packages don't overlap? I'm aware that it's not a goal of modularity to
>  > support parallel installability. But what if the packages themselves
>  > already allow parallel installability?
>  > 
>  > Example:
>  > 
>  > Module name: jdk
>  > Streams: 8, 11
>  > 
>  > 1) "dnf module install jdk:8/default" installs these packages:
>  > 
>  > java-1.8.0-openjdk
>  > java-1.8.0-openjdk-devel
>  > java-1.8.0-openjdk-headless
>  > 
>  > 2) "dnf module install jdk:11/default" installs these packages:
>  > 
>  > java-11-openjdk
>  > java-11-openjdk-headless
>  > java-11-openjdk-devel
>  > 
>  > After 1) *and* 2) would I have both module streams' packages on my
>  > system? Would I have only the set from 2) with 1) removed?
>  
>  I think you would only have the set from 2.
>  
>  Parallel installation can't be done via streams, so it must be done at
>  the module level.  So you'd have jdk8 and jdk11 modules, each with a
>  default stream.  Given the default is specified, an installation would
>  look something like:
>  
>  "dnf module install jdk8; dnf module install jdk11" 
That's interesting, thanks!

FWIW, I've tried the reverse :) I had jdk10 and jdk11 both specifying
packages java-openjdk{,-devel,-headless} and the install of jdk11
*upgraded* packages from the other (jdk10) module.

Cheers,
Severin

[Petr Šabata]

On Wed, Jun 20, 2018 at 02:15:00PM +0200, Severin Gehwolf wrote:
>  On Wed, 2018-06-20 at 07:59 -0400, Josh Boyer wrote:
>  > On Wed, Jun 20, 2018 at 7:52 AM Severin Gehwolf <sgehwolf(a)redhat.com&gt; wrote:
>  > > 
>  > > Hi,
>  > > 
>  > > I'm exploring the Fedora Modularity world and would like to know
>  > > whether it's possible to have two different module streams install two
>  > > different sets of packages? If that's possible, would installing one
>  > > stream remove the packages from the other stream even though the set of
>  > > packages don't overlap? I'm aware that it's not a goal of
> modularity to
>  > > support parallel installability. But what if the packages themselves
>  > > already allow parallel installability?
>  > > 
>  > > Example:
>  > > 
>  > > Module name: jdk
>  > > Streams: 8, 11
>  > > 
>  > > 1) "dnf module install jdk:8/default" installs these packages:
>  > > 
>  > > java-1.8.0-openjdk
>  > > java-1.8.0-openjdk-devel
>  > > java-1.8.0-openjdk-headless
>  > > 
>  > > 2) "dnf module install jdk:11/default" installs these packages:
>  > > 
>  > > java-11-openjdk
>  > > java-11-openjdk-headless
>  > > java-11-openjdk-devel
>  > > 
>  > > After 1) *and* 2) would I have both module streams' packages on my
>  > > system? Would I have only the set from 2) with 1) removed?
>  > 
>  > I think you would only have the set from 2.
>  > 
>  > Parallel installation can't be done via streams, so it must be done at
>  > the module level.  So you'd have jdk8 and jdk11 modules, each with a
>  > default stream.  Given the default is specified, an installation would
>  > look something like:
>  > 
>  > "dnf module install jdk8; dnf module install jdk11"
>  
>  That's interesting, thanks!
>  
>  FWIW, I've tried the reverse :) I had jdk10 and jdk11 both specifying
>  packages java-openjdk{,-devel,-headless} and the install of jdk11
>  *upgraded* packages from the other (jdk10) module. 
You could still do it with streams and simply encourage users
who need both to use containers, one with each stream stream
enabled :)

Parallel installation of streams on a single system indeed
isn't supported at this point and isn't planned anytime in the
near future.  In general it's a more complicated problem than
it might seem at first.

P

[Mikolaj Izdebski]

On 06/20/2018 02:30 PM, Petr Šabata wrote:
>  Parallel installation of streams on a single system indeed
>  isn't supported at this point and isn't planned anytime in the
>  near future.  In general it's a more complicated problem than
>  it might seem at first. 
Could you elaborate and explain what's so complicated about it?

-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Stephen Gallagher]

On Wed, Jun 20, 2018 at 10:06 AM Mikolaj Izdebski <mizdebsk(a)redhat.com&gt;
wrote:

>  On 06/20/2018 02:30 PM, Petr Šabata wrote:
>  > Parallel installation of streams on a single system indeed
>  > isn't supported at this point and isn't planned anytime in the
>  > near future.  In general it's a more complicated problem than
>  > it might seem at first.
>
>  Could you elaborate and explain what's so complicated about it?
>  
The short answer is that there exists no generic solution for
parallel-installation. Many packages rely on well-known resources[1] and
cannot be parallel-installed at all. Other packages *may* support
parallel-installation but consumers must take special steps to enable
support for it.

I don't have a good link right now, but folks at Red Hat did a number of
customer studies and determined that in real-world deployments,
parallel-installation was very rarely used. Generally, the OS was
established with a standard set of packages and then anything that needed
an alternate version was deployed as a VM or container.

So, when we started looking at ways to provide alternative software, we
determined that parallel-installation was a non-goal. Not needing to solve
an unsolvable problem meant that we could focus on the parts that really
matter: allowing people to select which single version of the software
meets their needs.

[1] I can't install two packages that try to serve the same well-known
D-BUS name or both provide a local DNS cache, for example. I can't install
two packages that both own the same executable name without using the
"alternatives" system. I can't install two versions of a package that
provide a shared object plugin with the same name. And so on and so forth.

All of these have possible solutions, but no single generic solution
exists. In the general case, we figure that the triumvirate of modules,
containers and VMs solves this problem for 99% of cases. The remaining
exception cases have to be dealt with individually.

[Mikolaj Izdebski]

On 06/20/2018 04:15 PM, Stephen Gallagher wrote:
>  On Wed, Jun 20, 2018 at 10:06 AM Mikolaj Izdebski
> <mizdebsk(a)redhat.com&gt;
>  wrote:
>  
> > On 06/20/2018 02:30 PM, Petr Šabata wrote:
> >> Parallel installation of streams on a single system indeed
> >> isn't supported at this point and isn't planned anytime in the
> >> near future.  In general it's a more complicated problem than
> >> it might seem at first.
> >
> > Could you elaborate and explain what's so complicated about it?
> >
>  
>  The short answer is that there exists no generic solution for
>  parallel-installation. Many packages rely on well-known resources[1] and
>  cannot be parallel-installed at all. Other packages *may* support
>  parallel-installation but consumers must take special steps to enable
>  support for it. 
In general case, when packages conflict, I would agree. But as subject
says, this is about a special case - non-overlapping, non-conflicting
package sets, which can be installed in parallel without taking any
special steps, like java-1.8.0-openjdk and java-11-openjdk that were
brought up in the first message.

>  I don't have a good link right now, but folks at Red Hat did a
> number of
>  customer studies and determined that in real-world deployments,
>  parallel-installation was very rarely used. Generally, the OS was
>  established with a standard set of packages and then anything that needed
>  an alternate version was deployed as a VM or container. 
Some packages are designed to be parallel-installable for a reason -
because users expect and require that. OpenJDK is an example of such
package. Even different version-releases of the same component can be,
by design, installed in parallel.

>  So, when we started looking at ways to provide alternative software,
> we
>  determined that parallel-installation was a non-goal. Not needing to solve
>  an unsolvable problem meant that we could focus on the parts that really
>  matter: allowing people to select which single version of the software
>  meets their needs. 
So the conclusion is that inability to have parallel-installable streams
of the same module is not due to technical difficulties to implement
that feature, but rather a consequence of prioritizing different
requirements.

-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Stephen Gallagher]

On Thu, Jun 21, 2018 at 11:56 AM Mikolaj Izdebski <mizdebsk(a)redhat.com&gt;
wrote:

>  On 06/20/2018 04:15 PM, Stephen Gallagher wrote:
>  > On Wed, Jun 20, 2018 at 10:06 AM Mikolaj Izdebski <mizdebsk(a)redhat.com&gt;
>  > wrote:
>  >
>  >> On 06/20/2018 02:30 PM, Petr Šabata wrote:
>  >>> Parallel installation of streams on a single system indeed
>  >>> isn't supported at this point and isn't planned anytime in the
>  >>> near future.  In general it's a more complicated problem than
>  >>> it might seem at first.
>  >>
>  >> Could you elaborate and explain what's so complicated about it?
>  >>
>  >
>  > The short answer is that there exists no generic solution for
>  > parallel-installation. Many packages rely on well-known resources[1] and
>  > cannot be parallel-installed at all. Other packages *may* support
>  > parallel-installation but consumers must take special steps to enable
>  > support for it.
>
>  In general case, when packages conflict, I would agree. But as subject
>  says, this is about a special case - non-overlapping, non-conflicting
>  package sets, which can be installed in parallel without taking any
>  special steps, like java-1.8.0-openjdk and java-11-openjdk that were
>  brought up in the first message.
>
>  
Yes, but in order to support that in Modularity, we would need a
general-case solution that would work for all packages. Just because *some*
RPMs don't conflict between versions doesn't mean that a generic solution
can be built for all RPMs.


>  > I don't have a good link right now, but folks at Red Hat did
> a number of
>  > customer studies and determined that in real-world deployments,
>  > parallel-installation was very rarely used. Generally, the OS was
>  > established with a standard set of packages and then anything that needed
>  > an alternate version was deployed as a VM or container.
>
>  Some packages are designed to be parallel-installable for a reason -
>  because users expect and require that. OpenJDK is an example of such
>  package. Even different version-releases of the same component can be,
>  by design, installed in parallel.
>
>  
Sure, and that's why we recommend that in THIS case, you would probably
build a separate module (or use standard RPMs) instead.


>  > So, when we started looking at ways to provide alternative
> software, we
>  > determined that parallel-installation was a non-goal. Not needing to
>  solve
>  > an unsolvable problem meant that we could focus on the parts that really
>  > matter: allowing people to select which single version of the software
>  > meets their needs.
>
>  So the conclusion is that inability to have parallel-installable streams
>  of the same module is not due to technical difficulties to implement
>  that feature, but rather a consequence of prioritizing different
>  requirements.
>
>  
You drew a conclusion that is EXACTLY the opposite of what I said above. We
determined that parallel-installation was basically impossible to build in
a generic way. Once that was eliminated as a possible requirement, the rest
fell into place fairly neatly.

[Mikolaj Izdebski]

On 06/21/2018 06:26 PM, Stephen Gallagher wrote:
> >> So, when we started looking at ways to provide alternative
> software, we
> >> determined that parallel-installation was a non-goal. Not needing to
> > solve
> >> an unsolvable problem meant that we could focus on the parts that really
> >> matter: allowing people to select which single version of the software
> >> meets their needs.
> >
> > So the conclusion is that inability to have parallel-installable streams
> > of the same module is not due to technical difficulties to implement
> > that feature, but rather a consequence of prioritizing different
> > requirements.
> >
> >
>  You drew a conclusion that is EXACTLY the opposite of what I said above. We
>  determined that parallel-installation was basically impossible to build in
>  a generic way. Once that was eliminated as a possible requirement, the rest
>  fell into place fairly neatly. 
Allowing parallel installation of two distinct package sets, provided
that they don't conflict in any way - how is that impossible?  I get
that it's not a goal for modularity to support this, but I don't see any
technical reason not to allow it.


-- 
Mikolaj Izdebski
Senior Software Engineer, Red Hat
IRC: mizdebsk

[Petr Šabata]

On Thu, Jun 21, 2018 at 06:55:54PM +0200, Mikolaj Izdebski wrote:
>  On 06/21/2018 06:26 PM, Stephen Gallagher wrote:
>  >>> So, when we started looking at ways to provide alternative software, we
>  >>> determined that parallel-installation was a non-goal. Not needing to
>  >> solve
>  >>> an unsolvable problem meant that we could focus on the parts that really
>  >>> matter: allowing people to select which single version of the software
>  >>> meets their needs.
>  >>
>  >> So the conclusion is that inability to have parallel-installable streams
>  >> of the same module is not due to technical difficulties to implement
>  >> that feature, but rather a consequence of prioritizing different
>  >> requirements.
>  >>
>  >>
>  > You drew a conclusion that is EXACTLY the opposite of what I said above. We
>  > determined that parallel-installation was basically impossible to build in
>  > a generic way. Once that was eliminated as a possible requirement, the rest
>  > fell into place fairly neatly.
>  
>  Allowing parallel installation of two distinct package sets, provided
>  that they don't conflict in any way - how is that impossible?  I get
>  that it's not a goal for modularity to support this, but I don't see any
>  technical reason not to allow it. 
If those package sets are completely distinct, most of the
arguments Stephen provided indeed don't apply, however, you
still need to thoroughly check that your module and all of its
dependencies are installable in this scenario and you need to
be able to do that in the buildroot as well.  That includes
both modular and ursine buildroot.  Furthermore you need to
come up with a new tracking mechanism in the software database
as well as usable UI and APIs.  All this for only a small set
of modules that could easily delivery via some other method.

A generic solution appears to be impossible.  This kind of a
limited solution can be done but as I said earlier, it's not
currently on the agenda.

P

[Nicolas Mailhot]

Le jeudi 21 juin 2018 à 18:55 +0200, Mikolaj Izdebski a écrit :
>  
>  Allowing parallel installation of two distinct package sets, provided
>  that they don't conflict in any way - how is that impossible?  I get
>  that it's not a goal for modularity to support this, but I don't see
>  any technical reason not to allow it. 
It's the usual leaf vs trunk dilemma. Everyone wants to be treated as a
leaf (top of the ecosystem), but free software is deeply interlinked,
and you get complexity explosion as soon as you allow alternative links.

Regards,

-- 
Nicolas Mailhot