Renewing the Modularity objective

[Ben Cotton]

Now that Modularity is available for all Fedora variants, it's time to
address issues discovered and improve the experience for packagers and
users. The Modularity team identified a number of projects that will
improve the usefulness of Modularity and the experience of creating
modules for packagers. Thoe team is proposing a renewed objective to
the Fedora Council.

You can read the proposal at:
https://pagure.io/Fedora-Council/council-docs/pull-request/61#request_diff

The Council will vote on this in two weeks.

This is also posted to the Community Blog:
https://communityblog.fedoraproject.org/renewing-the-modularity-objective/

-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Michal Schorm]

I'd suggest that the Modularity team could preapre a list of example
use cases that will present the strenghts of the modularity.
I believe, there are currently many examples of packages that took a
path to become only modular and it always created more issues than it
solved.
Let's focus on a simple use cases first, which only modularity can
solve the best and leave the more complicated ones for later - after a
careful consideration if turning some regular packages into only
modules would really help both packager and user experience.

Keep in mind, there are still wide gaps in the modularity packager
experience, new bugs spawning every now and then.
In light of this persistent situation, I honor the new goal currently
set of focusing at those issues.

--

I'll start with my use case, which IMHO can be used as a great case
advocating for modularity.
I'm a maintainer of MariaDB, MySQL and some software around.
While the new major version of the database is being developed, I'd
love to pack it in Fedora, test it, offer it to the users and provide
feedback to the upstream, solving the uprising issues with them way
before the GA.
Because I want to keep a stable version in the base Fedora, I'm using
modules to provide both of them.
E.g. when MySQL 8 was being developed ( pre-GA releases ), I normally
maintained the MySQL 5.7 (latest stable major version) in the base
Fedora, while having MySQL 8 as a module.

When the MySQL 8 went GA, I had MySQL 5.7 in Fedora N and MySQL 8 in Fedora N+1.
However at the same time I also provided MySQL 5.7 as a module in both
Fedora N and N+1. Same for MySQL 8 module (in both Fedora N & N+1).
That way, the users weren't forced to upgrade right away (once the
Fedora N went EOL), but they got time to prepare and / or test it
first.
In a case, when user is hungry for the very latest version, he has the
MySQL 8 available before it went ot the base Fedora, and even before
the MySQL 8 went GA, so he can provide feedback to the upstream either
directly or through me (via BZ).
Since the upgrades between Fedora releases with modules when the
version in Fedora base changes are still not really thought through,
you (as a user) need the same module in the both Fedora N & N+1,
beacuse you can't upgrade Fedora version from MySQL 5.7 that was in
base to MySQL 5.7 module, since newly there is MySQL 8 is in the base
Fedora.

I believe no other applicable solution is as elegant for my use case,
as modularity has.
COPR repositories are hidden from the users (you first need to know
which repo you want to use before getting it's content), and the
builds from there are not pushed through the updtae system (BODHI) to
discover bugs early.
New package (named e.g. 'mysql-8' ) is also far from "elegant"
solution. Even further when I (the pkg mainatiner) plan to soon (once
GA is released) update to that version in the original package.

The same for MariaDB 10.3 -> 10.4; future 10.4 -> 10.5 ...

--

I strogly believe a list of use cases like this would make the
modularity much more clear to both mainatiners and users.

Stick to Unix philosophy ("Do One Thing and Do It Well") and don't
rush or even try to make modules from everything.

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--

On Wed, Sep 18, 2019 at 9:32 PM Ben Cotton <bcotton(a)redhat.com&gt; wrote:
>  Now that Modularity is available for all Fedora variants, it's time to
>  address issues discovered and improve the experience for packagers and
>  users. The Modularity team identified a number of projects that will
>  improve the usefulness of Modularity and the experience of creating
>  modules for packagers. Thoe team is proposing a renewed objective to
>  the Fedora Council.
>
>  You can read the proposal at:
>  https://pagure.io/Fedora-Council/council-docs/pull-request/61#request_diff
>
>  The Council will vote on this in two weeks.
>
>  This is also posted to the Community Blog:
>  https://communityblog.fedoraproject.org/renewing-the-modularity-objective/
>
>  --
>  Ben Cotton
>  He / Him / His
>  Fedora Program Manager
>  Red Hat
>  TZ=America/Indiana/Indianapolis
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Petr Pisar]

On 2019-09-19, Michal Schorm <mschorm(a)redhat.com&gt; wrote:
>  While the new major version of the database is being developed,
> I'd
>  love to pack it in Fedora, test it, offer it to the users and provide
>  feedback to the upstream, solving the uprising issues with them way
>  before the GA.
>  Because I want to keep a stable version in the base Fedora, I'm using
>  modules to provide both of them.
>  E.g. when MySQL 8 was being developed ( pre-GA releases ), I normally
>  maintained the MySQL 5.7 (latest stable major version) in the base
>  Fedora, while having MySQL 8 as a module.
>  
When MySQL 8 is being developed and being packages as module, do you
build the module for Rawhide only or for all Fedoras?

If you build it for all Fedoras, how do you deal with incompatible
changes during the MySQL 8 developement. I'm hitting on the Fedora
Updates Policy that forbids incompatible changes in stable Fedoras.

If you build it for Rawhide only, how do you ensure that the module is
not inheritted into a stable Fedora on branching. Because in case of
branching F31 relengs tried very hard to branch the module and rebuild
them. (Despite I told them not to do that with perl:5.26.)

I'm still missing an offical recognition that there can be modules under
development in stable Fedora. Otherwise we have no way of developing new
modules. Fedora tries very hard to align module lifecycle to Fedora
lifecycle. It does not work for me.

-- Petr

[Michal Schorm]

>  When MySQL 8 is being developed and being packages as module, do you
>  build the module for Rawhide only or for all Fedoras?
>
>  If you build it for all Fedoras, how do you deal with incompatible
>  changes during the MySQL 8 developement. I'm hitting on the Fedora
>  Updates Policy that forbids incompatible changes in stable Fedoras.
>
>  If you build it for Rawhide only, how do you ensure that the module is
>  not inheritted into a stable Fedora on branching. Because in case of
>  branching F31 relengs tried very hard to branch the module and rebuild
>  them. (Despite I told them not to do that with perl:5.26.) 
That's a great observation.
Ususally when the package (module in this case) is prone to breaking
bugs, I develop it in Rawhide and only later, (e.g. Beta, but it
depends from upstream to upstream) I extend the build to other
Fedoras.

>  I'm still missing an offical recognition that there can be
> modules under
>  development in stable Fedora. Otherwise we have no way of developing new
>  modules. Fedora tries very hard to align module lifecycle to Fedora
>  lifecycle. It does not work for me. 
That's surely an *absolute need* to have an option to mark a module as
"under developement" or something simmilar and have that anchored in
the guidelines, if we want to use this chance a modularity technically
offers.

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--

On Thu, Sep 19, 2019 at 2:33 PM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>
> On 2019-09-19, Michal Schorm <mschorm(a)redhat.com&gt; wrote:
> > While the new major version of the database is being developed, I'd
> > love to pack it in Fedora, test it, offer it to the users and provide
> > feedback to the upstream, solving the uprising issues with them way
> > before the GA.
> > Because I want to keep a stable version in the base Fedora, I'm using
> > modules to provide both of them.
> > E.g. when MySQL 8 was being developed ( pre-GA releases ), I normally
> > maintained the MySQL 5.7 (latest stable major version) in the base
> > Fedora, while having MySQL 8 as a module.
> >
>  When MySQL 8 is being developed and being packages as module, do you
>  build the module for Rawhide only or for all Fedoras?
>
>  If you build it for all Fedoras, how do you deal with incompatible
>  changes during the MySQL 8 developement. I'm hitting on the Fedora
>  Updates Policy that forbids incompatible changes in stable Fedoras.
>
>  If you build it for Rawhide only, how do you ensure that the module is
>  not inheritted into a stable Fedora on branching. Because in case of
>  branching F31 relengs tried very hard to branch the module and rebuild
>  them. (Despite I told them not to do that with perl:5.26.) 
>
>  I'm still missing an offical recognition that there can be
> modules under
>  development in stable Fedora. Otherwise we have no way of developing new
>  modules. Fedora tries very hard to align module lifecycle to Fedora
>  lifecycle. It does not work for me. 
>
> -- Petr
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Jun Aruga]

>  I'd suggest that the Modularity team could preapre a list of
> example
>  use cases that will present the strenghts of the modularity. 
I just share below my project for me to investigate use cases to
switch a modules stream with Fedora (and RHEL) container and Travis
CI.
It's not general module examples but only examples to switch a module.
But I believe that the way to show the example with actual command
lines with the Fedora container and Travis CI helps someone to
advocate the list of the example use cases.

https://github.com/junaruga/switch-module-stream

-- 
Jun | He - His - Him

[Kevin Fenzi]

On Wed, Sep 18, 2019 at 03:30:58PM -0400, Ben Cotton wrote:
>  Now that Modularity is available for all Fedora variants, it's
> time to
>  address issues discovered and improve the experience for packagers and
>  users. The Modularity team identified a number of projects that will
>  improve the usefulness of Modularity and the experience of creating
>  modules for packagers. Thoe team is proposing a renewed objective to
>  the Fedora Council.
>  
>  You can read the proposal at:
>  https://pagure.io/Fedora-Council/council-docs/pull-request/61#request_diff 
Is it really good to replace the old objective with the new one?
Shouldn't we archive off that one and call this one something else so
you can see what was done when?

Did you want comments here or on the PR? Or both?
(I see a number of folks have commented on the PR... which is awesome!)

I really like the goals/ideas here. We definitely need to improve
modules. 

kevin

Hello,

Could you please also resolve breaking upgrade paths for people who
never run `dnf module` command?

I'll try to explain what happened to me. As I wrote above I never run
`dnf module` until I was forced to run `dnf module reset` to fix my
issue. I'm running Fedora 30.

If I understand correctly what happened was that my zanata-client
package switched to module (or maybe dependencies, not sure) and then
the module was removed and replaced by packages.

It happened like this:

1) `dnf update` updated zanata to use module
2) module was removed
3) `dnf update` has conflict with module packages because it tried to
update to non module packages which were in conflict with the module
installed ones
4) Had to call my first module command and that was `dnf module reset`
5) Call `dnf distrosync` to remove the packages from a non-existing
module
6) Finally working `dnf update`

This is really not nice user experience to start with the modules from
a user perspective. Could you please prevent this issue for future.
There should be a way how to remove module without blocking upgrade
path, especially when the module was automatically dragged in by a
normal update.

What I missed in the above for simplification is that I used the `
--best --allowerasing` with a hope that I will be able to install the
new zanata then. By doing that I've effectively blocked zanata-client
from being installed on my laptop.

I also want to thank DNF team, they helped me to fix my NB otherwise I
wouldn't be able to make a new releases thanks to the missing zanata-
client.

Best regards,
Jirka

On Wed, 2019-09-18 at 15:30 -0400, Ben Cotton wrote:
>  Now that Modularity is available for all Fedora variants, it's
> time
>  to
>  address issues discovered and improve the experience for packagers
>  and
>  users. The Modularity team identified a number of projects that will
>  improve the usefulness of Modularity and the experience of creating
>  modules for packagers. Thoe team is proposing a renewed objective to
>  the Fedora Council.
>  
>  You can read the proposal at:
>  https://pagure.io/Fedora-Council/council-docs/pull-request/61#request_diff
>  
>  The Council will vote on this in two weeks.
>  
>  This is also posted to the Community Blog:
>  https://communityblog.fedoraproject.org/renewing-the-modularity-objective/
>  
>  -- 
>  Ben Cotton
>  He / Him / His
>  Fedora Program Manager
>  Red Hat
>  TZ=America/Indiana/Indianapolis
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]

[Matthew Miller]

On Sun, Sep 22, 2019 at 02:10:11PM -0700, Kevin Fenzi wrote:
>  Is it really good to replace the old objective with the new one?
>  Shouldn't we archive off that one and call this one something else so
>  you can see what was done when? 
Possibly! We've done the modularity objective in phases so far, with greater
and lesser attention to tying off some of the different phases. Do you have
a suggestion for an alternate name?

>  Did you want comments here or on the PR? Or both?
>  (I see a number of folks have commented on the PR... which is awesome!) 
Big picture comments here, details on the PR? 


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Stephen Gallagher]

On Thu, Sep 19, 2019 at 8:33 AM Petr Pisar <ppisar(a)redhat.com&gt; wrote:
>  I'm still missing an offical recognition that there can be modules under
>  development in stable Fedora. Otherwise we have no way of developing new
>  modules. Fedora tries very hard to align module lifecycle to Fedora
>  lifecycle. It does not work for me.
>  

I just opened https://pagure.io/modularity/issue/156 to make a proper
policy decision on this, including my recommendation:

***
* It is not permissible to make a prerelease stream the default stream
for a module in any release.(*)
* It is permissible to include a pre-release stream in Rawhide at any time.
* It is permissible to include a pre-release stream in a stable branch
of Fedora if the stream name clearly identifies it as unstable in some
manner. When the stream is ready to be treated as stable, a new stream
should be created with an [appropriate
name](https://docs.fedoraproject.org/en-US/modularity/making-modules/nami...
(such as a version number or other indicator of API/UX compatibility).

(*) Note that this is different from a "rolling" stream, in that the
expectation on rolling streams are that all of the updates are stable,
not that they are experimental and might change.
***