Splitting alternative Python packages into subpackages, e.g. python3.11{,libs,devel,...}

[Tomáš Orsava]

Hi Python-devel,
we are considering splitting the alternative Python versions from a 
single-package format (e.g. python3.11) to multiple subpackages (e.g. 
python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with 
the main `python3` package: it requires less disk space to install and 
speeds up download times, because you can chose which bits are important 
to you. For example, if you decide you don't need python3-tkinter, you 
save yourself ~18 dependent packages leading to a total savings of 
~20MBs, while skipping python3-test saves you further ~10MBs.

What do you think?

The push came from [BZ#2063227] where the reporters would welcome to 
have a smaller python3.11 package for containers and VMs for local 
testing, CI purposes and more.
This would be a larger amount of work, so our initial reaction was 
hesitant. We'll have to change the already complicated spec file %bcond 
logic, and adjust the ecosystem to work with the new subpackages. For 
example tox would need to Recommend `python3.11-devel`, as `python3.11` 
would bring in only the bare interpreter. And of course a thorough 
integration testing would be in order.

However, we already do separate subpackages for alternative stacks in 
Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we 
consider it good to have fewer differences between Fedora and EL. This 
helps to test things earlier, and there are fewer surprises in user 
experience. So perhaps the effort in doing this would be well spent.

To cut down on the amount of work, we're considering changing only the 
`python3.11` package (and any future newer versions) right now. If later 
we consider it worth it, we could switch the older alternative 
interpreters as well, or we might let them die out as they are.

We're currently in the brainstorming stage, so you're feedback is welcome.

[BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227

All the best,
Tomáš

[Miro Hrončok]

On 16. 03. 22 17:12, Tomáš Orsava wrote:
> Hi Python-devel,
> we are considering splitting the alternative Python versions from a 
> single-package format (e.g. python3.11) to multiple subpackages (e.g. 
> python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with the 
> main `python3` package: it requires less disk space to install and speeds up 
> download times, because you can chose which bits are important to you. For 
> example, if you decide you don't need python3-tkinter, you save yourself ~18 
> dependent packages leading to a total savings of ~20MBs, while skipping 
> python3-test saves you further ~10MBs.
>
> What do you think?
>
> The push came from [BZ#2063227] where the reporters would welcome to have a 
> smaller python3.11 package for containers and VMs for local testing, CI 
> purposes and more.
> This would be a larger amount of work, so our initial reaction was hesitant. 
> We'll have to change the already complicated spec file %bcond logic, and adjust 
> the ecosystem to work with the new subpackages. For example tox would need to 
> Recommend `python3.11-devel`, as `python3.11` would bring in only the bare 
> interpreter. And of course a thorough integration testing would be in order.
>
> However, we already do separate subpackages for alternative stacks in 
> Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we consider 
> it good to have fewer differences between Fedora and EL. This helps to test 
> things earlier, and there are fewer surprises in user experience. So perhaps 
> the effort in doing this would be well spent.
>
> To cut down on the amount of work, we're considering changing only the 
> `python3.11` package (and any future newer versions) right now. If later we 
> consider it worth it, we could switch the older alternative interpreters as 
> well, or we might let them die out as they are.
>
> We're currently in the brainstorming stage, so you're feedback is welcome.
>
> [BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227

Note that I was originally against this because I though it would make things 
more complex/complicated and the "flat" Python packages are simple.

However, we build the "flat" package vs. the "split" package based on a %bcond 
-- the spec is needlessly complex/complicated already and this could make it 
much simpler (so it would no longer resemble a nuclear power plant).

My only remaining fear was that it will be a huge amount of work across all our 
interpreters, but the idea to only do it in 3.11+ for now is intriguing. (In 
fact, I'd do it in 3.10+ so when we update main python to 3.10 in Fedora 37, 
Python 3.10 remains split.)

Hence, I am in favor of this idea. Let's see how much cleanup this can bring to 
the spec file and how much integration work would be needed (we use reverse 
weak deps in tox, so it should be painless).

Thanks, Tomáš, for writing it down.

[Neal Gompa]

On Fri, Mar 18, 2022 at 4:28 PM Miro Hrončok  wrote:
> On 16. 03. 22 17:12, Tomáš Orsava wrote:
> > Hi Python-devel,
> > we are considering splitting the alternative Python versions from a
> > single-package format (e.g. python3.11) to multiple subpackages (e.g.
> > python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with the
> > main `python3` package: it requires less disk space to install and speeds up
> > download times, because you can chose which bits are important to you. For
> > example, if you decide you don't need python3-tkinter, you save yourself ~18
> > dependent packages leading to a total savings of ~20MBs, while skipping
> > python3-test saves you further ~10MBs.
> >
> > What do you think?
> >
> > The push came from [BZ#2063227] where the reporters would welcome to have a
> > smaller python3.11 package for containers and VMs for local testing, CI
> > purposes and more.
> > This would be a larger amount of work, so our initial reaction was hesitant.
> > We'll have to change the already complicated spec file %bcond logic, and adjust
> > the ecosystem to work with the new subpackages. For example tox would need to
> > Recommend `python3.11-devel`, as `python3.11` would bring in only the bare
> > interpreter. And of course a thorough integration testing would be in order.
> >
> > However, we already do separate subpackages for alternative stacks in
> > Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we consider
> > it good to have fewer differences between Fedora and EL. This helps to test
> > things earlier, and there are fewer surprises in user experience. So perhaps
> > the effort in doing this would be well spent.
> >
> > To cut down on the amount of work, we're considering changing only the
> > `python3.11` package (and any future newer versions) right now. If later we
> > consider it worth it, we could switch the older alternative interpreters as
> > well, or we might let them die out as they are.
> >
> > We're currently in the brainstorming stage, so you're feedback is welcome.
> >
> > [BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227
>
> Note that I was originally against this because I though it would make things
> more complex/complicated and the "flat" Python packages are simple.
>
> However, we build the "flat" package vs. the "split" package based on a %bcond
> -- the spec is needlessly complex/complicated already and this could make it
> much simpler (so it would no longer resemble a nuclear power plant).
>
> My only remaining fear was that it will be a huge amount of work across all our
> interpreters, but the idea to only do it in 3.11+ for now is intriguing. (In
> fact, I'd do it in 3.10+ so when we update main python to 3.10 in Fedora 37,
> Python 3.10 remains split.)
>
> Hence, I am in favor of this idea. Let's see how much cleanup this can bring to
> the spec file and how much integration work would be needed (we use reverse
> weak deps in tox, so it should be painless).
>
> Thanks, Tomáš, for writing it down.

I'm in favor of it myself as well. I brought it up a while back on
this list last year even:
https://lists.fedoraproject.org/archives/list/[e-mail]...

So I'd love to see this happen.

[Miro Hrončok]

On 18. 03. 22 21:34, Neal Gompa wrote:
> On Fri, Mar 18, 2022 at 4:28 PM Miro Hrončok  wrote:
> > On 16. 03. 22 17:12, Tomáš Orsava wrote:
> > > Hi Python-devel,
> > > we are considering splitting the alternative Python versions from a
> > > single-package format (e.g. python3.11) to multiple subpackages (e.g.
> > > python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with the
> > > main `python3` package: it requires less disk space to install and speeds up
> > > download times, because you can chose which bits are important to you. For
> > > example, if you decide you don't need python3-tkinter, you save yourself ~18
> > > dependent packages leading to a total savings of ~20MBs, while skipping
> > > python3-test saves you further ~10MBs.
> > >
> > > What do you think?
> > >
> > > The push came from [BZ#2063227] where the reporters would welcome to have a
> > > smaller python3.11 package for containers and VMs for local testing, CI
> > > purposes and more.
> > > This would be a larger amount of work, so our initial reaction was hesitant.
> > > We'll have to change the already complicated spec file %bcond logic, and adjust
> > > the ecosystem to work with the new subpackages. For example tox would need to
> > > Recommend `python3.11-devel`, as `python3.11` would bring in only the bare
> > > interpreter. And of course a thorough integration testing would be in order.
> > >
> > > However, we already do separate subpackages for alternative stacks in
> > > Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we consider
> > > it good to have fewer differences between Fedora and EL. This helps to test
> > > things earlier, and there are fewer surprises in user experience. So perhaps
> > > the effort in doing this would be well spent.
> > >
> > > To cut down on the amount of work, we're considering changing only the
> > > `python3.11` package (and any future newer versions) right now. If later we
> > > consider it worth it, we could switch the older alternative interpreters as
> > > well, or we might let them die out as they are.
> > >
> > > We're currently in the brainstorming stage, so you're feedback is welcome.
> > >
> > > [BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227
> >
> > Note that I was originally against this because I though it would make things
> > more complex/complicated and the "flat" Python packages are simple.
> >
> > However, we build the "flat" package vs. the "split" package based on a %bcond
> > -- the spec is needlessly complex/complicated already and this could make it
> > much simpler (so it would no longer resemble a nuclear power plant).
> >
> > My only remaining fear was that it will be a huge amount of work across all our
> > interpreters, but the idea to only do it in 3.11+ for now is intriguing. (In
> > fact, I'd do it in 3.10+ so when we update main python to 3.10 in Fedora 37,
> > Python 3.10 remains split.)
> >
> > Hence, I am in favor of this idea. Let's see how much cleanup this can bring to
> > the spec file and how much integration work would be needed (we use reverse
> > weak deps in tox, so it should be painless).
> >
> > Thanks, Tomáš, for writing it down.
>
> I'm in favor of it myself as well. I brought it up a while back on
> this list last year even:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> So I'd love to see this happen.

I am quite convinced about the subpackages split, but I haven't thought about 
(not) providing python(abi). I don't want packages that fail to rebuild with 
new Python version to depend on the older one.

[Miro Hrončok]

On 16. 03. 22 17:12, Tomáš Orsava wrote:
> Hi Python-devel,
> we are considering splitting the alternative Python versions from a 
> single-package format (e.g. python3.11) to multiple subpackages (e.g. 
> python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with the 
> main `python3` package: it requires less disk space to install and speeds up 
> download times, because you can chose which bits are important to you. For 
> example, if you decide you don't need python3-tkinter, you save yourself ~18 
> dependent packages leading to a total savings of ~20MBs, while skipping 
> python3-test saves you further ~10MBs.
>
> What do you think?
>
> The push came from [BZ#2063227] where the reporters would welcome to have a 
> smaller python3.11 package for containers and VMs for local testing, CI 
> purposes and more.
> This would be a larger amount of work, so our initial reaction was hesitant. 
> We'll have to change the already complicated spec file %bcond logic, and adjust 
> the ecosystem to work with the new subpackages. For example tox would need to 
> Recommend `python3.11-devel`, as `python3.11` would bring in only the bare 
> interpreter. And of course a thorough integration testing would be in order.
>
> However, we already do separate subpackages for alternative stacks in 
> Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we consider 
> it good to have fewer differences between Fedora and EL. This helps to test 
> things earlier, and there are fewer surprises in user experience. So perhaps 
> the effort in doing this would be well spent.
>
> To cut down on the amount of work, we're considering changing only the 
> `python3.11` package (and any future newer versions) right now. If later we 
> consider it worth it, we could switch the older alternative interpreters as 
> well, or we might let them die out as they are.
>
> We're currently in the brainstorming stage, so you're feedback is welcome.
>
> [BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227

This is now in rawhide and updates testing, for Python 3.11.

https://bodhi.fedoraproject.org/updates/FEDORA-2022-69ddddac59
https://bodhi.fedoraproject.org/updates/FEDORA-2022-a6cb44ffeb
https://bodhi.fedoraproject.org/updates/FEDORA-2022-77db03acc5
https://bodhi.fedoraproject.org/updates/FEDORA-2022-be990fe88f

If all goes well, we plan to make Python 3.10 the same in Fedora 37 when the 
main Python will be 3.11.

[Victor Stinner]

That's great! It makes Python 3.11 usable in more cases.

Victor

On Tue, May 3, 2022 at 7:19 PM Miro Hrončok  wrote:
> On 16. 03. 22 17:12, Tomáš Orsava wrote:
> > Hi Python-devel,
> > we are considering splitting the alternative Python versions from a
> > single-package format (e.g. python3.11) to multiple subpackages (e.g.
> > python3.11{,-libs,-devel,-tkinter,-test,-idle}). We do this already with the
> > main `python3` package: it requires less disk space to install and speeds up
> > download times, because you can chose which bits are important to you. For
> > example, if you decide you don't need python3-tkinter, you save yourself ~18
> > dependent packages leading to a total savings of ~20MBs, while skipping
> > python3-test saves you further ~10MBs.
> >
> > What do you think?
> >
> > The push came from [BZ#2063227] where the reporters would welcome to have a
> > smaller python3.11 package for containers and VMs for local testing, CI
> > purposes and more.
> > This would be a larger amount of work, so our initial reaction was hesitant.
> > We'll have to change the already complicated spec file %bcond logic, and adjust
> > the ecosystem to work with the new subpackages. For example tox would need to
> > Recommend `python3.11-devel`, as `python3.11` would bring in only the bare
> > interpreter. And of course a thorough integration testing would be in order.
> >
> > However, we already do separate subpackages for alternative stacks in
> > Enterprise Linux (CentOS /Stream, RHEL, EPEL) and as a general rule we consider
> > it good to have fewer differences between Fedora and EL. This helps to test
> > things earlier, and there are fewer surprises in user experience. So perhaps
> > the effort in doing this would be well spent.
> >
> > To cut down on the amount of work, we're considering changing only the
> > `python3.11` package (and any future newer versions) right now. If later we
> > consider it worth it, we could switch the older alternative interpreters as
> > well, or we might let them die out as they are.
> >
> > We're currently in the brainstorming stage, so you're feedback is welcome.
> >
> > [BZ#2063227] https://bugzilla.redhat.com/show_bug.cgi?id=2063227
>
> This is now in rawhide and updates testing, for Python 3.11.
>
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-69ddddac59
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-a6cb44ffeb
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-77db03acc5
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-be990fe88f
>
> If all goes well, we plan to make Python 3.10 the same in Fedora 37 when the
> main Python will be 3.11.
>
> --
> Miro Hrončok
> --
> Phone: [phone]
> IRC: mhroncok
> _______________________________________________
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure