Current plan: Build python3, python3-libs etc. from python39 SRPM on F33+

[Miro Hrončok]

Hello Pythonistas.

(I've CC'ed the devel list for further exposure. But let's discuss this on 
python-devel list please to avoid noise.)


We  would like ro rename the "python3" component (SRPM) to "python39"
to  make 
maintaining various Python versions in various Fedora versions  easier.
The names of binary RPMs would be unchanged; you still do `dnf install python3`.

The change should only affect packagers of Python itself; not users or packagers 
of software written in Python.
The only user-visible or general-packager-visible change is a different 
component (SRPM) name.

Details follow.



Currently, we build the Python interpreters this way:

  - the "main" Python 3 interpreter component is called python3
    - it builds a couple of subpackages, like python3-libs or python3-devel
  - the "alternate" Python 3 versions components are called python3X (python37,

python39...)

For example, Fedora 31 has:

- python36
- python3  (the "main" Python, which is 3.7.x)
- python38
- python39

(python37 is retired in f31, but present in Fedora versions where python3 != 3.7)

This makes updating the "main" Python fairly complicated. To upgrade from 3.7 to

3.8 in f32, we had to:

  1) retire python38
  2) upgrade python3 from 3.7 to 3.8 (this makes python3 obsolete python38)
  3) re-introduce (unretire) python37

Such an update comes every few years:
  - https://fedoraproject.org/wiki/Changes/Python3.7 (f29)
  - https://fedoraproject.org/wiki/Changes/Python3.8 (f32)
  - https://fedoraproject.org/wiki/Changes/Python3.9 (f33)

What's even more complicated is backporting patches across Fedora releases. 
Assume there is an important upgrade/fix for Python 3.7 we wish to apply for all 
Fedoras including stable. That currently means we need to apply the fix in:

  - master branch of python37
  - f32 branch of python37
  - f31 branch of python3 (!)
  - f30 branch of python3

While cherry-picking or merging patches between branches of one git repo is 
quite straight-forward (the only major obstacle is the release and %chaneglog), 
doing this across multiple git repos (and spec filenames) is very tedious and 
error-prone (albeit possible).

With the 3.9 update approaching in Fedora 33, we would end up with the following 
scheme when updating Python 3.8:

  - master branch of python38
  - f32 branch of python3 (!)
  - f31 branch of python38 (!)
  - f30 branch of python38

Apart from technological obstacles, this causes nontrivial cognitive overhead. 
So far, I've been carrying that overhead myself, but others (who don't work on 
the Fedora Python packages daily) always need my assistance with this, because 
it's overly complicated and hard to understand without drawing pictures.



For this to be easier for us, while keeping the change minimal for others, we 
have decided to retire the python3 component and only continue maintaining the 
interpreters in "python3X" components. The names of "binary" packages
remain 
unchanged.

Before:

SRPM: python3
builds:
  - python3
  - python-unversioned-command
  - python3-{libs,devel,tkinter,idle,test,debug}
  - python3-debug{info,source}

After:

SRPM: python39
builds:
  - python3 (unchanged)
  - python-unversioned-command (unchanged)
  - python3-{libs,devel,tkinter,idle,test,debug} (unchanged)
  - python39-debug{info,source} (usually not installed by name)

To avoid further confusion, the "python3-libs" etc. packages will provide 
"python39-libs" as well.

We plan to do this together with the Python 3.9 upgrade. I'll update the change 
page after some feedback happens here.

The initial implementation is in:
   https://src.fedoraproject.org/rpms/python39/pull-request/32


Here are other approaches we have considered:


Why not rename the packages to python39-libs etc. (and only provide python3-libs)?

   This would be inconsistent with the way other Python 3 packages are called 
(e.g. python39-setuptools). We consider that confusing.

   The goal here is to make our work easier while introducing minimal changes to 
our users and contributors.


Why not rename all the Python packages to python39-setuptools etc.?

   This would create a mess upon distro boundary Python upgrades.
   All python39-foo packages would need to obsolete all python38-foo packages.
   Even if we add the obsoletes metadata to all the packages, it would prevent 
users from having the two parallel stacks of Python installed on the same OS. 
(We don't have such stacks in Fedora, but we intend to continue to make this 
possible in third party repos, downstreams of Fedora or other systems built on 
top or Fedora).

   See also the "minimal changes" point above.


Why not package the alternate stacks as modular streams and keep everything 
called just python3- (or even python-)?

   1) The Python stacks are naturally parallel-installable. This would not be 
possible if they were different streams of one module.
   2) Vital system tools (dnf, anaconda, packaging tools) use Python, so the 
"main" Python can't be in a module. (We could solve this by a separate 
"platform Python" just for these system tools, but that makes everything more 
complex and harder to work with -- the opposite of what this change is about.)

   See also the "minimal changes" point above.

Why are you breaking the world once again?

   Sorry if this breaks anything for you. Could you be as specific as possible? 
We anticipate no changes in Fedora packages (other than python3/python39).

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Neal Gompa]

On Tue, Mar 24, 2020 at 8:40 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  Hello Pythonistas.
>
>  (I've CC'ed the devel list for further exposure. But let's discuss this on
>  python-devel list please to avoid noise.)
>
>
>  We  would like ro rename the "python3" component (SRPM) to "python39"
> to  make
>  maintaining various Python versions in various Fedora versions  easier.
>  The names of binary RPMs would be unchanged; you still do `dnf install python3`.
>
>  The change should only affect packagers of Python itself; not users or packagers
>  of software written in Python.
>  The only user-visible or general-packager-visible change is a different
>  component (SRPM) name.
>
>  Details follow.
>
>
>
>  Currently, we build the Python interpreters this way:
>
>    - the "main" Python 3 interpreter component is called python3
>      - it builds a couple of subpackages, like python3-libs or python3-devel
>    - the "alternate" Python 3 versions components are called python3X
> (python37,
>  python39...)
>
>  For example, Fedora 31 has:
>
>  - python36
>  - python3  (the "main" Python, which is 3.7.x)
>  - python38
>  - python39
>
>  (python37 is retired in f31, but present in Fedora versions where python3 != 3.7)
>
>  This makes updating the "main" Python fairly complicated. To upgrade from 3.7
> to
>  3.8 in f32, we had to:
>
>    1) retire python38
>    2) upgrade python3 from 3.7 to 3.8 (this makes python3 obsolete python38)
>    3) re-introduce (unretire) python37
>
>  Such an update comes every few years:
>    - https://fedoraproject.org/wiki/Changes/Python3.7 (f29)
>    - https://fedoraproject.org/wiki/Changes/Python3.8 (f32)
>    - https://fedoraproject.org/wiki/Changes/Python3.9 (f33)
>
>  What's even more complicated is backporting patches across Fedora releases.
>  Assume there is an important upgrade/fix for Python 3.7 we wish to apply for all
>  Fedoras including stable. That currently means we need to apply the fix in:
>
>    - master branch of python37
>    - f32 branch of python37
>    - f31 branch of python3 (!)
>    - f30 branch of python3
>
>  While cherry-picking or merging patches between branches of one git repo is
>  quite straight-forward (the only major obstacle is the release and %chaneglog),
>  doing this across multiple git repos (and spec filenames) is very tedious and
>  error-prone (albeit possible).
>
>  With the 3.9 update approaching in Fedora 33, we would end up with the following
>  scheme when updating Python 3.8:
>
>    - master branch of python38
>    - f32 branch of python3 (!)
>    - f31 branch of python38 (!)
>    - f30 branch of python38
>
>  Apart from technological obstacles, this causes nontrivial cognitive overhead.
>  So far, I've been carrying that overhead myself, but others (who don't work on
>  the Fedora Python packages daily) always need my assistance with this, because
>  it's overly complicated and hard to understand without drawing pictures.
>
>
>
>  For this to be easier for us, while keeping the change minimal for others, we
>  have decided to retire the python3 component and only continue maintaining the
>  interpreters in "python3X" components. The names of "binary" packages
> remain
>  unchanged.
>
>  Before:
>
>  SRPM: python3
>  builds:
>    - python3
>    - python-unversioned-command
>    - python3-{libs,devel,tkinter,idle,test,debug}
>    - python3-debug{info,source}
>
>  After:
>
>  SRPM: python39
>  builds:
>    - python3 (unchanged)
>    - python-unversioned-command (unchanged)
>    - python3-{libs,devel,tkinter,idle,test,debug} (unchanged)
>    - python39-debug{info,source} (usually not installed by name)
>
>  To avoid further confusion, the "python3-libs" etc. packages will provide
>  "python39-libs" as well.
>
>  We plan to do this together with the Python 3.9 upgrade. I'll update the change
>  page after some feedback happens here.
>
>  The initial implementation is in:
>     https://src.fedoraproject.org/rpms/python39/pull-request/32
>
>
>  Here are other approaches we have considered:
>
>
>  Why not rename the packages to python39-libs etc. (and only provide python3-libs)?
>
>     This would be inconsistent with the way other Python 3 packages are called
>  (e.g. python39-setuptools). We consider that confusing.
>
>     The goal here is to make our work easier while introducing minimal changes to
>  our users and contributors.
>
>
>  Why not rename all the Python packages to python39-setuptools etc.?
>
>     This would create a mess upon distro boundary Python upgrades.
>     All python39-foo packages would need to obsolete all python38-foo packages.
>     Even if we add the obsoletes metadata to all the packages, it would prevent
>  users from having the two parallel stacks of Python installed on the same OS.
>  (We don't have such stacks in Fedora, but we intend to continue to make this
>  possible in third party repos, downstreams of Fedora or other systems built on
>  top or Fedora).
>
>     See also the "minimal changes" point above.
>
>
>  Why not package the alternate stacks as modular streams and keep everything
>  called just python3- (or even python-)?
>
>     1) The Python stacks are naturally parallel-installable. This would not be
>  possible if they were different streams of one module.
>     2) Vital system tools (dnf, anaconda, packaging tools) use Python, so the
>  "main" Python can't be in a module. (We could solve this by a separate
>  "platform Python" just for these system tools, but that makes everything more
>  complex and harder to work with -- the opposite of what this change is about.)
>
>     See also the "minimal changes" point above.
>
>  Why are you breaking the world once again?
>
>     Sorry if this breaks anything for you. Could you be as specific as possible?
>  We anticipate no changes in Fedora packages (other than python3/python39).
>  
As long as there's a flag for doing fully versioned not-flat packages,
I think I'd be good with this. That is: python3Y-* packages.


-- 
真実はいつも一つ！/ Always, there's only one truth!