F39 proposal: Python 3.12 (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Python3.12

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Update the Python stack in Fedora from Python 3.11 to Python 3.12, the
newest major release of the Python programming language.

== Owner ==
* Name: [[User:Thrnciar|Tomáš Hrnčiar]]
* Name: [[User:Churchyard|Miro Hrončok]]
* Email: python-maint(a)redhat.com


== Detailed Description ==
We would like to upgrade Python to 3.12 in Fedora 39 thus we are
proposing this plan early.

See the upstream notes at
[https://peps.python.org/pep-0693/#features-for-3-12 Features for
3.12] and [https://docs.python.org/3.12/whatsnew/3.12.html What's new
in 3.12].

=== Important dates and plan ===

* 2022-05-08: Python 3.12 development begins
* 2022-10-24: Python 3.12.0 alpha 1
** Package it as {{package|python3.12}} for testing purposes
** Start the bootstrap procedure in Copr
** Do a mass rebuild against every future release in Copr
* 2022-11-14: Python 3.12.0 alpha 2
* 2022-12-05: Python 3.12.0 alpha 3
* 2023-01-09: Python 3.12.0 alpha 4
* 2023-02-06: Python 3.12.0 alpha 5
* 2023-02-07: Branch Fedora 38, Rawhide becomes future Fedora 39
** The earliest point when we can start rebuilding in Koji side-tag
* 2023-03-06: Python 3.12.0 alpha 6
* 2023-04-03: Python 3.12.0 alpha 7
* 2023-05-08: Python 3.12.0 beta 1
** No new features beyond this point
* 2023-05-29: Python 3.12.0 beta 2
** The ideal point when we can start rebuilding in Koji
* 2023-06-05: Expected side tag-merge (optimistic)
* 2023-06-19: Python 3.12.0 beta 3
* 2023-06-26: Expected side tag-merge (realistic)
* 2023-07-10: Python 3.12.0 beta 4
* 2023-07-17: Expected side tag-merge (pessimistic)
* 2023-07-19: Fedora 39 Mass Rebuild
** The mass rebuild happens with the fourth beta. We might need to
rebuild Python packages later in exceptional case.
** If the Koji side-tag is not merged yet at this point, we defer the
change to Fedora 38.
* 2023-07-31: Python 3.12.0 candidate 1
** This serves as "final" for our purposes.
* 2023-08-08: Branch Fedora 39, Rawhide becomes future Fedora 40
* 2023-08-08: Fedora 39 Change Checkpoint: Completion deadline (testable)
* 2023-08-22: Fedora Beta Freeze
** If rebuild with 3.12.0rc1 is needed, we should strive to do it
before the freeze - there is a window of 3 weeks.
* 2023-09-04: Python 3.12.0 candidate 2
* 2023-09-12: Fedora 39 Beta Release (Preferred Target)
** Beta will likely be released with 3.12.0rc2.
* 2023-09-19: Fedora 39 Beta Target date #1
* 2023-10-02: Python 3.12.0 final
* 2023-10-03: Fedora 39 Final Freeze
** We'll update to 3.12.0 final using a freeze exception.
* 2023-10-17: Fedora 39 Preferred Final Target date
* 2023-10-24: Fedora 39 Final Target date #1


(From [https://peps.python.org/pep-0693/#release-schedule Python 3.12
Release Schedule] and
[https://fedorapeople.org/groups/schedule/f-39/f-39-key-tasks.html
Fedora 39 Release Schedule].)

The schedule might appear somewhat tight for Fedora 39, but Python's
[https://lists.fedoraproject.org/archives/list/[e-mail].o...
annual release cycle was adapted for Fedora] and this worked fine
since Python 3.9 and Fedora 33. It is now common that Python is
upgraded on a similar schedule in every odd-numbered Fedora release.

Note that upstream's "release candidates" are frozen except for
blocker bugs. Since we can and will backport blocker fixes between
Fedora and upstream, we essentially treat the Release Candidate as the
final release.

=== Notes from the previous upgrade ===

There are notes from the previous upgrade available, so this upgrade
may go smoother: [[SIGs/Python/UpgradingPython]]

== Benefit to Fedora ==

Fedora aims to showcase the latest in free and open-source software -
we should have the most recent release of Python 3. Packages in Fedora
can use the new features from 3.12.

There's also a benefit to the larger Python ecosystem: by building
Fedora's packages against 3.12 while it's still in development, we can
catch critical bugs before the final 3.12.0 release.

== Scope ==

We will coordinate the work in a side tag and merge when ready.

* Proposal owners:
*# Introduce {{package|python3.12}} for all Fedoras
*# Prepare stuff in Copr as explained in description.
*# Update {{package|python-rpm-macros}} so {{package|python3.12}}
builds {{package|python3}}
*# Build {{package|python3.12}} as the main Python
*# Mass rebuild all the packages that runtime require `python(abi) =
3.11` and/or `libpython3.11.so.1.0` (~3900 known packages in October
2022)
*# Build {{package|python3.12}} as a non-main Python

* Other developers: Maintainers of packages that fail to rebuild
during the rebuilds will be asked, using e-mail and bugzilla, to fix
or remove their packages from the distribution. If any issues appear,
they should be solvable either by communicating with the respective
upstreams first and/or applying downstream patches. Also, the package
maintainers should have a look at:
[https://docs.python.org/3.12/whatsnew/3.12.html#porting-to-python-3-12
Porting to Python 3.12]. The python-maint team will be available to
help with fixing issues.

* Release engineering: [TBD]
* Policies and guidelines: nope
* Trademark approval: nope


== Upgrade/compatibility impact ==
All the packages that depend on Python 3 must be rebuilt. User written
Python 3 scripts/applications may require a small amount of porting,
but mostly Python 3.11 is forward compatible with Python 3.12.

=== The Python standard library distutils module will be removed ===
For many years distutils module was providing support for building and
installing additional modules into a Python installation.
Since Python 3.10 distutils package is deprecated, and will be removed
in Python 3.12. Its functionality for specifying package builds has
already been completely replaced by third-party packages setuptools
and packaging, and most other commonly used APIs are available
elsewhere in the standard library (such as platform, shutil,
subprocess or sysconfig).

Affected packages will be failing with `ModuleNotFoundError: No module
named 'distutils'`.

The python3-setutpools package provides a distutils module, so
sometimes "simply" adding BuildRequires: python3-setuptools might
workaround the problem. Unfortunately, it is not 100 % compatible with
the removed standard library one distutils:
https://github.com/pypa/setuptools/issues/3532

Fedora packagers can check if their packages build without distutils
by removing it form Python 3.11:

# `fedpkg clone <package name> && cd <package name>`
# `mock -r fedora-rawhide-x86_64 init`
# `mock -r fedora-rawhide-x86_64 install python3-devel`
# `sudo rm -rf /var/lib/mock/fedora-rawhide-x86_64/root/usr/lib64/python3.11/distutils/`
# `fedpkg mockbuild -N`

Later, when [https://copr.fedorainfracloud.org/coprs/g/python/python3.12/
Python 3.12 COPR] is available, you can use it for testing.

See
https://lists.fedoraproject.org/archives/list/[e-mail]...
for known Fedora packages that'll need changes.

== How To Test ==
Interested testers do not need special hardware. If you have a
favourite Python 3 script, module, or application, please test it with
Python 3.12 and verify that it still works as you would expect. If the
application you are testing does not require any other modules, you
can test it using {{package|python3.12}} even before this change is
implemented, in Fedora 36, 37 or 38.

In case your application requires other modules, or if you are testing
an rpm package, it is necessary to install the 3.12 version of the
python3 rpm. Right now that rpm is available in copr, along with all
other python packages that build successfully with python 3.12. See
https://copr.fedorainfracloud.org/coprs/g/python/python3.12/ for
detailed instructions on how to enable Python 3.12 copr for mock.

Once the change is in place, test if your favourite Python apps are
working as they were before. File bugs if they don't.

== User Experience ==
Regular distro users shouldn't notice any change in system behaviour
other than the Python 3 interpreter will be in version 3.12.

== Dependencies ==
4000+ packages depend on Python 3 and ~3900 packages need rebuilding
when Python is upgraded. See scope section.

== Contingency Plan ==
* Contingency mechanism: Do not merge the side tag with rawhide. If
the side tag has been merged and issues arise, that will justify a
downgrade, then use an epoch tag to revert to 3.11 version (never
needed before) * Contingency deadline: TBD
* Blocks release? Yes, we'd like to block Fedora 39 release on at
least 3.12.0rc1
* Blocks product? See above

== Documentation ==
[https://peps.python.org/pep-0693/ Python 3.12 Release Schedule]

[https://peps.python.org/pep-0693/#features-for-3-12 Features for 3.12]

[https://docs.python.org/3.12/whatsnew/3.12.html What's new in 3.12]

[https://docs.python.org/3.12/whatsnew/3.12.html#porting-to-python-3-12
Porting to Python 3.12]

== Release Notes ==
* Migrating user installed packages -
https://pagure.io/fedora-docs/release-notes/issue/503


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Maxwell G]

On Tue Oct 25, 2022 at 15:46 -0400, Ben Cotton wrote:
>  * 2023-07-17: Expected side tag-merge (pessimistic)
>  * 2023-07-19: Fedora 39 Mass Rebuild
>  ** The mass rebuild happens with the fourth beta. We might need to
>  rebuild Python packages later in exceptional case.
>  ** If the Koji side-tag is not merged yet at this point, we defer the
>  change to Fedora 38. 
Should this be Fedora 40?

--
Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Miro Hrončok]

On 25. 10. 22 23:10, Maxwell G via devel wrote:
>  On Tue Oct 25, 2022 at 15:46 -0400, Ben Cotton wrote:
> > * 2023-07-17: Expected side tag-merge (pessimistic)
> > * 2023-07-19: Fedora 39 Mass Rebuild
> > ** The mass rebuild happens with the fourth beta. We might need to
> > rebuild Python packages later in exceptional case.
> > ** If the Koji side-tag is not merged yet at this point, we defer the
> > change to Fedora 38.
>  
>  Should this be Fedora 40? 
Yes. Amended.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Michael J Gruber]

Did this work fine for python 3.11 in Fedora 37? Asking for a friend ...

[Miro Hrončok]

On 26. 10. 22 18:01, Michael J Gruber wrote:
>  Did this work fine for python 3.11 in Fedora 37? Asking for a friend
> ... 
Reasonably fine.

1 package still requires Python 3.11:
https://bugzilla.redhat.com/show_bug.cgi?id=2045255

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok