F31 System-Wide Change proposal: F31 Mass Python 2 Package Removal

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal

== Summary ==

Python 2 is reaching end of life, and the current maintainers would like to
orphan it.

To prevent massive breakage, and to plan help with porting to Python 3,
we will systematically track and remove packages that still depend on Python 2.
The Fedora 30 change, [[Changes/Mass Python 2 Package Removal]],
continues in Fedora 31.

We allow packagers to more easily abandon Python 2 parts of their
packages (as an alternative to orphaning the whole package).

We also start systematically collecting info on what remaining packages need to
switch to Python 3

Finally, we'll use an expedited process to remove non-installable
Python 2 packages from the distro.

== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]]
* Name: [[User:Pviktori|Petr Viktorin]]
* Name: [[User:Cstratak|Charalampos Stratakis]]
* Name: [[User:Zbyszek|Zbigniew Jędrzejewski-Szmek]] (proposed)
* Name: [[User:Ignatenkobrain|Igor Gnatenko]]
* Name: [[User:Ngompa|Neal Gompa]]

* Email: [e-mail]

== Detailed Description ==

Over a thousand packages in Fedora still depend on Python 2, which
will reach End of Life (EOL) on 2020-01-01, shortly after the expected
Fedora 31 release date.

Without upstream help, we (the current maintainers) cannot support Python 2
effectively, especially with such a vast ecosystem.
There are also still too many packages to help porting them individually.

We would ideally like to orphan and retire Python 2, but to avoid
breaking the distro,
we choose a more controlled path to removing it:

* Python 2 subpackages can be abandoned easily, by a process similar
to orphaning
* Python 2 modules with no additional functionality will be removed
* Non-essential requirements will be removed
* We will collect information to help with "project management"
* Non-installable Python 2 packages will be removed from the distro

The Change describes several processes that, if approved, will be in
force for Fedora 31 (until Final Freeze), or until this Change is
reverted.
(The processes are quite wordy and lawyery. Sorry! Unfortunately, with
so many packages to handle, we need to formalize things.)

This Change is just a more controlled version of orphaning the Python 2 package.
For anyone who disagrees with the Change, there are several alternatives to
following it:

* Requesting an exception from the Python SIG (with reasons).
* Suggesting changes to the process.
* Taking over python2 now (with a better plan).
* Expressing interest in maintaining Python 2, or just ignoring the
Change. (When only these packages and their dependencies remain, the
current Python maintainers will orphan python2 and many fundamental
packages.)

=== Process for abandoning Python 2 subpackages ===

Many packages in Fedora contain Python 2 and Python 3 subpackages in
the same component (SRPM).
If a maintainer no longer wishes to maintain the Python 2 parts (e.g.
a <code>python2-*</code> subpackage),
it is currently easier to orphan/retire the entire package than to either:

* remove only the Python 2 parts (which might break dependent
packages, and so is not allowed), or
* split the Python 2 parts out into a separate component (which would
require formal review).

We make abandoning Python 2 parts easier by introducing the following
formal process.

Both FESCo and FPC need to agree to this process, which will be in
force for Fedora 31
(and may be extended to Fedrora 32).

Highlights from the process:

* The maintainer does not (have to) package the Python 2 parts, but
must give a opportunity for others to do so.
** Prior to the split, Python 2 parts must be conditionalized (mainly
to help others locate them).
** The package with the split-out Python 2 parts are granted several
review exceptions.
* If anything depends on the Python 2 parts, people are given a chance
to claim them before they're removed.
** There is a 3-week period of waiting for responses to e-mails. (The
normal time between oprhaning and retiring a package is 6 weeks.)

==== Removing Python 2 parts ====

# If the package has co-maintainers, discuss your intention with them.
# Add <code>%bcond_without python2</code> to your specfile, and
conditionalize all Python 2 parts using <code>%if %{with
python2}</code>. (If the package already is conditionalized in some
other way, you may keep the other way instead.) Commit the result to
the master branch.
# Check that your package builds and works without python2 (for
example, by passing <code>--without python2</code> to <code>fedpkg
mockbuild</code>).
# Using a scratch build, check that both the Python 2 and Python 3
parts still build and work with default settings.
# Invite people to claim the Python 2 parts. This process can be
stopped if/when someone claims them:
## Announce on the fedora-devel mailing list that you wish to stop
maintaining the Python 2 parts of your package, and invite others to
claim them.
## If any packages depend on the functionality you are removing, then:
### After every 7 days, e-mail fedora-devel again. CC maintainers of
all packages that depend on the parts you are removing (even
transitively). Include a breakdown of the affected packages and
maintainers.
### After 3 such e-mails (3 weeks since the first mail to
fedora-devel), consider the Python 2 parts unclaimed and go on with
the removal.
### If any *new* dependency on the functionality you are removing
appears, restart the counter. (This should not happen often: python2
is deprecated, so adding new dependencies on it is generally against
policy).
# If anyone claimed the Python 2 parts, wait two weeks or until they
are ready. See the following section for the claiming process.
# Remove the Python 2 parts from Fedora Rawhide (i.e. the master branch).
## (The implementation is up to you: remove the conditionalized parts
entirely, change the bcond to <code>%bcond_with python2</code>, or do
something else to that effect.)
# Scratch-build all dependent packages. If any fail to build from
source, or to install, file a bug in Bugzilla (or comment on an
existing bug).

(If and when the Python 2 parts are claimed and packaged in Rawhide,
you may backport the changes to released/branched Fedoras. The Bodhi
update for such backport (if any) MUST contain both packages (Python 3
and Python 2).)

==== Claiming Python 2 parts of a package ====

If you wish to maintain Python 2 parts of a package which were (or are
planned to be) removed through the above process:

# Start with the spec file of the combined package
# Remove everything that is Python 3-specific and/or would conflict
with sub-packages of the Python3-only package.
# Rename the package to start with "python2-" (instead of "python-",
if applicable). Transform the python2- subpackage to the main package
(if applicable).
# Go through the [[Package_Review_Process]], noting to the reviewers
that several exceptions apply (see below).

Since this is a package split, any points where the Python 2 parts of
the original subpackage did not meet the
[https://docs.fedoraproject.org/en-US/packaging-guidelines/ Packaging
Policy] or [[Packaging:ReviewGuidelines]] can be ignored for the new
package, except:
* The package MUST still be named according to the Package Naming Guidelines.
* The source package for the software targeted at Python2 MUST take a
<code>python2-</code> prefix. (This is stronger than the guidelines,
which only say "should".)
* Python2 binary packages MUST be named using a <code>python2-</code> prefix.

And, of course: the new package MAY depend on <code>python2</code>,
and/or any package that depends on <code>python2</code>, even if those
are deprecated.

==== Notes ====

It is OK for a maintainer to split a package and keep maintaining both
parts. The e-mail notifications can be skipped in this case. The
package review exceptions still apply.

It is OK for a maintainer to split a package, formally keep
maintaining both parts, and immediately
[[Orphaned_package_that_need_new_maintainers#Orphaning_Procedure|orphan]]
the Python 2 parts after the split. Again, the e-mail notifications
can be skipped in this case. We ask you to be nice and make your
intentions clear.

Note that leaf (not depended upon) packages only providing Python 2
modules are still being removed from Fedora.

=== Removing Modules ===

(Sub-)packages that only provide a python2 importable module, and are
not required for other packages, will be removed.

The details are specified in the Fedora 30 change,
[[Changes/Mass Python 2 Package Removal]]. This effort continues in Fedora 31.

=== Removing Requirements ===

Requirements (Requires and BuildRequires) on Python2-only packages that bring
little or no impact on features and quality of packages will be removed.
This covers things like:

* Tests of seldom-used features (for example, BuildRequiring a
framework only to test integration with that framework).
* Non-essential test-only functaionality (e.g. a test runner plugin
for distributed multi-process testing).
* Performance enhancements (e.g. compiled modules that are drop-in
replacements for pure-Python modules).
* Linters and code style checkers (these should be run in upstream CIs
but, arguably, are not appropriate in <code>%check</code> at all).
* Code and content generators (such as Cython, Sphinx, custom include
file generators), especially if upstream ships pre-generated
code/docs.

Where possible, we will try to switch to the Python 3 versions of these
dependencies rather than remove them.

=== Information on Remaining Packages ===

To help manage the removal, we ask maintainers of remaining Python2-only
packages to provide some or all of the following information:

* What is the reason for the Python2 dependency? (Is it software
written in Python, or does it just provide Python bindings, or use
Python in the build system or test runner?)
* What are the upstream/community plans/timelines regarding Python 3?
* What is the guidance for porting to Python 3? (Assuming that there
is someone who generally knows how to port to Python 3, but doesn't
know anything about the particular package, what are the next steps to
take?)

Generally, we will open Bugzillas asking for this information when needed,
and we expect it to be provided and kept up to date.
(Yes, mainaining a Python 2 package is a burden.)
The bug may contain other questions as well.

Every week (7 days), if the 3 questions are still unanswered, anyone
may add a comment asking
again for a response and linking to this process.
After 3 such comments, the package can be orphaned (through a releng ticket,
similar to the last step of the
[[Policy_for_nonresponsive_package_maintainers]]).
Weeks when the packager is on vacation (according to the
[https://apps.fedoraproject.org/calendar/list/vacation/ calendar])
don't count. (If we think the packager is deliberately stalling, we
file a FESCo ticket and ask to orphan the package immediately.)

If the package is updated to remove the Python 2 dependency, this
process immediately stops.

This is more drastic version of the
[[Policy_for_nonresponsive_package_maintainers]]:
* Comments that don't answer the questions are ignored.
* Instead of being taken over, the package is orphaned.
* There are less steps and waiting.

Both FESCo and FPC need to agree to this process, which will be in
force for Fedora 31
(and may be extended to Fedora 32).

(We will try to be nice and relax the process for packagers who just need more
time and resources, but unfortunately we ultimately need a "big stick"
to get things done.)

=== Removing non-installable packages from the distro ===

This process applies to packages that depend on Python 2.
A similar distro-wide policy is suggested in
https://pagure.io/fesco/issue/2109, but it has longer waiting times.
If the distro-wide policy is accepted, this process may be adjusted
(with FESCo approval) to better align with it.

Both FESCo and FPC need to agree to this process, which will be in
force for Fedora 31
(and may be extended to Fedora 32).

If a package Fails To Build Or Install due to a missing Python 2 dependncy
(for example becasue the maintainers ignored the announcements about
removing a Python 2 subpackage)
anyone can file a a bugzilla blocking the
[https://bugzilla.redhat.com/show_bug.cgi?id=PY2FTBI PY2FTBI tracker
bug].

After the bug reamins in the NEW state for a week, anyone can send a
reminder with NEEDINFO.
After 3 such reminders (and at least 3 weeks in the NEW state), the
package gets orphaned,
effectivelly short-circuiting the
[[Policy_for_nonresponsive_package_maintainers]].

Packages still failing to install due to a missing Python 2 dependncy
will be retired at Beta Freeze, assuming they have a Bugzilla bug open
for at least 2 weeks.
The package maintainer may pospone this retirement to the Final Freeze
by promising to fix it until then.

=== Note about Deprecation ===

As a reminder, Python 2 is deprecated since Fedora 30 (see
[[Packaging:Deprecating_Packages]]),
which means that no package depends on it may be added to Fedora (though
an exception might be granted by the packaging committee).
This applies for indirect/transitive dependencies as well, so Python 2 packages
generally don't need to be marked deprecated.

== Benefit to Fedora ==

Python 2 will not be orphaned/retired suddenly, but in a controlled manner.

== Scope ==
* Proposal owners:
** Track a list of packages to remove
** Track a graph of dependencies to break
** Retire python2-only packages
** Remove python2 subpackages from dual-python packages
** Break python2-only dependencies
** File bugs to request and track information

* Other developers:
** Ask for exceptions if needed (with reasons)
** Help with the steps above (not needed, but helpful)
** Provide the requested information
** Optionaly remove Python 2 parts form their packages
** Optionaly mark remaining python2 packages deprecated

* [[FPC]]: Formally agree with the processes above.

* Release engineering: [https://pagure.io/releng/issue/8221 #8221] (a
check of an impact with Release Engineering is needed)
** Releng may be asked to orphan some packages, according to the
process above. Miro Hrončok will ask for permissions to orphan
packages to not overwhelm releng.

* Policies and guidelines: handled in the previous Change

* Trademark approval: not needed for this Change

== Upgrade/compatibility impact ==
Packages removed from Fedora repos will remain on the installed OS
until explicitly removed by the user or obsoleted by the packagers.
We'll only obsolete packages that would break upgrade (from
fedora-obsolete-packages).

== How To Test ==

Any program written in Python and any program that has Python plugins
could potentially be influenced by this change. Testing is different
for software that is still using Python 2 and for software that has
been switched to Python 3.

For any python2 programs, make sure those programs are still
functional after the package removal. Since various packages that are
no longer built will not be removed from an existing system, just
upgrading and checking packages is not enough. Either a new
installation should be used, or after a branching point, any packages
which haven't been rebuilt for F31 (any packages with .fc30 or lower
release suffix) should be removed from the system before testing.

For the python3 programs, install all upgrades and check if the software works.

Upgrade failures because of missing dependencies should be treated as
bugs. Any removed python2 subpackages which break upgrades need to be
added to Obsoletes in fedora-obsolete-packages.

== User Experience ==
There will be less Python 2 RPMs in Fedora repos. Users are encouraged
to switch to Python 3 and/or use Python 2 virtual environments and pip
for development.

== Dependencies ==
Ideally, all programs that use python2 would be switched to use
python3. Although we don't expect everything to be switched over, as
much as possible should be, so that the set of remaining python2
subpackages is as small as possible.

== Contingency Plan ==
* Contingency mechanism:
** If for some reason not everything is removed, nothing happens, it
can be removed later. If for some reason something breaks, some
packages can be unretired or restored.
** If someone steps up to maintain Python 2 (including the full
ecosystem of packages now in Fedora), they can decide to discontinue
removing packages, revert the Change, or come up with another plan.
(Note that in this case, current maintainers will most likely orphan
many fundamental python2 packages.)

* Contingency deadline: Final freeze
* Blocks release? No
* Blocks product? No

== Documentation ==
This page is the main documentation.

Also see: https://pythonclock.org