Fedora 30 System-Wide Change proposal: Mass Python 2 Package Removal

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal

== Summary ==
(Sub-)packages only providing python2 importable modules without
additional functionality will be removed from Fedora unless some other
package(s) depends on them.

Python 2 will be deprecated in Fedora. Packagers can mark any other
Python 2 packages as deprecated as well.

== Owner ==
* Miro Hrončok (Churchyard)
* Petr Viktorin (Pviktori)
* Charalampos Stratakis (Cstratak)
* Zbigniew Jędrzejewski-Szmek (Zbyszek)
* Igor Gantenko (Ignatenkobrain)
* Neal Gompa (Ngompa)

== Detailed Description ==
Python 2 reaches End of Life on 2020-01-01. Its current maintainers
would like to orphan it before that date, and so far no one else has
stepped up to maintain Python 2 (with the full ecosystem) past 2020.
Since thousands of packages still depend on python2, we need a more
careful approach than normal orphaning.
Some of those packages are abandoned and/or the Python 2 version is
unnecessary. Others are useful and just need more time to port to
Python 3.
Hence we set up criteria for python2 packages that can remain in the
distribution and we remove everything else. This should allow us to
keep python2 for limited use, not break everything, but should also
send a strong message that it is no longer a first class citizen, and
filter packages we need to focus Python 3 porting efforts on.

(Sub-)packages that only provide a python2 importable module, and are
not required for other packages, will be removed.

Examples of situations where a (sub-)package does not provide only the
importable module:

* A package also provides an application, mostly likely in /usr/bin,
/usr/libexec...
** (Note that according to current guidelines, if the Python 3 version
of a package provides the same functionality, the Python 3 package
should provide the application.)
** (In certain situations the provided application is only useful to
boostrap or manage projects using the module. Such applications don't
count.)
* A package provides a plugin for another application, most likely
trough a setuptools entrypoint interface or some custom location on
disk.

Our process will be:

* File bugs for all packages that look like they only provide a Python
2 importable module. (This includes those needed by other packages we
plan to remove.)
* Leave at least a week for packagers to respond to the bugs.
* If the packager approves or there is no response for a week, we will
remove the package.

There are currently ~3000 source packages that generate python2 dependent RPMs.
Automation is being created to be able to provide us a rough list of
packages to remove.

=== Packages to remove ===
The list is at https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal/List
not to disturb the reading experience of this page.

Packagers can block some packages from removal by responding on
Bugzilla and providing reasons. If no general consensus is reached,
FESCo will decide (as Python SIG does not have a formal process for
such decisions).

We'll also actively try to remove unused or optional dependencies to
reduce the number of packages that need to be kept because other
packages depend on them.

As dependencies evolve over time, we will regularly repeat this proces
on rawhide from now on even on future releases.

Packagers are strongly encouraged to help port their applications to
Python 3. Removing old Python 2 only applications from Fedora is also
encouraged, especially if the upstream is dead. Python SIG is
available to help with porting.

Packaging guidelines will be updated to reflect that packaging for
Python 3 only is the default. Instructions for Python 2 and
dual-support will be moved to the Appendix.

Python 2 will be marked deprecated (see
https://fedoraproject.org/wiki/Packaging:Deprecating_Packages).

Any package that only provides a Python 2 importable module may be
marked as deprecated as well if the maintainer(s) want to.

== Benefit to Fedora ==
A giant pile of unneeded software running on a legacy interperter will
be removed from Fedora before the interpreter stops being supported
upstream.

While we would very much like to remove Python 2 entirely, this way we
are not breaking Fedora, and we can accomodate some exceptions.

== Scope ==
* Proposal owners:
** Provide a list of packages to remove
** Collect feedback from affected packagers
** Retire the python2 only packages
** Remove python2 subpackages from dual-python packages

* Other developers:
** Ask for packages to be removed from the list if needed (with reasons).
** Remove python2 subpackages from dual-python packages (not needed,
but helpful)
** Optionaly mark remainign python2 packages deprecated

* Policies and guidelines:
** Python packaging guidelines will be changed to only describe Python 3
** Python 2 and dual-support packaging will be moved to the appendix
** Only packages providing additional features different than
importable modules will be allowed to be added in Fedora (with
exception for dependenecies of other packages)
** Python 2 will be marked deprecated
https://fedoraproject.org/wiki/Packaging:Deprecating_Packages
* Trademark approval: not needed for this Change

== Upgrade/compatibility impact ==
Packages removed form Fedora repos will remain on the installed OS
until explicitly removed by the user or obsoleted by the packagers.
We'll only obsolete packages that would break upgrade (from
fedora-obsolete-packages).

== How To Test ==
Any program written in Python and any program that has Python plugins
could potentially be influenced by this change. Testing is different
for software that is still using Python 2 and for software that has
been switched to Python 3.

For any ''python2'' programs, make sure those programs are still
functional after the package removal. Since various packages that are
no longer built will not be removed from an existing system, just
upgrading and checking packages is not enough. Either a new
installation should be used, or after a branching point, any packages
which haven't been rebuilt for F30, so any packages with .fc29 or
lower release suffix should be removed from the system before testing.

For the ''python3'' programs, install all upgrades and check if the
software works.

Upgrade failures because of missing dependencies should be treated as
bugs. Any removed python2 subpackages which break upgrades need to be
added to Obsoletes in fedora-obsolete-packages.

== User Experience ==
There will be less Python 2 RPMs in Fedora repos. Users are encouraged
to switch to Python 3 and/or use Python 2 virtual environments and pip
for development.

== Dependencies ==
Ideally, all programs that use python2 would be switched to use
python3. Although we don't expect everything to be switched over, as
much as possible should be, so that the set of remaining python2
subpackages is as small as possible.

== Contingency Plan ==
* Contingency mechanism:
** If for some reason not everything is removed, nothing happens, it
can be removed later. If for some reason something breaks, some
packages can be unretired or restored.
** If someone steps up to maintain Python 2 (including the full
ecosystem of packages now in Fedora), they can decide to discontinue
removing packages, revert the Change, or come up with another plan.
(Note that in this case, current maintainers will most likely orphan
many fundamental python2 packages.)

== Documentation ==
This page is the main documentation.

Also see: https://pythonclock.org/

-- 
Ben Cotton
Fedora Program Manager
TZ=America/Indiana/Indianapolis

[Randy Barlow]

On 08/20/2018 04:41 PM, Ben Cotton wrote:
>  Python 2 will be deprecated in Fedora. Packagers can mark any other
>  Python 2 packages as deprecated as well. 
I didn't know there was a way to do this, but the change page does link
to a policy for it:

https://fedoraproject.org/wiki/Packaging:Deprecating_Packages

Neat ☺

One question: is there a date we should put on the deprecation, or
should we just mark Python 2 packages as deprecated for now?

[Miro Hrončok]

On 7.9.2018 19:45, Randy Barlow wrote:
>  On 08/20/2018 04:41 PM, Ben Cotton wrote:
> > Python 2 will be deprecated in Fedora. Packagers can mark any other
> > Python 2 packages as deprecated as well.
>  
>  I didn't know there was a way to do this, but the change page does link
>  to a policy for it:
>  
>  https://fedoraproject.org/wiki/Packaging:Deprecating_Packages
>  
>  Neat ☺
>  
>  One question: is there a date we should put on the deprecation, or
>  should we just mark Python 2 packages as deprecated for now? 
There is no date when we remove python2 from Fedora, at least not yet. 
Hence we've only put deprecated() in Python.

As for your Python 2 packages, I guess consult with consumers (dependent 
packages) and set a date that works for all of them, or do the same 
thing as we did with python2.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok

[Sérgio Basto]

Hi,

On remove python2-foo, shouldn't we add "Obsoletes: python2-foo" in foo
package ? 

Thanks.


On Mon, 2018-08-20 at 16:41 -0400, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
>  
>  == Summary ==
>  (Sub-)packages only providing python2 importable modules without
>  additional functionality will be removed from Fedora unless some
>  other
>  package(s) depends on them.
>  
>  Python 2 will be deprecated in Fedora. Packagers can mark any other
>  Python 2 packages as deprecated as well.
>  
>  == Owner ==
>  * Miro Hrončok (Churchyard)
>  * Petr Viktorin (Pviktori)
>  * Charalampos Stratakis (Cstratak)
>  * Zbigniew Jędrzejewski-Szmek (Zbyszek)
>  * Igor Gantenko (Ignatenkobrain)
>  * Neal Gompa (Ngompa)
>  
>  == Detailed Description ==
>  Python 2 reaches End of Life on 2020-01-01. Its current maintainers
>  would like to orphan it before that date, and so far no one else has
>  stepped up to maintain Python 2 (with the full ecosystem) past 2020.
>  Since thousands of packages still depend on python2, we need a more
>  careful approach than normal orphaning.
>  Some of those packages are abandoned and/or the Python 2 version is
>  unnecessary. Others are useful and just need more time to port to
>  Python 3.
>  Hence we set up criteria for python2 packages that can remain in the
>  distribution and we remove everything else. This should allow us to
>  keep python2 for limited use, not break everything, but should also
>  send a strong message that it is no longer a first class citizen, and
>  filter packages we need to focus Python 3 porting efforts on.
>  
>  (Sub-)packages that only provide a python2 importable module, and are
>  not required for other packages, will be removed.
>  
>  Examples of situations where a (sub-)package does not provide only
>  the
>  importable module:
>  
>  * A package also provides an application, mostly likely in /usr/bin,
>  /usr/libexec...
>  ** (Note that according to current guidelines, if the Python 3
>  version
>  of a package provides the same functionality, the Python 3 package
>  should provide the application.)
>  ** (In certain situations the provided application is only useful to
>  boostrap or manage projects using the module. Such applications don't
>  count.)
>  * A package provides a plugin for another application, most likely
>  trough a setuptools entrypoint interface or some custom location on
>  disk.
>  
>  Our process will be:
>  
>  * File bugs for all packages that look like they only provide a
>  Python
>  2 importable module. (This includes those needed by other packages we
>  plan to remove.)
>  * Leave at least a week for packagers to respond to the bugs.
>  * If the packager approves or there is no response for a week, we
>  will
>  remove the package.
>  
>  There are currently ~3000 source packages that generate python2
>  dependent RPMs.
>  Automation is being created to be able to provide us a rough list of
>  packages to remove.
>  
>  === Packages to remove ===
>  The list is at 
>  https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal/List
>  not to disturb the reading experience of this page.
>  
>  Packagers can block some packages from removal by responding on
>  Bugzilla and providing reasons. If no general consensus is reached,
>  FESCo will decide (as Python SIG does not have a formal process for
>  such decisions).
>  
>  We'll also actively try to remove unused or optional dependencies to
>  reduce the number of packages that need to be kept because other
>  packages depend on them.
>  
>  As dependencies evolve over time, we will regularly repeat this
>  proces
>  on rawhide from now on even on future releases.
>  
>  Packagers are strongly encouraged to help port their applications to
>  Python 3. Removing old Python 2 only applications from Fedora is also
>  encouraged, especially if the upstream is dead. Python SIG is
>  available to help with porting.
>  
>  Packaging guidelines will be updated to reflect that packaging for
>  Python 3 only is the default. Instructions for Python 2 and
>  dual-support will be moved to the Appendix.
>  
>  Python 2 will be marked deprecated (see
>  https://fedoraproject.org/wiki/Packaging:Deprecating_Packages).
>  
>  Any package that only provides a Python 2 importable module may be
>  marked as deprecated as well if the maintainer(s) want to.
>  
>  == Benefit to Fedora ==
>  A giant pile of unneeded software running on a legacy interperter
>  will
>  be removed from Fedora before the interpreter stops being supported
>  upstream.
>  
>  While we would very much like to remove Python 2 entirely, this way
>  we
>  are not breaking Fedora, and we can accomodate some exceptions.
>  
>  == Scope ==
>  * Proposal owners:
>  ** Provide a list of packages to remove
>  ** Collect feedback from affected packagers
>  ** Retire the python2 only packages
>  ** Remove python2 subpackages from dual-python packages
>  
>  * Other developers:
>  ** Ask for packages to be removed from the list if needed (with
>  reasons).
>  ** Remove python2 subpackages from dual-python packages (not needed,
>  but helpful)
>  ** Optionaly mark remainign python2 packages deprecated
>  
>  * Policies and guidelines:
>  ** Python packaging guidelines will be changed to only describe
>  Python 3
>  ** Python 2 and dual-support packaging will be moved to the appendix
>  ** Only packages providing additional features different than
>  importable modules will be allowed to be added in Fedora (with
>  exception for dependenecies of other packages)
>  ** Python 2 will be marked deprecated
>  https://fedoraproject.org/wiki/Packaging:Deprecating_Packages
>  * Trademark approval: not needed for this Change
>  
>  == Upgrade/compatibility impact ==
>  Packages removed form Fedora repos will remain on the installed OS
>  until explicitly removed by the user or obsoleted by the packagers.
>  We'll only obsolete packages that would break upgrade (from
>  fedora-obsolete-packages).
>  
>  == How To Test ==
>  Any program written in Python and any program that has Python plugins
>  could potentially be influenced by this change. Testing is different
>  for software that is still using Python 2 and for software that has
>  been switched to Python 3.
>  
>  For any ''python2'' programs, make sure those programs are still
>  functional after the package removal. Since various packages that are
>  no longer built will not be removed from an existing system, just
>  upgrading and checking packages is not enough. Either a new
>  installation should be used, or after a branching point, any packages
>  which haven't been rebuilt for F30, so any packages with .fc29 or
>  lower release suffix should be removed from the system before
>  testing.
>  
>  For the ''python3'' programs, install all upgrades and check if the
>  software works.
>  
>  Upgrade failures because of missing dependencies should be treated as
>  bugs. Any removed python2 subpackages which break upgrades need to be
>  added to Obsoletes in fedora-obsolete-packages.
>  
>  == User Experience ==
>  There will be less Python 2 RPMs in Fedora repos. Users are
>  encouraged
>  to switch to Python 3 and/or use Python 2 virtual environments and
>  pip
>  for development.
>  
>  == Dependencies ==
>  Ideally, all programs that use python2 would be switched to use
>  python3. Although we don't expect everything to be switched over, as
>  much as possible should be, so that the set of remaining python2
>  subpackages is as small as possible.
>  
>  == Contingency Plan ==
>  * Contingency mechanism:
>  ** If for some reason not everything is removed, nothing happens, it
>  can be removed later. If for some reason something breaks, some
>  packages can be unretired or restored.
>  ** If someone steps up to maintain Python 2 (including the full
>  ecosystem of packages now in Fedora), they can decide to discontinue
>  removing packages, revert the Change, or come up with another plan.
>  (Note that in this case, current maintainers will most likely orphan
>  many fundamental python2 packages.)
>  
>  == Documentation ==
>  This page is the main documentation.
>  
>  Also see: https://pythonclock.org/
>  
>  -- 
>  Ben Cotton
>  Fedora Program Manager
>  TZ=America/Indiana/Indianapolis
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
-- 
Sérgio M. B.

[Miro Hrončok]

On 16. 02. 19 22:01, Sérgio Basto wrote:
>  Hi,
>  
>  On remove python2-foo, shouldn't we add "Obsoletes: python2-foo" in foo
>  package ? 
To fedora-obsolete-packages and only in certain cases.

See https://pagure.io/packaging-committee/issue/754


-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok