Making sudo pip Safe

[Michal Cyprian]

Hello,

there is a long-standing problem that `sudo pip install` cannot be safely used in Fedora.
Many users don't know about this and break python packages on theirs systems. Packages
installed using this command can conflict and overwrite Python rpm packages.
This is a major problem and we have seen several systems broken by people using "sudo
pip". Unfortunately, telling people to not use it will not work: "sudo pip"
appears in documentation of too many projects online.
We plan to solve this in Fedora 26. A more precise description of the problem follows.

Current behavior of python packages installation tools in Fedora:
1) sudo dnf install python3-foo  # root installs foo from rpm to
/usr/lib/pythonX.Y/site-packages
2) sudo pip3 install foo  # root installs foo from PyPI to
/usr/lib/pythonX.Y/site-packages
3) sudo pip3 install -t /usr/local/lib/pythonX.Y/site-packages foo  # root install package
to selected location
4) pip3 install --user foo # user install foo from PyPI to
~/.local/lib/python3.5/site-packages

Using  the --user option with pip would be the ideal solution. However, it is reportedly
broken in some versions of Ubuntu, so it is hard to convince software authors to recommend
it.

Packages installed using `sudo pip` (2) under /usr can be overwritten or removed by dnf.
`dnf install python3-foo` fails if foo was `sudo pip installed` before etc. This problem
has been reported many times. [6]
Another issue is that packages installed under usr/local (3) cannot be imported unless
PYTHONPATH or sys.path is set explicitly.

The default install location of pip/distutils-installed packages depends on the value of
the
 sys.prefix variable [4].
There were several discussions on Bugzilla [1] and the pypa-dev mailing list [2].
Interesting solutions were conceived at the CPython level:
    - addition of sys.local_prefix [2]
    - simplification of CPython startup sequence [4] 
Unfortunately none of them were realized and both solutions require many changes of Python
and Python Standard Library. 

We discussed all the possible solutions with colleagues from the Python maint team.
It would be great to fix this upstream, but it won't happen in a reasonable time
frame.
We realized that System Python [3], which was announced in Fedora 24, can help us reach
the goal. We came up with the following solution:
  - sys.prefix of the python3 executable will be set to /usr/local
  - sys.prefix of system-python will be set to /usr
  - all rpm python packages will be installed using system-python (value of rpm macro
%{__python3} will be changed to point to system-python) 
  - The path /usr/lib/.../site_packages will be included in sys.path of /usr/bin/python3

Note that no packages will be affected by other work going on around system-python, namely
that packages can declare that they are only using a smaller subset of the Python stdlib.
That option is still opt-in.

Behavior of install methods after mentioned changes:
1) sudo dnf install foo  # root installs foo from rpm to /usr/lib/pythonX.Y/site-packages
2) sudo pip3 install foo  # root installs foo from PyPI to
/usr/local/lib/pythonX.Y/site-packages
- pip never touches /usr/lib/python3.5/site-packages
- /usr/bin/python3 can import modules from
    - Python Standard Library
    - site-packages under /usr/local (priority)
    - site-packages under /usr
- system-python (i.e. what all programs installed via DNF will use) is limited to
site-packages under /usr, so DNF-installed software is unaffected by anything installed
with pip


[1] https://bugzilla.redhat.com/show_bug.cgi?id=662034
[2] https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0
[3] https://fedoraproject.org/wiki/Changes/System_Python
[4] https://docs.python.org/3/library/sys.html#sys.prefix
[5] https://www.python.org/dev/peps/pep-0432/

[6] For example:
* https://bugzilla.redhat.com/show_bug.cgi?id=1396158
* https://bugzilla.redhat.com/show_bug.cgi?id=1397575
* https://bugzilla.redhat.com/show_bug.cgi?id=1400377
* More: https://www.google.cz/search?q="sudo+pip"+site%3Abugzilla.redha...
And there are many anecdotal stories of frustrated users who didn't report

Michal Cyprian