Thanks for all suggestions and comments. Debian's patch is very complex compared to
our solution.
Our patch is 12 lines long including docstring and it doesn't break any tests.
Behavior of
/usr/bin/python3 executable with customized sys.prefix will be a bit similar to virtual
environment
created with option --system-site-packages. Early results of whole environment
(system-python + patched python3)
tests in mock are promising.
Something similar to this proposal:
CFLAGS="%{optflags}" %{__system_python} %{py_setup}
%{?py_setup_args} build
--executable="%{__python3} %{py3_shbang_opts}" %{?*}
could help us to enable `sudo pip`-installed extensions of DNF-installed programs.
Unfortunately --executable doesn't affect scripts listed as 'entry-points' in
setup.py script.
We will probably have to patch setuptools to make this work.
Michal Cyprian
----- Original Message -----
From: "Petr Viktorin" <pviktori(a)redhat.com>
To: python-devel(a)lists.fedoraproject.org, "Michal Cyprian"
<mcyprian(a)redhat.com>
Sent: Monday, December 12, 2016 11:57:23 AM
Subject: Re: Making sudo pip Safe
On 12/09/2016 06:09 PM, Orion Poplawski wrote:
> On 12/07/2016 05:53 AM, Michal Cyprian wrote:
>> Hello,
>>
>> there is a long-standing problem that `sudo pip install` cannot be safely used in
Fedora. Many users don't know about this and break python packages on theirs systems.
Packages installed using this command can conflict and overwrite Python rpm packages.
>> This is a major problem and we have seen several systems broken by people using
"sudo pip". Unfortunately, telling people to not use it will not work:
"sudo pip" appears in documentation of too many projects online.
>> We plan to solve this in Fedora 26. A more precise description of the problem
follows.
>>
>> Current behavior of python packages installation tools in Fedora:
>> 1) sudo dnf install python3-foo # root installs foo from rpm to
/usr/lib/pythonX.Y/site-packages
>> 2) sudo pip3 install foo # root installs foo from PyPI to
/usr/lib/pythonX.Y/site-packages
>> 3) sudo pip3 install -t /usr/local/lib/pythonX.Y/site-packages foo # root
install package to selected location
>> 4) pip3 install --user foo # user install foo from PyPI to
~/.local/lib/python3.5/site-packages
>>
>> Using the --user option with pip would be the ideal solution. However, it is
reportedly broken in some versions of Ubuntu, so it is hard to convince software authors
to recommend it.
>>
>> Packages installed using `sudo pip` (2) under /usr can be overwritten or removed
by dnf. `dnf install python3-foo` fails if foo was `sudo pip installed` before etc. This
problem has been reported many times. [6]
>> Another issue is that packages installed under usr/local (3) cannot be imported
unless PYTHONPATH or sys.path is set explicitly.
>>
>> The default install location of pip/distutils-installed packages depends on the
value of the
>> sys.prefix variable [4].
>> There were several discussions on Bugzilla [1] and the pypa-dev mailing list
[2].
>> Interesting solutions were conceived at the CPython level:
>> - addition of sys.local_prefix [2]
>> - simplification of CPython startup sequence [4]
>> Unfortunately none of them were realized and both solutions require many changes
of Python and Python Standard Library.
>>
>> We discussed all the possible solutions with colleagues from the Python maint
team.
>> It would be great to fix this upstream, but it won't happen in a reasonable
time frame.
>> We realized that System Python [3], which was announced in Fedora 24, can help us
reach the goal. We came up with the following solution:
>> - sys.prefix of the python3 executable will be set to /usr/local
>> - sys.prefix of system-python will be set to /usr
>> - all rpm python packages will be installed using system-python (value of rpm
macro %{__python3} will be changed to point to system-python)
>> - The path /usr/lib/.../site_packages will be included in sys.path of
/usr/bin/python3
>>
>> Note that no packages will be affected by other work going on around
system-python, namely that packages can declare that they are only using a smaller subset
of the Python stdlib. That option is still opt-in.
>>
>> Behavior of install methods after mentioned changes:
>> 1) sudo dnf install foo # root installs foo from rpm to
/usr/lib/pythonX.Y/site-packages
>> 2) sudo pip3 install foo # root installs foo from PyPI to
/usr/local/lib/pythonX.Y/site-packages
>> - pip never touches /usr/lib/python3.5/site-packages
>> - /usr/bin/python3 can import modules from
>> - Python Standard Library
>> - site-packages under /usr/local (priority)
>> - site-packages under /usr
>> - system-python (i.e. what all programs installed via DNF will use) is limited to
site-packages under /usr, so DNF-installed software is unaffected by anything installed
with pip
>>
>
>> Michal Cyprian
>
>
> I applaud this effort. My concern comes from the question of whether or not
> one should be able to extend the available library for system tools. If I'm
> using the dnf installed jupyter notebook and I want to use a newer version of
> some other package or something that isn't packaged and I install it via
"sudo
> pip install foo" I can't import it without changing PYTHONPATH, which
brings
> us back to the problems with method #3 above. Or can we get around this by
> not changing %__python3 but changing %py3_build to be:
>
CFLAGS="%{optflags}" %{__system_python} %{py_setup}
%{?py_setup_args} build
--executable="%{__python3} %{py3_shbang_opts}" %{?*}
>
> And still having the installed shebangs reference /usr/bin/python3.X?
Right, it makes sense for DNF-installed programs that expose a Python
shell to the user or are otherwise extensible using pip should have
access to `sudo pip`-installed packages, and there should probably be a
nice macro to make that easy.
Michal, could you make sure that case is covered in the proposal?
We had a discussion a while back about adding -E or -I to the
py3_shbang_opts,
but this was felt to be too disruptive. I think having everything installed
with system_python shbangs would be fairly similar. Or am I missing something?
Yes, it would make a lot of sense for system-python to always behave as
if -I was specified.
Debian deals with this by having dist-packages
(
https://wiki.debian.org/Python). Is this not worth adopting?
This diverges from upstream even more, and AFAICS the benefit it brings
is a that if you compile Python from source with the default settings,
it is isolated. With Python 3's built-in isolation between different
*versions* of Python, I think there's no problem with separating
packages installed with `sudo pip` from the system Python and from a
Python compiled from source.
--
Petr Viktorin