F27 Self Contained Change: Making sudo pip Safe (Again)

[Jan Kurik]

= Proposed Self Contained Change: Making sudo pip Safe (Again) =
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Change owner(s):
* Michal Cyprian <mcyprian AT redhat DOT com>
* Petr Viktorin <pviktori AT redhat DOT com>
* Tomas Orsava <torsava AT redhat DOT com>
* Miro Hroncok <mhroncok AT redhat DOT com>


At the present time, running sudo pip3 in Fedora is not safe. Pip
shares its installation directory with dnf, can remove dnf-managed
files and generally break the Python 3 interpreter. We propose a
series of measures that will make it safe to use.


== Detailed Description ==
The danger of using sudo pip3 stems from the fact that both Python dnf
packages and sudo pip3 install modules to the same location, namely
/usr/lib/pythonX.Y/site-packages.

We aim to move the working directory for sudo pip3 to a more
appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
modify the Python 3 interpreter in Fedora to scan both above mentioned
locations when importing modules. In addition, system-python—a
stripped down version of Python 3 for use by system tools—will not
read the sudo pip3 install location, making it more secure by being
less susceptible to interference by user-downloaded modules.

From the technical standpoint, this will be accomplished by changing
the install prefix setting of the distutils install command in the
/usr/bin/python3 executable from /usr/ to /usr/local. pip3 and
distutils will thereafter use this prefix when determining where to
install modules. In addition, the paths
/usr/local/lib/pythonX.Y/site-packages and
/usr/local/lib64/pythonX.Y/site-packages will be added to the front of
the sys.path variable so that modules are imported preferentially from
there. These settings, however, will not be modified for the
system-python binary, the /usr/bin/python3 executable when running
with -I option specified, nor when an RPM build is detected.
Therefore, Python RPM packages will continue to be built with the
correct installation path for system modules.

The purpose of this change is not to make sudo pip a standard way to
install Python packages. Virtual environments and pip3 install --user
should still be the prefered options. Nevertheless, sudo pip is far
too prevalent an instruction in various guides and installation notes
throughout the Internet that there is little hope of changing users'
behaviour in this regard.


== Scope ==
* Proposal owners:
Modify the distutils install command as described above.
Modify the site.py script to add additional paths to sys.path when it is needed.

* Other developers:
N/A (not a System Wide Change)

* Release engineering:
https://pagure.io/releng/issue/6820

* List of deliverables:
N/A (not a System Wide Change)

* Policies and guidelines:
N/A (not a System Wide Change)

* Trademark approval:
Not needed for this Change


-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

[Björn 'besser82' Esser]

Am 31.05.2017 um 16:20 schrieb Jan Kurik:
>  = Proposed Self Contained Change: Making sudo pip Safe (Again) =
>  https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
>
>  Change owner(s):
>  * Michal Cyprian <mcyprian AT redhat DOT com>
>  * Petr Viktorin <pviktori AT redhat DOT com>
>  * Tomas Orsava <torsava AT redhat DOT com>
>  * Miro Hroncok <mhroncok AT redhat DOT com>
>
>
>  At the present time, running sudo pip3 in Fedora is not safe. Pip
>  shares its installation directory with dnf, can remove dnf-managed
>  files and generally break the Python 3 interpreter. We propose a
>  series of measures that will make it safe to use.
>
>
>  == Detailed Description ==
>  The danger of using sudo pip3 stems from the fact that both Python dnf
>  packages and sudo pip3 install modules to the same location, namely
>  /usr/lib/pythonX.Y/site-packages.
>
>  We aim to move the working directory for sudo pip3 to a more
>  appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
>  modify the Python 3 interpreter in Fedora to scan both above mentioned
>  locations when importing modules. In addition, system-python—a
>  stripped down version of Python 3 for use by system tools—will not
>  read the sudo pip3 install location, making it more secure by being
>  less susceptible to interference by user-downloaded modules.
>
>   From the technical standpoint, this will be accomplished by changing
>  the install prefix setting of the distutils install command in the
>  /usr/bin/python3 executable from /usr/ to /usr/local. pip3 and
>  distutils will thereafter use this prefix when determining where to
>  install modules. In addition, the paths
>  /usr/local/lib/pythonX.Y/site-packages and
>  /usr/local/lib64/pythonX.Y/site-packages will be added to the front of
>  the sys.path variable so that modules are imported preferentially from
>  there. These settings, however, will not be modified for the
>  system-python binary, the /usr/bin/python3 executable when running
>  with -I option specified, nor when an RPM build is detected.
>  Therefore, Python RPM packages will continue to be built with the
>  correct installation path for system modules.
>
>  The purpose of this change is not to make sudo pip a standard way to
>  install Python packages. Virtual environments and pip3 install --user
>  should still be the prefered options. Nevertheless, sudo pip is far
>  too prevalent an instruction in various guides and installation notes
>  throughout the Internet that there is little hope of changing users'
>  behaviour in this regard.
>
>
>  == Scope ==
>  * Proposal owners:
>  Modify the distutils install command as described above.
>  Modify the site.py script to add additional paths to sys.path when it is needed.
>
>  * Other developers:
>  N/A (not a System Wide Change)
>
>  * Release engineering:
>  https://pagure.io/releng/issue/6820
>
>  * List of deliverables:
>  N/A (not a System Wide Change)
>
>  * Policies and guidelines:
>  N/A (not a System Wide Change)
>
>  * Trademark approval:
>  Not needed for this Change 
+1 for this proposal

[Nico Kadel-Garcia]

On Wed, May 31, 2017 at 10:46 AM, Björn 'besser82' Esser
<besser82(a)fedoraproject.org&gt; wrote:
>  Am 31.05.2017 um 16:20 schrieb Jan Kurik:
> >
> > = Proposed Self Contained Change: Making sudo pip Safe (Again) =
> > https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
> >
> > Change owner(s):
> > * Michal Cyprian <mcyprian AT redhat DOT com>
> > * Petr Viktorin <pviktori AT redhat DOT com>
> > * Tomas Orsava <torsava AT redhat DOT com>
> > * Miro Hroncok <mhroncok AT redhat DOT com>
> >
> >
> > At the present time, running sudo pip3 in Fedora is not safe. Pip
> > shares its installation directory with dnf, can remove dnf-managed
> > files and generally break the Python 3 interpreter. We propose a
> > series of measures that will make it safe to use. 
> >  From the technical standpoint, this will be accomplished by
> changing
> > the install prefix setting of the distutils install command in the
> > /usr/bin/python3 executable from /usr/ to /usr/local. pip3 and
> > distutils will thereafter use this prefix when determining where to
> > install modules. In addition, the paths
> > /usr/local/lib/pythonX.Y/site-packages and
> > /usr/local/lib64/pythonX.Y/site-packages will be added to the front of
> > the sys.path variable so that modules are imported preferentially from
> > there. These settings, however, will not be modified for the
> > system-python binary, the /usr/bin/python3 executable when running
> > with -I option specified, nor when an RPM build is detected.
> > Therefore, Python RPM packages will continue to be built with the
> > correct installation path for system modules. 
I wrote about this idea before: I'll re-iterate my concerns once, for
people new to the thread.

I still think it's a bad idea. What you're describing is essentially
reimplementing pyenv to point to /usr/local/, and automatically
activating that pyenv-like target as enabled by default for all users
of that system. It could, if engineered well, help avoid the
overwriting of RPM bundled components of precisely the same version
with pip installed versions, and vice versa. What it provides no help
with, because this shareable location will be activated by default
ahead of the more standard RPM bundled modules, is when newer versions
of modules installed by "pip3 install" are with existing modules of
any sort from any default discoverable location. The problem also
exists with pyvenv, but on a much smaller scale and only when the user
specifically selects to use that local installation. It remains very
difficult for older modules to include module dependencies to avoid
this, since at the time the older module was written the newer and
incompatible module may not have existed.

Been there, done that, with awscli and various Sphinx module versions
in the airflow modules dependency tree. With airflow, the Sphinx
upgrades broke quite a few previously working pip module
installations. And with some modules, you have to update "pip" itself.
Activating a new pip module this way is *begging* to create pain
building tools that previously built well. As I said, been there, done
that.

I acknowledge that it could make cleaning up after that kind of "sudo
pip3 install pip" safer. But I'd worry more about "sudo pip3 install
[module-that-requires-Sphinx-upgrade]" and having to dig for where the
bugs are for other modules.

> > The purpose of this change is not to make sudo pip a standard way
> to
> > install Python packages. Virtual environments and pip3 install --user
> > should still be the prefered options. Nevertheless, sudo pip is far
> > too prevalent an instruction in various guides and installation notes
> > throughout the Internet that there is little hope of changing users'
> > behaviour in this regard. 
*That* part is very real. I'm not clear that this actually solves
anything, since it leaves this shared location as the system default.


>>
>>
>> == Scope ==
>> * Proposal owners:
>> Modify the distutils install command as described above.
>> Modify the site.py script to add additional paths to sys.path when it is
>> needed.
>>
>> * Other developers:
>> N/A (not a System Wide Change)
>>
>> * Release engineering:
>> https://pagure.io/releng/issue/6820
>>
>> * List of deliverables:
>> N/A (not a System Wide Change)
>>
>> * Policies and guidelines:
>> N/A (not a System Wide Change)
>>
>> * Trademark approval:
>> Not needed for this Change
>
>
> +1 for this proposal
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Michal Cyprian]

There are plenty of bugs [0] reported by users, who break their 
systems using sudo pip and new bugs appear very often.
For example the bug [1] was opened a week ago. The biggest 
issue is that in the current state it is not easy or even possible 
to recover from such situations.
This issue is well known upstream as well, unfortunately there 
is not an elegant and simple solution to fix it for all the distributions.
Debian and its derivatives use [2] patch similar to our proposal for
Fedora since Python version 2.6.

This topic has been discussed many times on Fedora mailing lists, 
for example [3] and all the reasons why the Change is needed were 
described in detail. We believe this is a good first step on the way
to a safe Python environment.

I would also like to make clear some points discussed on the last
FESCo meeting [4]. This proposal was approved as a Change for F26.
However, some issues with the chosen implementation surfaced and there 
was not enough time for a proper investigation and fix of the problem
and so we decided to postpone it for F27.

The implementation was redesigned so the problems encountered previously
won't resurface. The main parts of the patch for F27 are similar 
to the Debian's fix, which was proved to work in many different contexts by now.

[0] https://www.google.cz/search?q="sudo+pip"+site%3Abugzilla.redha...
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1459244
[2] https://wiki.debian.org/Python
[3]
https://lists.fedoraproject.org/archives/list/[e-mail].o...
[4]
https://meetbot.fedoraproject.org/fedora-meeting/2017-06-09/fesco.2017-06...

Michal Cyprian

----- Original Message -----
From: "Nico Kadel-Garcia" <nkadel(a)gmail.com&gt;
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org&gt;
Sent: Thursday, June 1, 2017 12:07:16 PM
Subject: Re: F27 Self Contained Change: Making sudo pip Safe (Again)

On Wed, May 31, 2017 at 10:46 AM, Björn 'besser82' Esser
<besser82(a)fedoraproject.org&gt; wrote:
>  Am 31.05.2017 um 16:20 schrieb Jan Kurik:
> >
> > = Proposed Self Contained Change: Making sudo pip Safe (Again) =
> > https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
> >
> > Change owner(s):
> > * Michal Cyprian <mcyprian AT redhat DOT com>
> > * Petr Viktorin <pviktori AT redhat DOT com>
> > * Tomas Orsava <torsava AT redhat DOT com>
> > * Miro Hroncok <mhroncok AT redhat DOT com>
> >
> >
> > At the present time, running sudo pip3 in Fedora is not safe. Pip
> > shares its installation directory with dnf, can remove dnf-managed
> > files and generally break the Python 3 interpreter. We propose a
> > series of measures that will make it safe to use. 
> >  From the technical standpoint, this will be accomplished by
> changing
> > the install prefix setting of the distutils install command in the
> > /usr/bin/python3 executable from /usr/ to /usr/local. pip3 and
> > distutils will thereafter use this prefix when determining where to
> > install modules. In addition, the paths
> > /usr/local/lib/pythonX.Y/site-packages and
> > /usr/local/lib64/pythonX.Y/site-packages will be added to the front of
> > the sys.path variable so that modules are imported preferentially from
> > there. These settings, however, will not be modified for the
> > system-python binary, the /usr/bin/python3 executable when running
> > with -I option specified, nor when an RPM build is detected.
> > Therefore, Python RPM packages will continue to be built with the
> > correct installation path for system modules. 
I wrote about this idea before: I'll re-iterate my concerns once, for
people new to the thread.

I still think it's a bad idea. What you're describing is essentially
reimplementing pyenv to point to /usr/local/, and automatically
activating that pyenv-like target as enabled by default for all users
of that system. It could, if engineered well, help avoid the
overwriting of RPM bundled components of precisely the same version
with pip installed versions, and vice versa. What it provides no help
with, because this shareable location will be activated by default
ahead of the more standard RPM bundled modules, is when newer versions
of modules installed by "pip3 install" are with existing modules of
any sort from any default discoverable location. The problem also
exists with pyvenv, but on a much smaller scale and only when the user
specifically selects to use that local installation. It remains very
difficult for older modules to include module dependencies to avoid
this, since at the time the older module was written the newer and
incompatible module may not have existed.

Been there, done that, with awscli and various Sphinx module versions
in the airflow modules dependency tree. With airflow, the Sphinx
upgrades broke quite a few previously working pip module
installations. And with some modules, you have to update "pip" itself.
Activating a new pip module this way is *begging* to create pain
building tools that previously built well. As I said, been there, done
that.

I acknowledge that it could make cleaning up after that kind of "sudo
pip3 install pip" safer. But I'd worry more about "sudo pip3 install
[module-that-requires-Sphinx-upgrade]" and having to dig for where the
bugs are for other modules.

> > The purpose of this change is not to make sudo pip a standard way
> to
> > install Python packages. Virtual environments and pip3 install --user
> > should still be the prefered options. Nevertheless, sudo pip is far
> > too prevalent an instruction in various guides and installation notes
> > throughout the Internet that there is little hope of changing users'
> > behaviour in this regard. 
*That* part is very real. I'm not clear that this actually solves
anything, since it leaves this shared location as the system default.


> >
> >
> > == Scope ==
> > * Proposal owners:
> > Modify the distutils install command as described above.
> > Modify the site.py script to add additional paths to sys.path when it is
> > needed.
> >
> > * Other developers:
> > N/A (not a System Wide Change)
> >
> > * Release engineering:
> > https://pagure.io/releng/issue/6820
> >
> > * List of deliverables:
> > N/A (not a System Wide Change)
> >
> > * Policies and guidelines:
> > N/A (not a System Wide Change)
> >
> > * Trademark approval:
> > Not needed for this Change
>
>
>  +1 for this proposal
>
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org