F26 Self Contained Change: Making sudo pip Safe (Again)

[Jan Kurik]

= Proposed Self Contained Change: Making sudo pip Safe (Again) =
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe

Change owner(s):
* Michal Cyprian <mcyprian AT redhat DOT com>
* Petr Viktorin <pviktori AT redhat DOT com>
* Tomas Orsava <torsava AT redhat DOT com>
* Miro Hroncok <mhroncok AT redhat DOT com>


At the present time, running sudo pip3 in Fedora is not safe. Pip
shares its installation directory with dnf, can remove dnf-managed
files and generally break the Python 3 interpreter. We propose a
series of measures that will make it safe to use.


== Detailed Description ==
The danger of using sudo pip3 stems from the fact that both Python dnf
packages and sudo pip3 install modules to the same location, namely
/usr/lib/pythonX.Y/site-packages.

We aim to move the working directory for sudo pip3 to a more
appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
modify the Python 3 interpreter in Fedora to scan both above mentioned
locations when importing modules. In addition, system-python—a
stripped down version of Python 3 for use by system tools—will not
read the sudo pip3 install location, making it more secure by being
less susceptible to interference by user-downloaded modules.

From the technical standpoint, this will be accomplished by changing
the sys.prefix setting in the /usr/bin/python3 executable from /usr/
to /usr/local. pip3 will thereafter use this prefix when determining
where to install modules. In addition, the original path
/usr/lib/pythonX.Y/site-packages will be added to the sys.path
variable (so that modules at that location are still processed when
importing), because this path will not be automatically scanned
anymore as it no longer lies inside the sys.prefix path. These
settings, however, will not be modified for the system-python binary,
and the %{__python3} macro will be changed from /usr/bin/python3 to
/usr/libexec/system-python. Therefore, Python dnf packages will
continue to be built with the correct installation path for system
modules.

Note that using sudo pip3 is not strictly necessary, as using pip3
install --user would satisfy the vast majority of use cases.
Nevertheless, sudo pip is far too prevalent an instruction in various
guides and installation notes throughout the Internet that there is
little hope of changing users' behaviour in this regard.



== Scope ==
* Proposal owners:
Modify the Python 3 executable as described above.
Modify the %{__python3} macro so that it points to /usr/libexec/system-python

* Other developers:
Spec files that use pip3 install without the use of a macro will need
to be modified accordingly. Only 3 like packages were identified
(python-flit, python-entrypoints, python-setuptools).

* Release engineering:
A rebuild of all Python packages will be necessary.

* List of deliverables:
All Fedora deliverables will be affected in a minor way that does not
jeopardize their delivery.

* Policies and guidelines:
The definition of the %{__python3} macro will be updated as mentioned above.

* Trademark approval:
Not needed for this Change
-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic

[Adam Williamson]

On Fri, 2017-01-20 at 12:07 +0100, Jan Kurik wrote:
>  = Proposed Self Contained Change: Making sudo pip Safe (Again) =
>  https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
>  
>  Change owner(s):
>  * Michal Cyprian <mcyprian AT redhat DOT com>
>  * Petr Viktorin <pviktori AT redhat DOT com>
>  * Tomas Orsava <torsava AT redhat DOT com>
>  * Miro Hroncok <mhroncok AT redhat DOT com>
>  
>  
>  At the present time, running sudo pip3 in Fedora is not safe. Pip
>  shares its installation directory with dnf, can remove dnf-managed
>  files and generally break the Python 3 interpreter. We propose a
>  series of measures that will make it safe to use.
>  
>  
>  == Detailed Description ==
>  The danger of using sudo pip3 stems from the fact that both Python dnf
>  packages and sudo pip3 install modules to the same location, namely
>  /usr/lib/pythonX.Y/site-packages.
>  
>  We aim to move the working directory for sudo pip3 to a more
>  appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
>  modify the Python 3 interpreter in Fedora to scan both above mentioned
>  locations when importing modules. 
This might also mean that we start using Python modules installed from
self-compiled applications, which might not be intended (we do not
include /usr/local/lib(64) in the default ldconfig path, AFAIK).

>   In addition, system-python—a
>  stripped down version of Python 3 for use by system tools—will not
>  read the sudo pip3 install location, making it more secure by being
>  less susceptible to interference by user-downloaded modules.
>  
>  From the technical standpoint, this will be accomplished by changing
>  the sys.prefix setting in the /usr/bin/python3 executable from /usr/
>  to /usr/local. pip3 will thereafter use this prefix when determining
>  where to install modules. 
This seems like quite a significant change. Have you investigated any
potential unexpected consequences of this? Do setuptools etc. use this
setting in any way? Have you checked for existing code reading it for
any reason? Have you checked what else Python itself uses it for, and
if any of that could be negatively affected?

>   In addition, the original path
>  /usr/lib/pythonX.Y/site-packages will be added to the sys.path
>  variable (so that modules at that location are still processed when
>  importing), because this path will not be automatically scanned
>  anymore as it no longer lies inside the sys.prefix path. These
>  settings, however, will not be modified for the system-python binary,
>  and the %{__python3} macro will be changed from /usr/bin/python3 to
>  /usr/libexec/system-python. Therefore, Python dnf packages will
>  continue to be built with the correct installation path for system
>  modules. 
>  Note that using sudo pip3 is not strictly necessary, as using pip3
>  install --user would satisfy the vast majority of use cases.
>  Nevertheless, sudo pip is far too prevalent an instruction in various
>  guides and installation notes throughout the Internet that there is
>  little hope of changing users' behaviour in this regard. 
Presumably this change is applied only to Python 3 because there is no
'system-python-2'?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Mathieu Bridon]

On Fri, 2017-01-20 at 09:12 -0800, Adam Williamson wrote:
>  On Fri, 2017-01-20 at 12:07 +0100, Jan Kurik wrote:
>  > = Proposed Self Contained Change: Making sudo pip Safe (Again) =
>  > https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
>  > 
>  > Change owner(s):
>  > * Michal Cyprian <mcyprian AT redhat DOT com>
>  > * Petr Viktorin <pviktori AT redhat DOT com>
>  > * Tomas Orsava <torsava AT redhat DOT com>
>  > * Miro Hroncok <mhroncok AT redhat DOT com>
>  > 
>  > 
>  > At the present time, running sudo pip3 in Fedora is not safe. Pip
>  > shares its installation directory with dnf, can remove dnf-managed
>  > files and generally break the Python 3 interpreter. We propose a
>  > series of measures that will make it safe to use.
>  > 
>  > 
>  > == Detailed Description ==
>  > The danger of using sudo pip3 stems from the fact that both Python
>  > dnf packages and sudo pip3 install modules to the same location,
>  > namely /usr/lib/pythonX.Y/site-packages.
>  > 
>  > We aim to move the working directory for sudo pip3 to a more
>  > appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
>  > modify the Python 3 interpreter in Fedora to scan both above
>  > mentioned locations when importing modules.
>  
>  This might also mean that we start using Python modules installed
>  from self-compiled applications, which might not be intended (we do
>  not include /usr/local/lib(64) in the default ldconfig path, AFAIK). 
In addition to that, if someone compiles and installs their own Python,
it goes into /usr/local by default. (with the typical ./configure &&
make && make install)

Which means that its modules would go into
/usr/local/lib/pythonX.Y/site-packages.

The suggested change means that the system Python would use the modules
intended for that local Python. (if they are the same version)


-- 
Mathieu

[Michal Cyprian]

----- Original Message -----
From: "Adam Williamson" <adamwill(a)fedoraproject.org&gt;
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org&gt;
Sent: Friday, January 20, 2017 6:12:33 PM
Subject: Re: F26 Self Contained Change: Making sudo pip Safe (Again)

...

> >  In addition, system-python—a
> > stripped down version of Python 3 for use by system tools—will not
> > read the sudo pip3 install location, making it more secure by being
> > less susceptible to interference by user-downloaded modules.
> > 
> > From the technical standpoint, this will be accomplished by changing
> > the sys.prefix setting in the /usr/bin/python3 executable from /usr/
> > to /usr/local. pip3 will thereafter use this prefix when determining
> > where to install modules. 
>  This seems like quite a significant change. Have you investigated
> any
>  potential unexpected consequences of this? Do setuptools etc. use this
>  setting in any way? Have you checked for existing code reading it for
>  any reason? Have you checked what else Python itself uses it for, and
>  if any of that could be negatively affected? 
We have checked how the main python packages and libraries will be affected 
by this change. Whole environment including python3, system-python, setuptools, 
pip and virtualenv was tested and it works fine.

...

>  Presumably this change is applied only to Python 3 because there is
> no
>  'system-python-2'? 
Yes, this change will be applied only to Python 3.

-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

Change owners
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Jason L Tibbitts III]

> >>>> "JK" == Jan Kurik <jkurik(a)redhat.com&gt;
> writes: 
JK> We aim to move the working directory for sudo pip3 to a more
JK> appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
JK> modify the Python 3 interpreter in Fedora to scan both above
JK> mentioned locations when importing modules.

I wanted to point out a potential caveat with using /usr/local for
things; some sites manage things under there in ways you can't really
predict.  For example, all of my machines used to rsync the entire
/usr/local tree from a master host.  Before that, it was an NFS mount.

However, I no longer do that and I don't think Fedora would ever
actually ship anything under /usr/local besides the few directories that
the filesystem package creates, so I'm certainly not going to object.  I
did, however, want make sure folks were aware that we should be careful
not to assume too much about the structure of /usr/local.

 - J<

[Michal Cyprian]

----- Original Message -----
From: "Jason L Tibbitts III" <tibbs(a)math.uh.edu&gt;
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org&gt;
Sent: Friday, January 20, 2017 6:33:36 PM
Subject: Re: F26 Self Contained Change: Making sudo pip Safe (Again)

> >>>> "JK" == Jan Kurik <jkurik(a)redhat.com&gt;
> writes: 
>  JK> We aim to move the working directory for sudo pip3 to a more
>  JK> appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
>  JK> modify the Python 3 interpreter in Fedora to scan both above
>  JK> mentioned locations when importing modules. 
>  I wanted to point out a potential caveat with using /usr/local for
>  things; some sites manage things under there in ways you can't really
>  predict.  For example, all of my machines used to rsync the entire
>  /usr/local tree from a master host.  Before that, it was an NFS mount. 
>  However, I no longer do that and I don't think Fedora would ever
>  actually ship anything under /usr/local besides the few directories that
>  the filesystem package creates, so I'm certainly not going to object.  I
>  did, however, want make sure folks were aware that we should be careful
>  not to assume too much about the structure of /usr/local. 
 - J<

Ruby on Fedora uses /usr/local/share/gems/ for packages installed as a root 
via gem tool for many years.

Sudo pip installed packages on Debian are stored under
/usr/local/lib/pythonX.Y/site-packages. 
We consider their switch to dist-packages unnecessary.

Placing user-installed modules into /usr/local is compliant with 
the Filesystem Hierarchy Standard, as pip-installed modules are 
specific to the host and not part of the distribution.

Change owners
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Vít Ondruch]

Dne 7.2.2017 v 11:32 Michal Cyprian napsal(a):
>  ----- Original Message -----
>  From: "Jason L Tibbitts III" <tibbs(a)math.uh.edu&gt;
>  To: "Development discussions related to Fedora"
> <devel(a)lists.fedoraproject.org&gt;
>  Sent: Friday, January 20, 2017 6:33:36 PM
>  Subject: Re: F26 Self Contained Change: Making sudo pip Safe (Again)
>
> >>>>> "JK" == Jan Kurik <jkurik(a)redhat.com&gt; writes:
> > JK> We aim to move the working directory for sudo pip3 to a more
> > JK> appropriate location: /usr/local/lib/pythonX.Y/site-packages, and
> > JK> modify the Python 3 interpreter in Fedora to scan both above
> > JK> mentioned locations when importing modules.
> > I wanted to point out a potential caveat with using /usr/local for
> > things; some sites manage things under there in ways you can't really
> > predict.  For example, all of my machines used to rsync the entire
> > /usr/local tree from a master host.  Before that, it was an NFS mount.
> > However, I no longer do that and I don't think Fedora would ever
> > actually ship anything under /usr/local besides the few directories that
> > the filesystem package creates, so I'm certainly not going to object.  I
> > did, however, want make sure folks were aware that we should be careful
> > not to assume too much about the structure of /usr/local.
>   - J<
>
>  Ruby on Fedora uses /usr/local/share/gems/ for packages installed as a root 
>  via gem tool for many years. 
Since you refer to Ruby, I just want to highlight that
/usr/local/share/gems/ is for packages installed by *root* via gem too
(as you correctly said) [1]. We assume that root manages the entire
system and wants installed packages to be available system wide. I don't
expect this is widely used practice. Typical user installed packages
goes into $HOME/.gem. Is there any location similar to this? As far as I
know, our users very appreciate the possibility to do "gem install"
without administrator privileges.

Vít



[1]
http://pkgs.fedoraproject.org/cgit/rpms/ruby.git/tree/operating_system.rb...

[Mathieu Bridon]

On Tue, 2017-02-07 at 13:08 +0100, Vít Ondruch wrote:
>  Dne 7.2.2017 v 11:32 Michal Cyprian napsal(a):
>  > Ruby on Fedora uses /usr/local/share/gems/ for packages installed
>  > as a root via gem tool for many years.
>  
>  Since you refer to Ruby, I just want to highlight that
>  /usr/local/share/gems/ is for packages installed by *root* via gem
>  too (as you correctly said) [1]. We assume that root manages the
>  entire system and wants installed packages to be available system
>  wide. I don't expect this is widely used practice. Typical user
>  installed packages goes into $HOME/.gem. Is there any location
>  similar to this? 
Yes, under ~/.local/lib/

It's what you get with `pip install --user $module`.

>  As far as I know, our users very appreciate the possibility to do
>  "gem install" without administrator privileges. 
So do Python developers. It's very commonly used.

But even more common is to use virtual environments, one per
application.

---

I strongly feel this feature is a move in the wrong direction. I've
never met a Python developer (and I'm one myself) installing modules
system-wide with Pip. It seems to me the only people doing that are
people who follow bad documentations.

IMHO a much better (and simpler) way to fix the (very real) issue of
unsuspecting users damaging their systems would be for pip to refuse to
run with root permissions.


-- 
Mathieu

[Petr Viktorin]

On 02/07/2017 01:22 PM, Mathieu Bridon wrote:
>  On Tue, 2017-02-07 at 13:08 +0100, Vít Ondruch wrote:
> > Dne 7.2.2017 v 11:32 Michal Cyprian napsal(a):
> >> Ruby on Fedora uses /usr/local/share/gems/ for packages installed
> >> as a root via gem tool for many years.
> >
> > Since you refer to Ruby, I just want to highlight that
> > /usr/local/share/gems/ is for packages installed by *root* via gem
> > too (as you correctly said) [1]. We assume that root manages the
> > entire system and wants installed packages to be available system
> > wide. I don't expect this is widely used practice. Typical user
> > installed packages goes into $HOME/.gem. Is there any location
> > similar to this?
>
>  Yes, under ~/.local/lib/
>
>  It's what you get with `pip install --user $module`.
>
> > As far as I know, our users very appreciate the possibility to do
> > "gem install" without administrator privileges.
>
>  So do Python developers. It's very commonly used.
>
>  But even more common is to use virtual environments, one per
>  application.
>
>  ---
>
>  I strongly feel this feature is a move in the wrong direction. I've
>  never met a Python developer (and I'm one myself) installing modules
>  system-wide with Pip. It seems to me the only people doing that are
>  people who follow bad documentations. 
Well, I've met some that are writing the bad documentation. Their 
rationale was that pip's --user option doesn't work on Ubuntu, so they 
don't have many options. (This might have been fixed since, but old 
versions are still around.)

And, not just Python developers are installing software distributed over 
PyPI.

>  IMHO a much better (and simpler) way to fix the (very real) issue of
>  unsuspecting users damaging their systems would be for pip to refuse to
>  run with root permissions. 
I wouldn't say it's simpler, unless you also want to disable `sudo pip 
--user`.

I believe that a proper fix, which includes making --user the default, 
is planned upstream.


-- 
Petr Viktorin

[Matthew Miller]

On Tue, Feb 07, 2017 at 05:32:59AM -0500, Michal Cyprian wrote:
>  Ruby on Fedora uses /usr/local/share/gems/ for packages installed as
>  a root via gem tool for many years. 
And FWIW, this change was made for Perl, too, back in the dawn of time.
(Possibly in RHL pre-Fedora?)

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Chris Adams]

Once upon a time, Matthew Miller <mattdm(a)fedoraproject.org&gt; said:
>  On Tue, Feb 07, 2017 at 05:32:59AM -0500, Michal Cyprian wrote:
>  > Ruby on Fedora uses /usr/local/share/gems/ for packages installed as
>  > a root via gem tool for many years.
>  
>  And FWIW, this change was made for Perl, too, back in the dawn of time.
>  (Possibly in RHL pre-Fedora?) 
It was later than that - between RHEL 5 (based on FC 6) and RHEL 6
(based on F 12/13).

However, this is another case where sudo having a different PATH than
the normal shell comes into play as an irritant... installing modules
(or gems, etc.) that include executables puts them in /usr/local/bin,
which works from the shell but not sudo.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Matthew Miller]

On Tue, Feb 07, 2017 at 08:40:08AM -0600, Chris Adams wrote:
>  > And FWIW, this change was made for Perl, too, back in the dawn
> of time.
>  > (Possibly in RHL pre-Fedora?)
>  
>  It was later than that - between RHEL 5 (based on FC 6) and RHEL 6
>  (based on F 12/13).
>  
>  However, this is another case where sudo having a different PATH than
>  the normal shell comes into play as an irritant... installing modules
>  (or gems, etc.) that include executables puts them in /usr/local/bin,
>  which works from the shell but not sudo. 
Good thing that sudo path is easily configured. :)

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Mathieu Bridon]

On Tue, 2017-02-07 at 08:40 -0600, Chris Adams wrote:
>  However, this is another case where sudo having a different PATH
> than
>  the normal shell comes into play as an irritant... installing modules
>  (or gems, etc.) that include executables puts them in /usr/local/bin,
>  which works from the shell but not sudo. 
Unfortunately:

https://bugzilla.redhat.com/show_bug.cgi?id=1166185
https://pagure.io/fesco/issue/1646


-- 
Mathieu