F37 proposal: Python: Add -P to default shebangs (System-Wide Change proposal)

[Ben Cotton]

https://fedoraproject.org/wiki/Changes/PythonSafePath

== Summary ==
The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P `-P`
flag] will be added to the Python shebang macros
(`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that
adhere to those macros will change their Python shebangs from `#!
/usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result,
will no longer have the directory of the script (such as `/usr/bin`)
in `sys.path`. An opt-out mechanism exists.

== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]], [[User:Vstinner|Victor Stinner]]
* Email: python-maint(a)redhat.com

== Detailed Description ==
All Python 3 shebang RPM macros will be changed to contain one more
flag: `-P`. Previously, they contained `-s`, now they will contain
`-sP`.

From the [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
documentation for the `-P` option]:

:Don’t prepend a potentially unsafe path to `sys.path`:
:
:* `python -m module` command line: Don’t prepend the current working directory.
:* `python script.py` command line: Don’t prepend the script’s
directory. If it’s a symbolic link, resolve symbolic links.
:* `python -c code` and `python` (REPL) command lines: Don’t prepend
an empty string, which means the current working directory.

In shebangs, only the middle option (''don’t prepend the script’s
directory'') is relevant.

Consider the following executbale script installed as
`/usr/bin/let-there-be-fun`:

 #! /usr/bin/python3 -s
 import abc
 ...

When the script is directly executed (e.g. by running
`let-there-be-fun` from the console), the script's directory
(`/usr/bin`) is prepended to `sys.path`. Python tries to locate an
importable `abc` module in `/usr/bin` first. This can cause real
issues: [https://bugzilla.redhat.com/2057340 python3-notebook:
ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six].

When the shebang includes `-P`:

 #! /usr/bin/python3 -sP
 import abc
 ...

...the script's directory (`/usr/bin`) is '''not''' prepended
to
`sys.path`. The change owners consider this approach safer for the
majority of Fedora's RPM packages.

By default, '''all standardly RPM-packaged Python packages with
scripts in `/usr/bin` will gain the `-P` flag in their shebang''',
assuming the software is packaged in a way that respects the Python
shebang RPM macros (see below for opt-out and explicit opt-in
mechanisms). Due to the variety of ways such scripts can be
created/packaged, there will likely be packages that will not be
affected by the change automatically. (In other words, the change is
applied on RPM macro level, no added mechanics to force the flag, such
as BRP scripts, are planned as part of this change.)

=== List of RPM macros that will gain `-P` ===

* `%{py3_shbang_opts}`
* `%{py3_shbang_opts_nodash}`
* `%{py3_shebang_flags}`
* `%{py_shbang_opts}`
* `%{py_shbang_opts_nodash}`
* `%{py_shebang_flags}`

=== Opting out ===

If the new behavior is not desirable to your package amend the macros
(e.g. with `sed`) to remove the `P` flag.

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, use:

 # Don't add -P to Python shebang
 # This package only works when /usr/bin is in sys.path (use your own
rationale here)
 %global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, use:

 # Don't add -P to Python shebang
 # This package only works when /usr/bin is in sys.path (use your own
rationale here)
 %global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

=== Opting in ===

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
current Python packaging guidelines], e.g. `%pyproject_wheel` and
`%pyproject_install`, the standard set of Python shebang flags is
applied to all files with Python shebangs installed in `/usr/bin/`.

If you use the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python packaging guidelines], e.g. `%py3_build` and
`%py3_install`, the standard set of Python shebang flags might be
applied to some files and not applied to others depending on the exact
structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang
flags on a certain file that is not handled automatically, use
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#py3_she...
the `%py3_shebang_fix` macro].

=== What if the packager changes `%__python3` to an older version of Python ===

The `-P` flag was introduced in Python 3.11. When `%__python3` is
redefined to an older Python version, e.g. `/usr/bin/python3.10`,
including the `-P` flag in shebangs would break the scripts. Hence,
the flag will be included conditionally, presumably somehow like this:

 <nowiki>%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P'
if hasattr(sys.flags, 'safe_path') else '')")</nowiki>

=== What if the admin/user changes `/usr/bin/python3` to an older
version of Python ===

The `-P` flag was introduced in Python 3.11. When an admin/user
changes `/usr/bin/python3` to point to an older version of Python,
e.g. `/usr/bin/python3.10`, including the `-P` flag in shebangs would
break the scripts.

However, changing `/usr/bin/python3` to a different Python would
''brick'' a Fedora system even now. So we don't consider that an
issue. See an example that changes `/usr/bin/python3` to Python 3.9
(don't try this at home):

 [root@086a2804411a /]# head -n1 /usr/bin/dnf
 #!/usr/bin/python3

 [root@086a2804411a /]# dnf --version
 4.12.0
 ...

 [root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

 [root@086a2804411a /]# dnf --version
 Traceback (most recent call last):
   File "/usr/bin/dnf", line 61, in <module>
     from dnf.cli import main
 ModuleNotFoundError: No module named 'dnf'

=== Risks ===

As with any other change, there is a risk that this will break things.
The change owners plan to test the change extensively via Copr before
they deploy the change in Rawhide. If things go badly, they are
prepared to delay or cancel the change.

The [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/
201x-era Python RPM macros] set the shebang flags by a weird hack. As
a result, it is not well-defined what scripts will be affected by this
change. The change owners are aware that:

# not all Python scripts in `/usr/bin` will have the `-P` flag automatically
# some scripts '''not''' in `/usr/bin` might gain the `-P` flag as
well

The first point is an acceptable gradual deployment of the default
flag. The second point is not very dangerous because we don't except
users to directly execute Python scripts via shebangs when such
scripts are not in `$PATH`. If a problematic package is found, it can
opt-out easily. If this causes too much friction, we will only change
the flag used in the `%pyproject_install` macro, leaving packages with
the "legacy" macros intact.

== Feedback ==
* Relevant upstream discussion: [https://discuss.python.org/t/13896
Should console_scripts entry points exclude the scripts directory from
sys.path?]
* Adding `-P` to Python:
[https://github.com/python/cpython/issues/57684 GitHub issue],
[https://mail.python.org/archives/list/[e-mail]/thread/IU5Q2A...
email thread]
* [https://github.com/rpm-software-management/dnf/pull/1815#pullrequestrevie...
dnf maintainers response for doing this explicitly in dnf first]

Generally, people seem to think that this is a good thing. They are
afraid to change the default behavior of Python but welcome using this
flag for system-installed scripts.

== Benefit to Fedora ==
Python programs in `/usr/bin` will be less fragile to other random
files being present in `/usr/bin`.
Real hard-to-debug issues like [https://bugzilla.redhat.com/2057340
python3-notebook: ImportError: bad magic number in six] and
[https://github.com/benjaminp/six/issues/359 bad magic number in six]
will not happen.

== Scope ==
* Proposal owners:
** Test everything in Copr
** If everything works, change the flags either before the Python 3.11
rebuild or before the Fedora 37 Mass Rebuild
** Provide guidance to packagers, fix bugs if needed

* Other developers:
** Observe their packages, find and report bugs, opt-out if needed
** Volunteerily opt-in by calling the `%py3_shebang_fix` macro and/or
by converting their packages to `%pyproject_install`

* Release engineering: [https://pagure.io/releng/issue/10784 #10784]
* Policies and guidelines: The new flag needs to be documented in the
Python packaging guidelines (old and new)
* Trademark approval: not needed for this Change
* Alignment with Objectives: no


== Upgrade/compatibility impact ==
No impact is anticipated.

== How To Test ==
* Low level: Examine the value of the changed RPM macros, it should
contain the `-P` flag
* Middle level: Examine the shebang lines of RPM-installed Python
scripts in `/usr/bin`, it should contain the `-P` flag
* High level: Tests that RPM-installed Python scripts still behave as
expected but don't try to import stuff from `/usr/bin`

== User Experience ==
Users of RPM-installed scripts should get a safer experience by default.
Users of Python should not observe a difference, the behavior is not a
new default: the `-P` flag needs to be explicitly used.

== Dependencies ==
We need [[Changes/Python3.11]] first.

== Contingency Plan ==
* Contingency mechanism: defer to F38; only add `-P` to shebangs in
`%pyproject_install` to keep backward compatibility of the old macros;
revert and rebuild
* Contingency deadline: 1 week before the beta freeze
* Blocks release? No

== Documentation ==
* This page
* TBD Updated Python guidelines
* https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
* https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights
(Security improvements)

== Release Notes ==
TBD


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

[Robbie Harwood]

Ben Cotton <bcotton(a)redhat.com&gt; writes:

>  :Don’t prepend a potentially unsafe path to `sys.path`: 
If this is a safety/security issue, why not just make it the default for
python itself?

Be well,
--Robbie

[Zbigniew Jędrzejewski-Szmek]

On Wed, May 11, 2022 at 10:24:17AM -0400, Robbie Harwood wrote:
>  Ben Cotton <bcotton(a)redhat.com&gt; writes:
>  
>  > :Don’t prepend a potentially unsafe path to `sys.path`:
>  
>  If this is a safety/security issue, why not just make it the default for
>  python itself? 
Yeah, I agree. I think Python upstream should own up to the fact that
adding '.' to sys.path was always a mistake.

Just ask a random user: is

  echo 'import sys; print(sys.version)' >/tmp/test.py
  python /tmp/test.py

safe to execute on a multi-user system?

Zbyszek

P.S. If we can't get the proper fix, this Change proposal is better
than nothing.  So I'll vote +1 on the proposal. But I think we can do
better.

[Barry]

>  On 11 May 2022, at 15:25, Robbie Harwood <rharwood(a)redhat.com&gt;
> wrote:
>  
>  Ben Cotton <bcotton(a)redhat.com&gt; writes:
>  
> > :Don’t prepend a potentially unsafe path to `sys.path`:
>  
>  If this is a safety/security issue, why not just make it the default for
>  python itself? 
It will break normal user use of python.

There is an expectation that if I have a.py and b.py in
the current directory then this must work.

In a.py:
import b

Then I can just do:

$ python a.py

Barry

>  
>  Be well,
>  --Robbie
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Daniel P. Berrangé]

On Wed, May 11, 2022 at 10:24:17AM -0400, Robbie Harwood wrote:
>  Ben Cotton <bcotton(a)redhat.com&gt; writes:
>  
>  > :Don’t prepend a potentially unsafe path to `sys.path`:
>  
>  If this is a safety/security issue, why not just make it the default for
>  python itself? 
I presume that approach is considered too disruptive to users.
I know I'm running python apps which relying on './someapp' being
able to import modules under './'. Typically this is where I've
checked out $random git repo and don't want to actually run
a full install of it, instead just run straight from git, so I
can switch branches at will. It would be pretty annoying if this
broke, despite the understandable security benefit.

This proposal at least gets the security benefits for all system
shipped stuff, without breaking anything the user has been using
from non-packaged locations.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[Chris Adams]

Once upon a time, Zbigniew Jędrzejewski-Szmek <zbyszek(a)in.waw.pl&gt; said:
>  Yeah, I agree. I think Python upstream should own up to the fact
> that
>  adding '.' to sys.path was always a mistake. 
Yeah, perl bit that bullet a while ago now, dropping '.' from @INC.
It's really the only sane default.
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Miro Hrončok]

On 11. 05. 22 16:24, Robbie Harwood wrote:
>  Ben Cotton <bcotton(a)redhat.com&gt; writes:
>  
> > :Don’t prepend a potentially unsafe path to `sys.path`:
>  
>  If this is a safety/security issue, why not just make it the default for
>  python itself? 
I would like that, but -P is what we have now. If we pioneer this in Fedora, 
maybe we can convince upstream to make that the default, however, 
backwards-compatibility matters to many people and while I would appreciate 
such change, I don't really want to to drive it. Victor (the second change 
owner) might want to persuade that path, but it ain't gonna happen in Python 3.11.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 11. 05. 22 18:37, Daniel P. Berrangé wrote:
>  On Wed, May 11, 2022 at 10:24:17AM -0400, Robbie Harwood wrote:
> > Ben Cotton <bcotton(a)redhat.com&gt; writes:
> >
> >> :Don’t prepend a potentially unsafe path to `sys.path`:
> >
> > If this is a safety/security issue, why not just make it the default for
> > python itself?
>  
>  I presume that approach is considered too disruptive to users.
>  I know I'm running python apps which relying on './someapp' being
>  able to import modules under './'. Typically this is where I've
>  checked out $random git repo and don't want to actually run
>  a full install of it, instead just run straight from git, so I
>  can switch branches at will. It would be pretty annoying if this
>  broke, despite the understandable security benefit.
>  
>  This proposal at least gets the security benefits for all system
>  shipped stuff, without breaking anything the user has been using
>  from non-packaged locations. 
I've added this to https://fedoraproject.org/wiki/Changes/PythonSafePath#Feeedback

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Vít Ondruch]

Dne 11. 05. 22 v 20:45 Chris Adams napsal(a):
>  Once upon a time, Zbigniew Jędrzejewski-Szmek
> <zbyszek(a)in.waw.pl&gt; said:
> > Yeah, I agree. I think Python upstream should own up to the fact that
> > adding '.' to sys.path was always a mistake.
>  Yeah, perl bit that bullet a while ago now, dropping '.' from @INC.
>  It's really the only sane default.
>  

The same change happened for Ruby around Ruby 2.0, i.e. ~9 years ago.


Vít

[Victor Stinner]

Changing the default in Python was discussed multiple times over the
last 20 years. Every time, it was said that it's convenient and it's
not worth it to break use cases to increase the default security.

Python 3.4 already has -I (isolated mode) which imply Python 3.11 -P
option, but it also implies other options like -E which ignores
environment variables, and so cannot be used for Fedora use cases
(shebangs).

I'm thinking about proposing a PEP to change the defaut in Python
3.12, but I'm not confident that it would be accepted. It's more to
write down the rationale once ;-)

Victor

On Wed, May 11, 2022 at 10:32 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 11. 05. 22 16:24, Robbie Harwood wrote:
>  > Ben Cotton <bcotton(a)redhat.com&gt; writes:
>  >
>  >> :Don’t prepend a potentially unsafe path to `sys.path`:
>  >
>  > If this is a safety/security issue, why not just make it the default for
>  > python itself?
>
>  I would like that, but -P is what we have now. If we pioneer this in Fedora,
>  maybe we can convince upstream to make that the default, however,
>  backwards-compatibility matters to many people and while I would appreciate
>  such change, I don't really want to to drive it. Victor (the second change
>  owner) might want to persuade that path, but it ain't gonna happen in Python 3.11.
>
>  --
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>

[Miro Hrončok]

On 11. 05. 22 15:27, Ben Cotton wrote:
>  https://fedoraproject.org/wiki/Changes/PythonSafePath
>  
>  == Summary ==
>  The [https://docs.python.org/3.11/using/cmdline.html#cmdoption-P  `-P`
>  flag] will be added to the Python shebang macros
>  (`%{py3_shbang_opts}`, `%{py3_shebang_flags}`, ...). Packages that
>  adhere to those macros will change their Python shebangs from `#!
>  /usr/bin/python3 -s` to `#! /usr/bin/python3 -sP` and as a result,
>  will no longer have the directory of the script (such as `/usr/bin`)
>  in `sys.path`. An opt-out mechanism exists. 
python-rpm-macros-3.11-3.fc37 with this change just landed in Rawhide in time 
for the mass rebuild.

We have identified one package that needed to opt-out and one package that 
needed a workaround:

https://src.fedoraproject.org/rpms/llvm/pull-request/146
https://src.fedoraproject.org/rpms/python-sphinxcontrib-programoutput/pul...

To opt-out, if needed, see:

https://fedoraproject.org/wiki/Changes/PythonSafePath#Opting_out

The list of packages that will have different shebangs is at:

https://fedoraproject.org/wiki/Changes/PythonSafePath#List_of_source_RPM_...

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok