Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
Details:
On Python 3.13+ we only ship one downstream-only patch in Python. That is a patch that is not yet upstreamable, isn't a backport etc.
The patch is this:
https://src.fedoraproject.org/rpms/python3.13/blob/f39/f/00251-change-user-i... (Deliberately linking f39 to make the link stable. Rawhide is currently the same.)
Or as a commit: https://github.com/fedora-python/cpython/commit/ac3a9df5f8
We carry this patch in various forms for 7+ years. It originated with https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
The idea is this:
- sudo pip install etc. installs to /usr/local/lib(64)/python3.X/site-packages - RPM packages are installed in /usr/lib(64)/python3.X/site-packages - RPM installed software ignores the packages in /usr/local (unless they opt in not to) - other software uses packages from both, giving preference to /usr/local
There are several challenges to this approach:
One of them is upgrading. Ideally if pyfoo 1 is installed via RPM, sudo pip install --upgrade 'pyfoo>=2' should install pyfoo to /usr/local while leaving the RPM-installed pyfoo 1 be. In reality, pip is not ready for this and needs a patch as well: https://src.fedoraproject.org/rpms/python-pip/blob/f39/f/remove-existing-dis... This means that users who do sudo pip install --upgrade pip lose this patch. This also means that the fact that this appears to work in other Python package managers (such as uv) is probably purely accidental. This behavior also tends to differ for RPM-packaged Python packages with .dist-info and .egg-info metadata.
Another challenge is the mechanism we decided to use for RPM packages to ignore /usr/local. We use the -s flag in shebangs for this. That leads to problems like this: "global pip installed modules not visible if user site dir is disabled" https://bugzilla.redhat.com/2256190
We could probably solve this by introducing a custom flag instead of -s, but that would make the patch even worse -- our RPM-packaged software would not even execute without our Python with our patch. I don't want to do that.
A result of this problem (which is shared with Debian-based distros) was PEP 668 – Marking Python base environments as “externally managed” https://peps.python.org/pep-0668/ Except we cannot use it as is, because our install locations share a common stdlib location. I described this at https://discuss.python.org/t/pep-668-marking-python-base-environments-as-ext... and further.
At this point, after 2 years of failure, I really don't know how to get something from this that's upstreamable.
I spoke about this at PyCon PL '22 as well: https://www.youtube.com/watch?v=7SBH4PkbMhw
---
Recently it occurred to me that perhaps we could:
1. drop the patches from Python and pip 2. mark /usr/lib(64)/python3.X as externally managed (PEP 668)
And declare sudo-pip-installing unsupported. (That means, it only works with a custom pip flag and if users nuke their systems by using that flag, they shot themelves and we cannot help them.)
Personally, I think this would be the best solution: sudo-pip-installing is bad and should never be done (no, not even in containers).
However, I realize that others might disagree. For them, I can offer pip --break-system-packages.
If we want to do this, I think we need to do this on Python upgrade boundary. We don't want users sudo pip-installing packages with Python 3.13 on Fedora 41 only to find them ignored on Fedora 42. However, when we upgrade Python to 3.14 in Fedora 43, their packages would still be broken anyway.
On Tue, Dec 3, 2024 at 7:02 PM Miro Hrončok via python-devel < python-devel@lists.fedoraproject.org> wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
Details:
On Python 3.13+ we only ship one downstream-only patch in Python. That is a patch that is not yet upstreamable, isn't a backport etc.
The patch is this:
https://src.fedoraproject.org/rpms/python3.13/blob/f39/f/00251-change-user-i... (Deliberately linking f39 to make the link stable. Rawhide is currently the same.)
Or as a commit: https://github.com/fedora-python/cpython/commit/ac3a9df5f8
We carry this patch in various forms for 7+ years. It originated with https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
The idea is this:
- sudo pip install etc. installs to
/usr/local/lib(64)/python3.X/site-packages
- RPM packages are installed in /usr/lib(64)/python3.X/site-packages
- RPM installed software ignores the packages in /usr/local (unless they
opt in not to)
- other software uses packages from both, giving preference to /usr/local
There are several challenges to this approach:
One of them is upgrading. Ideally if pyfoo 1 is installed via RPM, sudo pip install --upgrade 'pyfoo>=2' should install pyfoo to /usr/local while leaving the RPM-installed pyfoo 1 be. In reality, pip is not ready for this and needs a patch as well:
https://src.fedoraproject.org/rpms/python-pip/blob/f39/f/remove-existing-dis... This means that users who do sudo pip install --upgrade pip lose this patch. This also means that the fact that this appears to work in other Python package managers (such as uv) is probably purely accidental. This behavior also tends to differ for RPM-packaged Python packages with .dist-info and .egg-info metadata.
Another challenge is the mechanism we decided to use for RPM packages to ignore /usr/local. We use the -s flag in shebangs for this. That leads to problems like this: "global pip installed modules not visible if user site dir is disabled" https://bugzilla.redhat.com/2256190
We could probably solve this by introducing a custom flag instead of -s, but that would make the patch even worse -- our RPM-packaged software would not even execute without our Python with our patch. I don't want to do that.
A result of this problem (which is shared with Debian-based distros) was PEP 668 – Marking Python base environments as “externally managed” https://peps.python.org/pep-0668/ Except we cannot use it as is, because our install locations share a common stdlib location. I described this at
https://discuss.python.org/t/pep-668-marking-python-base-environments-as-ext... and further.
At this point, after 2 years of failure, I really don't know how to get something from this that's upstreamable.
I spoke about this at PyCon PL '22 as well: https://www.youtube.com/watch?v=7SBH4PkbMhw
Recently it occurred to me that perhaps we could:
- drop the patches from Python and pip
- mark /usr/lib(64)/python3.X as externally managed (PEP 668)
So the implication here is that pip will not be allowed to install a package in the system directories that RPMs are installed in?
And declare sudo-pip-installing unsupported. (That means, it only works
with a custom pip flag and if users nuke their systems by using that flag, they shot themelves and we cannot help them.)
Honestly I don't see this working and I believe that ship sailed a long time ago unfortunately. Way too many tutorials, books and even workshops recommend "sudo pip install" and before our patch we always had a constant stream of bugs filled about users nuking their systems that way.
Personally, I think this would be the best solution:
sudo-pip-installing is bad and should never be done (no, not even in containers).
However, I realize that others might disagree. For them, I can offer pip --break-system-packages.
If we want to do this, I think we need to do this on Python upgrade boundary. We don't want users sudo pip-installing packages with Python 3.13 on Fedora 41 only to find them ignored on Fedora 42. However, when we upgrade Python to 3.14 in Fedora 43, their packages would still be broken anyway.
-- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok
-- _______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 12/4/24 23:28, Charalampos Stratakis via python-devel wrote:
On Tue, Dec 3, 2024 at 7:02 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas. tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead. Details: On Python 3.13+ we only ship one downstream-only patch in Python. That is a patch that is not yet upstreamable, isn't a backport etc. The patch is this: https://src.fedoraproject.org/rpms/python3.13/blob/f39/f/00251-change-user-install-location.patch (Deliberately linking f39 to make the link stable. Rawhide is currently the same.) Or as a commit: https://github.com/fedora-python/cpython/commit/ac3a9df5f8 We carry this patch in various forms for 7+ years. It originated with https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe The idea is this: - sudo pip install etc. installs to /usr/local/lib(64)/python3.X/site-packages - RPM packages are installed in /usr/lib(64)/python3.X/site-packages - RPM installed software ignores the packages in /usr/local (unless they opt in not to) - other software uses packages from both, giving preference to /usr/local There are several challenges to this approach: One of them is upgrading. Ideally if pyfoo 1 is installed via RPM, sudo pip install --upgrade 'pyfoo>=2' should install pyfoo to /usr/local while leaving the RPM-installed pyfoo 1 be. In reality, pip is not ready for this and needs a patch as well: https://src.fedoraproject.org/rpms/python-pip/blob/f39/f/remove-existing-dist-only-if-path-conflicts.patch This means that users who do sudo pip install --upgrade pip lose this patch. This also means that the fact that this appears to work in other Python package managers (such as uv) is probably purely accidental. This behavior also tends to differ for RPM-packaged Python packages with .dist-info and .egg-info metadata. Another challenge is the mechanism we decided to use for RPM packages to ignore /usr/local. We use the -s flag in shebangs for this. That leads to problems like this: "global pip installed modules not visible if user site dir is disabled" https://bugzilla.redhat.com/2256190 We could probably solve this by introducing a custom flag instead of -s, but that would make the patch even worse -- our RPM-packaged software would not even execute without our Python with our patch. I don't want to do that. A result of this problem (which is shared with Debian-based distros) was PEP 668 – Marking Python base environments as “externally managed” https://peps.python.org/pep-0668/ Except we cannot use it as is, because our install locations share a common stdlib location. I described this at https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/46 and further. At this point, after 2 years of failure, I really don't know how to get something from this that's upstreamable. I spoke about this at PyCon PL '22 as well: https://www.youtube.com/watch?v=7SBH4PkbMhw --- Recently it occurred to me that perhaps we could: 1. drop the patches from Python and pip 2. mark /usr/lib(64)/python3.X as externally managed (PEP 668)So the implication here is that pip will not be allowed to install a package in the system directories that RPMs are installed in?
And declare sudo-pip-installing unsupported. (That means, it only works with a custom pip flag and if users nuke their systems by using that flag, they shot themelves and we cannot help them.)Honestly I don't see this working and I believe that ship sailed a long time ago unfortunately. Way too many tutorials, books and even workshops recommend "sudo pip install" and before our patch we always had a constant stream of bugs filled about users nuking their systems that way.
Hi all, I agree with that. This is a righteous fight, but one we're going to lose. If we "break" `sudo pip install`, we're going to break a lot of people's (maybe wrong) use cases, gonna get a lot of angry and/or confused users, and Fedora might not be seen as loving Python so much anymore.
On top of that, I really worry that if our containers don't support sudo pip, we're going to lose users, because our competitors do support this common use case. I also talked to the PM for Python in RHEL, and he believes sudo pip must work in RHEL containers (that is of course not binding for this Fedora discussion though).
I fully agree that the status quo is not great. But I'm not convinced this change would make it better, especially not from the users' perspective.
One thing that comes to my mind is that we don't really need any safe guards in containers, because if you break a container with sudo pip, you just restart it and fix your install script. So perhaps, we could mark Python base environments as externally managed only in live installs, and keep it free in containers, which would solve part of the problem. On the other hand, it's not great to have a different container environment, and could lead to unforeseen issues.
Tomas
Personally, I think this would be the best solution: sudo-pip-installing is bad and should never be done (no, not even in containers). However, I realize that others might disagree. For them, I can offer pip --break-system-packages. If we want to do this, I think we need to do this on Python upgrade boundary. We don't want users sudo pip-installing packages with Python 3.13 on Fedora 41 only to find them ignored on Fedora 42. However, when we upgrade Python to 3.14 in Fedora 43, their packages would still be broken anyway. -- Miro Hrončok -- Phone: +420777974800 Fedora Matrix: mhroncok -- _______________________________________________ python-devel mailing list -- python-devel@lists.fedoraproject.org To unsubscribe send an email to python-devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue-- Regards,
Charalampos Stratakis Senior Software Engineer Python Maintenance Team, Red Hat
On Tue, Dec 3, 2024 at 7:03 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
We shouldn't cater to ancient / broken tutorials and / or use cases that have always been "wrong" for all eternity, IMO. virtualenvs have existed in some shape or form for almost two decades (the venv module was added in Python 3.3 - in 2012!) - people should just use those.
So I say "do it". If you make this a Change proposal, I'd vote +1 on it.
Fabio
On Fri, Dec 6, 2024 at 12:04 PM Fabio Valentini via python-devel python-devel@lists.fedoraproject.org wrote:
On Tue, Dec 3, 2024 at 7:03 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
We shouldn't cater to ancient / broken tutorials and / or use cases that have always been "wrong" for all eternity, IMO. virtualenvs have existed in some shape or form for almost two decades (the venv module was added in Python 3.3 - in 2012!) - people should just use those.
So I say "do it". If you make this a Change proposal, I'd vote +1 on it.
Another option would be to shift our stuff out of site-packages and into a new directory (dist-packages like debian, maybe?). If RPM packaged stuff went there and everything else went into site-packages, that would effectively retain the same spirit as what we do now.
On 06. 12. 24 18:31, Neal Gompa via python-devel wrote:
On Fri, Dec 6, 2024 at 12:04 PM Fabio Valentini via python-devel python-devel@lists.fedoraproject.org wrote:
On Tue, Dec 3, 2024 at 7:03 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
We shouldn't cater to ancient / broken tutorials and / or use cases that have always been "wrong" for all eternity, IMO. virtualenvs have existed in some shape or form for almost two decades (the venv module was added in Python 3.3 - in 2012!) - people should just use those.
So I say "do it". If you make this a Change proposal, I'd vote +1 on it.
Another option would be to shift our stuff out of site-packages and into a new directory (dist-packages like debian, maybe?). If RPM packaged stuff went there and everything else went into site-packages, that would effectively retain the same spirit as what we do now.
Yet it would not solve any of the problems.
On Fri, Dec 6, 2024 at 12:48 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
On 06. 12. 24 18:31, Neal Gompa via python-devel wrote:
On Fri, Dec 6, 2024 at 12:04 PM Fabio Valentini via python-devel python-devel@lists.fedoraproject.org wrote:
On Tue, Dec 3, 2024 at 7:03 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
We shouldn't cater to ancient / broken tutorials and / or use cases that have always been "wrong" for all eternity, IMO. virtualenvs have existed in some shape or form for almost two decades (the venv module was added in Python 3.3 - in 2012!) - people should just use those.
So I say "do it". If you make this a Change proposal, I'd vote +1 on it.
Another option would be to shift our stuff out of site-packages and into a new directory (dist-packages like debian, maybe?). If RPM packaged stuff went there and everything else went into site-packages, that would effectively retain the same spirit as what we do now.
Yet it would not solve any of the problems.
I thought EXTERNALLY-MANAGED works off the install scheme? If one scheme is locked but the other isn't, it should be fine, right?
On 06. 12. 24 18:56, Neal Gompa wrote:
On Fri, Dec 6, 2024 at 12:48 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
On 06. 12. 24 18:31, Neal Gompa via python-devel wrote:
On Fri, Dec 6, 2024 at 12:04 PM Fabio Valentini via python-devel python-devel@lists.fedoraproject.org wrote:
On Tue, Dec 3, 2024 at 7:03 PM Miro Hrončok via python-devel python-devel@lists.fedoraproject.org wrote:
Hello Pythonistas.
tl;dr I wonder if we should get rid of the last downstream-only patch in Fedora's Python, the one responsible for /usr/local/lib(64)/python... installation location. The removal would bring us closer to upstream, but perhaps cause a regression for our users. I propose to adapt PEP 668 (Marking Python base environments as “externally managed”) instead.
We shouldn't cater to ancient / broken tutorials and / or use cases that have always been "wrong" for all eternity, IMO. virtualenvs have existed in some shape or form for almost two decades (the venv module was added in Python 3.3 - in 2012!) - people should just use those.
So I say "do it". If you make this a Change proposal, I'd vote +1 on it.
Another option would be to shift our stuff out of site-packages and into a new directory (dist-packages like debian, maybe?). If RPM packaged stuff went there and everything else went into site-packages, that would effectively retain the same spirit as what we do now.
Yet it would not solve any of the problems.
I thought EXTERNALLY-MANAGED works off the install scheme? If one scheme is locked but the other isn't, it should be fine, right?
Yes, but we still hit all of the problems described in my original email.
python-devel@lists.fedoraproject.org
-
Charalampos Stratakis -
Fabio Valentini -
Miro Hrončok -
Neal Gompa -
Tomáš Orsava