Thanks for the detailed response!
To start with your final questions:
- Why is Python itself not installed in the /app prefix?
- Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one
mounted on /usr and one mounted on /app. The image mounted on /usr is
called the "runtime" and contains libraries and other files shared between
different applications. In Fedora, this is made out of standard
distribution packages - glibc, libjpeg, and so forth. It also contains
Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific
application - the application itself and libraries. In Fedora, we create
the /app filesystem by rebuilding packages inside a module that is
configured to have a special RPM macros package installed in the buildroot
-
https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed
at /etc/rpm/macros.flatpak so that it has higher priority than the macros
from /usr/lib/rpm)
(The advantage of this split over the docker layer system is that
applications don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and
less hacky things to make that actual work within the Fedora RPM ecosystem
- in particular for Python, it overrides
%python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is
installed from the same package. This combination worked pretty well for
F35 for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel
out of an RPM build - the particular breakage that triggered this
investigation was the Python bindings for zxing-cpp , but I think it would
apply to pretty much anything that has moved to the %pyproject macros.
On Thu, May 26, 2022 at 4:04 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
[...]
To read what changed in Fedora 36, I suggest reading the release
notes:
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/develope...
And this email thread:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora
36+.
The change in our patching also caused a naughty UX-nightmare:
https://bugzilla.redhat.com/show_bug.cgi?id=2026979
However, when $RPM_BUILD_ROOT is set, this should not matter. As long as
you
call `pip install` from the specfile (how *exactly* are you using pip
install?)
you should be not impacted (much). I am just mentioning this for
completeness,
as I am unaware of how you use pip exactly.
The fact that sysconfig behaves differently when $RPM_BUILD_ROOT is set
wasn't something I was expecting, and explains quite a bit :-)
I have dedicated my time in the upcoming months to try to rework
this
entirely
for Fedora 37+, as it seems I rather screwed this up in the effort to try
to
make it better. The new approach will however also patch sysconfig and not
distutils. Using distutils for anything has no future.
(My new idea is about distro-customizable default sysconfig installation
prefix
value, and I am happy to brag about it more -- we could possibly use it as
a
future-proof solution for /app as well, depending on what you want to
achieve.)
Having some standard way to configure a system by environment variable or
drop in file would definitely be nice ... and sounds like something we
could leverage.
Though I'm also wondering now if we could just change %py_install variants
and %pyproject variants to have --prefix %{_prefix} on the install command
line - that should have no effect for standard builds and make Flatpak
builds work?
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to
False.
RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my
change
that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this
- build your own forked pip package that defaults to this
We definitely have the *ability* to use a stream branch for python3 or
python-pip, and build it in the flatpak-runtime package. But I'd really
prefer to avoid that, since that creates a fork we need to maintain every
time that these packages are updated in Fedora 36, and there's a
possibility of missing security updates.
- set it via a Python module that is imported from
/usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof.
When looking for a permanent solution, you need to avoid distutils
entirely.
A pip with a backported patch (or a rebase) + this approach sounds most
maintainable for Fedora 36. But if we think that the --prefix %{_prefix}
approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36
B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime
module for F36 and reconverge for F37 - I'm less worried about carrying a
stream branch for macros packages, since they are unlikely to have security
fixes.
What do you think?
- Owen