[ At Miro's request, resending this to python-devel so the discussion can be public ]
Hi Miro -
When rebuilding a package to include in a Flatpak, we want to install *everything* under prefix=/app - that includes Python modules. (Paths will be adjusted properly when running the Flatpak.)
For years, the way we did that is by installing a /usr/lib64/python3.10/distutils/distutils.cfg: === [install] prefix=/app ===
This is part of the flatpak-rpm-macros package that gets installed in the buildroot for Flatpak rebuilds
In Fedora 36, this no longer works for 'pip install' - it seems that pip unconditionally uses 'sysconfig' rather than 'distutils' to determine install paths for Python >= 3.10.
I see that you filed https://github.com/pypa/pip/issues/10647 which landed in pip for pip-22, but Fedora 36 / pip-21 this isn't present? And in any case, that seems like it's something for 'sudo pip install' rather than package installation. I actually don't really follow how the Python macros are getting things installed into /usr/lib rather than /usr/local/lib.
Any advice about how we can make the installation into /app work? Hopefully in a future-proof way...
Thanks!!! Owen
On Thu, May 26, 2022 at 12:10 AM Owen Taylor otaylor@redhat.com wrote:
[ At Miro's request, resending this to python-devel so the discussion can be public ]
Hi Miro -
When rebuilding a package to include in a Flatpak, we want to install *everything* under prefix=/app - that includes Python modules. (Paths will be adjusted properly when running the Flatpak.)
For years, the way we did that is by installing a /usr/lib64/python3.10/distutils/distutils.cfg:
[install] prefix=/app ===
This is part of the flatpak-rpm-macros package that gets installed in the buildroot for Flatpak rebuilds
In Fedora 36, this no longer works for 'pip install' - it seems that pip unconditionally uses 'sysconfig' rather than 'distutils' to determine install paths for Python >= 3.10.
I see that you filed https://github.com/pypa/pip/issues/10647 which landed in pip for pip-22, but Fedora 36 / pip-21 this isn't present? And in any case, that seems like it's something for 'sudo pip install' rather than package installation. I actually don't really follow how the Python macros are getting things installed into /usr/lib rather than /usr/local/lib.
Any advice about how we can make the installation into /app work? Hopefully in a future-proof way...
What's stopping us from upgrading pip in F36? It's packaged separately from Python itself already...
On 26. 05. 22 6:27, Neal Gompa wrote:
On Thu, May 26, 2022 at 12:10 AM Owen Taylor otaylor@redhat.com wrote:
[ At Miro's request, resending this to python-devel so the discussion can be public ]
Hi Miro -
When rebuilding a package to include in a Flatpak, we want to install *everything* under prefix=/app - that includes Python modules. (Paths will be adjusted properly when running the Flatpak.)
For years, the way we did that is by installing a /usr/lib64/python3.10/distutils/distutils.cfg:
[install] prefix=/app ===
This is part of the flatpak-rpm-macros package that gets installed in the buildroot for Flatpak rebuilds
In Fedora 36, this no longer works for 'pip install' - it seems that pip unconditionally uses 'sysconfig' rather than 'distutils' to determine install paths for Python >= 3.10.
I see that you filed https://github.com/pypa/pip/issues/10647 which landed in pip for pip-22, but Fedora 36 / pip-21 this isn't present? And in any case, that seems like it's something for 'sudo pip install' rather than package installation. I actually don't really follow how the Python macros are getting things installed into /usr/lib rather than /usr/local/lib.
Any advice about how we can make the installation into /app work? Hopefully in a future-proof way...
What's stopping us from upgrading pip in F36? It's packaged separately from Python itself already...
Well, technically probably the *Updates policy*, but we could carefully evaluate the impact of such upgrade and eventually decide to do it.
However, it would make no difference standalone (see my longer previous reply to Owen).
On 26. 05. 22 6:10, Owen Taylor wrote:
[ At Miro's request, resending this to python-devel so the discussion can be public ]
Hi Miro -
Hey Owen.
[the same here]
When rebuilding a package to include in a Flatpak, we want to install *everything* under prefix=/app - that includes Python modules. (Paths will be adjusted properly when running the Flatpak.)
For years, the way we did that is by installing a /usr/lib64/python3.10/distutils/distutils.cfg: === [install] prefix=/app ===
This is part of the flatpak-rpm-macros package that gets installed in the buildroot for Flatpak rebuilds
In Fedora 36, this no longer works for 'pip install' - it seems that pip unconditionally uses 'sysconfig' rather than 'distutils' to determine install paths for Python >= 3.10.
I see that you filed https://github.com/pypa/pip/issues/10647 https://github.com/pypa/pip/issues/10647 which landed in pip for pip-22, but Fedora 36 / pip-21 this isn't present? And in any case, that seems like it's something for 'sudo pip install' rather than package installation. I actually don't really follow how the Python macros are getting things installed into /usr/lib rather than /usr/local/lib.
Any advice about how we can make the installation into /app work? Hopefully in a future-proof way...
Since pip 21.3, pip uses sysconfig rather than distutils (but only on Python 3.10+).
As we used to patch distutils in Fedora 35, we needed pip to use distutils there. When pip made this change it was too late for us to change our patches significantly to patch sysconfig instead. I've opened the issue you linked that allowed us to *delay* this pip change in Fedora 35. And we did. See https://src.fedoraproject.org/rpms/python3.10/blob/f35/f/00251-change-user-i... - it has: diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py +_PIP_USE_SYSCONFIG = False
Note that this was aimed purely for pip-installed or pip-upgraded pip (users tend to do that). The RPM-installed pip in Fedora 35 was < 21.3 so it was not affected at all and uses distutils.
As the Python ecosystem moves away from distutils we needed to stop patching it and instead, reworked our patching significantly to patch syconfig instead. This is needed because distutils is going to be removed and because pip won't support our hack forever.
To read what changed in Fedora 36, I suggest reading the release notes:
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/...
And this email thread:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora 36+.
The change in our patching also caused a naughty UX-nightmare: https://bugzilla.redhat.com/show_bug.cgi?id=2026979
However, when $RPM_BUILD_ROOT is set, this should not matter. As long as you call `pip install` from the specfile (how *exactly* are you using pip install?) you should be not impacted (much). I am just mentioning this for completeness, as I am unaware of how you use pip exactly.
I have dedicated my time in the upcoming months to try to rework this entirely for Fedora 37+, as it seems I rather screwed this up in the effort to try to make it better. The new approach will however also patch sysconfig and not distutils. Using distutils for anything has no future.
(My new idea is about distro-customizable default sysconfig installation prefix value, and I am happy to brag about it more -- we could possibly use it as a future-proof solution for /app as well, depending on what you want to achieve.)
------
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to False. RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this - build your own forked pip package that defaults to this - set it via a Python module that is imported from /usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof. When looking for a permanent solution, you need to avoid distutils entirely.
To be able to suggest alternatives, I need to understand what it is that you are doing.
- Why is Python itself not installed in the /app prefix? - Where and how is pip install invoked?
Thanks for the detailed response!
To start with your final questions:
- Why is Python itself not installed in the /app prefix?
- Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one mounted on /usr and one mounted on /app. The image mounted on /usr is called the "runtime" and contains libraries and other files shared between different applications. In Fedora, this is made out of standard distribution packages - glibc, libjpeg, and so forth. It also contains Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific application - the application itself and libraries. In Fedora, we create the /app filesystem by rebuilding packages inside a module that is configured to have a special RPM macros package installed in the buildroot - https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed at /etc/rpm/macros.flatpak so that it has higher priority than the macros from /usr/lib/rpm)
(The advantage of this split over the docker layer system is that applications don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and less hacky things to make that actual work within the Fedora RPM ecosystem - in particular for Python, it overrides %python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is installed from the same package. This combination worked pretty well for F35 for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel out of an RPM build - the particular breakage that triggered this investigation was the Python bindings for zxing-cpp , but I think it would apply to pretty much anything that has moved to the %pyproject macros.
On Thu, May 26, 2022 at 4:04 AM Miro Hrončok mhroncok@redhat.com wrote: [...]
To read what changed in Fedora 36, I suggest reading the release notes:
https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/...
And this email thread:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora 36+.
The change in our patching also caused a naughty UX-nightmare: https://bugzilla.redhat.com/show_bug.cgi?id=2026979
However, when $RPM_BUILD_ROOT is set, this should not matter. As long as you call `pip install` from the specfile (how *exactly* are you using pip install?) you should be not impacted (much). I am just mentioning this for completeness, as I am unaware of how you use pip exactly.
The fact that sysconfig behaves differently when $RPM_BUILD_ROOT is set wasn't something I was expecting, and explains quite a bit :-)
I have dedicated my time in the upcoming months to try to rework this entirely for Fedora 37+, as it seems I rather screwed this up in the effort to try to make it better. The new approach will however also patch sysconfig and not distutils. Using distutils for anything has no future.
(My new idea is about distro-customizable default sysconfig installation prefix value, and I am happy to brag about it more -- we could possibly use it as a future-proof solution for /app as well, depending on what you want to achieve.)
Having some standard way to configure a system by environment variable or drop in file would definitely be nice ... and sounds like something we could leverage.
Though I'm also wondering now if we could just change %py_install variants and %pyproject variants to have --prefix %{_prefix} on the install command line - that should have no effect for standard builds and make Flatpak builds work?
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to
False. RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change that reads this value to make it work (I have no problem doing that).
Those are the ideas that come to my mind about how to set that attribute:
- build your own forked Python package that sets this
- build your own forked pip package that defaults to this
We definitely have the *ability* to use a stream branch for python3 or python-pip, and build it in the flatpak-runtime package. But I'd really prefer to avoid that, since that creates a fork we need to maintain every time that these packages are updated in Fedora 36, and there's a possibility of missing security updates.
- set it via a Python module that is imported from /usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth
However, forcing pip to use distutils is the opposite of future-proof. When looking for a permanent solution, you need to avoid distutils entirely.
A pip with a backported patch (or a rebase) + this approach sounds most maintainable for Fedora 36. But if we think that the --prefix %{_prefix} approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36 B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime module for F36 and reconverge for F37 - I'm less worried about carrying a stream branch for macros packages, since they are unlikely to have security fixes.
What do you think?
- Owen
On 26. 05. 22 17:13, Owen Taylor wrote:
Thanks for the detailed response!
To start with your final questions:
- Why is Python itself not installed in the /app prefix? - Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one mounted on /usr and one mounted on /app. The image mounted on /usr is called the "runtime" and contains libraries and other files shared between different applications. In Fedora, this is made out of standard distribution packages - glibc, libjpeg, and so forth. It also contains Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific application - the application itself and libraries. In Fedora, we create the /app filesystem by rebuilding packages inside a module that is configured to have a special RPM macros package installed in the buildroot - https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed at /etc/rpm/macros.flatpak so that it has higher priority than the macros from /usr/lib/rpm)
(The advantage of this split over the docker layer system is that applications don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and less hacky things to make that actual work within the Fedora RPM ecosystem - in particular for Python, it overrides %python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is installed from the same package. This combination worked pretty well for F35 for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel out of an RPM build - the particular breakage that triggered this investigation was the Python bindings for zxing-cpp , but I think it would apply to pretty much anything that has moved to the %pyproject macros.
In that case I suppose we could change those macros to pass --prefix %{_prefix} explicitly. [I thought this before reading the rest of your emial, keeping it here.]
To read what changed in Fedora 36, I suggest reading the release notes: https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme <https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/developers/Development_Python/#sect-install_scheme> And this email thread: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/ <https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/AAGUFQZ4RZDU7KUN4HA43KQJCMSFR3GW/> When we did this, we have dropped sysconfig._PIP_USE_SYSCONFIG in Fedora 36+. The change in our patching also caused a naughty UX-nightmare: https://bugzilla.redhat.com/show_bug.cgi?id=2026979 <https://bugzilla.redhat.com/show_bug.cgi?id=2026979> However, when $RPM_BUILD_ROOT is set, this should not matter. As long as you call `pip install` from the specfile (how *exactly* are you using pip install?) you should be not impacted (much). I am just mentioning this for completeness, as I am unaware of how you use pip exactly.The fact that sysconfig behaves differently when $RPM_BUILD_ROOT is set wasn't something I was expecting, and explains quite a bit :-)
I have dedicated my time in the upcoming months to try to rework this entirely for Fedora 37+, as it seems I rather screwed this up in the effort to try to make it better. The new approach will however also patch sysconfig and not distutils. Using distutils for anything has no future. (My new idea is about distro-customizable default sysconfig installation prefix value, and I am happy to brag about it more -- we could possibly use it as a future-proof solution for /app as well, depending on what you want to achieve.)Having some standard way to configure a system by environment variable or drop in file would definitely be nice ... and sounds like something we could leverage.
Awesome.
Though I'm also wondering now if we could just change %py_install variants and %pyproject variants to have --prefix %{_prefix} on the install command line - that should have no effect for standard builds and make Flatpak builds work?
Now I see you are in agreement with me previous thought. That should indeed work.
As a hotfix, you could probably set sysconfig._PIP_USE_SYSCONFIG back to False. RPM-packaged pip in Fedora 36 is 21.3 so we would need to backport my change that reads this value to make it work (I have no problem doing that). Those are the ideas that come to my mind about how to set that attribute: - build your own forked Python package that sets this - build your own forked pip package that defaults to thisWe definitely have the *ability* to use a stream branch for python3 or python-pip, and build it in the flatpak-runtime package. But I'd really prefer to avoid that, since that creates a fork we need to maintain every time that these packages are updated in Fedora 36, and there's a possibility of missing security updates.
Makes sense. I kinda suspected that, that's why I proposed the next option.
- set it via a Python module that is imported from /usr/lib/python3.10/site-packages/_pip_use_sysconfig.pth However, forcing pip to use distutils is the opposite of future-proof. When looking for a permanent solution, you need to avoid distutils entirely.A pip with a backported patch (or a rebase) + this approach sounds most maintainable for Fedora 36. But if we think that the --prefix %{_prefix} approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36 B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime module for F36 and reconverge for F37 - I'm less worried about carrying a stream branch for macros packages, since they are unlikely to have security fixes.
What do you think?
Definitively A. There is no reason to fork the macros, the change should be perfectly backwards compatible.
The lines are:
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.pytho... https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.pytho... https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macro...
On Thu, May 26, 2022 at 1:07 PM Miro Hrončok mhroncok@redhat.com wrote:
On 26. 05. 22 17:13, Owen Taylor wrote:
A pip with a backported patch (or a rebase) + this approach sounds most maintainable for Fedora 36. But if we think that the --prefix %{_prefix} approach actually works, then perhaps we can either:
A) Just update python-rpm-macros and pygobject-rpm-macros in F36 B) Build a stream-branch (or even the f37 branch) in the
flatpak-runtime
module for F36 and reconverge for F37 - I'm less worried about carrying
a
stream branch for macros packages, since they are unlikely to have
security fixes.
What do you think?
Definitively A. There is no reason to fork the macros, the change should be perfectly backwards compatible.
The lines are:
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.pytho...
https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.pytho...
https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macro...
OK, I'll work on coming up with some patches. Do you have an opinion about whether it makes sense to make the %_prefix support complete with something like:
- %python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib'))") +%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', sysconfig.get_default_scheme(), {'platbase': '%{_prefix}', 'base': '%{_prefix}')})
Without that, the --prefix %{_prefix} additions won't work in isolation, and will require something else to redefine python3_sitelib/python3_sitearch.
Thanks! Owen
On 26. 05. 22 22:08, Owen Taylor wrote:
On Thu, May 26, 2022 at 1:07 PM Miro Hrončok <mhroncok@redhat.com mailto:mhroncok@redhat.com> wrote:
On 26. 05. 22 17:13, Owen Taylor wrote: > A pip with a backported patch (or a rebase) + this approach sounds most > maintainable for Fedora 36. But if we think that the --prefix %{_prefix} > approach actually works, then perhaps we can either: > > A) Just update python-rpm-macros and pygobject-rpm-macros in F36 > B) Build a stream-branch (or even the f37 branch) in the flatpak-runtime > module for F36 and reconverge for F37 - I'm less worried about carrying a > stream branch for macros packages, since they are unlikely to have security fixes. > > What do you think? Definitively A. There is no reason to fork the macros, the change should be perfectly backwards compatible. The lines are: https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python#_61 <https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python#_61> https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python3#_59 <https://src.fedoraproject.org/rpms/python-rpm-macros/blob/f36/f/macros.python3#_59> https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_57 <https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macros.pyproject#_57>OK, I'll work on coming up with some patches. Do you have an opinion about whether it makes sense to make the %_prefix support complete with something like:
- %python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig;
print(sysconfig.get_path('purelib'))") +%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', sysconfig.get_default_scheme(), {'platbase': '%{_prefix}', 'base': '%{_prefix}')})
Without that, the --prefix %{_prefix} additions won't work in isolation, and will require something else to redefine python3_sitelib/python3_sitearch.
Using vars={'platbase': '%{_prefix}', 'base': '%{_prefix}')} makes sense to me (you can omit the second parameter (scheme) if you pass vars by name).
On Thu, May 26, 2022 at 5:24 PM Miro Hrončok mhroncok@redhat.com wrote:
On 26. 05. 22 22:08, Owen Taylor wrote:
OK, I'll work on coming up with some patches. Do you have an opinion
about
whether it makes sense to make the %_prefix support complete with
something like:
- %python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig;
print(sysconfig.get_path('purelib'))") +%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', sysconfig.get_default_scheme(), {'platbase': '%{_prefix}', 'base': '%{_prefix}')})
Without that, the --prefix %{_prefix} additions won't work in isolation,
and
will require something else to redefine python3_sitelib/python3_sitearch.
Using vars={'platbase': '%{_prefix}', 'base': '%{_prefix}')} makes sense to me (you can omit the second parameter (scheme) if you pass vars by name).
Ah, yes, thanks.
I made a pair of pull requests: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/135 https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/288
that I believe have all the necessary changes, and did some testing (there's also a test added in pyproject/rpm-macros/tests, which seems to work following the instructions for local testing in the README.md)
Unfortunately, I need to take time off from work for personal reasons for a while, so if you could go ahead and make whatever changes are needed and merge them, that would be greatly appreciated. Stephan Bergmann can help you if you want confirmation that these are actually fixing the Flatpak build issues that we were running into.
Thanks! Owen
- Owen
On 07. 06. 22 20:20, Owen Taylor wrote:
On Thu, May 26, 2022 at 5:24 PM Miro Hrončok <mhroncok@redhat.com mailto:mhroncok@redhat.com> wrote:
On 26. 05. 22 22:08, Owen Taylor wrote: > OK, I'll work on coming up with some patches. Do you have an opinion about > whether it makes sense to make the %_prefix support complete with something like: > > - %python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; > print(sysconfig.get_path('purelib'))") > +%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; > print(sysconfig.get_path('purelib', sysconfig.get_default_scheme(), > {'platbase': '%{_prefix}', 'base': '%{_prefix}')}) > > Without that, the --prefix %{_prefix} additions won't work in isolation, and > will require something else to redefine python3_sitelib/python3_sitearch. Using vars={'platbase': '%{_prefix}', 'base': '%{_prefix}')} makes sense to me (you can omit the second parameter (scheme) if you pass vars by name).Ah, yes, thanks.
I made a pair of pull requests: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/135 https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/135 https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/288 https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/288
that I believe have all the necessary changes, and did some testing (there's also a test added in pyproject/rpm-macros/tests, which seems to work following the instructions for local testing in the README.md)
Unfortunately, I need to take time off from work for personal reasons for a while, so if you could go ahead and make whatever changes are needed and merge them, that would be greatly appreciated. Stephan Bergmann can help you if you want confirmation that these are actually fixing the Flatpak build issues that we were running into.
This is the Fedora 36 update:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-5eaabfc110
Stephan, could you please verify?
On Thu, Jun 9, 2022, 8:01 AM Miro Hrončok mhroncok@redhat.com wrote:
Unfortunately, I need to take time off from work for personal reasons
for a
while, so if you could go ahead and make whatever changes are needed and
merge
them, that would be greatly appreciated. Stephan Bergmann can help you
if you
want confirmation that these are actually fixing the Flatpak build
issues that
we were running into.
This is the Fedora 36 update:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-5eaabfc110
Stephan, could you please verify?
Thanks, Miro!
Stephan - here's how I would test
* Modify flatpak-rpm-macros to remove all the python-specific overrides, and build it locally. * Install that and the new python and pyobject macros in a throwaway container (or remove afterwards) * Dnf builddep and rebuild the affected package. If it rebuilds successfully and everything is under /app, we should be all set.
I don't think trying to rebuild locally under MBS with packages from updates-testing is going to be feasible.
Thanks! Owen
On 26. 05. 22 17:13, Owen Taylor wrote:
- Why is Python itself not installed in the /app prefix? - Where and how is pip install invoked?
The way that Flatpaks work is that there are *two* filesystem images, one mounted on /usr and one mounted on /app. The image mounted on /usr is called the "runtime" and contains libraries and other files shared between different applications. In Fedora, this is made out of standard distribution packages - glibc, libjpeg, and so forth. It also contains Python 3 and some modules.
The filesystem image at /app contains packages specific to a specific application - the application itself and libraries. In Fedora, we create the /app filesystem by rebuilding packages inside a module that is configured to have a special RPM macros package installed in the buildroot - https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 https://src.fedoraproject.org/rpms/flatpak-rpm-macros/tree/f36 (installed at /etc/rpm/macros.flatpak so that it has higher priority than the macros from /usr/lib/rpm)
(The advantage of this split over the docker layer system is that applications don't need to be rebuilt every time the base layer is updated.)
macros.flatpak redefines %_prefix to /app, and then does various more and less hacky things to make that actual work within the Fedora RPM ecosystem - in particular for Python, it overrides %python3_sitelib and %python3_sitearch. The distutils.cfg I mentioned is installed from the same package. This combination worked pretty well for F35 for Python.
pip install is being invoked is by %pyproject_install or %py_install_wheel out of an RPM build - the particular breakage that triggered this investigation was the Python bindings for zxing-cpp , but I think it would apply to pretty much anything that has moved to the %pyproject macros.
I have a followup question about this approach.
Consider I want to build 2 (or more) Python packages into /app because my Flatpak needs multiple libraries specific to my application.
For simplicity, let's assume the following dependency chain.
myapp -> python3-higher-level-lib -> python3-lower-level-lib
The build chain starts with python-lower-level-lib component. It set's %_prefix to /app and installs everything to %{python3_stielib} which is in /app. Python itself uses the /usr prefix, but we pass --prefix explicitly, so it works, the resulting python3-lower-level-lib package contains everything in /app.
The build chain proceeds with python-higher-level-lib. During the build python3-lower-level-lib is BuildRequired as it is required to run the %check section. In the %check section, tests are executed and they want to import a Python module from python3-lower-level-lib -- hewer, /app/.../site-pacakges is not on sys.path and the %check fails.
How is the second situation solved? I've seen slightly horrifying things like this in some specfiles:
%if ! 0%{?flatpak} %check make test ... %endif
Was that because of this problem?
python-devel@lists.fedoraproject.org
-
Miro Hrončok -
Neal Gompa -
Owen Taylor