%pyproject_buildrequires -r/-x: Attempt to read runtime dependencies from pyproject.toml?
Hello,
recently, we were suggested an improvement for %pyproject_buildrequires -r/-x. We could read the project's runtime dependencies (if they're not marked as `dynamic`) from pyproject.toml [project] table directly, without calling prepare_metadata_for_build_wheel or building the wheel to read the dependency metadata from it.
Reading the metadata via prepare_metadata_for_build_wheel is already quite fast, however implementing that hook is optional for the build backends. When the hook is not available, we need to build the wheel, which can be potentially very slow and resource-hungry (imagine building the entire scipy project). Metadata in pyproject.toml is standardized (PEP 621), however there is no way of telling ahead of time if the build backend already supports that standard.
See the details and discussion in bugzilla: https://bugzilla.redhat.com/2261939
Maxwell has raised some valid concerns there: - Other tools (build frontends, e.g. pip/build) call prepare_metadata_for_build_wheel, our macros would diverge in functionality compared to the rest of the landscape. - pyproject.toml's [project] may not be the source of metadata that the build backend uses. A project could switch to a build backend without PEP 621 support (e.g. poetry-core) and forget to remove the [project] table. - There can be potential differences between BuildRequires and Requires generators when one generates dependencies based on the pyproject.toml [project] table while the other on the packaged dist-info metadata. - Using macros to build on older systems: e.g. RHEL 9's old setuptools version silently ignores the pyproject.toml [project] table - %pyproject_buildrequires could still pull the dependency information from it, but the resulting package would be broken because it did not include those in the packaged dist-info metadata.
One way to mitigate would be to make the proposed behavior opt-in only, with the possibility to either build wheel with -w option (already existing) or e.g. -p (now-proposed: reading from pyproject.toml) in case backend doesn't have prepare_metadata_for_build_wheel. However, this adds a layer of complexity for packagers and macros maintainers.
The questions we'd love your input for: - Should %pyproject_buildrequires try to read dependencies from pyproject.toml first and fall back to calling hooks only if that fails? - Should %pyproject_buildrequires call the hook and try to fall back to reading dependencies from pyproject.toml when the hook is not availbale? - Should this behavior exist but not be the default (explicit flag would be required to opt-in)? - Can you think of a better alternative than the ones described here?
Cheers, Karolina
On Wed Mar 27, 2024 at 15:48 +0100, Karolina Surma wrote:
Hello,
Hi Karolina,
Thank you for bring this to the mailing list.
recently, we were suggested an improvement for %pyproject_buildrequires -r/-x. We could read the project's runtime dependencies (if they're not marked as `dynamic`) from pyproject.toml [project] table directly, without calling prepare_metadata_for_build_wheel or building the wheel to read the dependency metadata from it.
Reading the metadata via prepare_metadata_for_build_wheel is already quite fast, however implementing that hook is optional for the build backends.
I'll note that all build backends packaged in Fedora (setuptools, hatchling, flit-core, poetry-core, the pdm one, and scikit) other than meson-python support this hook. Only 9 packages BuildRequire python3-meson-python.
Maxwell has raised some valid concerns there:
- Other tools (build frontends, e.g. pip/build) call
prepare_metadata_for_build_wheel, our macros would diverge in functionality compared to the rest of the landscape.
- pyproject.toml's [project] may not be the source of metadata that the
build backend uses. A project could switch to a build backend without PEP 621 support (e.g. poetry-core) and forget to remove the [project] table.
- There can be potential differences between BuildRequires and Requires
generators when one generates dependencies based on the pyproject.toml [project] table while the other on the packaged dist-info metadata.
- Using macros to build on older systems: e.g. RHEL 9's old setuptools
version silently ignores the pyproject.toml [project] table - %pyproject_buildrequires could still pull the dependency information from it, but the resulting package would be broken because it did not include those in the packaged dist-info metadata.
I could not have said it better myself :D.
One way to mitigate would be to make the proposed behavior opt-in only, with the possibility to either build wheel with -w option (already existing) or e.g. -p (now-proposed: reading from pyproject.toml) in case backend doesn't have prepare_metadata_for_build_wheel.
Yeah, I think -p (for *p*yproject) is good flag name choice.
However, this adds a layer of complexity for packagers and macros maintainers.
The questions we'd love your input for: - Should this behavior exist but not be the default (explicit flag would be required to opt-in)?
I consider this the only reasonable solution other than not implementing the feature at all.
- Can you think of a better alternative than the ones described here?
I guess I will throw something out there, but I am not convinced it is a great idea: what about making the `-p` flag fail if `prepare_metadata_for_build_wheel` is available? In my opinion, this should only be a last resort for backends that do not implement the hook.
—Maxwell
On 27. 03. 24 23:15, Maxwell G wrote:
One way to mitigate would be to make the proposed behavior opt-in only, with the possibility to either build wheel with -w option (already existing) or e.g. -p (now-proposed: reading from pyproject.toml) in case backend doesn't have prepare_metadata_for_build_wheel.
Yeah, I think -p (for *p*yproject) is good flag name choice.
Or even for [*p*roject] table. It is double good.
I guess I will throw something out there, but I am not convinced it is a great idea: what about making the `-p` flag fail if `prepare_metadata_for_build_wheel` is available? In my opinion, this should only be a last resort for backends that do not implement the hook.
I am not particularly keen on this.
This means that once the backend starts supporting it, all the spec files using -p need to drop it. And if the backend only supports it in rawhide, the spec files need to diverge and/or %if-guard the flag.
If the backend followed PEP 621 before adding the hook and now it added the hook, it is unlikely the PEP 621 support was dropped.
OTHO if the backed was changed (e.g. meson -> poetry), this *could* happen. So I am not entirely opposed for this guard.
Hello,
recently, we were suggested an improvement for %pyproject_buildrequires -r/-x. We could read the project's runtime dependencies (if they're not marked as `dynamic`) from pyproject.toml [project] table directly, without calling prepare_metadata_for_build_wheel or building the wheel to read the dependency metadata from it.
We implemented the change in pyproject-rpm-macros-1.15.0-1:
F39: https://bodhi.fedoraproject.org/updates/FEDORA-2024-97e69afef1 F40: https://bodhi.fedoraproject.org/updates/FEDORA-2024-f17e2f559b F41: https://bodhi.fedoraproject.org/updates/FEDORA-2024-aa49f791e5
When invoking `%pyproject_buildrequires -p` the dependencies will be read from the pyproject.toml [project] table and in case pyproject.toml doesn't exist, the [project] table doesn't exist, or the metadata are defined as dynamic, the macro will fail. There is no fallback mechanism implemented. You can combine the `-p` flag with extra flags (`-x`). It is mutually exclusive with `-w`, `-R` and `-N`.
Cheers, Karolina
On 27-03-2024 15:48, Karolina Surma wrote:
One way to mitigate would be to make the proposed behavior opt-in only, with the possibility to either build wheel with -w option (already existing) or e.g. -p (now-proposed: reading from pyproject.toml) in case backend doesn't have prepare_metadata_for_build_wheel. However, this adds a layer of complexity for packagers and macros maintainers.
The questions we'd love your input for: - Should %pyproject_buildrequires try to read dependencies from pyproject.toml first and fall back to calling hooks only if that fails? - Should %pyproject_buildrequires call the hook and try to fall back to reading dependencies from pyproject.toml when the hook is not availbale? - Should this behavior exist but not be the default (explicit flag would be required to opt-in)? - Can you think of a better alternative than the ones described here?
I'd be in favor of option two in combination with option three. That is, set a flag explicitly two enable the behavior described in option two, fallback to pyproject.toml if hook is not available.
Doing it that way also means the current behavior remains unchanged for package maintainers being unaware similar to the -l flag of %pyproject_save_files.
I'm approaching this purely from a packager's point of view. I dislike having the wheel built twice, once for the metadata and again for the package itself.
The documentation could point to the build backends, where it makes sense enabling -p, e.g. meson-python.
-- Sandro
python-devel@lists.fedoraproject.org
-
Karolina Surma -
Maxwell G -
Miro Hrončok -
Sandro