python-devel April 2023

python-devel@lists.fedoraproject.org

5 participants
5 discussions

RPM 4.19: Dynamic subpackages with Python extras
by Miro Hrončok 17 May '23

17 May '23

Hello Python packagers. RPM 4.19 introduces this feature: https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html I decided to write this email to gather my thoughts. I believe that with this, we can turn manual Python extras subpackages like this: %package -n python3-... Summary: %{summary} %description -n python3-... %_description %pyproject_extras_subpkg -n python3-xxx extra1 extra2 (See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras for what that means.) Into something like this: %package -n python3-... Summary: %{summary} %description -n python3-... %_description ... %install %pyproject_install ... %pyproject_generate_extras_subpkgs -n python3-xxx The %pyproject_generate_extras_subpkgs macro would parse the installed .dist-info directory to find out what extras are available and generate subpackages for all of them. (Obviously, the macro name is open up for discussion.) ---------------- An API would be required to exclude extras: - that are not useful for other packages (for example build/development requirements, commonly named dev, doc or test) - that have requirements that are not packaged in Fedora For example (mimicking the API of %pyproject_check_import): %pyproject_generate_extras_subpkgs -n python3-xxx -e test -e 'nonfree*' ---------------- However, extras are also currently manually passed to %pyproject_buildrequires: %generate_buildrequires %pyproject_buildrequires -x extra1 -x extra2 -x test It should already be possible to implement automatic extras discovery in %pyproject_buildrequires with older RPM versions and allow it to be used this way: %generate_buildrequires %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*' RPM macros can only accept short flags, so <FLAG_TO_ENABLE_ALL_EXTRAS> can either be -x '*' (if we start treating -x values as globs, which is backwards compatible and probably generally useful), or a single-letter switch such as -a (but honestly we are running out of meaningful letters). (When -X is used, <FLAG_TO_ENABLE_ALL_EXTRAS> can probably be implied. However, an explicit form needs to exist for packages that don't need to exclude any extras at all.) Eventually, I'd like to make <FLAG_TO_ENABLE_ALL_EXTRAS> the default, once RPM 4.19 is omnipresent. ---------------- Combined, this would mean that the packager needs to: 1. specify extras that are not supposed to be used as BRs 2. specify extras that are not supposed to be packaged In the ideal word (2) is a superset of (1). Should %pyproject_generate_extras_subpkgs somehow inherit the -Xes from %pyproject_buildrequires? When a package has extra1, extra2, nonfree1, nonfree2 and test extras, one could do: %generate_buildrequires %pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*' ... %pyproject_install ... %pyproject_generate_extras_subpkgs -X test That would mean: - extra1 is BRed and packaged - extra2 is BRed and packaged - test is BRed but not packaged - nonfree1 is neither - nonfree2 is neither ---------------- Alternatively the information could be supplied by %globals: %global _python_ignored_extras nonfree* %global _python_unpackaged_extras test However, I somehow dislike this approach. ---------------- I'd appreciate your thoughts on the matter. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

4 4

Package naming for vkbasalt-cli
by Sandro 16 May '23

16 May '23

Hi, I need to package vkbasalt-cli [1] as a dependency for Bottles. Bottles will use the Python library, but vkbasalt-cli also provides a command for independent/direct usage. My plan is to name the Fedora package 'vkbasalt-cli' in line with upstream and add a virtual provide for 'python3-vkbasalt-cli'. Could someone confirm that this is the right approach for vkbasalt-cli? Thanks in advance, -- Sandro FAS: gui1ty IRC: Penguinpee Elsewhere: [Pp]enguinpee

4 12

Numpy version pinned despite numpy>=1.22.3
by Sandro 01 May '23

01 May '23

Hi, I'm trying to build/update a package using mesonpy [1]. While going through the build process, exploring how the existing macros behave together with mesonpy, I came across a weird build error, for which I have not yet found a cause or explanation. The build fails with [2]: No matching package to install: 'python3dist(numpy) = 1.23.2' The version in rawhide is 1.24.3, yet the requirement in pyproject.toml [3] is: "numpy>=1.22.3" So, why is it insisting on using 1.23.2 while the existing 1.24.3 satisfies the requirement? There's one thing I noticed [4], which I believe is unrelated. But I asked upstream for clarification anyway. The spec file is currently work in progress. I simply triggered a Copr build, so I could provide links here and verify that this is not a local issue in my build environment. [1] So fresh it's still warm: https://src.fedoraproject.org/rpms/python-meson-python [2] https://download.copr.fedorainfracloud.org/results/gui1ty/neuro-sig/fedora-… [3] https://github.com/has2k1/scikit-misc/blob/8f829888ecf0fa96eb18aeb3f723fc98… [4] https://github.com/has2k1/scikit-misc/issues/24 Cheers, -- Sandro FAS: gui1ty IRC: Penguinpee Elsewhere: [Pp]enguinpee

3 3

Let's deprecate the %{pyproject_build_lib} macro... ?
by Miro Hrončok 18 Apr '23

18 Apr '23

Hello. The %{pyproject_build_lib} macro is a *provisional* macro from pyproject-rpm-macros. It was added to solve the problem of out-of-tree pip builds. From the pyproject-rpm-macros README: """ Sometimes, it is desired to be able to import the just-built extension modules in the %build section, e.g. to build the documentation with Sphinx. %build %pyproject_wheel ... build the docs here ... With pure Python packages, it might be possible to set PYTHONPATH=${PWD} or PYTHONPATH=${PWD}/src. However, it is a bit more complicated with extension modules. The location of just-built modules might differ depending on Python version, architecture, pip version, etc. Hence, the macro %{pyproject_build_lib} exists to be used like this: %build %pyproject_wheel PYTHONPATH=%{pyproject_build_lib} ... build the docs here ... """ When this macro was introduced, the built extension module lived in a folder like /builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10, hence it made sense to include in the pyproject-rpm-macros and call it %pyproject_something. ------------------------------------------------------------ Today, on Rawhide, pip default to in-tree-builds and the value for extension modules will most likely be: $PWD/build/lib.%{python3_platform}-cpython-%{python3_version_nodots} And for pure Python: $PWD/build/lib And it turns out, the value is setuptools-specific, see https://bugzilla.redhat.com/show_bug.cgi?id=2176393 Other build backends might store the built extension modules elsewhere or simply pack the pure Python files into wheel directly from the source tree. ------------------------------------------------------------ In Rawhide, 41 packages use PYTHONPATH="%{pyproject_build_lib}" and 2 packages have a comment with %%{pyproject_build_lib} in them. Considering the macro - is not build-backend-agnostic, and - is not so much needed as it once was I propose we deprecate it with no further fixes going in, but no scheduled removal. The 41 packages can be fixed once Fedora 36 goes EOL by expanding the macro to the values above, or if desired, we could macronize this in pythohn3-setuptools as %{setuptools_build_lib}. One problem is that the macro is unfortunately still needed on EL 9. Thoughts? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

2 2

Re: [Fedora-packaging] Re: RPM 4.19: Dynamic subpackages with Python extras
by Miro Hrončok 04 Apr '23

04 Apr '23

On 04. 04. 23 19:24, Dan Čermák wrote: > Hi Miro, > > On March 31, 2023 10:06:14 AM UTC, "Miro Hrončok" <mhroncok(a)redhat.com> wrote: >> On 30. 03. 23 23:55, Miro Hrončok wrote: >>> Hello Python packagers. >>> >>> RPM 4.19 introduces this feature: >>> >>> https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html >>> >>> I decided to write this email to gather my thoughts. I believe that with this, we can turn manual Python extras subpackages like this: >>> >>> %package -n python3-... >>> Summary: %{summary} >>> >>> %description -n python3-... %_description >>> >>> %pyproject_extras_subpkg -n python3-xxx extra1 extra2 >>> >>> (See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras for what that means.) >>> >>> Into something like this: >>> >>> %package -n python3-... >>> Summary: %{summary} >>> >>> %description -n python3-... %_description >>> ... >>> >>> %install >>> %pyproject_install >>> ... >>> %pyproject_generate_extras_subpkgs -n python3-xxx >>> >>> >>> The %pyproject_generate_extras_subpkgs macro would parse the installed .dist-info directory to find out what extras are available and generate subpackages for all of them. >> >> One challenge: When an extra is removed upstream, the packager might not notice that it needs to be provided/obsole> > > That is indeed a problem. What if we made the macro require the list of subpackages (e.g. as a comma separated string) as an argument? The if the packager wouldn't notice the removal, they'd get a build error. We already have that today without dynamic subpackages, because when you know the list ahead, you can "generate" subpackages already. $ rpm --eval '%pyproject_extras_subpkg -n python3-... extra1 extra2' %package -n python3-...+extra1 Summary: Metapackage for python3-...: extra1 extras Requires: python3-... = %{version}-%{release} %description -n python3-...+extra1 This is a metapackage bringing in extra1 extras requires for python3-.... It makes sure the dependencies are installed. %files -n python3-...+extra1 -f .../BUILD/%{name}-%{version}-%{release}.x86_64-pyproject-ghost-distinfo %package -n python3-...+extra2 Summary: Metapackage for python3-...: extra2 extras Requires: python3-... = %{version}-%{release} %description -n python3-...+extra2 This is a metapackage bringing in extra2 extras requires for python3-.... It makes sure the dependencies are installed. %files -n python3-...+extra2 -f .../%{name}-%{version}-%{release}.x86_64-pyproject-ghost-distinfo -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel April 2023