RPM 4.19: Dynamic subpackages with Python extras
by Miro Hrončok
Hello Python packagers.
RPM 4.19 introduces this feature:
https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html
I decided to write this email to gather my thoughts. I believe that with this,
we can turn manual Python extras subpackages like this:
%package -n python3-...
Summary: %{summary}
%description -n python3-... %_description
%pyproject_extras_subpkg -n python3-xxx extra1 extra2
(See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras
for what that means.)
Into something like this:
%package -n python3-...
Summary: %{summary}
%description -n python3-... %_description
...
%install
%pyproject_install
...
%pyproject_generate_extras_subpkgs -n python3-xxx
The %pyproject_generate_extras_subpkgs macro would parse the installed
.dist-info directory to find out what extras are available and generate
subpackages for all of them.
(Obviously, the macro name is open up for discussion.)
----------------
An API would be required to exclude extras:
- that are not useful for other packages
(for example build/development requirements, commonly named dev, doc or test)
- that have requirements that are not packaged in Fedora
For example (mimicking the API of %pyproject_check_import):
%pyproject_generate_extras_subpkgs -n python3-xxx -e test -e 'nonfree*'
----------------
However, extras are also currently manually passed to %pyproject_buildrequires:
%generate_buildrequires
%pyproject_buildrequires -x extra1 -x extra2 -x test
It should already be possible to implement automatic extras discovery in
%pyproject_buildrequires with older RPM versions and allow it to be used this way:
%generate_buildrequires
%pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
RPM macros can only accept short flags, so <FLAG_TO_ENABLE_ALL_EXTRAS> can
either be -x '*' (if we start treating -x values as globs, which is backwards
compatible and probably generally useful), or a single-letter switch such as -a
(but honestly we are running out of meaningful letters).
(When -X is used, <FLAG_TO_ENABLE_ALL_EXTRAS> can probably be implied. However,
an explicit form needs to exist for packages that don't need to exclude any
extras at all.)
Eventually, I'd like to make <FLAG_TO_ENABLE_ALL_EXTRAS> the default, once RPM
4.19 is omnipresent.
----------------
Combined, this would mean that the packager needs to:
1. specify extras that are not supposed to be used as BRs
2. specify extras that are not supposed to be packaged
In the ideal word (2) is a superset of (1).
Should %pyproject_generate_extras_subpkgs somehow inherit the -Xes from
%pyproject_buildrequires?
When a package has extra1, extra2, nonfree1, nonfree2 and test extras, one
could do:
%generate_buildrequires
%pyproject_buildrequires <FLAG_TO_ENABLE_ALL_EXTRAS> -X 'nonfree*'
...
%pyproject_install
...
%pyproject_generate_extras_subpkgs -X test
That would mean:
- extra1 is BRed and packaged
- extra2 is BRed and packaged
- test is BRed but not packaged
- nonfree1 is neither
- nonfree2 is neither
----------------
Alternatively the information could be supplied by %globals:
%global _python_ignored_extras nonfree*
%global _python_unpackaged_extras test
However, I somehow dislike this approach.
----------------
I'd appreciate your thoughts on the matter.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
2 weeks
Package naming for vkbasalt-cli
by Sandro
Hi,
I need to package vkbasalt-cli [1] as a dependency for Bottles. Bottles
will use the Python library, but vkbasalt-cli also provides a command
for independent/direct usage.
My plan is to name the Fedora package 'vkbasalt-cli' in line with
upstream and add a virtual provide for 'python3-vkbasalt-cli'.
Could someone confirm that this is the right approach for vkbasalt-cli?
Thanks in advance,
--
Sandro
FAS: gui1ty
IRC: Penguinpee
Elsewhere: [Pp]enguinpee
2 weeks, 1 day
Let's deprecate the %{pyproject_build_lib} macro... ?
by Miro Hrončok
Hello.
The %{pyproject_build_lib} macro is a *provisional* macro from
pyproject-rpm-macros.
It was added to solve the problem of out-of-tree pip builds. From the
pyproject-rpm-macros README:
"""
Sometimes, it is desired to be able to import the just-built extension modules
in the %build section, e.g. to build the documentation with Sphinx.
%build
%pyproject_wheel
... build the docs here ...
With pure Python packages, it might be possible to set PYTHONPATH=${PWD} or
PYTHONPATH=${PWD}/src. However, it is a bit more complicated with extension
modules.
The location of just-built modules might differ depending on Python version,
architecture, pip version, etc. Hence, the macro %{pyproject_build_lib} exists
to be used like this:
%build
%pyproject_wheel
PYTHONPATH=%{pyproject_build_lib} ... build the docs here ...
"""
When this macro was introduced, the built extension module lived in a folder
like
/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10,
hence it made sense to include in the pyproject-rpm-macros and call it
%pyproject_something.
------------------------------------------------------------
Today, on Rawhide, pip default to in-tree-builds and the value for extension
modules will most likely be:
$PWD/build/lib.%{python3_platform}-cpython-%{python3_version_nodots}
And for pure Python:
$PWD/build/lib
And it turns out, the value is setuptools-specific, see
https://bugzilla.redhat.com/show_bug.cgi?id=2176393
Other build backends might store the built extension modules elsewhere or
simply pack the pure Python files into wheel directly from the source tree.
------------------------------------------------------------
In Rawhide, 41 packages use PYTHONPATH="%{pyproject_build_lib}" and 2 packages
have a comment with %%{pyproject_build_lib} in them.
Considering the macro
- is not build-backend-agnostic, and
- is not so much needed as it once was
I propose we deprecate it with no further fixes going in, but no scheduled removal.
The 41 packages can be fixed once Fedora 36 goes EOL by expanding the macro to
the values above, or if desired, we could macronize this in pythohn3-setuptools
as %{setuptools_build_lib}.
One problem is that the macro is unfortunately still needed on EL 9.
Thoughts?
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 month, 1 week
Re: [Fedora-packaging] Re: RPM 4.19: Dynamic subpackages with Python
extras
by Miro Hrončok
On 04. 04. 23 19:24, Dan Čermák wrote:
> Hi Miro,
>
> On March 31, 2023 10:06:14 AM UTC, "Miro Hrončok" <mhroncok(a)redhat.com> wrote:
>> On 30. 03. 23 23:55, Miro Hrončok wrote:
>>> Hello Python packagers.
>>>
>>> RPM 4.19 introduces this feature:
>>>
>>> https://rpm-software-management.github.io/rpm/manual/dynamic_specs.html
>>>
>>> I decided to write this email to gather my thoughts. I believe that with this, we can turn manual Python extras subpackages like this:
>>>
>>> %package -n python3-...
>>> Summary: %{summary}
>>>
>>> %description -n python3-... %_description
>>>
>>> %pyproject_extras_subpkg -n python3-xxx extra1 extra2
>>>
>>> (See https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#Extras for what that means.)
>>>
>>> Into something like this:
>>>
>>> %package -n python3-...
>>> Summary: %{summary}
>>>
>>> %description -n python3-... %_description
>>> ...
>>>
>>> %install
>>> %pyproject_install
>>> ...
>>> %pyproject_generate_extras_subpkgs -n python3-xxx
>>>
>>>
>>> The %pyproject_generate_extras_subpkgs macro would parse the installed .dist-info directory to find out what extras are available and generate subpackages for all of them.
>>
>> One challenge: When an extra is removed upstream, the packager might not notice that it needs to be provided/obsole>
>
> That is indeed a problem. What if we made the macro require the list of subpackages (e.g. as a comma separated string) as an argument? The if the packager wouldn't notice the removal, they'd get a build error.
We already have that today without dynamic subpackages, because when you know
the list ahead, you can "generate" subpackages already.
$ rpm --eval '%pyproject_extras_subpkg -n python3-... extra1 extra2'
%package -n python3-...+extra1
Summary: Metapackage for python3-...: extra1 extras
Requires: python3-... = %{version}-%{release}
%description -n python3-...+extra1
This is a metapackage bringing in extra1 extras requires for python3-....
It makes sure the dependencies are installed.
%files -n python3-...+extra1 -f
.../BUILD/%{name}-%{version}-%{release}.x86_64-pyproject-ghost-distinfo
%package -n python3-...+extra2
Summary: Metapackage for python3-...: extra2 extras
Requires: python3-... = %{version}-%{release}
%description -n python3-...+extra2
This is a metapackage bringing in extra2 extras requires for python3-....
It makes sure the dependencies are installed.
%files -n python3-...+extra2 -f
.../%{name}-%{version}-%{release}.x86_64-pyproject-ghost-distinfo
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
1 month, 3 weeks
