Hi folks,
I'm in the process of updating the nest package to the latest release. It uses cmake as its build system, but then the cmake calls `python setup.py install` and so on to install the Python API. So, we strip that bit out and call these commands ourselves in the right places in the spec[1,2,3].
As you'll also see from the spec, nest supports MPI, so we build it 3 times: - without MPI - once with openmpi - once with mpich
For the MPI builds, we need to pass the `--install-lib=$MPI_SITEARCH` option to `python setup.py install` to get it to install the bits in the right MPI related locations.
I've noticed that using this system now gives us deprecation warnings during the build since calling `setup.py ...` explicitly is deprecated.
However, I can't figure out how I would replicate the build using the `pyproject` macros. I guess `pyproject_wheel` usage is straightforward (?), but how do I get `pyproject_install` to install in the $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to store the files in three different lists that I can use in the different `%files` sections?
[1] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/0001-Disable-python-s... [2] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_342 [3] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_424
Quick addendum:
Upstream has moved away from calling `setup.py` and now uses `pip`:
https://github.com/nest/nest-simulator/blob/v3.4/pynest/CMakeLists.txt#L91
Still not sure how to use the pyproject macros though.
On Mon, Jun 12, 2023 15:04:54 +0100, Ankur Sinha wrote:
Hi folks,
I'm in the process of updating the nest package to the latest release. It uses cmake as its build system, but then the cmake calls `python setup.py install` and so on to install the Python API. So, we strip that bit out and call these commands ourselves in the right places in the spec[1,2,3].
As you'll also see from the spec, nest supports MPI, so we build it 3 times:
- without MPI
- once with openmpi
- once with mpich
For the MPI builds, we need to pass the `--install-lib=$MPI_SITEARCH` option to `python setup.py install` to get it to install the bits in the right MPI related locations.
I've noticed that using this system now gives us deprecation warnings during the build since calling `setup.py ...` explicitly is deprecated.
However, I can't figure out how I would replicate the build using the `pyproject` macros. I guess `pyproject_wheel` usage is straightforward (?), but how do I get `pyproject_install` to install in the $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to store the files in three different lists that I can use in the different `%files` sections?
[1] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/0001-Disable-python-s... [2] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_342 [3] https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_424
-- Thanks, Regards, Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha Time zone: Europe/London
On Mon, Jun 12, 2023 15:25:57 +0100, Ankur Sinha wrote:
Quick addendum:
Upstream has moved away from calling `setup.py` and now uses `pip`:
https://github.com/nest/nest-simulator/blob/v3.4/pynest/CMakeLists.txt#L91
Still not sure how to use the pyproject macros though.
No, that's wrong. They still call `setup.py install_egg_info` there.
On 12. 06. 23 16:04, Ankur Sinha wrote:
However, I can't figure out how I would replicate the build using the `pyproject` macros. I guess `pyproject_wheel` usage is straightforward (?), but how do I get `pyproject_install` to install in the $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to store the files in three different lists that I can use in the different `%files` sections?
This is currently not possible. Unfortunately, support for this would require somebody else than me to drive it.
(If you just need to install the same files in multiple locations, you can cp -a them around after %pyproject_install.)
On Mon, Jun 12, 2023 21:07:05 +0200, Miro Hrončok wrote:
On 12. 06. 23 16:04, Ankur Sinha wrote:
However, I can't figure out how I would replicate the build using the `pyproject` macros. I guess `pyproject_wheel` usage is straightforward (?), but how do I get `pyproject_install` to install in the $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to store the files in three different lists that I can use in the different `%files` sections?
This is currently not possible. Unfortunately, support for this would require somebody else than me to drive it.
(If you just need to install the same files in multiple locations, you can cp -a them around after %pyproject_install.)
Thanks for your reply, Miro. I guess I can go through the pyproject macro sources and see if it's possible to replicate their flow in the spec. If not, I'll try and replace the `setup.py` commands with `pip` commands in the spec and see how that goes.
Hi Ankur,
Not sure whether that helps, but BOUT++ (requires MPI) switched to a custom back end [1] when it switched to cmake [2] from configure + calling setup.py. The main motivation was to enable packaging in fedora, but it also allowed to just pip install the whole package, as BOUT++ is now PEP-517 enabled.
This doesn't call the pyproject macros, but just the cmake ones [3], so not sure this is applicable to what you need, but maybe it offers at least some insight into how it is done for other MPI codes ...
Best, David
[1] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil... [2] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil... [3] https://src.fedoraproject.org/rpms/bout++/blob/rawhide/f/bout++.spec
On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote:
Hi Ankur,
Hi David,
Thanks for your reply.
Not sure whether that helps, but BOUT++ (requires MPI) switched to a custom back end [1] when it switched to cmake [2] from configure + calling setup.py. The main motivation was to enable packaging in fedora, but it also allowed to just pip install the whole package, as BOUT++ is now PEP-517 enabled.
This doesn't call the pyproject macros, but just the cmake ones [3], so not sure this is applicable to what you need, but maybe it offers at least some insight into how it is done for other MPI codes ...
That's interesting. So it builds the cpython as part of the cmake build and then simply installs it in the required directory.
How does it generate the dist-info metadata, though? That's the bit where nest uses `python setup.py install_egg_info`.
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_info...
https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e237...
On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote: That's interesting. So it builds the cpython as part of the cmake build and then simply installs it in the required directory.
How does it generate the dist-info metadata, though? That's the bit where nest uses `python setup.py install_egg_info`.
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_i...
https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e...
Unless I am mistaken, that is done in the prepare_metadata_for_build [1], which gets called from here [2]
It is currently missing the readme, thus the project description is empty on pypi [3]. I had troubles finding the appropriate documentation how to do that.
[1] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil... [2] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil... [3] https://pypi.org/project/boutpp/
On Wed, Jun 14, 2023 12:59:40 -0000, David Schwörer wrote:
On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote: That's interesting. So it builds the cpython as part of the cmake build and then simply installs it in the required directory.
How does it generate the dist-info metadata, though? That's the bit where nest uses `python setup.py install_egg_info`.
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_i...
https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e...
Unless I am mistaken, that is done in the prepare_metadata_for_build [1], which gets called from here [2]
It is currently missing the readme, thus the project description is empty on pypi [3]. I had troubles finding the appropriate documentation how to do that.
Ah, indeed it is generated there. I asked because I was looking at the Fedora package to try and figure it out and couldn't find the metadata in there:
$ sudo dnf repoquery -l python3-bout++ | grep info # returns nothing
Thanks again for all your replies. I'm going to spend some time tinkering with the nest sources to see what I can do.
$ sudo dnf repoquery -l python3-bout++ | grep info # returns nothing
Thanks again for all your replies. I'm going to spend some time tinkering with the nest sources to see what I can do.
They should be though in the mpi packages: $ sudo dnf repoquery -l python3-bout++-mpich | grep info # returns nothing But they are there missing as well. I think the issue is that not the wheel gets installed with cmake, but the installation is done with cmake: https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
I just notice that the wheel is not build at all, as that invokes cmake itself, thus a different wheel would need to be build by cmake, but then I am not sure how that can be installed?
On Thu, Jun 15, 2023 10:13:28 -0000, David Schwörer wrote:
$ sudo dnf repoquery -l python3-bout++ | grep info # returns nothing
Thanks again for all your replies. I'm going to spend some time tinkering with the nest sources to see what I can do.
They should be though in the mpi packages: $ sudo dnf repoquery -l python3-bout++-mpich | grep info # returns nothing But they are there missing as well. I think the issue is that not the wheel gets installed with cmake, but the installation is done with cmake: https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
I just notice that the wheel is not build at all, as that invokes cmake itself, thus a different wheel would need to be build by cmake, but then I am not sure how that can be installed?
So the wheel is built manually: https://github.com/boutproject/BOUT-dev/blob/cbd197e78f7d52721188badfd7c38a0...
but I don't see where they then install it. Maybe that's the step that's missing, and would install the dist-info etc.?
Looking at `pyproject_install`:
$ rpm -E %{pyproject_install} \ specifier=$(ls /home/asinha/rpmbuild/BUILD/pyproject-wheeldir/*.whl | xargs basename --multiple | sed -E 's/([^-]+)-([^-]+)-.+.whl/\1==\2/') TMPDIR="/home/asinha/rpmbuild/BUILD/.pyproject-builddir" /usr/bin/python3 -m pip install --root /home/asinha/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 --prefix /usr --no-deps --disable-pip-version-check --progress-bar off --verbose --ignore-installed --no-warn-script-location --no-index --no-cache-dir --find-links /home/asinha/rpmbuild/BUILD/pyproject-wheeldir $specifier
Source here: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macro...
So, for nest, I'm thinking I can build and install the wheel manually perhaps similar to how I build and install the package manually now, only at the moment I do it with `setup.py`.
python-devel@lists.fedoraproject.org