nest: python package still setup.py; also needs to be built for MPI

[Ankur Sinha]

Hi folks,

I'm in the process of updating the nest package to the latest release.
It uses cmake as its build system, but then the cmake calls `python
setup.py install` and so on to install the Python API. So, we strip that
bit out and call these commands ourselves in the right places in the
spec[1,2,3].

As you'll also see from the spec, nest supports MPI, so we build it 3
times:
- without MPI
- once with openmpi
- once with mpich

For the MPI builds, we need to pass the `--install-lib=$MPI_SITEARCH`
option to `python setup.py install` to get it to install the bits in the
right MPI related locations.

I've noticed that using this system now gives us deprecation warnings
during the build since calling `setup.py ...` explicitly is deprecated.

However, I can't figure out how I would replicate the build using the
`pyproject` macros. I guess `pyproject_wheel` usage is straightforward
(?), but how do I get `pyproject_install` to install in the
$MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to
store the files in three different lists that I can use in the different
`%files` sections?

[1]
https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/0001-Disable-python-s...
[2]
https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_342
[3]
https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_424

[Ankur Sinha]

Quick addendum:

Upstream has moved away from calling `setup.py` and now uses `pip`:

https://github.com/nest/nest-simulator/blob/v3.4/pynest/CMakeLists.txt#L91

Still not sure how to use the pyproject macros though.

On Mon, Jun 12, 2023 15:04:54 +0100, Ankur Sinha wrote:
> Hi folks,
>
> I'm in the process of updating the nest package to the latest release.
> It uses cmake as its build system, but then the cmake calls `python
> setup.py install` and so on to install the Python API. So, we strip that
> bit out and call these commands ourselves in the right places in the
> spec[1,2,3].
>
> As you'll also see from the spec, nest supports MPI, so we build it 3
> times:
> without MPI
> once with openmpi
> once with mpich
>
> For the MPI builds, we need to pass the `--install-lib=$MPI_SITEARCH`
> option to `python setup.py install` to get it to install the bits in the
> right MPI related locations.
>
> I've noticed that using this system now gives us deprecation warnings
> during the build since calling `setup.py ...` explicitly is deprecated.
>
> However, I can't figure out how I would replicate the build using the
> `pyproject` macros. I guess `pyproject_wheel` usage is straightforward
> (?), but how do I get `pyproject_install` to install in the
> $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to
> store the files in three different lists that I can use in the different
> `%files` sections?
>
> [1]
> https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/0001-Disable-python-s...
> [2]
> https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_342
> [3]
> https://src.fedoraproject.org/rpms/nest/blob/rawhide/f/nest.spec#_424
>
> -- 
> Thanks,
> Regards,
> Ankur Sinha "FranciscoD" (He / Him / His) | https://fedoraproject.org/wiki/User:Ankursinha
> Time zone: Europe/London

[Ankur Sinha]

On Mon, Jun 12, 2023 15:25:57 +0100, Ankur Sinha wrote:
> Quick addendum:
>
> Upstream has moved away from calling `setup.py` and now uses `pip`:
>
> https://github.com/nest/nest-simulator/blob/v3.4/pynest/CMakeLists.txt#L91
>
> Still not sure how to use the pyproject macros though.

No, that's wrong. They still call `setup.py install_egg_info` there.

[Miro Hrončok]

On 12. 06. 23 16:04, Ankur Sinha wrote:
> However, I can't figure out how I would replicate the build using the
> `pyproject` macros. I guess `pyproject_wheel` usage is straightforward
> (?), but how do I get `pyproject_install` to install in the
> $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to
> store the files in three different lists that I can use in the different
> `%files` sections?

This is currently not possible. Unfortunately, support for this would require 
somebody else than me to drive it.

(If you just need to install the same files in multiple locations, you can cp 
-a them around after %pyproject_install.)

[Ankur Sinha]

On Mon, Jun 12, 2023 21:07:05 +0200, Miro Hrončok wrote:
> On 12. 06. 23 16:04, Ankur Sinha wrote:
> > However, I can't figure out how I would replicate the build using the
> > `pyproject` macros. I guess `pyproject_wheel` usage is straightforward
> > (?), but how do I get `pyproject_install` to install in the
> > $MPI_SITEARCH locations, and then how do I get `pyproject_save_files` to
> > store the files in three different lists that I can use in the different
> > `%files` sections?
>
> This is currently not possible. Unfortunately, support for this would
> require somebody else than me to drive it.
>
> (If you just need to install the same files in multiple locations, you can
> cp -a them around after %pyproject_install.)

Thanks for your reply, Miro. I guess I can go through the pyproject
macro sources and see if it's possible to replicate their flow in the
spec. If not, I'll try and replace the `setup.py` commands with `pip`
commands in the spec and see how that goes.

[David Schwörer]

Hi Ankur,

Not sure whether that helps, but BOUT++ (requires MPI) switched to a custom back end [1] when it switched to cmake [2] from configure + calling setup.py. The main motivation was to enable packaging in fedora, but it also allowed to just pip install the whole package, as BOUT++ is now PEP-517 enabled.

This doesn't call the pyproject macros, but just the cmake ones [3], so not sure this is applicable to what you need, but maybe it offers at least some insight into how it is done for other MPI codes ...

Best,
David

[1] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
[2] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
[3] https://src.fedoraproject.org/rpms/bout++/blob/rawhide/f/bout++.spec

[Ankur Sinha]

On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote:
> Hi Ankur,

Hi David,

Thanks for your reply.
> Not sure whether that helps, but BOUT++ (requires MPI) switched to a
> custom back end [1] when it switched to cmake [2] from configure +
> calling setup.py. The main motivation was to enable packaging in
> fedora, but it also allowed to just pip install the whole package, as
> BOUT++ is now PEP-517 enabled.
>
> This doesn't call the pyproject macros, but just the cmake ones [3],
> so not sure this is applicable to what you need, but maybe it offers
> at least some insight into how it is done for other MPI codes ...

That's interesting. So it builds the cpython as part of the cmake build
and then simply installs it in the required directory.

How does it generate the dist-info metadata, though? That's the bit
where nest uses `python setup.py install_egg_info`.

https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_info...

https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e237...

[David Schwörer]

> On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote: 
> That's interesting. So it builds the cpython as part of the cmake build
> and then simply installs it in the required directory.
>
> How does it generate the dist-info metadata, though? That's the bit
> where nest uses `python setup.py install_egg_info`.
>
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_i...
>
> https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e...

Unless I am mistaken, that is done in the prepare_metadata_for_build [1], which gets called from here [2]

It is currently missing the readme, thus the project description is empty on pypi [3]. I had troubles finding the appropriate documentation how to do that.

[1] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
[2] https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
[3] https://pypi.org/project/boutpp/

[Ankur Sinha]

On Wed, Jun 14, 2023 12:59:40 -0000, David Schwörer wrote:
> > On Tue, Jun 13, 2023 08:31:49 -0000, David Schwörer wrote: 
> > That's interesting. So it builds the cpython as part of the cmake build
> > and then simply installs it in the required directory.
> >
> > How does it generate the dist-info metadata, though? That's the bit
> > where nest uses `python setup.py install_egg_info`.
> >
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_dist_i...
> >
> > https://github.com/sanjayankur31/nest-simulator/blob/41892a50f95ddf50966e...
>
> Unless I am mistaken, that is done in the prepare_metadata_for_build [1], which gets called from here [2]
>
> It is currently missing the readme, thus the project description is empty on pypi [3]. I had troubles finding the appropriate documentation how to do that.

Ah, indeed it is generated there. I asked because I was looking at the
Fedora package to try and figure it out and couldn't find the metadata
in there:

$ sudo dnf repoquery -l python3-bout++ | grep info
# returns nothing

Thanks again for all your replies. I'm going to spend some time
tinkering with the nest sources to see what I can do.

[David Schwörer]

> $ sudo dnf repoquery -l python3-bout++ | grep info
> # returns nothing
>
> Thanks again for all your replies. I'm going to spend some time
> tinkering with the nest sources to see what I can do.

They should be though in the mpi packages:
$ sudo dnf repoquery -l python3-bout++-mpich | grep info
# returns nothing
But they are there missing as well.
I think the issue is that not the wheel gets installed with cmake, but the installation is done with cmake:
https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...

I just notice that the wheel is not build at all, as that invokes cmake itself, thus a different wheel would need to be build by cmake, but then I am not sure how that can be installed?

[Ankur Sinha]

On Thu, Jun 15, 2023 10:13:28 -0000, David Schwörer wrote:
> > $ sudo dnf repoquery -l python3-bout++ | grep info
> > # returns nothing
> >
> > Thanks again for all your replies. I'm going to spend some time
> > tinkering with the nest sources to see what I can do.
>
> They should be though in the mpi packages:
> $ sudo dnf repoquery -l python3-bout++-mpich | grep info
> # returns nothing
> But they are there missing as well.
> I think the issue is that not the wheel gets installed with cmake, but the installation is done with cmake:
> https://github.com/boutproject/BOUT-dev/blob/master/tools/pylib/_boutpp_buil...
> I just notice that the wheel is not build at all, as that invokes
> cmake itself, thus a different wheel would need to be build by cmake,
> but then I am not sure how that can be installed?

So the wheel is built manually:
https://github.com/boutproject/BOUT-dev/blob/cbd197e78f7d52721188badfd7c38a0...

but I don't see where they then install it. Maybe that's the step that's
missing, and would install the dist-info etc.?

Looking at `pyproject_install`:

$ rpm -E %{pyproject_install}
\
specifier=$(ls /home/asinha/rpmbuild/BUILD/pyproject-wheeldir/*.whl | xargs basename --multiple | sed -E 's/([^-]+)-([^-]+)-.+.whl/\1==\2/')
TMPDIR="/home/asinha/rpmbuild/BUILD/.pyproject-builddir" /usr/bin/python3 -m pip install --root /home/asinha/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 --prefix /usr --no-deps --disable-pip-version-check --progress-bar off --verbose --ignore-installed --no-warn-script-location --no-index --no-cache-dir --find-links /home/asinha/rpmbuild/BUILD/pyproject-wheeldir $specifier

Source here:
https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/macro...

So, for nest, I'm thinking I can build and install the wheel manually
perhaps similar to how I build and install the package manually now,
only at the moment I do it with `setup.py`.