Hello fellow Python packagers. This is an announcement about a new set of RPM macros you can use to build PEP 517/518 enabled packages, that is Python packages that have the pyproject.toml file.
https://www.python.org/dev/peps/pep-0517/ https://www.python.org/dev/peps/pep-0518/ https://snarky.ca/clarifying-pep-518/
The set of macros is designed for modern packaging with dynamic buildrequires in mind. The macros are in the pyproject-rpm-macros package and you can use them like this:
BuildRequires: pyproject-rpm-macros
%prep ...
%generate_buildrequires %pyproject_buildrequires -t
%build %pyproject_wheel
%install %pyproject_install
%check %tox
See the full documentation of the macros: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/master/f/README...
See example spec files: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/master/f/tests
(These use setuptools (setup.py), flit and poetry for build backends, but you cannot tell that from the specfiles - BuildRequires are generated dynamically from upstream metadata.)
The macros are **provisional**, i.e. their API may be changed upon feedback received from you. We are not (yet) interested in a general "update all the Python packages" hunt, but rather in early adopters.
If you have questions, ask here. We'll gladly extend the docs if something is not clear. If you find bugs, report them in bugzilla or here. Likewise for RFEs.
Happy hacking, Python Maintenance
Hi,
the Python (3) bindings are missing on RHEL/CentOS/EPEL 8 for the protobuf package (https://src.fedoraproject.org/rpms/protobuf). A bug request has been created on Bugzilla ( https://bugzilla.redhat.com/show_bug.cgi?id=1765844), but as no status has been given, I was wondering whether someone could shed some light on the context.
Since protobuf is a RedHat core package (maintained by RedHat and therefore not managed by Fedora/EPEL), it appears as a kind of black box from Fedora perspective. On Fedora (Rawhide, 31), the Python (3) bindings are generated/packaged (see for instance https://bodhi.fedoraproject.org/updates/FEDORA-2019-e3a662fe8b and https://koji.fedoraproject.org/koji/rpminfo?rpmID=19440119), but for some reason, those Python bindings are not generated by RedHat for RHEL/CentOS 8.
1. Would anyone from RedHat be able to provide some heads up on why those Python 3 bindings are missing for Protobuf, and/or an approximate timeline for when it would be generated? 2. Would RedHat need help with packaging protobuf on RHEL/CentOS/EPEL 8? 3. Would you recommend another way for Fedora packagers/users to get their hands on the python3-protobuf/protobuf-python3 package? For instance, through COPR, or some module we may have missed.
Kind regards
Denis
Hi, Miro.
On Friday, 04 October 2019 at 13:50, Miro Hrončok wrote:
Hello fellow Python packagers. This is an announcement about a new set of RPM macros you can use to build PEP 517/518 enabled packages, that is Python packages that have the pyproject.toml file.
Thanks for your (and whoever else's) work on this. This is exactly the kind of automation we need to make packaging easier.
Having said that, I tried to use it in one of my packages (python-gsd) which started providing the pyproject.toml file in a recent release.
Two observations:
1. It actually generates more BRs than I specify manually (I had 3). It adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
2. It would be useful if it generated the file list automatically, too. I had to drop .egg-info and .dist-info manually.
Perhaps I'll try to use the macros in a package with more BRs again.
Regards, Dominik
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
- It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
python3-pytoml or python3-toml is used to parse the pyproject.toml file. Unfortunately, while the toml format has been selected for this file, a standardized python packaging file, there is no toml parser in the stdlib. This might change in the future some day, but this is what we have now.
Toml parsing is bundled in pip. We could possibly use that instead of an additional package, but the API for this is "private" and might change any time.
pip itself is used to build the package via pyproject.toml specified build backend. We could possibly re-implement everything in the pyproject macros project, but given that pip is driving the pyproject.toml standard, we would just be in constant need to follow up pip recent development. Not to mention NIH. We didnt want to go that way.
Setuptools and wheel is your build backend specified in pyproject.toml (or the assumed default). If the upstream chooses to use e.g. flit, sheel would not be needed.
Technically, the package gained more dependencies when compared with plain "python setup.py build/install". However it is now built differently, more close to how things are done in upstream. Frankly speaking the "python setup.py build/install" is becoming so heavily obsoleted that I suspect modern projects to drop setup.py soon-ish.
The entire idea here is that there are more build backends to choose from. Hence depending on the upstream choices, you'll get:
(py)toml - to parse the files in our macros pip - the build frontend "whatever" - the build backed
- It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with something like this:
%files -n python3-foo -f %{pyproject_files}
Note however that it has a certain drawback. If your "foo" package decides to suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm, you will not be able to prevent it via a more strict file rules. That's why I would also like to offer more "soft" option of this, that would be used somewhat like this:
%files -n python3-foo -f %{pyproject_metadata} %{_bindir}/abc %{python3_sitelib}/xyz/
I.e. it would only contain the dist-info directory. Possibly pycache.
With the dist info directory file list generated (in both concepts), we would be able to do things like this:
%dir %{python_sitlib}/foo-%{py_version}.dist-info %license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt %{python_sitlib}/foo-%{py_version}.dist-info/...
That would however require an upstream way to mark the license file as a license file. I plan to explore that option early next year.
On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok mhroncok@redhat.com wrote:
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
- It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
python3-pytoml or python3-toml is used to parse the pyproject.toml file. Unfortunately, while the toml format has been selected for this file, a standardized python packaging file, there is no toml parser in the stdlib. This might change in the future some day, but this is what we have now.
Has anyone considered proposing a PEP to mainline a TOML parser implementation into the standard library yet? We're now nearly two years into PEP 517 being in force and with Python 2's EOL, I suspect we'll see the classical setup.py method go away very rapidly. I know that for my personal projects, the only reason I haven't migrated to Poetry is because I'm being asked to keep Python 2 compatibility upstream and dealing with environments running Python 2 makes pyproject.toml stuff annoying (as they're usually quite old and don't support it).
Toml parsing is bundled in pip. We could possibly use that instead of an additional package, but the API for this is "private" and might change any time.
pip itself is used to build the package via pyproject.toml specified build backend. We could possibly re-implement everything in the pyproject macros project, but given that pip is driving the pyproject.toml standard, we would just be in constant need to follow up pip recent development. Not to mention NIH. We didnt want to go that way.
Setuptools and wheel is your build backend specified in pyproject.toml (or the assumed default). If the upstream chooses to use e.g. flit, sheel would not be needed.
Technically, the package gained more dependencies when compared with plain "python setup.py build/install". However it is now built differently, more close to how things are done in upstream. Frankly speaking the "python setup.py build/install" is becoming so heavily obsoleted that I suspect modern projects to drop setup.py soon-ish.
The entire idea here is that there are more build backends to choose from. Hence depending on the upstream choices, you'll get:
(py)toml - to parse the files in our macros pip - the build frontend "whatever" - the build backed
- It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with something like this:
%files -n python3-foo -f %{pyproject_files}
Note however that it has a certain drawback. If your "foo" package decides to suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm, you will not be able to prevent it via a more strict file rules. That's why I would also like to offer more "soft" option of this, that would be used somewhat like this:
%files -n python3-foo -f %{pyproject_metadata} %{_bindir}/abc %{python3_sitelib}/xyz/
I.e. it would only contain the dist-info directory. Possibly pycache.
With the dist info directory file list generated (in both concepts), we would be able to do things like this:
%dir %{python_sitlib}/foo-%{py_version}.dist-info %license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt %{python_sitlib}/foo-%{py_version}.dist-info/...
That would however require an upstream way to mark the license file as a license file. I plan to explore that option early next year.
I'm generally not a fan of the whole generated filelist thing, but I'll reserve judgement until we have a better idea of what this would look like.
On 30. 11. 19 18:54, Neal Gompa wrote:
On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok mhroncok@redhat.com wrote:
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
- It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
python3-pytoml or python3-toml is used to parse the pyproject.toml file. Unfortunately, while the toml format has been selected for this file, a standardized python packaging file, there is no toml parser in the stdlib. This might change in the future some day, but this is what we have now.
Has anyone considered proposing a PEP to mainline a TOML parser implementation into the standard library yet? We're now nearly two years into PEP 517 being in force and with Python 2's EOL, I suspect we'll see the classical setup.py method go away very rapidly. I know that for my personal projects, the only reason I haven't migrated to Poetry is because I'm being asked to keep Python 2 compatibility upstream and dealing with environments running Python 2 makes pyproject.toml stuff annoying (as they're usually quite old and don't support it).
Yes and no. See
https://mail.python.org/archives/list/python-ideas@python.org/thread/HOZNLCL...
https://github.com/python/peps/issues/1133
- It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with something like this:
%files -n python3-foo -f %{pyproject_files}Note however that it has a certain drawback. If your "foo" package decides to suddenly starts installing %{python3_sitelib}/requests or even %{_bindir}/rpm, you will not be able to prevent it via a more strict file rules. That's why I would also like to offer more "soft" option of this, that would be used somewhat like this:
%files -n python3-foo -f %{pyproject_metadata} %{_bindir}/abc %{python3_sitelib}/xyz/I.e. it would only contain the dist-info directory. Possibly pycache.
With the dist info directory file list generated (in both concepts), we would be able to do things like this:
%dir %{python_sitlib}/foo-%{py_version}.dist-info %license %{python_sitlib}/foo-%{py_version}.dist-info/LICENSE.txt %{python_sitlib}/foo-%{py_version}.dist-info/...That would however require an upstream way to mark the license file as a license file. I plan to explore that option early next year.
I'm generally not a fan of the whole generated filelist thing, but I'll reserve judgement until we have a better idea of what this would look like.
The general idea I start to like is that the "allowed" (in Fedora) option would be %{pyproject_metadata} and that would only ever include the dist-info directory and assertions would be made about it, such as it is only one of them.
Technically all files macro would be possible for fully automated setups, but I would probably like it banned in regular Fedora.
That is however just my personal view, not an actual plan.
Le samedi 30 novembre 2019 à 12:54 -0500, Neal Gompa a écrit :
On Sat, Nov 30, 2019 at 12:49 PM Miro Hrončok mhroncok@redhat.com wrote:
On 30. 11. 19 17:26, Dominik 'Rathann' Mierzejewski wrote:
- It would be useful if it generated the file list
automatically, too. I had to drop .egg-info and .dist-info manually.
It would. I don't know if we track this somewhere but it is certainly on our radar. Patrik (CCed) is working on it.
I have done some API drafting and based on RPM limitations, we might end up with something like this:
You have exemples of working implementations in go-rpm-macros and fonts-rpm-macros https://pagure.io/fonts-rpm-macros/ https://pagure.io/go-rpm-macros/ (see rpm/templates in those projects to look how the macros are used)
Basically, you need to generate a set of rpm variables that contain at least the target subpackage name, and a way to deduce from it the filelist name (ideally, another clean rpm variable easy to manipulate)
Then you just define a <foopkgfile> macro that generates the %file call for all <foopkg> variable sets.
As long a you do it this way it’s fairly easy to manipulate the filelist in %build (or even %install) if the automated process made mistakes.
But, generally, a well tested automated generation process will make less errors than a human toiling by rote.
Regards,
On Sat, Nov 30, 2019 at 05:26:14PM +0100, Dominik 'Rathann' Mierzejewski wrote:
Hi, Miro.
On Friday, 04 October 2019 at 13:50, Miro Hrončok wrote:
Hello fellow Python packagers. This is an announcement about a new set of RPM macros you can use to build PEP 517/518 enabled packages, that is Python packages that have the pyproject.toml file.
Thanks for your (and whoever else's) work on this. This is exactly the kind of automation we need to make packaging easier.
Having said that, I tried to use it in one of my packages (python-gsd) which started providing the pyproject.toml file in a recent release.
Two observations:
- It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
BR or R? If it just a few additional BR on small python packages, I don't it matters much.
- It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
That, OTOH, sounds like a bug...
Zbyszek
On Sunday, 01 December 2019 at 00:07, Zbigniew Jędrzejewski-Szmek wrote:
On Sat, Nov 30, 2019 at 05:26:14PM +0100, Dominik 'Rathann' Mierzejewski wrote:
[...]
Two observations:
- It actually generates more BRs than I specify manually (I had 3). It
adds python3-wheel and it brings in python3-pip and python3-pytoml on it own, so my package ends up with 4 additional BRs for no apparent gain.
BR or R? If it just a few additional BR on small python packages, I don't it matters much.
BR and it matters in this case, because it increases the number of BRs by 100%, precisely because it is a small package. It makes the build setup slower, especially on ARM.
- It would be useful if it generated the file list automatically, too.
I had to drop .egg-info and .dist-info manually.
I meant:
I had to drop .egg-info and *add* .dist-info manually.
That, OTOH, sounds like a bug...
Nah. It's just the difference between setup.py and pyproject builds.
Regards, Dominik
-
Denis Arnaud -
Dominik 'Rathann' Mierzejewski -
Miro Hrončok -
Neal Gompa -
Nicolas Mailhot -
Zbigniew Jędrzejewski-Szmek