Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats.
I'm kind of out of the loop on the state of the art of python packaging, and wondered if some kind Python SIG person would like to take it on for me.
On Fri, Feb 19, 2021 at 4:57 PM Matthew Miller mattdm@fedoraproject.org wrote:
Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats.
I'm kind of out of the loop on the state of the art of python packaging, and wondered if some kind Python SIG person would like to take it on for me.
It looks like it's just a classic setuptools build, so the same packaging style that's been in use for ~5+ years should work fine here.
On 19. 02. 21 23:42, Neal Gompa wrote:
On Fri, Feb 19, 2021 at 4:57 PM Matthew Miller mattdm@fedoraproject.org wrote:
Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats.
I'm kind of out of the loop on the state of the art of python packaging, and wondered if some kind Python SIG person would like to take it on for me.
It looks like it's just a classic setuptools build, so the same packaging style that's been in use for ~5+ years should work fine here.
Side note: Carfull, the package bundles fontawesome.
On 19. 02. 21 22:57, Matthew Miller wrote:
Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats.
I'm kind of out of the loop on the state of the art of python packaging, and wondered if some kind Python SIG person would like to take it on for me.
Hey Matthew.
I'm sure that if you hit specific issues when trying to package PyWaffle, some kind Python SIG person will help you to resolve them.
My personal opinion is that packing Python libraries that are not required by packaged applications/tools brings very little benefit to the project. I suggest to use a Python virtual environment instead and install the libraries you need with pip.
See Fedora Developer portal for more info:
https://github.com/developer-portal/content/blob/master/tech/languages/pytho...
(Unfortunately the content has not yet been refreshed on the website, hence sharing a link to GitHub instead.)
Hi! I'm using this https://github.com/gyli/PyWaffle for some visualizations for Fedora Project stats.
So, I've finally gotten around to packaging this up -- review request at:
https://bugzilla.redhat.com/show_bug.cgi?id=2033481
it was a little annoying as I had to de-bundle fontawesome, but I think it's all covered.
The modern Python macros are ... kind of a lot to pick up as a casual packager, but I think I've gotten it right. If someone could review, I'd appreciate it!
On 17. 12. 21 17:58, Matthew Miller wrote:
The modern Python macros are ... kind of a lot to pick up as a casual packager
If you could share a more constructive feedback, I'd gladly iterate on the documentation or the macros themselves.
On Sun, Dec 19, 2021 at 12:14:29PM +0100, Miro Hrončok wrote:
On 17. 12. 21 17:58, Matthew Miller wrote:
The modern Python macros are ... kind of a lot to pick up as a casual packager
If you could share a more constructive feedback, I'd gladly iterate on the documentation or the macros themselves.
I guess I mostly meant that literally: it's a lot! I started using pyp2rpm, which says
Convert Python packages to RPM SPECFILES. The packages can be downloaded from PyPI and the produced SPEC is in line with Fedora Packaging Guidelines or Mageia Python Policy.
... which was probably my first mistake, because that uses a previous generation, and I should know better than to trust that random package descriptions are current. :)
I should have just thrown all of that out and gone with the template https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_empty_spe... .
On that template in specific:
1. It's not immediately clear that "..." means "put package name here".
2. You mentioned in the review request — and Zbyszek said something similar — that it'd be better to repeat the package name throughout the spec file in the various place it's needed. That doesn't feel right to me.
I know the macros replace a lot of boilerplate, which is a good direction, but having the name be different in every file _throughout_ the file makes it harder to pick up when something is different from standard. It seems that a lot of python packages could be identical _except_ for the header. (And maybe then all of that eventually hidden completely with just some thing near the top that indicates "standard python package".)
3. The machinery to make every package be python-... while generating an actual python3-... subpackage is awkward. Are we stuck with that forever?
Overall in the guideline doc, back to the "it's a lot!" feeling...
What do you think about having a "Quick Version" at the top, which includes the spec file template, followed by a line-by line commentary — or at least, section by section. Like this:
Name: python-... Version: ... Release: 0%{?dist} Summary: ...
Packages primarily providing a python library must have names starting with `python-` and that should be followed by the name of the main importable module. Applications which happen to be written in Python should follow the standard Fedora guidelines. See [Naming] for details
Version, Release, and Summary are as standard for Fedora packages.
License: ... URL: https://... Source0: %{url}/archive/v%{version}/...-%{version}.tar.gz / %{pypi_source ...}
These are also as normal, but also see [Source files from PyPI].
BuildArch: noarch BuildRequires: gcc
Pick **one** of the above, depending on the package. (Note on the generate-buildrequires macro here. Will that work for archful devel package requirements?)
BuildRequires: python3-devel
This is mandatory for every package that uses Python at runtime.
... and so on.
In general, I agree that the docs could use a short "tl;dr — the common case" section at the top. This also applies to the "simplified" versioning guidelines with full version info the the Version field. I wrote a chunk of the latest text and I'm having trouble finding the right paragraph in all that prose. The descriptions that we have are correct, but just not very easy to consume casually. This is part of a bigger problem: docs have regularly been coming up in various surveys as the number one problem in Fedora.
IMO, one of the causes is that we've been keeping the descriptions of older workflows. In case of Versioning it's because some ancient RHEL baggage doesn't support the new stuff, and also because some people just *like* the old approach. In case of Python macros, it's because not all packages can use the new build systems, etc. But I think this isn't working, and we should just settle on the best version and prune the rest or move to appendices. People can look at archived versions of the docs if they want to do technoarcheology.
Right now, rpmdevtools recommends 'rm -rf $RPM_BUILD_ROOT' and outdated python macros. Almost no guides have been updated for rpmautospec. (Though thankfully at least fedora-review has been fixed not to warn about %autorelease.) Etc, etc. Experienced packagers know what to ignore, but new packagers are very likely to fall into the numerous traps we set out for them.
On Sun, Dec 19, 2021 at 02:26:15PM -0500, Matthew Miller wrote:
- You mentioned in the review request — and Zbyszek said something similar — that it'd be better to repeat the package name throughout the spec
file in the various place it's needed. That doesn't feel right to me.
I know the macros replace a lot of boilerplate, which is a good direction, but having the name be different in every file _throughout_ the file makes it harder to pick up when something is different from standard. It seems that a lot of python packages could be identical _except_ for the header. (And maybe then all of that eventually hidden completely with just some thing near the top that indicates "standard python package".)
I find 'pywaffle' easier to read that '%{pypi_name}'. When I see e.g. '%{_bindir}/%{pypi_name}', I *can* do the substitutions in my head and arrive at '/usr/bin/pywaffle', but it's certainly a bit of an overhead.
Using macros for things that *change* is reasonable: we certainly wouldn't want to search&replace the version string when it changes during version upgrades, and we can't hardcode the library path, because it changes between architectures. But for things that *never change* during the lifetime of the package, what is the point of using macros? When you create a package for a different python module, you're going to start afresh. The spec file should be as readable as possible for _that particular package_. This also means not using unnecessary macros and making the spec file as short as is reasonable.
It seems that a lot of python packages could be identical _except_ for the header.
Not really. There's still a lot of variation between packages: even Python packages have different build systems, various workarounds and required, etc. The spec files have been becoming shorter, but right now it isn't practical to completely hide all the details. If the technology improves, _and_ upstreams clean up their metadata, maybe we'll be able to handle common Python packaging scenarios with a one-line boilerplate.
But right now we're not there yet. In your spec file, we have 'pywaffle', '%{pypi_name}', and '%{githubname}' used in various places. When reading the spec file, I need to jump up to the header to check the definition of the macros to remember which one is which.
(There's also the tiny annoyance that I can't ctrl-click on https://github.com/gyli/%%7Bgithubname%7D to open the repo, I need to do the substitution… Firefox doesn't understand the macro either.)
- The machinery to make every package be python-... while generating an actual python3-... subpackage is awkward. Are we stuck with that forever?
Yes, I think we'll have this for the foreseeable future. On EPEL, you can have multiple python versions, so we *need* the subpackages there. Maybe we'll be able to hide the binary package name once we have package generators (https://github.com/rpm-software-management/rpm/issues/329).
Zbyszek
On Mon, Dec 20, 2021 at 08:48:17AM +0000, Zbigniew Jędrzejewski-Szmek wrote:
IMO, one of the causes is that we've been keeping the descriptions of older workflows.
Yeah, absolutely. I see that some of that was moved to a "201x-era" entirely separate doc — it might be good to be more aggressive with that.
[...]
Using macros for things that *change* is reasonable: we certainly wouldn't want to search&replace the version string when it changes during version upgrades, and we can't hardcode the library path, because it changes between architectures. But for things that *never change* during the lifetime of the package, what is the point of using macros? When you create a package for a different python module, you're going to start afresh. The spec file should be as readable as possible for _that
I think a lot of people actually _do_ want to start with a working example and modify. Actually, my first instinct was to start with the "python-pello" example https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_example_s... rather than from the blank one.
- The machinery to make every package be python-... while generating an actual python3-... subpackage is awkward. Are we stuck with that forever?
Yes, I think we'll have this for the foreseeable future. On EPEL, you can have multiple python versions, so we *need* the subpackages there. Maybe we'll be able to hide the binary package name once we have package generators (https://github.com/rpm-software-management/rpm/issues/329).
That would be amazing.
On 19. 12. 21 20:26, Matthew Miller wrote:
On Sun, Dec 19, 2021 at 12:14:29PM +0100, Miro Hrončok wrote:
On 17. 12. 21 17:58, Matthew Miller wrote:
The modern Python macros are ... kind of a lot to pick up as a casual packager
If you could share a more constructive feedback, I'd gladly iterate on the documentation or the macros themselves.
I guess I mostly meant that literally: it's a lot! I started using pyp2rpm, which says
Convert Python packages to RPM SPECFILES. The packages can be downloaded from PyPI and the produced SPEC is in line with Fedora Packaging Guidelines or Mageia Python Policy.
... which was probably my first mistake, because that uses a previous generation, and I should know better than to trust that random package descriptions are current. :)
OK, let me try to correct that:
https://src.fedoraproject.org/rpms/pyp2rpm/pull-request/2
But there are plenty of other tools that you can use to generate a wrong specfile. It is hard to keep them up to date -- e.g. rpmdevtools maintains compatibility with other distros without our macros.
I should have just thrown all of that out and gone with the template https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_empty_spe...
Yes, that's why the empty spec file exists in the first place, so packagers can copy and adapt.
On that template in specific:
- It's not immediately clear that "..." means "put package name here".
Oh. See, to me this was rather obvious. What do you actually expected it meant? Do you have an idea how to make it more clear?
You mentioned in the review request — and Zbyszek said something similar — that it'd be better to repeat the package name throughout the spec file in the various place it's needed. That doesn't feel right to me.
I know the macros replace a lot of boilerplate, which is a good direction, but having the name be different in every file _throughout_ the file makes it harder to pick up when something is different from standard. It seems that a lot of python packages could be identical _except_ for the header. (And maybe then all of that eventually hidden completely with just some thing near the top that indicates "standard python package".)
As does Zbyszek, I think this is a common misconception. It seems that packagers assume that if they define %pypi_name, the spec file is somewhat more universal. It kinda makes sense, I used to do it as well.
Except suddenly, you also need to define %github_name, which happens to have a different case. And %module_name, because the imported module does not contain the py prefix. Or there are multiple modules shipped in this package.... and so on. And as soon as you start defining 2, 3, ... more different macros to mean "some kind of name", the spec file turns out to be much less readable and less universal.
In reality, I have not seen many identical Python packages except for the header that were simple or readable.
It's much easier to search for the name(s) and replace them once, if you are into cop-pasting existing spec files into new packages, rather than the need to do it over and over again in your head.
This idea that packagers could copy one specfile and change header to make it another package is also one reason why packagers keep adding stuff that has not been necessary for years.
- The machinery to make every package be python-... while generating an actual python3-... subpackage is awkward. Are we stuck with that forever?
We are stuck with it for now.
Overall in the guideline doc, back to the "it's a lot!" feeling...
I guess that's because they were written as guidelines and not as a tutorial. Writing a tutorial and having that in the preamble of the guidelines is actuall an idea that has been lurking in my head for a while.
What do you think about having a "Quick Version" at the top, which includes the spec file template, followed by a line-by line commentary — or at least, section by section. Like this:
Name: python-... Version: ... Release: 0%{?dist} Summary: ... Packages primarily providing a python library must have names starting with `python-` and that should be followed by the name of the main importable module. Applications which happen to be written in Python should follow the standard Fedora guidelines. See [Naming] for details Version, Release, and Summary are as standard for Fedora packages. License: ... URL: https://... Source0: %{url}/archive/v%{version}/...-%{version}.tar.gz / %{pypi_source ...} These are also as normal, but also see [Source files from PyPI]. BuildArch: noarch BuildRequires: gcc Pick **one** of the above, depending on the package. (Note on the generate-buildrequires macro here. Will that work for archful devel package requirements?)
I am not sure I understand this question. You can use the %pyproject_buildrequires macro with archful packages, but it will only generate requirements on other Python packages(setuptools, Cython, etc.). If your package needs libmagic-devel to build, you still need to BuildRequire that one manually.
BuildRequires: python3-devel This is mandatory for every package that uses Python at runtime. ... and so on.
Let's do that. I've put that on my TODO.
On Mon, Dec 20, 2021 at 06:53:59PM +0100, Miro Hrončok wrote:
- It's not immediately clear that "..." means "put package name here".
Oh. See, to me this was rather obvious. What do you actually expected it meant? Do you have an idea how to make it more clear?
Maybe something like "{...}", with a note at the top?
In reality, I have not seen many identical Python packages except for the header that were simple or readable.
Okay, I'll follow your lead on this then. :)
Pick **one** of the above, depending on the package. (Note on thegenerate-buildrequires macro here. Will that work for archful devel package requirements?)
I am not sure I understand this question. You can use the %pyproject_buildrequires macro with archful packages, but it will only generate requirements on other Python packages(setuptools, Cython, etc.). If your package needs libmagic-devel to build, you still need to BuildRequire that one manually.
Yep, that was my question. :)
Let's do that. I've put that on my TODO.
Cool, thanks!
On 20. 12. 21 21:04, Matthew Miller wrote:
On Mon, Dec 20, 2021 at 06:53:59PM +0100, Miro Hrončok wrote:
- It's not immediately clear that "..." means "put package name here".
Oh. See, to me this was rather obvious. What do you actually expected it meant? Do you have an idea how to make it more clear?
Maybe something like "{...}", with a note at the top?
We can add a note if you think it'll help. OTOH I don't think the curly brackets make it any more obvious. It resembles macro syntax, but not quite.
On Mon, Dec 20, 2021 at 10:50:39PM +0100, Miro Hrončok wrote:
Maybe something like "{...}", with a note at the top?
We can add a note if you think it'll help. OTOH I don't think the curly brackets make it any more obvious. It resembles macro syntax, but not quite.
Or [...] or even PACKAGENAME. The ellipsis is just too easy to miss.
python-devel@lists.fedoraproject.org
-
Matthew Miller -
Miro Hrončok -
Neal Gompa -
Zbigniew Jędrzejewski-Szmek