https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x
== Summary == The Python Packaging guidelines will be rewritten, with the major changes being '''PyPI parity''' and usage of '''upstream metadata'''.
A new set of macros, {{package|pyproject-rpm-macros}}, written in mind with the new guidelines and with upstream best practices, are documented in the new guidelines as practical examples.
The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect as an option (until retired by another Fedora Change).
== Owner == * Name: [[User:pviktori| Petr Viktorin]] ** Email: pviktori@redhat.com
* Name: [[SIGs/Python| Python SIG]] ** Email: python-devel@lists.fedoraproject.org
== Detailed Description ==
The proposed new Python packaging guidelines are available at this Pull request: https://pagure.io/packaging-committee/pull-request/1072
A rendered preview (which might be deleted by the time you read this) is at: http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/
=== Not removing older guidelines ===
The current ("201x-era") guidelines will stay in effect as an option for packages that haven't migrated yet or those that cannot follow the new guidelines for whatever unforeseen reason.
They will be retired in another Fedora Change, some time after the vast majority of Python packages follow the new guidelines and there are no known blockers for the remaining ones. There's no rush; it might well take a decade.
=== Guideline Changes ===
See [https://hackmd.io/@python-maint/rJmQQc4DP an external document] for a detailed list of changes to '''MUST'''/'''SHOULD''' rules. The major ones are listed here:
==== Requiring python3-devel ====
All packages that use Python at run- or build-time will need to BuildRequire {{package|python3-devel}}. This package brings the necessary macros and settings, and it will enable automated or manual checks. (For example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes.)
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
==== Upstream metadata ====
Upstream “dist-info” metadata for Python packages is now standardized and shareable with other distributors (be it Linux distros or others). As much as possible, RPM metadata should be automatically taken from there, rather be duplicated in the spec file (and likely diverge over time). This includes run-time and build-time requirements for Python packages, tests and test requirements for packages that use the <code>tox</code> tool, and ''Extras'' (optional features).
Packagers are expected to treat metadata bugs as any other bugs. (That is, ideally: patch them and present the patches upstream).
==== Naming ====
The new guidelines explain the various names a Python package can have (Fedora package name, project name, import name) and begs developers/maintainers to keep them synchronized if possible.
==== Python 3 ====
The new guidelines only cover Python 3.
Python 2 does not need its own guidelines. {{package|python2.7}} is deprecated and as of this writing, only about 35 packages use it, usually under individual FESCo exceptions.
Python 4 is not planned upstream.
==== Tests, linters and coverage ====
Running upstream tests is mandatory. Running linters (e.g. {{package|pylint}}, {{package|pyflakes}}, {{package|python-black}}) or test coverage measurements (e.g. {{package|python-coverage}}) is discouraged.
=== Non-Guideline Changes ===
The {{package|python3-devel}} package, a mandatory build-time requirement for all packages that use Python, will require {{package|pyproject-rpm-macros}}, helpers designed for the new guidelines.
While this small package is not always necessary, it will make life easier for most Python packagers.
{{package|pyproject-rpm-macros}} is currently [https://src.fedoraproject.org/rpms/pyproject-rpm-macros documented as ''provisional'']; that label will be removed and a 1.0 release will be made.
=== Example spec file with changes ===
Example of a spec file following the old guidelines:
Name: python-pello Version: 1.0.2 Release: 1%{?dist} Summary: Example Python library
License: MIT URL: https://github.com/fedora-python/Pello Source0: %{url}/archive/v%{version}/Pello-%{version}.tar.gz
BuildArch: noarch BuildRequires: python3-devel
# Build dependencies needed to be specified manually BuildRequires: python3-setuptools
# Test dependencies needed to be specified manually BuildRequires: python3-pytest >= 3
# Runtime dependencies needed to be specified manually on build time to run tests BuildRequires: python3-blessings
%global _description %{expand: A python module which provides a convenient example. This description provides some details.}
%description %_description
%package -n python3-pello Summary: %{summary} Recommends: python3-pello+color
%description -n python3-pello %_description
# The metadata directory needed to be specified manually for Python extras %python_extras_subpkg -n python3-pello color -i %{python3_sitelib}/*.egg-info
%prep %autosetup -p1 -n Pello-%{version}
%build # The macro only supported projects with setup.py %py3_build
%install # The macro only supported projects with setup.py %py3_install
%check # The old guidelines mentioned one way of running tests which is deprecated # and in many cases lead to unexpected results (such as: 0 tests run, success) # Alternatively, %%pytest can be used with the old macros as well %{python3} setup.py test
%files -n python3-pello %doc README.md %license LICENSE.txt %{_bindir}/pello_greeting
# The library files needed to be listed manually, # often with extra care about the .pyc cache (although not in this example) %{python3_sitelib}/pello/
# The metadata files needed to be listed manually %{python3_sitelib}/Pello-*.egg-info/
Example of a spec file following the new guidelines:
Name: python-pello Version: 1.0.2 Release: 1%{?dist} Summary: Example Python library
License: MIT URL: https://github.com/fedora-python/Pello Source0: %{url}/archive/v%{version}/Pello-%{version}.tar.gz
BuildArch: noarch BuildRequires: python3-devel
%global _description %{expand: A python module which provides a convenient example. This description provides some details.}
%description %_description
%package -n python3-pello Summary: %{summary} Recommends: python3-pello+color
%description -n python3-pello %_description
%pyproject_extras_subpkg -n python3-pello color
%prep %autosetup -p1 -n Pello-%{version}
%generate_buildrequires # The build/test/runtime BuildRequires are generated from upstream metadata %pyproject_buildrequires -t
%build # The macro supports setup.py-based and pyproject.toml-based build %pyproject_wheel
%install # The macro supports setup.py-based and pyproject.toml-based build %pyproject_install
# Library and metadata files can be saved automatically %pyproject_save_files pello
%check # test can run via the upstream de-facto standard way # tox based tests run in Fedora's environment, not a Python virtual environment # for projects without tox, %%pytest is preferred %tox
# For python3-pello, %%{pyproject_files} handles code files, but # executables, documentation and license must be listed in the spec file: %files -n python3-pello -f %{pyproject_files} %doc README.md %license LICENSE.txt %{_bindir}/pello_greeting
== Feedback ==
The new guidelines wede discussed extensively on [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... the Python SIG list discussion thread].
Two major objections remain. Both were best voiced by Neal Gompa:
=== Burdening packagers ===
[https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... Neal] does “not necessarily disagree that PyPI and Fedora pythonXdist() names should match, but [he disagrees] on burdening packagers with this.”
However, actually pushing a package to PyPI (and maintaining it there) is not necessary. The minimal-effort solution is to ''reserve'' the name on PyPI so that a conflicting package can not be created there.
The author of this change is willing to do this work for all Python packages whose upstream is not on PyPI; packagers need to send a request to the Python SIG list. If something happens to me, Red Hat's python-maint team is also ready to do this, and in the event of catastrophic management change, PyPI maintainers can be e-mailed directly.
This is all described in the new guidelines (see the ''PyPI parity'' section).
The new guidelines discourage the minimal-effort solution because it is suboptimal ''for users'' (and for the wider Python ecosystem).
(It was further suggested that the registration process should be ''automatic'' to minimize burden on packagers. It will be automated if it's too much for humans to handle, but we expect there won't be enough requests to make automation worth it.)
=== Conditioning names on PyPI ===
Neal also [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... fundamentally disagrees] “on conditioning those names on PyPI, because frankly no other ecosystem in Fedora works that way. Not Rust, not Go, not Perl, not OCaml, not anything.”
It's true that no other ecosystem in Fedora does this, but the author of this change believes that they should: if the ecosystem uses a flat namespace of dependencies, its Fedora packages ''should'' use the ecosystem-wide namespace rather than a Fedora-specific one. RPM predates global per-ecosystem directories of free/open software, but they are here and users (at least those affected by these issues) are used to them.
For Python specifically, PyPI is already a de-facto standard: if you see a name in <code>install_requires</code> in <code>setup.py</code> of some random upstream project you want to package, you can assume that it's the PyPI name.
It is in the best interest of Fedora's users to use PyPI names for dependency resolution.
== Benefit to Fedora ==
The new packaging guidelines should encourage sharing packaging work with upstream developers (and through them, with other distros). Hopefully, they'll also free up some packagers' time.
PyPI parity will hopefully enable automated ways of mixing Fedora packages with upstream ones.
The updated examples and notes will be more useful to today's packagers.
The older (a.k.a. "201x-era") Python Packaging guidelines only cover projects that use `setup.py`-based installation (via `setuptools` or `distutils`). The new guidelines and macros support more "build backends" (such as `setup.py`-less `setuptools`, `poetry`, `flit` or others) transparently via [https://www.python.org/dev/peps/pep-0517/ PEP 517] (and they continue to work with projects that use `setup.py` to install).
== Scope == * Policies and guidelines: https://pagure.io/packaging-committee/pull-request/1072
* Proposal owners: ** Make {{package|python3-devel}} require {{package|pyproject-rpm-macros}} ** Remove ''provisional'' status from {{package|pyproject-rpm-macros}}; release & package version 1.0. ** Handle PyPI name registration/reservation requests from other packagers.
* Other developers: Maintainers of packages that use Python should gradually switch to the new guidelines
* Release engineering: https://pagure.io/releng/issue/10164 * Trademark approval: N/A (not needed for this Change) * Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
None expected. Any compatibility impact on individual packages would be a bug.
== How To Test ==
If you maintain a package that uses Python, try to follow the new guidelines. Any major omission or ambiguity is a bug. (Minor/unique cases should be discussed on the [mailto:python-devel@lists.fedoraproject.org Pytohn SIG list] instead; the document is already quite long.)
Until this change proposal is approved and enabled, you'll need to BuildRequire `pyproject-rpm-macros` to have the new macros available.
Updated packages should be tested normally, as with any other package update.
== User Experience ==
No direct impact on end users.
== Dependencies ==
PyPI maintainers [https://github.com/pypa/pypi-support/issues/355 are aware] of the general direction the Guidelines, and have been supportive.
== Contingency Plan == * Contingency mechanism: Revert to the older guidelines * Contingency deadline: N/A (Any time until old guidelines are retired, in a separate Change) * Blocks release? No
== Documentation == The new guidelines are the documentation. See ''Detailed Description'' above for links.
== Release Notes == Fedora's Python Packaging guidelines have been rewritten to align with modern Python packaging practices and better integrate with the wider Python ecosystem (specifically, the Python Package Index).
Questions, questions ...
These new guidelines seem to be fine for pure Python packages, but I'm maintaining a couple of packages where Python bindings are built as subpackages of existing C libraries:
https://src.fedoraproject.org/rpms/libnbd https://src.fedoraproject.org/rpms/libguestfs
[Yes there are good reasons for this. No we are not going to decouple them and package the Python stuff separately.]
Running %pyproject_install etc is not really an option for us. However I guess some of these macros could make sense (%pyproject_save_files? %pyproject_files?). Also I don't think these projects generate the extra *.egg-info files or how to create them.
=== Not removing older guidelines === There's no rush; it might well take a decade.
I guess we've got a while ...
While I've got your attention, one other package is interesting:
https://src.fedoraproject.org/rpms/nbdkit
This has Python 3 bindings but they work in completely the opposite way to normal bindings, namely a Python interpreter is embedded in a C program. As far as I know none of the Python guidelines in Fedora address this scenario directly, and indeed we don't currently use any of the special %python* macros.
Rich.
On 14. 06. 21 17:39, Richard W.M. Jones wrote:
Questions, questions ...
These new guidelines seem to be fine for pure Python packages, but I'm maintaining a couple of packages where Python bindings are built as subpackages of existing C libraries:
https://src.fedoraproject.org/rpms/libnbd https://src.fedoraproject.org/rpms/libguestfs
[Yes there are good reasons for this. No we are not going to decouple them and package the Python stuff separately.]
Running %pyproject_install etc is not really an option for us. However I guess some of these macros could make sense (%pyproject_save_files? %pyproject_files?). Also I don't think these projects generate the extra *.egg-info files or how to create them.
Note that the macros are helpers/implementation details. They're not necessary; you can use other tools to be compatible with the guidelines. Some of those tools are still to be written. That's the main reason why adopting the guidelines might take a decade. We do intend to add helpers for more use cases, but it's not a priority right now.
Like the change says, there's no rush; the old guidelines will be retired "some time after the vast majority of Python packages follow the new guidelines and there are no known blockers for the remaining ones". I hope it's clear that I do not expect individual packagers to solve this problem. (Though if y'all want to help, I won't object.)
=== Not removing older guidelines === There's no rush; it might well take a decade.
I guess we've got a while ...
While I've got your attention, one other package is interesting:
https://src.fedoraproject.org/rpms/nbdkit
This has Python 3 bindings but they work in completely the opposite way to normal bindings, namely a Python interpreter is embedded in a C program. As far as I know none of the Python guidelines in Fedora address this scenario directly, and indeed we don't currently use any of the special %python* macros.
Thanks for the examples!
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
From the proposed guidelines:
If a test suite exists upstream, it MUST be run in the %check section and/or in Fedora CI. You MAY exclude specific failing tests.
You MUST NOT disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you MAY disable tests with an appropriate %if conditional (e.g. bcond) when bootstrapping.
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
Assuming only some of the tests require network access, you should exclude them in %check (and possibly run them on Fedora CI instead).
On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
And if I don't want to patch tests in downstream or not interested in runing tests at all?
For example my package wloc[1] has upstream tests, but they cannot run on Fedora due to the lack of a test suite file and secret keys.
Another example is a package that requires special equipment to run tests.
I'm strongly against this part of proposal. The package maintainers should decide whether they need tests or not.
[1]: https://github.com/xvitaly/wloc
On 14. 06. 21 18:32, Vitaly Zaitsev via devel wrote:
On 14.06.2021 17:56, Miro Hrončok wrote:
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
And if I don't want to patch tests in downstream or not interested in runing tests at all?
For example my package wloc[1] has upstream tests, but they cannot run on Fedora due to the lack of a test suite file and secret keys.
Another example is a package that requires special equipment to run tests.
I'm strongly against this part of proposal. The package maintainers should decide whether they need tests or not.
There are two situations described here:
1) Running tests in %check is impossible, unreasonably hard or useless
In that case, I am inclined to agree running them should not be hard requirement, the packagers should instead be required to document the reasoning in the spec file.
2) Packager not interested in running tests at all
That is exactly the thing we need to avoid. Python packages without tests always cause trouble when we measure impact of our changes. E.g. when we continuously rebuild packages with the next Python release, pure Python packages with incompatibilities but without %check always succeed the build, but they break other packages.
Example Python 3.10 failures that only manifest themselves in the depending packages because %check is non-existent or commented out:
https://bugzilla.redhat.com/show_bug.cgi?id=1968737 https://bugzilla.redhat.com/show_bug.cgi?id=1969311 https://bugzilla.redhat.com/show_bug.cgi?id=1919789 https://bugzilla.redhat.com/show_bug.cgi?id=1926112
On 14.06.2021 19:16, Miro Hrončok wrote:
That is exactly the thing we need to avoid. Python packages without tests always cause trouble when we measure impact of our changes. E.g. when we continuously rebuild packages with the next Python release, pure Python packages with incompatibilities but without %check always succeed the build, but they break other packages.
OK, I will orphan all my Python packages than. I don't want to deal with tests at all.
Dne 14. 06. 21 v 19:30 Vitaly Zaitsev via devel napsal(a):
On 14.06.2021 19:16, Miro Hrončok wrote:
That is exactly the thing we need to avoid. Python packages without tests always cause trouble when we measure impact of our changes. E.g. when we continuously rebuild packages with the next Python release, pure Python packages with incompatibilities but without %check always succeed the build, but they break other packages.
OK, I will orphan all my Python packages than. I don't want to deal with tests at all.
I think that "use your bests judgement" still applies. So whatever is in guidelines should be respected, but sometimes there needs to be exceptions.
Nevertheless, I definitely support Python team in making %check section mandatory and should if disabled, it should be well reasoned. This is what Ruby guidelines says on this topic:
~~~
If there is test suite available for the package (even separately, for example not included in the gem but available in the upstream repository), it SHOULD be run in |%check|. The test suite is the only automated tool which can assure basic functionality of the package. Running it is especially helpful when mass rebuilds are required. You MAY skip test suite execution when not all build dependencies are met but this MUST be documented in the specfile. The missing build dependencies to enable the test suite SHOULD be packaged for Fedora as soon as possible and the test suite re-enabled.
~~~
Vít
On 15.06.2021 10:14, Vít Ondruch wrote:
I think that "use your bests judgement" still applies. So whatever is in guidelines should be respected, but sometimes there needs to be exceptions.
The word "MUST" should be replaced by "SHOULD" then.
If upstream tests works fine, I always enable them, but if I need to do a lot of work by patching/rewriting them, I would rather skip them altogether, because I'm not getting paid for this unnecessary work.
Also, some upstream tests are designed for their own CI only. You will need to do a lot of patching to reproduce their build/tests environment.
On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok mhroncok@redhat.com wrote:
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
From the proposed guidelines:
If a test suite exists upstream, it MUST be run in the %check section and/or in Fedora CI. You MAY exclude specific failing tests.
You MUST NOT disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you MAY disable tests with an appropriate %if conditional (e.g. bcond) when bootstrapping.
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
Assuming only some of the tests require network access, you should exclude them in %check (and possibly run them on Fedora CI instead).
This is unreasonable, since Fedora CI isn't usable across all architectures. Unless Fedora CI is at platform parity, then we have to have an allowance for not running tests.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 14. 06. 21 18:35, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok mhroncok@redhat.com wrote:
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
From the proposed guidelines:
If a test suite exists upstream, it MUST be run in the %check section and/or in Fedora CI. You MAY exclude specific failing tests.
You MUST NOT disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you MAY disable tests with an appropriate %if conditional (e.g. bcond) when bootstrapping.
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
Assuming only some of the tests require network access, you should exclude them in %check (and possibly run them on Fedora CI instead).
This is unreasonable, since Fedora CI isn't usable across all architectures. Unless Fedora CI is at platform parity, then we have to have an allowance for not running tests.
Yeah, the situation wrt. platform parity is not good. However I still think that running the network-needing tests on x86_64 only is a magnitude better than "%%check requires internet, commented out".
That said, requiring to run tests on the Fedora CI if they cannot run in %check is a new thing and I would not like the new Python packaging guidelines to be blocked exactly on this. I don't event think this was the intention.
However, I have different reasons than platform parity: I don't want Python packagers to be blocked by figuring out how to do Fedora CI, since it is non-trivial.
Personally, I still prefer tests in %check because they are easily verifiable e.g. in Copr when we try to measure the impact of some of our changes. Tests in the CI are unusable for this use-case, because there is no tooling around it.
I suppose allowing to not run tests (at all) if they (all of them) require resources not available in Koji during rpmbuild (such as internet access or secret keys) is a reasonable compromise, however I'd like to give Petr a chance to pitch in as well.
On ma, 14 kesä 2021, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 11:57 AM Miro Hrončok mhroncok@redhat.com wrote:
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
From the proposed guidelines:
If a test suite exists upstream, it MUST be run in the %check section and/or in Fedora CI. You MAY exclude specific failing tests.
You MUST NOT disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you MAY disable tests with an appropriate %if conditional (e.g. bcond) when bootstrapping.
Assuming that *all the tests* require network access, you should run them on Fedora CI instead.
Assuming only some of the tests require network access, you should exclude them in %check (and possibly run them on Fedora CI instead).
This is unreasonable, since Fedora CI isn't usable across all architectures. Unless Fedora CI is at platform parity, then we have to have an allowance for not running tests.
More to that, for packages like FreeIPA, running a FreeIPA installation -- which is what required to have a reasonable test run -- is simply not possible in %check. We could run a subset of unit tests but that is not going to produce a meaningful result.
Current Fedora CI capabilities also pretty limited with regards to multi-server deployments. We rely on OpenQA testing at the Bodhi update time for a reason.
I agree that running the tests wherever practical is the best practice. I do put my time where my mouth is—I maintain 17 packages that now use the pyproject-rpm-macros, and you’ll find that in general I’ve added a lot of %check sections to packages that were previously lacking them.
However, let me point to pipx as an example where I recently gave up on maintaining an ever-shifting list of 50 tests that require interaction with PyPI (https://src.fedoraproject.org/rpms/pipx/c/55c6c9debf3039747327686a2f07dc31cd...). It just wasn’t worth it to keep doing this in order to run the small minority of tests that can work offline. Even worse, upstream recently added an offline bundle of a huge number of PyPI wheels to make these tests reproducible offline, but this bundle is arch-specific with compiled binary wheels (a non-starter for including as a Source# file), and it would now require a nontrivial patch to run ANY of the tests without either the wheel bundle or a network connection.
Now, running these fussy tests in CI would be great! I would accept a reasonable PR to do so. But I’m not excited about learning Yet Another YAML-Based Programming-As-Configuration Language being an absolute requirement in order to maintain packages that have impractical upstream tests like these. There are an increasing number of upstreams with test suites that are really only designed to run in upstream’s own fickle CI, and reproducing these environments could be quite a burden. Realistically, I picked up pipx and cleaned it up after it was orphaned, and I’m likely just to file a Bugzilla and orphan it again if CI is mandated. Unfortunately, I suspect there are many packagers who would choose quiet noncompliance instead.
Another example is python-strictyaml. It builds its documentation using an idiosyncratic build system (see https://hitchdev.com/) that is hopelessly entangled with the idea of downloading dependencies from PyPI. An offline documentation build is therefore nearly impossible. Currently, it has no tests upstream, but if it ever adds them, they will be run via the same system and will also be effectively impossible to run offline.
A third example is python-pgspecial. Most of the tests require a postgresql database, which I cannot set up in %check because this requires root privileges. This package actually seems like it would be compliant today, because it has a few tests that work without the database connection, and automatically skips the rest—but I am sure there are others where every single test requires a privileged operation. That would be the case for python-databases if it did not support SQLite.
A fourth example is python-absl-py. There are a couple of “smoke tests” that I can and do run, but the core test suite requires Bazel—which, as has been discussed before, will probably never be successfully packaged for Fedora.
I’m still in favor of running every test that is even vaguely practical in %check, but upstream Python packaging practices are wildly diverse (arguably, a mess) and it seems like a strongly worded SHOULD with a fallback of “trust the packager” would be a better approach than forcing packagers to build complicated CI configurations and go to great lengths to implement and debug network-connected tests they cannot reproduce locally.
It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area.
On 14. 06. 21 19:35, Benjamin Beasley wrote:
I’m still in favor of running every test that is even vaguely practical in %check, but upstream Python packaging practices are wildly diverse (arguably, a mess) and it seems like a strongly worded SHOULD with a fallback of “trust the packager” would be a better approach than forcing packagers to build complicated CI configurations and go to great lengths to implement and debug network-connected tests they cannot reproduce locally.
I don't disagree with you.
However I think we should at least strictly require a smoke test (such as %python3 -c "import foo, foo.bar") in such cases, for reasons described below...
It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area.
As written elsewhere in this thread, because we regularly got bitten by pure Python packages without tests.
With Python packages, when there is no (useful) %check, the build basically just copy-pastes some files around and bytecompiles them. If there are no syntax errors, the bytecompilation always succeeds.
Python packages without tests always cause trouble when we measure impact of our changes. E.g. when we continuously rebuild packages with the next Python release, pure Python packages with incompatibilities but without %check always succeed the build, but they break other packages.
Example Python 3.10 failures that only manifest themselves in the depending packages because %check is non-existent or commented out:
https://bugzilla.redhat.com/show_bug.cgi?id=1968737 https://bugzilla.redhat.com/show_bug.cgi?id=1969311 https://bugzilla.redhat.com/show_bug.cgi?id=1919789 https://bugzilla.redhat.com/show_bug.cgi?id=1926112
On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:
On 14. 06. 21 19:35, Benjamin Beasley wrote: However I think we should at least strictly require a smoke test (such as %python3 -c "import foo, foo.bar") in such cases, for reasons described below... […]
It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area.
As written elsewhere in this thread, because we regularly got bitten by pure Python packages without tests. […]
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok
I tentatively agree with the idea of requiring an “import foo.bar” smoke test in cases where the upstream tests cannot be used, especially since pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add runtime dependencies as BR’s. (It may not always be practical to explicitly import all subpackages/modules in a complicated package, but even importing the top-level package is a good start). It would catch a large portion of the FTI bugs that appear in practice.
The NodeJS packaging guidelines have a similar requirement (https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_te...).
Even just mandating that the runtime requirements must be BR’s (%pyproject_buildrequires -r or “better”) would catch most such issues at build time. I did this in pipx for exactly that reason.
– Ben Beasley
On 14. 06. 21 20:28, Ben Beasley wrote:
On Mon, Jun 14, 2021, at 2:07 PM, Miro Hrončok wrote:
On 14. 06. 21 19:35, Benjamin Beasley wrote: However I think we should at least strictly require a smoke test (such as %python3 -c "import foo, foo.bar") in such cases, for reasons described below... […]
It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area.
As written elsewhere in this thread, because we regularly got bitten by pure Python packages without tests. […]
I tentatively agree with the idea of requiring an “import foo.bar” smoke test in cases where the upstream tests cannot be used, especially since pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add runtime dependencies as BR’s. (It may not always be practical to explicitly import all subpackages/modules in a complicated package, but even importing the top-level package is a good start). It would catch a large portion of the FTI bugs that appear in practice.
We could very well add a macro helper that imports modules from the %{buildroot} and:
- when given positional arguments, import the given names - when no arguments given, imports all modules saved by %pyproject_save_files
The NodeJS packaging guidelines have a similar requirement (https://docs.fedoraproject.org/en-US/packaging-guidelines/Node.js/#_build_te...).
Awesome requirement. Will discuss this with Petr.
Even just mandating that the runtime requirements must be BR’s (%pyproject_buildrequires -r or “better”) would catch most such issues at build time. I did this in pipx for exactly that reason.
Recently I was considering if -r shouldn't be the default (with -R or similar for opt-out). That should catch most of the fails to install issues.
On 14. 06. 21 21:00, Miro Hrončok wrote:
I tentatively agree with the idea of requiring an “import foo.bar” smoke test in cases where the upstream tests cannot be used, especially since pyproject-rpm-macros with %pyproject_buildrequires makes it much easier to add runtime dependencies as BR’s. (It may not always be practical to explicitly import all subpackages/modules in a complicated package, but even importing the top-level package is a good start). It would catch a large portion of the FTI bugs that appear in practice.
We could very well add a macro helper that imports modules from the %{buildroot} and:
- when given positional arguments, import the given names
Proof of concept: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
Miro Hrončok mhroncok@redhat.com writes:
On 14. 06. 21 19:35, Benjamin Beasley wrote:
I’m still in favor of running every test that is even vaguely practical in %check, but upstream Python packaging practices are wildly diverse (arguably, a mess) and it seems like a strongly worded SHOULD with a fallback of “trust the packager” would be a better approach than forcing packagers to build complicated CI configurations and go to great lengths to implement and debug network-connected tests they cannot reproduce locally.
I don't disagree with you.
However I think we should at least strictly require a smoke test (such as %python3 -c "import foo, foo.bar") in such cases, for reasons described below...
I would then suggest to change the wording from "Running upstream tests is mandatory." to "Upstream tests SHOULD be run unless there are compelling reasons. In that case basic smoke tests MUST be added to %check".
We could consider suggesting Fedora CI for tests that require network access, but given the voiced concerns its unlikely to be a viable alternative for both package maintainers and the Python SIG conducting Python version bump rebuilds.
Cheers,
Dan
On 14.06.2021 22:33, Dan Čermák wrote:
I would then suggest to change the wording from "Running upstream tests is mandatory." to "Upstream tests SHOULD be run unless there are compelling reasons. In that case basic smoke tests MUST be added to %check".
+1 for this.
I'll address the larger "mandatory tests" issue later; thanks for all your concerns! This point deserves a reply on its own:
On 14. 06. 21 19:35, Benjamin Beasley wrote: [...]
It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area.
Two reasons: - Python is a dynamic language; most errors only happen at runtime. In C, just compiling and linking might work as a reasonable smoke test. - I'd like the Python guidelines to be a bit "ahead" of the Fedora-wide ones. The proposal contains a few ideas that might be better Fedora-wide, but I'd rather introduce and test them in the ecosystem I know.
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
Thanks for this and all the other concerns about mandatory tests! I updated the proposal to mane them not mandatory, but require a smoke test if they're not there. I also removed the mention of Fedora CI: it was meant as an escape hatch for tests that need network, but that's now unnecessary.
Diff: https://pagure.io/fork/pviktori/packaging-committee/c/ec9643873c989e0ff26412... History including other minor changes: https://pagure.io/fork/pviktori/packaging-committee/commits/new-py-guideline...
New wording:
If a test suite exists upstream, it *SHOULD* be run in the `+%check+` section. If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
You *MAY* exclude specific failing tests. You *MUST NOT* disable the entire testsuite or ignore its result to solve a build failure.
As an exception, you *MAY* disable tests with an appropriate `+%if+` conditional (e.g. http://rpm.org/user_doc/conditional_builds.html%5Bbcond]) when xref:index.adoc#bootstrapping[bootstrapping].
Most errors in Python happen at run-time, so tests are extremely important to root out issues, especially when mass rebuilds are required.
Common reasons for skipping tests in `+%check+` include requiring network access, dependencies not packaged in Fedora, and/or specialized hardware or resources.
On 15.06.2021 13:33, Petr Viktorin wrote:
If a test suite exists upstream, it *SHOULD* be run in the `+%check+` section.
LGTM now. Many thanks.
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
And for those of us who also maintain packages for EL7/8, what's the availability of these macros?
On 15. 06. 21 19:34, Ewoud Kohl van Wijngaarden wrote:
And for those of us who also maintain packages for EL7/8, what's the availability of these macros?
Whenever technically possible, we add our new Python macros to EPEL 7+. Sometimes, we even backport them to plain RHEL 8+.
This can (and will) be done for the proposed "import test" macro.
However, the pyproject RPM macros use RPM features not yet available in RHEL 8, so we won't be adding those.
On Tue, Jun 15, 2021 at 7:41 PM Miro Hrončok mhroncok@redhat.com wrote:
On 15. 06. 21 19:34, Ewoud Kohl van Wijngaarden wrote:
And for those of us who also maintain packages for EL7/8, what's the availability of these macros?
Whenever technically possible, we add our new Python macros to EPEL 7+. Sometimes, we even backport them to plain RHEL 8+.
This can (and will) be done for the proposed "import test" macro.
I can assume the new import test macro will be included in Centos stream 9 macros, right?
However, the pyproject RPM macros use RPM features not yet available in RHEL 8, so we won't be adding those.
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 16. 06. 21 10:10, Alfredo Moralejo Alonso wrote:
On Tue, Jun 15, 2021 at 7:41 PM Miro Hrončok <mhroncok@redhat.com mailto:mhroncok@redhat.com> wrote:
On 15. 06. 21 19:34, Ewoud Kohl van Wijngaarden wrote: > And for those of us who also maintain packages for EL7/8, what's the > availability of these macros? Whenever technically possible, we add our new Python macros to EPEL 7+. Sometimes, we even backport them to plain RHEL 8+. This can (and will) be done for the proposed "import test" macro.I can assume the new import test macro will be included in Centos stream 9 macros, right?
Yes, once it is available in Fedora Rawhide, I'll continue with:
- Fedora 34 - Fedora 33 - CentOS Stream 9 - EPEL 8 - EPEL 7
Later, maybe, it might (or not) be added to RHEL 8 proper (and removed from EPEL 8).
That is the normal procedure for new Python macros that are backwards compatible.
Dne 15. 06. 21 v 19:34 Ewoud Kohl van Wijngaarden napsal(a):
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
No, there is no similar macro for Ruby. Historically, it was not clear what should be actually required. With Bundler putting into place more conventions, the situation is better.
Nevertheless, what is the specific issue you are referring to above? There are many examples of fixing such issue, e.g.
https://src.fedoraproject.org/rpms/rubygem-bcrypt/blob/rawhide/f/rubygem-bcr...
And there are probably more complex cases.
And for those of us who also maintain packages for EL7/8, what's the availability of these macros?
RHEL7 is in Maintenance Support 2 Phase [1], so don't expect too much. There are better chances to get such macro into RHEL8, but in the context of Ruby, there are 3 supported versions ATM, therefore it might get complex.
Anyway, it is good idea to use such macros in the following way:
~~~
%{?import_smoke_test}
~~~
This does the right thing where such macro is supported and is ignored elsewhere, where such macro is not available.
Vít
On Wed, Jun 16, 2021 at 01:17:31PM +0200, Vít Ondruch wrote:
Dne 15. 06. 21 v 19:34 Ewoud Kohl van Wijngaarden napsal(a):
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
No, there is no similar macro for Ruby. Historically, it was not clear what should be actually required. With Bundler putting into place more conventions, the situation is better.
Nevertheless, what is the specific issue you are referring to above? There are many examples of fixing such issue, e.g.
https://src.fedoraproject.org/rpms/rubygem-bcrypt/blob/rawhide/f/rubygem-bcr...
And there are probably more complex cases.
Perhaps we should take this to a separate discussion, but we found that there are various bugs in SCL macros that s.o ended up in the wrong directories so it wouldn't load.
Our specific case is that we have our own SCL that depends on rh-ruby27 and must override GEM_PATH[1], GEM_HOME[2] and replace %%gem_install[3]. We forgot this at some point. Providing the explicit paths masked this failure. Perhaps this simply can't work inside RPM builds and you need to really install it in some chroot to test it.
[1]: https://github.com/theforeman/foreman-packaging/blob/7b1dd67c08f4f17b818dd0b... [2]: https://github.com/theforeman/foreman-packaging/blob/7b1dd67c08f4f17b818dd0b... [3]: https://github.com/theforeman/foreman-packaging/blob/7b1dd67c08f4f17b818dd0b...
And for those of us who also maintain packages for EL7/8, what's the availability of these macros?
RHEL7 is in Maintenance Support 2 Phase [1], so don't expect too much. There are better chances to get such macro into RHEL8, but in the context of Ruby, there are 3 supported versions ATM, therefore it might get complex.
Anyway, it is good idea to use such macros in the following way:
%{?import_smoke_test}This does the right thing where such macro is supported and is ignored elsewhere, where such macro is not available.
Thanks! That makes sense.
Dne 16. 06. 21 v 13:36 Ewoud Kohl van Wijngaarden napsal(a):
On Wed, Jun 16, 2021 at 01:17:31PM +0200, Vít Ondruch wrote:
Dne 15. 06. 21 v 19:34 Ewoud Kohl van Wijngaarden napsal(a):
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
No, there is no similar macro for Ruby. Historically, it was not clear what should be actually required. With Bundler putting into place more conventions, the situation is better.
Nevertheless, what is the specific issue you are referring to above? There are many examples of fixing such issue, e.g.
https://src.fedoraproject.org/rpms/rubygem-bcrypt/blob/rawhide/f/rubygem-bcr...
And there are probably more complex cases.
Perhaps we should take this to a separate discussion, but we found that there are various bugs in SCL macros that s.o ended up in the wrong directories so it wouldn't load.
SCLs or not, this is what always happens for several reasons:
1) While upstream leaves the .so files along side other parts of the library, we place the platform independent code in /usr/share while placing the platform dependent code into /usr/lib. This is done via [1] and it was actually supposed to be default in RubyGems 3, but unfortunately with change of upstream maintainers, it got forgotten [2].
2) The paths used during rpmbuild are never on Ruby LOAD_PATH. Upstream might add the `lib` directory on the load path, which be defaults contains the .so file. But that is not case on Fedora as I explained in (1), therefore it is always necessary to add the directory with .so file on the LOAD_PATH explicitly.
In theory, we could somehow extend the GEM_PATH to include the RPM directories, but upstream test suites are not prepared for this scenario, so it would not help.
Really, majority of Ruby packages has fragments like `-Ilib` or `-Ilib:test` in the `%check` section. This is grep across the Fedora .spec files [3]:
~~~
$ find . -name rubygem* -exec grep ' -I' {} ; | wc -l 311
~~~
You just need to add the path to the .so file on the LOAD_PATH similarly.
Vít
[1] https://src.fedoraproject.org/rpms/ruby/blob/rawhide/f/operating_system.rb#_...
[2] https://src.fedoraproject.org/rpms/ruby/blob/rawhide/f/operating_system.rb#_...
[3] https://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
On Wed, Jun 16, 2021 at 02:21:18PM +0200, Vít Ondruch wrote:
Dne 16. 06. 21 v 13:36 Ewoud Kohl van Wijngaarden napsal(a):
On Wed, Jun 16, 2021 at 01:17:31PM +0200, Vít Ondruch wrote:
Dne 15. 06. 21 v 19:34 Ewoud Kohl van Wijngaarden napsal(a):
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote:
>If that is not possible with reasonable effort, >at least a basic smoke test (such as importing the packaged module) >*MUST* be run in `+%check+`.
A simple scriplet should be introduced I think:
%check %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
No, there is no similar macro for Ruby. Historically, it was not clear what should be actually required. With Bundler putting into place more conventions, the situation is better.
Nevertheless, what is the specific issue you are referring to above? There are many examples of fixing such issue, e.g.
https://src.fedoraproject.org/rpms/rubygem-bcrypt/blob/rawhide/f/rubygem-bcr...
And there are probably more complex cases.
Perhaps we should take this to a separate discussion, but we found that there are various bugs in SCL macros that s.o ended up in the wrong directories so it wouldn't load.
SCLs or not, this is what always happens for several reasons:
- While upstream leaves the .so files along side other parts of the
library, we place the platform independent code in /usr/share while placing the platform dependent code into /usr/lib. This is done via [1] and it was actually supposed to be default in RubyGems 3, but unfortunately with change of upstream maintainers, it got forgotten [2].
- The paths used during rpmbuild are never on Ruby LOAD_PATH.
Upstream might add the `lib` directory on the load path, which be defaults contains the .so file. But that is not case on Fedora as I explained in (1), therefore it is always necessary to add the directory with .so file on the LOAD_PATH explicitly.
In theory, we could somehow extend the GEM_PATH to include the RPM directories, but upstream test suites are not prepared for this scenario, so it would not help.
Really, majority of Ruby packages has fragments like `-Ilib` or `-Ilib:test` in the `%check` section. This is grep across the Fedora .spec files [3]:
$ find . -name rubygem\* -exec grep ' \-I' {} \; | wc -l 311You just need to add the path to the .so file on the LOAD_PATH similarly.
That does leave a feeling that we can't test the correct .so location at all in %check.
It feels like we need a phase where the package was installed (respecting %files and all). If we can *then* import the library using Ruby, then we're content. While it may not be RPMs job, an easy verification point that's also usable outside of some fixed infrastructure (such as Fedora) would be great. Is there such a thing today?
Dne 17. 06. 21 v 13:26 Ewoud Kohl van Wijngaarden napsal(a):
On Wed, Jun 16, 2021 at 02:21:18PM +0200, Vít Ondruch wrote:
Dne 16. 06. 21 v 13:36 Ewoud Kohl van Wijngaarden napsal(a):
On Wed, Jun 16, 2021 at 01:17:31PM +0200, Vít Ondruch wrote:
Dne 15. 06. 21 v 19:34 Ewoud Kohl van Wijngaarden napsal(a):
On Tue, Jun 15, 2021 at 01:51:12PM +0200, Miro Hrončok wrote:
On 15. 06. 21 13:46, Vitaly Zaitsev via devel wrote: >> If that is not possible with reasonable effort, >> at least a basic smoke test (such as importing the packaged >> module) >> *MUST* be run in `+%check+`. > > A simple scriplet should be introduced I think: > > %check > %do_import_test
Already on it: https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/99
This may be not be the right place, but in Foreman's Ruby packaging we've also felt a similar pain. Mostly with C extensions that were built in the wrong directory. We also added a %check section to do a basic "import" (require) test.
Is there a similar macro for Ruby smoke testing?
No, there is no similar macro for Ruby. Historically, it was not clear what should be actually required. With Bundler putting into place more conventions, the situation is better.
Nevertheless, what is the specific issue you are referring to above? There are many examples of fixing such issue, e.g.
https://src.fedoraproject.org/rpms/rubygem-bcrypt/blob/rawhide/f/rubygem-bcr...
And there are probably more complex cases.
Perhaps we should take this to a separate discussion, but we found that there are various bugs in SCL macros that s.o ended up in the wrong directories so it wouldn't load.
SCLs or not, this is what always happens for several reasons:
- While upstream leaves the .so files along side other parts of the
library, we place the platform independent code in /usr/share while placing the platform dependent code into /usr/lib. This is done via [1] and it was actually supposed to be default in RubyGems 3, but unfortunately with change of upstream maintainers, it got forgotten [2].
- The paths used during rpmbuild are never on Ruby LOAD_PATH.
Upstream might add the `lib` directory on the load path, which be defaults contains the .so file. But that is not case on Fedora as I explained in (1), therefore it is always necessary to add the directory with .so file on the LOAD_PATH explicitly.
In theory, we could somehow extend the GEM_PATH to include the RPM directories, but upstream test suites are not prepared for this scenario, so it would not help.
Really, majority of Ruby packages has fragments like `-Ilib` or `-Ilib:test` in the `%check` section. This is grep across the Fedora .spec files [3]:
$ find . -name rubygem\* -exec grep ' \-I' {} \; | wc -l 311You just need to add the path to the .so file on the LOAD_PATH similarly.
That does leave a feeling that we can't test the correct .so location at all in %check.
Right, you can't. You never work with correct locations during rpmbuild. The location can be just faked for better or worse. The correct .so location can be tested only via integration tests which are run outside of rpmbuild environment.
Please note that neither upstreams test the "correct" location, they are faking them just to the extend to have their test suite green. That does not mean the test suite necessarily follows all conventions.
It feels like we need a phase where the package was installed (respecting %files and all). If we can *then* import the library using Ruby, then we're content. While it may not be RPMs job, an easy verification point that's also usable outside of some fixed infrastructure (such as Fedora) would be great. Is there such a thing today?
Yes, you can use CI:
https://docs.fedoraproject.org/en-US/ci/
And Fedora actually runs some test by default:
https://bodhi.fedoraproject.org/updates/FEDORA-2021-be0f7a99dd
But there is unfortunately no generic smoke test, which would try to load the library from the right location.
Vít
Petr Viktorin pviktori@redhat.com writes:
On 14. 06. 21 17:52, Vitaly Zaitsev via devel wrote:
On 14.06.2021 15:32, Ben Cotton wrote:
Running upstream tests is mandatory.
What about tests that require network access?
Thanks for this and all the other concerns about mandatory tests! I updated the proposal to mane them not mandatory, but require a smoke test if they're not there. I also removed the mention of Fedora CI: it was meant as an escape hatch for tests that need network, but that's now unnecessary.
Diff: https://pagure.io/fork/pviktori/packaging-committee/c/ec9643873c989e0ff26412... History including other minor changes: https://pagure.io/fork/pviktori/packaging-committee/commits/new-py-guideline...
New wording:
If a test suite exists upstream, it *SHOULD* be run in the `+%check+` section. If that is not possible with reasonable effort, at least a basic smoke test (such as importing the packaged module) *MUST* be run in `+%check+`.
You *MAY* exclude specific failing tests. You *MUST NOT* disable the entire testsuite or ignore its result to solve a build failure.
As an exception, you *MAY* disable tests with an appropriate `+%if+` conditional (e.g. http://rpm.org/user_doc/conditional_builds.html%5Bbcond]) when xref:index.adoc#bootstrapping[bootstrapping].
Most errors in Python happen at run-time, so tests are extremely important to root out issues, especially when mass rebuilds are required.
Common reasons for skipping tests in `+%check+` include requiring network access, dependencies not packaged in Fedora, and/or specialized hardware or resources.
This is much better, thank you! And thanks Miro for the new macro!
Cheers,
Dan
On ma, 14 kesä 2021, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x
== Summary == The Python Packaging guidelines will be rewritten, with the major changes being '''PyPI parity''' and usage of '''upstream metadata'''.
A new set of macros, {{package|pyproject-rpm-macros}}, written in mind with the new guidelines and with upstream best practices, are documented in the new guidelines as practical examples.
The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect as an option (until retired by another Fedora Change).
== Owner ==
- Name: [[User:pviktori| Petr Viktorin]]
** Email: pviktori@redhat.com
- Name: [[SIGs/Python| Python SIG]]
** Email: python-devel@lists.fedoraproject.org
== Detailed Description ==
The proposed new Python packaging guidelines are available at this Pull request: https://pagure.io/packaging-committee/pull-request/1072
A rendered preview (which might be deleted by the time you read this) is at: http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/
=== Not removing older guidelines ===
The current ("201x-era") guidelines will stay in effect as an option for packages that haven't migrated yet or those that cannot follow the new guidelines for whatever unforeseen reason.
They will be retired in another Fedora Change, some time after the vast majority of Python packages follow the new guidelines and there are no known blockers for the remaining ones. There's no rush; it might well take a decade.
=== Guideline Changes ===
See [https://hackmd.io/@python-maint/rJmQQc4DP an external document] for a detailed list of changes to '''MUST'''/'''SHOULD''' rules. The major ones are listed here:
==== Requiring python3-devel ====
All packages that use Python at run- or build-time will need to BuildRequire {{package|python3-devel}}. This package brings the necessary macros and settings, and it will enable automated or manual checks. (For example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes.)
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
=== Burdening packagers ===
[https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... Neal] does “not necessarily disagree that PyPI and Fedora pythonXdist() names should match, but [he disagrees] on burdening packagers with this.”
However, actually pushing a package to PyPI (and maintaining it there) is not necessary. The minimal-effort solution is to ''reserve'' the name on PyPI so that a conflicting package can not be created there.
The author of this change is willing to do this work for all Python packages whose upstream is not on PyPI; packagers need to send a request to the Python SIG list. If something happens to me, Red Hat's python-maint team is also ready to do this, and in the event of catastrophic management change, PyPI maintainers can be e-mailed directly.
This is all described in the new guidelines (see the ''PyPI parity'' section).
The new guidelines discourage the minimal-effort solution because it is suboptimal ''for users'' (and for the wider Python ecosystem).
In case of Samba I would argue that going with anything but minimal-effort is actually suboptimal to users.
(It was further suggested that the registration process should be ''automatic'' to minimize burden on packagers. It will be automated if it's too much for humans to handle, but we expect there won't be enough requests to make automation worth it.)
=== Conditioning names on PyPI ===
Neal also [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... fundamentally disagrees] “on conditioning those names on PyPI, because frankly no other ecosystem in Fedora works that way. Not Rust, not Go, not Perl, not OCaml, not anything.”
It's true that no other ecosystem in Fedora does this, but the author of this change believes that they should: if the ecosystem uses a flat namespace of dependencies, its Fedora packages ''should'' use the ecosystem-wide namespace rather than a Fedora-specific one. RPM predates global per-ecosystem directories of free/open software, but they are here and users (at least those affected by these issues) are used to them.
For Python specifically, PyPI is already a de-facto standard: if you see a name in <code>install_requires</code> in <code>setup.py</code> of some random upstream project you want to package, you can assume that it's the PyPI name.
It is in the best interest of Fedora's users to use PyPI names for dependency resolution.
== Benefit to Fedora ==
The new packaging guidelines should encourage sharing packaging work with upstream developers (and through them, with other distros). Hopefully, they'll also free up some packagers' time.
PyPI parity will hopefully enable automated ways of mixing Fedora packages with upstream ones.
The updated examples and notes will be more useful to today's packagers.
The older (a.k.a. "201x-era") Python Packaging guidelines only cover projects that use `setup.py`-based installation (via `setuptools` or `distutils`). The new guidelines and macros support more "build backends" (such as `setup.py`-less `setuptools`, `poetry`, `flit` or others) transparently via [https://www.python.org/dev/peps/pep-0517/ PEP 517] (and they continue to work with projects that use `setup.py` to install).
== Scope ==
- Policies and guidelines:
https://pagure.io/packaging-committee/pull-request/1072
- Proposal owners:
** Make {{package|python3-devel}} require {{package|pyproject-rpm-macros}} ** Remove ''provisional'' status from {{package|pyproject-rpm-macros}}; release & package version 1.0. ** Handle PyPI name registration/reservation requests from other packagers.
- Other developers: Maintainers of packages that use Python should
gradually switch to the new guidelines
- Release engineering: https://pagure.io/releng/issue/10164
- Trademark approval: N/A (not needed for this Change)
- Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
None expected. Any compatibility impact on individual packages would be a bug.
== How To Test ==
If you maintain a package that uses Python, try to follow the new guidelines. Any major omission or ambiguity is a bug. (Minor/unique cases should be discussed on the [mailto:python-devel@lists.fedoraproject.org Pytohn SIG list] instead; the document is already quite long.)
Until this change proposal is approved and enabled, you'll need to BuildRequire `pyproject-rpm-macros` to have the new macros available.
Updated packages should be tested normally, as with any other package update.
== User Experience ==
No direct impact on end users.
== Dependencies ==
PyPI maintainers [https://github.com/pypa/pypi-support/issues/355 are aware] of the general direction the Guidelines, and have been supportive.
== Contingency Plan ==
- Contingency mechanism: Revert to the older guidelines
- Contingency deadline: N/A (Any time until old guidelines are
retired, in a separate Change)
- Blocks release? No
== Documentation == The new guidelines are the documentation. See ''Detailed Description'' above for links.
== Release Notes == Fedora's Python Packaging guidelines have been rewritten to align with modern Python packaging practices and better integrate with the wider Python ecosystem (specifically, the Python Package Index).
-- Ben Cotton He / Him / His Fedora Program Manager Red Hat TZ=America/Indiana/Indianapolis _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 14. 06. 21 20:09, Alexander Bokovoy wrote:
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
As long as the RPM package doesn't contain dist-info/egg-info that says "this is a Python package called samba" (and hence doesn't provide e.g. python3dist(samba)), this rule does not apply to samba's Python bindings.
On Mon, Jun 14, 2021 at 3:05 PM Miro Hrončok mhroncok@redhat.com wrote:
On 14. 06. 21 20:09, Alexander Bokovoy wrote:
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
As long as the RPM package doesn't contain dist-info/egg-info that says "this is a Python package called samba" (and hence doesn't provide e.g. python3dist(samba)), this rule does not apply to samba's Python bindings.
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I reiterate that I did not *ever* envision the Python SIG using the dependency generator I introduced as a means to act as a gatekeeper for Python software packaging in Fedora.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 6/14/21 1:53 PM, Neal Gompa wrote:
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I don't think we can reasonably separate the two.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
(I believe this is the "namespace" issue mentioned in the proposed change.)
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 6/14/21 1:53 PM, Neal Gompa wrote:
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I don't think we can reasonably separate the two.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
(I believe this is the "namespace" issue mentioned in the proposed change.)
You can. When it comes to distribution packages, we can maintain consistency of the package names referenced within the distribution. The point of this was to be able to use the names Python packages call themselves for dependencies. PyPI is merely a distribution center for such things. Fedora is another distribution center for such things.
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
On 6/14/21 5:11 PM, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
Yes, that was my point. That's exactly how Alex Birsan was able to infiltrate and exploit "dozens" of tech companies.
On Mon, Jun 14, 2021 at 8:35 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 6/14/21 5:11 PM, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
Yes, that was my point. That's exactly how Alex Birsan was able to infiltrate and exploit "dozens" of tech companies.
That only worked because public was default, not private. That's not how it works for Fedora packages at all.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 6/14/21 6:07 PM, Neal Gompa wrote:
That only worked because public was default, not private. That's not how it works for Fedora packages at all.
You're much more experienced in this area than I am, so I'm inclined to think that you're right and I'm missing something. The guidelines do make sense to me though, as a packager, because if a package in Fedora provides python3dist(samba), then I infer that other packages record a dependency on a python package named "samba". And as long as those packages are *only* built as RPMs and installed in Fedora, everything is fine. But if one of those packages is also published on PyPI, and records a dependency on "samba", then anyone who installs that package through "pip" is going to pull in a bad dependency, and we should be trying to avoid that situation, typically by either publishing packages in PyPI or at least reserving the names.
It's unfortunate that there is a "samba" module on PyPI. I don't know if we could negotiate with its owner to transfer that name. The "homepage" links to an absent github project, and the module does not appear to be actively developed.
Hi Neal, We had this conversation in the past (and you can see it in the change). I don't think I can convince you, but I'll reiterate since it's new for devel@.
Unlike the "mandatory tests" issue elsewhere in this thread, using the PyPI namespace is the main point of the change. I can't take it out.
On 15. 06. 21 2:11, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 6/14/21 1:53 PM, Neal Gompa wrote:
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I don't think we can reasonably separate the two.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
(I believe this is the "namespace" issue mentioned in the proposed change.)
You can. When it comes to distribution packages, we can maintain consistency of the package names referenced within the distribution. The point of this was to be able to use the names Python packages call themselves for dependencies. PyPI is merely a distribution center for such things. Fedora is another distribution center for such things.
You can only separate the namespaces if you sacrifice interoberability, i.e. allowing pip-installable packages to depend on Fedora packages. It turns out you can already do that (venv --system-site-packages), and users expect that dist-info project names use the PyPI namespace. This is a current problem that we need to solve.
There are other (harder) solutions for the issue than reusing the PyPI namespace in Fedora, but why should we bother with them? There already is an ecosystem-wide namespace; why should we add a Fedora-specific one? (Or a Linux-distro-specific one, if we're interested in cross-distro collaboration.)
You seem to assume that registering/blocking a name on PyPI is hard. I say it is not and I am willing to do it for packagers who need it. (And to mitigate the bus factor, I'll make it a responsibility of Red Hat's python-maint team.)
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
I agree, in fact, I think Fedora's problems here are a subset of the problems the private organizations have: if issues of proprietary corps are solved, we can use the solution as well. (However, it'll need more work than is necessary for Fedora/FOSS needs, so I don't want to drive the effort.) BUT, if the issues are solved, it'll likely be through namespacing: we'd need to prefix our names with `fedora-` or `fedora:`. I still think it is better for Fedora to reuse the public PyPI namespace rather than start its own.
On Tue, Jun 15, 2021 at 6:50 AM Petr Viktorin pviktori@redhat.com wrote:
Hi Neal, We had this conversation in the past (and you can see it in the change). I don't think I can convince you, but I'll reiterate since it's new for devel@.
Unlike the "mandatory tests" issue elsewhere in this thread, using the PyPI namespace is the main point of the change. I can't take it out.
On 15. 06. 21 2:11, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 6/14/21 1:53 PM, Neal Gompa wrote:
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I don't think we can reasonably separate the two.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
(I believe this is the "namespace" issue mentioned in the proposed change.)
You can. When it comes to distribution packages, we can maintain consistency of the package names referenced within the distribution. The point of this was to be able to use the names Python packages call themselves for dependencies. PyPI is merely a distribution center for such things. Fedora is another distribution center for such things.
You can only separate the namespaces if you sacrifice interoberability, i.e. allowing pip-installable packages to depend on Fedora packages. It turns out you can already do that (venv --system-site-packages), and users expect that dist-info project names use the PyPI namespace. This is a current problem that we need to solve.
There are other (harder) solutions for the issue than reusing the PyPI namespace in Fedora, but why should we bother with them? There already is an ecosystem-wide namespace; why should we add a Fedora-specific one? (Or a Linux-distro-specific one, if we're interested in cross-distro collaboration.)
You seem to assume that registering/blocking a name on PyPI is hard. I say it is not and I am willing to do it for packagers who need it. (And to mitigate the bus factor, I'll make it a responsibility of Red Hat's python-maint team.)
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
I agree, in fact, I think Fedora's problems here are a subset of the problems the private organizations have: if issues of proprietary corps are solved, we can use the solution as well. (However, it'll need more work than is necessary for Fedora/FOSS needs, so I don't want to drive the effort.) BUT, if the issues are solved, it'll likely be through namespacing: we'd need to prefix our names with `fedora-` or `fedora:`. I still think it is better for Fedora to reuse the public PyPI namespace rather than start its own.
As long as you don't make packagers deal with it, I'm fine with it. I'm okay with the PyPI contract as long as you don't make me or any other Fedora packager specifically deal with it. And I strongly suspect you're going to want automation in the future, but that choice is up to you.
-- 真実はいつも一つ！/ Always, there's only one truth!
On 15. 06. 21 13:48, Neal Gompa wrote:
On Tue, Jun 15, 2021 at 6:50 AM Petr Viktorin pviktori@redhat.com wrote:
Hi Neal, We had this conversation in the past (and you can see it in the change). I don't think I can convince you, but I'll reiterate since it's new for devel@.
Unlike the "mandatory tests" issue elsewhere in this thread, using the PyPI namespace is the main point of the change. I can't take it out.
On 15. 06. 21 2:11, Neal Gompa wrote:
On Mon, Jun 14, 2021 at 8:02 PM Gordon Messmer gordon.messmer@gmail.com wrote:
On 6/14/21 1:53 PM, Neal Gompa wrote:
This is completely unreasonable. The dist-info/egg-info data of a Python module is for generating dependencies, not for forcing people to deal with PyPI.
I don't think we can reasonably separate the two.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
(I believe this is the "namespace" issue mentioned in the proposed change.)
You can. When it comes to distribution packages, we can maintain consistency of the package names referenced within the distribution. The point of this was to be able to use the names Python packages call themselves for dependencies. PyPI is merely a distribution center for such things. Fedora is another distribution center for such things.
You can only separate the namespaces if you sacrifice interoberability, i.e. allowing pip-installable packages to depend on Fedora packages. It turns out you can already do that (venv --system-site-packages), and users expect that dist-info project names use the PyPI namespace. This is a current problem that we need to solve.
There are other (harder) solutions for the issue than reusing the PyPI namespace in Fedora, but why should we bother with them? There already is an ecosystem-wide namespace; why should we add a Fedora-specific one? (Or a Linux-distro-specific one, if we're interested in cross-distro collaboration.)
You seem to assume that registering/blocking a name on PyPI is hard. I say it is not and I am willing to do it for packagers who need it. (And to mitigate the bus factor, I'll make it a responsibility of Red Hat's python-maint team.)
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
I agree, in fact, I think Fedora's problems here are a subset of the problems the private organizations have: if issues of proprietary corps are solved, we can use the solution as well. (However, it'll need more work than is necessary for Fedora/FOSS needs, so I don't want to drive the effort.) BUT, if the issues are solved, it'll likely be through namespacing: we'd need to prefix our names with `fedora-` or `fedora:`. I still think it is better for Fedora to reuse the public PyPI namespace rather than start its own.
As long as you don't make packagers deal with it, I'm fine with it. I'm okay with the PyPI contract as long as you don't make me or any other Fedora packager specifically deal with it. And I strongly suspect you're going to want automation in the future, but that choice is up to you.
When packaging something that isn't on PyPI, you can let me know (on python-devel@lists.fedoraproject.org ) and I'll get the name blocked. I expect this to be rare enough to not need automation.
If the name *conflicts* with something on PyPI, then also let me know. Dealing with that issue might be more difficult, and it might involve you more. Still, this situation is confusing for users, so I think it's fair to want to solve it. (And again, I'm not expecting existing packages to do this *right* now.)
On Tue, Jun 15, 2021 at 12:50:03PM +0200, Petr Viktorin wrote:
On 15. 06. 21 2:11, Neal Gompa wrote:
It's not terribly different from how organizations may have private Python package indexes that may use whatever names they want for Python software they build and release.
I agree, in fact, I think Fedora's problems here are a subset of the problems the private organizations have: if issues of proprietary corps are solved, we can use the solution as well. (However, it'll need more work than is necessary for Fedora/FOSS needs, so I don't want to drive the effort.) BUT, if the issues are solved, it'll likely be through namespacing: we'd need to prefix our names with `fedora-` or `fedora:`. I still think it is better for Fedora to reuse the public PyPI namespace rather than start its own.
Registering on PyPI for private packages can be useful to avoid dependency confusion attacks[1]. Essentially we're talking about the same problem here.
[1]: https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
On 14. 06. 21 20:09, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Ben Cotton wrote:
[...]
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
Then I'd recommend that the project name in dist-info metadata would be, for example "samba-bindings". This matters in cases like a pip-installable package declaring a dependency on Samba bindings: it now can't use `samba` because that means something else on PyPI.
I fully agree about not installing Samba itself via pip.
(Also, this is one of the cases that might take a decade -- I'm not putting pressure on you to implement this, but pointing out a direction for the future.)
=== Burdening packagers ===
[https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
Neal] does “not necessarily disagree that PyPI and Fedora pythonXdist() names should match, but [he disagrees] on burdening packagers with this.”
However, actually pushing a package to PyPI (and maintaining it there) is not necessary. The minimal-effort solution is to ''reserve'' the name on PyPI so that a conflicting package can not be created there.
The author of this change is willing to do this work for all Python packages whose upstream is not on PyPI; packagers need to send a request to the Python SIG list. If something happens to me, Red Hat's python-maint team is also ready to do this, and in the event of catastrophic management change, PyPI maintainers can be e-mailed directly.
This is all described in the new guidelines (see the ''PyPI parity'' section).
The new guidelines discourage the minimal-effort solution because it is suboptimal ''for users'' (and for the wider Python ecosystem).
In case of Samba I would argue that going with anything but minimal-effort is actually suboptimal to users.
I agree; Samba is a special case.
On ti, 15 kesä 2021, Petr Viktorin wrote:
On 14. 06. 21 20:09, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Ben Cotton wrote:
[...]
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
Then I'd recommend that the project name in dist-info metadata would be, for example "samba-bindings".
Samba Python bindings have no dist-info metadata. Does it need to? The bindings are consumed by few projects around Samba but none of them has anything related on PyPI. What exists on PyPI is either an outdated stuff without upstream sources or a reference to a similar outdated stuff, none of which is using Samba Python bindings.
This matters in cases like a pip-installable package declaring a dependency on Samba bindings: it now can't use `samba` because that means something else on PyPI.
There are already unrelated projects with Samba name on PyPI. This change in Fedora guidelines cannot affect those. It cannot also affect actual use of Samba Python bindings. It looks like there is no real use of PyPI for our use case either, so why the fuss for this particular package?
I suspect at best we'd submit a request for exception, if pressed.
I fully agree about not installing Samba itself via pip.
(Also, this is one of the cases that might take a decade -- I'm not putting pressure on you to implement this, but pointing out a direction for the future.)
Frankly, I do not see a real use case for what you might see in other projects. We had a similar issue with FreeIPA and what we provide on PyPI is a crippled subset of FreeIPA Python modules that mostly does not use real FreeIPA features. To date, I am not sure it is really bringing any benefit to us (upstream).
This does not mean the whole effort is useless, not at all. It means, in my opinion, that proposed guidelines are failing to address real world use cases present in Fedora. Invalidating these use cases is as bad as dropping the guidelines' changes in one go. ;)
On 15. 06. 21 13:32, Alexander Bokovoy wrote:
On ti, 15 kesä 2021, Petr Viktorin wrote:
On 14. 06. 21 20:09, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Ben Cotton wrote:
[...]
==== PyPI Parity ====
Machine-readable metadata (''distribution'' names in <code>dist-info</code> directories on disk and the corresponding <code>python3.Xdist(foo)</code> RPM provides) will match the Python Package Index (PyPI).
This solves a ''namespace'' issue. Python packaging tools use a flat namespace, and PyPI is ''the'' place where open-source Python packages are generally published, so users and tools assume a package called <code>requests</code> is whatever <code>requests</code> means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.
Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.
The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.
Note that names found in Fedora but not on PyPI [https://github.com/pypa/pypi-support/issues/355 have been reserved on PyPI] to avoid being taken by unrelated projects.
samba has extensive Python C bindings but does not use PyPI at all. We don't want to, we don't need to, it is technically not possible without building Samba from scratch and it would not make it usable for PIP install without a stricter coordination of the non-Python dependencies -- that's what Linux distributions do.
In addition, 'samba' name is taken by an unrelated package on PyPI which was not updated since 2019. For us this namespacing enforcement would only be a problem.
Then I'd recommend that the project name in dist-info metadata would be, for example "samba-bindings".
Samba Python bindings have no dist-info metadata. Does it need to? The bindings are consumed by few projects around Samba but none of them has anything related on PyPI. What exists on PyPI is either an outdated stuff without upstream sources or a reference to a similar outdated stuff, none of which is using Samba Python bindings.
It doesn't need to have dist-info metadata, but I believe it would be better if it did, so that it's *possible* to encode "requires Samba" in Python packaging metadata. (*Should* the bindings be consumed by more than a few projects around Samba? If yes, it dist-info would make it easier.)
There should also be a good way to add dist-info metadata to projects *like* Samba, even if Samba packagers decide they don't want it.
This matters in cases like a pip-installable package declaring a dependency on Samba bindings: it now can't use `samba` because that means something else on PyPI.
There are already unrelated projects with Samba name on PyPI. This change in Fedora guidelines cannot affect those. It cannot also affect actual use of Samba Python bindings. It looks like there is no real use of PyPI for our use case either, so why the fuss for this particular package?
I suspect at best we'd submit a request for exception, if pressed.
No pressure; this is exactly why the old guidelines aren't going anywhere.
I fully agree about not installing Samba itself via pip.
(Also, this is one of the cases that might take a decade -- I'm not putting pressure on you to implement this, but pointing out a direction for the future.)
Frankly, I do not see a real use case for what you might see in other projects. We had a similar issue with FreeIPA and what we provide on PyPI is a crippled subset of FreeIPA Python modules that mostly does not use real FreeIPA features. To date, I am not sure it is really bringing any benefit to us (upstream).
This does not mean the whole effort is useless, not at all. It means, in my opinion, that proposed guidelines are failing to address real world use cases present in Fedora. Invalidating these use cases is as bad as dropping the guidelines' changes in one go. ;)
Don't put Samba or FreeIPA up on PyPI. But provide a name in dist-info, so packages on PyPI can depend on Samba/FreeIPA (which need to be installed separately). And reserve that name, so no one else can put it on PyPI either.
Again, no pressure, just a direction for the future.
-
Alexander Bokovoy -
Alfredo Moralejo Alonso -
Ben Beasley -
Ben Cotton -
Benjamin Beasley
-
Dan Čermák -
Ewoud Kohl van Wijngaarden -
Gordon Messmer -
Miro Hrončok -
Neal Gompa -
Petr Viktorin -
Richard W.M. Jones -
Vitaly Zaitsev -
Vít Ondruch