Re: Rust bindings for Python (pyo3 versions <0.19, cpython) broken with Python 3.12
by Fabio Valentini
On Fri, Nov 17, 2023 at 1:34 AM Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> On Mon, Nov 13, 2023 at 07:25:10PM +0100, Fabio Valentini wrote:
> >
> > Yup, I've mentioned that in the bug I filed for python-bcrypt -
> > It might be as simple as bumping the dependency on pyo3 from v0.15 to v0.19.
>
> I've made an attempt here:
> https://src.fedoraproject.org/rpms/python-bcrypt/pull-request/9
All dependent packages have now moved off of pyo3 v0.15 to at least
pyo3 v0.19.2+ or the latest v0.20, both of which should be fine with
Python 3.12.
Thank you all for your help! I will retire the packages for pyo3 v0.15
from rawhide now.
Fabio
4 months, 4 weeks
Rust crates with optional namespacing and how to map them to package names
by Fabio Valentini
Hi all,
I'm sorry in advance for the wall of text below, but I think it adds
some necessary context to this issue.
The cargo RFC for "Packages as (optional) namespaces" was recently
approved, and will likely be implemented in a future version of cargo,
and on the official package registry, crates.io:
c.f. https://github.com/rust-lang/cargo/issues/12901
c.f. https://github.com/rust-lang/rfcs/pull/3243
With this being an *optional* namespacing, this raises a few problems
for packaging. For example, the 1:1 mapping of crate name <-> package
name is no longer possible, since there will be zero or one namespace
associated with crates. I have found no other language-specific
package registries that use "zero or one" namespaces other than NPM,
so I don't think there's good "prior art" for this problem. Other
language-specific registries I've looked at either use *no*
namespacing at all, or always use *exactly one* level of namespacing -
but not "zero or one", as will likely be implemented in the Rust crate
ecosystem.
After having been made aware of this RFC, I commented on the upstream
tickets, warning them that this is a rather unusual setup, and will
likely cause issues with how Fedora packages Rust crates. One of the
Rust package maintainers from Debian added a similar comment. It looks
like these decisions were made without considering the broader
ecosystem and / or downstream consumers of Rust crates, and I only
became aware of this "pending approval" RFC very late in the process,
since there doesn't seem to have been any outreach to downstream
consumers of Rust crates (neither any Fedora, nor any Debian Rust
packagers were aware).
With the RFC having been approved, it now raises the issue of how to
represent the new optional namespace name in package names in Fedora.
There was some discussion in the upstream ticket(s) about this, and I
don't much like my conclusions - but here's a summary:
- The only characters allowed in names of crates that are published on
crates.io are a-z, A-Z, 0-9, hyphen, and underscore. These are all
acceptable for RPM package names, which has allowed us to map "project
published as $crate" -> "crate packaged as rust-$crate" without
collisions (disregarding any possible collisions due to compat version
suffices, which luckily has not happened yet).
- The proposed separator between crate namespace and crate name is
`::`(two colons, just like the import path separators in "use"
statements in Rust code). This is not a valid character in RPM package
names.
- The intersection of the three sets "ascii characters", "characters
that are valid in RPM package names", and "characters that cannot be
part of a crate name" is vanishingly small. To the best of my
knowledge, the intersection only contains two characters - the plus
sign `+` and the dot `.`.
- Rust crates as they are packaged in Fedora already use the `+`
character as a separator for a different purpose in "feature
subpackages'' (i.e. rust-$crate+$feature-devel), similarly to how
Python packages handle "extras" with additional subpackages.
As best as I can tell, this leaves us with only two options:
The first option would be to adopt the `.` character as the separator
character between crate namespace and crate name, as that appears to
be the only choice that does not lead to any ambiguities (hyphen and
underscore characters can be part of both crate namespace and crate
name, and the plus sign is already used for a different purpose) -
i.e. source packages being named like `rust-$namespace.$crate`, and
subpackages being named like `rust-$namespace.$crate+$feature-devel`.
The second option would be to "overload" the meaning of the `+` sign,
and use it both as a separator between "crate namespace" and "crate
name" *and* as the separator between "crate name" and "feature name".
In this case, both the source package name and the binary package
names will contain the `+` character. Using the `.` as the new
separator, the source package names would not contain any `+`
characters.
However, this would not result in unambiguous package names in all
cases (in particular, for "binary" packages). A `+` sign in the source
package name unambiguously means that the project consists of both
namespace *and* name, but in built "-devel" packages, only the
occurrence of two `+` characters is unambiguous. However, for "-devel"
packages that contain only one `+` character, it is possible for both
`rust-$namespace+$crate-devel` (i.e. the "main" binary package, which
encodes all mandatory dependencies for project "$namespace::$crate")
and `rust-$crate+$feature-devel` (i.e. the binary package that encodes
dependencies for the feature "$feature" for project "$crate") to
coincide, with two source packages for different projects resulting in
binary packages of the same name.
I also don't consider this an "unlikely" thing to happen - rather, I
think it will be *very likely*. For example, a future `serde::derive`
crate and the subpackage for the "derive" feature of `serde` crate
would both result in binary packages named `rust-serde+derive-devel`.
Using the `.` as the separator character, these two cases would result
in unique package names instead (i.e. `rust-serde.derive-devel` and
`rust-serde+derive-devel`, respectively).
So, the only choice that does not lead to ambiguities and name clashes
appears to be to use the `.` character as the new separator between
namespace name and crate name.
The Packaging Guidelines are not really helpful in this case - they
seem to only stipulate that the plus sign `+`, the dot `.`, and the
underscore `_` MUST NOT be used as separators when packaging a
project, but this only seems to apply to the source package name (?)
if the upstream project's name consists of multiple parts, and already
doesn't fit with "extra" feature packages like the ones that are used
in Python or Rust packaging already. If we really need to adopt the
`.` character as a new separator for namespaced Rust crates, I will
ask the FPC to amend this section of the Packaging Guidelines to allow
uses case like this, where there really is no other option:
https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_separa...
I'm open to discuss other options, but I don't see any. Please let me
know what you think.
Fabio
5 months, 1 week
Rust bindings for Python (pyo3 versions <0.19, cpython) broken with Python 3.12
by Fabio Valentini
Hello Pythonistas and Rustaceans,
TL;DR: Only PyO3 v0.19.2 (and later) will ever properly support Python
3.12. Port your Python projects to v0.19 **NOW**.
Older versions of PyO3 (especially pyo3 v0.15, v0.16, v0.17, and
v0.18) are *not* compatible with Python 3.12 due to some ABI changes
in unicode strings and behavioural changes wrt/ "immortal" objects.
This also affects all current versions of the "cpython" Rust bindings,
with no timeline for Python 3.12 support.
As far as I can tell, extensions that use pyo3 < v0.19 or the
"cpython" bindings can (and likely will) not work as expected on
Python 3.12 if they use the affected APIs (either by producing garbage
data in strings that are passed over the FFI boundary, or by
crashing).
There are applications in Fedora that still rely on *ancient* versions
of PyO3, potentially affected by this:
- cpython: mercurial
- pyo3 v0.15: fapolicy-analyzer, python-bcrypt, python-cryptography
- pyo3 v0.16: python-y-py
- pyo3 v0.17: unused compat packages, will be retired
- pyo3 v0.18: matrix-synapse
I *stongly* recommend to move all of these packages to pyo3 v0.19 in
Rawhide as soon as possible. I will try to submit pull requests with
the required changes for affected packages (except mercurial, since
there's no version of the "cpython" crate that supports Python 3.12 in
sight).
There's already a few packages that depend on pyo3 v0.19, which I will
rebuild in rawhide for pyo3 v0.19.2, which has much better support for
Python 3.12 than v0.19.0 and v0.19.1 (breezy, python-rpds-py, orjson)
unless there are any objections.
As soon as no packages depend on the compat packages for old versions
of pyo3 any longer, I will retire them from Rawhide (and F39,
depending on the timing), since they will never work with Python 3.12
and nothing should use them.
I've added <package>-maintainers(a)fedoraproject.org for all these
packages to the CC of this message.
Fabio
Rust SIG / PyO3 maintainer in Fedora
5 months, 2 weeks