Hi all,
I'm sorry in advance for the wall of text below, but I think it adds some necessary context to this issue.
The cargo RFC for "Packages as (optional) namespaces" was recently approved, and will likely be implemented in a future version of cargo, and on the official package registry, crates.io:
c.f. https://github.com/rust-lang/cargo/issues/12901 c.f. https://github.com/rust-lang/rfcs/pull/3243
With this being an *optional* namespacing, this raises a few problems for packaging. For example, the 1:1 mapping of crate name <-> package name is no longer possible, since there will be zero or one namespace associated with crates. I have found no other language-specific package registries that use "zero or one" namespaces other than NPM, so I don't think there's good "prior art" for this problem. Other language-specific registries I've looked at either use *no* namespacing at all, or always use *exactly one* level of namespacing - but not "zero or one", as will likely be implemented in the Rust crate ecosystem.
After having been made aware of this RFC, I commented on the upstream tickets, warning them that this is a rather unusual setup, and will likely cause issues with how Fedora packages Rust crates. One of the Rust package maintainers from Debian added a similar comment. It looks like these decisions were made without considering the broader ecosystem and / or downstream consumers of Rust crates, and I only became aware of this "pending approval" RFC very late in the process, since there doesn't seem to have been any outreach to downstream consumers of Rust crates (neither any Fedora, nor any Debian Rust packagers were aware).
With the RFC having been approved, it now raises the issue of how to represent the new optional namespace name in package names in Fedora. There was some discussion in the upstream ticket(s) about this, and I don't much like my conclusions - but here's a summary:
- The only characters allowed in names of crates that are published on crates.io are a-z, A-Z, 0-9, hyphen, and underscore. These are all acceptable for RPM package names, which has allowed us to map "project published as $crate" -> "crate packaged as rust-$crate" without collisions (disregarding any possible collisions due to compat version suffices, which luckily has not happened yet). - The proposed separator between crate namespace and crate name is `::`(two colons, just like the import path separators in "use" statements in Rust code). This is not a valid character in RPM package names. - The intersection of the three sets "ascii characters", "characters that are valid in RPM package names", and "characters that cannot be part of a crate name" is vanishingly small. To the best of my knowledge, the intersection only contains two characters - the plus sign `+` and the dot `.`. - Rust crates as they are packaged in Fedora already use the `+` character as a separator for a different purpose in "feature subpackages'' (i.e. rust-$crate+$feature-devel), similarly to how Python packages handle "extras" with additional subpackages.
As best as I can tell, this leaves us with only two options:
The first option would be to adopt the `.` character as the separator character between crate namespace and crate name, as that appears to be the only choice that does not lead to any ambiguities (hyphen and underscore characters can be part of both crate namespace and crate name, and the plus sign is already used for a different purpose) - i.e. source packages being named like `rust-$namespace.$crate`, and subpackages being named like `rust-$namespace.$crate+$feature-devel`.
The second option would be to "overload" the meaning of the `+` sign, and use it both as a separator between "crate namespace" and "crate name" *and* as the separator between "crate name" and "feature name". In this case, both the source package name and the binary package names will contain the `+` character. Using the `.` as the new separator, the source package names would not contain any `+` characters.
However, this would not result in unambiguous package names in all cases (in particular, for "binary" packages). A `+` sign in the source package name unambiguously means that the project consists of both namespace *and* name, but in built "-devel" packages, only the occurrence of two `+` characters is unambiguous. However, for "-devel" packages that contain only one `+` character, it is possible for both `rust-$namespace+$crate-devel` (i.e. the "main" binary package, which encodes all mandatory dependencies for project "$namespace::$crate") and `rust-$crate+$feature-devel` (i.e. the binary package that encodes dependencies for the feature "$feature" for project "$crate") to coincide, with two source packages for different projects resulting in binary packages of the same name.
I also don't consider this an "unlikely" thing to happen - rather, I think it will be *very likely*. For example, a future `serde::derive` crate and the subpackage for the "derive" feature of `serde` crate would both result in binary packages named `rust-serde+derive-devel`. Using the `.` as the separator character, these two cases would result in unique package names instead (i.e. `rust-serde.derive-devel` and `rust-serde+derive-devel`, respectively).
So, the only choice that does not lead to ambiguities and name clashes appears to be to use the `.` character as the new separator between namespace name and crate name.
The Packaging Guidelines are not really helpful in this case - they seem to only stipulate that the plus sign `+`, the dot `.`, and the underscore `_` MUST NOT be used as separators when packaging a project, but this only seems to apply to the source package name (?) if the upstream project's name consists of multiple parts, and already doesn't fit with "extra" feature packages like the ones that are used in Python or Rust packaging already. If we really need to adopt the `.` character as a new separator for namespaced Rust crates, I will ask the FPC to amend this section of the Packaging Guidelines to allow uses case like this, where there really is no other option: https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_separator...
I'm open to discuss other options, but I don't see any. Please let me know what you think.
Fabio
On Fri, Nov 17, 2023 at 4:37 PM Fabio Valentini decathorpe@gmail.com wrote:
Hi all,
I'm sorry in advance for the wall of text below, but I think it adds some necessary context to this issue.
The cargo RFC for "Packages as (optional) namespaces" was recently approved, and will likely be implemented in a future version of cargo, and on the official package registry, crates.io:
c.f. https://github.com/rust-lang/cargo/issues/12901 c.f. https://github.com/rust-lang/rfcs/pull/3243
With this being an *optional* namespacing, this raises a few problems for packaging. For example, the 1:1 mapping of crate name <-> package name is no longer possible, since there will be zero or one namespace associated with crates. I have found no other language-specific package registries that use "zero or one" namespaces other than NPM, so I don't think there's good "prior art" for this problem. Other language-specific registries I've looked at either use *no* namespacing at all, or always use *exactly one* level of namespacing - but not "zero or one", as will likely be implemented in the Rust crate ecosystem.
After having been made aware of this RFC, I commented on the upstream tickets, warning them that this is a rather unusual setup, and will likely cause issues with how Fedora packages Rust crates. One of the Rust package maintainers from Debian added a similar comment. It looks like these decisions were made without considering the broader ecosystem and / or downstream consumers of Rust crates, and I only became aware of this "pending approval" RFC very late in the process, since there doesn't seem to have been any outreach to downstream consumers of Rust crates (neither any Fedora, nor any Debian Rust packagers were aware).
With the RFC having been approved, it now raises the issue of how to represent the new optional namespace name in package names in Fedora. There was some discussion in the upstream ticket(s) about this, and I don't much like my conclusions - but here's a summary:
- The only characters allowed in names of crates that are published on
crates.io are a-z, A-Z, 0-9, hyphen, and underscore. These are all acceptable for RPM package names, which has allowed us to map "project published as $crate" -> "crate packaged as rust-$crate" without collisions (disregarding any possible collisions due to compat version suffices, which luckily has not happened yet).
- The proposed separator between crate namespace and crate name is
`::`(two colons, just like the import path separators in "use" statements in Rust code). This is not a valid character in RPM package names.
- The intersection of the three sets "ascii characters", "characters
that are valid in RPM package names", and "characters that cannot be part of a crate name" is vanishingly small. To the best of my knowledge, the intersection only contains two characters - the plus sign `+` and the dot `.`.
- Rust crates as they are packaged in Fedora already use the `+`
character as a separator for a different purpose in "feature subpackages'' (i.e. rust-$crate+$feature-devel), similarly to how Python packages handle "extras" with additional subpackages.
As best as I can tell, this leaves us with only two options:
The first option would be to adopt the `.` character as the separator character between crate namespace and crate name, as that appears to be the only choice that does not lead to any ambiguities (hyphen and underscore characters can be part of both crate namespace and crate name, and the plus sign is already used for a different purpose) - i.e. source packages being named like `rust-$namespace.$crate`, and subpackages being named like `rust-$namespace.$crate+$feature-devel`.
The second option would be to "overload" the meaning of the `+` sign, and use it both as a separator between "crate namespace" and "crate name" *and* as the separator between "crate name" and "feature name". In this case, both the source package name and the binary package names will contain the `+` character. Using the `.` as the new separator, the source package names would not contain any `+` characters.
However, this would not result in unambiguous package names in all cases (in particular, for "binary" packages). A `+` sign in the source package name unambiguously means that the project consists of both namespace *and* name, but in built "-devel" packages, only the occurrence of two `+` characters is unambiguous. However, for "-devel" packages that contain only one `+` character, it is possible for both `rust-$namespace+$crate-devel` (i.e. the "main" binary package, which encodes all mandatory dependencies for project "$namespace::$crate") and `rust-$crate+$feature-devel` (i.e. the binary package that encodes dependencies for the feature "$feature" for project "$crate") to coincide, with two source packages for different projects resulting in binary packages of the same name.
I also don't consider this an "unlikely" thing to happen - rather, I think it will be *very likely*. For example, a future `serde::derive` crate and the subpackage for the "derive" feature of `serde` crate would both result in binary packages named `rust-serde+derive-devel`. Using the `.` as the separator character, these two cases would result in unique package names instead (i.e. `rust-serde.derive-devel` and `rust-serde+derive-devel`, respectively).
So, the only choice that does not lead to ambiguities and name clashes appears to be to use the `.` character as the new separator between namespace name and crate name.
The Packaging Guidelines are not really helpful in this case - they seem to only stipulate that the plus sign `+`, the dot `.`, and the underscore `_` MUST NOT be used as separators when packaging a project, but this only seems to apply to the source package name (?) if the upstream project's name consists of multiple parts, and already doesn't fit with "extra" feature packages like the ones that are used in Python or Rust packaging already. If we really need to adopt the `.` character as a new separator for namespaced Rust crates, I will ask the FPC to amend this section of the Packaging Guidelines to allow uses case like this, where there really is no other option: https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_separator...
I'm open to discuss other options, but I don't see any. Please let me know what you think.
The guidelines in question need to be dropped anyway, since we already use dots and underscores in source package names for a variety of ecosystems. I don't see a reason to maintain an illusion that they cannot be used.
-
Fabio Valentini -
Neal Gompa