Rust packaging tools "next generation" ready for testing

[Fabio Valentini]

Hello fellow Rustaceans and / or frustrated Rust packagers,

My work on the "next generation" of Rust packaging tools has finally
progressed to the point where it can be tested by actual people. :)

Adding support for new features of cargo - "namespaced dependencies"
(`dep:foo`) and "weak dependency features" (`foo?/bar`) syntax for
feature dependencies; both stable since Rust 1.60.0 - proved to be
very difficult in the existing rust2rpm code base, which is why I
implemented a new backend for RPM BuildRequires, Requires, and
Provides generators (the cargo2rpm binary and Python package).

I have already adapted all RPM macros to depend on cargo2rpm instead
of rust2rpm / cargo-inspector. Building RPM packages no longer pulls
in any part of rust2rpm, which is only needed for generating spec
files (and patches) now.

For now, only changes that were absolutely necessary to make rust2rpm
work for the new cargo features were implemented. In the near future,
I plan to pull out the old cargo metadata handling from rust2rpm, and
replace it with API calls to cargo2rpm.

Project structure and packaging now looks like this:

- cargo2rpm: provides /usr/bin/cargo2rpm and cargo2rpm Python package
for low-level functionality
- rust-packaging: provides rust-rpm-macros, rust-srpm-macros, and
cargo-rpm-macros; depends on cargo2rpm for implementing the backend
for some RPM macros and generators
- rust2rpm: spec file generator (some parts of the backend are still
going to be modernized and / or replaced by functions now implemented
in cargo2rpm)

I've prepared a COPR with test builds of these three packages:
https://copr.fedorainfracloud.org/coprs/decathorpe/rust-packaging-ng/moni...

Running test builds against these new RPM macros is easy with this mock config:
https://github.com/decathorpe/rust-packaging-ng/blob/main/copr.cfg

For testing spec file generation, packages from the COPR will need to
be installed on the host system where "rust2rpm" is run.

For example: After enabling the COPR repo and updating all packages to
the latest version, running "rust2rpm" for existing Rust package
should produce the same results as before, and for crates that use the
new syntax in their feature dependencies, it should produce valid spec
files.

Building the packages against the new RPM macro implementations can
then be done with something like "mock -r ./path/to/copr.cfg
./path/to/src.rpm` (with paths pointing at the mock config linked
above and the SRPM to be built, respectively).

All packages should have their Provides and Obsoletes set up correctly
for upgrades from existing installs of rust2rpm to work as expected.
Version numbers are also intended to be transparently upgraded to the
"stable" release of these tools once they are pushed to Fedora.

At this point, I still expect there to be some rough edges. I only
finished implementing the final pieces earlier today. Please report
back if you see anything that doesn't work as expected. The  new code
in cargo2rpm has over 95% test coverage - getting more weird data as
input to tickle out the last edge cases would be great ;)

The latest version of the RPM macros now lives in a repository
separate from rust2rpm: https://pagure.io/fedora-rust/rust-packaging

The cargo2rpm executable and Python package are maintained here:
https://pagure.io/fedora-rust/cargo2rpm

And rust2rpm continues to be maintained here (though I have not pushed
my latest patches there yet, and the RPM macros are still in the repo
even though I moved further development to a separate project):
https://pagure.io/fedora-rust/rust2rpm

The spec files and patches for the packages in COPR are also public:
https://github.com/decathorpe/rust-packaging-ng

I will continue working on these three components until I think the
new versions / packages are ready to be pushed to rawhide, which I
will announce in advance.

Until the updates are ready, please DO NOT push packages that use
`dep:foo` or `foo?/bar` syntax in their feature dependencies to
rawhide. Doing so *might* work with or without workarounds, but the
resulting packages *will definitely be broken*.

Happy testing,
Fabio

[Fabio Valentini]

On Sat, Feb 11, 2023 at 2:40 AM Fabio Valentini <decathorpe(a)gmail.com&gt; wrote:
>  Hello fellow Rustaceans and / or frustrated Rust packagers,
>
>  My work on the "next generation" of Rust packaging tools has finally
>  progressed to the point where it can be tested by actual people. :) 
After a few more days of code cleanups, bug fixes, more testing, and
even more regression fixes ... I think everything is ready now.
cargo2rpm v0.1.1, rust-packaging v24, and rust2rpm v24.0.1 will be
available in rawhide later today.

I have done extensive testing of the new RPM macro and generator
implementations in COPR, and have verified that rust2rpm v24 produces
spec files that are either equivalent to those from earlier versions,
or actually produce working spec files for projects where earlier
versions crashed or produced broken spec files. I really hope that
this update will not be disruptive, even though almost all of the code
involved was either refactored or rewritten entirely.

I will monitor koschei for any breakages for a while before I push
these updates to stable branches as well (i.e. at least a few days for
f38, and even longer for f37 and f36).

But why push a big update like that to non-rawhide branches at all,
you might ask?
First, all these components are either only used at build-time, or are
tools which are exclusively used by Rust packagers, so the update
should not affect any users of Fedora. Second, without these updates,
some Rust libraries will not be update-able in stable branches at all
(this might also include security fixes), which is a situation that I
would like to avoid.

If you spot any regressions that are caused by this update, please
either ping me on IRC / Matrix (the low-traffic "Fedora Rust" room is
a good choice), or file a ticket on bugzilla or with the upstream
project on pagure:

- cargo2rpm (RPM macro backend / low-level Python module):
https://pagure.io/fedora-rust/cargo2rpm
- rust-packaging (RPM macros and generators):
https://pagure.io/fedora-rust/rust-packaging
- rust2rpm (RPM spec file generator): https://pagure.io/fedora-rust/rust2rpm

Fabio