musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]

[Matthew Miller]

On Thu, Oct 13, 2022 at 03:46:49PM +0200, Fabio Valentini wrote:
>  > The dependency on LLVM is not even the worst issue in my eyes.
> LLVM is also
>  > used by other core projects, e.g., mesa, these days.
>  >
>  > The worst issue I see with Rust is the way libraries are "packaged",
> which
>  > just implies installing source code and recompiling that source code for
>  > every single application. (And as a result, the output obviously gets
>  > statically linked into the application, with all the drawbacks of static
>  > linking.) I consider a language with no usable shared library support to be
>  > entirely unpackageable and hence entirely useless.
>  
>  Do you have suggestions for improving this situation? I think we're
>  pretty close to doing the best we can with packaging Rust projects,
>  given the current limitations of the language (i.e. the support for
>  building "true shared Rust libraries" is still very limited and
>  unstable; but that might improve in the future). 
I _very much_ appreciate all the work you and the other Rust SIG folks
(Igor and Zbyszek in particular but I'm sure others as well!) have put into
packaging rust apps and crates and all of the systems around that.

I fundamentally disagree with Kevin on a deep level about "entirely
useless", but ... find myself kind of agreeing about the "unpackagable"
part. I mean: clearly we've found a way, but I'm really not sure we're
providing a lot of _value_ in this approach, and I'm also not sure it's as
successful as it could be.

There are three ways having things packaged in Fedora repos _can_ be
helpful:

1. End-user applications and tools
2. Useful development environment
3. As convenience for ourselves for building packages for #1 or #2

I am not discounting the value of #3 -- making a shared thing that we all
work on together is kind of the whole point, and the nicer we can make that
the better we can bring in more people, and those of us already here have a
lighter load and can work on the things we're most interested in. But
ultimately, we're doing it so we make a useful system for users. That means
the first two.

I'll start with the second: our system for Rust doesn't really do that.
Developers are going to use cargo and crates.io and we're not going to
convince them that they should do otherwise. (I don't expect anyone
disagrees with this.)

We're doing okay with #1, but... I think #3 _even_ with all of the work in
Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
engine and will probably have a few things it would be nice to package to
make available in Fedora Linux. I might not even mind maintaining Bevy
itself.

But running `cargo fetch` with a clean cache pulls down *390* crates. Of
these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
which is *amazing*. But... that still is hundreds that I'd have to add. And
mostly they are things I don't know _anything_ about. 

*This is what open source winning looks like.*

I remember a Byte magazine article from the 1990 (I just checked!) with the
title "There Is a Silver Bullet: The birth of interchangeable, reusable
software components will bring software into the information age". [1] 
This was about the newly-hot idea of Object Oriented Programming. It was
very exciting. But, of course, that vision of the world did not happen. It
turns out proprietary software *can't* do this.

But now we have it! I don't have to reinvent every basic wheel — but even
more than that, I do not have to be an expert in the intricacies of safe
concurrency to write an app that uses that under the hood. That's amazing! I
can do such powerful things from high-level interfaces and trust the
expertise of those who really understand the deep computer science some of
this requires.

I am competent enough to write a silly toy game using Bevy. It might be good
enough that others will enjoy it. *I am not competent to maintain many of
these dependencies.* I don't even know what most of them DO. "anyhow"?
"bytemuck"?

Worse, many of the Bevy deps are specified with exact versions. Maybe I
could make the package work with the packaged versions, but ... that
requires deep expertise and even then might lead to unexpected behavior and
has a high chance of putting me at odds with both the engine upstream and
any other games which use it. The packaging guidelines say that I SHOULD
create patches to update to latest versions of dependencies, and that I
should further convince the upstream to take them. Candidly, that seems like
a waste of everone's time.

The guidelines provide for creating compat packages, but that means 1) the
existing shared work is less useful, 2) requires even more extra steps, and
3) even without reviews for compat has extra administrative overhead.

So, going back to Kevin's point: it _does_ feel like this is unpackagable.
But that's because the barrier to participation seems too high. It's not
because it's statically-linked binaries [2] can't or shouldn't exist in
Fedora — that's not one of our core principles! In fact and quite to the
contrary, we need to adapt to handle this amazing open source success story
better.

And, I led with: I appreciate all the work you've all done to make this
work. That's definitely true — I think it was super-valuable to pilot this
approach. But I think that the Rust ecosystem would be a great place to
pilot a different way. Something lightweight where we cache crates and use
them _directly_ in the build process for _application_ RPMs.

Rust packages include a lot of machine-readable metadata. We should be able
to watch for CVEs, RustSec, and other security notices even without encoding
the metadata in RPMs. License review could also be automated — the field in
Cargo.toml is supposed to be SPDX, so that's convenient. [3]

We could also attach other metadata to the packages in the cache. Maybe some
popularity, update frequency from Cargo.io, but also package review flags:
checked license against source, and whatever other auditing we think should
be done. This moves the focus from specfile-correctness to the package
itself, and the effort from packaging to reviewing. (I'd suggest that for
the experiment, we not make any deep auditing manditory, but instead
encouraged.) And these flags should be able to be added by anyone in the
Rust SIG, not necessarily just at import.

Maybe we could get involved in Cargo Vet [4] — we could be both a consumer
_and_ a data source.

Fedora _needs_ to adapt to stay relevant in the world where every language
stack has developed a packaging ecosystem which effectively ignores us. Some
of them are missing lessons they could have learned, ah well — but they also
have a lot of nice new ideas we're missing. And, no matter what we think,
we're clearly not going to stop them.

Rust packaging seems like a great place to lead the way — and then we can
maybe expand to Go, which has similar issues, and then Java (where, you
know, things have already collapsed despite heroic effort.)

What do you all think?


------

[1] https://archive.org/details/sim_byte_1990-10_15_10/page/n249/mode/2up

[2] Also, actually: https://docs.rs/bevy_dylib/0.8.1/bevy_dylib/. They don't 
    recommend this for release builds... but that looks to be mainly for
    distribution reasons we could easily handle?

[3]
https://communityblog.fedoraproject.org/important-changes-to-software-lic...

[4] https://mozilla.github.io/cargo-vet/   

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader