On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
>
> I _very much_ appreciate all the work you and the other Rust SIG folks
> (Igor and Zbyszek in particular but I'm sure others as well!) have put into
[… and Fabio in particular]
> I'll start with the second: our system for Rust doesn't
really do that.
> Developers are going to use cargo and crates.io and we're not going to
> convince them that they should do otherwise. (I don't expect anyone
> disagrees with this.)
This is true, and probably also not "fixable".
Also, arguably it's not something we'd like to "fix" at all.
I think not-using-the-rpm-packaged-software-during-development is
something that is equally true for Rust, C, Python, Java, and pretty
much any language. As long as Rust is not exclusive to Fedora/CentOS/RHEL
family, rpms are just going to be one of the possible ways to get rust
code.
For me, the goal is to make the automatic conversion from rust crates
to rpms as smooth as possible. I want the rpm package database to be
the CDN we use to get verified Rust code.
> *This is what open source winning looks like.*
[Snip various details here. I think Fabio's points are very good.]
> We could also attach other metadata to the packages in the
cache. Maybe some
> popularity, update frequency from Cargo.io, but also package review flags:
> checked license against source, and whatever other auditing we think should
> be done. This moves the focus from specfile-correctness to the package
> itself, and the effort from packaging to reviewing. (I'd suggest that for
> the experiment, we not make any deep auditing manditory, but instead
> encouraged.) And these flags should be able to be added by anyone in the
> Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
> Fedora _needs_ to adapt to stay relevant in the world where every language
> stack has developed a packaging ecosystem which effectively ignores us. Some
> of them are missing lessons they could have learned, ah well — but they also
> have a lot of nice new ideas we're missing. And, no matter what we think,
> we're clearly not going to stop them.
>
> Rust packaging seems like a great place to lead the way — and then we can
> maybe expand to Go, which has similar issues, and then Java (where, you
> know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
Given that, I think we're actually in kind of a *good* situation with
Rust packaging, especially compared to other language ecosystems - not
only right now, but also looking at the future. And looking at the
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
Yep. I expect some power-law distribution of package popularity. It
seems like there's an endless supply of crates, but most just don't
matter. If we get some threshold fraction of the most popular libaries
packaged, that should cover all or almost all dependencies of new leaf packages.
We're still hitting cases where there's a lot of stuff to package,
but as Fabio describes, those cases are not that common.
> Maybe we could get involved in Cargo Vet [4] — we could be both
a consumer
> _and_ a data source.
I like the idea. It'd be great to get for example a CI check for
package updates, and going the other way, use our review, QA, and
karma process as a source of trust. But this can (and IMO should) be
done in addition to our packaging, not as a replacement.
Zbyszek