Why we package Rust crates
by Igor Raits
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hello,
This email attempts to answer some frequently asked questions about
Rust SIG packaging of crates. For those who don't know what a "crate"
is: it is the name for a collection of functionality in Rust, similar
to libraries (C/C++), modules (Python/Perl/PHP), gems (Ruby), etc.
In Fedora, we package each crate separately and then use them as
BuildRequires to build the binaries (applications/utilities). The
crates are built and linked statically to each other, so they are not
present on user’s systems. The resulting binary build only has non-Rust
dependencies at runtime (standard library, C/C++ libraries, etc.).
Many folks ask the Rust SIG many variations of the same questions, so
we decided to provide a common set of questions and answers for that.
## Q: Why do you package those small bits separately than just
vendoring (bundling) them?
You (hopefully) all know that bundling is bad. There are many reasons
for this ranging from security to maintainability to licensing
compliance. This problem does not get better when your dependency
graphs get very wide (as they do for languages like Rust, Go, Nodejs,
etc.).
When we package something, we:
1. Check licensing
We make sure that acceptable (for Fedora) open source licenses are
used, license files exist and contain proper license texts, ensure
compatibility of the licenses in resulting binaries, keep track of
changes and properly announcing any license changes to ensure that
distribution of Fedora stays legal.
2. Build the software and run the tests
We have found quite amount of
* Architecture-related bugs
* 32bit vs 64bit
* big endian vs little endian
* Dependency information not being correct
* Breakages or issues in new versions of the Rust compiler
And many others.
Of course it would be nice if upstreams have extensive test coverage on
all distros and platforms, but this is unfortunately not the case. Much
of it has to do with the limitations of CI/CD services that are in
common use (such as Travis CI, Azure DevOps, GitHub Actions, etc.). For
example, Travis CI does not offer Fedora so people often only test
things with some old version of Ubuntu LTS with a hodgepodge of tools
and libraries with varying degrees of age. Unfortunately, this is
simply not something the Rust SIG is capable of fixing. But when we do
package it ourselves in Fedora, we catch these things and send fixes
upstream to ensure that everyone benefits from the work we *can* do to
make things better.
3. Integrate it with the rest of the distribution
For example, many crates that are bindings to the C libraries tend to
bundle C libraries instead of linking with system-provided shared
libraries. Sometimes they will use the system-provided shared libraries
only under specific circumstances. We patch them to always use system
libraries and run the test suites to ensure that they actually work.
This allows us to leverage the work that has been done to maintain
system libraries to keep Rust software in good shape.
As an example, libgit2-sys used to use system libgit2 only if some
environment variable was set, we had patch from day 0 to change that
condition and use system libgit2. As libgit2 is a library that tends to
need security fixes often, doing this allows us to avoid having to
rebuild any Rust applications that link to libgit2 whenever we fix
libgit2 and roll out fixes *very* quickly.
Now imagine that you'd need to apply all those fixes to each
application which depends on libgit2-sys. Today, we have following in
Fedora:
* bat
* cargo-c
* exa
* pretty-git-prompt
* silver
* starship
Even though they are statically linked (in terms of crates), they use
the standardized way of building packages in Fedora and it is possible
to know which versions of which component was used to build them so
that we can make adjustments as needed.
## Q: You patch crates and that harms the reputations of the upstream
projects!
Firstly, it's important to recognize that patching software in itself
is not inherently bad. There are many reasons to do this, and it's a
matter of having the right discipline to figure out what the right
thing to do is.
Secondly, when we *do* patch crates, we always send patches to the
upstream projects and most of them are accepted. There are a few
patches which have been rejected or not accepted. The most common
reason for this is that upstream is dead, plain and simple.
Finally, since we try to keep only the latest version of a crate in
Fedora, the bulk of our patching is oriented around porting crates /
applications to latest version of their dependencies, which is
generally appreciated by upstream (unless they have some unusual
compatibility promises).
So we are often doing a good service to the upstream projects
developing and maintaining those crates by sharing a bit of that burden
and helping them move forward!
## Q: Why not to have mirror of crates.io in Fedora?
I would ask same for every ecosystem, CPAN, RubyGems, PyPI, …
It is simply not useful to make a 1:1 copy of upstream (see reasons
above). There are things which must be done before content can appear
in Fedora.
If we can make it for crates.io, we will need to make it generic enough
to support PyPI, CPAN and others. All of them have their own specifics
and the tool will need to handle all of them. And then it would be
different for ecosystems which do not have a language-specific package
manager.
We already have RPM packaging format and we can encode any kind of
information in it, so you could analyze, do queries across all
languages the same way. Licensing, dependencies, everything. Same way.
I would be happy to have crates.fedoraproject.org which would be
populated from RPMs so that people can use Cargo.toml and that would
download crates with Fedora specifics.
RPM spec files for Rust crates are generated using rust2rpm, then you
just need to do manual steps which you need to do anyway even if you
would have it on crates.fedoraproject.org.
The pain points here are complicated importing process and manual build
process in Fedora. Most of crates can be packaged as RPM in 5-10
minutes, but then it takes enormous time to actually get it reviewed
and built in Koji, from an hour up to days.
Again, this is not something Rust SIG can work on, but we are happy to
provide feedback and help where we can.
## Q: Why don't you do dynamic linking?
The answer is quite simple. There are 2 reasons:
1. Rust does not have stable ABI:
https://github.com/rust-lang/rfcs/issues/600
Building same thing twice with different versions of the compiler
results in different (incompatible!) libraries, so you need to
recompile them anyway. If the Rust language defines a standard ABI for
Rust libraries, then we would change this. However, the Rust language
developer team is not (currently) interested in that.
2. Features
One of the greatest features of Rust is the standardized way of
selecting features which library should have. That is very similar
(same?) to what Python "extras" are. Think of it like extra features
library has (e.g. JSON/YAML/TOML input/output support, colorful
output). This usually means some new dependencies would be pulled in.
That means, we would need to package all combinations of libraries or
find a way how to teach Rust to load additional symbols from such
"features" or even override them and what to do in case of conflicts.
This is complicated and is not something what can be done in a few
weeks.
## Q: All you do is just complicates packaging Rust software in Fedora!
Yes and no. That is true that packaging some application can require
adding tens of other dependencies and their dependencies. But in the
end what you get is standardized build of an RPM with records of which
packages were used in the build environment, proper integration done
for every crate, and all the other benefits that packaging software in
the distribution offers.
- --
On behalf of Rust SIG,
Igor Raits <ignatenkobrain(a)fedoraproject.org>
Neal Gompa <ngompa13(a)gmail.com>
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl69nXkACgkQEV1auJxc
Hh5vyA//SoShI5Yi+MqmCoLHlXGo/64YnH0RZBZoc7VEDpBN8oXFVn0Dcn4A+ipq
vZ8Avj/WpchzZQf1AiLjiVYZ1NOzpyKOznPx2c9fQBcAZDIFTY9n98C741V5C8Qj
6TXh5ie9ga5tbfglcES48PoX0u+mRDYBs6PBt1XezPlUCIiIBfzv0UuM0aKdStEX
TUCvGwJn/BAyXfVz5In0VwLv6RlL5rDqb9Yi1u5GbxWNYvKh26TvFnCvI1vHhD/Y
NHcEStzcxcNiDyM/Vd3c+w5N+d5FRAbvBuobWhy/2Pa22D6Y1e8T2q7Sj7u11vZf
NuxGXC/JymbAGaB7i5gIkGwkhvhUmgvLf7h40MrtR5Y3G9Z2iWCsTwpwhG1lv4kr
tyMDI6m2a7Wb2yWV4clAFJKHYP3IQMwWfNHEMLBunAooOEORYdhIUHZvvTHbX5sp
zX5Lt/n0O55x4YqA5YH/IcgMBJyxk9yOZNWojCCUKrfQ6WZ1QklIf3wZeSi9XV6x
shSMCnqpbU285GYa2ne8ieUYH9U5cMuqrIrJEU1wafUsrHDN2Vm7Q0S1zUlebDId
OgmUkS7qOZ7/aGB2eRXm4PomFSar7fUqnhvkPmDpwb7XhzqS8PAn7vTZzxwEpRZg
FnM0vm1bcuY467zQqUJYZ4TDBMiXo7yJE6YYRWrtPjSuCbO6j0Q=
=e324
-----END PGP SIGNATURE-----
3 years, 6 months
- 1
- 0
- ← Newer
- 1
- Older →