F25 Self Contained Change: Rust Compiler

[Jan Kurik]

= Proposed Self Contained Change: Rust Compiler =
https://fedoraproject.org/wiki/Changes/RustCompiler

Change owner(s):
* Josh Stone <jistone at redhat dot com>

Introduce packages for the Rust compiler and its Cargo build system.

== Detailed Description ==
In the project's own words, "Rust is a systems programming language
that runs blazingly fast, prevents segfaults, and guarantees thread
safety." Cargo is Rust's primary package manager and build system.
This change is to bring both into Fedora.

Rust 1.10.0 was released on July 7 along with Cargo 0.11.0. These will
be the initial targets to package. Rust's next release is scheduled
for August 18 on a 6-week cycle. Backwards compatibility is taken very
seriously, so it should be possible to keep upgrading throughout a
Fedora release's lifetime. It might be debatable whether Fedora users
want those upgrades or not.

More information:
* Bug request to package rust #915043
* Work in progress rust-spec for the latest release 1.10
* COPR jistone/rust demonstrating the bootstrap from a binary then pure source

== Scope ==
* Proposal owners: Get rust and cargo into Fedora, and maintain them
in the future.

* Other developers: N/A (not a System Wide Change)

* Release engineering: N/A (not a System Wide Change)

* List of deliverables: N/A (not a System Wide Change)

* Policies and guidelines:
- Possibly add new guidelines for packages written in Rust. This can
be done after.
- Considerations are pretty similar to PackagingDrafts/Go, especially
static linking and the idea of source-based -devel libraries.

* Trademark approval: N/A (not needed for this Change)

[Dan Horák]

On Wed, 13 Jul 2016 06:18:17 +0200
Jan Kurik  wrote:
> = Proposed Self Contained Change: Rust Compiler =
> https://fedoraproject.org/wiki/Changes/RustCompiler
>
> Change owner(s):
> Josh Stone <jistone at redhat dot com>
>
> Introduce packages for the Rust compiler and its Cargo build system.
>
> == Detailed Description ==
> In the project's own words, "Rust is a systems programming language
> that runs blazingly fast, prevents segfaults, and guarantees thread
> safety." Cargo is Rust's primary package manager and build system.
> This change is to bring both into Fedora.

How does the new language (and compiler) affect secondary architectures?
Will it just work?

Dan

[Stefan Nuxoll]

Rust uses LLVM for codegen, so in theory, yes. This excludes any potential platform-specific bugs that may affect rustc, which are certainly a possibility.

[Adam Jackson]

On Wed, 2016-07-13 at 07:26 +0000, Stefan Nuxoll wrote:
> Rust uses LLVM for codegen, so in theory, yes. This excludes any
> potential platform-specific bugs that may affect rustc, which are
> certainly a possibility.

At the moment llvm has codegen support for every Fedora architecture,
primary and secondary. Of architectures we've formerly supported it's
missing s390 and ia64.

- ajax

[Josh Stone]

On 07/13/2016 07:50 AM, Adam Jackson wrote:
> On Wed, 2016-07-13 at 07:26 +0000, Stefan Nuxoll wrote:
> > Rust uses LLVM for codegen, so in theory, yes. This excludes any
> > potential platform-specific bugs that may affect rustc, which are
> > certainly a possibility.
>
> At the moment llvm has codegen support for every Fedora architecture,
> primary and secondary. Of architectures we've formerly supported it's
> missing s390 and ia64.

Perhaps I'm looking in the wrong place, but F24 "llc -version" does not
show mips, which is in the list of secondary architectures here:

https://fedoraproject.org/wiki/Architectures#Structure

Should "Mips" be added to LLVM_TARGETS_TO_BUILD?

[Adam Jackson]

On Wed, 2016-07-13 at 12:42 -0700, Josh Stone wrote:
> On 07/13/2016 07:50 AM, Adam Jackson wrote:
> > On Wed, 2016-07-13 at 07:26 +0000, Stefan Nuxoll wrote:
> > > Rust uses LLVM for codegen, so in theory, yes. This excludes any
> > > potential platform-specific bugs that may affect rustc, which are
> > > certainly a possibility.
> >
> > At the moment llvm has codegen support for every Fedora architecture,
> > primary and secondary. Of architectures we've formerly supported it's
> > missing s390 and ia64.
>
> Perhaps I'm looking in the wrong place, but F24 "llc -version" does not
> show mips, which is in the list of secondary architectures here:
>
> https://fedoraproject.org/wiki/Architectures#Structure
>
> Should "Mips" be added to LLVM_TARGETS_TO_BUILD?

Eep, you are correct. I have another llvm build I need to submit
shortly anyway, I'll add mips to the list. Thanks for the catch.

- ajax

[Igor Gnatenko]

On Wed, Jul 13, 2016 at 8:45 AM, Dan Horák  wrote:
> On Wed, 13 Jul 2016 06:18:17 +0200
> Jan Kurik  wrote:
> > = Proposed Self Contained Change: Rust Compiler =
> > https://fedoraproject.org/wiki/Changes/RustCompiler
> >
> > Change owner(s):
> > Josh Stone <jistone at redhat dot com>
> >
> > Introduce packages for the Rust compiler and its Cargo build system.
> >
> > == Detailed Description ==
> > In the project's own words, "Rust is a systems programming language
> > that runs blazingly fast, prevents segfaults, and guarantees thread
> > safety." Cargo is Rust's primary package manager and build system.
> > This change is to bring both into Fedora.
>
> How does the new language (and compiler) affect secondary architectures?
> Will it just work?

They make archives for bootstrapping for aarch64. Not sure about others...
>             Dan
>
> --
> devel mailing list
> [e-mail]
> https://lists.fedoraproject.org/admin/lists/[e-mail]

[Josh Stone]

On 07/12/2016 11:45 PM, Dan Horák wrote:
> How does the new language (and compiler) affect secondary architectures?
> Will it just work?

Rust has their own tier system of architecture support:

https://doc.rust-lang.org/book/getting-started.html#platform-support

Only i686 and x86_64 are Tier 1 at this time.

Tier 2 gets a little spottier, but at least armv7-hf and aarch64 do have
rustc and cargo to bootstrap from.  For ppc64/le and mipsel, we'd have
to prepare our own cross-compiled rustc and cargo for bootstrap.  Then
mips64el and s390x are completely missing.

If folks would like to see better Rust support for their favorite
architecture, it seems there are no real blockers* -- they just need to
pitch in.  Get it working, then add it to rust-buildbot to stay working.

https://internals.rust-lang.org/t/suggestion-make-real-world-tier-1-platform...

(*no real blockers except for LLVM support, I should say)

[Dan Horák]

On Wed, 13 Jul 2016 09:02:06 -0700
Josh Stone  wrote:
> On 07/12/2016 11:45 PM, Dan Horák wrote:
> > How does the new language (and compiler) affect secondary
> > architectures? Will it just work?
>
> Rust has their own tier system of architecture support:
>
> https://doc.rust-lang.org/book/getting-started.html#platform-support
>
> Only i686 and x86_64 are Tier 1 at this time.
>
> Tier 2 gets a little spottier, but at least armv7-hf and aarch64 do
> have rustc and cargo to bootstrap from.  For ppc64/le and mipsel,
> we'd have to prepare our own cross-compiled rustc and cargo for
> bootstrap.  Then mips64el and s390x are completely missing.

sad, time to ask whether IBM hasn't already been cooking something
internally ... LLVM itself has s390x support for quite some time.

Dan
> If folks would like to see better Rust support for their favorite
> architecture, it seems there are no real blockers* -- they just need
> to pitch in.  Get it working, then add it to rust-buildbot to stay
> working.
>
> https://internals.rust-lang.org/t/suggestion-make-real-world-tier-1-platform...
> (*no real blockers except for LLVM support, I should say)
>
> devel mailing list
> [e-mail]
> https://lists.fedoraproject.org/admin/lists/[e-mail]

[Stefan Nuxoll]

Rust supports dynamic linking, would it be worth investigating that route instead of copying the proposed Go guidelines? I'm not against static linking at all costs, but it would make package maintenance (especially security updates) a lot more pleasant.

[Igor Gnatenko]

Currently it's not possible for dynamic linking...

On Wed, Jul 13, 2016 at 9:29 AM, Stefan Nuxoll  wrote:
> Rust supports dynamic linking, would it be worth investigating that route instead of copying the proposed Go guidelines? I'm not against static linking at all costs, but it would make package maintenance (especially security updates) a lot more pleasant.
>
> devel mailing list
> [e-mail]
> https://lists.fedoraproject.org/admin/lists/[e-mail]

[Alexander Bokovoy]

On Wed, 13 Jul 2016, Igor Gnatenko wrote:
> Currently it's not possible for dynamic linking...

Why?

Dynamic linking is supported by Rust and according to its documentation,
is enabled for system libraries on Linux by default.

Can you explain why you are claiming it is not possible to enable
dynamic linking.
> On Wed, Jul 13, 2016 at 9:29 AM, Stefan Nuxoll  wrote:
> > Rust supports dynamic linking, would it be worth investigating that
> > route instead of copying the proposed Go guidelines? I'm not against
> > static linking at all costs, but it would make package maintenance
> > (especially security updates) a lot more pleasant.  --
> > devel mailing list
> > [e-mail]
> > https://lists.fedoraproject.org/admin/lists/[e-mail]
>
> --
> -Igor Gnatenko
>
> devel mailing list
> [e-mail]
> https://lists.fedoraproject.org/admin/lists/[e-mail]

[Igor Gnatenko]

I meant that rust itself links dynamically to system libraries. But
libraries built by rust are not supposed to be dynamically linked by
programs.

At least at this moment. So for now I don't see alternatives to have same
packaging guidelines  as for Go.

-Igor Gnatenko

On Jul 13, 2016 11:05 AM, "Alexander Bokovoy"  wrote:
> On Wed, 13 Jul 2016, Igor Gnatenko wrote:
> > Currently it's not possible for dynamic linking...
>
> Why?
>
> Dynamic linking is supported by Rust and according to its documentation,
> is enabled for system libraries on Linux by default.
>
> Can you explain why you are claiming it is not possible to enable
> dynamic linking.
> > On Wed, Jul 13, 2016 at 9:29 AM, Stefan Nuxoll  wrote:
> > > Rust supports dynamic linking, would it be worth investigating that
> > > route instead of copying the proposed Go guidelines? I'm not against
> > > static linking at all costs, but it would make package maintenance
> > > (especially security updates) a lot more pleasant.  --
> > > devel mailing list
> > > [e-mail]

https://lists.fedoraproject.org/admin/lists/[e-mail]
> > --
> > -Igor Gnatenko
> >
> > devel mailing list
> > [e-mail]
> > https://lists.fedoraproject.org/admin/lists/[e-mail]
>
> --
> / Alexander Bokovoy
>
> --
> devel mailing list
> [e-mail]
> https://lists.fedoraproject.org/admin/lists/[e-mail]

-Igor Gnatenko

[Dridi Boukelmoune]

On Wed, Jul 13, 2016 at 11:11 AM, Igor Gnatenko  wrote:
> I meant that rust itself links dynamically to system libraries. But
> libraries built by rust are not supposed to be dynamically linked by
> programs.

Hello,

Since 1.10 (the target of this change) it has become possible to
create "cdylib" (C dynamic library) crates, basically shared objects
that can be linked to by non-Rust programs.

It isn't supported by cargo yet, but it's available..

Dridi

[Josh Stone]

On 07/13/2016 02:24 AM, Dridi Boukelmoune wrote:
> Since 1.10 (the target of this change) it has become possible to
> create "cdylib" (C dynamic library) crates, basically shared objects
> that can be linked to by non-Rust programs.
>
> It isn't supported by cargo yet, but it's available..

Yes, you can use this to provide a library of extern-C functions.  I see
no problem Rust-based packages using this in Fedora, but they will need
to adhere to the usual stability policies.

Actually, I think you could do that even with the "dylib" target.  It's
just that this would include all the metadata and symbols for its
reachable Rust items too, which is irrelevant for an FFI library.

There's more info in the RFC:
https://github.com/rust-lang/rfcs/blob/master/text/1510-rdylib.md

[Josh Stone]

On 07/13/2016 12:29 AM, Stefan Nuxoll wrote:
> Rust supports dynamic linking, would it be worth investigating that
> route instead of copying the proposed Go guidelines? I'm not against
> static linking at all costs, but it would make package maintenance
> (especially security updates) a lot more pleasant.

To be clear, we will still dynamically link to foreign libraries.

Rust does support dynamic linking, but there's no stable ABI.  So from a
packaging perspective, that really defeats the point.  Every rebuild of
a library would potentially need a rebuild of its dependents.

There's also support for creating static libraries (*.rlib), but that
will surely have ABI issues too.

So with static linking from source-based devel packages, we only have to
rebuild the leaves that actually ship an executable.  I think that's the
best scenario for now.

[Stefan Nuxoll]

ABI stability is entirely dependent on compiler version, AFAIK. Within the same version of rustc there should be no issues with dynamic linking, of course whenever we upgrade rustc we'd need a mass rebuild of every rust package but we'd have to do the same thing with static linking.

[Josh Stone]

On 07/13/2016 09:10 AM, Stefan Nuxoll wrote:
> ABI stability is entirely dependent on compiler version, AFAIK.
> Within the same version of rustc there should be no issues with
> dynamic linking,

I suspect that may true in general, but I don't think it's promised, and
I don't really trust it.  We need to be able to patch bugs and security
fixes, and it's really hard to tell what will change the ABI.

For instance, all reachable symbols are exposed in the dynamic library,
not just public ones.  This is necessary for generics that are
instantiated elsewhere to get to their internals.  So you'd have to make
sure that whatever you're patching doesn't change reachability.

Inline tags can also affect symbols, and I'm sure there are other things
that I'm not thinking of right now.  I really think we should stay away
from this until Rust gets a true stable ABI.
> of course whenever we upgrade rustc we'd need a mass rebuild of every
> rust package but we'd have to do the same thing with static linking.

With unstable dynamic libraries, we'd have to rebuild everything all at
once, and push them through bodhi together, with precise Requires tags
to keep them all in sync.

With static linking, we can relax a little.  Only leaf packages have to
be rebuilt, and only if the update actually requires propagating, which
can be left to maintainer discretion.  For instance, even a full rustc
rebase doesn't necessarily need to rebuild all rust packages.  And since
there's no runtime dependency, the rebuilt packages can be pushed
through bodhi independently.

[Josh Stone]

On 07/12/2016 09:18 PM, Jan Kurik wrote:
> Rust 1.10.0 was released on July 7 along with Cargo 0.11.0. These will
> be the initial targets to package. Rust's next release is scheduled
> for August 18 on a 6-week cycle. Backwards compatibility is taken very
> seriously, so it should be possible to keep upgrading throughout a
> Fedora release's lifetime. It might be debatable whether Fedora users
> want those upgrades or not.

I would specifically like feedback on this point, the 6-week release
cycle.  We should certainly keep rawhide rolling along, but do people
want stable Fedora to keep getting Rust updates?

We do keep things like Firefox current, but an application definitely
has different considerations than a compiler.

[Simo Sorce]

On Wed, 2016-07-13 at 09:12 -0700, Josh Stone wrote:
> On 07/12/2016 09:18 PM, Jan Kurik wrote:
> > Rust 1.10.0 was released on July 7 along with Cargo 0.11.0. These
> > will
> > be the initial targets to package. Rust's next release is scheduled
> > for August 18 on a 6-week cycle. Backwards compatibility is taken
> > very
> > seriously, so it should be possible to keep upgrading throughout a
> > Fedora release's lifetime. It might be debatable whether Fedora
> > users
> > want those upgrades or not.
>
> I would specifically like feedback on this point, the 6-week release
> cycle.  We should certainly keep rawhide rolling along, but do people
> want stable Fedora to keep getting Rust updates?
>
> We do keep things like Firefox current, but an application definitely
> has different considerations than a compiler.

Until the Rust ecosystem becomes more stable, it is better to upgrade
it regularly than have a stale tool nobody can really use.
There are even crates that work only with nightlies for now, so trying
to stay on par is necessary to make it useful. Also, although I am big
fan of dynamic linking we should probably keep Rust program built
statically for now, until Rust gets a stable ABI, or the breakages will
defeat any advantage we may get.

Simo.

[Josh Stone]

On 07/14/2016 05:15 PM, Simo Sorce wrote:
> Until the Rust ecosystem becomes more stable, it is better to upgrade
> it regularly than have a stale tool nobody can really use.
> There are even crates that work only with nightlies for now, so trying
> to stay on par is necessary to make it useful. Also, although I am big
> fan of dynamic linking we should probably keep Rust program built
> statically for now, until Rust gets a stable ABI, or the breakages will
> defeat any advantage we may get.

Thanks, that's pretty much how I feel too.  We can't do much for those
that want nightly (though I might start a copr), but for everyone else
it's probably best to keep shipping the newest Rust possible.

We agree on static compilation too.  Even once there is an ABI, there's
a lot to think about how generics are instantiated -- although at that
point I guess it's not too different from C++ template considerations.

I've also been experimenting with how crates could be packaged, and
while I still think source-based devel rpms are the answer, the tooling
doesn't seem to be there yet.  I think we'll need directory registries
from here[1], and until then only we should only ship rustc and cargo.
[1] https://github.com/rust-lang/cargo/pull/2857

(That won't prevent Firefox packagers from using Rust, if they want to,
since their code is currently vendored into the source tarball anyway.)