8:57 a.m.
Hi,
before I write the proposal itself I just want to stress the fact that
it isn’t my intention to change the current packaging workflow and
definitely not the user experience. Also if you have C or Python
packages it would not affect your work at all (I’m not familiar with all
interpreted languages in Fedora, but I guess it is similar to Python and
therefore it is not affected by the problems I am going to describe).
First of all, let me describe the problem I see in our Fedora ecosystem
with relation to Go and Rust language ecosystems. More specifically in
the relation between RPM buildroot and packages in these ecosystems.
Both of these languages follow the idea that packages should be small
and only have a limited set of features. Developers then use a lot of
these packages to write the final executable that is meant for end-users
[1]. Also both of these languages use static linking of the final
binaries meaning that Fedora users don’t install RPM packages of these
libraries as they are already baked inside of the binary [2].
The 1st problem is that if we want to build RPMs of the final
executables the way we do now, we need to package all these small
libraries into RPM even though they are just build dependencies and
users never install RPMs of these libraries. Many of these RPMs are
automatically generated from the upstream packages meaning that we don’t
do anything except for unpacking the upstream package (e.g. plain
tarball in case of crates.io) and then we package the same into RPM.
This process is unfortunately not fully automated and therefore requires
a certain amount of human effort.
To sum up the previous paragraph, I don’t think it is necessary to
repackage upstream tarball into a downstream RPM.
The 2nd problem is present only in the Rust ecosystem (as far as my
knowledge goes). Cargo, the official package manager for Rust, can
handle the scenario where an executable depends on a single library in
two different versions [3]. Dnf, on the other hand, will not install two
versions of the same RPM package. What we do now is, that we patch the
affected executables and libraries to only use the newest versions
everywhere. This is again an additional maintenance cost and we create
differences from upstream, because these patches are not necessarily
merged. See this as an example:
https://src.fedoraproject.org/rpms/rust-bstr/blob/master/f/rust-bstr.spec...
https://github.com/BurntSushi/bstr/pull/23
To sum up the 2nd problem: we are using dnf instead of the upstream
package manager to install dependencies. These two approaches can be
incompatible (and they are in case of Rust).
The 3rd problem I see is that issues like this are not going away, they
are only going to get worse as other ecosystems emerge. e.g. if the
Swift language became popular (for Linux development) it would again
have it’s own package manager and probably its own set of issues in
relation to our build system. (I mention Swift specifically because it
is a compiled language that have stable ABI only for std, other
libraries are statically linked)
Now, I’d like to point out that Go and Rust packaging works well in
Fedora due to the enormous effort of certain people and I very much
admire the work they do. But on the other hand I’m afraid where the
ecosystem would go without them. This is where we get to the situation
in the enterprise derivatives, specifically RHEL and CentOS. Their
solution to this problem is not to package all libraries separately but
to bundle all libraries directly into source RPMs of each executable. So
the bundling is not present only on the binary file level, but also on
the source RPM level. Go went even further in this case and it is common
to bundle all the dependencies as a source code directly in the upstream
repository. See this repo as an example:
https://github.com/containers/libpod/tree/master/vendor
It is fair to say, that my first motivation was the current state of
packaging in RHEL but I’d prefer to discuss this in Fedora first.
The proposal itself is fairly simple: Let’s stop packaging all Go and
Rust libraries into RPM and install them to the buildroot in the
upstream format instead.
The specific implementation is up for a discussion but I think it is
logical to start by asking what features do we want from the build
system? My answers are:
* We want to know what exactly was used to build the RPM to ensure
integrity. This is possible with upstream tarballs as well as with RPMs.
Just store a hash of the tarball.
* There should be no maintenance cost. If we would avoid modifying the
upstream package it would be the ideal case.
* It should be possible to patch the library in case of severe issues
like CVEs. This is a contradiction to the previous point, but it could
be solved by unpacking the upstream package into a git repository and
creating a new, modified package in upstream format from it.
* It should be possible to run the build locally in exactly the same
way Koji does it. This is just about exposing our “registry” of packages
to the public Internet.
Finally, the implementation could be something like this:
A service like release-monitoring could monitor the upstream projects
for new releases (e.g. NPM has a RSS feed). Every time there is a new
release it would automatically synchronize our own Fedora-specific
registry, which would in turn be accessible in buildroot. Then the RPM
macros like “cargo_build” could be modified to use this registry.
If we wanted to have the possibility to patch these libraries we could
synchronize the upstream release into our git repository and generate
packages in upstream format from it.
What do you think about the way we build RPM packages for Go and Rust?
Do you see other solutions or is it ok the way it is? (Please do not
suggest to “fix” the upstream communities)
Thanks for your opinions!
Martin
FAQ:
* Does it mean you want to get rid of -devel subpackages for C libraries?
No. First of all C language does not have a single package manager and
it is still common to install dependencies using the distro specific
package manager. But packaging into RPM also comes with the advantage of
having shared libraries so in this case it has a positive effect on the
end-user experience. Rust and Go libraries on the other hand are mostly
useless for end-users.
* I prefer to install development libraries using the distro package
manager instead of the upstream package manager. Why do you want to
change it?
This is the case for C development, but it is different in most of the
modern languages. For example we ship Python packages, but we discourage
people from using them for their own development. Python developers are
instead encouraged to use pip to create a virtual environment and use
upstream packages. In the case of Rust, using the system libraries would
be much harder than using the upstream package manager.
* Does it mean you want to get rid of Rust packages like ripgrep?
No. The end user experience stays unchanged. Packages that we ship to
Fedora users must be in RPM format, it is only the buildroot that changes.
Notes:
[1] Please don’t discuss if it is a good idea or not. This is a
discussion for upstream of those languages.
[2] Again, please don’t discuss static vs. dynamic linking here.
[3] https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell
9:11 a.m.
On 2/21/20 9:57 AM, Martin Sehnoutka wrote:
Every time there is a new release it would automatically synchronize
our
own Fedora-specific registry, which would in turn be accessible in
buildroot.
One thing that comes to my mind with this proposal is that we still need
some way to vet licenses. Today, this is done via the package review
process, and in my mind is the primary purpose of package review. If we
started having upstream compatible registries, I suppose we could
introduce a review process for adding packages to the registries to
solve this concern.
I think it's a good idea.
4:35 a.m.
On Fri, Feb 21, 2020 10:11:47 -0500, Randy Barlow wrote:
On 2/21/20 9:57 AM, Martin Sehnoutka wrote:
> Every time there is a new release it would automatically synchronize our
> own Fedora-specific registry, which would in turn be accessible in
> buildroot.
One thing that comes to my mind with this proposal is that we still need
some way to vet licenses. Today, this is done via the package review
process, and in my mind is the primary purpose of package review. If we
started having upstream compatible registries, I suppose we could introduce
a review process for adding packages to the registries to solve this
concern.
Vetting of licenses is only one aspect of the review but not its sole
purpose. Apart from all the other checks to ensure that the software
follows current best practices in software development, we also verify
the correctness of the software during the review.
--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
1:41 p.m.
On Mon, Feb 24, 2020 at 10:35:08AM +0000, Ankur Sinha wrote:
> One thing that comes to my mind with this proposal is that we
still need
> some way to vet licenses. Today, this is done via the package review
> process, and in my mind is the primary purpose of package review. If we
> started having upstream compatible registries, I suppose we could introduce
> a review process for adding packages to the registries to solve this
> concern.
Vetting of licenses is only one aspect of the review but not its sole
purpose. Apart from all the other checks to ensure that the software
follows current best practices in software development, we also verify
the correctness of the software during the review.
That's true, but I think fundamentally I agree with Randy: if the software
is terrible in some way but still useful to users (hello, Chromium!), we
probably want to find some room for it in Fedora somehow. But we absolutely
need to be sure about licenses.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
2:12 p.m.
On Tue, Feb 25, 2020 14:41:34 -0500, Matthew Miller wrote:
On Mon, Feb 24, 2020 at 10:35:08AM +0000, Ankur Sinha wrote:
> > One thing that comes to my mind with this proposal is that we still need
> > some way to vet licenses. Today, this is done via the package review
> > process, and in my mind is the primary purpose of package review. If we
> > started having upstream compatible registries, I suppose we could introduce
> > a review process for adding packages to the registries to solve this
> > concern.
> Vetting of licenses is only one aspect of the review but not its sole
> purpose. Apart from all the other checks to ensure that the software
> follows current best practices in software development, we also verify
> the correctness of the software during the review.
That's true, but I think fundamentally I agree with Randy: if the software
is terrible in some way but still useful to users (hello, Chromium!), we
probably want to find some room for it in Fedora somehow.
Sure, but only if it works (correctness).
But we absolutely need to be sure about licenses.
Yes, the legal requirement merely sets the lowest bar for inclusion of
software: it must be legally allowed to be even considered for
inclusion. So, it is a compulsory part of the review process, but so are
other checks. I give correctness as an example because it's easiest to
understand: if the software is legally acceptable, but doesn't build or
work properly, then it doesn't pass the review either.
Basically, packages do not pass review merely because they use good
licenses.
--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
2:09 a.m.
On Tue, Feb 25, 2020 22:26:24 -0500, Randy Barlow wrote:
On 2/25/20 3:12 PM, Ankur Sinha wrote:
> Basically, packages do not pass review merely because they use good
> licenses.
Note that I just said that I thought it was the primary purpose, not
the only purpose.
Sure, but I'm arguing that this isn't correct.
Maybe it's to do with how I interpret the phrase: "the primary". To me
(so it could be very wrong) "the primary purpose" also implies that the
other purposes are "secondary" and that isn't true. So, I would be
happier with saying "a primary purpose" instead of "the primary
purpose".
Sorry, got a bit pedantic with language here. :/
--
Thanks,
Regards,
Ankur Sinha "FranciscoD" (He / Him / His) |
https://fedoraproject.org/wiki/User:Ankursinha
Time zone: Europe/London
9:21 a.m.
Le vendredi 21 février 2020 à 15:57 +0100, Martin Sehnoutka a écrit :
Hi,
Hi,
Go went even further in this case and it is
common
to bundle all the dependencies as a source code directly in the
upstream
repository. See this repo as an example:
https://github.com/containers/libpod/tree/master/vendor
Since you take Go in example – vendoring (bundling) proved such a
maintenance headache upstream that Google had to create a complete new
packaging system to get rid of it. Not for Fedora or Linux people, but
because they had huge problems with unmaintained, holed and non-legal
code lurking in this kind of vendor pile.
So Go (upstream) is going away from the kind of practice you’re
advocating. And Google’s implementation of versionning in Go modules is
even stricter and more rigid than the one in rpm (precisely because
they tried the lala-lala let’s bundle everything without caring about
versions, and were badly burned).
From a technical POW, we need to improve the mapping of their packaging
system and then integrating the Go ecosystem in Fedora will be no
harder than integrating any other language component (easier, because
the language-specific package manager produces very regular sources).
From a maintenance POW dumping huge piles of unmanaged third-party code
on build farms just does not work long term. That’s why Google had to
break its vendor piles in atomic Go modules.
From a process POW, Fedora is not ready. Not because out tech does not
work but because our processes can not handle the pile of atomic
components generated by upstream practices. Since the problem is
process, not technical side, please work on the process side, instead
of trying to bypass the process with bad technical solutions.
Regards,
--
Nicolas Mailhot
9:42 a.m.
On Fri, 21 Feb 2020 at 09:58, Martin Sehnoutka <msehnout(a)redhat.com> wrote:
Hi,
before I write the proposal itself I just want to stress the fact that
it isn’t my intention to change the current packaging workflow and
definitely not the user experience. Also if you have C or Python
packages it would not affect your work at all (I’m not familiar with all
interpreted languages in Fedora, but I guess it is similar to Python and
therefore it is not affected by the problems I am going to describe).
First of all, let me describe the problem I see in our Fedora ecosystem
with relation to Go and Rust language ecosystems. More specifically in
the relation between RPM buildroot and packages in these ecosystems.
Both of these languages follow the idea that packages should be small
and only have a limited set of features. Developers then use a lot of
these packages to write the final executable that is meant for end-users
[1]. Also both of these languages use static linking of the final
binaries meaning that Fedora users don’t install RPM packages of these
libraries as they are already baked inside of the binary [2].
Ok here are some of the issues we have to deal with for these sorts of changes:
1. RPMs are built in Fedora without any access to the Internet. They
do not and can not pull in anything outside of their chroot as they
are building. This is a security requirement for multiple reasons
where everytime we relax it.. bad things happen. [Seriously there are
enough people with bad intentions to spoil everything for everyone
multiple times.]
2. To deal with 1, you need to now set up local registry and local
copies of the cargo/go/etc code and have mock learn how to 'dump' that
into the chroot in a way that it can see.
3. We need to then have a copy of the code in the Fedora build area.
That code will have had to have been vetted for license problems and
all the other reviews we do for getting a package because you dont'
want to have a cargo or any other thing with an rm -rf embedded in it.
4. We have now created 2-3 build systems which will need monitoring,
reviews, equipment, and tooling:
a. Ruby/perl/python build system (since they have the same issues).
b. Go build system
c. Rust build system
d. RPM build system.
e. Code repositories and registries for each of these so that the
outer mock could do the right thing and put things into the chroot.
5. Deal with all the duct tape with all the other tools that are
needed to get anything done.
The fact that you have to add a lot for 2,3,4 and 5.. it has always
been faster to just stick with doing everything in 1 and make RPMs.
The tasks are not impossible ones, but they are not simple and they
need someone to do that work. If you are going to back that proposal
with a working prototype we can shove into production with all our
other prototypes.. that will help a lot.
--
Stephen J Smoogen.
9:46 a.m.
On Fri, Feb 21, 2020 at 3:58 PM Martin Sehnoutka <msehnout(a)redhat.com> wrote:
Hi,
Hi!
I think several of your assumptions are not necessarily correct, which
might lead to wrong conclusions.
My responses are inline below.
before I write the proposal itself I just want to stress the fact
that
it isn’t my intention to change the current packaging workflow and
definitely not the user experience. Also if you have C or Python
packages it would not affect your work at all (I’m not familiar with all
interpreted languages in Fedora, but I guess it is similar to Python and
therefore it is not affected by the problems I am going to describe).
(snip)
First of all, let me describe the problem I see in our Fedora
ecosystem
with relation to Go and Rust language ecosystems. More specifically in
the relation between RPM buildroot and packages in these ecosystems.
Both of these languages follow the idea that packages should be small
and only have a limited set of features. Developers then use a lot of
these packages to write the final executable that is meant for end-users
[1]. Also both of these languages use static linking of the final
binaries meaning that Fedora users don’t install RPM packages of these
libraries as they are already baked inside of the binary [2].
One big difference between Go and Rust is that Go only namespaces
libraries by their import path (with some exceptions, like gopkg.in),
whereas Rust/cargo namespaces everything by name *and* version, which
makes some things a lot easier. Then there's also the fact that a lot
of Go libraries don't even *have* versions, and most only reference
dependencies by "this git repository". There's some effort to make
this better, but Go modules are not ready yet.
The 1st problem is that if we want to build RPMs of the final
executables the way we do now, we need to package all these small
libraries into RPM even though they are just build dependencies and
users never install RPMs of these libraries. Many of these RPMs are
automatically generated from the upstream packages meaning that we don’t
do anything except for unpacking the upstream package (e.g. plain
tarball in case of crates.io) and then we package the same into RPM.
This process is unfortunately not fully automated and therefore requires
a certain amount of human effort.
To sum up the previous paragraph, I don’t think it is necessary to
repackage upstream tarball into a downstream RPM.
Of course this requires a certain amount of human effort - because
that's where the packager does their work. This includes:
- verifying that the specified license is correct and suitable for
(re)distribution in fedora
- verify that the package builds and doesn't do crazy things
- fix upstream and integration issues with patches
And none of these three things is 100% automatable.
The 2nd problem is present only in the Rust ecosystem (as far as my
knowledge goes). Cargo, the official package manager for Rust, can
handle the scenario where an executable depends on a single library in
two different versions [3]. Dnf, on the other hand, will not install two
versions of the same RPM package. What we do now is, that we patch the
affected executables and libraries to only use the newest versions
everywhere. This is again an additional maintenance cost and we create
differences from upstream, because these patches are not necessarily
merged. See this as an example:
https://src.fedoraproject.org/rpms/rust-bstr/blob/master/f/rust-bstr.spec...
https://github.com/BurntSushi/bstr/pull/23
To sum up the 2nd problem: we are using dnf instead of the upstream
package manager to install dependencies. These two approaches can be
incompatible (and they are in case of Rust).
That's not true. Due to the name-version namespacing that Cargo uses,
it's pretty easy to create "compat" packages for different versions of
a library.
This approach is in use in fedora *today*, and it's pretty easy to do
- I've done it myself.
Using "compat" packages also maps upstream versions to RPM versions
exactly, so I don't understand why you call rpm/dnf and Cargo
incompatible here.
The 3rd problem I see is that issues like this are not going away,
they
are only going to get worse as other ecosystems emerge. e.g. if the
Swift language became popular (for Linux development) it would again
have it’s own package manager and probably its own set of issues in
relation to our build system. (I mention Swift specifically because it
is a compiled language that have stable ABI only for std, other
libraries are statically linked)
Of course, not every language ecosystem is equally well-suited for
distribution packaging.
In my experience, C, C++, Python, and Rust libraries all map to RPM
packages pretty well due to the upstream's design of "package
management" (or lack thereof, in the case of C / C++). The only thing
we can really do about this, is work with upstream projects to improve
suitability with distribution packaging.
Now, I’d like to point out that Go and Rust packaging works well in
Fedora due to the enormous effort of certain people and I very much
admire the work they do. But on the other hand I’m afraid where the
ecosystem would go without them. This is where we get to the situation
in the enterprise derivatives, specifically RHEL and CentOS. Their
solution to this problem is not to package all libraries separately but
to bundle all libraries directly into source RPMs of each executable. So
the bundling is not present only on the binary file level, but also on
the source RPM level. Go went even further in this case and it is common
to bundle all the dependencies as a source code directly in the upstream
repository. See this repo as an example:
https://github.com/containers/libpod/tree/master/vendor
I agree that in some cases, using vendored dependencies is the only
workable solution. I ship one such package myself (syncthing), but
only because upstream frequently bumps or adds new dependencies, and I
could not keep up with packaging everything separately (or some
packages were not maintained by me and were outdated).
However, Go packaging is now in a pretty good state in fedora compared
to 1-2 years ago, with pretty good tooling in place to make packaging
easier.
Rust packaging is a bit special here, since Igor decided to only
package things for rawhide, not do compat packages, and submit stable
builds via koji hackery. I've talked with him about this situation,
but we disagree about the approach to take for packaging rust for
stable releases (where I wouldn't mind keeping compat packages around
and, in general, do things like other language ecosystems do them,
e.g. only introduce big version bumps of libraries in rawhide).
It is fair to say, that my first motivation was the current state of
packaging in RHEL but I’d prefer to discuss this in Fedora first.
The proposal itself is fairly simple: Let’s stop packaging all Go and
Rust libraries into RPM and install them to the buildroot in the
upstream format instead.
As I said (and others have pointed out as well), this is not possible
right now, primarily because automation cannot verify licensing and
write and apply necessary patches without human intervention. And when
you start curating your own Cargo registry, you might as well just
package them as RPMs ... and the whole "use upstream format instead"
probably won't work for Go unless you want to spin up fake github.com
/ bitbucket.com / gitlab.com servers for Go to download sources from.
The specific implementation is up for a discussion but I think it is
logical to start by asking what features do we want from the build
system? My answers are:
* We want to know what exactly was used to build the RPM to ensure
integrity. This is possible with upstream tarballs as well as with RPMs.
Just store a hash of the tarball.
* There should be no maintenance cost. If we would avoid modifying the
upstream package it would be the ideal case.
* It should be possible to patch the library in case of severe issues
like CVEs. This is a contradiction to the previous point, but it could
be solved by unpacking the upstream package into a git repository and
creating a new, modified package in upstream format from it.
I'd argue that effectively maintaining a downstream fork is a bigger
maintenance burden than building a normal RPM package with a patch
file.
* It should be possible to run the build locally in exactly the
same
way Koji does it. This is just about exposing our “registry” of packages
to the public Internet.
That won't work for Go, as explained above (since go downloads sources
based on import path, which is just a git repository somewhere).
Finally, the implementation could be something like this:
A service like release-monitoring could monitor the upstream projects
for new releases (e.g. NPM has a RSS feed). Every time there is a new
release it would automatically synchronize our own Fedora-specific
registry, which would in turn be accessible in buildroot. Then the RPM
macros like “cargo_build” could be modified to use this registry.
If we wanted to have the possibility to patch these libraries we could
synchronize the upstream release into our git repository and generate
packages in upstream format from it.
What do you think about the way we build RPM packages for Go and Rust?
Do you see other solutions or is it ok the way it is? (Please do not
suggest to “fix” the upstream communities)
First, why not?
Additionally, the Go project has for years been working on a module
system that would make these things easier, and Rust/cargo already
maps quite nicely onto RPM packaging ...
Thanks for your opinions!
Martin
FAQ:
* Does it mean you want to get rid of -devel subpackages for C libraries?
No. First of all C language does not have a single package manager and
it is still common to install dependencies using the distro specific
package manager. But packaging into RPM also comes with the advantage of
having shared libraries so in this case it has a positive effect on the
end-user experience. Rust and Go libraries on the other hand are mostly
useless for end-users.
They might be useless for end users, but they still provide benefit
for the distribution and packagers.
Primarily, it provides a single "source of truth" for package dependencies.
Secondarily, it provides a consistent way to run test suites for *all*
libraries. When using "bundled" or "upstream" dependencies, the test
suites for the *libraries* are never run. When you package them
separately, the test suites are run for everything. This ensures
quality, and lets packagers spot regressions and breakage sooner.
* I prefer to install development libraries using the distro
package
manager instead of the upstream package manager. Why do you want to
change it?
This is the case for C development, but it is different in most of the
modern languages. For example we ship Python packages, but we discourage
people from using them for their own development. Python developers are
instead encouraged to use pip to create a virtual environment and use
upstream packages. In the case of Rust, using the system libraries would
be much harder than using the upstream package manager.
I don't know how others do this, but I exclusively develop and run my
projects against the distro's python3 packages. That's where I expect
my projects to run, so that's what I'm developing them against. I've
never used pip or virtualenv to get dependencies.
Fabio
> * Does it mean you want to get rid of Rust packages like ripgrep.
>
> No. The end user experience stays unchanged. Packages that we ship to
> Fedora users must be in RPM format, it is only the buildroot that changes.
>
> Notes:
> [1] Please don’t discuss if it is a good idea or not. This is a
> discussion for upstream of those languages.
> [2] Again, please don’t discuss static vs. dynamic linking here.
> [3] https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
7:11 a.m.
----- Original Message -----
From: "Fabio Valentini" <decathorpe(a)gmail.com>
To: "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Friday, February 21, 2020 4:46:52 PM
Subject: Re: Include non-RPM content in buildroot
On Fri, Feb 21, 2020 at 3:58 PM Martin Sehnoutka <msehnout(a)redhat.com> wrote:
>
> Hi,
Hi!
I think several of your assumptions are not necessarily correct, which
might lead to wrong conclusions.
My responses are inline below.
I second that ;).
> before I write the proposal itself I just want to stress the fact that
> it isn’t my intention to change the current packaging workflow and
> definitely not the user experience. Also if you have C or Python
> packages it would not affect your work at all (I’m not familiar with all
> interpreted languages in Fedora, but I guess it is similar to Python and
> therefore it is not affected by the problems I am going to describe).
(snip)
> First of all, let me describe the problem I see in our Fedora ecosystem
> with relation to Go and Rust language ecosystems. More specifically in
> the relation between RPM buildroot and packages in these ecosystems.
> Both of these languages follow the idea that packages should be small
> and only have a limited set of features. Developers then use a lot of
> these packages to write the final executable that is meant for end-users
> [1]. Also both of these languages use static linking of the final
> binaries meaning that Fedora users don’t install RPM packages of these
> libraries as they are already baked inside of the binary [2].
One big difference between Go and Rust is that Go only namespaces
libraries by their import path (with some exceptions, like gopkg.in),
whereas Rust/cargo namespaces everything by name *and* version, which
makes some things a lot easier. Then there's also the fact that a lot
of Go libraries don't even *have* versions, and most only reference
dependencies by "this git repository". There's some effort to make
this better, but Go modules are not ready yet.
With Go modules("upstream ones") you are getting that name + version/tag(and
rules what to use). Definitively many packages will not move to the modules/best
practices, if that happen, we should try to move them to modules/best practices or move
depending package to different package that is still maintained or in worst case scenario
take over the maintenance -> fork.
> The 1st problem is that if we want to build RPMs of the final
> executables the way we do now, we need to package all these small
> libraries into RPM even though they are just build dependencies and
> users never install RPMs of these libraries. Many of these RPMs are
> automatically generated from the upstream packages meaning that we don’t
> do anything except for unpacking the upstream package (e.g. plain
> tarball in case of crates.io) and then we package the same into RPM.
> This process is unfortunately not fully automated and therefore requires
> a certain amount of human effort.
>
>
> To sum up the previous paragraph, I don’t think it is necessary to
> repackage upstream tarball into a downstream RPM.
Of course this requires a certain amount of human effort - because
that's where the packager does their work. This includes:
- verifying that the specified license is correct and suitable for
(re)distribution in fedora
- verify that the package builds and doesn't do crazy things
- fix upstream and integration issues with patches
And none of these three things is 100% automatable.
Seconding that.
> The 2nd problem is present only in the Rust ecosystem (as far as my
> knowledge goes). Cargo, the official package manager for Rust, can
> handle the scenario where an executable depends on a single library in
> two different versions [3]. Dnf, on the other hand, will not install two
> versions of the same RPM package. What we do now is, that we patch the
> affected executables and libraries to only use the newest versions
> everywhere. This is again an additional maintenance cost and we create
> differences from upstream, because these patches are not necessarily
> merged. See this as an example:
> https://src.fedoraproject.org/rpms/rust-bstr/blob/master/f/rust-bstr.spec...
> https://github.com/BurntSushi/bstr/pull/23
(snip)
> It is fair to say, that my first motivation was the current state of
> packaging in RHEL but I’d prefer to discuss this in Fedora first.
>
>
> The proposal itself is fairly simple: Let’s stop packaging all Go and
> Rust libraries into RPM and install them to the buildroot in the
> upstream format instead.
As I said (and others have pointed out as well), this is not possible
right now, primarily because automation cannot verify licensing and
write and apply necessary patches without human intervention. And when
you start curating your own Cargo registry, you might as well just
package them as RPMs ... and the whole "use upstream format instead"
probably won't work for Go unless you want to spin up fake github.com
/ bitbucket.com / gitlab.com servers for Go to download sources from.
I think you could spin up Go proxy with whitelisted packages(name/versions/tags/hashes)
inside the Fedora's/Koji's network, but I think that with that we would be
creating totally separate and different package build workflow to current dist-git and
lookaside cache, that could work, but it would needed to be backed by lot of committed
resources(people, HW,...).
> The specific implementation is up for a discussion but I think
it is
> logical to start by asking what features do we want from the build
> system? My answers are:
>
>
> * We want to know what exactly was used to build the RPM to ensure
> integrity. This is possible with upstream tarballs as well as with RPMs.
> Just store a hash of the tarball.
> * There should be no maintenance cost. If we would avoid modifying the
> upstream package it would be the ideal case.
> * It should be possible to patch the library in case of severe issues
> like CVEs. This is a contradiction to the previous point, but it could
> be solved by unpacking the upstream package into a git repository and
> creating a new, modified package in upstream format from it.
I'd argue that effectively maintaining a downstream fork is a bigger
maintenance burden than building a normal RPM package with a patch
file.
Effectively they are the same thing. IMHO maintaining "fork", i.e. Git repo
with "Fedora" patches, is easier that just having some patch files laying around
in dist git, thanks to the git cli tools(rebases and patch file generation for dist-git)
and better discover-ability by outsiders(and automation possibilities,...), if present in
public.
> * It should be possible to run the build locally in exactly
the same
> way Koji does it. This is just about exposing our “registry” of packages
> to the public Internet.
That won't work for Go, as explained above (since go downloads sources
based on import path, which is just a git repository somewhere).
It can, see Go proxy(just to illustrate, one of the possible implementations
https://goproxy.io/).
> Finally, the implementation could be something like this:
> A service like release-monitoring could monitor the upstream projects
> for new releases (e.g. NPM has a RSS feed). Every time there is a new
> release it would automatically synchronize our own Fedora-specific
> registry, which would in turn be accessible in buildroot. Then the RPM
> macros like “cargo_build” could be modified to use this registry.
> If we wanted to have the possibility to patch these libraries we could
> synchronize the upstream release into our git repository and generate
> packages in upstream format from it.
>
>
> What do you think about the way we build RPM packages for Go and Rust?
> Do you see other solutions or is it ok the way it is? (Please do not
> suggest to “fix” the upstream communities)
First, why not?
Additionally, the Go project has for years been working on a module
system that would make these things easier, and Rust/cargo already
maps quite nicely onto RPM packaging ...
You have to always start with the upstream(however that might be hard and tedious, to
build the relationships,...), if you don't, you will just end up swamped with
"unupstreamable" patches and continuously slip in to the "I don't
care" mode forcing the upstream to slip in to the "I don't care" mode
too.
Overall I think that automating the packaging(including most/all of review) is the way to
go so packagers don't need to really write any biolerplate(even cleaned up one), but
that is really complex and still lots and lots of work away.
JC
> Thanks for your opinions!
> Martin
>
>
> FAQ:
> * Does it mean you want to get rid of -devel subpackages for C libraries?
>
> No. First of all C language does not have a single package manager and
> it is still common to install dependencies using the distro specific
> package manager. But packaging into RPM also comes with the advantage of
> having shared libraries so in this case it has a positive effect on the
> end-user experience. Rust and Go libraries on the other hand are mostly
> useless for end-users.
They might be useless for end users, but they still provide benefit
for the distribution and packagers.
Primarily, it provides a single "source of truth" for package dependencies.
Secondarily, it provides a consistent way to run test suites for *all*
libraries. When using "bundled" or "upstream" dependencies, the test
suites for the *libraries* are never run. When you package them
separately, the test suites are run for everything. This ensures
quality, and lets packagers spot regressions and breakage sooner.
> * I prefer to install development libraries using the distro package
> manager instead of the upstream package manager. Why do you want to
> change it?
>
> This is the case for C development, but it is different in most of the
> modern languages. For example we ship Python packages, but we discourage
> people from using them for their own development. Python developers are
> instead encouraged to use pip to create a virtual environment and use
> upstream packages. In the case of Rust, using the system libraries would
> be much harder than using the upstream package manager.
I don't know how others do this, but I exclusively develop and run my
projects against the distro's python3 packages. That's where I expect
my projects to run, so that's what I'm developing them against. I've
never used pip or virtualenv to get dependencies.
Fabio
> * Does it mean you want to get rid of Rust packages like ripgrep.
>
> No. The end user experience stays unchanged. Packages that we ship to
> Fedora users must be in RPM format, it is only the buildroot that changes.
>
> Notes:
> [1] Please don’t discuss if it is a good idea or not. This is a
> discussion for upstream of those languages.
> [2] Again, please don’t discuss static vs. dynamic linking here.
> [3] https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
3:55 a.m.
Dne 21. 02. 20 v 15:57 Martin Sehnoutka napsal(a):
This process is unfortunately not fully automated and therefore
requires a certain amount of human effort.
This is an important sentence. Let's save it for later as [1]
The proposal itself is fairly simple: Let’s stop packaging all Go and
Rust libraries into RPM and install them to the
buildroot in the upstream format instead.
...
* We want to know what exactly was used to build the RPM to ensure integrity. This is
possible with upstream tarballs
as well as with RPMs. Just store a hash of the tarball.
* There should be no maintenance cost. If we would avoid modifying the upstream package
it would be the ideal case.
* It should be possible to patch the library in case of severe issues like CVEs. This is
a contradiction to the
previous point, but it could be solved by unpacking the upstream package into a git
repository and creating a new,
modified package in upstream format from it.
* It should be possible to run the build locally in exactly the same way Koji does it.
This is just about exposing our
“registry” of packages to the public Internet.
Whoa. That is a lots of requirements. You can easily get lost in them. And very likely
unable to reach your goal.
Finally, the implementation could be something like this:
A service like release-monitoring could monitor the upstream projects for new releases
(e.g. NPM has a RSS feed). Every
time there is a new release it would automatically synchronize our own Fedora-specific
registry, which would in turn be
accessible in buildroot.
So bits in wild internet suddenly become clean when we duplicate them into our
infrastructure?
And this is a lot of bits. The storage has a price :(
Additionally, even when the build in Koji (or Mock in general) is offline, the
dependencies are installed with internet
enabled. If you teach Mock how to call native crate/rubygem/.. before the actual build
start, you will have most of the
work done.
Many people tried to achieve this (non-RPM content in rpm). It may sound simple, but this
is a **huge** task. And for
example: 1) making this technically possible and 2) allow this in Fedora are completely
different task with different
ETA and different chances for success.
Why are you trying to come with this proposal? Do you think it will be much easier than
automating the packaging? (see
[1] above). I, personally, think automating the packaging is a better way to go. Still
huge, though.
What I would recommend is either:
1) Focus on the automation. Especially automation of license check. There has been several
mentiond on this list in past
few months. We can learn a lot from OpenSuse regarding this.
2) Improve the automation of packaging.
See https://copr.fedorainfracloud.org/coprs/iucar/cran/ or our previous attempts to
rebuild PYPI and Rubygems where we
imporoved spec generators a lot.
3) If you really want to focus on non-RPM content - you may add something like this to
SPEC files:
BuildRequires: native::crate(foo)
with the semantics that rpm-build:
1) will ignore anything with native:: prefix
2) or check if foo is present on system using crate (and not as rpm)
This has a benefit that you need to modify only rpm-build, but not rpm itself.
When you choose the first option, then you can levarage
https://fedoraproject.org/wiki/Changes/DynamicBuildRequires
and emit this dependency and it can be catched by Mock and installed before the rpm-build
is called.
Focus on making this technically possible. And only then start discussion if we want to
have this enabled in Fedora.
This will give you a grace period where this feature can mature.
--
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
4:17 a.m.
On 24.02.2020 10:55, Miroslav Suchý wrote:
Additionally, even when the build in Koji (or Mock in general) is
offline, the dependencies are installed with internet
enabled. If you teach Mock how to call native crate/rubygem/.. before the actual build
start, you will have most of the
work done.
All packages must be build with disabled network access to ensure, that
it use packaged dependencies and not downloaded from outside.
Remember about malwares/breakages in npm.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
6:41 a.m.
Dne 24. 02. 20 v 11:17 Vitaly Zaitsev via devel napsal(a):
All packages must be build with disabled network access to ensure,
that
it use packaged dependencies and not downloaded from outside.
Yes. But it is only rpm-build what cannot access network. Mock itself can access network.
In other words. (see comments inline of output)
###### internet access here ####################
Mock Version: 2.0
INFO: Mock Version: 2.0
Start: dnf install
No matches found for the following disable plugin patterns: local, spacewalk
fedora
16 MB/s | 71 MB 00:04
updates
14 MB/s | 21 MB 00:01
....
Complete!
Finish: dnf install
Start: creating root cache
Finish: creating root cache
Finish: chroot init
INFO: Installed packages:
Start: build phase for mock-core-configs-32.1-1.fc31.src.rpm
Start: build setup for mock-core-configs-32.1-1.fc31.src.rpm
########### No internet from here - Mock executing rpm-build #########
Building target platforms: x86_64
Building for target x86_64
setting SOURCE_DATE_EPOCH=1581120000
Wrote: /builddir/build/SRPMS/mock-core-configs-32.1-1.fc31.src.rpm
########### Internet enabled again #########
No matches found for the following disable plugin patterns: local, spacewalk
fedora
42 kB/s | 24 kB 00:00
updates
45 kB/s | 21 kB 00:00
Dependencies resolved.
Nothing to do.
Complete!
Finish: build setup for mock-core-configs-32.1-1.fc31.src.rpm
Start: Outputting list of installed packages
Finish: Outputting list of installed packages
Start: rpmbuild mock-core-configs-32.1-1.fc31.src.rpm
########### No internet from here - Mock executing rpm-build #########
Building target platforms: x86_64
Building for target x86_64
setting SOURCE_DATE_EPOCH=1581120000
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.iN2SBw
....
Finish: rpmbuild mock-core-configs-32.1-1.fc31.src.rpm
Finish: build phase for mock-core-configs-32.1-1.fc31.src.rpm
########### Internet enabled again #########
INFO: Done(/tmp/tito/mock-core-configs-32.1-1.fc31.src.rpm) Config(default) 3 minutes 28
seconds
INFO: Results and/or logs in: /var/lib/mock/fedora-31-x86_64/result
Finish: run
--
Miroslav Suchy, RHCA
Red Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys
7:46 a.m.
On 24.02.2020 13:41, Miroslav Suchý wrote:
Yes. But it is only rpm-build what cannot access network. Mock itself
can access network.
Mock is using Internet connection only for downloading metadata and
packages from repositories. Then it use systemd-nspawn to create a
protected isolated container and run rpmbuild there with no Internet access.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
12:40 p.m.
On Mon, Feb 24, 2020 at 02:46:02PM +0100, Vitaly Zaitsev via devel wrote:
On 24.02.2020 13:41, Miroslav Suchý wrote:
> Yes. But it is only rpm-build what cannot access network. Mock itself can access
network.
Mock is using Internet connection only for downloading metadata and
packages from repositories. Then it use systemd-nspawn to create a
protected isolated container and run rpmbuild there with no Internet access.
To further clarify here... yes, mock can access the internet.
However, Fedora builders also have a restrictive firewall set, allowing
them to only access a few specific sites. They can reach koji/kojipkgs,
they can reach the master mirrors, they can reach pagure.io (for things
like kickstarts and comps), they can reach src.fedoraproject.org (for
things like spec files/sources files), they can reach the fedora
regestry. Thats about it. Everything else is just denied.
kevin
2:50 a.m.
Hi,
thank you all for your input! I have few notes:
Review process:
I don't think that the review process is a problem. It is independent of
the file format we use. I mean we can have regular review process and
store the package in a tarball as well as we do it with RPM.
Go package management:
I know that Go has a package management now, but the question is if
upstream communities are going to adopt it.
Storage requirements:
Regarding the amount of packages, of course I don't want to clone the
whole registry, I just want to do the same we already do, but without RPM.
-compat packages for Rust:
I, personally, don't like the proposed solution because it requires a
lot of manual work and my goal is to remove as much manual work as possible.
mock:
I am aware that we don't have Internet access during RPM build, but as I
said above, we can do the same with tarball and RPM.
Thanks for this email thread I also had few discussions off-line and it
seems to me that there is a certain shift in the way people want to
distribute their software. More specifically I could see more people
focus on shipping their software in containers and trying to avoid RPM
completely.
Regards,
Martin
On 2/21/20 3:57 PM, Martin Sehnoutka wrote:
Hi,
before I write the proposal itself I just want to stress the fact that
it isn’t my intention to change the current packaging workflow and
definitely not the user experience. Also if you have C or Python
packages it would not affect your work at all (I’m not familiar with all
interpreted languages in Fedora, but I guess it is similar to Python and
therefore it is not affected by the problems I am going to describe).
First of all, let me describe the problem I see in our Fedora ecosystem
with relation to Go and Rust language ecosystems. More specifically in
the relation between RPM buildroot and packages in these ecosystems.
Both of these languages follow the idea that packages should be small
and only have a limited set of features. Developers then use a lot of
these packages to write the final executable that is meant for end-users
[1]. Also both of these languages use static linking of the final
binaries meaning that Fedora users don’t install RPM packages of these
libraries as they are already baked inside of the binary [2].
The 1st problem is that if we want to build RPMs of the final
executables the way we do now, we need to package all these small
libraries into RPM even though they are just build dependencies and
users never install RPMs of these libraries. Many of these RPMs are
automatically generated from the upstream packages meaning that we don’t
do anything except for unpacking the upstream package (e.g. plain
tarball in case of crates.io) and then we package the same into RPM.
This process is unfortunately not fully automated and therefore requires
a certain amount of human effort.
To sum up the previous paragraph, I don’t think it is necessary to
repackage upstream tarball into a downstream RPM.
The 2nd problem is present only in the Rust ecosystem (as far as my
knowledge goes). Cargo, the official package manager for Rust, can
handle the scenario where an executable depends on a single library in
two different versions [3]. Dnf, on the other hand, will not install two
versions of the same RPM package. What we do now is, that we patch the
affected executables and libraries to only use the newest versions
everywhere. This is again an additional maintenance cost and we create
differences from upstream, because these patches are not necessarily
merged. See this as an example:
https://src.fedoraproject.org/rpms/rust-bstr/blob/master/f/rust-bstr.spec...
https://github.com/BurntSushi/bstr/pull/23
To sum up the 2nd problem: we are using dnf instead of the upstream
package manager to install dependencies. These two approaches can be
incompatible (and they are in case of Rust).
The 3rd problem I see is that issues like this are not going away, they
are only going to get worse as other ecosystems emerge. e.g. if the
Swift language became popular (for Linux development) it would again
have it’s own package manager and probably its own set of issues in
relation to our build system. (I mention Swift specifically because it
is a compiled language that have stable ABI only for std, other
libraries are statically linked)
Now, I’d like to point out that Go and Rust packaging works well in
Fedora due to the enormous effort of certain people and I very much
admire the work they do. But on the other hand I’m afraid where the
ecosystem would go without them. This is where we get to the situation
in the enterprise derivatives, specifically RHEL and CentOS. Their
solution to this problem is not to package all libraries separately but
to bundle all libraries directly into source RPMs of each executable. So
the bundling is not present only on the binary file level, but also on
the source RPM level. Go went even further in this case and it is common
to bundle all the dependencies as a source code directly in the upstream
repository. See this repo as an example:
https://github.com/containers/libpod/tree/master/vendor
It is fair to say, that my first motivation was the current state of
packaging in RHEL but I’d prefer to discuss this in Fedora first.
The proposal itself is fairly simple: Let’s stop packaging all Go and
Rust libraries into RPM and install them to the buildroot in the
upstream format instead.
The specific implementation is up for a discussion but I think it is
logical to start by asking what features do we want from the build
system? My answers are:
* We want to know what exactly was used to build the RPM to ensure
integrity. This is possible with upstream tarballs as well as with RPMs.
Just store a hash of the tarball.
* There should be no maintenance cost. If we would avoid modifying the
upstream package it would be the ideal case.
* It should be possible to patch the library in case of severe issues
like CVEs. This is a contradiction to the previous point, but it could
be solved by unpacking the upstream package into a git repository and
creating a new, modified package in upstream format from it.
* It should be possible to run the build locally in exactly the same
way Koji does it. This is just about exposing our “registry” of packages
to the public Internet.
Finally, the implementation could be something like this:
A service like release-monitoring could monitor the upstream projects
for new releases (e.g. NPM has a RSS feed). Every time there is a new
release it would automatically synchronize our own Fedora-specific
registry, which would in turn be accessible in buildroot. Then the RPM
macros like “cargo_build” could be modified to use this registry.
If we wanted to have the possibility to patch these libraries we could
synchronize the upstream release into our git repository and generate
packages in upstream format from it.
What do you think about the way we build RPM packages for Go and Rust?
Do you see other solutions or is it ok the way it is? (Please do not
suggest to “fix” the upstream communities)
Thanks for your opinions!
Martin
FAQ:
* Does it mean you want to get rid of -devel subpackages for C libraries?
No. First of all C language does not have a single package manager and
it is still common to install dependencies using the distro specific
package manager. But packaging into RPM also comes with the advantage of
having shared libraries so in this case it has a positive effect on the
end-user experience. Rust and Go libraries on the other hand are mostly
useless for end-users.
* I prefer to install development libraries using the distro package
manager instead of the upstream package manager. Why do you want to
change it?
This is the case for C development, but it is different in most of the
modern languages. For example we ship Python packages, but we discourage
people from using them for their own development. Python developers are
instead encouraged to use pip to create a virtual environment and use
upstream packages. In the case of Rust, using the system libraries would
be much harder than using the upstream package manager.
* Does it mean you want to get rid of Rust packages like ripgrep?
No. The end user experience stays unchanged. Packages that we ship to
Fedora users must be in RPM format, it is only the buildroot that changes.
Notes:
[1] Please don’t discuss if it is a good idea or not. This is a
discussion for upstream of those languages.
[2] Again, please don’t discuss static vs. dynamic linking here.
[3] https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
3:39 a.m.
Le 2020-02-26 09:50, Martin Sehnoutka a écrit :
Hi,
Hi,
Go package management:
I know that Go has a package management now, but the question is if
upstream communities are going to adopt it.
Upstream communities won’t have any choice if they want their software
to be trusted by third parties, because all the upstream "free" security
checking provided by Google in its own registry relies on the new
component model.
Software security is hard. Security of huge piles of unmanaged third
party bundled code is not economically feasible. That’s why Google moved
to a formal component system, for the exact same reasons distros moved
to formal packages 20+ years ago. (and it was all done in a NIH way
Google side, they’re re-discovering all the packaging lessons distros
learnt long ago one by one).
Thanks for this email thread I also had few discussions off-line and
it seems to me that there is a certain shift in the way people want to
distribute their software. More specifically I could see more people
focus on shipping their software in containers and trying to avoid RPM
completely.
You can see it because devs keep hoping for a free lunch. That does not
exist in the real life. Real world software has maintenance and security
issues. Managing those requires a finer grained component model than
shoveling piles of unmanaged code in a container and hoping for the
best.
Upstreams that do not make the effort to manage the third party code
they rely on, condemn themselves to obscurity, or to revenue capture by
third parties that *will* transform their code in something manageable
(typically, by breaking it in components that can be audited).
Typically, Amazon, Google, Red Hat, etc.
Regards,
--
Nicolas Mailhot
4:18 a.m.
Hi Martin,
I was quite busy lately so did not have time to reply.
(Most of the time I'll speak for Rust ecosystem below)
On Fri, Feb 21, 2020 at 4:06 PM Martin Sehnoutka <msehnout(a)redhat.com> wrote:
Hi,
before I write the proposal itself I just want to stress the fact that
it isn’t my intention to change the current packaging workflow and
definitely not the user experience. Also if you have C or Python
packages it would not affect your work at all (I’m not familiar with all
interpreted languages in Fedora, but I guess it is similar to Python and
therefore it is not affected by the problems I am going to describe).
I disagree, Python has same registry (PyPI) as Rust does (crates.io)
and. Of course, there are different problems in different ecosystems,
but approach should be same to all.
First of all, let me describe the problem I see in our Fedora ecosystem
with relation to Go and Rust language ecosystems. More specifically in
the relation between RPM buildroot and packages in these ecosystems.
Both of these languages follow the idea that packages should be small
and only have a limited set of features. Developers then use a lot of
these packages to write the final executable that is meant for end-users
[1]. Also both of these languages use static linking of the final
binaries meaning that Fedora users don’t install RPM packages of these
libraries as they are already baked inside of the binary [2].
Technically, we can do dynamic linking. The problem is that there is
no stable ABI, so even rebuild of the same sources with different
version of compiler can result into a different thing.
I can't speak for Go, but in Rust the crate (aka library) can be used
(compiled) with different features. So we would either need to produce
libraries with all combinations of those features set or ship one big
library with all of them included. That would mean, the dependency set
will grow and most likely we won't save any space, or even worse,
waste it. Until some amount of packages which are common on all
systems, of course.
By the way, Haskell has very similar problem, they just put hashes all
over the place to ensure that all packages are rebuilt.
The 1st problem is that if we want to build RPMs of the final
executables the way we do now, we need to package all these small
libraries into RPM even though they are just build dependencies and
users never install RPMs of these libraries. Many of these RPMs are
automatically generated from the upstream packages meaning that we don’t
do anything except for unpacking the upstream package (e.g. plain
tarball in case of crates.io) and then we package the same into RPM.
This process is unfortunately not fully automated and therefore requires
a certain amount of human effort.
As others pointed out, we do audit licenses. But apart from that, we also do:
1) cargo build and cargo test to ensure that the crate actually works
2) Try to patch them to rely on latest version of crates
3) Add patches to use system versions of libraries (libcurl,
oniguruma, libssh2, …)
We did found many issues related to 32bit, endianess and even when
bumping version of dependency, you can find that there is actual bug
in the code of crate
(https://github.com/budziq/rust-skeptic/pull/116#issuecomment-590009805)
To sum up the previous paragraph, I don’t think it is necessary to
repackage upstream tarball into a downstream RPM.
If it is automated, why do you think so? If you have different
ecosystems, being able to check something across all ecosystems in the
same way (aka using just rpm instead of pip, cargo, go, …).
The 2nd problem is present only in the Rust ecosystem (as far as my
knowledge goes). Cargo, the official package manager for Rust, can
handle the scenario where an executable depends on a single library in
two different versions [3]. Dnf, on the other hand, will not install two
versions of the same RPM package. What we do now is, that we patch the
affected executables and libraries to only use the newest versions
everywhere. This is again an additional maintenance cost and we create
differences from upstream, because these patches are not necessarily
merged. See this as an example:
https://src.fedoraproject.org/rpms/rust-bstr/blob/master/f/rust-bstr.spec...
https://github.com/BurntSushi/bstr/pull/23
As others, pointed out - we can (and we do) compat packages.
What do you propose to do if there is, let's say vulnerability in the
version some crate depends on and upstream is not responsibe?
95+% of our patches are merged into upstreams sooner or later
(excluding dead upstreams).
To sum up the 2nd problem: we are using dnf instead of the upstream
package manager to install dependencies. These two approaches can be
incompatible (and they are in case of Rust).
That's not exactly true, we do mirror what Cargo does with RPM. And
that works pretty well.
The 3rd problem I see is that issues like this are not going away, they
are only going to get worse as other ecosystems emerge. e.g. if the
Swift language became popular (for Linux development) it would again
have it’s own package manager and probably its own set of issues in
relation to our build system. (I mention Swift specifically because it
is a compiled language that have stable ABI only for std, other
libraries are statically linked)
So would it have the same problem if we want to have mirror/patched
registry of the language X in Fedora which would comply with our
policies / requirements.
Now, I’d like to point out that Go and Rust packaging works well in
Fedora due to the enormous effort of certain people and I very much
admire the work they do. But on the other hand I’m afraid where the
ecosystem would go without them. This is where we get to the situation
in the enterprise derivatives, specifically RHEL and CentOS. Their
solution to this problem is not to package all libraries separately but
to bundle all libraries directly into source RPMs of each executable. So
the bundling is not present only on the binary file level, but also on
the source RPM level. Go went even further in this case and it is common
to bundle all the dependencies as a source code directly in the upstream
repository. See this repo as an example:
https://github.com/containers/libpod/tree/master/vendor
Bundling is bad, that's it. I'm not going to write why it is bad and
what disadvantages it has.
I think the real problem here is that our Fedora Build System just
suck. The hours I spent on building rust packages in Fedora is because
of this, nothing more.
Another part of that time is spent on things you've described above
(like patching crate to use latest versions of deps). Technically I
could have created bunch of compat packages, but I call this just
lazyness, if you port to latest versions of deps, you have to care
about just one copy of a library and that's it.
It is fair to say, that my first motivation was the current state of
packaging in RHEL but I’d prefer to discuss this in Fedora first.
The proposal itself is fairly simple: Let’s stop packaging all Go and
Rust libraries into RPM and install them to the buildroot in the
upstream format instead.
No?
The specific implementation is up for a discussion but I think it is
logical to start by asking what features do we want from the build
system? My answers are:
* We want to know what exactly was used to build the RPM to ensure
integrity. This is possible with upstream tarballs as well as with RPMs.
Just store a hash of the tarball.
That's not enough, you need to also store all sources.
* There should be no maintenance cost. If we would avoid modifying
the
upstream package it would be the ideal case.
Of course we can't. If you automate importing of the packages in RPM
format in Fedora in 90%, that would be much more beneficial.
* It should be possible to patch the library in case of severe
issues
like CVEs. This is a contradiction to the previous point, but it could
be solved by unpacking the upstream package into a git repository and
creating a new, modified package in upstream format from it.
* It should be possible to run the build locally in exactly the same
way Koji does it. This is just about exposing our “registry” of packages
to the public Internet.
Finally, the implementation could be something like this:
A service like release-monitoring could monitor the upstream projects
for new releases (e.g. NPM has a RSS feed). Every time there is a new
release it would automatically synchronize our own Fedora-specific
registry, which would in turn be accessible in buildroot. Then the RPM
macros like “cargo_build” could be modified to use this registry.
If we wanted to have the possibility to patch these libraries we could
synchronize the upstream release into our git repository and generate
packages in upstream format from it.
You forgot one very important part here: you need to run tests on
every package of the registry every time their dependencies get update
and/or buildroot changes.
What do you think about the way we build RPM packages for Go and Rust?
Do you see other solutions or is it ok the way it is? (Please do not
suggest to “fix” the upstream communities)
I think we:
* build them fine
* should continue on improving RPM to hide all non-interesting details
from maintainers
* should improve our buildsystem
* automate most of the things
Thanks for your opinions!
Martin
FAQ:
* Does it mean you want to get rid of -devel subpackages for C libraries?
No. First of all C language does not have a single package manager and
it is still common to install dependencies using the distro specific
package manager. But packaging into RPM also comes with the advantage of
having shared libraries so in this case it has a positive effect on the
end-user experience. Rust and Go libraries on the other hand are mostly
useless for end-users.
* I prefer to install development libraries using the distro package
manager instead of the upstream package manager. Why do you want to
change it?
This is the case for C development, but it is different in most of the
modern languages. For example we ship Python packages, but we discourage
people from using them for their own development. Python developers are
instead encouraged to use pip to create a virtual environment and use
upstream packages. In the case of Rust, using the system libraries would
be much harder than using the upstream package manager.
That's not true. It is pretty easy to use them in Rust ecosystem, just
drop one configuration file. It is just that no one yet implemented
support for having multiple registries in Cargo (in the sense you can
use system crates together with crates.io one).
>
> * Does it mean you want to get rid of Rust packages like ripgrep?
>
> No. The end user experience stays unchanged. Packages that we ship to
> Fedora users must be in RPM format, it is only the buildroot that changes.
>
> Notes:
> [1] Please don’t discuss if it is a good idea or not. This is a
> discussion for upstream of those languages.
> [2] Again, please don’t discuss static vs. dynamic linking here.
> [3] https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
1526
days inactive
1531
days old
18 comments
12 participants
participants (12)
-
Ankur Sinha -
Fabio Valentini -
Igor Gnatenko -
Jakub Cajka -
Kevin Fenzi -
Martin Sehnoutka -
Matthew Miller -
Miroslav Suchý -
Nicolas Mailhot -
Randy Barlow -
Stephen John Smoogen -
Vitaly Zaitsev