9:32 a.m.
https://fedoraproject.org/wiki/Changes/RpmSequoia
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.
== Summary ==
Change RPM to use [https://sequoia-pgp.org/ Sequoia] based OpenPGP
parser instead of it's own, flawed and limited implementation.
== Owner ==
* Name: [[User:pmatilai| Panu Matilainen]]
* Email: pmatilai(a)redhat.com
== Detailed Description ==
For the last 20 years or so, RPM has used a home-grown OpenPGP parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
At this point the change is mostly invisible in normal daily use.
== Feedback ==
== Benefit to Fedora ==
The main, direct benefit to Fedora is improved security and
standards-compliance (RFC-4880) in one of the corner-stones of the
whole distribution. Longer term, we can expect better error messages
and other functional improvements regarding key and signature
handling.
== Scope ==
* Proposal owners:
** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499
package/review rpm-sequoia]
** Build rpm with --with-crypto=sequoia
** Watch out for the unexpected
* Other developers:
** Help [https://bugzilla.redhat.com/show_bug.cgi?id=2087499
package/review rpm-sequoia]
* Release engineering: [https://pagure.io/releng/issue/11077 #11077]
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
Within Fedora package set, this has no impact as everything is already
using sufficiently strong crypto. Third party repositories / packages
could be signed with insecure crypto, and those may require working
around with --nosignature. However this incidentally overlaps with
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2
which has effectively the same effect on rpm.
== How To Test ==
In general, normal rpm/dnf use provides sufficient test coverage. For
more advanced testers: try signing and verifying with different keys
and their subkeys, using different algorithms etc.
== User Experience ==
For normal usage, the change is quite invisible. The notable exceptions are
- Some old, insecure (MD5/SHA1 based) signatures are rejected (this is
in line with the stronger crypto settings proposed elsewhere for F38)
- Key import may accept some previously rejected keys, in part due to
limitations of old parser etc but in particular, the old
implementation verifies self-signatures at import time whereas Sequoia
verifies them at time of use.
- Key import may reject some previously accepted keys due to better validation.
== Dependencies ==
The change introduces one new direct dependency:
[https://github.com/rpm-software-management/rpm-sequoia/ rpm-sequoia].
The rpm-sequoia package also takes over other crypto besides OpenPGP,
currently Sequoia uses nettle as its low-level crypto provider, but
work is underway to
[https://gitlab.com/sequoia-pgp/sequoia/-/merge_requests/1361 support
openssl in Sequoia], and the plan is to have Sequoia in Fedora use
that once it becomes available. This plan
[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
has support of the crypto team].
== Contingency Plan ==
* Contingency mechanism: Revert back to the internal PGP parser
* Contingency deadline: Beta release
* Blocks release? No
== Documentation ==
There's not much in the way of documentation as there's not much to
document, except for the deprecation of the internal parser:
https://github.com/rpm-software-management/rpm/issues/1935
rpm-sequoia build instructions can be found in
https://github.com/rpm-software-management/rpm-sequoia/
== Release Notes ==
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
12:02 p.m.
Hi Ben,
Ben Cotton <bcotton(a)redhat.com> wrote:
Within Fedora package set, this has no impact as everything is
already
using sufficiently strong crypto. Third party repositories / packages
could be signed with insecure crypto, and those may require working
around with --nosignature. However this incidentally overlaps with
https://fedoraproject.org/wiki/Changes/StrongCryptoSettings3Forewarning2
which has effectively the same effect on rpm.
Note that the StrongCryptoSettings3Forewarning2 proposal recently failed to
gather enough votes to be accepted, so it will likely not be happening (or
not in this form) for Fedora 38.
Additionally, crypto-policies would have supported switching to LEGACY to
allow installation of non-conforming RPMs, so you should at least provide a
method to also install such old RPMs, ideally while still verifying the old
SHA-1 signature rather than ignoring it completely.
HTH,
Clemens
--
Clemens Lang
RHEL Crypto Team
Red Hat
11:18 p.m.
For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
Why are you using a new library written in Rust? Can you not use one of the
existing mature C implementations of OpenPGP? gpgme maybe?
At this point the change is mostly invisible in normal daily use.
Not really, because it makes some packages uninstallable.
- Some old, insecure (MD5/SHA1 based) signatures are rejected (this
is
in line with the stronger crypto settings proposed elsewhere for F38)
Such a hardcoded restriction, without a way for the local administrator to
allow the legacy signatures, is not acceptable.
Kevin Kofler
2:29 a.m.
On 10/13/22 07:18, Kevin Kofler via devel wrote:
> For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
> for dealing with keys and signatures. That parser is rather infamous
> for its limitations and flaws, and especially in recent years has
> proven a significant burden to RPM development. In order to improve
> security and free developer resources for dealing with RPM's "core
> business" instead, RPM upstream is in the process of deprecating the
> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
> based solution written in Rust.
Why are you using a new library written in Rust? Can you not use one of the
existing mature C implementations of OpenPGP? gpgme maybe?
Had there been such an option, we would've switched over years ago.
Gpgme is based around calling and communicating with an external gpg
process, which is a setup you do NOT want in the rpm context where
chroots come and go etc. Also, rpm requires access to the "low-level"
digests-in-progress because it calculates multiple things on a single read.
> At this point the change is mostly invisible in normal daily
use.
Not really, because it makes some packages uninstallable.
Not uninstallable. You can use --nosignature, or resign such packages
with something stronger. Which one is the better course depends on the
situation of course.
> - Some old, insecure (MD5/SHA1 based) signatures are rejected
(this is
> in line with the stronger crypto settings proposed elsewhere for F38)
Such a hardcoded restriction, without a way for the local administrator to
allow the legacy signatures, is not acceptable.
Mind you, I don't exactly agree with this style of explicit disabling
either (see
https://lists.rpm.org/pipermail/rpm-maint/2021-October/018344.html and
onwards). But. I doubt many people realize just how thin the ice is (and
has always been) with the existing parser. I consider this step a matter
of survival, and ultimately some legacy content becoming harder to use
is an acceptable tradeoff for *that*.
I don't know how deep this all is wired inside Sequoia, but I totally
agree (as you see in the thread linked above) that this should be based
on the system crypto policy. As explained in the change, nettle (which
doesn't support the system crypto policies AIUI) should be seen as a
temporary stepstone in Fedora, with a plan to switch to openssl (which
does) in the nearish future.
So technically this is a matter of "Sequoia should honor system crypto
policy", rpm is just a dumb API user here that sometimes get told "nope"
by the underlying libraries, whether due to crypto policy, FIPS or whatever.
- Panu -
2:53 a.m.
On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
On 10/13/22 07:18, Kevin Kofler via devel wrote:
>> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
>> for dealing with keys and signatures. That parser is rather infamous
>> for its limitations and flaws, and especially in recent years has
>> proven a significant burden to RPM development. In order to improve
>> security and free developer resources for dealing with RPM's "core
>> business" instead, RPM upstream is in the process of deprecating the
>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>> based solution written in Rust.
>
> Why are you using a new library written in Rust? Can you not use one of the
> existing mature C implementations of OpenPGP? gpgme maybe?
Had there been such an option, we would've switched over years ago.
Gpgme is based around calling and communicating with an external gpg
process, which is a setup you do NOT want in the rpm context where
chroots come and go etc. Also, rpm requires access to the "low-level"
digests-in-progress because it calculates multiple things on a single read.
The real problem is that all other OpenPGP implementations died out
because of GnuPG. And then GnuPG made the choice to force an
inter-process model.
At work, I deal with this on Debian systems, which do indeed use GnuPG
for this. It creates a lot of problems, especially for building images
and dealing with chroots, which is why in the context of RPM PGP
upstream, I never pushed to consider it.
The most serious problems with PackageKit memory leaks and hangs are
actually caused by GnuPG, since DNF uses it for some GPG stuff because
there's no API for using RPM's PGP capabilities. There's no fix unless
the RPM and DNF teams agree on an API and build it out so that libdnf
and librepo no longer need to use GnuPG through gpgme anymore.
This is also the underlying reason why Red Hat has resisted
implementing signed repository metadata and enforcing it by default.
Of course this is a bit of a catch-22 as well, as there's no
motivation to find a solution because neither Fedora nor RHEL offer
signed repository metadata despite repeated calls for it over the past
decade.
Now, don't get me wrong: I'm personally extremely unhappy about having
to depend on the Sequoia stack for RPM PGP. I have a strong distaste
for the Rust community ecosystem these days, and I don't love the idea
of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
will be in place soon enough!). But we literally don't have any other
options. GnuPG/GPGME is out of the question for the reasons Panu and I
stated, and NeoPG died several years ago. There are no C/C++ libraries
for OpenPGP verification.
If there was *any other choice*, we would have taken it. Even
splitting out RPM's internal OpenPGP code into its own project for
someone to rework and upgrade would be an option if someone actually
wanted to. The reality is that nobody wants to work on that.
So here we are, in a subpar situation created by bad tools because
nobody cares enough about security anyway.
--
真実はいつも一つ！/ Always, there's only one truth!
3:23 a.m.
On 10/13/22 10:53, Neal Gompa wrote:
On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen
<pmatilai(a)redhat.com> wrote:
>
> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
>>> for dealing with keys and signatures. That parser is rather infamous
>>> for its limitations and flaws, and especially in recent years has
>>> proven a significant burden to RPM development. In order to improve
>>> security and free developer resources for dealing with RPM's "core
>>> business" instead, RPM upstream is in the process of deprecating the
>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>> based solution written in Rust.
>>
>> Why are you using a new library written in Rust? Can you not use one of the
>> existing mature C implementations of OpenPGP? gpgme maybe?
>
> Had there been such an option, we would've switched over years ago.
> Gpgme is based around calling and communicating with an external gpg
> process, which is a setup you do NOT want in the rpm context where
> chroots come and go etc. Also, rpm requires access to the "low-level"
> digests-in-progress because it calculates multiple things on a single read.
>
The real problem is that all other OpenPGP implementations died out
because of GnuPG. And then GnuPG made the choice to force an
inter-process model.
At work, I deal with this on Debian systems, which do indeed use GnuPG
for this. It creates a lot of problems, especially for building images
and dealing with chroots, which is why in the context of RPM PGP
upstream, I never pushed to consider it.
The most serious problems with PackageKit memory leaks and hangs are
actually caused by GnuPG, since DNF uses it for some GPG stuff because
there's no API for using RPM's PGP capabilities. There's no fix unless
the RPM and DNF teams agree on an API and build it out so that libdnf
and librepo no longer need to use GnuPG through gpgme anymore.
This is also the underlying reason why Red Hat has resisted
implementing signed repository metadata and enforcing it by default.
Of course this is a bit of a catch-22 as well, as there's no
motivation to find a solution because neither Fedora nor RHEL offer
signed repository metadata despite repeated calls for it over the past
decade.
Now, don't get me wrong: I'm personally extremely unhappy about having
to depend on the Sequoia stack for RPM PGP. I have a strong distaste
for the Rust community ecosystem these days, and I don't love the idea
of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
will be in place soon enough!). But we literally don't have any other
options. GnuPG/GPGME is out of the question for the reasons Panu and I
stated, and NeoPG died several years ago. There are no C/C++ libraries
for OpenPGP verification.
There's RNP (in C++, used by Thunderbird at least), but alas that
doesn't expose the digest-in-progress either. So at least in it's
current form, it's not an option for rpm. Also, the API appears to have
all manner of quirks and gotchas that aren't welcome in a
security-critical piece.
As for bootstrap, there will always (have to) be a way to build rpm
without depending on Rust. Even if that meant no signature verification
support in such a configuration.
If there was *any other choice*, we would have taken it. Even
splitting out RPM's internal OpenPGP code into its own project for
someone to rework and upgrade would be an option if someone actually
wanted to. The reality is that nobody wants to work on that.
Yep. In case there's any doubt, I too would've much, MUCH, VERY MUCH
preferred a C (or even C++) library but in the ~15 years I've been on
the lookout, no viable and credible candidates have appeared.
Rpm will remain open to alternative backends, should such a thing
emerge, but I'm not holding my breath.
- Panu -
So here we are, in a subpar situation created by bad tools because
nobody cares enough about security anyway.
--
真実はいつも一つ！/ Always, there's only one truth!
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
7:14 a.m.
On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
On 10/13/22 10:53, Neal Gompa wrote:
> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
>>
>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
>>>> for dealing with keys and signatures. That parser is rather infamous
>>>> for its limitations and flaws, and especially in recent years has
>>>> proven a significant burden to RPM development. In order to improve
>>>> security and free developer resources for dealing with RPM's
"core
>>>> business" instead, RPM upstream is in the process of deprecating
the
>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>>> based solution written in Rust.
>>>
>>> Why are you using a new library written in Rust? Can you not use one of the
>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>
>> Had there been such an option, we would've switched over years ago.
>> Gpgme is based around calling and communicating with an external gpg
>> process, which is a setup you do NOT want in the rpm context where
>> chroots come and go etc. Also, rpm requires access to the "low-level"
>> digests-in-progress because it calculates multiple things on a single read.
>>
>
> The real problem is that all other OpenPGP implementations died out
> because of GnuPG. And then GnuPG made the choice to force an
> inter-process model.
>
> At work, I deal with this on Debian systems, which do indeed use GnuPG
> for this. It creates a lot of problems, especially for building images
> and dealing with chroots, which is why in the context of RPM PGP
> upstream, I never pushed to consider it.
>
> The most serious problems with PackageKit memory leaks and hangs are
> actually caused by GnuPG, since DNF uses it for some GPG stuff because
> there's no API for using RPM's PGP capabilities. There's no fix unless
> the RPM and DNF teams agree on an API and build it out so that libdnf
> and librepo no longer need to use GnuPG through gpgme anymore.
>
> This is also the underlying reason why Red Hat has resisted
> implementing signed repository metadata and enforcing it by default.
> Of course this is a bit of a catch-22 as well, as there's no
> motivation to find a solution because neither Fedora nor RHEL offer
> signed repository metadata despite repeated calls for it over the past
> decade.
>
> Now, don't get me wrong: I'm personally extremely unhappy about having
> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
> for the Rust community ecosystem these days, and I don't love the idea
> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
> will be in place soon enough!). But we literally don't have any other
> options. GnuPG/GPGME is out of the question for the reasons Panu and I
> stated, and NeoPG died several years ago. There are no C/C++ libraries
> for OpenPGP verification.
There's RNP (in C++, used by Thunderbird at least), but alas that
doesn't expose the digest-in-progress either. So at least in it's
current form, it's not an option for rpm. Also, the API appears to have
all manner of quirks and gotchas that aren't welcome in a
security-critical piece.
Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend
and at least the verification API (which is what RPM would use) seems
to be getting love lately. Insofar as quirks and gotchas, I'm not a
great judge of that at the moment, but I don't think it could be worse
than what we have with GnuPG.
The missing "digest-in-progress" thing is an issue, I guess. Have we
raised the issue with them about it?
As for bootstrap, there will always (have to) be a way to build rpm
without depending on Rust. Even if that meant no signature verification
support in such a configuration.
Eck. What about the x509 based stuff we were talking about last year?
All the crypto backends RPM supports now support that stuff out of the
box.
Embedded x509 signatures (certs) to replace GPG signatures could work
as an alternative.
--
真実はいつも一つ！/ Always, there's only one truth!
8:30 a.m.
Neal Gompa wrote:
This is also the underlying reason why Red Hat has resisted
implementing signed repository metadata and enforcing it by default.
Of course this is a bit of a catch-22 as well, as there's no
motivation to find a solution because neither Fedora nor RHEL offer
signed repository metadata despite repeated calls for it over the past
decade.
Is signed repository metadata not basically moot now that pretty much all
the world has moved on from unencrypted HTTP to secure HTTPS?
Now, don't get me wrong: I'm personally extremely unhappy
about having
to depend on the Sequoia stack for RPM PGP. I have a strong distaste
for the Rust community ecosystem these days, and I don't love the idea
of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
will be in place soon enough!).
The dependency on LLVM is not even the worst issue in my eyes. LLVM is also
used by other core projects, e.g., mesa, these days.
The worst issue I see with Rust is the way libraries are "packaged", which
just implies installing source code and recompiling that source code for
every single application. (And as a result, the output obviously gets
statically linked into the application, with all the drawbacks of static
linking.) I consider a language with no usable shared library support to be
entirely unpackageable and hence entirely useless.
And then of course there is the issue that it is yet another language with
yet another syntax (and an only partially C-like one, so the learning curve
is unnecessarily high), yet another library ecosystem, etc. C has been the
de facto lingua franca all this time, now we are back into a tower-of-babel
scenario with tons of programming languages, which will necessarily bloat
the core system over time.
So here we are, in a subpar situation created by bad tools because
nobody cares enough about security anyway.
Sounds like a mess indeed.
Kevin Kofler
8:46 a.m.
On Thu, Oct 13, 2022 at 9:31 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> This is also the underlying reason why Red Hat has resisted
> implementing signed repository metadata and enforcing it by default.
> Of course this is a bit of a catch-22 as well, as there's no
> motivation to find a solution because neither Fedora nor RHEL offer
> signed repository metadata despite repeated calls for it over the past
> decade.
Is signed repository metadata not basically moot now that pretty much all
the world has moved on from unencrypted HTTP to secure HTTPS?
No, because when you do things like mirror repositories (especially
for private mirrors), that signature is the only way to verify the
integrity. HTTPS is only transport encryption from a particular
connection.
Also, a ton of Fedora mirrors still don't use HTTPS for various reasons.
> Now, don't get me wrong: I'm personally extremely
unhappy about having
> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
> for the Rust community ecosystem these days, and I don't love the idea
> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
> will be in place soon enough!).
The dependency on LLVM is not even the worst issue in my eyes. LLVM is also
used by other core projects, e.g., mesa, these days.
The worst issue I see with Rust is the way libraries are "packaged", which
just implies installing source code and recompiling that source code for
every single application. (And as a result, the output obviously gets
statically linked into the application, with all the drawbacks of static
linking.) I consider a language with no usable shared library support to be
entirely unpackageable and hence entirely useless.
And then of course there is the issue that it is yet another language with
yet another syntax (and an only partially C-like one, so the learning curve
is unnecessarily high), yet another library ecosystem, etc. C has been the
de facto lingua franca all this time, now we are back into a tower-of-babel
scenario with tons of programming languages, which will necessarily bloat
the core system over time.
> So here we are, in a subpar situation created by bad tools because
> nobody cares enough about security anyway.
Sounds like a mess indeed.
Well, it might still be worthwhile to split out RPM's OpenPGP
implementation into its own project and allow people to contribute to
it. The worst that can happen is that nothing changes.
--
真実はいつも一つ！/ Always, there's only one truth!
8:46 a.m.
On Thu, Oct 13, 2022 at 3:31 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
The dependency on LLVM is not even the worst issue in my eyes. LLVM is also
used by other core projects, e.g., mesa, these days.
The worst issue I see with Rust is the way libraries are "packaged", which
just implies installing source code and recompiling that source code for
every single application. (And as a result, the output obviously gets
statically linked into the application, with all the drawbacks of static
linking.) I consider a language with no usable shared library support to be
entirely unpackageable and hence entirely useless.
Do you have suggestions for improving this situation? I think we're
pretty close to doing the best we can with packaging Rust projects,
given the current limitations of the language (i.e. the support for
building "true shared Rust libraries" is still very limited and
unstable; but that might improve in the future).
Additionally, the way the Sequoia GPG backend for RPM is implemented,
it's actually shipped as a standard shared library with a C ABI and
accompanying C headers (which are binary compatible with the existing
in-house GPG backend code in RPM). No Rust code will be statically
linked into /usr/bin/rpm.
And then of course there is the issue that it is yet another language
with
yet another syntax (and an only partially C-like one, so the learning curve
is unnecessarily high), yet another library ecosystem, etc. C has been the
de facto lingua franca all this time, now we are back into a tower-of-babel
scenario with tons of programming languages, which will necessarily bloat
the core system over time.
I'm sorry to disappoint you, but Rust is here to stay, whether you
like it or not.
For example, it's been voted the "most loved" and "most wanted"
language for a few years in a row in Stack Overflow's surveys:
https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-w...
So, do you have any actionable / constructive criticism for how we do
Rust packaging in Fedora, or did you just want to post that "I hate
Rust because it's not C"?
Fabio
8:57 a.m.
Neal Gompa wrote:
No, because when you do things like mirror repositories (especially
for private mirrors), that signature is the only way to verify the
integrity. HTTPS is only transport encryption from a particular
connection.
HTTPS protects against a MITM on the connection introducing invalid
repository contents, which I would assume to be the biggest threat here. But
sure, it by design does not guarantee that the data on the remote end is
valid to begin with.
Also, a ton of Fedora mirrors still don't use HTTPS for various
reasons.
I would say that those mirrors ought to be kicked out of the mirror list
immediately.
With Let's Encrypt having been available for years, there is really no
excuse for not offering HTTPS. Assuming you own a domain name (which I
assume to already be the case for all mirrors), setting up HTTPS with Let's
Encrypt does not cost you a dime. Even if you are a commercial entity.
Well, it might still be worthwhile to split out RPM's OpenPGP
implementation into its own project and allow people to contribute to
it. The worst that can happen is that nothing changes.
If that implementation is really as awfully broken as Panu is saying, I do
not think that that would be of much use, unfortunately.
Kevin Kofler
9:31 a.m.
On Thu, Oct 13, 2022 at 9:58 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> No, because when you do things like mirror repositories (especially
> for private mirrors), that signature is the only way to verify the
> integrity. HTTPS is only transport encryption from a particular
> connection.
HTTPS protects against a MITM on the connection introducing invalid
repository contents, which I would assume to be the biggest threat here. But
sure, it by design does not guarantee that the data on the remote end is
valid to begin with.
> Also, a ton of Fedora mirrors still don't use HTTPS for various reasons.
I would say that those mirrors ought to be kicked out of the mirror list
immediately.
With Let's Encrypt having been available for years, there is really no
excuse for not offering HTTPS. Assuming you own a domain name (which I
assume to already be the case for all mirrors), setting up HTTPS with Let's
Encrypt does not cost you a dime. Even if you are a commercial entity.
I'm not going to get into this too much, but suffice to say, it's not
universally accessible as a CA. And using Let's Encrypt for private
mirrors is sufficiently painful that I wouldn't recommend it.
> Well, it might still be worthwhile to split out RPM's
OpenPGP
> implementation into its own project and allow people to contribute to
> it. The worst that can happen is that nothing changes.
If that implementation is really as awfully broken as Panu is saying, I do
not think that that would be of much use, unfortunately.
There have been attempts to fix things, but Panu doesn't feel
qualified to review the changes. That doesn't mean someone else who
would be willing to do so couldn't. But because of... reasons, as long
as it's in the RPM codebase, it's unlikely someone else will be
trusted enough to do those reviews.
--
真実はいつも一つ！/ Always, there's only one truth!
10:12 a.m.
Neal Gompa wrote:
I'm not going to get into this too much, but suffice to say,
it's not
universally accessible as a CA.
I would very much be interested in those details though. I do not see
anybody being excluded from Let's Encrypt, not even countries under US
embargo (e.g., over 300000 sites in Iran are apparently using it
successfully).
And using Let's Encrypt for private mirrors is sufficiently
painful that I
wouldn't recommend it.
Set up a subdomain like vpn.example.com, point it to the public IP, then
configure the VPN's internal DNS to resolve vpn.example.com to the VPN-
internal address instead, the /etc/hosts on the VPN server itself to resolve
it to 127.0.0.1, and the mirror server on port 443 (whereas port 80 is
reserved for certbot's builtin temporary (and world-readable) webserver with
the http-01 challenge) to accept connections only from the VPN and from
localhost and to use the Let's Encrypt certificate. Been there, done that
(not for a repository mirror though, my employer is small enough for that
not to be worthwhile). I assume that this approach should also work for a
physical LAN in lieu of the VPN.
There have been attempts to fix things, but Panu doesn't feel
qualified to review the changes. That doesn't mean someone else who
would be willing to do so couldn't. But because of... reasons, as long
as it's in the RPM codebase, it's unlikely someone else will be
trusted enough to do those reviews.
I see. So splitting might be worthwhile then. Assuming someone will care
enough to actually maintain the code.
Kevin Kofler
10:18 a.m.
Fabio Valentini wrote:
Do you have suggestions for improving this situation? I think
we're
pretty close to doing the best we can with packaging Rust projects,
given the current limitations of the language (i.e. the support for
building "true shared Rust libraries" is still very limited and
unstable; but that might improve in the future).
Building true shared libraries is pretty much a prerequisite for any sane
form of packaging.
Additionally, the way the Sequoia GPG backend for RPM is
implemented,
it's actually shipped as a standard shared library with a C ABI and
accompanying C headers (which are binary compatible with the existing
in-house GPG backend code in RPM). No Rust code will be statically
linked into /usr/bin/rpm.
That at least makes sense. (Though I assume that that C-style .so still
internally depends on a whole bunch of Rust crates that are statically
compiled and linked into the .so, does it not?)
I'm sorry to disappoint you, but Rust is here to stay, whether
you
like it or not.
For example, it's been voted the "most loved" and "most wanted"
language for a few years in a row in Stack Overflow's surveys:
https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-w...
And according to the same statistics, the majority of developers runs
Windows, yet hopefully you do not propose that Fedora ship Windows…
We need to ship what we can reasonably ship, not what the (relative)
majority of developers in the world (most of whom do not even run Fedora)
happens to prefer.
Kevin Kofler
10:22 a.m.
On Thu, Oct 13, 2022 at 3:13 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Neal Gompa wrote:
> I'm not going to get into this too much, but suffice to say, it's not
> universally accessible as a CA.
I would very much be interested in those details though.
As I recall, the current LE root certificate is not (and in some
cases may not be able to be) installed in all clients. When
the cross signing cert (from a well known/trusted CA)
expired a year ago a number of clients suddenly found
themselves getting errors. It is easy to say "install new
root certs", but that is not actually easy to do in all cases.
10:36 a.m.
On 10/13/22 08:14, Neal Gompa wrote:
On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen
<pmatilai(a)redhat.com> wrote:
>
> On 10/13/22 10:53, Neal Gompa wrote:
>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com>
wrote:
>>>
>>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
>>>>> for dealing with keys and signatures. That parser is rather infamous
>>>>> for its limitations and flaws, and especially in recent years has
>>>>> proven a significant burden to RPM development. In order to improve
>>>>> security and free developer resources for dealing with RPM's
"core
>>>>> business" instead, RPM upstream is in the process of deprecating
the
>>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>>>> based solution written in Rust.
>>>>
>>>> Why are you using a new library written in Rust? Can you not use one of
the
>>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>>
>>> Had there been such an option, we would've switched over years ago.
>>> Gpgme is based around calling and communicating with an external gpg
>>> process, which is a setup you do NOT want in the rpm context where
>>> chroots come and go etc. Also, rpm requires access to the
"low-level"
>>> digests-in-progress because it calculates multiple things on a single read.
>>>
>>
>> The real problem is that all other OpenPGP implementations died out
>> because of GnuPG. And then GnuPG made the choice to force an
>> inter-process model.
>>
>> At work, I deal with this on Debian systems, which do indeed use GnuPG
>> for this. It creates a lot of problems, especially for building images
>> and dealing with chroots, which is why in the context of RPM PGP
>> upstream, I never pushed to consider it.
>>
>> The most serious problems with PackageKit memory leaks and hangs are
>> actually caused by GnuPG, since DNF uses it for some GPG stuff because
>> there's no API for using RPM's PGP capabilities. There's no fix
unless
>> the RPM and DNF teams agree on an API and build it out so that libdnf
>> and librepo no longer need to use GnuPG through gpgme anymore.
>>
>> This is also the underlying reason why Red Hat has resisted
>> implementing signed repository metadata and enforcing it by default.
>> Of course this is a bit of a catch-22 as well, as there's no
>> motivation to find a solution because neither Fedora nor RHEL offer
>> signed repository metadata despite repeated calls for it over the past
>> decade.
>>
>> Now, don't get me wrong: I'm personally extremely unhappy about having
>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
>> for the Rust community ecosystem these days, and I don't love the idea
>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
>> will be in place soon enough!). But we literally don't have any other
>> options. GnuPG/GPGME is out of the question for the reasons Panu and I
>> stated, and NeoPG died several years ago. There are no C/C++ libraries
>> for OpenPGP verification.
>
> There's RNP (in C++, used by Thunderbird at least), but alas that
> doesn't expose the digest-in-progress either. So at least in it's
> current form, it's not an option for rpm. Also, the API appears to have
> all manner of quirks and gotchas that aren't welcome in a
> security-critical piece.
>
Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend
and at least the verification API (which is what RPM would use) seems
to be getting love lately. Insofar as quirks and gotchas, I'm not a
great judge of that at the moment, but I don't think it could be worse
than what we have with GnuPG.
The missing "digest-in-progress" thing is an issue, I guess. Have we
raised the issue with them about it?
> As for bootstrap, there will always (have to) be a way to build rpm
> without depending on Rust. Even if that meant no signature verification
> support in such a configuration.
>
Eck. What about the x509 based stuff we were talking about last year?
All the crypto backends RPM supports now support that stuff out of the
box.
Embedded x509 signatures (certs) to replace GPG signatures could work
as an alternative.
OpenPGP is not great, but X.509 is an absolute disaster in comparison.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
10:43 a.m.
On Thu, Oct 13, 2022 at 11:38 AM Demi Marie Obenour
<demiobenour(a)gmail.com> wrote:
On 10/13/22 08:14, Neal Gompa wrote:
> On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
>>
>> On 10/13/22 10:53, Neal Gompa wrote:
>>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com>
wrote:
>>>>
>>>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
>>>>>> for dealing with keys and signatures. That parser is rather
infamous
>>>>>> for its limitations and flaws, and especially in recent years
has
>>>>>> proven a significant burden to RPM development. In order to
improve
>>>>>> security and free developer resources for dealing with RPM's
"core
>>>>>> business" instead, RPM upstream is in the process of
deprecating the
>>>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia
PGP]
>>>>>> based solution written in Rust.
>>>>>
>>>>> Why are you using a new library written in Rust? Can you not use one
of the
>>>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>>>
>>>> Had there been such an option, we would've switched over years ago.
>>>> Gpgme is based around calling and communicating with an external gpg
>>>> process, which is a setup you do NOT want in the rpm context where
>>>> chroots come and go etc. Also, rpm requires access to the
"low-level"
>>>> digests-in-progress because it calculates multiple things on a single
read.
>>>>
>>>
>>> The real problem is that all other OpenPGP implementations died out
>>> because of GnuPG. And then GnuPG made the choice to force an
>>> inter-process model.
>>>
>>> At work, I deal with this on Debian systems, which do indeed use GnuPG
>>> for this. It creates a lot of problems, especially for building images
>>> and dealing with chroots, which is why in the context of RPM PGP
>>> upstream, I never pushed to consider it.
>>>
>>> The most serious problems with PackageKit memory leaks and hangs are
>>> actually caused by GnuPG, since DNF uses it for some GPG stuff because
>>> there's no API for using RPM's PGP capabilities. There's no fix
unless
>>> the RPM and DNF teams agree on an API and build it out so that libdnf
>>> and librepo no longer need to use GnuPG through gpgme anymore.
>>>
>>> This is also the underlying reason why Red Hat has resisted
>>> implementing signed repository metadata and enforcing it by default.
>>> Of course this is a bit of a catch-22 as well, as there's no
>>> motivation to find a solution because neither Fedora nor RHEL offer
>>> signed repository metadata despite repeated calls for it over the past
>>> decade.
>>>
>>> Now, don't get me wrong: I'm personally extremely unhappy about
having
>>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
>>> for the Rust community ecosystem these days, and I don't love the idea
>>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
>>> will be in place soon enough!). But we literally don't have any other
>>> options. GnuPG/GPGME is out of the question for the reasons Panu and I
>>> stated, and NeoPG died several years ago. There are no C/C++ libraries
>>> for OpenPGP verification.
>>
>> There's RNP (in C++, used by Thunderbird at least), but alas that
>> doesn't expose the digest-in-progress either. So at least in it's
>> current form, it's not an option for rpm. Also, the API appears to have
>> all manner of quirks and gotchas that aren't welcome in a
>> security-critical piece.
>>
>
> Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend
> and at least the verification API (which is what RPM would use) seems
> to be getting love lately. Insofar as quirks and gotchas, I'm not a
> great judge of that at the moment, but I don't think it could be worse
> than what we have with GnuPG.
>
> The missing "digest-in-progress" thing is an issue, I guess. Have we
> raised the issue with them about it?
>
>> As for bootstrap, there will always (have to) be a way to build rpm
>> without depending on Rust. Even if that meant no signature verification
>> support in such a configuration.
>>
>
> Eck. What about the x509 based stuff we were talking about last year?
> All the crypto backends RPM supports now support that stuff out of the
> box.
>
> Embedded x509 signatures (certs) to replace GPG signatures could work
> as an alternative.
OpenPGP is not great, but X.509 is an absolute disaster in comparison.
But there are more implementations of the latter. I'll take that.
--
真実はいつも一つ！/ Always, there's only one truth!
11:28 a.m.
On 10/13/22 11:18, Kevin Kofler via devel wrote:
Fabio Valentini wrote:
> Do you have suggestions for improving this situation? I think we're
> pretty close to doing the best we can with packaging Rust projects,
> given the current limitations of the language (i.e. the support for
> building "true shared Rust libraries" is still very limited and
> unstable; but that might improve in the future).
Building true shared libraries is pretty much a prerequisite for any sane
form of packaging.
I strongly recommend that you bring this up on the rust-lang Zulip
(https://rust-lang.zulipchat.com) and see if you can find people who can
improve this.
> Additionally, the way the Sequoia GPG backend for RPM is
implemented,
> it's actually shipped as a standard shared library with a C ABI and
> accompanying C headers (which are binary compatible with the existing
> in-house GPG backend code in RPM). No Rust code will be statically
> linked into /usr/bin/rpm.
That at least makes sense. (Though I assume that that C-style .so still
internally depends on a whole bunch of Rust crates that are statically
compiled and linked into the .so, does it not?)
It does.
> I'm sorry to disappoint you, but Rust is here to stay,
whether you
> like it or not.
> For example, it's been voted the "most loved" and "most
wanted"
> language for a few years in a row in Stack Overflow's surveys:
> https://survey.stackoverflow.co/2022/#technology-most-loved-dreaded-and-w...
And according to the same statistics, the majority of developers runs
Windows, yet hopefully you do not propose that Fedora ship Windows…
We need to ship what we can reasonably ship, not what the (relative)
majority of developers in the world (most of whom do not even run Fedora)
happens to prefer.
There is now Rust code in the Linux kernel. It currently is not
needed by any shipping driver, but that is not a matter of *if*,
but of *when*. The Asahi GPU driver is being written in Rust,
for example, so anyone building Linux for Apple Silicon will need
CONFIG_RUST=y in the not-too-distant future. systemd (yes, systemd)
is considering using Rust, though it has not yet started including it,
and there is already Rust code in Mesa IIRC.
C and C++ are being moved away from because they have unfixable
security problems. Memory unsafety, which is predominantly (though
not exclusively) found in C and C++ codebases, accounts for well
over half of all security vulnerabilities. Google tried to retrofit
memory safety into C++ and failed, and the existing methods for
making C memory-safe either have massive overheads (4x slowdown
for a single-threaded application, worse for multi-threaded, IIRC)
or require a completely unreasonable amount of developer effort for
a volunteer project (formal methods).
--
Sincerely,
Demi Marie Obenour (she/her/hers)
11:35 a.m.
On 10/13/22 04:23, Panu Matilainen wrote:
On 10/13/22 10:53, Neal Gompa wrote:
> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
>>
>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
>>>> for dealing with keys and signatures. That parser is rather infamous
>>>> for its limitations and flaws, and especially in recent years has
>>>> proven a significant burden to RPM development. In order to improve
>>>> security and free developer resources for dealing with RPM's
"core
>>>> business" instead, RPM upstream is in the process of deprecating
the
>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>>> based solution written in Rust.
>>>
>>> Why are you using a new library written in Rust? Can you not use one of the
>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>
>> Had there been such an option, we would've switched over years ago.
>> Gpgme is based around calling and communicating with an external gpg
>> process, which is a setup you do NOT want in the rpm context where
>> chroots come and go etc. Also, rpm requires access to the "low-level"
>> digests-in-progress because it calculates multiple things on a single read.
>>
>
> The real problem is that all other OpenPGP implementations died out
> because of GnuPG. And then GnuPG made the choice to force an
> inter-process model.
>
> At work, I deal with this on Debian systems, which do indeed use GnuPG
> for this. It creates a lot of problems, especially for building images
> and dealing with chroots, which is why in the context of RPM PGP
> upstream, I never pushed to consider it.
>
> The most serious problems with PackageKit memory leaks and hangs are
> actually caused by GnuPG, since DNF uses it for some GPG stuff because
> there's no API for using RPM's PGP capabilities. There's no fix unless
> the RPM and DNF teams agree on an API and build it out so that libdnf
> and librepo no longer need to use GnuPG through gpgme anymore.
>
> This is also the underlying reason why Red Hat has resisted
> implementing signed repository metadata and enforcing it by default.
> Of course this is a bit of a catch-22 as well, as there's no
> motivation to find a solution because neither Fedora nor RHEL offer
> signed repository metadata despite repeated calls for it over the past
> decade.
>
> Now, don't get me wrong: I'm personally extremely unhappy about having
> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
> for the Rust community ecosystem these days, and I don't love the idea
> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
> will be in place soon enough!). But we literally don't have any other
> options. GnuPG/GPGME is out of the question for the reasons Panu and I
> stated, and NeoPG died several years ago. There are no C/C++ libraries
> for OpenPGP verification.
There's RNP (in C++, used by Thunderbird at least), but alas that
doesn't expose the digest-in-progress either. So at least in it's
current form, it's not an option for rpm. Also, the API appears to have
all manner of quirks and gotchas that aren't welcome in a
security-critical piece.
As for bootstrap, there will always (have to) be a way to build rpm
without depending on Rust. Even if that meant no signature verification
support in such a configuration.
RPM could keep its own internal parser for bootstrap only. Bootstrap
does not need support for revocation, etc.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
11:56 a.m.
On Thu Oct 13, 2022 at 17:12 +0200, Kevin Kofler via devel wrote:
> And using Let's Encrypt for private mirrors is sufficiently
painful that I
> wouldn't recommend it.
Set up a subdomain like vpn.example.com, point it to the public IP, then
configure the VPN's internal DNS to resolve vpn.example.com to the VPN-
internal address instead, the /etc/hosts on the VPN server itself to resolve
it to 127.0.0.1, and the mirror server on port 443 (whereas port 80 is
reserved for certbot's builtin temporary (and world-readable) webserver with
the http-01 challenge) to accept connections only from the VPN and from
localhost and to use the Let's Encrypt certificate. Been there, done that
(not for a repository mirror though, my employer is small enough for that
not to be worthwhile). I assume that this approach should also work for a
physical LAN in lieu of the VPN.
Let's Encrypt also supports the dns-01 challenge[1] that doesn't require
any publicly available IPs. Using dns verification is required to obtain a
Let's Encrypt wildcard certificate.
[1]: https://letsencrypt.org/docs/challenge-types/#dns-01-challenge
--
Maxwell G (@gotmax23)
Pronouns: He/Him/His
11:56 a.m.
On 10/13/22 11:28, Demi Marie Obenour wrote:
systemd (yes, systemd)
is considering using Rust, though it has not yet started including it,
and there is already Rust code in Mesa IIRC.
Don't forget the Python 'cryptography' package... also a Rust user. It's
here to stay, at least for the foreseeable future.
--
Kevin P. Fleming
He/Him/His
Principal Program Manager, RHEL
Red Hat US/Eastern Time Zone
12:15 p.m.
On Thu, Oct 13, 2022 at 4:57 PM Maxwell G via devel
<devel(a)lists.fedoraproject.org> wrote:
Let's Encrypt also supports the dns-01 challenge[1] that
doesn't require
any publicly available IPs. Using dns verification is required to obtain a
Let's Encrypt wildcard certificate.
While I tend to prefer using the dns-01 challenge approach
when possible, not all DNS providers have made it easy to
accomplish (the certbot folk have implementations for a
number of the major DNS providers, and one can sometimes
find other 3rd party code for others, but it can still be hard
to setup and use, which means just enough additional
impedance that sometimes people will choose not to use it;
I can't blame them, as sometimes free has a higher cost
than having someone else order the cert from one of
the non-free CAs).
1:17 p.m.
> Let's Encrypt also supports the dns-01 challenge[1] that
doesn't require
> any publicly available IPs. Using dns verification is required to obtain a
> Let's Encrypt wildcard certificate.
While I tend to prefer using the dns-01 challenge approach
when possible, not all DNS providers have made it easy to
accomplish (the certbot folk have implementations for a
number of the major DNS providers, and one can sometimes
find other 3rd party code for others, but it can still be hard
to setup and use, which means just enough additional
impedance that sometimes people will choose not to use it;
I can't blame them, as sometimes free has a higher cost
than having someone else order the cert from one of
the non-free CAs).
fwiw, IME, one of the lowest-friction dns-challenge tools I've recommended, and see
actually getting used by clients, is acme.sh,
https://github.com/acmesh-official/acme.sh#user-content-8-automatic-dns-a...
which supports 'most' of the big dns apis,
https://github.com/acmesh-official/acme.sh/wiki/dnsapi
and, when not an option, is fairly trivial to use manually
https://github.com/acmesh-official/acme.sh#user-content-9-use-dns-manual-...
https://github.com/acmesh-official/acme.sh/wiki/dns-manual-mode
all of this with no cumbersome python, go, webserver, etc deps. just bash shell.
1:09 a.m.
On 10/13/22 19:35, Demi Marie Obenour wrote:
On 10/13/22 04:23, Panu Matilainen wrote:
> On 10/13/22 10:53, Neal Gompa wrote:
>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com>
wrote:
>>>
>>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
>>>>> for dealing with keys and signatures. That parser is rather infamous
>>>>> for its limitations and flaws, and especially in recent years has
>>>>> proven a significant burden to RPM development. In order to improve
>>>>> security and free developer resources for dealing with RPM's
"core
>>>>> business" instead, RPM upstream is in the process of deprecating
the
>>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>>>> based solution written in Rust.
>>>>
>>>> Why are you using a new library written in Rust? Can you not use one of
the
>>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>>
>>> Had there been such an option, we would've switched over years ago.
>>> Gpgme is based around calling and communicating with an external gpg
>>> process, which is a setup you do NOT want in the rpm context where
>>> chroots come and go etc. Also, rpm requires access to the
"low-level"
>>> digests-in-progress because it calculates multiple things on a single read.
>>>
>>
>> The real problem is that all other OpenPGP implementations died out
>> because of GnuPG. And then GnuPG made the choice to force an
>> inter-process model.
>>
>> At work, I deal with this on Debian systems, which do indeed use GnuPG
>> for this. It creates a lot of problems, especially for building images
>> and dealing with chroots, which is why in the context of RPM PGP
>> upstream, I never pushed to consider it.
>>
>> The most serious problems with PackageKit memory leaks and hangs are
>> actually caused by GnuPG, since DNF uses it for some GPG stuff because
>> there's no API for using RPM's PGP capabilities. There's no fix
unless
>> the RPM and DNF teams agree on an API and build it out so that libdnf
>> and librepo no longer need to use GnuPG through gpgme anymore.
>>
>> This is also the underlying reason why Red Hat has resisted
>> implementing signed repository metadata and enforcing it by default.
>> Of course this is a bit of a catch-22 as well, as there's no
>> motivation to find a solution because neither Fedora nor RHEL offer
>> signed repository metadata despite repeated calls for it over the past
>> decade.
>>
>> Now, don't get me wrong: I'm personally extremely unhappy about having
>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
>> for the Rust community ecosystem these days, and I don't love the idea
>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
>> will be in place soon enough!). But we literally don't have any other
>> options. GnuPG/GPGME is out of the question for the reasons Panu and I
>> stated, and NeoPG died several years ago. There are no C/C++ libraries
>> for OpenPGP verification.
>
> There's RNP (in C++, used by Thunderbird at least), but alas that
> doesn't expose the digest-in-progress either. So at least in it's
> current form, it's not an option for rpm. Also, the API appears to have
> all manner of quirks and gotchas that aren't welcome in a
> security-critical piece.
>
> As for bootstrap, there will always (have to) be a way to build rpm
> without depending on Rust. Even if that meant no signature verification
> support in such a configuration.
RPM could keep its own internal parser for bootstrap only. Bootstrap
does not need support for revocation, etc.
I very intentionally left the possibilities open here. Rust-free
bootstrap is a hard requirement for rpm, how that is achieved is a whole
other topic.
- Panu -
1:30 a.m.
On 10/13/22 17:31, Neal Gompa wrote:
On Thu, Oct 13, 2022 at 9:58 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> Neal Gompa wrote:
>> No, because when you do things like mirror repositories (especially
>> for private mirrors), that signature is the only way to verify the
>> integrity. HTTPS is only transport encryption from a particular
>> connection.
>
> HTTPS protects against a MITM on the connection introducing invalid
> repository contents, which I would assume to be the biggest threat here. But
> sure, it by design does not guarantee that the data on the remote end is
> valid to begin with.
>
>> Also, a ton of Fedora mirrors still don't use HTTPS for various reasons.
>
> I would say that those mirrors ought to be kicked out of the mirror list
> immediately.
>
> With Let's Encrypt having been available for years, there is really no
> excuse for not offering HTTPS. Assuming you own a domain name (which I
> assume to already be the case for all mirrors), setting up HTTPS with Let's
> Encrypt does not cost you a dime. Even if you are a commercial entity.
>
I'm not going to get into this too much, but suffice to say, it's not
universally accessible as a CA. And using Let's Encrypt for private
mirrors is sufficiently painful that I wouldn't recommend it.
>> Well, it might still be worthwhile to split out RPM's OpenPGP
>> implementation into its own project and allow people to contribute to
>> it. The worst that can happen is that nothing changes.
>
> If that implementation is really as awfully broken as Panu is saying, I do
> not think that that would be of much use, unfortunately.
>
There have been attempts to fix things, but Panu doesn't feel
qualified to review the changes. That doesn't mean someone else who
would be willing to do so couldn't. But because of... reasons, as long
as it's in the RPM codebase, it's unlikely someone else will be
trusted enough to do those reviews.
This misses the point by a mile.
Parsing OpenPGP is not rpm's "core business", by any stretch of
imagination. As such, I refuse to let it take up any significant portion
of rpm development resources.
The parser has been largely dormant in the codebase for like 20 years,
except for a couple of cleanup rounds when people have pointed out
random flaws and vulnerabilities. Last year when people started to
really look into it, it's become obvious that the effort to fix & review
it all is just nowhere near justifiable.
Consider this the RPM burning platform memo if you like ;)
- Panu -
2:14 a.m.
On 10/13/22 15:14, Neal Gompa wrote:
On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen
<pmatilai(a)redhat.com> wrote:
>
> On 10/13/22 10:53, Neal Gompa wrote:
>> On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen <pmatilai(a)redhat.com>
wrote:
>>>
>>> On 10/13/22 07:18, Kevin Kofler via devel wrote:
>>>>> For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
>>>>> for dealing with keys and signatures. That parser is rather infamous
>>>>> for its limitations and flaws, and especially in recent years has
>>>>> proven a significant burden to RPM development. In order to improve
>>>>> security and free developer resources for dealing with RPM's
"core
>>>>> business" instead, RPM upstream is in the process of deprecating
the
>>>>> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
>>>>> based solution written in Rust.
>>>>
>>>> Why are you using a new library written in Rust? Can you not use one of
the
>>>> existing mature C implementations of OpenPGP? gpgme maybe?
>>>
>>> Had there been such an option, we would've switched over years ago.
>>> Gpgme is based around calling and communicating with an external gpg
>>> process, which is a setup you do NOT want in the rpm context where
>>> chroots come and go etc. Also, rpm requires access to the
"low-level"
>>> digests-in-progress because it calculates multiple things on a single read.
>>>
>>
>> The real problem is that all other OpenPGP implementations died out
>> because of GnuPG. And then GnuPG made the choice to force an
>> inter-process model.
>>
>> At work, I deal with this on Debian systems, which do indeed use GnuPG
>> for this. It creates a lot of problems, especially for building images
>> and dealing with chroots, which is why in the context of RPM PGP
>> upstream, I never pushed to consider it.
>>
>> The most serious problems with PackageKit memory leaks and hangs are
>> actually caused by GnuPG, since DNF uses it for some GPG stuff because
>> there's no API for using RPM's PGP capabilities. There's no fix
unless
>> the RPM and DNF teams agree on an API and build it out so that libdnf
>> and librepo no longer need to use GnuPG through gpgme anymore.
>>
>> This is also the underlying reason why Red Hat has resisted
>> implementing signed repository metadata and enforcing it by default.
>> Of course this is a bit of a catch-22 as well, as there's no
>> motivation to find a solution because neither Fedora nor RHEL offer
>> signed repository metadata despite repeated calls for it over the past
>> decade.
>>
>> Now, don't get me wrong: I'm personally extremely unhappy about having
>> to depend on the Sequoia stack for RPM PGP. I have a strong distaste
>> for the Rust community ecosystem these days, and I don't love the idea
>> of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
>> will be in place soon enough!). But we literally don't have any other
>> options. GnuPG/GPGME is out of the question for the reasons Panu and I
>> stated, and NeoPG died several years ago. There are no C/C++ libraries
>> for OpenPGP verification.
>
> There's RNP (in C++, used by Thunderbird at least), but alas that
> doesn't expose the digest-in-progress either. So at least in it's
> current form, it's not an option for rpm. Also, the API appears to have
> all manner of quirks and gotchas that aren't welcome in a
> security-critical piece.
>
Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend
and at least the verification API (which is what RPM would use) seems
to be getting love lately. Insofar as quirks and gotchas, I'm not a
great judge of that at the moment, but I don't think it could be worse
than what we have with GnuPG.
The missing "digest-in-progress" thing is an issue, I guess. Have we
raised the issue with them about it?
No, because beneath the surface it didn't seem all that appetizing
afterall. See
https://sequoia-pgp.org/blog/2021/05/06/202105-thunderbird-rnp-and-the-im...
for an example (even bearing in mind that a blog post from a competitor
may be necessarily a little opionated), but its reputation/security
record doesn't seem that great.
It's something to keep an eye on of course, it would be good to have
alternatives available.
> As for bootstrap, there will always (have to) be a way to build rpm
> without depending on Rust. Even if that meant no signature verification
> support in such a configuration.
>
Eck. What about the x509 based stuff we were talking about last year?
All the crypto backends RPM supports now support that stuff out of the
box.
Embedded x509 signatures (certs) to replace GPG signatures could work
as an alternative.
x509 seems to be loathed even more than PGP, so it doesn't seem that
appetizing either.
If someone with known crypto-clue would send patches they would be
looked at, *I* have no prejudice about x509 because I also have no clue
about it. Ditto for Signify, which often gets brought up in these
discussions.
And yet, that all is largely irrelevant for the subject at hand: no
matter what, rpm will need OpenPGP support for years to come because
existing content will need to remain usable for a long, long time.
- Panu -
11:28 a.m.
On Fri, 2022-10-14 at 10:14 +0300, Panu Matilainen wrote:
On 10/13/22 15:14, Neal Gompa wrote:
> On Thu, Oct 13, 2022 at 4:24 AM Panu Matilainen <pmatilai(a)redhat.com> wrote:
> >
> > On 10/13/22 10:53, Neal Gompa wrote:
> > > On Thu, Oct 13, 2022 at 3:29 AM Panu Matilainen
<pmatilai(a)redhat.com> wrote:
> > > >
> > > > On 10/13/22 07:18, Kevin Kofler via devel wrote:
> > > > > > For the last 20 years or so, RPM has used a home-grown
OpenPGP parser
> > > > > > for dealing with keys and signatures. That parser is rather
infamous
> > > > > > for its limitations and flaws, and especially in recent
years has
> > > > > > proven a significant burden to RPM development. In order to
improve
> > > > > > security and free developer resources for dealing with
RPM's "core
> > > > > > business" instead, RPM upstream is in the process of
deprecating the
> > > > > > internal parser in favor of [https://sequoia-pgp.org/
Sequoia PGP]
> > > > > > based solution written in Rust.
> > > > >
> > > > > Why are you using a new library written in Rust? Can you not use
one of the
> > > > > existing mature C implementations of OpenPGP? gpgme maybe?
> > > >
> > > > Had there been such an option, we would've switched over years
ago.
> > > > Gpgme is based around calling and communicating with an external gpg
> > > > process, which is a setup you do NOT want in the rpm context where
> > > > chroots come and go etc. Also, rpm requires access to the
"low-level"
> > > > digests-in-progress because it calculates multiple things on a single
read.
> > > >
> > >
> > > The real problem is that all other OpenPGP implementations died out
> > > because of GnuPG. And then GnuPG made the choice to force an
> > > inter-process model.
> > >
> > > At work, I deal with this on Debian systems, which do indeed use GnuPG
> > > for this. It creates a lot of problems, especially for building images
> > > and dealing with chroots, which is why in the context of RPM PGP
> > > upstream, I never pushed to consider it.
> > >
> > > The most serious problems with PackageKit memory leaks and hangs are
> > > actually caused by GnuPG, since DNF uses it for some GPG stuff because
> > > there's no API for using RPM's PGP capabilities. There's no
fix unless
> > > the RPM and DNF teams agree on an API and build it out so that libdnf
> > > and librepo no longer need to use GnuPG through gpgme anymore.
> > >
> > > This is also the underlying reason why Red Hat has resisted
> > > implementing signed repository metadata and enforcing it by default.
> > > Of course this is a bit of a catch-22 as well, as there's no
> > > motivation to find a solution because neither Fedora nor RHEL offer
> > > signed repository metadata despite repeated calls for it over the past
> > > decade.
> > >
> > > Now, don't get me wrong: I'm personally extremely unhappy about
having
> > > to depend on the Sequoia stack for RPM PGP. I have a strong distaste
> > > for the Rust community ecosystem these days, and I don't love the
idea
> > > of having to have LLVM in the core bootstrap chain (hopefully gcc-rs
> > > will be in place soon enough!). But we literally don't have any other
> > > options. GnuPG/GPGME is out of the question for the reasons Panu and I
> > > stated, and NeoPG died several years ago. There are no C/C++ libraries
> > > for OpenPGP verification.
> >
> > There's RNP (in C++, used by Thunderbird at least), but alas that
> > doesn't expose the digest-in-progress either. So at least in it's
> > current form, it's not an option for rpm. Also, the API appears to have
> > all manner of quirks and gotchas that aren't welcome in a
> > security-critical piece.
> >
>
> Huh, I'd forgotten about RNP. It seems it now has an OpenSSL backend
> and at least the verification API (which is what RPM would use) seems
> to be getting love lately. Insofar as quirks and gotchas, I'm not a
> great judge of that at the moment, but I don't think it could be worse
> than what we have with GnuPG.
>
> The missing "digest-in-progress" thing is an issue, I guess. Have we
> raised the issue with them about it?
No, because beneath the surface it didn't seem all that appetizing
afterall. See
https://sequoia-pgp.org/blog/2021/05/06/202105-thunderbird-rnp-and-the-im...
for an example (even bearing in mind that a blog post from a competitor
may be necessarily a little opionated), but its reputation/security
record doesn't seem that great.
It's something to keep an eye on of course, it would be good to have
alternatives available.
>
> > As for bootstrap, there will always (have to) be a way to build rpm
> > without depending on Rust. Even if that meant no signature verification
> > support in such a configuration.
> >
>
> Eck. What about the x509 based stuff we were talking about last year?
> All the crypto backends RPM supports now support that stuff out of the
> box.
>
> Embedded x509 signatures (certs) to replace GPG signatures could work
> as an alternative.
x509 seems to be loathed even more than PGP, so it doesn't seem that
appetizing either.
If someone with known crypto-clue would send patches they would be
looked at, *I* have no prejudice about x509 because I also have no clue
about it. Ditto for Signify, which often gets brought up in these
discussions.
And yet, that all is largely irrelevant for the subject at hand: no
matter what, rpm will need OpenPGP support for years to come because
existing content will need to remain usable for a long, long time.
Just to throw a little fire in here :-D
We are starting to look at what it means to sign content with PQC
algorithms.
One of the factors (not the only one) is the recent CNSA 2.0
recommendations:
https://media.defense.gov/2022/Sep/07/2003071834/-1/-1/0/CSA_CNSA_2.0_ALG...
One of the possible impediments, besides finding a decent
implementation, zeroing in on the algorithm(s) to select and support,
and a gazillion other small but critical details, is the format.
At this time, as far as I know, there is no OpenPGP work of any kind on
supporting PQC algorithms. Furthermore the way we use signatures in RPM
really has no resemblance to the scenarios OpenPGP was built for.
So we should consider whether moving to PQC will also mean moving off
OpenPGP as our signature format and into something simpler.
Panu,
I will try to engage the RPM team once we have a settle a little more
on the cryptography part.
Simo.
--
Simo Sorce
RHEL Crypto Team
Red Hat, Inc
3:36 p.m.
On Fri, 14 Oct 2022 18:28:01 +0200,
Simo Sorce wrote:
At this time, as far as I know, there is no OpenPGP work of any kind
on
supporting PQC algorithms.
The German BSI contracted MTG AG to design and implement PQC for
OpenPGP. They presented their work at IETF 113, and at the OpenPGP
email summit this past May.
https://datatracker.ietf.org/meeting/113/materials/agenda-113-openpgp-01
Furthermore the way we use signatures in RPM
really has no resemblance to the scenarios OpenPGP was built for.
Could you elaborate on this, please.
Thanks,
Neal
4:09 a.m.
New subject: RPM Sequoia - respect system's crypt policy
On Thu, 13 Oct 2022 09:29:27 +0200,
Panu Matilainen wrote:
>> - Some old, insecure (MD5/SHA1 based) signatures are
rejected (this is
>> in line with the stronger crypto settings proposed elsewhere for F38)
>
> Such a hardcoded restriction, without a way for the local administrator to
> allow the legacy signatures, is not acceptable.
Mind you, I don't exactly agree with this style of explicit disabling
either (see
https://lists.rpm.org/pipermail/rpm-maint/2021-October/018344.html and
onwards). But. I doubt many people realize just how thin the ice is
(and has always been) with the existing parser. I consider this step a
matter of survival, and ultimately some legacy content becoming harder
to use is an acceptable tradeoff for *that*.
I don't know how deep this all is wired inside Sequoia, but I totally
agree (as you see in the thread linked above) that this should be
based on the system crypto policy. As explained in the change, nettle
(which doesn't support the system crypto policies AIUI) should be seen
as a temporary stepstone in Fedora, with a plan to switch to openssl
(which does) in the nearish future.
So technically this is a matter of "Sequoia should honor system crypto
policy", rpm is just a dumb API user here that sometimes get told
"nope" by the underlying libraries, whether due to crypto policy, FIPS
or whatever.
I opened [1] to track this issue.
It should be relatively straightforward to implement this. Sequoia
already has first class policy objects that are consulted on every
cryptograph operation [2]. What needs to be done is to read the
Fedora system policy and configure the rpm-sequoia's policy object [3]
appropriately.
:) Neal
[1] https://github.com/rpm-software-management/rpm-sequoia/issues/14
[2] https://docs.sequoia-pgp.org/sequoia_openpgp/policy/index.html
https://docs.sequoia-pgp.org/sequoia_openpgp/policy/struct.StandardPolicy...
[3] https://github.com/rpm-software-management/rpm-sequoia/blob/main/src/lib....
2:06 a.m.
New subject: http[s] mirrors [was Re: F38 proposal: RPM Sequoia (System-Wide
Change proposal)]
On Thu, Oct 13, 2022 at 03:57:41PM +0200, Kevin Kofler via devel wrote:
> Also, a ton of Fedora mirrors still don't use HTTPS for
various reasons.
I would say that those mirrors ought to be kicked out of the mirror list
immediately.
There are also a lot of rsync mirrors. I don't think any of them are using
rsync-ssl.
I think "kicked out" is a bit harsh -- but we should definitely suggest it.
And I think we should also do the metadata signing as soon as practical...
defense in depth and all that.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
2:33 a.m.
On Wed, Oct 19, 2022 at 8:17 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 13/10/2022 15:46, Neal Gompa wrote:
> Also, a ton of Fedora mirrors still don't use HTTPS for various reasons.
I think such insecure mirrors should be removed from metalink.
Why are they insecure? This is public open data, not banking data,
where the data being downloaded is verifiable by the rpm signatures
and signing keys. There is of course a chance an MitM attacker could
work out you're installing sendmail or tuxracer but there's other
mechanisms in place to ensure it's not compromisable mid stream.
The flip side is we remove the non https mirrors and the mirror system
slows down considerably, or in some cases is even unavailable, for
users which in IMO is more of a problem because people then really do
have insecure systems.
Peter
2:40 a.m.
On 19/10/2022 09:33, Peter Robinson wrote:
Why are they insecure? This is public open data, not banking data,
where the data being downloaded is verifiable by the rpm signatures
and signing keys.
ISPs or anyone on the the same network can view, intercept or even
modify HTTP/rsync traffic.
The flip side is we remove the non https mirrors and the mirror
system
slows down considerably, or in some cases is even unavailable, for
users which in IMO is more of a problem because people then really do
have insecure systems.
We can give mirror owners some time (for example, a month) to migrate
their mirrors to HTTPS.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
2:48 a.m.
On Wed, Oct 19, 2022 at 8:40 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 19/10/2022 09:33, Peter Robinson wrote:
> Why are they insecure? This is public open data, not banking data,
> where the data being downloaded is verifiable by the rpm signatures
> and signing keys.
ISPs or anyone on the the same network can view, intercept or even
modify HTTP/rsync traffic.
Sure but as mentioned it's public data, and the modification, and I
covered that in my reply, would be picked up by the other mechanisms.
> The flip side is we remove the non https mirrors and the mirror
system
> slows down considerably, or in some cases is even unavailable, for
> users which in IMO is more of a problem because people then really do
> have insecure systems.
We can give mirror owners some time (for example, a month) to migrate
their mirrors to HTTPS.
The ones that actually care have already migrated, they've been
actively encouraged to do this previously and haven't, see point above
about a lack of mirrors. Ultimately bandwidth is expensive in a lot of
parts of the world for commercial entities to provide, that's why
there's mirrors. There isn't actually that many mirrors left do we
really want to reduce the number more for end users for no actual
improvement in security?
While where you are there may be lots of mirrors there are places
where there are very few and having less will likely make Fedora
unusable, I had a report on the arm channel just this week of problems
with lack of available mirrors just this week.
3:01 a.m.
On 19/10/2022 09:48, Peter Robinson wrote:
Sure but as mentioned it's public data, and the modification, and
I
covered that in my reply, would be picked up by the other mechanisms.
They can collect a lot of sensitive information: your IP, Fedora
version, packages version, etc. This can help with recon for attackers.
There isn't actually that many mirrors left do we
really want to reduce the number more for end users for no actual
improvement in security?
It will improve privacy at least.
Ultimately bandwidth is expensive in a lot of
parts of the world for commercial entities to provide, that's why
there's mirrors.
Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can
switch to a CDN too?
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
3:31 a.m.
On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 19/10/2022 09:48, Peter Robinson wrote:
> Sure but as mentioned it's public data, and the modification, and I
> covered that in my reply, would be picked up by the other mechanisms.
They can collect a lot of sensitive information: your IP, Fedora
version, packages version, etc. This can help with recon for attackers.
HTTPS does not help with that. It's just a transport protocol.
> There isn't actually that many mirrors left do we
> really want to reduce the number more for end users for no actual
> improvement in security?
It will improve privacy at least.
Not in any meaningful way, and in most cases HTTPS makes mirrors slower too.
> Ultimately bandwidth is expensive in a lot of
> parts of the world for commercial entities to provide, that's why
> there's mirrors.
Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can
switch to a CDN too?
We don't have a "main mirror" for that to work.
--
真実はいつも一つ！/ Always, there's only one truth!
3:50 a.m.
On 19-10-2022 10:31, Neal Gompa wrote:
On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> On 19/10/2022 09:48, Peter Robinson wrote:
>> Sure but as mentioned it's public data, and the modification, and I
>> covered that in my reply, would be picked up by the other mechanisms.
>
> They can collect a lot of sensitive information: your IP, Fedora
> version, packages version, etc. This can help with recon for attackers.
>
HTTPS does not help with that. It's just a transport protocol.
Security is covered, as probinson already mentioned, by internal
verification.
>> There isn't actually that many mirrors left do we
>> really want to reduce the number more for end users for no actual
>> improvement in security?
>
> It will improve privacy at least.
>
Not in any meaningful way, and in most cases HTTPS makes mirrors slower too.
If privacy is your concern, which is something the user has to decide,
there's the torproxy plugin for DNF. I think that would protect your
privacy and allow use of plain http - win/win.
>> Ultimately bandwidth is expensive in a lot of
>> parts of the world for commercial entities to provide, that's why
>> there's mirrors.
>
> Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can
> switch to a CDN too?
>
We don't have a "main mirror" for that to work.
So, this has been looked into already? It definitely sounds like it
could help in sparsely served parts of the world at a reasonable cost.
-- Sandro (just throwing in my $0.02)
4:25 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Thu, Oct 13, 2022 at 03:46:49PM +0200, Fabio Valentini wrote:
> The dependency on LLVM is not even the worst issue in my eyes.
LLVM is also
> used by other core projects, e.g., mesa, these days.
>
> The worst issue I see with Rust is the way libraries are "packaged",
which
> just implies installing source code and recompiling that source code for
> every single application. (And as a result, the output obviously gets
> statically linked into the application, with all the drawbacks of static
> linking.) I consider a language with no usable shared library support to be
> entirely unpackageable and hence entirely useless.
Do you have suggestions for improving this situation? I think we're
pretty close to doing the best we can with packaging Rust projects,
given the current limitations of the language (i.e. the support for
building "true shared Rust libraries" is still very limited and
unstable; but that might improve in the future).
I _very much_ appreciate all the work you and the other Rust SIG folks
(Igor and Zbyszek in particular but I'm sure others as well!) have put into
packaging rust apps and crates and all of the systems around that.
I fundamentally disagree with Kevin on a deep level about "entirely
useless", but ... find myself kind of agreeing about the "unpackagable"
part. I mean: clearly we've found a way, but I'm really not sure we're
providing a lot of _value_ in this approach, and I'm also not sure it's as
successful as it could be.
There are three ways having things packaged in Fedora repos _can_ be
helpful:
1. End-user applications and tools
2. Useful development environment
3. As convenience for ourselves for building packages for #1 or #2
I am not discounting the value of #3 -- making a shared thing that we all
work on together is kind of the whole point, and the nicer we can make that
the better we can bring in more people, and those of us already here have a
lighter load and can work on the things we're most interested in. But
ultimately, we're doing it so we make a useful system for users. That means
the first two.
I'll start with the second: our system for Rust doesn't really do that.
Developers are going to use cargo and crates.io and we're not going to
convince them that they should do otherwise. (I don't expect anyone
disagrees with this.)
We're doing okay with #1, but... I think #3 _even_ with all of the work in
Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
engine and will probably have a few things it would be nice to package to
make available in Fedora Linux. I might not even mind maintaining Bevy
itself.
But running `cargo fetch` with a clean cache pulls down *390* crates. Of
these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
which is *amazing*. But... that still is hundreds that I'd have to add. And
mostly they are things I don't know _anything_ about.
*This is what open source winning looks like.*
I remember a Byte magazine article from the 1990 (I just checked!) with the
title "There Is a Silver Bullet: The birth of interchangeable, reusable
software components will bring software into the information age". [1]
This was about the newly-hot idea of Object Oriented Programming. It was
very exciting. But, of course, that vision of the world did not happen. It
turns out proprietary software *can't* do this.
But now we have it! I don't have to reinvent every basic wheel — but even
more than that, I do not have to be an expert in the intricacies of safe
concurrency to write an app that uses that under the hood. That's amazing! I
can do such powerful things from high-level interfaces and trust the
expertise of those who really understand the deep computer science some of
this requires.
I am competent enough to write a silly toy game using Bevy. It might be good
enough that others will enjoy it. *I am not competent to maintain many of
these dependencies.* I don't even know what most of them DO. "anyhow"?
"bytemuck"?
Worse, many of the Bevy deps are specified with exact versions. Maybe I
could make the package work with the packaged versions, but ... that
requires deep expertise and even then might lead to unexpected behavior and
has a high chance of putting me at odds with both the engine upstream and
any other games which use it. The packaging guidelines say that I SHOULD
create patches to update to latest versions of dependencies, and that I
should further convince the upstream to take them. Candidly, that seems like
a waste of everone's time.
The guidelines provide for creating compat packages, but that means 1) the
existing shared work is less useful, 2) requires even more extra steps, and
3) even without reviews for compat has extra administrative overhead.
So, going back to Kevin's point: it _does_ feel like this is unpackagable.
But that's because the barrier to participation seems too high. It's not
because it's statically-linked binaries [2] can't or shouldn't exist in
Fedora — that's not one of our core principles! In fact and quite to the
contrary, we need to adapt to handle this amazing open source success story
better.
And, I led with: I appreciate all the work you've all done to make this
work. That's definitely true — I think it was super-valuable to pilot this
approach. But I think that the Rust ecosystem would be a great place to
pilot a different way. Something lightweight where we cache crates and use
them _directly_ in the build process for _application_ RPMs.
Rust packages include a lot of machine-readable metadata. We should be able
to watch for CVEs, RustSec, and other security notices even without encoding
the metadata in RPMs. License review could also be automated — the field in
Cargo.toml is supposed to be SPDX, so that's convenient. [3]
We could also attach other metadata to the packages in the cache. Maybe some
popularity, update frequency from Cargo.io, but also package review flags:
checked license against source, and whatever other auditing we think should
be done. This moves the focus from specfile-correctness to the package
itself, and the effort from packaging to reviewing. (I'd suggest that for
the experiment, we not make any deep auditing manditory, but instead
encouraged.) And these flags should be able to be added by anyone in the
Rust SIG, not necessarily just at import.
Maybe we could get involved in Cargo Vet [4] — we could be both a consumer
_and_ a data source.
Fedora _needs_ to adapt to stay relevant in the world where every language
stack has developed a packaging ecosystem which effectively ignores us. Some
of them are missing lessons they could have learned, ah well — but they also
have a lot of nice new ideas we're missing. And, no matter what we think,
we're clearly not going to stop them.
Rust packaging seems like a great place to lead the way — and then we can
maybe expand to Go, which has similar issues, and then Java (where, you
know, things have already collapsed despite heroic effort.)
What do you all think?
------
[1] https://archive.org/details/sim_byte_1990-10_15_10/page/n249/mode/2up
[2] Also, actually: https://docs.rs/bevy_dylib/0.8.1/bevy_dylib/. They don't
recommend this for release builds... but that looks to be mainly for
distribution reasons we could easily handle?
[3]
https://communityblog.fedoraproject.org/important-changes-to-software-lic...
[4] https://mozilla.github.io/cargo-vet/
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
4:27 a.m.
New subject: Amazon mirrors [was Re: F38 proposal: RPM Sequoia (System-Wide
Change proposal)]
On Wed, Oct 19, 2022 at 10:50:19AM +0200, Sandro wrote:
>We don't have a "main mirror" for that to work.
So, this has been looked into already? It definitely sounds like it
could help in sparsely served parts of the world at a reasonable
cost.
Yes. I think this is basically just a matter of _doing_.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
6:04 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
I _very much_ appreciate all the work you and the other Rust SIG folks
(Igor and Zbyszek in particular but I'm sure others as well!) have put into
packaging rust apps and crates and all of the systems around that.
I'll respond inline.
I fundamentally disagree with Kevin on a deep level about
"entirely
useless", but ... find myself kind of agreeing about the "unpackagable"
part. I mean: clearly we've found a way, but I'm really not sure we're
providing a lot of _value_ in this approach, and I'm also not sure it's as
successful as it could be.
We *do* provide value to both users *and* developers by doing things
the way we do, but the benefits might not be obvious to people who
don't know how (Rust) packaging works, and what we as package
maintainers do.
There are three ways having things packaged in Fedora repos _can_ be
helpful:
1. End-user applications and tools
2. Useful development environment
3. As convenience for ourselves for building packages for #1 or #2
I am not discounting the value of #3 -- making a shared thing that we all
work on together is kind of the whole point, and the nicer we can make that
the better we can bring in more people, and those of us already here have a
lighter load and can work on the things we're most interested in. But
ultimately, we're doing it so we make a useful system for users. That means
the first two.
This I can agree with :)
I'll start with the second: our system for Rust doesn't
really do that.
Developers are going to use cargo and crates.io and we're not going to
convince them that they should do otherwise. (I don't expect anyone
disagrees with this.)
This is true, and probably also not "fixable". We need to make some
amount of non-upstreamable patches to some crates (most notably,
removing Windows- or mac OS-specific dependencies, because we don't
want to package those), but in some cases, these are "incompatible"
changes, and Rust *developers* should not be targeting our downstream
sources that have these differences with actual upstream sources.
This is due to a limitation of how cargo handles target-specific
dependencies - all dependencies that are *mentioned in any way* need
to be *present* for it to resolve dependencies / enabled optional
features / update its lockfile etc. But since we don't want to package
bindings for Windows and mac OS system APIs, we need to actually patch
them out, otherwise builds will fail.
We're doing okay with #1, but... I think #3 _even_ with all of
the work in
Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
engine and will probably have a few things it would be nice to package to
make available in Fedora Linux. I might not even mind maintaining Bevy
itself.
Somebody actually already started packaging Bevy components - some
packages are already approved and some are still pending review. Not
sure what the progress has been there, but it's not *impossible*.
But running `cargo fetch` with a clean cache pulls down *390* crates.
Of
these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
which is *amazing*. But... that still is hundreds that I'd have to add. And
mostly they are things I don't know _anything_ about.
You must realize that this is an extreme case. For many Rust
applications that people want to package for Fedora, the number of
dependencies that are missing is rather small, *because* most popular
libraries are already packaged.
Bevy is a bit special, because it (presumably) pulls in lots of GPU /
OpenGL / Vulkan related libraries, which we didn't need to package for
anything else yet, and it's also split into dozens of small libraries
itself, which can be painful to package, that is true.
We might need to reconsider how to package projects like this. I'm
pretty sure we could find a way to package them in a way that's
compatible with how we're currently doing things but would be much
less busywork.
*This is what open source winning looks like.*
I remember a Byte magazine article from the 1990 (I just checked!) with the
title "There Is a Silver Bullet: The birth of interchangeable, reusable
software components will bring software into the information age". [1]
This was about the newly-hot idea of Object Oriented Programming. It was
very exciting. But, of course, that vision of the world did not happen. It
turns out proprietary software *can't* do this.
But now we have it! I don't have to reinvent every basic wheel — but even
more than that, I do not have to be an expert in the intricacies of safe
concurrency to write an app that uses that under the hood. That's amazing! I
can do such powerful things from high-level interfaces and trust the
expertise of those who really understand the deep computer science some of
this requires.
I am competent enough to write a silly toy game using Bevy. It might be good
enough that others will enjoy it. *I am not competent to maintain many of
these dependencies.* I don't even know what most of them DO. "anyhow"?
"bytemuck"?
Sure, but isn't that the case for most projects that a newcomer wants
to package, regardless of programming language? Say, somebody wants to
package some cool new Python project for machine learning, then
there's probably also some linear algebra package or SIMD math library
in the dependency tree that's missing from Fedora. How is that
different?
Worse, many of the Bevy deps are specified with exact versions. Maybe
I
could make the package work with the packaged versions, but ... that
requires deep expertise and even then might lead to unexpected behavior and
has a high chance of putting me at odds with both the engine upstream and
any other games which use it.
For intra-project dependencies (i.e. bevy components depending on
exact versions of bevy components), this is kind of expected, and we
have tools to deal with this kind of situation (though bevy is on a
different scale). For dependencies on third-party libraries, this is
kind of unexpected, and I wonder why they do things like that? Locking
some dependencies to exact versions is usually handled by relying on
the lockfile, instead.
Still, for large projects that are split into many crates like bevy, I
think we could find a way to make packaging them a lot easier (i.e.
not use one source package per crate, but use one source package for
building a project that contains many crates).
The packaging guidelines say that I SHOULD
create patches to update to latest versions of dependencies, and that I
should further convince the upstream to take them. Candidly, that seems like
a waste of everone's time.
This is *not* a waste of time.
If we don't invest time to do that, many project's dependencies grow
stale, and actually *increase* the need for us to maintain compat
packages.
The guidelines provide for creating compat packages, but that means
1) the
existing shared work is less useful, 2) requires even more extra steps, and
3) even without reviews for compat has extra administrative overhead.
We only maintain compat packages where porting to the new version (and
submitting the changes upstream) is not feasible. Again, isn't that
how Fedora is supposed to work?
So, going back to Kevin's point: it _does_ feel like this is
unpackagable.
But that's because the barrier to participation seems too high. It's not
because it's statically-linked binaries [2] can't or shouldn't exist in
Fedora — that's not one of our core principles! In fact and quite to the
contrary, we need to adapt to handle this amazing open source success story
better.
The barrier for participation is too high in some cases, I agree.
However, in my experience, that's for a different reason:
The "shiny new things that happen to be written in Rust" that new
contributors want to have in Fedora are often very complicated
projects that even experienced Rust packagers would need to spend a
lot of time on.
Examples of that might be:
- wasmtime: I ultimately abandoned the attempt to package it "because
Fedora Legal", but the packages themselves worked fine
- deno: requires dozens of new packages, some of which also have
unclear / questionable licenses as well, but the packages themselves
worked
On the other hand, many "nice" CLI tools that people want to package
often require minimal knowledge of Rust packaging (our tools are
pretty nice for "standard" projects), and often only need very few new
dependencies to be packaged.
Just as an example, I just today started reviewing a "simple" Rust
application here:
https://bugzilla.redhat.com/show_bug.cgi?id=1990713
The spec file is very simple and almost entirely automatically
generated (with the exception of the missing License breakdown for the
statically linked binary), no dependencies were missing from Fedora.
Even Rust newbies would not have trouble packaging this, and that
would be a way better entry point than packaging stuff like Bevy.
And, I led with: I appreciate all the work you've all done to
make this
work. That's definitely true — I think it was super-valuable to pilot this
approach. But I think that the Rust ecosystem would be a great place to
pilot a different way. Something lightweight where we cache crates and use
them _directly_ in the build process for _application_ RPMs.
We have talked about this multiple times, but it won't work.
I think this was tried with first-class maven artifact support in
koji, but we all know how the Java packaging fiasco ended.
Or even if making Rust crates first-class deliverables *did work*, it
wouldn't give us the benefits of the current approach:
- we ensure that all crates in Fedora *build* on all architectures
- we ensure that most crates in Fedora pass their test suites on all
architectures
- we check all crates for objectionable content, licensing problems, etc.
- we change build flags to default to dynamically linking to system
libraries instead of statically linking against vendored copies
This would mean that we basically stop contributing things to the
upstream Rust ecosystem:
- we diagnose / report / fix architecture support issues
- we port projects to new versions of dependencies
- etc.
I see this work in the upstream ecosystem as an important part of the
work we do in packaging Rust crates for Fedora,
and I would not want to endorse an approach that meant we no longer do
these things.
Rust packages include a lot of machine-readable metadata. We should
be able
to watch for CVEs, RustSec, and other security notices even without encoding
the metadata in RPMs. License review could also be automated — the field in
Cargo.toml is supposed to be SPDX, so that's convenient. [3]
I already monitor RustSec advisories and check *all of them* against
Fedora packages. This takes up a miniscule amount of the time I spend
on Rust packaging (because there's so few Rust security advisories).
If I remember correctly, there were only 2-3 CVE issues in the Rust
stack that actually affected our packages, and dealing with those was
very simple:
1) Push the patched version of the library, 2) rebuild dependent
applications, 3) submit to bodhi.
There's some amount of automation that *could* be done (mostly in
figuring out which applications need to be rebuilt for a given library
change), but that's also pretty easily done with a "dnf repoquery" or
two.
On the other hand, license review is still important, even if it's
already available in SPDX format in the upstream metadata.
Just because sometimes, that metadata is either wrong or incomplete.
And even more often, package review flags other problems (like missing
LICENSE files for licenses that *require* redistributed sources to
contain a copy of the license text). Relying on SPDX metadata alone is
*not* safe.
We could also attach other metadata to the packages in the cache.
Maybe some
popularity, update frequency from Cargo.io, but also package review flags:
checked license against source, and whatever other auditing we think should
be done. This moves the focus from specfile-correctness to the package
itself, and the effort from packaging to reviewing. (I'd suggest that for
the experiment, we not make any deep auditing manditory, but instead
encouraged.) And these flags should be able to be added by anyone in the
Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
Maybe we could get involved in Cargo Vet [4] — we could be both a
consumer
_and_ a data source.
Fedora _needs_ to adapt to stay relevant in the world where every language
stack has developed a packaging ecosystem which effectively ignores us. Some
of them are missing lessons they could have learned, ah well — but they also
have a lot of nice new ideas we're missing. And, no matter what we think,
we're clearly not going to stop them.
Rust packaging seems like a great place to lead the way — and then we can
maybe expand to Go, which has similar issues, and then Java (where, you
know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
Given that, I think we're actually in kind of a *good* situation with
Rust packaging, especially compared to other language ecosystems - not
only right now, but also looking at the future. And looking at the
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
Fabio
6:56 a.m.
On 19/10/2022 10:31, Neal Gompa wrote:
HTTPS does not help with that. It's just a transport protocol.
It will. All requests will be encrypted. ISP will only see server's
IP-address and its hostname (only if SNI is enabled).
Not in any meaningful way, and in most cases HTTPS makes mirrors
slower too.
No. All modern servers support AES-NI, so encryption doesn't slow down
servers.
We don't have a "main mirror" for that to work.
Then where do mirrors get information about updates?
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
7:14 a.m.
On Wed, Oct 19, 2022 at 01:56:47PM +0200, Vitaly Zaitsev via devel wrote:
On 19/10/2022 10:31, Neal Gompa wrote:
> HTTPS does not help with that. It's just a transport protocol.
It will. All requests will be encrypted. ISP will only see server's
IP-address and its hostname (only if SNI is enabled).
> Not in any meaningful way, and in most cases HTTPS makes mirrors slower too.
No. All modern servers support AES-NI, so encryption doesn't slow down
servers.
AES-NI does not provide infinite speed. Each CPU has a limit
to how much data it can shuffle through AES-NI in a given
timeframe. AES-NI may well be x10 faster than doing AES
in software with generic instructions, but it still has a
performance upper bound.
In my previous testing of AES-NI for QEMU live migration, I
was unable to saturate the max available NIC bandwidth
available. It was massively better than not using AES-NI,
but not encrypting at all was still faster by a significant
degree.
IOW, the impact of AES on server peformance will vary depending
on CPU models, NIC models / network switches and whether other
workloads are competing for CPU time. Admins need to decide
what tradeoffs are important to them.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:46 a.m.
On 19/10/2022 14:14, Daniel P. Berrangé wrote:
IOW, the impact of AES on server peformance will vary depending
on CPU models, NIC models / network switches and whether other
workloads are competing for CPU time. Admins need to decide
what tradeoffs are important to them.
In future, modern web browsers will start rejecting unencrypted HTTP
traffic (except localhost and intranet, ofc). All server owners should
switch to HTTPS as soon as possible.
--
Sincerely,
Vitaly Zaitsev (vitaly(a)easycoding.org)
7:47 a.m.
Once upon a time, Vitaly Zaitsev <vitaly(a)easycoding.org> said:
On 19/10/2022 09:33, Peter Robinson wrote:
>Why are they insecure? This is public open data, not banking data,
>where the data being downloaded is verifiable by the rpm signatures
>and signing keys.
ISPs or anyone on the the same network can view, intercept or even
modify HTTP/rsync traffic.
And so?
The mirror network is essentially already an untrusted man in the middle
between the Fedora build servers and you. Most mirror servers are
hosted by large organizations (companies and schools with extra
bandwidth) - in many cases, they ARE the ISPs. At a previous ISP job, I
ran a mirror and marked it as "preferred" for my networks, which meant
all my customers used my mirror by default.
Anybody can set that up, with pretty minimal verification. You are
trying to insist on a trust layer for an already-untrusted system.
--
Chris Adams <linux(a)cmadams.net>
7:51 a.m.
On Wed, 19 Oct 2022 at 05:09, Sandro <lists(a)penguinpee.nl> wrote:
On 19-10-2022 10:31, Neal Gompa wrote:
> On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel
> We don't have a "main mirror" for that to work.
dl.fedoraproject.org would be what this would aim at but we would probably
have to change the disk layout for what the CDN works best for.
So, this has been looked into already? It definitely sounds like it
could help in sparsely served parts of the world at a reasonable cost.
Please do not use the word 'reasonable' cost without actually knowing what
the costs are :).
1. Amazon is currently donating bandwidth and systems for our work but it
is in a non-written format. If Amazon tomorrow decided that they needed to
concentrate on other things or look at what we are costing and say 'well it
was time to stop', we would have to turn it all off then. If we require
various flags in the CDN we have to go through their change management and
accounting to get them done.
2. Akamai and other CDN's are expensive. You pay to put content into them,
you pay to clear out caches, you pay differently for the areas where the
CDN is covering, and you pay for the amount of data come out. You also pay
for them to block bad actors whose job is to try and make you go out of
business by downloading constantly. For even a 'niche' market like Fedora,
we would be looking at least 1.5 million per year in costs.
3. CDN's work well for software which does not change a lot but if you have
updates daily you end up having to force cache cleanouts and new upload
costs. Even then you end up with needing to deal with 'broken' parts of the
CDN and people getting old data. This happens regularly with COPR and
CentOS parts which have been using the Amazon CDN at times.
All of these are possible to get past, but they will take time, effort and
may be more than what anyone wants to be 'reasonable'.
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
8:34 a.m.
On Wed, Oct 19, 2022 at 9:33 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> On 19/10/2022 09:48, Peter Robinson wrote:
> > Sure but as mentioned it's public data, and the modification, and I
> > covered that in my reply, would be picked up by the other mechanisms.
>
> They can collect a lot of sensitive information: your IP, Fedora
> version, packages version, etc. This can help with recon for attackers.
>
HTTPS does not help with that. It's just a transport protocol.
> > There isn't actually that many mirrors left do we
> > really want to reduce the number more for end users for no actual
> > improvement in security?
>
> It will improve privacy at least.
>
Not in any meaningful way, and in most cases HTTPS makes mirrors slower too.
> > Ultimately bandwidth is expensive in a lot of
> > parts of the world for commercial entities to provide, that's why
> > there's mirrors.
>
> Fedora COPR has moved to Amazon CDN. Maybe Fedora's main mirror can
> switch to a CDN too?
>
We don't have a "main mirror" for that to work.
We do, it's called download.fedoraproject.org (or dl.fedoraproject.org
for the non redirect to mirror*) but who is going to pay for the AWS
traffic, the mirror already is plugged into AWS for AWS traffic so if
you're running instances in the cloud you actually get it directly
from AWS as that's in their best interest but it's not covered for
everyone on the general internet.
8:43 a.m.
On Wed, Oct 19, 2022 at 1:47 PM Vitaly Zaitsev via devel
<devel(a)lists.fedoraproject.org> wrote:
On 19/10/2022 14:14, Daniel P. Berrangé wrote:
> IOW, the impact of AES on server peformance will vary depending
> on CPU models, NIC models / network switches and whether other
> workloads are competing for CPU time. Admins need to decide
> what tradeoffs are important to them.
In future, modern web browsers will start rejecting unencrypted HTTP
traffic (except localhost and intranet, ofc). All server owners should
switch to HTTPS as soon as possible.
We don't use a browser for general updates, we use an application
which can configure libcurl for it's desired requirements.
The use of AES is not just required for client side, but on server
side too, it takes non inconsiderable amount of compute to saturate a
10G+ link using https, that compute needs to come from somewhere and
against most of the mirrors are being provided as a community donation
and likely have requirements where if it hits a threshold it likely
makes it unviable for that company to provide where they don't get a
return (advertising, or something pro-rata)
11 a.m.
On 19-10-2022 14:51, Stephen Smoogen wrote:
On Wed, 19 Oct 2022 at 05:09, Sandro <lists(a)penguinpee.nl>
wrote:
> On 19-10-2022 10:31, Neal Gompa wrote:
>> On Wed, Oct 19, 2022 at 4:01 AM Vitaly Zaitsev via devel
>
>> We don't have a "main mirror" for that to work.
>
>
dl.fedoraproject.org would be what this would aim at but we would probably
have to change the disk layout for what the CDN works best for.
> So, this has been looked into already? It definitely sounds like it
> could help in sparsely served parts of the world at a reasonable cost.
>
Please do not use the word 'reasonable' cost without actually knowing what
the costs are :).
1. Amazon is currently donating bandwidth and systems for our work but it
is in a non-written format. If Amazon tomorrow decided that they needed to
concentrate on other things or look at what we are costing and say 'well it
was time to stop', we would have to turn it all off then. If we require
various flags in the CDN we have to go through their change management and
accounting to get them done.
2. Akamai and other CDN's are expensive. You pay to put content into them,
you pay to clear out caches, you pay differently for the areas where the
CDN is covering, and you pay for the amount of data come out. You also pay
for them to block bad actors whose job is to try and make you go out of
business by downloading constantly. For even a 'niche' market like Fedora,
we would be looking at least 1.5 million per year in costs.
3. CDN's work well for software which does not change a lot but if you have
updates daily you end up having to force cache cleanouts and new upload
costs. Even then you end up with needing to deal with 'broken' parts of the
CDN and people getting old data. This happens regularly with COPR and
CentOS parts which have been using the Amazon CDN at times.
All of these are possible to get past, but they will take time, effort and
may be more than what anyone wants to be 'reasonable'.
Point taken. ;)
Thanks for the very detailed answer. I only glanced at the pricing and
just now noticed the *most Important* part: '_per GB_'. So, yeah, this
will add up for sure. I should have been more careful.
-- Sandro
3:13 a.m.
On 19/10/2022 13:56, Vitaly Zaitsev via devel wrote:
On 19/10/2022 10:31, Neal Gompa wrote:
> HTTPS does not help with that. It's just a transport protocol.
It will. All requests will be encrypted. ISP will only see server's
IP-address and its hostname (only if SNI is enabled).
HTTPS does not automatically add privacy for users of mirrors. See for
example the excellent Debian related website and OpenBSD presentation
about how you can still identify downloaded packages even when using HTTPS.
[1] https://whydoesaptnotusehttps.com/
[2] https://www.openbsd.org/papers/eurobsdcon_2018_https.pdf
4:03 a.m.
On 10. 10. 22 16:32, Ben Cotton wrote:
For the last 20 years or so, RPM has used a home-grown OpenPGP
parser
for dealing with keys and signatures. That parser is rather infamous
for its limitations and flaws, and especially in recent years has
proven a significant burden to RPM development. In order to improve
security and free developer resources for dealing with RPM's "core
business" instead, RPM upstream is in the process of deprecating the
internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
based solution written in Rust.
At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100% available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
to build rpms. 3) seems somewhat tedious for no good reason.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
5:11 a.m.
On Thu, Oct 20, 2022 at 11:03 AM Miro Hrončok <mhroncok(a)redhat.com> wrote:
On 10. 10. 22 16:32, Ben Cotton wrote:
> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
> for dealing with keys and signatures. That parser is rather infamous
> for its limitations and flaws, and especially in recent years has
> proven a significant burden to RPM development. In order to improve
> security and free developer resources for dealing with RPM's "core
> business" instead, RPM upstream is in the process of deprecating the
> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
> based solution written in Rust.
> At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100% available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
to build rpms. 3) seems somewhat tedious for no good reason.
I think 2) would be the easiest solution in the long term. rustc and
LLVM now already support all targets that we *might* want to add in
the near-to-mid-future (riscv64gc-unknown-linux-gnu comes to mind -
its support is at the same level as other targets we already have).
Fabio
6:49 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
Lots of good wisdom here, thank you. IMHO Rust will benefit from whatever
"adult supervision" Fedora can provide.
-Blaise (currently undergoing treatment for injuries sustained supporting
npm in production)
On Wed, Oct 19, 2022 at 7:05 AM Fabio Valentini <decathorpe(a)gmail.com>
wrote:
On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
>
> I _very much_ appreciate all the work you and the other Rust SIG folks
> (Igor and Zbyszek in particular but I'm sure others as well!) have put
into
> packaging rust apps and crates and all of the systems around that.
I'll respond inline.
> I fundamentally disagree with Kevin on a deep level about "entirely
> useless", but ... find myself kind of agreeing about the
"unpackagable"
> part. I mean: clearly we've found a way, but I'm really not sure we're
> providing a lot of _value_ in this approach, and I'm also not sure it's
as
> successful as it could be.
We *do* provide value to both users *and* developers by doing things
the way we do, but the benefits might not be obvious to people who
don't know how (Rust) packaging works, and what we as package
maintainers do.
> There are three ways having things packaged in Fedora repos _can_ be
> helpful:
>
> 1. End-user applications and tools
> 2. Useful development environment
> 3. As convenience for ourselves for building packages for #1 or #2
>
> I am not discounting the value of #3 -- making a shared thing that we all
> work on together is kind of the whole point, and the nicer we can make
that
> the better we can bring in more people, and those of us already here
have a
> lighter load and can work on the things we're most interested in. But
> ultimately, we're doing it so we make a useful system for users. That
means
> the first two.
This I can agree with :)
> I'll start with the second: our system for Rust doesn't really do that.
> Developers are going to use cargo and crates.io and we're not going to
> convince them that they should do otherwise. (I don't expect anyone
> disagrees with this.)
This is true, and probably also not "fixable". We need to make some
amount of non-upstreamable patches to some crates (most notably,
removing Windows- or mac OS-specific dependencies, because we don't
want to package those), but in some cases, these are "incompatible"
changes, and Rust *developers* should not be targeting our downstream
sources that have these differences with actual upstream sources.
This is due to a limitation of how cargo handles target-specific
dependencies - all dependencies that are *mentioned in any way* need
to be *present* for it to resolve dependencies / enabled optional
features / update its lockfile etc. But since we don't want to package
bindings for Windows and mac OS system APIs, we need to actually patch
them out, otherwise builds will fail.
> We're doing okay with #1, but... I think #3 _even_ with all of the work
in
> Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
> engine and will probably have a few things it would be nice to package to
> make available in Fedora Linux. I might not even mind maintaining Bevy
> itself.
Somebody actually already started packaging Bevy components - some
packages are already approved and some are still pending review. Not
sure what the progress has been there, but it's not *impossible*.
> But running `cargo fetch` with a clean cache pulls down *390* crates. Of
> these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
> which is *amazing*. But... that still is hundreds that I'd have to add.
And
> mostly they are things I don't know _anything_ about.
You must realize that this is an extreme case. For many Rust
applications that people want to package for Fedora, the number of
dependencies that are missing is rather small, *because* most popular
libraries are already packaged.
Bevy is a bit special, because it (presumably) pulls in lots of GPU /
OpenGL / Vulkan related libraries, which we didn't need to package for
anything else yet, and it's also split into dozens of small libraries
itself, which can be painful to package, that is true.
We might need to reconsider how to package projects like this. I'm
pretty sure we could find a way to package them in a way that's
compatible with how we're currently doing things but would be much
less busywork.
> *This is what open source winning looks like.*
>
> I remember a Byte magazine article from the 1990 (I just checked!) with
the
> title "There Is a Silver Bullet: The birth of interchangeable, reusable
> software components will bring software into the information age". [1]
> This was about the newly-hot idea of Object Oriented Programming. It was
> very exciting. But, of course, that vision of the world did not happen.
It
> turns out proprietary software *can't* do this.
>
> But now we have it! I don't have to reinvent every basic wheel — but even
> more than that, I do not have to be an expert in the intricacies of safe
> concurrency to write an app that uses that under the hood. That's
amazing! I
> can do such powerful things from high-level interfaces and trust the
> expertise of those who really understand the deep computer science some
of
> this requires.
>
> I am competent enough to write a silly toy game using Bevy. It might be
good
> enough that others will enjoy it. *I am not competent to maintain many of
> these dependencies.* I don't even know what most of them DO.
"anyhow"?
> "bytemuck"?
Sure, but isn't that the case for most projects that a newcomer wants
to package, regardless of programming language? Say, somebody wants to
package some cool new Python project for machine learning, then
there's probably also some linear algebra package or SIMD math library
in the dependency tree that's missing from Fedora. How is that
different?
> Worse, many of the Bevy deps are specified with exact versions. Maybe I
> could make the package work with the packaged versions, but ... that
> requires deep expertise and even then might lead to unexpected behavior
and
> has a high chance of putting me at odds with both the engine upstream and
> any other games which use it.
For intra-project dependencies (i.e. bevy components depending on
exact versions of bevy components), this is kind of expected, and we
have tools to deal with this kind of situation (though bevy is on a
different scale). For dependencies on third-party libraries, this is
kind of unexpected, and I wonder why they do things like that? Locking
some dependencies to exact versions is usually handled by relying on
the lockfile, instead.
Still, for large projects that are split into many crates like bevy, I
think we could find a way to make packaging them a lot easier (i.e.
not use one source package per crate, but use one source package for
building a project that contains many crates).
> The packaging guidelines say that I SHOULD
> create patches to update to latest versions of dependencies, and that I
> should further convince the upstream to take them. Candidly, that seems
like
> a waste of everone's time.
This is *not* a waste of time.
If we don't invest time to do that, many project's dependencies grow
stale, and actually *increase* the need for us to maintain compat
packages.
> The guidelines provide for creating compat packages, but that means 1)
the
> existing shared work is less useful, 2) requires even more extra steps,
and
> 3) even without reviews for compat has extra administrative overhead.
We only maintain compat packages where porting to the new version (and
submitting the changes upstream) is not feasible. Again, isn't that
how Fedora is supposed to work?
> So, going back to Kevin's point: it _does_ feel like this is
unpackagable.
> But that's because the barrier to participation seems too high. It's not
> because it's statically-linked binaries [2] can't or shouldn't exist in
> Fedora — that's not one of our core principles! In fact and quite to the
> contrary, we need to adapt to handle this amazing open source success
story
> better.
The barrier for participation is too high in some cases, I agree.
However, in my experience, that's for a different reason:
The "shiny new things that happen to be written in Rust" that new
contributors want to have in Fedora are often very complicated
projects that even experienced Rust packagers would need to spend a
lot of time on.
Examples of that might be:
- wasmtime: I ultimately abandoned the attempt to package it "because
Fedora Legal", but the packages themselves worked fine
- deno: requires dozens of new packages, some of which also have
unclear / questionable licenses as well, but the packages themselves
worked
On the other hand, many "nice" CLI tools that people want to package
often require minimal knowledge of Rust packaging (our tools are
pretty nice for "standard" projects), and often only need very few new
dependencies to be packaged.
Just as an example, I just today started reviewing a "simple" Rust
application here:
https://bugzilla.redhat.com/show_bug.cgi?id=1990713
The spec file is very simple and almost entirely automatically
generated (with the exception of the missing License breakdown for the
statically linked binary), no dependencies were missing from Fedora.
Even Rust newbies would not have trouble packaging this, and that
would be a way better entry point than packaging stuff like Bevy.
> And, I led with: I appreciate all the work you've all done to make this
> work. That's definitely true — I think it was super-valuable to pilot
this
> approach. But I think that the Rust ecosystem would be a great place to
> pilot a different way. Something lightweight where we cache crates and
use
> them _directly_ in the build process for _application_ RPMs.
We have talked about this multiple times, but it won't work.
I think this was tried with first-class maven artifact support in
koji, but we all know how the Java packaging fiasco ended.
Or even if making Rust crates first-class deliverables *did work*, it
wouldn't give us the benefits of the current approach:
- we ensure that all crates in Fedora *build* on all architectures
- we ensure that most crates in Fedora pass their test suites on all
architectures
- we check all crates for objectionable content, licensing problems, etc.
- we change build flags to default to dynamically linking to system
libraries instead of statically linking against vendored copies
This would mean that we basically stop contributing things to the
upstream Rust ecosystem:
- we diagnose / report / fix architecture support issues
- we port projects to new versions of dependencies
- etc.
I see this work in the upstream ecosystem as an important part of the
work we do in packaging Rust crates for Fedora,
and I would not want to endorse an approach that meant we no longer do
these things.
> Rust packages include a lot of machine-readable metadata. We should be
able
> to watch for CVEs, RustSec, and other security notices even without
encoding
> the metadata in RPMs. License review could also be automated — the field
in
> Cargo.toml is supposed to be SPDX, so that's convenient. [3]
I already monitor RustSec advisories and check *all of them* against
Fedora packages. This takes up a miniscule amount of the time I spend
on Rust packaging (because there's so few Rust security advisories).
If I remember correctly, there were only 2-3 CVE issues in the Rust
stack that actually affected our packages, and dealing with those was
very simple:
1) Push the patched version of the library, 2) rebuild dependent
applications, 3) submit to bodhi.
There's some amount of automation that *could* be done (mostly in
figuring out which applications need to be rebuilt for a given library
change), but that's also pretty easily done with a "dnf repoquery" or
two.
On the other hand, license review is still important, even if it's
already available in SPDX format in the upstream metadata.
Just because sometimes, that metadata is either wrong or incomplete.
And even more often, package review flags other problems (like missing
LICENSE files for licenses that *require* redistributed sources to
contain a copy of the license text). Relying on SPDX metadata alone is
*not* safe.
> We could also attach other metadata to the packages in the cache. Maybe
some
> popularity, update frequency from Cargo.io, but also package review
flags:
> checked license against source, and whatever other auditing we think
should
> be done. This moves the focus from specfile-correctness to the package
> itself, and the effort from packaging to reviewing. (I'd suggest that for
> the experiment, we not make any deep auditing manditory, but instead
> encouraged.) And these flags should be able to be added by anyone in the
> Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
> Maybe we could get involved in Cargo Vet [4] — we could be both a
consumer
> _and_ a data source.
>
> Fedora _needs_ to adapt to stay relevant in the world where every
language
> stack has developed a packaging ecosystem which effectively ignores us.
Some
> of them are missing lessons they could have learned, ah well — but they
also
> have a lot of nice new ideas we're missing. And, no matter what we think,
> we're clearly not going to stop them.
>
> Rust packaging seems like a great place to lead the way — and then we can
> maybe expand to Go, which has similar issues, and then Java (where, you
> know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
Given that, I think we're actually in kind of a *good* situation with
Rust packaging, especially compared to other language ecosystems - not
only right now, but also looking at the future. And looking at the
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
Fabio
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
LinkedIn <https://www.linkedin.com/in/blaisepabon/> | Quora
<https://www.quora.com/profile/Blaise-Pabon> | Github
<https://github.com/blaisep>
“If you want to go fast, go alone. If you want to go far, go
together.” --African
proverb
7:26 a.m.
On 10/20/22 12:03, Miro Hrončok wrote:
On 10. 10. 22 16:32, Ben Cotton wrote:
> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
> for dealing with keys and signatures. That parser is rather infamous
> for its limitations and flaws, and especially in recent years has
> proven a significant burden to RPM development. In order to improve
> security and free developer resources for dealing with RPM's "core
> business" instead, RPM upstream is in the process of deprecating the
> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
> based solution written in Rust.
> At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100%
available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't be
able to build rpms. 3) seems somewhat tedious for no good reason.
I was under the impression Rust was available for all architectures (for
Fedora anyway), no? There's no Rust code in rpm now either this didn't
even cross my mind really :D
Technically, I guess the right thing to do is 1) when Sequoia is
enabled. Ie:
%if %{with sequoia}
%global crypto sequoia
BuildRequires: rpm-sequoia-devel >= 1.0.0
ExclusiveArch: %{rust_archves}
%else
%global crypto openssl
BuildRequires: openssl-devel
%endif
This is already in rawhide, except for the ExclusiveArch thing.
That said, the non-sequoia options should be considered only a bootstrap
aid, we're not going to provide security support for the internal parser
for some fringe architectures only.
I'm not sure that answered your questions though.
- Panu -
8:39 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Matthew Miller wrote:
I fundamentally disagree with Kevin on a deep level about
"entirely
useless", but ... find myself kind of agreeing about the "unpackagable"
part. I mean: clearly we've found a way, but I'm really not sure we're
providing a lot of _value_ in this approach, and I'm also not sure it's as
successful as it could be.
We cannot ship what we cannot package. (Just repackaging upstream blobs is a
no-go, it is explicitly banned by Fedora Packaging Guidelines for good
reasons.)
I'll start with the second: our system for Rust doesn't
really do that.
Developers are going to use cargo and crates.io and we're not going to
convince them that they should do otherwise. (I don't expect anyone
disagrees with this.)
That is one of the issues (along with lack of proper shared library support)
that makes Rust such a painful language: it makes it way too easy for
developers to just add a dependency on some random unpackaged (by
distributions) library with some random license and with random transitive
dependencies.
In a reasonable programming language, you have to think twice before you add
yet another dependency to your project. Tools like cargo make it way too
easy, just add a line and rebuild. So then you end up with a mess like this:
But running `cargo fetch` with a clean cache pulls down *390* crates.
390 dependencies is absolutely insane! No project should ever depend on so
many libraries. This is completely unmaintainable.
*This is what open source winning looks like.*
I remember a Byte magazine article from the 1990 (I just checked!) with
the title "There Is a Silver Bullet: The birth of interchangeable,
reusable software components will bring software into the information
age". [1] This was about the newly-hot idea of Object Oriented
Programming. It was very exciting. But, of course, that vision of the
world did not happen. It turns out proprietary software *can't* do this.
But now we have it! I don't have to reinvent every basic wheel — but even
more than that, I do not have to be an expert in the intricacies of safe
concurrency to write an app that uses that under the hood. That's amazing!
I can do such powerful things from high-level interfaces and trust the
expertise of those who really understand the deep computer science some of
this requires.
Code reuse is great when it makes sense, but this "NPM culture" where
developers happily depend on tons of packages containing trivial one-line
functions is completely insane.
If this is now the essence of "Open Source", then it will ultimately lose.
Proprietary software companies will just be waiting for the next paradigm
shift to lock us into their proprietary ecosystem again. For a large part,
this has already happened with smartphones: you get to choose only between
the Android walled garden (with an open core and lots of proprietary Google
services and (Google and third-party) apps on top) and the iOS walled garden
(which is completely proprietary and closed, even strictly enforcing an App
Store monopoly). We are struggling to escape from that with devices like the
PinePhone. But with development practices relying on an unmaintainable
dependency hell, it will never happen.
I am competent enough to write a silly toy game using Bevy. It might
be
good enough that others will enjoy it. *I am not competent to maintain
many of these dependencies.* I don't even know what most of them DO.
"anyhow"? "bytemuck"?
Welcome to dependency hell!
Worse, many of the Bevy deps are specified with exact versions. Maybe
I
could make the package work with the packaged versions, but ... that
requires deep expertise and even then might lead to unexpected behavior
and has a high chance of putting me at odds with both the engine upstream
and any other games which use it. The packaging guidelines say that I
SHOULD create patches to update to latest versions of dependencies, and
that I should further convince the upstream to take them. Candidly, that
seems like a waste of everone's time.
Hardcoding exact versions of dependencies is one of the worst misfeatures of
those language-specific build tools. It effectively prevents you from fixing
security issues without patching every single application. And of course it
can cause version conflicts between different applications or even between
transitive dependencies within an application. (And, while the former ones
can typically be resolved by shipping both versions of the library, the
latter ones cannot, because you cannot usually link two versions of the same
library into the same program.)
So, going back to Kevin's point: it _does_ feel like this is
unpackagable.
But that's because the barrier to participation seems too high.
No, the answer to upstream shipping an unpackageable mess cannot possibly be
to make it easier to smuggle unpackageable messes into Fedora!
It's not because it's statically-linked binaries [2]
can't or shouldn't
exist in Fedora — that's not one of our core principles!
It is actually one of our Packaging Guidelines that static linking is
discouraged:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-stat...
as is bundling libraries:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling
(So they can indeed exist, but they definitely SHOULD NOT, the Guidelines
undisputably say so.)
In fact and quite to the contrary, we need to adapt to handle this
amazing
open source success story better.
I fail to see a success story here. I see a failure story. Dependency hell
is not a success.
And, I led with: I appreciate all the work you've all done to
make this
work. That's definitely true — I think it was super-valuable to pilot this
approach. But I think that the Rust ecosystem would be a great place to
pilot a different way. Something lightweight where we cache crates and use
them _directly_ in the build process for _application_ RPMs.
Fabio has already explained very well why that is a horrible idea.
Fedora _needs_ to adapt to stay relevant in the world where every
language
stack has developed a packaging ecosystem which effectively ignores us.
Some of them are missing lessons they could have learned, ah well — but
they also have a lot of nice new ideas we're missing. And, no matter what
we think, we're clearly not going to stop them.
Rust needs to adapt to become relevant in GNU/Linux distributions.
Kevin Kofler
9:01 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
Matthew Miller wrote:
> I fundamentally disagree with Kevin on a deep level about "entirely
> useless", but ... find myself kind of agreeing about the
"unpackagable"
> part. I mean: clearly we've found a way, but I'm really not sure we're
> providing a lot of _value_ in this approach, and I'm also not sure it's as
> successful as it could be.
We cannot ship what we cannot package. (Just repackaging upstream blobs is a
no-go, it is explicitly banned by Fedora Packaging Guidelines for good
reasons.)
> I'll start with the second: our system for Rust doesn't really do that.
> Developers are going to use cargo and crates.io and we're not going to
> convince them that they should do otherwise. (I don't expect anyone
> disagrees with this.)
That is one of the issues (along with lack of proper shared library support)
that makes Rust such a painful language: it makes it way too easy for
developers to just add a dependency on some random unpackaged (by
distributions) library with some random license and with random transitive
dependencies.
In a reasonable programming language, you have to think twice before you add
yet another dependency to your project. Tools like cargo make it way too
easy, just add a line and rebuild. So then you end up with a mess like this:
> But running `cargo fetch` with a clean cache pulls down *390* crates.
390 dependencies is absolutely insane! No project should ever depend on so
many libraries. This is completely unmaintainable.
> *This is what open source winning looks like.*
>
> I remember a Byte magazine article from the 1990 (I just checked!) with
> the title "There Is a Silver Bullet: The birth of interchangeable,
> reusable software components will bring software into the information
> age". [1] This was about the newly-hot idea of Object Oriented
> Programming. It was very exciting. But, of course, that vision of the
> world did not happen. It turns out proprietary software *can't* do this.
>
> But now we have it! I don't have to reinvent every basic wheel — but even
> more than that, I do not have to be an expert in the intricacies of safe
> concurrency to write an app that uses that under the hood. That's amazing!
> I can do such powerful things from high-level interfaces and trust the
> expertise of those who really understand the deep computer science some of
> this requires.
Code reuse is great when it makes sense, but this "NPM culture" where
developers happily depend on tons of packages containing trivial one-line
functions is completely insane.
If this is now the essence of "Open Source", then it will ultimately lose.
Proprietary software companies will just be waiting for the next paradigm
shift to lock us into their proprietary ecosystem again. For a large part,
this has already happened with smartphones: you get to choose only between
the Android walled garden (with an open core and lots of proprietary Google
services and (Google and third-party) apps on top) and the iOS walled garden
(which is completely proprietary and closed, even strictly enforcing an App
Store monopoly). We are struggling to escape from that with devices like the
PinePhone. But with development practices relying on an unmaintainable
dependency hell, it will never happen.
> I am competent enough to write a silly toy game using Bevy. It might be
> good enough that others will enjoy it. *I am not competent to maintain
> many of these dependencies.* I don't even know what most of them DO.
> "anyhow"? "bytemuck"?
Welcome to dependency hell!
> Worse, many of the Bevy deps are specified with exact versions. Maybe I
> could make the package work with the packaged versions, but ... that
> requires deep expertise and even then might lead to unexpected behavior
> and has a high chance of putting me at odds with both the engine upstream
> and any other games which use it. The packaging guidelines say that I
> SHOULD create patches to update to latest versions of dependencies, and
> that I should further convince the upstream to take them. Candidly, that
> seems like a waste of everone's time.
Hardcoding exact versions of dependencies is one of the worst misfeatures of
those language-specific build tools. It effectively prevents you from fixing
security issues without patching every single application. And of course it
can cause version conflicts between different applications or even between
transitive dependencies within an application. (And, while the former ones
can typically be resolved by shipping both versions of the library, the
latter ones cannot, because you cannot usually link two versions of the same
library into the same program.)
> So, going back to Kevin's point: it _does_ feel like this is unpackagable.
> But that's because the barrier to participation seems too high.
No, the answer to upstream shipping an unpackageable mess cannot possibly be
to make it easier to smuggle unpackageable messes into Fedora!
> It's not because it's statically-linked binaries [2] can't or
shouldn't
> exist in Fedora — that's not one of our core principles!
It is actually one of our Packaging Guidelines that static linking is
discouraged:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-stat...
as is bundling libraries:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling
(So they can indeed exist, but they definitely SHOULD NOT, the Guidelines
undisputably say so.)
> In fact and quite to the contrary, we need to adapt to handle this amazing
> open source success story better.
I fail to see a success story here. I see a failure story. Dependency hell
is not a success.
> And, I led with: I appreciate all the work you've all done to make this
> work. That's definitely true — I think it was super-valuable to pilot this
> approach. But I think that the Rust ecosystem would be a great place to
> pilot a different way. Something lightweight where we cache crates and use
> them _directly_ in the build process for _application_ RPMs.
Fabio has already explained very well why that is a horrible idea.
> Fedora _needs_ to adapt to stay relevant in the world where every language
> stack has developed a packaging ecosystem which effectively ignores us.
> Some of them are missing lessons they could have learned, ah well — but
> they also have a lot of nice new ideas we're missing. And, no matter what
> we think, we're clearly not going to stop them.
Rust needs to adapt to become relevant in GNU/Linux distributions.
There is nobody pushing for Rust to improve anymore. When Igor and I
were building out Fedora Rust, we did some engagement with Rust
upstream about stabilizing Rust's ABI so we could ship dynamic
libraries. While one or two members of the Rust core team were
sympathetic, most of the Rust community attacked me for "trying to
make Rust like C" and I got crap from people on the Rust community
channels, Twitter, and other places. Eventually, I flamed out because
there's only so much punishment someone can take over it.
A couple of years ago, there was a revival of the topic[1], but it
went nowhere again.
Until the situation changes, I'm very firmly anti-Rust. Unfortunately,
sometimes I have no choice but to deal with it.
[1]: https://github.com/slightknack/rust-abi-wiki
--
真実はいつも一つ！/ Always, there's only one truth!
11:57 a.m.
On 20. 10. 22 12:11, Fabio Valentini wrote:
> Which of the following will happen:
>
> 1) rpm will gain ExclusiveArch: %{rust_arches}
> 2) we will stop requiring the above in Rust packages, as Rust is 100% available
> 3) rpm will %ifarch %{rust_arches} this change
> 4) something else (what?)
>
> IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
> to build rpms. 3) seems somewhat tedious for no good reason.
I think 2) would be the easiest solution in the long term. rustc and
LLVM now already support all targets that we*might* want to add in
the near-to-mid-future (riscv64gc-unknown-linux-gnu comes to mind -
its support is at the same level as other targets we already have).
I've opened https://pagure.io/packaging-committee/issue/1220
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
noon
On 20. 10. 22 14:26, Panu Matilainen wrote:
>
> Which of the following will happen:
>
> 1) rpm will gain ExclusiveArch: %{rust_arches}
> 2) we will stop requiring the above in Rust packages, as Rust is 100% available
> 3) rpm will %ifarch %{rust_arches} this change
> 4) something else (what?)
>
> IMHO if we do 1) we could as well do 2) because without rpm, we won't be able
> to build rpms. 3) seems somewhat tedious for no good reason.
I was under the impression Rust was available for all architectures (for Fedora
anyway), no? There's no Rust code in rpm now either this didn't even cross my
mind really :D
Technically, I guess the right thing to do is 1) when Sequoia is enabled. Ie:
%if %{with sequoia}
%global crypto sequoia
BuildRequires: rpm-sequoia-devel >= 1.0.0
ExclusiveArch: %{rust_archves}
%else
%global crypto openssl
BuildRequires: openssl-devel
%endif
This is already in rawhide, except for the ExclusiveArch thing.
That said, the non-sequoia options should be considered only a bootstrap aid,
we're not going to provide security support for the internal parser for some
fringe architectures only.
I'm not sure that answered your questions though.
Partially.
My main point is: If we don't support rust-free (stainless?) RPM, what is the
point in mandating the use of ExclusiveArch: %{rust_arches} in our packages,
when it is by definition always contain the architecture this is build on?
Hence, I proposed to the FPC that we should no longer make the use of
ExclusiveArch: %{rust_arches} mandatory:
https://pagure.io/packaging-committee/issue/1220
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
11:27 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
To Neal's point, I had the audacity to suggest some improvements along the
lines of docutils and the response was underwhelming.
https://users.rust-lang.org/t/rust-analog-to-the-python-compilers-docutil...
On Thu, Oct 20, 2022 at 10:02 AM Neal Gompa <ngompa13(a)gmail.com> wrote:
On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
>
> Matthew Miller wrote:
> > I fundamentally disagree with Kevin on a deep level about "entirely
> > useless", but ... find myself kind of agreeing about the
"unpackagable"
> > part. I mean: clearly we've found a way, but I'm really not sure
we're
> > providing a lot of _value_ in this approach, and I'm also not sure
it's as
> > successful as it could be.
>
> We cannot ship what we cannot package. (Just repackaging upstream blobs
is a
> no-go, it is explicitly banned by Fedora Packaging Guidelines for good
> reasons.)
>
> > I'll start with the second: our system for Rust doesn't really do
that.
> > Developers are going to use cargo and crates.io and we're not going to
> > convince them that they should do otherwise. (I don't expect anyone
> > disagrees with this.)
>
> That is one of the issues (along with lack of proper shared library
support)
> that makes Rust such a painful language: it makes it way too easy for
> developers to just add a dependency on some random unpackaged (by
> distributions) library with some random license and with random
transitive
> dependencies.
>
> In a reasonable programming language, you have to think twice before you
add
> yet another dependency to your project. Tools like cargo make it way too
> easy, just add a line and rebuild. So then you end up with a mess like
this:
>
> > But running `cargo fetch` with a clean cache pulls down *390* crates.
>
> 390 dependencies is absolutely insane! No project should ever depend on
so
> many libraries. This is completely unmaintainable.
>
> > *This is what open source winning looks like.*
> >
> > I remember a Byte magazine article from the 1990 (I just checked!) with
> > the title "There Is a Silver Bullet: The birth of interchangeable,
> > reusable software components will bring software into the information
> > age". [1] This was about the newly-hot idea of Object Oriented
> > Programming. It was very exciting. But, of course, that vision of the
> > world did not happen. It turns out proprietary software *can't* do
this.
> >
> > But now we have it! I don't have to reinvent every basic wheel — but
even
> > more than that, I do not have to be an expert in the intricacies of
safe
> > concurrency to write an app that uses that under the hood. That's
amazing!
> > I can do such powerful things from high-level interfaces and trust the
> > expertise of those who really understand the deep computer science
some of
> > this requires.
>
> Code reuse is great when it makes sense, but this "NPM culture" where
> developers happily depend on tons of packages containing trivial one-line
> functions is completely insane.
>
> If this is now the essence of "Open Source", then it will ultimately
lose.
> Proprietary software companies will just be waiting for the next paradigm
> shift to lock us into their proprietary ecosystem again. For a large
part,
> this has already happened with smartphones: you get to choose only
between
> the Android walled garden (with an open core and lots of proprietary
Google
> services and (Google and third-party) apps on top) and the iOS walled
garden
> (which is completely proprietary and closed, even strictly enforcing an
App
> Store monopoly). We are struggling to escape from that with devices like
the
> PinePhone. But with development practices relying on an unmaintainable
> dependency hell, it will never happen.
>
> > I am competent enough to write a silly toy game using Bevy. It might be
> > good enough that others will enjoy it. *I am not competent to maintain
> > many of these dependencies.* I don't even know what most of them DO.
> > "anyhow"? "bytemuck"?
>
> Welcome to dependency hell!
>
> > Worse, many of the Bevy deps are specified with exact versions. Maybe I
> > could make the package work with the packaged versions, but ... that
> > requires deep expertise and even then might lead to unexpected behavior
> > and has a high chance of putting me at odds with both the engine
upstream
> > and any other games which use it. The packaging guidelines say that I
> > SHOULD create patches to update to latest versions of dependencies, and
> > that I should further convince the upstream to take them. Candidly,
that
> > seems like a waste of everone's time.
>
> Hardcoding exact versions of dependencies is one of the worst
misfeatures of
> those language-specific build tools. It effectively prevents you from
fixing
> security issues without patching every single application. And of course
it
> can cause version conflicts between different applications or even
between
> transitive dependencies within an application. (And, while the former
ones
> can typically be resolved by shipping both versions of the library, the
> latter ones cannot, because you cannot usually link two versions of the
same
> library into the same program.)
>
> > So, going back to Kevin's point: it _does_ feel like this is
unpackagable.
> > But that's because the barrier to participation seems too high.
>
> No, the answer to upstream shipping an unpackageable mess cannot
possibly be
> to make it easier to smuggle unpackageable messes into Fedora!
>
> > It's not because it's statically-linked binaries [2] can't or
shouldn't
> > exist in Fedora — that's not one of our core principles!
>
> It is actually one of our Packaging Guidelines that static linking is
> discouraged:
>
https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-stat...
> as is bundling libraries:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling
>
> (So they can indeed exist, but they definitely SHOULD NOT, the Guidelines
> undisputably say so.)
>
> > In fact and quite to the contrary, we need to adapt to handle this
amazing
> > open source success story better.
>
> I fail to see a success story here. I see a failure story. Dependency
hell
> is not a success.
>
> > And, I led with: I appreciate all the work you've all done to make this
> > work. That's definitely true — I think it was super-valuable to pilot
this
> > approach. But I think that the Rust ecosystem would be a great place to
> > pilot a different way. Something lightweight where we cache crates and
use
> > them _directly_ in the build process for _application_ RPMs.
>
> Fabio has already explained very well why that is a horrible idea.
>
> > Fedora _needs_ to adapt to stay relevant in the world where every
language
> > stack has developed a packaging ecosystem which effectively ignores us.
> > Some of them are missing lessons they could have learned, ah well — but
> > they also have a lot of nice new ideas we're missing. And, no matter
what
> > we think, we're clearly not going to stop them.
>
> Rust needs to adapt to become relevant in GNU/Linux distributions.
>
There is nobody pushing for Rust to improve anymore. When Igor and I
were building out Fedora Rust, we did some engagement with Rust
upstream about stabilizing Rust's ABI so we could ship dynamic
libraries. While one or two members of the Rust core team were
sympathetic, most of the Rust community attacked me for "trying to
make Rust like C" and I got crap from people on the Rust community
channels, Twitter, and other places. Eventually, I flamed out because
there's only so much punishment someone can take over it.
A couple of years ago, there was a revival of the topic[1], but it
went nowhere again.
Until the situation changes, I'm very firmly anti-Rust. Unfortunately,
sometimes I have no choice but to deal with it.
[1]: https://github.com/slightknack/rust-abi-wiki
--
真実はいつも一つ！/ Always, there's only one truth!
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
--
LinkedIn <https://www.linkedin.com/in/blaisepabon/> | Quora
<https://www.quora.com/profile/Blaise-Pabon> | Github
<https://github.com/blaisep>
“If you want to go fast, go alone. If you want to go far, go
together.” --African
proverb
3:12 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On 10/20/22 10:01, Neal Gompa wrote:
On Thu, Oct 20, 2022 at 9:39 AM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org> wrote:
> Rust needs to adapt to become relevant in GNU/Linux distributions.
>
There is nobody pushing for Rust to improve anymore. When Igor and I
were building out Fedora Rust, we did some engagement with Rust
upstream about stabilizing Rust's ABI so we could ship dynamic
libraries. While one or two members of the Rust core team were
sympathetic, most of the Rust community attacked me for "trying to
make Rust like C" and I got crap from people on the Rust community
channels, Twitter, and other places. Eventually, I flamed out because
there's only so much punishment someone can take over it.
Personal attacks are not okay.
A couple of years ago, there was a revival of the topic[1], but it
went nowhere again.
Until the situation changes, I'm very firmly anti-Rust. Unfortunately,
sometimes I have no choice but to deal with it.
[1]: https://github.com/slightknack/rust-abi-wiki
There are a couple major constraints that apply to Rust:
1. Rust implements generics via monomorphisation. This means that e.g. Vec *has no ABI
at all*: code will only be generated for Vec<T> when it is instantiated.
2. Rust relies heavily on cross-crate inlining to get good performance. If e.g.
Option::map is not inlined into the caller, performance will be terrible.
Neither of those constraints are unique to Rust. C++ template libraries have exactly
the same problems. It’s just that we think of C++ headers as being the interface to a
library, instead of the implementation. Trying to ship the implementation of types
like Vec and Option in a shared library makes no more sense than doing so with C++’s
std::vector and std::optional, which is to say, none at all.
Another factor is maintaining a stable ABI severely limits library evolution. In C++,
ABI stability has resulted in std::regex being *far* slower than third-party regular
expression libraries. I am not surprised that Rust wants to avoid similar problems.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
5:03 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
On Wed, Oct 19, 2022 at 11:25 AM Matthew Miller
<mattdm(a)fedoraproject.org> wrote:
>
> I _very much_ appreciate all the work you and the other Rust SIG folks
> (Igor and Zbyszek in particular but I'm sure others as well!) have put into
[… and Fabio in particular]
> I'll start with the second: our system for Rust doesn't
really do that.
> Developers are going to use cargo and crates.io and we're not going to
> convince them that they should do otherwise. (I don't expect anyone
> disagrees with this.)
This is true, and probably also not "fixable".
Also, arguably it's not something we'd like to "fix" at all.
I think not-using-the-rpm-packaged-software-during-development is
something that is equally true for Rust, C, Python, Java, and pretty
much any language. As long as Rust is not exclusive to Fedora/CentOS/RHEL
family, rpms are just going to be one of the possible ways to get rust
code.
For me, the goal is to make the automatic conversion from rust crates
to rpms as smooth as possible. I want the rpm package database to be
the CDN we use to get verified Rust code.
> *This is what open source winning looks like.*
[Snip various details here. I think Fabio's points are very good.]
> We could also attach other metadata to the packages in the
cache. Maybe some
> popularity, update frequency from Cargo.io, but also package review flags:
> checked license against source, and whatever other auditing we think should
> be done. This moves the focus from specfile-correctness to the package
> itself, and the effort from packaging to reviewing. (I'd suggest that for
> the experiment, we not make any deep auditing manditory, but instead
> encouraged.) And these flags should be able to be added by anyone in the
> Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
> Fedora _needs_ to adapt to stay relevant in the world where every language
> stack has developed a packaging ecosystem which effectively ignores us. Some
> of them are missing lessons they could have learned, ah well — but they also
> have a lot of nice new ideas we're missing. And, no matter what we think,
> we're clearly not going to stop them.
>
> Rust packaging seems like a great place to lead the way — and then we can
> maybe expand to Go, which has similar issues, and then Java (where, you
> know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
Given that, I think we're actually in kind of a *good* situation with
Rust packaging, especially compared to other language ecosystems - not
only right now, but also looking at the future. And looking at the
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
Yep. I expect some power-law distribution of package popularity. It
seems like there's an endless supply of crates, but most just don't
matter. If we get some threshold fraction of the most popular libaries
packaged, that should cover all or almost all dependencies of new leaf packages.
We're still hitting cases where there's a lot of stuff to package,
but as Fabio describes, those cases are not that common.
> Maybe we could get involved in Cargo Vet [4] — we could be both
a consumer
> _and_ a data source.
I like the idea. It'd be great to get for example a CI check for
package updates, and going the other way, use our review, QA, and
karma process as a source of trust. But this can (and IMO should) be
done in addition to our packaging, not as a replacement.
Zbyszek
3:23 p.m.
Hi, maybe it was already answered.
Not long ago Thunderbird switched from using installed GPG to its own
implementation inside. I think I have found the library part and it
seems to be in C++, which should be much more easier to integrate than
rust libraries.
I think the project link is:
https://github.com/rnpgp/rnp
Wouldn't it solve the problems cause in more compatible way? Is has
relatively recent release, so it does not seem abandoned. Is there a
specific reason, why is a Rust implementation chosen instead? I like
Rust language, but its integration into a core system component does not
seem easy.
Cheers,
Petr
On 20. 10. 22 11:03, Miro Hrončok wrote:
On 10. 10. 22 16:32, Ben Cotton wrote:
> For the last 20 years or so, RPM has used a home-grown OpenPGP parser
> for dealing with keys and signatures. That parser is rather infamous
> for its limitations and flaws, and especially in recent years has
> proven a significant burden to RPM development. In order to improve
> security and free developer resources for dealing with RPM's "core
> business" instead, RPM upstream is in the process of deprecating the
> internal parser in favor of [https://sequoia-pgp.org/ Sequoia PGP]
> based solution written in Rust.
> At this point the change is mostly invisible in normal daily use.
Which of the following will happen:
1) rpm will gain ExclusiveArch: %{rust_arches}
2) we will stop requiring the above in Rust packages, as Rust is 100%
available
3) rpm will %ifarch %{rust_arches} this change
4) something else (what?)
IMHO if we do 1) we could as well do 2) because without rpm, we won't
be able to build rpms. 3) seems somewhat tedious for no good reason.
--
Petr Menšík
Software Engineer, RHEL
Red Hat, http://www.redhat.com/
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB
12:36 a.m.
On 10/24/22 23:23, Petr Menšík wrote:
Hi, maybe it was already answered.
Not long ago Thunderbird switched from using installed GPG to its own
implementation inside. I think I have found the library part and it
seems to be in C++, which should be much more easier to integrate than
rust libraries.
I think the project link is:
https://github.com/rnpgp/rnp
Wouldn't it solve the problems cause in more compatible way? Is has
relatively recent release, so it does not seem abandoned. Is there a
specific reason, why is a Rust implementation chosen instead?
Yes it was already answered, see
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
and
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
We would've, *of course*, gone for something C-nativeish if that had
been an option at all. As I said in some other post in this thread, I've
been on the lookout for a viable C-native option for 15+ years. Yet here
we are.
And as I've also said elsewhere in this thread, the plan is to keep the
options open for the future. I don't like the shotgun marriage to Rust
any more than the next person out there.
I like Rust language, but its integration into a core system
component does not seem easy.
Except for the matter of bootstrap dependencies (which has also been
discussed here already), I don't know what the difficulty in this case
is supposed to be.
- Panu
9:40 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
I'll respond inline.
Me too -- and apologies for the delay.
> I fundamentally disagree with Kevin on a deep level about
"entirely
> useless", but ... find myself kind of agreeing about the
"unpackagable"
> part. I mean: clearly we've found a way, but I'm really not sure we're
> providing a lot of _value_ in this approach, and I'm also not sure it's
> as successful as it could be.
We *do* provide value to both users *and* developers by doing things
the way we do, but the benefits might not be obvious to people who
don't know how (Rust) packaging works, and what we as package
maintainers do.
Let me rephrase: I absolutely think you've provided value and are providing
value (and I appreciate it). I am not convinced that the value is in the
RPM-izing part, though.
[...]
This is due to a limitation of how cargo handles target-specific
dependencies - all dependencies that are *mentioned in any way* need
to be *present* for it to resolve dependencies / enabled optional
features / update its lockfile etc. But since we don't want to package
bindings for Windows and mac OS system APIs, we need to actually patch
them out, otherwise builds will fail.
Theoretically, if we had our own crate repository, we could either make
those changes there (possibly using something like packit to carry the
patches) -- or, just, not make the changes and not worry because we know
those won't end up used anyway?
You must realize that this is an extreme case. For many Rust
applications that people want to package for Fedora, the number of
dependencies that are missing is rather small, *because* most popular
libraries are already packaged.
It may be that I just hear about the difficult cases.
We might need to reconsider how to package projects like this.
I'm
pretty sure we could find a way to package them in a way that's
compatible with how we're currently doing things but would be much
less busywork.
Okay, I'm open to that.
Sure, but isn't that the case for most projects that a newcomer
wants
to package, regardless of programming language? Say, somebody wants to
package some cool new Python project for machine learning, then
there's probably also some linear algebra package or SIMD math library
in the dependency tree that's missing from Fedora. How is that
different?
Rust tends to be more fine-grained. I don't think this is necessarily
rust-specific _really_ — I think it's a trend as people get more used to
this way of doing things. With Python, there are some big packages
(including "batteries included" standard Python itself) which tend to group
big related sets of functionality. (notably: numpy, scipy, pandas...)
For intra-project dependencies (i.e. bevy components depending on
exact versions of bevy components), this is kind of expected, and we
have tools to deal with this kind of situation (though bevy is on a
different scale). For dependencies on third-party libraries, this is
kind of unexpected, and I wonder why they do things like that? Locking
some dependencies to exact versions is usually handled by relying on
the lockfile, instead.
I was wrong about this. I actually didn't realize that the ^ was optional. I
was, um, cargo-culting that around. Ah well. Anyway, that's less of a
problem than I worried.
> The packaging guidelines say that I SHOULD create patches to
update to
> latest versions of dependencies, and that I should further convince the
> upstream to take them. Candidly, that seems like a waste of everone's
> time.
This is *not* a waste of time. If we don't invest time to do that, many
project's dependencies grow stale, and actually *increase* the need for us
to maintain compat packages.
I have not tried this with any Rust package. My experience in the past is
that many upstreams find this the kind of thing that makes them go on long
blog rants about distro packaging -- they picked a version, it works fine,
they don't need the distraction of being told they must update it.
But even when this doesn't happen, it gets into the matter of expertise. If
I need to update a dependency for a newer-version of the sub-dependency, and
I don't know enough about either code base to do anything other than file a
"please update" bug, then everything is blocked on that.
I don't dispute that helping projects keep up to the latest is valuable
work. It even seems like it might be in-scope work for Fedora. But couldn't
we do that as something _separate_ from blocking ourselves (either literally
or through the extra overhead of compat packages) from packaging the
dependent app?
> The guidelines provide for creating compat packages, but that
means 1) the
> existing shared work is less useful, 2) requires even more extra steps, and
> 3) even without reviews for compat has extra administrative overhead.
We only maintain compat packages where porting to the new version (and
submitting the changes upstream) is not feasible. Again, isn't that
how Fedora is supposed to work?
I guess it depends on how broadly one reads "feasible". :)
The barrier for participation is too high in some cases, I agree.
However, in my experience, that's for a different reason:
The "shiny new things that happen to be written in Rust" that new
contributors want to have in Fedora are often very complicated
projects that even experienced Rust packagers would need to spend a
lot of time on.
Examples of that might be:
- wasmtime: I ultimately abandoned the attempt to package it "because
Fedora Legal", but the packages themselves worked fine
An aside, but: did I miss something with this on the Legal list? The only
thing I'm finding is a question about how to phrase `Apache-2.0 WITH
LLVM-exception`.
- deno: requires dozens of new packages, some of which also have
unclear / questionable licenses as well, but the packages themselves
worked
I'm not sure this example isn't agreeing with me. :)
On the other hand, many "nice" CLI tools that people want
to package
often require minimal knowledge of Rust packaging (our tools are
pretty nice for "standard" projects), and often only need very few new
dependencies to be packaged.
Just as an example, I just today started reviewing a "simple" Rust
application here:
https://bugzilla.redhat.com/show_bug.cgi?id=1990713
The spec file is very simple and almost entirely automatically
generated (with the exception of the missing License breakdown for the
statically linked binary), no dependencies were missing from Fedora.
Even Rust newbies would not have trouble packaging this, and that
would be a way better entry point than packaging stuff like Bevy.
How many lines of that are unique to that package.
I guess my impression of this is a little big colored by a Python packaging
adventure from a little bit ago:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
... where it kind of turns out that what looks like it could be automated
ends up with a hand-tuned specfile with a lot of exceptions.
I'm hopeful that maybe the Rust version of this could be more streamlined.
What if, instead of the specfile + boilerplate, there was a toml (or yaml,
or whatever) file that just listed whatever is unique to the package?
For this package, maybe it is... nothing? Just need an indicator that this
is a Rust package.
I know it looks simple to an experienced packager, but that specfile has a
_lot_ of complicated domain knowledge -- both general Fedora RPM packaging
and the rust packaging macros.
> And, I led with: I appreciate all the work you've all done
to make this
> work. That's definitely true — I think it was super-valuable to pilot this
> approach. But I think that the Rust ecosystem would be a great place to
> pilot a different way. Something lightweight where we cache crates and use
> them _directly_ in the build process for _application_ RPMs.
We have talked about this multiple times, but it won't work.
I think this was tried with first-class maven artifact support in
koji, but we all know how the Java packaging fiasco ended.
I would rather see it as: we learned some lessons from that approach and can
do it better.
Or even if making Rust crates first-class deliverables *did work*,
it
wouldn't give us the benefits of the current approach:
- we ensure that all crates in Fedora *build* on all architectures
- we ensure that most crates in Fedora pass their test suites on all
architectures
But those things aren't attached to making them into RPMs.
- we check all crates for objectionable content, licensing problems,
etc.
Nor is this. And I don't think we _should_ skip this part. It is clear
value.
- we change build flags to default to dynamically linking to system
libraries instead of statically linking against vendored copies
This too.
Mostly, at least. Assuming this isn't _prebuilt binaries_ or similar,
upstream may or may not have a good reason or strong opinion. Like any
bundling, we need a system which can track and react to security problems
with those libraries, though. (And we don't meaningfully have that for RPMs
now either.)
This would mean that we basically stop contributing things to the
upstream Rust ecosystem:
- we diagnose / report / fix architecture support issues
- we port projects to new versions of dependencies
- etc.
Why? I think it would give us _more_ time to do those things.
I see this work in the upstream ecosystem as an important part of
the
work we do in packaging Rust crates for Fedora,
and I would not want to endorse an approach that meant we no longer do
these things.
Sure!
> Rust packages include a lot of machine-readable metadata. We
should be
> able to watch for CVEs, RustSec, and other security notices even without
> encoding the metadata in RPMs. License review could also be automated —
> the field in Cargo.toml is supposed to be SPDX, so that's convenient.
> [3]
I already monitor RustSec advisories and check *all of them* against
Fedora packages. This takes up a miniscule amount of the time I spend
on Rust packaging (because there's so few Rust security advisories).
If I remember correctly, there were only 2-3 CVE issues in the Rust
stack that actually affected our packages, and dealing with those was
very simple:
1) Push the patched version of the library, 2) rebuild dependent
applications, 3) submit to bodhi.
There's some amount of automation that *could* be done (mostly in
figuring out which applications need to be rebuilt for a given library
change), but that's also pretty easily done with a "dnf repoquery" or
two.
I appreciate that you do this (very much!). But it seems like this could be
_entirely_ automated (with alerts for dependency or license changes, etc).
On the other hand, license review is still important, even if
it's
already available in SPDX format in the upstream metadata.
Just because sometimes, that metadata is either wrong or incomplete.
And even more often, package review flags other problems (like missing
LICENSE files for licenses that *require* redistributed sources to
contain a copy of the license text). Relying on SPDX metadata alone is
*not* safe.
Again, yes -- and we should talk to Jilanye and others on the Legal list,
because we can do this better generally. For most packages, there's a
one-time review gate, applied with various diligence depending on the
packager and reviewer. Then, maybe never looked at again. For packages that
go into RHEL, there's another review by RH that _hopefully_ (and by policy!)
should go back to the Fedora packages.
> We could also attach other metadata to the packages in the cache. Maybe some
> popularity, update frequency from Cargo.io, but also package review flags:
> checked license against source, and whatever other auditing we think should
> be done. This moves the focus from specfile-correctness to the package
> itself, and the effort from packaging to reviewing. (I'd suggest that for
> the experiment, we not make any deep auditing manditory, but instead
> encouraged.) And these flags should be able to be added by anyone in the
> Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
See my comment above -- there are a lot of steps and a l
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
But our mechanism is really complicated -- a barrier to entry, lots of
places for mistakes, and even with collaboration with other distros, very
Fedora-specific. So I think there is something to gain.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
I really hope we can look at these and learn how to do it better, instead of
deciding that better isn't possible. And — while I'm not really up on node —
I have pretty good hindsight on what went wrong with modularity. (Not enough
to try modularity _again_ just yet... but that's a different thing. A whole
talk for next year's Nest/Flock, maybe....)
> Rust packaging seems like a great place to lead the way — and
then we can
> maybe expand to Go, which has similar issues, and then Java (where, you
> know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
I think we can both be right, here: the simple mapping seems like it makes
it good to experiment with.
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
I'll take "for now". :)
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
1:06 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On 11/1/22 10:40, Matthew Miller wrote:
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
> I'll respond inline.
Me too -- and apologies for the delay.
>> I fundamentally disagree with Kevin on a deep level about "entirely
>> useless", but ... find myself kind of agreeing about the
"unpackagable"
>> part. I mean: clearly we've found a way, but I'm really not sure
we're
>> providing a lot of _value_ in this approach, and I'm also not sure it's
>> as successful as it could be.
> We *do* provide value to both users *and* developers by doing things
> the way we do, but the benefits might not be obvious to people who
> don't know how (Rust) packaging works, and what we as package
> maintainers do.
Let me rephrase: I absolutely think you've provided value and are providing
value (and I appreciate it). I am not convinced that the value is in the
RPM-izing part, though.
Please do not stop packaging Rust crates. Qubes OS at least will be relying
on these packages to build its own Rust code in the not-too-distant future.
[...]
> This is due to a limitation of how cargo handles target-specific
> dependencies - all dependencies that are *mentioned in any way* need
> to be *present* for it to resolve dependencies / enabled optional
> features / update its lockfile etc. But since we don't want to package
> bindings for Windows and mac OS system APIs, we need to actually patch
> them out, otherwise builds will fail.
Theoretically, if we had our own crate repository, we could either make
those changes there (possibly using something like packit to carry the
patches) -- or, just, not make the changes and not worry because we know
those won't end up used anyway?
> You must realize that this is an extreme case. For many Rust
> applications that people want to package for Fedora, the number of
> dependencies that are missing is rather small, *because* most popular
> libraries are already packaged.
It may be that I just hear about the difficult cases.
Probably so.
> Sure, but isn't that the case for most projects that a
newcomer wants
> to package, regardless of programming language? Say, somebody wants to
> package some cool new Python project for machine learning, then
> there's probably also some linear algebra package or SIMD math library
> in the dependency tree that's missing from Fedora. How is that
> different?
Rust tends to be more fine-grained. I don't think this is necessarily
rust-specific _really_ — I think it's a trend as people get more used to
this way of doing things. With Python, there are some big packages
(including "batteries included" standard Python itself) which tend to group
big related sets of functionality. (notably: numpy, scipy, pandas...)
There is another factor that I think needs to be mentioned here:
In Rust, a crate is not only the unit of packaging, but also of
compilation. A project may well have several crates purely for
internal code organization or to reduce build time. Some of these
crates may never be intended for use outside of the larger project.
In this case, bundling them is absolutely the correct decision, as
nobody else should ever be depending on them. However, bundling is
*not* the right decision if a crate *is* intended to have outside
users.
> For intra-project dependencies (i.e. bevy components depending
on
> exact versions of bevy components), this is kind of expected, and we
> have tools to deal with this kind of situation (though bevy is on a
> different scale). For dependencies on third-party libraries, this is
> kind of unexpected, and I wonder why they do things like that? Locking
> some dependencies to exact versions is usually handled by relying on
> the lockfile, instead.
I was wrong about this. I actually didn't realize that the ^ was optional. I
was, um, cargo-culting that around. Ah well. Anyway, that's less of a
problem than I worried.
Will the bevy components ever be used outside of bevy? If not,
then they should be bundled. if so, they should not be bundled.
>> The packaging guidelines say that I SHOULD create patches to
update to
>> latest versions of dependencies, and that I should further convince the
>> upstream to take them. Candidly, that seems like a waste of everone's
>> time.
> This is *not* a waste of time. If we don't invest time to do that, many
> project's dependencies grow stale, and actually *increase* the need for us
> to maintain compat packages.
I have not tried this with any Rust package. My experience in the past is
that many upstreams find this the kind of thing that makes them go on long
blog rants about distro packaging -- they picked a version, it works fine,
they don't need the distraction of being told they must update it.
I heavily doubt this will be an issue with Rust. There is a reason
that dependabot is so popular.
But even when this doesn't happen, it gets into the matter of
expertise. If
I need to update a dependency for a newer-version of the sub-dependency, and
I don't know enough about either code base to do anything other than file a
"please update" bug, then everything is blocked on that.
Fair. Obviously, a patch that actually updates them to the latest
version would be awesome, but a bunch of bug reports seem to be more
of a job for a bot.
I don't dispute that helping projects keep up to the latest is
valuable
work. It even seems like it might be in-scope work for Fedora. But couldn't
we do that as something _separate_ from blocking ourselves (either literally
or through the extra overhead of compat packages) from packaging the
dependent app?
I don’t think it should be a blocker unless e.g. the out-of-date
dependency has a serious security vulnerability.
--
Sincerely,
Demi Marie Obenour (she/her/hers)
1:24 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Tue, Nov 1, 2022 at 3:40 PM Matthew Miller <mattdm(a)fedoraproject.org> wrote:
On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
> I'll respond inline.
Me too -- and apologies for the delay.
> > I fundamentally disagree with Kevin on a deep level about "entirely
> > useless", but ... find myself kind of agreeing about the
"unpackagable"
> > part. I mean: clearly we've found a way, but I'm really not sure
we're
> > providing a lot of _value_ in this approach, and I'm also not sure
it's
> > as successful as it could be.
> We *do* provide value to both users *and* developers by doing things
> the way we do, but the benefits might not be obvious to people who
> don't know how (Rust) packaging works, and what we as package
> maintainers do.
Let me rephrase: I absolutely think you've provided value and are providing
value (and I appreciate it). I am not convinced that the value is in the
RPM-izing part, though.
[...]
> This is due to a limitation of how cargo handles target-specific
> dependencies - all dependencies that are *mentioned in any way* need
> to be *present* for it to resolve dependencies / enabled optional
> features / update its lockfile etc. But since we don't want to package
> bindings for Windows and mac OS system APIs, we need to actually patch
> them out, otherwise builds will fail.
Theoretically, if we had our own crate repository, we could either make
those changes there (possibly using something like packit to carry the
patches) -- or, just, not make the changes and not worry because we know
those won't end up used anyway?
That won't do. It would mean that we end up having to review and keep
track of *more* crates instead of fewer.
> You must realize that this is an extreme case. For many Rust
> applications that people want to package for Fedora, the number of
> dependencies that are missing is rather small, *because* most popular
> libraries are already packaged.
It may be that I just hear about the difficult cases.
Of course. People who manager to package their favourite Rust app
without problems don't come to complain, do they?
> We might need to reconsider how to package projects like this.
I'm
> pretty sure we could find a way to package them in a way that's
> compatible with how we're currently doing things but would be much
> less busywork.
Okay, I'm open to that.
I'll have more time to spend on this soon, I hope, so I'll try to come
back with a prototype ASAP.
> Sure, but isn't that the case for most projects that a
newcomer wants
> to package, regardless of programming language? Say, somebody wants to
> package some cool new Python project for machine learning, then
> there's probably also some linear algebra package or SIMD math library
> in the dependency tree that's missing from Fedora. How is that
> different?
Rust tends to be more fine-grained. I don't think this is necessarily
rust-specific _really_ — I think it's a trend as people get more used to
this way of doing things. With Python, there are some big packages
(including "batteries included" standard Python itself) which tend to group
big related sets of functionality. (notably: numpy, scipy, pandas...)
It's probably a misconception that the Rust standard library is small.
It's not small, it's just not not very *broad*.
And there's some libraries that are also quite "batteries included",
for example, hyper and tokio, which could be considered "standard" at
this point.
(snip)
I have not tried this with any Rust package. My experience in the
past is
that many upstreams find this the kind of thing that makes them go on long
blog rants about distro packaging -- they picked a version, it works fine,
they don't need the distraction of being told they must update it.
But even when this doesn't happen, it gets into the matter of expertise. If
I need to update a dependency for a newer-version of the sub-dependency, and
I don't know enough about either code base to do anything other than file a
"please update" bug, then everything is blocked on that.
It's not, though. This is what SIGs are for.
I explicitly tell people who are new to Rust packaging that I'll help
them get their dependencies reviewed and keep them up-to-date so they
can focus on their own stuff.
That some people don't accept help when it is offered is a different problem ...
I don't dispute that helping projects keep up to the latest is
valuable
work. It even seems like it might be in-scope work for Fedora. But couldn't
we do that as something _separate_ from blocking ourselves (either literally
or through the extra overhead of compat packages) from packaging the
dependent app?
> > The guidelines provide for creating compat packages, but that means 1) the
> > existing shared work is less useful, 2) requires even more extra steps, and
> > 3) even without reviews for compat has extra administrative overhead.
>
> We only maintain compat packages where porting to the new version (and
> submitting the changes upstream) is not feasible. Again, isn't that
> how Fedora is supposed to work?
I guess it depends on how broadly one reads "feasible". :)
Many "ports" to new versions require only changing names of structs /
methods, import paths, or even only bumping the dependency.
I'd say that's pretty easy, particularly if upstream projects publish
good Release notes.
> The barrier for participation is too high in some cases, I
agree.
> However, in my experience, that's for a different reason:
>
> The "shiny new things that happen to be written in Rust" that new
> contributors want to have in Fedora are often very complicated
> projects that even experienced Rust packagers would need to spend a
> lot of time on.
>
> Examples of that might be:
> - wasmtime: I ultimately abandoned the attempt to package it "because
> Fedora Legal", but the packages themselves worked fine
An aside, but: did I miss something with this on the Legal list? The only
thing I'm finding is a question about how to phrase `Apache-2.0 WITH
LLVM-exception`.
You might remember the fedora-council ticket tagged "OMG Urgent!!!1":
https://pagure.io/Fedora-Council/tickets/issue/408
It was closed, but I still haven't received any actionable information
from Fedora Legal to this day.
> - deno: requires dozens of new packages, some of which also
have
> unclear / questionable licenses as well, but the packages themselves
> worked
I'm not sure this example isn't agreeing with me. :)
How so? Even if packaged from bundled dependencies, the unclear status
of some components would have blocked the package from being added to
Fedora.
> On the other hand, many "nice" CLI tools that people
want to package
> often require minimal knowledge of Rust packaging (our tools are
> pretty nice for "standard" projects), and often only need very few new
> dependencies to be packaged.
>
> Just as an example, I just today started reviewing a "simple" Rust
> application here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1990713
>
> The spec file is very simple and almost entirely automatically
> generated (with the exception of the missing License breakdown for the
> statically linked binary), no dependencies were missing from Fedora.
> Even Rust newbies would not have trouble packaging this, and that
> would be a way better entry point than packaging stuff like Bevy.
How many lines of that are unique to that package.
Few. Most of it is boilerplate.
You don't *need* to understand what's automatically generated, you
only need to know what you're doing if you manually change things.
That's the whole point of having a spec file generator.
I guess my impression of this is a little big colored by a Python
packaging
adventure from a little bit ago:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedorapr...
... where it kind of turns out that what looks like it could be automated
ends up with a hand-tuned specfile with a lot of exceptions.
I'm hopeful that maybe the Rust version of this could be more streamlined.
It already is.
What if, instead of the specfile + boilerplate, there was a toml (or
yaml,
or whatever) file that just listed whatever is unique to the package?
For this package, maybe it is... nothing? Just need an indicator that this
is a Rust package.
That's basically already the case for many packages.
We have rust2rpm.conf files committed into dist-git which control some
aspects of spec file generation, making many manual edits unnecessary.
I know it looks simple to an experienced packager, but that specfile
has a
_lot_ of complicated domain knowledge -- both general Fedora RPM packaging
and the rust packaging macros.
And you don't really need to understand most of these things, unless
the upstream project is doing something weird, and you need to adapt
the packaging to that.
> > And, I led with: I appreciate all the work you've all
done to make this
> > work. That's definitely true — I think it was super-valuable to pilot this
> > approach. But I think that the Rust ecosystem would be a great place to
> > pilot a different way. Something lightweight where we cache crates and use
> > them _directly_ in the build process for _application_ RPMs.
>
> We have talked about this multiple times, but it won't work.
> I think this was tried with first-class maven artifact support in
> koji, but we all know how the Java packaging fiasco ended.
I would rather see it as: we learned some lessons from that approach and can
do it better.
So would Red Hat be willing to invest more resources into CPE to
implement this for Fedora infra?
> Or even if making Rust crates first-class deliverables *did
work*, it
> wouldn't give us the benefits of the current approach:
> - we ensure that all crates in Fedora *build* on all architectures
> - we ensure that most crates in Fedora pass their test suites on all
> architectures
But those things aren't attached to making them into RPMs.
Even if they're not - this is how they are already implemented, and
they already work.
Changing things to a completely different paradigm would require a
massive amount of work just to get back to the state *we already
have*.
I won't waste my time on that, sorry.
> - we check all crates for objectionable content, licensing
problems, etc.
Nor is this. And I don't think we _should_ skip this part. It is clear
value.
> - we change build flags to default to dynamically linking to system
> libraries instead of statically linking against vendored copies
This too.
Mostly, at least. Assuming this isn't _prebuilt binaries_ or similar,
upstream may or may not have a good reason or strong opinion. Like any
bundling, we need a system which can track and react to security problems
with those libraries, though. (And we don't meaningfully have that for RPMs
now either.)
There are no prebuilt binaries in the Rust stack. And there's only
very few cases where we don't dynamically link against system
libraries, but statically link bundled libraries. libgit2 is one of
them, where the package in Fedora is chronically under-maintained and
out-of-date (and hence too old for what we need in the Rust stack).
> This would mean that we basically stop contributing things to
the
> upstream Rust ecosystem:
> - we diagnose / report / fix architecture support issues
> - we port projects to new versions of dependencies
> - etc.
Why? I think it would give us _more_ time to do those things.
Would it? So you suggest that in addition to working out a new
delivery mechanism for Rust crates, we *also* devise a new way to
build and test them?
> I see this work in the upstream ecosystem as an important part
of the
> work we do in packaging Rust crates for Fedora,
> and I would not want to endorse an approach that meant we no longer do
> these things.
Sure!
> > Rust packages include a lot of machine-readable metadata. We should be
> > able to watch for CVEs, RustSec, and other security notices even without
> > encoding the metadata in RPMs. License review could also be automated —
> > the field in Cargo.toml is supposed to be SPDX, so that's convenient.
> > [3]
>
> I already monitor RustSec advisories and check *all of them* against
> Fedora packages. This takes up a miniscule amount of the time I spend
> on Rust packaging (because there's so few Rust security advisories).
> If I remember correctly, there were only 2-3 CVE issues in the Rust
> stack that actually affected our packages, and dealing with those was
> very simple:
> 1) Push the patched version of the library, 2) rebuild dependent
> applications, 3) submit to bodhi.
> There's some amount of automation that *could* be done (mostly in
> figuring out which applications need to be rebuilt for a given library
> change), but that's also pretty easily done with a "dnf repoquery" or
> two.
I appreciate that you do this (very much!). But it seems like this could be
_entirely_ automated (with alerts for dependency or license changes, etc).
It could be, but it's not high priority for me, just because it's such
a small amount of time I spend on that compared to other things.
(snip)
> If manual changes *are* required, then these changes would also
be
> required in the "first-class crate artifact" scenario, so you don't
> gain anything.
> And if there's other problems that are caught during package review,
> the distribution mechanism doesn't matter, either.
But our mechanism is really complicated -- a barrier to entry, lots of
places for mistakes, and even with collaboration with other distros, very
Fedora-specific. So I think there is something to gain.
No. :)
> In my experience, changing the distribution mechanism or
packaging
> paradigm will often make things *worse* instead of better. For
> example, the implosion of the NodeJS package ecosystem in Fedora was
> not only caused by the horrid state NPM, but also because the new
> packaging guidelines which prefer bundling essentially made it
> impossible for packagers to verify that objectionable content is
> present in vendored dependencies. For Java, Modularity was seen as a
> "solution", but the result was that basically everybody - except for
> the Red Hat maintainers who maintained the modules - just stopped
> doing Java packaging because of the hostile environment.
I really hope we can look at these and learn how to do it better, instead of
deciding that better isn't possible. And — while I'm not really up on node —
I have pretty good hindsight on what went wrong with modularity. (Not enough
to try modularity _again_ just yet... but that's a different thing. A whole
talk for next year's Nest/Flock, maybe....)
Please, no. It's time to stop beating that dead horse ...
Fabio
1:30 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
Hi all,
Just a note that over the summer, our intern did a project to try and
address some of these issues (namely, that while it's trivial to convert
a single crate to an RPM, trying to automate packaging all the
dependencies and making sure that you don't break anything else while
doing so is tedious and error-prone). Matt and Fabio might recall me
getting their inputs on this several months ago.
The tool:
- recursively checking out packages
- updating existing packages while carrying over manual changes
- creating compatibility packages when upgrading packages with dependents
- automate parallel COPR test builds of sets of packages in dependency order
- chain-build all packages in Koji with requested side tag
- merging and chain-building all packages across release branches
- help review a rust update in Bodhi and verify no compatibility issues are introduced
It does not automatically commit anything, of course, and right now is a
bit opinionated in favor of picking the lowest satisfactory version
possible, which is probably not what we want long term -- but it
hopefully provides a nice foundation on which to build on
I've finally gotten round to doing some polishing and getting it
packaged:
- updates for Fedora 36, 37, and Rawhide:
https://bodhi.fedoraproject.org/updates/?search=rust-update-set-0.0.1&...
- Pagure repo: https://pagure.io/fedora-rust/rust-update-set
There are some fixes for corner cases I encountered while trying to update our
`below` packages with this, and some changes needed to get this packageable,
but those are minor. It mostly works really well, and is at a stage where
hopefully people can take a look and make suggestions for improvements.
Best regards,
--
Michel Alexandre Salim
identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2
1:32 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Tue, Nov 1, 2022 at 7:07 PM Demi Marie Obenour <demiobenour(a)gmail.com> wrote:
On 11/1/22 10:40, Matthew Miller wrote:
> On Wed, Oct 19, 2022 at 01:04:39PM +0200, Fabio Valentini wrote:
>> For intra-project dependencies (i.e. bevy components depending on
>> exact versions of bevy components), this is kind of expected, and we
>> have tools to deal with this kind of situation (though bevy is on a
>> different scale). For dependencies on third-party libraries, this is
>> kind of unexpected, and I wonder why they do things like that? Locking
>> some dependencies to exact versions is usually handled by relying on
>> the lockfile, instead.
>
> I was wrong about this. I actually didn't realize that the ^ was optional. I
> was, um, cargo-culting that around. Ah well. Anyway, that's less of a
> problem than I worried.
Will the bevy components ever be used outside of bevy? If not,
then they should be bundled. if so, they should not be bundled.
Yeah, this is something that I hope to be able to improve soon.
>>> The packaging guidelines say that I SHOULD create
patches to update to
>>> latest versions of dependencies, and that I should further convince the
>>> upstream to take them. Candidly, that seems like a waste of everone's
>>> time.
>> This is *not* a waste of time. If we don't invest time to do that, many
>> project's dependencies grow stale, and actually *increase* the need for us
>> to maintain compat packages.
>
> I have not tried this with any Rust package. My experience in the past is
> that many upstreams find this the kind of thing that makes them go on long
> blog rants about distro packaging -- they picked a version, it works fine,
> they don't need the distraction of being told they must update it.
I heavily doubt this will be an issue with Rust. There is a reason
that dependabot is so popular.
Exactly. I forgot to reply to this in my previous post, but many Rust
projects on GitHub use dependabot (or similar bots) to automatically
bump their dependencies to the latest version. The projects that
*don'*t do that are the odd ones out, but still, PRs submitted to
these projects are usually received positively.
(snip)
> I don't dispute that helping projects keep up to the latest
is valuable
> work. It even seems like it might be in-scope work for Fedora. But couldn't
> we do that as something _separate_ from blocking ourselves (either literally
> or through the extra overhead of compat packages) from packaging the
> dependent app?
I don’t think it should be a blocker unless e.g. the out-of-date
dependency has a serious security vulnerability.
Yeah, I tend to be rather pragmatic in this regard.
Patch and submit to upstream if it's easy, keep a compat for the older
version around if it's not.
The only exception to this rule I recently made was to retire some
outdated versions of HTTP libraries, which were only used by obsolete
packages anyway.
Fabio
1:38 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
Matthew Miller wrote:
Rust tends to be more fine-grained. I don't think this is
necessarily
rust-specific _really_ — I think it's a trend as people get more used to
this way of doing things.
And this is inherently a PITA to package, unfortunately.
It is indeed not Rust-specific, other new fancy languages are a similar
mess, see Node.js (with NPM), Go, etc. Another reason why GNU/Linux
developers should stick to C and/or C++.
That said, both KDE and GNOME have also gone through a phase of splitting
craze whose consequences we are still suffering from: The software from the
KDE project used to be a dozen packages that could be updated and built by
one person by hand in a day. Now we have hundreds of packages released on 3
different release cycles (4 if you include the new Plasma Mobile Gear) that
need scripts to update, take days to build even with scripts, and lead to
updates on whose sheer number of packages Bodhi and OpenQA frequently choke.
And this goes also and especially for the libraries: kdelibs used to be one
package, now there are dozens of kf5-* packages, likewise Qt. And it is no
better in GNOME land, they started splitting stuff even before KDE did.
But the situation is worse in all those "modern" languages like Rust or Go.
(And also that language that calls itself "modern" even though it has been
designed decades ago to allow for pointlessly jumping text on websites. NPM
is infamous for containing "packages" with a single one-line function.)
I have not tried this with any Rust package. My experience in the
past is
that many upstreams find this the kind of thing that makes them go on long
blog rants about distro packaging -- they picked a version, it works fine,
they don't need the distraction of being told they must update it.
That unhelpfulness (and complete lack of understanding of and for how
distributions work) by upstreams is a big issue in those parallel
ecosystems.
But even when this doesn't happen, it gets into the matter of
expertise.
If I need to update a dependency for a newer-version of the
sub-dependency, and I don't know enough about either code base to do
anything other than file a "please update" bug, then everything is blocked
on that.
Normally, it should just be a matter of changing the version number. If that
fails to build, IMHO, the dependency (the library) is broken. Library
upstream "maintainers" with a complete disregard for backwards compatibility
are a PITA, no matter in what language.
> We only maintain compat packages where porting to the new version
(and
> submitting the changes upstream) is not feasible. Again, isn't that
> how Fedora is supposed to work?
I guess it depends on how broadly one reads "feasible". :)
We normally try pretty aggressively to port packages to new library versions
where the incompatibilities are not too bad. I do not see why it should be
any different when the library happens to be written in Rust.
> Examples of that might be:
> - wasmtime: I ultimately abandoned the attempt to package it "because
> Fedora Legal", but the packages themselves worked fine
An aside, but: did I miss something with this on the Legal list? The only
thing I'm finding is a question about how to phrase `Apache-2.0 WITH
LLVM-exception`.
See: https://bugzilla.redhat.com/show_bug.cgi?id=2130953
> We have talked about this multiple times, but it won't work.
> I think this was tried with first-class maven artifact support in
> koji, but we all know how the Java packaging fiasco ended.
I would rather see it as: we learned some lessons from that approach and
can do it better.
Without a concrete proposal on how you want to "do it better", there is
really nothing to discuss, because the only thing that we can talk about as
is is the approach that we know did not work. So, suggest a new approach and
we can analyze whether it has any chance of working any better or not.
My guess is that any working approach to allow foreign artifact types in
Koji, and also reliably deliver them to users (including ones that want to
build or rebuild software), would ultimately be more work than just using
RPMs.
> - we change build flags to default to dynamically linking to
system
> libraries instead of statically linking against vendored copies
This too.
Mostly, at least. Assuming this isn't _prebuilt binaries_ or similar,
upstream may or may not have a good reason or strong opinion.
Why would we care about a "strong opinion"? Either there is a good reason or
there is not. Irrational demands by unreasonable, uncooperative upstreams
ought to be just ignored. Free Software means we can adapt the software to
our needs. If upstream will not allow that, it is not Free Software.
I really hope we can look at these and learn how to do it better,
instead
of deciding that better isn't possible. And — while I'm not really up on
node — I have pretty good hindsight on what went wrong with modularity.
(Not enough to try modularity _again_ just yet... but that's a different
thing. A whole talk for next year's Nest/Flock, maybe....)
Why would trying modularity ever again even be up for discussion? Can we not
finally stop beating that dead horse?
Modularity failed exactly because of the fundamental design issue I had
warned about from day one: Modules can contain non-leaf packages, other
modules can depend on a those and typically depend on a specific version,
and those specific versions cannot coexist on the same system. That
necessarily leads to conflicts. And in the non-modular world, we already
have ways to solve exactly those conflicts (e.g., compat packages).
> alternatives, all attempts at trying different approaches (maven
> artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
> have *failed* and ultimately made things worse instead of improving
> the situation - the only thing that has proven to be sustainable (for
> now) is ... maybe surprisingly, plain RPM packages.
I'll take "for now". :)
I do not expect that to change, ever.
Kevin Kofler
2:50 a.m.
Hi Simo,
On Fri, 14 Oct 2022 22:36:09 +0200,
Neal H. Walfield wrote:
On Fri, 14 Oct 2022 18:28:01 +0200,
Simo Sorce wrote:
> At this time, as far as I know, there is no OpenPGP work of any kind on
> supporting PQC algorithms.
The German BSI contracted MTG AG to design and implement PQC for
OpenPGP. They presented their work at IETF 113, and at the OpenPGP
email summit this past May.
https://datatracker.ietf.org/meeting/113/materials/agenda-113-openpgp-01
The OpenPGP session at IETF 115 (Tuesday Nov 8th, 1300-1430 UTC) will
include a discussion on PQC in OpenPGP:
https://datatracker.ietf.org/doc/agenda-115-openpgp/
That will be followed by a side meeting dedicated to PQC in OpenPGP:
https://mailarchive.ietf.org/arch/msg/openpgp/ZTPCQQ13OookbjmSNeyptR4ItcM/
I hope you'll be able to attend and raise your concerns.
Neal
10:21 a.m.
New subject: [Rust] Re: musings on rust packaging [was Re: F38 proposal: RPM
Sequoia (System-Wide Change proposal)]
On Tue, Nov 01, 2022 at 01:30:01PM -0500, Michel Alexandre Salim wrote:
I've finally gotten round to doing some polishing and getting it
packaged:
- updates for Fedora 36, 37, and Rawhide:
https://bodhi.fedoraproject.org/updates/?search=rust-update-set-0.0.1&...
- Pagure repo: https://pagure.io/fedora-rust/rust-update-set
There are some fixes for corner cases I encountered while trying to update our
`below` packages with this, and some changes needed to get this packageable,
but those are minor. It mostly works really well, and is at a stage where
hopefully people can take a look and make suggestions for improvements.
Cool -- glad to see this!
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
5:54 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
Hi Fabio,
Been meaning to reply to this, but it got lost in the mail pile.
> I _very much_ appreciate all the work you and the other Rust SIG
folks
> (Igor and Zbyszek in particular but I'm sure others as well!) have put into
> packaging rust apps and crates and all of the systems around that.
I'll respond inline.
Same.
> I fundamentally disagree with Kevin on a deep level about
"entirely
> useless", but ... find myself kind of agreeing about the
"unpackagable"
> part. I mean: clearly we've found a way, but I'm really not sure we're
> providing a lot of _value_ in this approach, and I'm also not sure it's as
> successful as it could be.
We *do* provide value to both users *and* developers by doing things
the way we do, but the benefits might not be obvious to people who
don't know how (Rust) packaging works, and what we as package
maintainers do.
As a rust packager I initially agreed but I am now having doubts here.
> There are three ways having things packaged in Fedora repos
_can_ be
> helpful:
>
> 1. End-user applications and tools
> 2. Useful development environment
> 3. As convenience for ourselves for building packages for #1 or #2
>
> I am not discounting the value of #3 -- making a shared thing that we all
> work on together is kind of the whole point, and the nicer we can make that
> the better we can bring in more people, and those of us already here have a
> lighter load and can work on the things we're most interested in. But
> ultimately, we're doing it so we make a useful system for users. That means
> the first two.
This I can agree with :)
> I'll start with the second: our system for Rust doesn't really do that.
> Developers are going to use cargo and crates.io and we're not going to
> convince them that they should do otherwise. (I don't expect anyone
> disagrees with this.)
This is true, and probably also not "fixable". We need to make some
amount of non-upstreamable patches to some crates (most notably,
removing Windows- or mac OS-specific dependencies, because we don't
want to package those), but in some cases, these are "incompatible"
changes, and Rust *developers* should not be targeting our downstream
sources that have these differences with actual upstream sources.
Yet you say above "We *do* provide value to both users *and*
developers" yet you say developers shouldn't be targeting that work?
This is due to a limitation of how cargo handles target-specific
dependencies - all dependencies that are *mentioned in any way* need
to be *present* for it to resolve dependencies / enabled optional
features / update its lockfile etc. But since we don't want to package
bindings for Windows and mac OS system APIs, we need to actually patch
them out, otherwise builds will fail.
And that ends up being quite a bit of work from my point of view. Also
the way the packaging works with options things like devel or optional
features ends up being very painful. I will often drop out optional
features just so I can do less packaging.
> We're doing okay with #1, but... I think #3 _even_ with all
of the work in
> Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
> engine and will probably have a few things it would be nice to package to
> make available in Fedora Linux. I might not even mind maintaining Bevy
> itself.
Somebody actually already started packaging Bevy components - some
packages are already approved and some are still pending review. Not
sure what the progress has been there, but it's not *impossible*.
Well nothing is *impossible* if you have enough stamina, resources or
whatever else. I don't find saying something isn't *impossible*
necessarily makes it compelling.
> But running `cargo fetch` with a clean cache pulls down *390*
crates. Of
> these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
> which is *amazing*. But... that still is hundreds that I'd have to add. And
> mostly they are things I don't know _anything_ about.
You must realize that this is an extreme case. For many Rust
applications that people want to package for Fedora, the number of
dependencies that are missing is rather small, *because* most popular
libraries are already packaged.
In my experience, and I'm not packaging any form of gaming engine,
it's not an extreme case, for a few small things and one slightly
larger thing I *have* packaged I now maintain over 60 more packages. I
have another one I want to package and AFAICT I get to package another
50-100 but frankly it's hard to tell, and this is something I have
control over and have been actively trying to do upstream reduction of
deps.
And requests to make things like this easier have been open for over a year:
https://pagure.io/fedora-rust/rust2rpm/issue/140
Bevy is a bit special, because it (presumably) pulls in lots of GPU
/
OpenGL / Vulkan related libraries, which we didn't need to package for
anything else yet, and it's also split into dozens of small libraries
itself, which can be painful to package, that is true.
We might need to reconsider how to package projects like this. I'm
pretty sure we could find a way to package them in a way that's
compatible with how we're currently doing things but would be much
less busywork.
Well if we're finding a solution for those sorts of projects we should
be doing it for all rust projects. A lot of projects have internal
virtual/workspace bits where they split up things up internally but
may have a dozen or sub projects. These make absolutely no sense to
split up into separate packages because they're all bundled together,
have to be bumped together yet as it standard at the moment you either
have to package them all separately or do nasty rpm spec hacking to
try and glue it into a single rpm. Again RFEs to make less busy work
have sat for well over a year, some even 3 years, with no progress:
https://pagure.io/fedora-rust/rust2rpm/issue/141
> *This is what open source winning looks like.*
>
> I remember a Byte magazine article from the 1990 (I just checked!) with the
> title "There Is a Silver Bullet: The birth of interchangeable, reusable
> software components will bring software into the information age". [1]
> This was about the newly-hot idea of Object Oriented Programming. It was
> very exciting. But, of course, that vision of the world did not happen. It
> turns out proprietary software *can't* do this.
>
> But now we have it! I don't have to reinvent every basic wheel — but even
> more than that, I do not have to be an expert in the intricacies of safe
> concurrency to write an app that uses that under the hood. That's amazing! I
> can do such powerful things from high-level interfaces and trust the
> expertise of those who really understand the deep computer science some of
> this requires.
>
> I am competent enough to write a silly toy game using Bevy. It might be good
> enough that others will enjoy it. *I am not competent to maintain many of
> these dependencies.* I don't even know what most of them DO.
"anyhow"?
> "bytemuck"?
Sure, but isn't that the case for most projects that a newcomer wants
to package, regardless of programming language? Say, somebody wants to
package some cool new Python project for machine learning, then
there's probably also some linear algebra package or SIMD math library
in the dependency tree that's missing from Fedora. How is that
different?
I think the big difference here from my experience, and I've packaged
right across the Fedora spectrum, is that most of the python style
projects are able to use a pretty comprehensive standard library and
crypto library which helps minimise a lot of the extras I've seen in
the rust ecosystem.
> Worse, many of the Bevy deps are specified with exact versions.
Maybe I
> could make the package work with the packaged versions, but ... that
> requires deep expertise and even then might lead to unexpected behavior and
> has a high chance of putting me at odds with both the engine upstream and
> any other games which use it.
For intra-project dependencies (i.e. bevy components depending on
exact versions of bevy components), this is kind of expected, and we
have tools to deal with this kind of situation (though bevy is on a
different scale). For dependencies on third-party libraries, this is
kind of unexpected, and I wonder why they do things like that? Locking
some dependencies to exact versions is usually handled by relying on
the lockfile, instead.
Still, for large projects that are split into many crates like bevy, I
think we could find a way to make packaging them a lot easier (i.e.
not use one source package per crate, but use one source package for
building a project that contains many crates).
See my point above about workspaces.
> The packaging guidelines say that I SHOULD
> create patches to update to latest versions of dependencies, and that I
> should further convince the upstream to take them. Candidly, that seems like
> a waste of everone's time.
This is *not* a waste of time.
If we don't invest time to do that, many project's dependencies grow
stale, and actually *increase* the need for us to maintain compat
packages.
Well it depends on your opinion of a waste of time. The whole need for
packaging and maintaining compat packages is a waste of time and this
point just proves the whole problem with busyness. I've wasted so much
time dealing with other people's projects just so I can update my own
packages to achieve something like a minor version bump for a bug fix
or CVE..... time that I personally no longer have to spare, which for
me and the time I have *now* to spend on Fedora I feel is not well
spent, I would sooner be spending my spare time doing Arm enablement
or adding features, not just basic package bumps with provide *none*
of that, or in some cases I have to wait a bunch of time for someone
(generally yourself - and yes I very much appreciate the work you do)
else to do the bumps because it's large and I don't have the in depth
rust knowledge to handle 100s of package updates.
> The guidelines provide for creating compat packages, but that
means 1) the
> existing shared work is less useful, 2) requires even more extra steps, and
> 3) even without reviews for compat has extra administrative overhead.
We only maintain compat packages where porting to the new version (and
submitting the changes upstream) is not feasible. Again, isn't that
how Fedora is supposed to work?
To some degree, but when it blocks package updates or means someone
has to do 100s of PRs upstream to update one package it's not
something most people enjoy or have the time to do.
> So, going back to Kevin's point: it _does_ feel like this is
unpackagable.
> But that's because the barrier to participation seems too high. It's not
> because it's statically-linked binaries [2] can't or shouldn't exist in
> Fedora — that's not one of our core principles! In fact and quite to the
> contrary, we need to adapt to handle this amazing open source success story
> better.
The barrier for participation is too high in some cases, I agree.
However, in my experience, that's for a different reason:
In my experience it's not.
The "shiny new things that happen to be written in Rust"
that new
contributors want to have in Fedora are often very complicated
projects that even experienced Rust packagers would need to spend a
lot of time on.
Examples of that might be:
- wasmtime: I ultimately abandoned the attempt to package it "because
Fedora Legal", but the packages themselves worked fine
- deno: requires dozens of new packages, some of which also have
unclear / questionable licenses as well, but the packages themselves
worked
On the other hand, many "nice" CLI tools that people want to package
often require minimal knowledge of Rust packaging (our tools are
pretty nice for "standard" projects), and often only need very few new
dependencies to be packaged.
The tools are nice, but it doesn't mean that it's still not a
magnitude of work to get it done. Having packaged parsec [1] and
updated it a few times it's honestly not something I'm prepared to do
again, and even then there are some features I've actively disabled
because of the amount of extra packaging work required for those.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1859994
Just as an example, I just today started reviewing a
"simple" Rust
application here:
https://bugzilla.redhat.com/show_bug.cgi?id=1990713
The spec file is very simple and almost entirely automatically
generated (with the exception of the missing License breakdown for the
statically linked binary), no dependencies were missing from Fedora.
Even Rust newbies would not have trouble packaging this, and that
would be a way better entry point than packaging stuff like Bevy.
> And, I led with: I appreciate all the work you've all done to make this
> work. That's definitely true — I think it was super-valuable to pilot this
> approach. But I think that the Rust ecosystem would be a great place to
> pilot a different way. Something lightweight where we cache crates and use
> them _directly_ in the build process for _application_ RPMs.
We have talked about this multiple times, but it won't work.
I think this was tried with first-class maven artifact support in
koji, but we all know how the Java packaging fiasco ended.
Or even if making Rust crates first-class deliverables *did work*, it
wouldn't give us the benefits of the current approach:
- we ensure that all crates in Fedora *build* on all architectures
Bundling would also ensure this too.
- we ensure that most crates in Fedora pass their test suites on all
architectures
Again I don't see how bundling stops this.
- we check all crates for objectionable content, licensing problems,
etc.
That's a requirement of packaging in Fedora anyway.
- we change build flags to default to dynamically linking to system
libraries instead of statically linking against vendored copies
I don't see what that value provides to end users.
This would mean that we basically stop contributing things to the
upstream Rust ecosystem:
- we diagnose / report / fix architecture support issues
I don't see how the bundling of things to build on all architectures,
vs the build it all as single packages changes this, if it fails to
build on Z-series either as a vendored bundle or not it would still be
discoverable and hence reportable/fixable by the Fedora community for
benefit of the wider community.
- we port projects to new versions of dependencies
That's valuable for other projects but not Fedora, and it's not
something I am going to do personally as a rust packager.
I see this work in the upstream ecosystem as an important part of
the
work we do in packaging Rust crates for Fedora,
and I would not want to endorse an approach that meant we no longer do
these things.
I don't actually see how a change stops this.
> Rust packages include a lot of machine-readable metadata. We
should be able
> to watch for CVEs, RustSec, and other security notices even without encoding
> the metadata in RPMs. License review could also be automated — the field in
> Cargo.toml is supposed to be SPDX, so that's convenient. [3]
I already monitor RustSec advisories and check *all of them* against
Fedora packages. This takes up a miniscule amount of the time I spend
on Rust packaging (because there's so few Rust security advisories).
So if there's so few sec advisories, probably because of the style of
rust programming, again I don't see why even for single packages this
answers the question/statement Matthew made.
If I remember correctly, there were only 2-3 CVE issues in the Rust
stack that actually affected our packages, and dealing with those was
very simple:
1) Push the patched version of the library, 2) rebuild dependent
applications, 3) submit to bodhi.
There's some amount of automation that *could* be done (mostly in
figuring out which applications need to be rebuilt for a given library
change), but that's also pretty easily done with a "dnf repoquery" or
two.
On the other hand, license review is still important, even if it's
already available in SPDX format in the upstream metadata.
Just because sometimes, that metadata is either wrong or incomplete.
And even more often, package review flags other problems (like missing
LICENSE files for licenses that *require* redistributed sources to
contain a copy of the license text). Relying on SPDX metadata alone is
*not* safe.
> We could also attach other metadata to the packages in the cache. Maybe some
> popularity, update frequency from Cargo.io, but also package review flags:
> checked license against source, and whatever other auditing we think should
> be done. This moves the focus from specfile-correctness to the package
> itself, and the effort from packaging to reviewing. (I'd suggest that for
> the experiment, we not make any deep auditing manditory, but instead
> encouraged.) And these flags should be able to be added by anyone in the
> Rust SIG, not necessarily just at import.
This is already the case, though?
Writing a spec file for a new crate is already automated to the point
where "standard" crates can be 100% automatically generated and need
zero manual edits.
If manual changes *are* required, then these changes would also be
required in the "first-class crate artifact" scenario, so you don't
gain anything.
And if there's other problems that are caught during package review,
the distribution mechanism doesn't matter, either.
I agree with most of this, but what is not automated is that I need to
submit and get reviewed dozens of packages vs just one and it's a
completely different work flow than what I need to do in c9s and RHEL.
In my experience, changing the distribution mechanism or packaging
paradigm will often make things *worse* instead of better. For
example, the implosion of the NodeJS package ecosystem in Fedora was
not only caused by the horrid state NPM, but also because the new
packaging guidelines which prefer bundling essentially made it
impossible for packagers to verify that objectionable content is
present in vendored dependencies. For Java, Modularity was seen as a
"solution", but the result was that basically everybody - except for
the Red Hat maintainers who maintained the modules - just stopped
doing Java packaging because of the hostile environment.
Frankly I don't think modularity solved any of those problems. I also
personally believed that most of the collapse is because RHEL doesn't
require separate packaging now and allows vendoring so there was
suddenly no need to package a million dependencies except for Fedora
so why bother? It saved a lot of enforced busyness and I can
completely see why that was done, because doing things like actual
feature development is more useful and the end users AKA customers
want actual features not perceived features like granular packaging.
> Maybe we could get involved in Cargo Vet [4] — we could be both
a consumer
> _and_ a data source.
>
> Fedora _needs_ to adapt to stay relevant in the world where every language
> stack has developed a packaging ecosystem which effectively ignores us. Some
> of them are missing lessons they could have learned, ah well — but they also
> have a lot of nice new ideas we're missing. And, no matter what we think,
> we're clearly not going to stop them.
>
> Rust packaging seems like a great place to lead the way — and then we can
> maybe expand to Go, which has similar issues, and then Java (where, you
> know, things have already collapsed despite heroic effort.)
Oh, actually, I don't think Rust packaging is a good place to start
here at all. :)
The way cargo works already maps very neatly onto how RPM packages
work, which is definitely *not* the case for other language
ecosystems. I also think we could even massively improve handling of
"large" projects with many sub-components (like bevy, zola, wasmtime,
deno, etc.) - which are currently the only projects that are "painful"
to package - *without* completely changing the underlying packaging
paradigm or distribution mechanism. (I've been wanting to actually
write better tooling for this use case, but alas, Bachelor thesis is
more important for now.)
Given that, I think we're actually in kind of a *good* situation with
Rust packaging, especially compared to other language ecosystems - not
only right now, but also looking at the future. And looking at the
alternatives, all attempts at trying different approaches (maven
artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
have *failed* and ultimately made things worse instead of improving
the situation - the only thing that has proven to be sustainable (for
now) is ... maybe surprisingly, plain RPM packages.
I don't think it's as bad as other ecosystems but I don't agree with
your assessment of "kind of a *good* situation", I feel there may be a
little bit of Stockholm syndrome coming into play here to be honest.
As someone who started rust packaging around 2.5 years ago I feel I
spent the vast majority of my time doing things that don't achieve
what I need to achieve from actually packaging those packages in the
first place but rather waste a lot of my time having to do lots and
lots of packaging and fixing things elsewhere so I can update in a lot
of cases is what is a single cli package AKA what you call a simple
rust package so I can get a bug fix or a new feature out to my users
and I don't think that is a good use of my time.
Peter
6:54 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 11:54:10AM +0000, Peter Robinson wrote:
Hi Fabio,
Been meaning to reply to this, but it got lost in the mail pile.
> > But running `cargo fetch` with a clean cache pulls down
*390* crates. Of
> > these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
> > which is *amazing*. But... that still is hundreds that I'd have to add.
And
> > mostly they are things I don't know _anything_ about.
>
> You must realize that this is an extreme case. For many Rust
> applications that people want to package for Fedora, the number of
> dependencies that are missing is rather small, *because* most popular
> libraries are already packaged.
In my experience, and I'm not packaging any form of gaming engine,
it's not an extreme case, for a few small things and one slightly
larger thing I *have* packaged I now maintain over 60 more packages. I
have another one I want to package and AFAICT I get to package another
50-100 but frankly it's hard to tell, and this is something I have
control over and have been actively trying to do upstream reduction of
deps.
The magnitude of the number of deps is the real killer for not
only Rust, but also Go, and arguably anything that isn't a C
based application.
Packaging up all of a project's deps individually is viable when
there are a relatively small number of them, especially if most
of the deps are shared by other apps, and the libraries have good
API + ABI stability promises. This is common in the case of most
C applications/libraries, since shared libraries are relatively
speaking fairly broad in functional scope, and often long lived,
because that is the mindset of the C ecosystem in general.
Modern languages & their ecosystems though have brought about
a world where the granularity of deps is waaaaay smaller, where
the focus is on being able to evolve rapidly, with less focus
on long term stable API/ABI, as apps are expected to just fix
against specific versions they've tested on.
So functionality that lives in single library in C, often ends
up being spread across 20-200 modules in any of the modern
non-C languages. We've been experiancing this problem in Fedora
for a very long time and while we have spec auto-generators,
that still leaves masses of busy work, and also still leaves
the problem that what we ship doesn't match what upstream has
tested, so the result is often less reliable.
IMHO the only thing that can make packaging such fine grained
modules sustainable long term, is if the process could be 100%
automated. I don't mean just having a tool to auto-generate
a spec file. It needs to be fully automated to the extent that
you just tell a robot "i need deps for this package" and it
does everything automatically, except for human approval of
the results of a code license audit.
That of course would be a non-trivial service to build, and
it still begs the question of whether its really beneficial
in the long term.
> Sure, but isn't that the case for most projects that a
newcomer wants
> to package, regardless of programming language? Say, somebody wants to
> package some cool new Python project for machine learning, then
> there's probably also some linear algebra package or SIMD math library
> in the dependency tree that's missing from Fedora. How is that
> different?
I think the big difference here from my experience, and I've packaged
right across the Fedora spectrum, is that most of the python style
projects are able to use a pretty comprehensive standard library and
crypto library which helps minimise a lot of the extras I've seen in
the rust ecosystem.
Python isn't that different from the Rust world to be honest.
For a short while we had OpenStack in Fedora, but it wasn't
sustainable to maintain its large set of python module deps,
fighting between the versions Fedora already had, vs the
versions that OpenStack actually wanted (and was tested
against by upstream. It was never ending busy-work for the
people trying to keep it working in Fedora, taking time
away from more beneficial work, and delivering a system that
was often broken because module version combinations we used
were not tested.
> - we port projects to new versions of dependencies
That's valuable for other projects but not Fedora, and it's not
something I am going to do personally as a rust packager.
It isn't sustainable in the general case, as it requires a level
of expertize / knowledge that most packagers can't be assumed
to have. If they have that knowledge and the time to invest in
this, they can do it directly in upstream, regardless of what
Fedora packaging approach is used.
I don't think it's as bad as other ecosystems but I don't
agree with
your assessment of "kind of a *good* situation", I feel there may be a
little bit of Stockholm syndrome coming into play here to be honest.
I feel like Fedora is successfull in packaging huge numbers of
deps across the various language ecosystems, not because it is
easy, but often as a result of a relatively small number of people
in the respect SIGs investing a heroic amount of effort to keep
everything ticking along. That isn't long term sustainable and
leaves us vulnerable to maintainers no longer being able or willing
to invest their time in future, which risks huge chains of packages
being orphaned and falling out of the distro.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:15 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 12:54 PM Peter Robinson <pbrobinson(a)gmail.com> wrote:
> This is true, and probably also not "fixable". We need to make some
> amount of non-upstreamable patches to some crates (most notably,
> removing Windows- or mac OS-specific dependencies, because we don't
> want to package those), but in some cases, these are "incompatible"
> changes, and Rust *developers* should not be targeting our downstream
> sources that have these differences with actual upstream sources.
Yet you say above "We *do* provide value to both users *and*
developers" yet you say developers shouldn't be targeting that work?
We provide value to developers by basically running a huge free CI
across a wide range of architectures.
That doesn't mean that they should develop against crate sources we
have in Fedora, similarly to how upstream Python developers probably
should use pip/venv instead of "whatver is in Fedora". The same
applies to basically every language ecosystem except C/C++, where the
package manager story is just so bad that system libraries are
actually the only reliable development environment.
> This is due to a limitation of how cargo handles
target-specific
> dependencies - all dependencies that are *mentioned in any way* need
> to be *present* for it to resolve dependencies / enabled optional
> features / update its lockfile etc. But since we don't want to package
> bindings for Windows and mac OS system APIs, we need to actually patch
> them out, otherwise builds will fail.
And that ends up being quite a bit of work from my point of view. Also
the way the packaging works with options things like devel or optional
features ends up being very painful. I will often drop out optional
features just so I can do less packaging.
Recent versions of rust2rpm automatically generate a patch to remove
non-linux dependencies, so that work has effectively been reduced to
zero.
Disabling unused optional dependencies can also be done with either a
rust2rpm config option or with a simple patch, so that should not be a
problem, either. I also don't understand why disabling some optional
features is such a bad thing? I don't think we have a policy in Fedora
that says we *must* enable *all possible functionality and optional
features*.
> > We're doing okay with #1, but... I think #3 _even_ with
all of the work in
> > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy game
> > engine and will probably have a few things it would be nice to package to
> > make available in Fedora Linux. I might not even mind maintaining Bevy
> > itself.
>
> Somebody actually already started packaging Bevy components - some
> packages are already approved and some are still pending review. Not
> sure what the progress has been there, but it's not *impossible*.
Well nothing is *impossible* if you have enough stamina, resources or
whatever else. I don't find saying something isn't *impossible*
necessarily makes it compelling.
I agree. And I think we can do better, as I said in my original post.
> > But running `cargo fetch` with a clean cache pulls down
*390* crates. Of
> > these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
> > which is *amazing*. But... that still is hundreds that I'd have to add.
And
> > mostly they are things I don't know _anything_ about.
>
> You must realize that this is an extreme case. For many Rust
> applications that people want to package for Fedora, the number of
> dependencies that are missing is rather small, *because* most popular
> libraries are already packaged.
In my experience, and I'm not packaging any form of gaming engine,
it's not an extreme case, for a few small things and one slightly
larger thing I *have* packaged I now maintain over 60 more packages. I
have another one I want to package and AFAICT I get to package another
50-100 but frankly it's hard to tell, and this is something I have
control over and have been actively trying to do upstream reduction of
deps.
As far as I know, salimma's intern worked on a tool to determine
missing dependencies for Rust projects in Fedora recursively, which
should make it easier to determine the exact set of things that you
would need to do here.
And requests to make things like this easier have been open for over
a year:
https://pagure.io/fedora-rust/rust2rpm/issue/140
Right. I'm sorry about the issue reports that have been open for a long time.
rust2rpm / rust-packaging is an entirely community-run project. We
don't get any support from Red Hat (or other places) for anything
except the Rust compiler itself. And with the original developer of
rust2rpm (Igor) being gone, nobody who understands the low-level stuff
in there is still around.
zbyszek and I are trying our best to keep things running, but at this
point, we'll need basically a rewrite of both rust2rpm and the RPM
dependency / provides generators to support new cargo features. And
since I don't have much time for that right now (busy finishing my
university degree), these things are still pending.
> Bevy is a bit special, because it (presumably) pulls in lots of
GPU /
> OpenGL / Vulkan related libraries, which we didn't need to package for
> anything else yet, and it's also split into dozens of small libraries
> itself, which can be painful to package, that is true.
>
> We might need to reconsider how to package projects like this. I'm
> pretty sure we could find a way to package them in a way that's
> compatible with how we're currently doing things but would be much
> less busywork.
Well if we're finding a solution for those sorts of projects we should
be doing it for all rust projects. A lot of projects have internal
virtual/workspace bits where they split up things up internally but
may have a dozen or sub projects. These make absolutely no sense to
split up into separate packages because they're all bundled together,
have to be bumped together yet as it standard at the moment you either
have to package them all separately or do nasty rpm spec hacking to
try and glue it into a single rpm. Again RFEs to make less busy work
have sat for well over a year, some even 3 years, with no progress:
https://pagure.io/fedora-rust/rust2rpm/issue/141
Same here. I *would like to work on this and improve the situation for
all of us (including me)*, but I just don't have the time to do that
right now.
> Sure, but isn't that the case for most projects that a
newcomer wants
> to package, regardless of programming language? Say, somebody wants to
> package some cool new Python project for machine learning, then
> there's probably also some linear algebra package or SIMD math library
> in the dependency tree that's missing from Fedora. How is that
> different?
I think the big difference here from my experience, and I've packaged
right across the Fedora spectrum, is that most of the python style
projects are able to use a pretty comprehensive standard library and
crypto library which helps minimise a lot of the extras I've seen in
the rust ecosystem.
The Rust standard library is intentionally narrow. I don't think that
will change any time soon, so we'll have to deal with it. Either way,
it looks like the ecosystem is starting to standardize around a few
common libraries (serde for de/serialization, tokio as async runtime,
hyper for HTTP clients + servers, etc.), so things might get less
sprawling in these areas.
> Still, for large projects that are split into many crates like
bevy, I
> think we could find a way to make packaging them a lot easier (i.e.
> not use one source package per crate, but use one source package for
> building a project that contains many crates).
See my point above about workspaces.
Glad that we're agreeing on something ;)
Still, we need somebody to actually do the work and implement these features.
Pity we don't know any big open-source-friendly companies that operate
in this space ...
> This is *not* a waste of time.
> If we don't invest time to do that, many project's dependencies grow
> stale, and actually *increase* the need for us to maintain compat
> packages.
Well it depends on your opinion of a waste of time. The whole need for
packaging and maintaining compat packages is a waste of time and this
point just proves the whole problem with busyness. I've wasted so much
I don't understand this point.
We *need* compat packages for certain cases, or we'd break people's
packages, simple as that.
I don't create compat packages if they're not needed?
time dealing with other people's projects just so I can update my
own
packages to achieve something like a minor version bump for a bug fix
or CVE..... time that I personally no longer have to spare, which for
me and the time I have *now* to spend on Fedora I feel is not well
spent, I would sooner be spending my spare time doing Arm enablement
or adding features, not just basic package bumps with provide *none*
of that, or in some cases I have to wait a bunch of time for someone
(generally yourself - and yes I very much appreciate the work you do)
else to do the bumps because it's large and I don't have the in depth
rust knowledge to handle 100s of package updates.
(snip)
> We only maintain compat packages where porting to the new
version (and
> submitting the changes upstream) is not feasible. Again, isn't that
> how Fedora is supposed to work?
To some degree, but when it blocks package updates or means someone
has to do 100s of PRs upstream to update one package it's not
something most people enjoy or have the time to do.
But that's exactly my point? Am I speaking in tongues?
If porting dependent packages to a new version of something is too
much work, then keeping the old one around with a compat package will
work just fine.
> Or even if making Rust crates first-class deliverables *did
work*, it
> wouldn't give us the benefits of the current approach:
> - we ensure that all crates in Fedora *build* on all architectures
Bundling would also ensure this too.
Ensuring that they *build*, yes. With the caveat that figuring out
where in your bundled dependencies a build error comes from is pretty
tedious, and patching bundled dependencies is a bit of a hassle.
> - we ensure that most crates in Fedora pass their test suites on
all
> architectures
Again I don't see how bundling stops this.
It just ... does? When using vendored dependencies, test suites of
bundled crates *aren't run* by cargo at all, and there's also no way
to do that, as far as I know.
> - we check all crates for objectionable content, licensing
problems, etc.
That's a requirement of packaging in Fedora anyway.
True, but with most projects that bundle stuff, people just seem to
assume things are "fine" and don't look too hard. I don't want to know
that horrors lurk in vendor tarballs for the packages that use this
approach ... (and this is part of the reason why NodeJS packaging in
Fedora died).
> - we change build flags to default to dynamically linking to
system
> libraries instead of statically linking against vendored copies
I don't see what that value provides to end users.
Do I really need to explain this point? I think linking against system
OpenSSL is *way better* than statically linking to a random vendored
copy of it.
> This would mean that we basically stop contributing things to
the
> upstream Rust ecosystem:
> - we diagnose / report / fix architecture support issues
I don't see how the bundling of things to build on all architectures,
vs the build it all as single packages changes this, if it fails to
build on Z-series either as a vendored bundle or not it would still be
discoverable and hence reportable/fixable by the Fedora community for
benefit of the wider community.
Errors like that bubble to the top eventually, but are much harder to
diagnose and fix in this environment than when you're dealing with a
single library in isolation.
> - we port projects to new versions of dependencies
That's valuable for other projects but not Fedora, and it's not
something I am going to do personally as a rust packager.
If being an exclusive consumer of Rust projects instead of
collaborating with upstream projects is your choice, then so be it.
But that's not usually the way things are supposed to work in Linux
distros, I think.
> I see this work in the upstream ecosystem as an important part
of the
> work we do in packaging Rust crates for Fedora,
> and I would not want to endorse an approach that meant we no longer do
> these things.
I don't actually see how a change stops this.
But it would? If you could just "cargo vendor" and yeet that
horror-tarball into dist-git, you just wouldn't know about vendored
copy 1.2.3 of library foo being vulnerable to CVE X or having bug
#1234, or just being outdated and no longer working on Linux 6 or
something ... Most of these things *only* happen if you deal with
libraries individually.
> > Rust packages include a lot of machine-readable metadata.
We should be able
> > to watch for CVEs, RustSec, and other security notices even without encoding
> > the metadata in RPMs. License review could also be automated — the field in
> > Cargo.toml is supposed to be SPDX, so that's convenient. [3]
>
> I already monitor RustSec advisories and check *all of them* against
> Fedora packages. This takes up a miniscule amount of the time I spend
> on Rust packaging (because there's so few Rust security advisories).
So if there's so few sec advisories, probably because of the style of
rust programming, again I don't see why even for single packages this
answers the question/statement Matthew made.
It's just not worth it?
I don't want to spend days of my time on implementing some automation
for a task that takes me literally a few minutes every few months?
> This is already the case, though?
> Writing a spec file for a new crate is already automated to the point
> where "standard" crates can be 100% automatically generated and need
> zero manual edits.
> If manual changes *are* required, then these changes would also be
> required in the "first-class crate artifact" scenario, so you don't
> gain anything.
> And if there's other problems that are caught during package review,
> the distribution mechanism doesn't matter, either.
I agree with most of this, but what is not automated is that I need to
submit and get reviewed dozens of packages vs just one and it's a
completely different work flow than what I need to do in c9s and RHEL.
The way Rust is packaged in RHEL is an abomination, and most likely
also not OK from a legal standpoint. That's the only thing I'll say to
this ...
> In my experience, changing the distribution mechanism or
packaging
> paradigm will often make things *worse* instead of better. For
> example, the implosion of the NodeJS package ecosystem in Fedora was
> not only caused by the horrid state NPM, but also because the new
> packaging guidelines which prefer bundling essentially made it
> impossible for packagers to verify that objectionable content is
> present in vendored dependencies. For Java, Modularity was seen as a
> "solution", but the result was that basically everybody - except for
> the Red Hat maintainers who maintained the modules - just stopped
> doing Java packaging because of the hostile environment.
Frankly I don't think modularity solved any of those problems. I also
personally believed that most of the collapse is because RHEL doesn't
require separate packaging now and allows vendoring so there was
suddenly no need to package a million dependencies except for Fedora
so why bother? It saved a lot of enforced busyness and I can
completely see why that was done, because doing things like actual
feature development is more useful and the end users AKA customers
want actual features not perceived features like granular packaging.
Frankly, I couldn't care less about RHEL, especially from a Rust
packaging point of view.
If they really cared about any of that, we'd get contributions to
rust2rpm or RPM packaging tools from RH. But we don't. They just do
their own franken-packaging by re-using some of the tools we developed
in a way that they weren't intended to be used.
> Given that, I think we're actually in kind of a *good*
situation with
> Rust packaging, especially compared to other language ecosystems - not
> only right now, but also looking at the future. And looking at the
> alternatives, all attempts at trying different approaches (maven
> artifacts in koji, vendoring NodeJS dependencies, Java Modules, etc.)
> have *failed* and ultimately made things worse instead of improving
> the situation - the only thing that has proven to be sustainable (for
> now) is ... maybe surprisingly, plain RPM packages.
I don't think it's as bad as other ecosystems but I don't agree with
your assessment of "kind of a *good* situation", I feel there may be a
little bit of Stockholm syndrome coming into play here to be honest.
Of course I'm invested in this stuff. I've been spending lots of my time on it.
Of course it's not perfect. There's much space for improvements.
The only thing we don't have is an abundance of people who work on it
to actually make it better.
As someone who started rust packaging around 2.5 years ago I feel I
spent the vast majority of my time doing things that don't achieve
what I need to achieve from actually packaging those packages in the
first place but rather waste a lot of my time having to do lots and
lots of packaging and fixing things elsewhere so I can update in a lot
of cases is what is a single cli package AKA what you call a simple
rust package so I can get a bug fix or a new feature out to my users
and I don't think that is a good use of my time.
Packages changing lots of their dependencies for a simple bugfix
release seems like really bad upstream practice, and I don't think
Rust is to blame here, but that's beside the point.
Fabio
7:28 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 8:16 AM Fabio Valentini <decathorpe(a)gmail.com> wrote:
On Wed, Nov 30, 2022 at 12:54 PM Peter Robinson <pbrobinson(a)gmail.com> wrote:
>
> > This is true, and probably also not "fixable". We need to make some
> > amount of non-upstreamable patches to some crates (most notably,
> > removing Windows- or mac OS-specific dependencies, because we don't
> > want to package those), but in some cases, these are "incompatible"
> > changes, and Rust *developers* should not be targeting our downstream
> > sources that have these differences with actual upstream sources.
>
> Yet you say above "We *do* provide value to both users *and*
> developers" yet you say developers shouldn't be targeting that work?
We provide value to developers by basically running a huge free CI
across a wide range of architectures.
That doesn't mean that they should develop against crate sources we
have in Fedora, similarly to how upstream Python developers probably
should use pip/venv instead of "whatver is in Fedora". The same
applies to basically every language ecosystem except C/C++, where the
package manager story is just so bad that system libraries are
actually the only reliable development environment.
> > This is due to a limitation of how cargo handles target-specific
> > dependencies - all dependencies that are *mentioned in any way* need
> > to be *present* for it to resolve dependencies / enabled optional
> > features / update its lockfile etc. But since we don't want to package
> > bindings for Windows and mac OS system APIs, we need to actually patch
> > them out, otherwise builds will fail.
>
> And that ends up being quite a bit of work from my point of view. Also
> the way the packaging works with options things like devel or optional
> features ends up being very painful. I will often drop out optional
> features just so I can do less packaging.
Recent versions of rust2rpm automatically generate a patch to remove
non-linux dependencies, so that work has effectively been reduced to
zero.
Disabling unused optional dependencies can also be done with either a
rust2rpm config option or with a simple patch, so that should not be a
problem, either. I also don't understand why disabling some optional
features is such a bad thing? I don't think we have a policy in Fedora
that says we *must* enable *all possible functionality and optional
features*.
> > > We're doing okay with #1, but... I think #3 _even_ with all of the
work in
> > > Rust-to-RPM packaging isn't sufficient. I've played with the Bevy
game
> > > engine and will probably have a few things it would be nice to package to
> > > make available in Fedora Linux. I might not even mind maintaining Bevy
> > > itself.
> >
> > Somebody actually already started packaging Bevy components - some
> > packages are already approved and some are still pending review. Not
> > sure what the progress has been there, but it's not *impossible*.
>
> Well nothing is *impossible* if you have enough stamina, resources or
> whatever else. I don't find saying something isn't *impossible*
> necessarily makes it compelling.
I agree. And I think we can do better, as I said in my original post.
> > > But running `cargo fetch` with a clean cache pulls down *390* crates. Of
> > > these, it looks like 199 (!) are already packaged as rust-[crate]-devel,
> > > which is *amazing*. But... that still is hundreds that I'd have to
add. And
> > > mostly they are things I don't know _anything_ about.
> >
> > You must realize that this is an extreme case. For many Rust
> > applications that people want to package for Fedora, the number of
> > dependencies that are missing is rather small, *because* most popular
> > libraries are already packaged.
>
> In my experience, and I'm not packaging any form of gaming engine,
> it's not an extreme case, for a few small things and one slightly
> larger thing I *have* packaged I now maintain over 60 more packages. I
> have another one I want to package and AFAICT I get to package another
> 50-100 but frankly it's hard to tell, and this is something I have
> control over and have been actively trying to do upstream reduction of
> deps.
As far as I know, salimma's intern worked on a tool to determine
missing dependencies for Rust projects in Fedora recursively, which
should make it easier to determine the exact set of things that you
would need to do here.
> And requests to make things like this easier have been open for over a year:
> https://pagure.io/fedora-rust/rust2rpm/issue/140
Right. I'm sorry about the issue reports that have been open for a long time.
rust2rpm / rust-packaging is an entirely community-run project. We
don't get any support from Red Hat (or other places) for anything
except the Rust compiler itself. And with the original developer of
rust2rpm (Igor) being gone, nobody who understands the low-level stuff
in there is still around.
zbyszek and I are trying our best to keep things running, but at this
point, we'll need basically a rewrite of both rust2rpm and the RPM
dependency / provides generators to support new cargo features. And
since I don't have much time for that right now (busy finishing my
university degree), these things are still pending.
I'm still around, but my interest in Rust has flagged significantly
after getting crap from upstream Rust developers and many other folks
in the Rust community when Igor and I did this work. I do still look
over PRs occasionally too.
> > Bevy is a bit special, because it (presumably) pulls in
lots of GPU /
> > OpenGL / Vulkan related libraries, which we didn't need to package for
> > anything else yet, and it's also split into dozens of small libraries
> > itself, which can be painful to package, that is true.
> >
> > We might need to reconsider how to package projects like this. I'm
> > pretty sure we could find a way to package them in a way that's
> > compatible with how we're currently doing things but would be much
> > less busywork.
>
> Well if we're finding a solution for those sorts of projects we should
> be doing it for all rust projects. A lot of projects have internal
> virtual/workspace bits where they split up things up internally but
> may have a dozen or sub projects. These make absolutely no sense to
> split up into separate packages because they're all bundled together,
> have to be bumped together yet as it standard at the moment you either
> have to package them all separately or do nasty rpm spec hacking to
> try and glue it into a single rpm. Again RFEs to make less busy work
> have sat for well over a year, some even 3 years, with no progress:
>
> https://pagure.io/fedora-rust/rust2rpm/issue/141
Same here. I *would like to work on this and improve the situation for
all of us (including me)*, but I just don't have the time to do that
right now.
> > Sure, but isn't that the case for most projects that a newcomer wants
> > to package, regardless of programming language? Say, somebody wants to
> > package some cool new Python project for machine learning, then
> > there's probably also some linear algebra package or SIMD math library
> > in the dependency tree that's missing from Fedora. How is that
> > different?
>
> I think the big difference here from my experience, and I've packaged
> right across the Fedora spectrum, is that most of the python style
> projects are able to use a pretty comprehensive standard library and
> crypto library which helps minimise a lot of the extras I've seen in
> the rust ecosystem.
The Rust standard library is intentionally narrow. I don't think that
will change any time soon, so we'll have to deal with it. Either way,
it looks like the ecosystem is starting to standardize around a few
common libraries (serde for de/serialization, tokio as async runtime,
hyper for HTTP clients + servers, etc.), so things might get less
sprawling in these areas.
I think a de-facto standard library will emerge over time. Rust is
young and most of its developers don't know what they want from the
language ecosystem yet. It'll take time to sort it out.
--
真実はいつも一つ！/ Always, there's only one truth!
7:53 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, 30 Nov 2022 at 07:54, Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
On Wed, Nov 30, 2022 at 11:54:10AM +0000, Peter Robinson wrote:
> Hi Fabio,
>
> Been meaning to reply to this, but it got lost in the mail pile.
>
> > > But running `cargo fetch` with a clean cache pulls down *390*
crates. Of
> > > these, it looks like 199 (!) are already packaged as
rust-[crate]-devel,
> > > which is *amazing*. But... that still is hundreds that I'd have to
add. And
> > > mostly they are things I don't know _anything_ about.
> >
> > You must realize that this is an extreme case. For many Rust
> > applications that people want to package for Fedora, the number of
> > dependencies that are missing is rather small, *because* most popular
> > libraries are already packaged.
>
> In my experience, and I'm not packaging any form of gaming engine,
> it's not an extreme case, for a few small things and one slightly
> larger thing I *have* packaged I now maintain over 60 more packages. I
> have another one I want to package and AFAICT I get to package another
> 50-100 but frankly it's hard to tell, and this is something I have
> control over and have been actively trying to do upstream reduction of
> deps.
The magnitude of the number of deps is the real killer for not
only Rust, but also Go, and arguably anything that isn't a C
based application.
Packaging up all of a project's deps individually is viable when
there are a relatively small number of them, especially if most
of the deps are shared by other apps, and the libraries have good
API + ABI stability promises. This is common in the case of most
C applications/libraries, since shared libraries are relatively
speaking fairly broad in functional scope, and often long lived,
because that is the mindset of the C ecosystem in general.
Modern languages & their ecosystems though have brought about
a world where the granularity of deps is waaaaay smaller, where
the focus is on being able to evolve rapidly, with less focus
on long term stable API/ABI, as apps are expected to just fix
against specific versions they've tested on.
I wonder if we should look at the standard libraries in the late 1960/1970
languages as being the same as 'opinionated' Operating Systems. Most of
them started out with a period where every mainframe might have a
'language' and a set of 'libraries' of routines but every site pretty
much
mangled them to fit their own 'local' needs. Software vendors which existed
usually ended up having consultants go out to 'fix' their code to match
whatever routines and tooling existed on each mainframe. This got to be
unworkable and various 'libraries' were put forward which were to
standardize things. However, going from the 'old war tales' my mentors and
professors from that era, the same arguments that the current languages
(Rust, etc) have against standards were existent. The issue was the scale
of the problem was much more on the side of the hardware/OS vendors putting
out a standard chosen library (and then fighting over getting those to
match up for the software vendors who still needed to spend a lot of
consulting time). Most of those arguments seem to have 'died' out as people
got tired of fighting the differences versus the delight of new challenges
that many programmers have when dealing with a 'new' language. [Same with
C++ back in the late 1980's where every vendor had a slightly different set
of libraries which for its proponents was the best thing over all the
others.. and then by the late 1990's was 'Oh no, not this again'. Java went
through this up until the late 00's. ]
I don't think we are at the state where enough of the new people who are
getting into Rust have gotten tired of fighting 'oh, I have to f'ing change
my code again to make it work with these 4 things?' Basically when enough
people HAVE to code in the language versus just 'WANT' to, but hate the
grind.. ability to standardize is going to happen.
So functionality that lives in single library in C, often ends
up being spread across 20-200 modules in any of the modern
non-C languages. We've been experiancing this problem in Fedora
for a very long time and while we have spec auto-generators,
that still leaves masses of busy work, and also still leaves
the problem that what we ship doesn't match what upstream has
tested, so the result is often less reliable.
IMHO the only thing that can make packaging such fine grained
modules sustainable long term, is if the process could be 100%
automated. I don't mean just having a tool to auto-generate
a spec file. It needs to be fully automated to the extent that
you just tell a robot "i need deps for this package" and it
does everything automatically, except for human approval of
the results of a code license audit.
That of course would be a non-trivial service to build, and
it still begs the question of whether its really beneficial
in the long term.
At the moment, I think that it is probably the best goal for both
opinionated operating system reasons, but also long term needs for a lot of
'customers'. This code is entering into a lot of highly regulated markets
where everything needs to be captured and kept for lifetimes of people
versus hardware and definitely longer than most 'software' sites
(sourceforge anyone?). Being able to accurately capture exactly what was
used, how it was used, and being able to independently audit it 20 years
down the road has been in many ways the hidden 'charm' (or solution to a
nightmare for the developer dealing with an audit of RHL6.2 in 2022) of
operating systems. It is one of the things I see problematic in various
central archives for code which people seem to rely on for their
deliverables.
--
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
11:20 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
Do I really need to explain this point? I think linking against
system
OpenSSL is *way better* than statically linking to a random vendored
copy of it.
There are maybe about 100-120 libraries for which this is obviously the case. openssl,
glibc, glib2, zlib, libxml2, libcurl, kde libraries, etc. Core system libraries that
pretty much everything depends on. Dynamically linking such libraries has real benefits.
For everything else though? No, not so much.
I feel like there is insufficient recognition of the extent to which C libraries do
"bundling". Not "bundling" in the sense of vendoring a whole library,
but in the sense of including one-off implementations of basic data structures,
configuration parsers, hashing algorithms, etc. I would love to hear anyone argue that
100 different variations of "sha256.c" across 100 different packages better
follows the spirit of the "no bundling" guidelines than a vendored crate named
"sha256" with 100x as many eyes on it, and a higher likelihood to actually be
updated if a problem is found.
Many of the tiny, "sprawling" Rust dependencies are like this - not all of them
of course, but many.
Torvalds has similar feelings:
https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=...
11:47 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley <dalley(a)redhat.com> wrote:
> Do I really need to explain this point? I think linking against system
> OpenSSL is *way better* than statically linking to a random vendored
> copy of it.
There are maybe about 100-120 libraries for which this is obviously the case. openssl,
glibc, glib2, zlib, libxml2, libcurl, kde libraries, etc. Core system libraries that
pretty much everything depends on. Dynamically linking such libraries has real benefits.
For everything else though? No, not so much.
These are actually all good examples of stuff that we dynamically link to.
This is the (not 100% exhaustive) list of system libraries we always
dynamically link to:
- all GTK / GNOME libraries (dbus, freetype, fontconfig, atk,
gdk-pixbuf, gdk, gtk3, gtk4, gio, glib2, graphene, gsk4, pango, cairo,
gstreamer, etc.)
- multimedia codecs / libraries (aom, dav1d, pipewire, pulseaudio, etc.)
- crypto libraries (openssl, libsodium, nettle, curl)
- compression libraries (bzip2, flate2, libz, lzma, zstd)
- database connectors (libsqlite, libpq)
- low-level device / storage libraries (devicemapper, libblkid)
I think almost all of these qualify as "Core system libraries that
pretty much everything depends on.".
Building their C dependencies from vendored copies (if that is even
supported) and statically linking them seems like a pretty bad idea in
almost all cases here, especially for things where the version of a
program on the "host" and the accompanying shared library should
match.
The only exception (that I know of) for "crate dependency built from
vendored sources and statically linked" in Fedora right now is
libgit2, because the version in Fedora is chronically outdated, and
dealing with that has become very painful - and started to block some
necessary updates to support more recent versions of Rust / cargo.
I feel like there is insufficient recognition of the extent to which
C libraries do "bundling". Not "bundling" in the sense of vendoring a
whole library, but in the sense of including one-off implementations of basic data
structures, configuration parsers, hashing algorithms, etc. I would love to hear anyone
argue that 100 different variations of "sha256.c" across 100 different packages
better follows the spirit of the "no bundling" guidelines than a vendored crate
named "sha256" with 100x as many eyes on it, and a higher likelihood to actually
be updated if a problem is found.
Many of the tiny, "sprawling" Rust dependencies are like this - not all of them
of course, but many.
But ... none of these "tiny" Rust crates are dynamically linked in
Fedora anyway - because Rust doesn't really support that?
So I fail to see your point there, unless you meant to say "C projects
don't 'bundle', they just often 'copy' some code into their
projects"?
Fabio
12:26 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia (System-Wide Change proposal)]
On Wednesday, 30 November 2022 17:47:16 GMT Fabio Valentini wrote:
On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley
<dalley(a)redhat.com> wrote:
> I feel like there is insufficient recognition of the extent to which C
> libraries do "bundling". Not "bundling" in the sense of
vendoring a
> whole library, but in the sense of including one-off implementations of
> basic data structures, configuration parsers, hashing algorithms, etc. I
> would love to hear anyone argue that 100 different variations of
> "sha256.c" across 100 different packages better follows the spirit of the
> "no bundling" guidelines than a vendored crate named "sha256"
with 100x
> as many eyes on it, and a higher likelihood to actually be updated if a
> problem is found.
>
>
> Many of the tiny, "sprawling" Rust dependencies are like this - not all
of
> them of course, but many.
But ... none of these "tiny" Rust crates are dynamically linked in
Fedora anyway - because Rust doesn't really support that?
So I fail to see your point there, unless you meant to say "C projects
don't 'bundle', they just often 'copy' some code into their
projects"?
I think the point he's making is that developers don't write common
functionality from scratch, in general. We reuse code from elsewhere.
It's just that in C, I'll copy-and-paste code from the web into my library or
application, not necessarily even bothering with a full "vendoring", whereas
in Rust, I'll use the crc crate (say), or the base64 crate, or other simple
utility crate.
The result is that I have N implementations of common functionality, each with
its own unique quirks and security risks, in my C binaries; but my C binaries
have only a small number of dependencies.
In Rust, however, I'm directly reusing the small utility crate, and while I
may use `cargo vendor` to import the crate's source into my tree, I'm unlikely
to edit it. The result is that a Rust binary has a larger number of
dependencies than the equivalent C program, because I'm depending on a crate
instead of copy-and-pasting the code and "hiding" it from the packager.
This is a challenge for Fedora: how do we cope with a world where instead of
having a few tens of dependencies, and a lot of copy-and-paste code, we have
hundreds of dependencies, but no copy-and-paste code?
One answer is to say that Rust is bad for encouraging developers to depend on
small crates instead of copying-and-pasting "small" utilities around. Another
(which you're doing a great job of) is to package up all the dependencies, so
that we represent the true dependency tree in RPM. Yet another would be to
manually decide which dependencies get bundled, and which don't - doing the
same thing as the C world does to keep its dependency count down.
--
Simon Farnsworth
2:18 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Wed, 2022-11-30 at 18:26 +0000, Simon Farnsworth via devel wrote:
On Wednesday, 30 November 2022 17:47:16 GMT Fabio Valentini wrote:
> On Wed, Nov 30, 2022 at 6:21 PM Daniel Alley <dalley(a)redhat.com> wrote:
> > I feel like there is insufficient recognition of the extent to which C
> > libraries do "bundling". Not "bundling" in the sense of
vendoring a
> > whole library, but in the sense of including one-off implementations of
> > basic data structures, configuration parsers, hashing algorithms, etc. I
> > would love to hear anyone argue that 100 different variations of
> > "sha256.c" across 100 different packages better follows the spirit of
the
> > "no bundling" guidelines than a vendored crate named
"sha256" with 100x
> > as many eyes on it, and a higher likelihood to actually be updated if a
> > problem is found.
>
> >
> >
> > Many of the tiny, "sprawling" Rust dependencies are like this - not
all of
> > them of course, but many.
>
> But ... none of these "tiny" Rust crates are dynamically linked in
> Fedora anyway - because Rust doesn't really support that?
> So I fail to see your point there, unless you meant to say "C projects
> don't 'bundle', they just often 'copy' some code into their
projects"?
>
I think the point he's making is that developers don't write common
functionality from scratch, in general. We reuse code from elsewhere.
It's just that in C, I'll copy-and-paste code from the web into my library or
application, not necessarily even bothering with a full "vendoring", whereas
in Rust, I'll use the crc crate (say), or the base64 crate, or other simple
utility crate.
The result is that I have N implementations of common functionality, each with
its own unique quirks and security risks, in my C binaries; but my C binaries
have only a small number of dependencies.
In Rust, however, I'm directly reusing the small utility crate, and while I
may use `cargo vendor` to import the crate's source into my tree, I'm unlikely
to edit it. The result is that a Rust binary has a larger number of
dependencies than the equivalent C program, because I'm depending on a crate
instead of copy-and-pasting the code and "hiding" it from the packager.
This is a challenge for Fedora: how do we cope with a world where instead of
having a few tens of dependencies, and a lot of copy-and-paste code, we have
hundreds of dependencies, but no copy-and-paste code?
One answer is to say that Rust is bad for encouraging developers to depend on
small crates instead of copying-and-pasting "small" utilities around. Another
(which you're doing a great job of) is to package up all the dependencies, so
that we represent the true dependency tree in RPM. Yet another would be to
manually decide which dependencies get bundled, and which don't - doing the
same thing as the C world does to keep its dependency count down.
This is a bit of a misleading characterization.
- the amount of copying in C programs is overblown in my opinion (no
data provided to back this up though)
- dependency minimization for C programs/libraries is assumed but no
data provided to back it up.
- the dilemma is how to manage rust programs not to decide what to
bundle or not, that's basically decided upstream
Although there are certainly instances of copying/paste in C code, the
vast majority of reuse comes through linking to utility libraries
dynamically, which makes distro-level maintenance much easier because
you have only once place to fix/rebuild/distribute when there are
serious issues.
The problems with Rust crates (or go modules, or any other
"vendoring"), is that not only you have to go and find each place where
a problematic crate was vendored; you also have to figure out, often
under pressure, if the crate can simply be summarily updated or if you
need a backport because the vendoring application can't cope with the
semantic changes that happened in the problematic crate's new version.
Multiply this by N packages using M different versions of the
problematic crate.
Although vendored crates can be tracked (this i much better than
copy/pasting), with additional tooling, the distribution remains on the
hook for solving the same problem in N packages, without easy
coordination. Some upstream may be quick and do the work for you, some
may not care, disappear etc... or are simply too slow for the urgency
the distribution has leading potentially to diverging solutions ...
Simo.
--
Simo Sorce
RHEL Crypto Team
Red Hat, Inc
3:18 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
I think almost all of these qualify as "Core system libraries
that
pretty much everything depends on.".
Building their C dependencies from vendored copies (if that is even
supported) and statically linking them seems like a pretty bad idea in
almost all cases here, especially for things where the version of a
program on the "host" and the accompanying shared library should
match.
Yes, we're in complete agreement. I'm not suggesting anything like that.
Vendoring libraries like openssl is a bad idea.
What I'm saying is that it's not very logically justified to say that just because
core system libraries like openssl shouldn't be vendored, all vendoring is disallowed
regardless of how small and focused they are or how few dependents they have.
Because - most C libraries have "dark dependencies" that are effectively the
same but worse, in some ways. Given the choice between 100 Rust packages vendoring 10
different copies of the sha256 crate and 100 C packages with 100 separate copies of
sha256.c sitting in their source trees (which seems like an entirely realistic
comparison), why would the latter be completely A-OK while the latter is completely
disallowed?
But ... none of these "tiny" Rust crates are dynamically
linked in
Fedora anyway - because Rust doesn't really support that?
So I fail to see your point there, unless you meant to say "C projects
don't 'bundle', they just often 'copy' some code into their
projects"?
Fabio
Yes, that's essentially what I'm saying. I feel like the "no bundling"
policy draws distinctions that don't entirely make sense, especially when it comes to
the small, focused leaf-node dependencies that people often complain about.
Clearly the "left-pad" scenario is bad and should be avoided, but on the other
hand is having 800 different linked list implementations, 500 different hash table
implementations, 25 different half-baked XML parsers, really so much "better",
or is it just what we're used to?
3:34 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
Once upon a time, Daniel Alley <dalley(a)redhat.com> said:
100 C packages with 100 separate copies of sha256.c sitting in their
source trees (which seems like an entirely realistic comparison)
You keep saying this - do you have any evidence that this is the case?
--
Chris Adams <linux(a)cmadams.net>
6:41 p.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
I'm quite not sure how one would go about empirically measuring something like that -
at least in the general case. It might be an interesting research topic. So no,
unfortunately I don't really have hard evidence for this.
I just know that of all the C libraries I've looked at, in my personal experience it
seems to be a very common phenomenon to copy or reimplement code that in Rust you would
just import and re-use.
It's just a pattern that one notices frequently when it comes to C libraries,
especially crossplatform ones that can't rely exclusively on the existence of a
Linux-like package manager.
If you want specific examples, the ones that pop to mind are:
* zchunk and deltarpm both reimplement / "bundle" multiple different hashing
algorithms
* libcomps implements about 4 different relatively common data structures
* GTK appears to contain a bundled, forked copy of the CRoaring library
11:04 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Thu, 2022-12-01 at 00:41 +0000, Daniel Alley wrote:
I'm quite not sure how one would go about empirically measuring
something like that - at least in the general case. It might be an
interesting research topic. So no, unfortunately I don't really have
hard evidence for this.
We did run discovery for this in RHEL when we started the SHA-1
deprecation process.
Some embedded implementations were found but the vast majority of
programs used one of the available libraries for SHA-1. I do not expect
SHA-256 to be any different.
so empirically I can tell you there isn't anywhere as much "vendoring"
in C as you claim. Using dynamically linked libraries is well establish
and the reason why sometimes the dependency chain is ... monstrous.
I just know that of all the C libraries I've looked at, in my
personal experience it seems to be a very common phenomenon to copy
or reimplement code that in Rust you would just import and re-use.
Perhaps this is true in the niche you are interested in?
It's just a pattern that one notices frequently when it comes to
C
libraries, especially crossplatform ones that can't rely exclusively
on the existence of a Linux-like package manager.
Yes, those kind of libraries tend to be quite bad in this regard, OTOH
it can be done right.
For example the NSS library generally carries copies of external
dependencies, but the configure script looks for a system version and
links to the one if available on build.
So if you looked at NSS you may think it vendors, and it does, but
smartly and in a way that is compatible with systems integration.
If you want specific examples, the ones that pop to mind are:
* zchunk and deltarpm both reimplement / "bundle" multiple different
hashing algorithms
* libcomps implements about 4 different relatively common data
structures
I am not sure this qualify as vendoring/bundling, kind of borderline
but I see why you added it as a case.
* GTK appears to contain a bundled, forked copy of the CRoaring
library
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Simo Sorce
RHEL Crypto Team
Red Hat, Inc
11:25 a.m.
New subject: musings on rust packaging [was Re: F38 proposal: RPM Sequoia
(System-Wide Change proposal)]
On Thu, 2022-12-01 at 00:41 +0000, Daniel Alley wrote:
<snip>
* zchunk and deltarpm both reimplement / "bundle" multiple different
hashing algorithms
zchunk does have bundled versions of various hashing algorithms, but,
if it's compiled against OpenSSL (as it is in Fedora), it uses the
OpenSSL hashing algorithms rather than the bundled versions.
Jonathan
11:07 a.m.
Hi Simo,
On Fri, 14 Oct 2022 18:28:01 +0200,
Simo Sorce wrote:
At this time, as far as I know, there is no OpenPGP work of any kind
on
supporting PQC algorithms. Furthermore the way we use signatures in RPM
really has no resemblance to the scenarios OpenPGP was built for.
So we should consider whether moving to PQC will also mean moving off
OpenPGP as our signature format and into something simpler.
A draft for PQC in OpenPGP was published today:
https://mailarchive.ietf.org/arch/msg/openpgp/zjbIqesTdiYfA7vkyMZ6EM9KWrA/
https://datatracker.ietf.org/doc/draft-wussler-openpgp-pqc/
Neal
489
days inactive
561
days old
85 comments
29 participants
participants (29)
-
Ben Cotton -
Blaise Pabon -
Chris Adams -
Clemens Lang -
Daniel Alley -
Daniel P. Berrangé -
Demi Marie Obenour -
Fabio Valentini -
Gary Buhrmaster -
Jelle van der Waa -
Jonathan Dieter -
Kevin Kofler -
Kevin P. Fleming -
Matthew Miller -
Maxwell G -
Michel Alexandre Salim -
Miro Hrončok -
Neal Gompa -
Neal H. Walfield -
Panu Matilainen -
Peter Robinson -
Petr Menšík -
PGNet Dev -
Sandro -
Simo Sorce -
Simon Farnsworth -
Stephen Smoogen -
Vitaly Zaitsev -
Zbigniew Jędrzejewski-Szmek