----- Original Message -----
From: "Nicolas Mailhot" nicolas.mailhot@laposte.net To: packaging@lists.fedoraproject.org Sent: Friday, March 29, 2019 8:33:39 PM Subject: [Fedora-packaging] Translating Go modules buildrequires in rpm syntax
Hi,
For Go modules, that will become the default upstream in August, Google invented holed BuildRequires version ranges: – anything starting with version – except a list of specific version exclusions (not exclusion ranges, specific versions)
Basically upstreams are allowed to declare incompatible versions, but Google would really like them to keep ascendent API compatibility (and Google requires module renaming on major version changes, so there is no upper limit to the API compatibility)
As I perceive this might be perceived by you as my personal attack I want to be clear that this is just my take on it and my technical opinion, so please keep this in mind. I would like to avoid any unnecessary misunderstandings and conflicts.
But do we really need to recreate everything that upstream put in to the constrains in specs? I think that we are carrying curated set of the packages so this is IMHO not really necessary(at least not in all cases, just ones that crept in to our package base and we should avoid them at all costs).
IMO if there are "holes" in the versions we should work with the upstreams to mitigate(fix) them and I believe they will be probably for some past/intermittent versions(do you have actual cases to show as examples?) and we can pretty easily handle them preventively with conflicts statements and proper comments accompanying them like "Based on constrain in the module ... caused by ...". Or is this something that is not possible/feasible?
Do you have any cases where we will have to carry them in 1:1 representation with current package set?
JC
PS: CC'ing wider Fedora's Go community
Is the correct way to represent those in rpm:
BuildRequires: ((golang-module(github.com/stretchr/testify) >= 1.3.0) and ((golang-module(github.com/stretchr/testify) < 1.3.2~0.20180906233101.161cd47e91fd) or (golang-module(github.com/stretchr/testify) > 1.3.2~0.20180906233101.161cd47e91fd)) and ((golang-module(github.com/stretchr/testify) < 1.3.2~1.pre1) or (golang-module(github.com/stretchr/testify) > 1.3.2~1.pre1)) and ((golang-module(github.com/stretchr/testify) < 1.3.2~1.pre1.0.20180628173108.788fd7840127) or (golang-module(github.com/stretchr/testify) > 1.3.2~1.pre1.0.20180628173108.788fd7840127)))
or am I missing something obvious?
Best regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2019-04-02 10:04, Jakub Cajka a écrit : Hi Jakub,
But do we really need to recreate everything that upstream put in to the constrains in specs? I think that we are carrying curated set of the packages so this is IMHO not really necessary(at least not in all cases, just ones that crept in to our package base and we should avoid them at all costs).
We will have to diverge from upstream, that's an absolute certainty, the quality of upstream releases is just not good enough for direct packaging in many cases.
However, just because we will diverge on some points, does not mean we have to diverge everywhere. Diverging makes engaging with upstream hard.
In that particular case it is possible to represent perfectly the upstream notion of holed version ranges as rpm constrains, and the upstream notion of holed version ranges is quite sane and limited¹, so there's no reason not to translate it accurately by default.
The packager can patch out exclusions from the upstream go.mod file in %prep if he finds them problematic.
Alternatively, he can add a sed (or other) filter to the buildrequires generator output, that's perfectly allowed by the https://github.com/rpm-software-management/rpm/issues/104 design.
Things like module replaces will be a lot more tricky to handle, and will almost always require packager decisions when present in upstream module files.
Likewise, things like “every file in the Go module participates in go module requirements, including tests and other os support files” will be awful for us and will require fine-grained file removal in %prep. We won't be able to install broken files and pretend they're not here like today (I warned against install-but-pretend-it’s-not-here strategies at the last SIG meeting, precisely because Go modules won’t let us do it free of deps).
So, don’t fret, we *will* diverge form upstream, just not on exclusions, unless we have to.
¹ Projects have to declare every single code state they want to blacklist, they can not issue blanket exclusion range statements, it's real blacklisting not “I don't want to look at other versions” stuff
----- Original Message -----
From: "Nicolas Mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Tuesday, April 2, 2019 12:07:58 PM Subject: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 10:04, Jakub Cajka a écrit : Hi Jakub,
But do we really need to recreate everything that upstream put in to the constrains in specs? I think that we are carrying curated set of the packages so this is IMHO not really necessary(at least not in all cases, just ones that crept in to our package base and we should avoid them at all costs).
We will have to diverge from upstream, that's an absolute certainty, the quality of upstream releases is just not good enough for direct packaging in many cases.
However, just because we will diverge on some points, does not mean we have to diverge everywhere. Diverging makes engaging with upstream hard.
In that particular case it is possible to represent perfectly the upstream notion of holed version ranges as rpm constrains, and the upstream notion of holed version ranges is quite sane and limited¹, so there's no reason not to translate it accurately by default.
The packager can patch out exclusions from the upstream go.mod file in %prep if he finds them problematic.
Alternatively, he can add a sed (or other) filter to the buildrequires generator output, that's perfectly allowed by the https://github.com/rpm-software-management/rpm/issues/104 design.
Things like module replaces will be a lot more tricky to handle, and will almost always require packager decisions when present in upstream module files.
Likewise, things like “every file in the Go module participates in go module requirements, including tests and other os support files” will be awful for us and will require fine-grained file removal in %prep. We won't be able to install broken files and pretend they're not here like today (I warned against install-but-pretend-it’s-not-here strategies at the last SIG meeting, precisely because Go modules won’t let us do it free of deps).
So, don’t fret, we *will* diverge form upstream, just not on exclusions, unless we have to.
¹ Projects have to declare every single code state they want to blacklist, they can not issue blanket exclusion range statements, it's real blacklisting not “I don't want to look at other versions” stuff
-- Nicolas Mailhot
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code, creating part that needs to be (manually) maintained up to date, while the GC tooling will enforce them anyway. We will have to workaround those constrains eventually in some cases, for sure as you outlined, but we don't need to IMHO "waste" time on mapping them 1:1 in to the RPM world, with RPM constrains.
JC
packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
creating part that needs to be (manually) maintained up to date,
There's nothing to maintain manually if you integrate properly with rpm. That's why spec generators like gofed, that try to avoid the rpm integration part, do not pass the maintainability test.
Integrating properly means https://github.com/rpm-software-management/rpm/issues/104
while the GC tooling will enforce them anyway.
ie builds will fail because the build environment is not populated correctly. It'd rather populate it automatically and correctly by default than wait for builds to fail and then spend time on manual workarounds because the tooling does not help me.
----- Original Message -----
From: "Nicolas Mailhot" nicolas.mailhot@laposte.net To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
creating part that needs to be (manually) maintained up to date,
There's nothing to maintain manually if you integrate properly with rpm. That's why spec generators like gofed, that try to avoid the rpm integration part, do not pass the maintainability test.
Integrating properly means https://github.com/rpm-software-management/rpm/issues/104
Which is not yet there, right?
while the GC tooling will enforce them anyway.
ie builds will fail because the build environment is not populated correctly. It'd rather populate it automatically and correctly by default than wait for builds to fail and then spend time on manual workarounds because the tooling does not help me.
Great to hear that you are working on the packaging automation.
JC
-- Nicolas Mailhot
Le 2019-04-02 14:51, Jakub Cajka a écrit :
----- Original Message -----
From: "Nicolas Mailhot" nicolas.mailhot@laposte.net To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
And upstream tools check and regenerate the go.mod file at the slightest occasion.
So, basically, having rpm parse exactly this file is not a problem, because if a packager disagrees with the content of this file, but does not fix/patch it, go tools will ignore the packager and obey the go.mod file anyway.
creating part that needs to be (manually) maintained up to date,
There's nothing to maintain manually if you integrate properly with rpm. That's why spec generators like gofed, that try to avoid the rpm integration part, do not pass the maintainability test.
Integrating properly means https://github.com/rpm-software-management/rpm/issues/104
Which is not yet there, right?
Which is being worked on by rpm upstream, just like I'm working on the corresponding generator for go modules (we already have a generator for pre-module world, even though it is based on golist not the dep analysis upstream created for go modules).
No having a generator at Go module time will mean every Go packager will have to perform go.mod analysis and translation to rpm deps manually, for every single Go project we package.
On Tue, Apr 2, 2019 at 6:17 PM Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le 2019-04-02 14:51, Jakub Cajka a écrit :
----- Original Message -----
From: "Nicolas Mailhot" nicolas.mailhot@laposte.net To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
And upstream tools check and regenerate the go.mod file at the slightest occasion.
So, basically, having rpm parse exactly this file is not a problem, because if a packager disagrees with the content of this file, but does not fix/patch it, go tools will ignore the packager and obey the go.mod file anyway.
I usually disagree with Nicolas, but this is the point where he gets +100 from me. Reflecting upstream constraints in RPM is essential. People do not just use distro repos, they build their own packages and such. In this case, we should be very strict about what content should be in buildroot and on user's systems. Otherwise you will get a lot of bugreports like "this golang module doesn't compile since last update" (which is because we didn't have proper info in RPM requires).
----- Original Message -----
From: "Igor Gnatenko" ignatenkobrain@fedoraproject.org To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 6:40:27 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
On Tue, Apr 2, 2019 at 6:17 PM Nicolas Mailhot < nicolas.mailhot@laposte.net
wrote:
Le 2019-04-02 14:51, Jakub Cajka a écrit :
----- Original Message -----
From: "Nicolas Mailhot" < nicolas.mailhot@laposte.net > To: "Discussion of RPM packaging standards and practices for Fedora" < packaging@lists.fedoraproject.org > Cc: golang@lists.fedoraproject.org , "Jakub Cajka" < jcajka@redhat.com > Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
And upstream tools check and regenerate the go.mod file at the slightest occasion.
So, basically, having rpm parse exactly this file is not a problem, because if a packager disagrees with the content of this file, but does not fix/patch it, go tools will ignore the packager and obey the go.mod file anyway.
I usually disagree with Nicolas, but this is the point where he gets +100 from me. Reflecting upstream constraints in RPM is essential. People do not just use distro repos, they build their own packages and such. In this case, we should be very strict about what content should be in buildroot and on user's systems. Otherwise you will get a lot of bugreports like "this golang module doesn't compile since last update" (which is because we didn't have proper info in RPM requires).
Actually same here as he is mentioning that he is working on the packaging automation. Without it IMO would be case for concerns. With it, it is awesome and I'm looking forward to it.
JC
golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/golang@lists.fedoraproject.org
On Wed, Apr 3, 2019 at 3:26 AM Jakub Cajka jcajka@redhat.com wrote:
----- Original Message -----
From: "Igor Gnatenko" ignatenkobrain@fedoraproject.org To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 6:40:27 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
On Tue, Apr 2, 2019 at 6:17 PM Nicolas Mailhot < nicolas.mailhot@laposte.net
wrote:
Le 2019-04-02 14:51, Jakub Cajka a écrit :
----- Original Message -----
From: "Nicolas Mailhot" < nicolas.mailhot@laposte.net > To: "Discussion of RPM packaging standards and practices for Fedora" < packaging@lists.fedoraproject.org > Cc: golang@lists.fedoraproject.org , "Jakub Cajka" < jcajka@redhat.com > Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
And upstream tools check and regenerate the go.mod file at the slightest occasion.
So, basically, having rpm parse exactly this file is not a problem, because if a packager disagrees with the content of this file, but does not fix/patch it, go tools will ignore the packager and obey the go.mod file anyway.
I usually disagree with Nicolas, but this is the point where he gets +100 from me. Reflecting upstream constraints in RPM is essential. People do not just use distro repos, they build their own packages and such. In this case, we should be very strict about what content should be in buildroot and on user's systems. Otherwise you will get a lot of bugreports like "this golang module doesn't compile since last update" (which is because we didn't have proper info in RPM requires).
Actually same here as he is mentioning that he is working on the packaging automation. Without it IMO would be case for concerns. With it, it is awesome and I'm looking forward to it.
To the original query, how I see it is that our (Build)Requires should structure as accordingly:
BuildRequires: (golang-module(foo) >= 1.0.0 unless (golang-module(foo) = 1.1.5 or golang-module(foo) = 1.2.0))
s/unless/without/
And then it looks good.
On Wed, Apr 3, 2019 at 2:19 PM Neal Gompa ngompa13@gmail.com wrote:
On Wed, Apr 3, 2019 at 3:26 AM Jakub Cajka jcajka@redhat.com wrote:
----- Original Message -----
From: "Igor Gnatenko" ignatenkobrain@fedoraproject.org To: "Discussion of RPM packaging standards and practices for Fedora" <
packaging@lists.fedoraproject.org>
Cc: golang@lists.fedoraproject.org, "Jakub Cajka" jcajka@redhat.com Sent: Tuesday, April 2, 2019 6:40:27 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules
buildrequires in rpm syntax
On Tue, Apr 2, 2019 at 6:17 PM Nicolas Mailhot <
nicolas.mailhot@laposte.net
wrote:
Le 2019-04-02 14:51, Jakub Cajka a écrit :
----- Original Message -----
From: "Nicolas Mailhot" < nicolas.mailhot@laposte.net > To: "Discussion of RPM packaging standards and practices for Fedora" < packaging@lists.fedoraproject.org > Cc: golang@lists.fedoraproject.org , "Jakub Cajka" <
jcajka@redhat.com >
Sent: Tuesday, April 2, 2019 1:27:09 PM Subject: Re: [Fedora-packaging] Re: Translating Go modules buildrequires in rpm syntax
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as
they
are already capture in the Go source code,
In the Go module world the constrains are in the module files, not
in
the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
And they are part of the checked in source tree of the respective Go project, if I'm not mistaken.
And upstream tools check and regenerate the go.mod file at the
slightest
occasion.
So, basically, having rpm parse exactly this file is not a problem, because if a packager disagrees with the content of this file, but does not fix/patch it, go tools will ignore the packager and obey the go.mod file anyway.
I usually disagree with Nicolas, but this is the point where he gets
+100
from me. Reflecting upstream constraints in RPM is essential. People
do not
just use distro repos, they build their own packages and such. In this
case,
we should be very strict about what content should be in buildroot and
on
user's systems. Otherwise you will get a lot of bugreports like "this
golang
module doesn't compile since last update" (which is because we didn't
have
proper info in RPM requires).
Actually same here as he is mentioning that he is working on the
packaging automation. Without it IMO would be case for concerns. With it, it is awesome and I'm looking forward to it.
To the original query, how I see it is that our (Build)Requires should structure as accordingly:
BuildRequires: (golang-module(foo) >= 1.0.0 unless (golang-module(foo) = 1.1.5 or golang-module(foo) = 1.2.0))
-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject....
On Wed, Apr 3, 2019 at 7:20 AM Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
s/unless/without/
And then it looks good.
Sorry, you're right. :)
Le mercredi 03 avril 2019 à 07:17 -0500, Neal Gompa a écrit :
To the original query, how I see it is that our (Build)Requires should structure as accordingly:
BuildRequires: (golang-module(foo) >= 1.0.0 unless (golang- module(foo) = 1.1.5 or golang-module(foo) = 1.2.0))
So I should drop the parenthesis around version qualifiers? Works for me
BuildRequires: (golang-module(foo) >= 1.0.0 without (golang-module(foo) = 1.1.5 or golang-module(foo) = 1.2.0))
Trivia time: an interesting twist of the golang module design is that exclusions do not propagate, they only apply in BuildRequires, not Requires
(I'm curious how that will work out in practice, it feels too clever by half, I don't see real-world devs understanding the implications, but I’ll be happy to be proven wrong)
On Tuesday, 2 April 2019 13:27:09 CEST Nicolas Mailhot wrote:
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
I don't intend to maintain X versions of the same Go package, only in some edge case should we have a compat package for an old versions, otherwise we should maintain the latest semver release (v1/v2 etc.. too) or the latest commit.
Le 2019-04-02 15:11, Robert-André Mauchin a écrit :
On Tuesday, 2 April 2019 13:27:09 CEST Nicolas Mailhot wrote:
Le 2019-04-02 12:52, Jakub Cajka a écrit :
I might have not been clear, sorry. My point is more that we don't need to recreate/capture the constrains in the spec files/RPM as they are already capture in the Go source code,
In the Go module world the constrains are in the module files, not in the source files. If you don't satisfy the constrains in your build root lots of go commands will break. So you better tell the rpm tooling what the constrains are so build roots are populated correctly.
I don't intend to maintain X versions of the same Go package, only in some edge case should we have a compat package for an old versions, otherwise we should maintain the latest semver release (v1/v2 etc.. too) or the latest commit.
Just to be clear, the upstream version range with exclusions model is intended to make upgrading to new versions safe. By default all deps are upgradable, an upstream can only exclude a specific set of upgrade versions.
So unless an upstream wants to watch for every new version or commit of the projects he uses, and add them to its exclusion list, it needs to accept they'll be eventually upgraded and the project code better be ready for it.
(not sure how Go projects will react when they realize Go upstream gave them a very short rope to hand themselves with, and exclusions do not lend themselves to blanket version blacklisting).
golang@lists.fedoraproject.org
-
Igor Gnatenko -
Jakub Cajka -
Neal Gompa -
Nicolas Mailhot -
Robert-André Mauchin