Hi,
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation — FPC ticket: https://pagure.io/packaging-committee/issue/719 (without the “hasdraft” tag because I don't know how to add it in pagure) — fedora-rpm-macros RFE with the macro file: https://bugzilla.redhat.com/show_bug.cgi?id=1523779
What it does: conversion of a forge url, version, tag, commit set to the values expected in rpm specfiles, in optional rpm variables. Computation of the corresponding %{dist}.
Objective: centralize forge structure know-how in a single technical place, deprecate all the complex manual forge URL handling spread over many Fedora spec files, simplify packaging and spend time on more interesting stuff.
What's currently implemented: definitions for github.com and code.googlesource.com (I didn't want to propose stuff I didn't use myself. Adding more definitions is trivial. The macros are in Lua which is change-friendly, no arcane rpm syntax knowledge is needed).
Please consult packaging draft for full information.
This is a spin-off of the work I'm currently doing on Go packaging, as Go is heavily forge-oriented. I took the time to extract the generic non-Go-specific forge knowledge in a separate macro file. The macros have been heavily tested on real-life Go projects with quite a lot of variance, on EL7 and rawhide. That's why they come with built-in error handling.
Regards,
On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
Could we be more specific about what "forges" are supported? I see github and googlesource.com in the examples, but I think this is more discoverable for people packaging if we call them out more clearly.
Also, is "forge" synonymous with "git hosting service" as used here? https://fedoraproject.org/wiki/Packaging:SourceURL#Git_Hosting_Services
Ah, here....
What's currently implemented: definitions for github.com and code.googlesource.com
Could open-source solutions pagure.io and gitlab.com be added, please?
De: "Matthew Miller" On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
Also, is "forge" synonymous with "git hosting service" as used here?
Here a "forge" is pretty much anything that allows access to version/commit/tag archives on normalized paths, that can be deduced from project root URL on the "forge" + version/commit/tag/scm info.
What's currently implemented: definitions for github.com and code.googlesource.com
Could open-source solutions pagure.io and gitlab.com be added, please?
Well I currently do not use those, so I wouldn't know what definitions to add :)
It takes maximum 5 min, for someone who knows the structure of a "forge" to add it to the macro, it's just a copy of an existing "if (forge == something) then" block, adaptation of the lua regexps to extract the correct variables from the URL path, and then use of the result in the inner if tag/version/commit block to set rpm variables
The hard part was to define a generic macro structure, with a packager-friendly API, sane rpm variable names, reasonable rule ordering and error handling.
Regards,
On 8.12.2017 20:38, nicolas.mailhot@laposte.net wrote:
De: "Matthew Miller" On Fri, Dec 08, 2017 at 07:03:48PM +0100, nicolas.mailhot@laposte.net wrote:
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
Also, is "forge" synonymous with "git hosting service" as used here?
Here a "forge" is pretty much anything that allows access to version/commit/tag archives on normalized paths, that can be deduced from project root URL on the "forge" + version/commit/tag/scm info.
This is a part of information, that I consider essential. Please put it to the wiki with the explanations and examples.
De: "Miro Hrončok"
On 8.12.2017 20:38, nicolas.mailhot wrote:
De: "Matthew Miller"
Also, is "forge" synonymous with "git hosting service" as used here?
Here a "forge" is pretty much anything that allows access to version/commit/tag archives on normalized paths, that can be deduced from project root URL on the "forge" + version/commit/tag/scm info.
This is a part of information, that I consider essential. Please put it to the wiki with the explanations and examples.
Done, thank you for the feedback
Regards,
De: "Matthew Miller"
Hi
Could open-source solutions pagure.io and gitlab.com be added, please?
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
As far as I've seen pagure.io does not allow direct download of release, tag or commit project archives, so it can not be supported right now. RFE: https://pagure.io/pagure/issue/2845
Regards,
On Sat, Dec 9, 2017 at 3:33 AM, nicolas.mailhot@laposte.net wrote:
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
I made an RFE for this a while back, people can thumbs it up so that it gets more priority: https://gitlab.com/gitlab-org/gitlab-ce/issues/38830
On Sat, Dec 09, 2017 at 09:33:17AM +0100, nicolas.mailhot@laposte.net wrote:
Since I'm a nice person I added GitLab support this morning (both gitlab.com and hosted gitlab). Releases are clearly an afterthought in GitLab, they depend on free-form tags and can't be used in rpm without knowing the corresponding hash (so worst-case, packaging a GitLab release requires declaration of version + tag + commit and not just version as on other forges).
As far as I've seen pagure.io does not allow direct download of release, tag or commit project archives, so it can not be supported right now. RFE: https://pagure.io/pagure/issue/2845
Cool -- thanks for both of these!
On 12/08/2017 08:03 PM, nicolas.mailhot@laposte.net wrote:
Hi,
I am proposing for inclusion a macro set aimed at automating the packaging of forge-hosted projects.
— Packaging draft: https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation — FPC ticket: https://pagure.io/packaging-committee/issue/719 (without the “hasdraft” tag because I don't know how to add it in pagure) — fedora-rpm-macros RFE with the macro file: https://bugzilla.redhat.com/show_bug.cgi?id=1523779
What it does: conversion of a forge url, version, tag, commit set to the values expected in rpm specfiles, in optional rpm variables. Computation of the corresponding %{dist}.
Objective: centralize forge structure know-how in a single technical place, deprecate all the complex manual forge URL handling spread over many Fedora spec files, simplify packaging and spend time on more interesting stuff.
Kudos for work on reducing repetitive complex error prone cruft in specs!
But don't override %setup. There's no need for such abuse (yes overriding rpm build-ins is abuse even if it still doesn't prevent it) and no good will come out of it.
Use something like %forgesetup instead which signifies it's something a bit special, and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
- Panu -
De: "Panu Matilainen"
Hi Panu,
Kudos for work on reducing repetitive complex error prone cruft in specs!
Thanks!
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
And, how many specs need to do that? Especially when the target project uses modern forge hosting? That's already deep rpm black magic land for most packagers.
I can add a guard so the override is only set when %{setupargs} is non empty if you want.
Use something like %forgesetup instead which signifies it's something a bit special
I don't want to signify it's a bit special, that's one more step the packager can do wrong, I'd rather have the common case easy and the complex cases possible.
Granted, the macro makes the complex cases a tad more complex, but if you're already in multiple-setup-call hell, how difficult is it to set %{setupargs} to "" ?
But, I'll bow to whatever FPC decides. *I* won't make the wrong macro call in my specs :).
and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
Interesting, are the changes described somewhere? Not that I want to break compat with el7 from the startup if I can avoid it
Regards,
De: "Panu Matilainen"
Hi Panu,
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
BTW the macro know even more than that, it knows the archive filename for which setup needs specific arguments so it could be made 100% transparent and safe if it was possible to specify
"if archive == %{archivename}%{archiveext} change %setup to %setup %{?setupargs}"
Regards,
On Mon, Dec 11, 2017 at 5:57 AM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
Hi Panu,
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
BTW the macro know even more than that, it knows the archive filename for which setup needs specific arguments so it could be made 100% transparent and safe if it was possible to specify
"if archive == %{archivename}%{archiveext} change %setup to %setup %{?setupargs}"
%setup is a very special macro. It's not actually written in macro language, (and neither is %patch, for that matter). It's really not a good idea to override it, especially if people need to do multi-source things.
And the issue you're having that requires %setupargs is not a problem in RPM 4.14, which ships in Fedora 27 and newer, and will be part of RHEL 8.
If you'd like this to be fixed in EL7 or something, then file a bug report against RHEL 7 and see if anything can be done about it.
Hi Neal,
And the issue you're having that requires %setupargs is not a problem in RPM 4.14
I don't have an issue with %setupargs, I have an issue with requiring packagers to change stuff in the spec header *and* at %prep level, which is not in the same place of the spec. That is something which has wasted huge quantities of man-hours in the past, even for experienced packagers.
The automation knows how the downloaded source archive will be named, what is the structure of the source archive (the arguments that need passing to %setup for this archive). The question is just how to pass that knowledge from the automation macro call to %setup or %autosetup.
Overriding %setup makes this work transparently with little risk. If there is a strong opposition to that what is the best way to achieve the same result?
Using a specific setup-ish macro name like suggested by Panu is trivial technically but has the huge drawback that it requires a specific call by the packager (and many will forget it, at least as first). So it de-optimizes the packager workflow. I'd frankly prefer to optimize the packager workflow over helping tooling – that's what costs actual money and potential contributors.
If there is now way to do it cleanly or safely in rpm, I'll de-optimize the packager side. I don't want to cause problems to anyone. But that would be pretty sad.
Regards,
On 12/11/2017 12:51 PM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
Hi Panu,
Kudos for work on reducing repetitive complex error prone cruft in specs!
Thanks!
But don't override %setup. There's no need for such abuse
It is really pretty safe, the macro controls the downloaded file, the file structure is known, the only time it won't "just work" is when a spec needs to call %setup several times (in that case the arguments will be wrong for the second call).
No. It's not about whether it's "pretty safe" or not, it's an rpm built-in and it's simply not yours to override (and what about the next guy who wants to override it for their own purposes, say, go-specific thing?). The fact that you can do so is basically a loophole in rpm that just hasn't gotten closed, yet. Don't bet the future on loopholes.
- Panu -
On 12/11/2017 12:51 PM, nicolas.mailhot@laposte.net wrote:
De: "Panu Matilainen"
and allows you to %autosetup underneath on versions where macro arguments are expanded (rpm >= 4.14)
Interesting, are the changes described somewhere? Not that I want to break compat with el7 from the startup if I can avoid it
Sorry, forgot to reply to this part.
http://rpm.org/wiki/Releases/4.14.0#Macros has the bullet points, other than that it's pretty scarce.
https://bugzilla.redhat.com/show_bug.cgi?id=1397209 is the original case with pointers to upstream commit etc.
- Panu -
packaging@lists.fedoraproject.org
-
Matthew Miller -
Miro Hrončok -
Neal Gompa -
nicolas.mailhot＠laposte.net -
Panu Matilainen