Proposed Fedora packaging guideline: More Go packaging

Hi,

I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.

- Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging
- https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721

This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.

What it does:

- drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
- simple, packager-friendly spec syntax
- automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
- working Go autoprovides. No forgotten provides anymore.
- working Go autorequires. No forgotten requires anymore.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
- easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
- guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
- centralized Go macros that can be audited and enhanced over time.
- aggressive leverage of upstream unit tests to detect quickly broken code.

Please consult packaging draft for full information.

The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.

I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft

Regards,

[Neal Gompa]

On Sun, Dec 17, 2017 at 2:11 AM,   wrote:
> Hi,
>
> I am proposing for inclusion a set of rpm technical files aimed at automating the packaging of forge-hosted projects.
> Packaging draft: https://fedoraproject.org/wiki/More_Go_packaging
> https://pagure.io/packaging-committee/issue/734
> go-srpm-macros RFE with the technical files: https://bugzilla.redhat.com/show_bug.cgi?id=1526721
>
> This proposal is integrated with and depends on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft
> It builds on the hard work of the Go SIG and reuses the rpm automation of https://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and produces compatible packages.
>
> What it does:
> drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
> simple, packager-friendly spec syntax
> automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
> working Go autoprovides. No forgotten provides anymore.
> working Go autorequires. No forgotten requires anymore.
> strict automated directory ownership (used by autorequires and autoprovides).
> centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
> easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
> guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
> guidelines-compliant bootstraping.
> systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
> flexibility, do the right thing transparently by default, leave room for special cases and overrides.
> no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
> centralized Go macros that can be audited and enhanced over time.
> aggressive leverage of upstream unit tests to detect quickly broken code.
>
> Please consult packaging draft for full information.
>
> The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.
>
> I hope posting the second part of the automation will answer some questions people had on the https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation draft

I really do like this. There are only two issues I have with it:

1. This seems to mandate that all packages must be named by their
import path. My golang package (snapd) is not, intentionally so. I
don't want to change this.

2. Mandating a forge is going to be tricky for self-hosted stuff, or
people who release Go code as tarballs (it's rare, but it happens).
How do you deal with that?

[Dridi Boukelmoune]

> I really do like this. There are only two issues I have with it:
> This seems to mandate that all packages must be named by their
>
> import path. My golang package (snapd) is not, intentionally so. I
> don't want to change this.
> Mandating a forge is going to be tricky for self-hosted stuff, or
>
> people who release Go code as tarballs (it's rare, but it happens).
> How do you deal with that?

By not using the macros for packages not fitting the model?

I think this is very helpful especially when it's the common practice,
and I certainly won't blame anyone doing proper releases and not
just a git tag with github releases notes ;)

Regarding naming, I think python packages must be prefixed with
python[23]- and can Provides: the upstream project name. On the
other hand we have packages like docker that are clearly named
after upstream's name, so I don't think that would be a problem for
snapd. (and maybe an exception needs to be granted?)

Dridi

[Neal Gompa]

On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
 wrote:
> > I really do like this. There are only two issues I have with it:
> > This seems to mandate that all packages must be named by their
> >
> > import path. My golang package (snapd) is not, intentionally so. I
> > don't want to change this.
> > Mandating a forge is going to be tricky for self-hosted stuff, or
> >
> > people who release Go code as tarballs (it's rare, but it happens).
> > How do you deal with that?
>
> By not using the macros for packages not fitting the model?

The issue is that the new Go macros are tightly wound into the forge
macros. I just want to be sure that we can leverage things like the
dependency generators without all the other stuff.
> I think this is very helpful especially when it's the common practice,
> and I certainly won't blame anyone doing proper releases and not
> just a git tag with github releases notes ;)
>
> Regarding naming, I think python packages must be prefixed with
> python[23]- and can Provides: the upstream project name. On the
> other hand we have packages like docker that are clearly named
> after upstream's name, so I don't think that would be a problem for
> snapd. (and maybe an exception needs to be granted?)

This rule only applies to Python packages that have modules that are
designed to be imported by other Python code. Otherwise, this is not
necessary.

[Marcin Dulak]

On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa  wrote:
> On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
>  wrote:
> > > I really do like this. There are only two issues I have with it:
> > > This seems to mandate that all packages must be named by their
> > >
> > > import path. My golang package (snapd) is not, intentionally so. I
> > > don't want to change this.
> > > Mandating a forge is going to be tricky for self-hosted stuff, or
> > >
> > > people who release Go code as tarballs (it's rare, but it happens).
> > > How do you deal with that?
> >
> > By not using the macros for packages not fitting the model?
>
> The issue is that the new Go macros are tightly wound into the forge
> macros. I just want to be sure that we can leverage things like the
> dependency generators without all the other stuff.
> > I think this is very helpful especially when it's the common practice,
> > and I certainly won't blame anyone doing proper releases and not
> > just a git tag with github releases notes ;)
> >
> > Regarding naming, I think python packages must be prefixed with
> > python[23]- and can Provides: the upstream project name.

I'm not sure if this matters in this discussion but an example Python3 part
of a spec file https://fedoraproject.org/wiki/Packaging:Python
to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with
python34 and not python3) would look like:

%package -n python%{python3_pkgversion}-%{pname}
%{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}

Macin
> On the
> > other hand we have packages like docker that are clearly named
> > after upstream's name, so I don't think that would be a problem for
> > snapd. (and maybe an exception needs to be granted?)
>
> This rule only applies to Python packages that have modules that are
> designed to be imported by other Python code. Otherwise, this is not
> necessary.
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> packaging mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Jakub Cajka]

----- Original Message -----
> From: "Marcin Dulak" 
> To: "Discussion of RPM packaging standards and practices for Fedora" 
> Cc: [e-mail], "Development discussions related to Fedora" 
> Sent: Monday, January 22, 2018 4:04:19 PM
> Subject: Re: [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
>
> On Mon, Jan 22, 2018 at 2:45 PM, Neal Gompa < [e-mail] > wrote:
>
> On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
> < [e-mail] > wrote:
> > > I really do like this. There are only two issues I have with it:
> > > This seems to mandate that all packages must be named by their
> > >
> > > import path. My golang package (snapd) is not, intentionally so. I
> > > don't want to change this.
> > > Mandating a forge is going to be tricky for self-hosted stuff, or
> > >
> > > people who release Go code as tarballs (it's rare, but it happens).
> > > How do you deal with that?
> >
> > By not using the macros for packages not fitting the model?
>
> The issue is that the new Go macros are tightly wound into the forge
> macros. I just want to be sure that we can leverage things like the
> dependency generators without all the other stuff.
> > I think this is very helpful especially when it's the common practice,
> > and I certainly won't blame anyone doing proper releases and not
> > just a git tag with github releases notes ;)
> >
> > Regarding naming, I think python packages must be prefixed with
> > python[23]- and can Provides: the upstream project name.
>
> I'm not sure if this matters in this discussion but an example Python3 part
> of a spec file https://fedoraproject.org/wiki/Packaging:Python
> to accommodate also EPEL (which on CentOS7 prefixes Python3 packages with
> python34 and not python3) would look like:
>
> %package -n python%{python3_pkgversion}-%{pname}
> %{?python_provide:%python_provide python%{python3_pkgversion}-%{pname}}
>
> Macin

Hopefully something like this will never happen as generally I'm strongly against shipping multiple versions(of one implementation) of Go concurrently.

JC
> On the
> > other hand we have packages like docker that are clearly named
> > after upstream's name, so I don't think that would be a problem for
> > snapd. (and maybe an exception needs to be granted?)
>
> This rule only applies to Python packages that have modules that are
> designed to be imported by other Python code. Otherwise, this is not
> necessary.
>
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> packaging mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

<DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Neal Gompa"
> On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
> > > I really do like this. There are only two issues I have with it:
> > > This seems to mandate that all packages must be named by their
> > >
> > > import path. My golang package (snapd) is not, intentionally so. I
> > > don't want to change this.
> > > Mandating a forge is going to be tricky for self-hosted stuff, or
> > >
> > > people who release Go code as tarballs (it's rare, but it happens).
> > > How do you deal with that?
> >
> > By not using the macros for packages not fitting the model?
> The issue is that the new Go macros are tightly wound into the forge
> macros. I just want to be sure that we can leverage things like the
> dependency generators without all the other stuff.

Hi Neal,

I should probably not let this pass without clarifying:

1.  if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't:
  — they include code to disable transparently and without exploding the associated bits if the hosting site is unknown
  — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work
  — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,

2. if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me

3. if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.

The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:

1. the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits

2. the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require

3. the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)

All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.

For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion  (and review by a human to check no problem code is deployed) first.

Regards,

<DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

De: "Neal Gompa"
> > The issue is that the new Go macros are tightly wound into the forge
> > macros. I just want to be sure that we can leverage things like the
> > dependency generators without all the other stuff.

Hi Neal,
> I should probably not let this pass without clarifying:
> 3. if your concern is that they won't ever be merged

And the forge macros are now available since redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream renaming the file). Heartfelt thanks to  Jason Tibbitts !

I'll focus on pushing the Go parts to go-srpm-macros now so everyone can play with them

Regards,

[Jason L Tibbitts III]

I wish this message wasn't crossposted everywhere, but I don't want to
lose any discussion by trimming the CC list.  Sorry if replies generate
bounces for some.
> > > > > "nm" == nicolas mailhot  writes:

nm> And the forge macros are now available since
nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream
nm> renaming the file). Heartfelt thanks to Jason Tibbitts !

Please don't forget to let me know when it's time to start thinking
about pushing this down to F27.  And maybe F26.  And as far as I can
tell it should work with only minor modification in EPEL7 (via
epel-rpm-macros).  I don't know about EPEL6, but we really should look
at it given some of the other discussions about specfile compatibility.
Some packagers wouldn't ever use it if it doesn't work everywhere.

Finally, we should also talk about whether there is any integration or
automation possible between fedpkg and specfiles configured with these
macros.

- J<

----- Mail original -----
De: "Jason L Tibbitts III"
> > > > > "nm" == nicolas mailhot  writes:
> nm> And the forge macros are now available since
> nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream
> nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
> Please don't forget to let me know when it's time to start thinking
> about pushing this down to F27.  And maybe F26.  And as far as I can
> tell it should work with only minor modification in EPEL7 (via
> epel-rpm-macros).

I don't know about EPEL6, but we use it as-is in EL7 and it works just as well (except maybe for the %autosetup bits but IIRC that's autosetup which is broken in EL7). In fact it has probably been used more heavily in EL7 than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never tried it. I guess it depends mostly on the level of lua support in EL6 rpm and rpm-related tools now the forge macro code is lua only. I'm pretty sure many of the problems in the early versions of the macro were due to non-lua code and its interactions with lua code once the lua-ification started.

It's a good idea to let people play with it in fedora-devel maybe a month, in case I missed something, but from a technical POW I'm prety sure it could be merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just like I submitted bitkeeper.org support today), right now the code is distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it for Go and the EL6 Go stack is really too old for a merge to be interesting. Anyway I'll certainly let you know when I feel the time is right (but do not block on me!)
> Finally, we should also talk about whether there is any integration or
> automation possible between fedpkg and specfiles configured with these
> macros.

I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of any use there. Though I'm not opposed to the idea at all.

Thanks again!

[Jason L Tibbitts III]

> > > > > "nm" == nicolas mailhot  writes:

nm> I don't know about EPEL6, but we use it as-is in EL7 and it works
nm> just as well (except maybe for the %autosetup bits but IIRC that's
nm> autosetup which is broken in EL7).

I had ported autosetup to EPEL6 and then at the next release the macros
showed up (without any discussion) in base RHEL6.  So they should be
there.  I'm not entirely sure what's actually broken about %autosetup in
EL7, though; I hadn't heard about breakage before this.  epel-rpm-macros
could conceivably carry %epel_autosetup or something which contains
fixes.  I mostly understand the internals of %autosetup so maybe there's
something I can do.

nm> Maybe it also works in EPEL 6 but I've never tried it. I guess it
nm> depends mostly on the level of lua support in EL6 rpm and
nm> rpm-related tools now the forge macro code is lua only.

I think it would be worth a try.  In my experience not all that much has
changed in the Lua interfaces since RHEL6 and even RHEL5.  (EL5 just
didn't have sources and patches in the Lua namespace.)

nm> For my part I doubt I'll ever use it in EL6 since I did it for Go
nm> and the EL6 Go stack is really too old for a merge to be
nm> interesting.

Well, sure, Go on EL6 is probably out but these macros were at least
presented as being far more general, and I'm sure there are plenty of
EPEL6 packages which could benefit.  Potential anything that packages a
git snapshot of something.

nm> I'm afraid my knowledge of recent fedpkg enhancements is too sparse
nm> to be of any use there. Though I'm not opposed to the idea at all.

It's just worth a brainstorm, I think.  I can imagine that loads of
people would love it if fedpkg could just auto-bump the bits necessary
to update a package to today's git head or some specific commit or
something like that.  Before we didn't really have a good standard way
to format a spec when you're using a checkout.  Now....

Doesn't necessarily have to start in fedpkg, either.  A standalone
utility for doing a couple of things like that could be useful as a
prototype.

- J<

[Jakub Cajka]

----- Original Message -----
> From: "nicolas mailhot" 
> To: "Jason L Tibbitts III" 
> Cc: [e-mail], "Development discussions related to Fedora" ,
> "Discussion of RPM packaging standards and practices for Fedora" 
> Sent: Tuesday, January 23, 2018 9:28:15 PM
> Subject: Re: Proposed Fedora packaging guideline: More Go packaging
>
> ----- Mail original -----
> De: "Jason L Tibbitts III"
> > > > > > "nm" == nicolas mailhot  writes:
> > nm> And the forge macros are now available since
> > nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream
> > nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
> > Please don't forget to let me know when it's time to start thinking
> > about pushing this down to F27.  And maybe F26.  And as far as I can
> > tell it should work with only minor modification in EPEL7 (via
> > epel-rpm-macros).
>
> I don't know about EPEL6, but we use it as-is in EL7 and it works just as
> well (except maybe for the %autosetup bits but IIRC that's autosetup which
> is broken in EL7). In fact it has probably been used more heavily in EL7
> than in fedora-devel so far. Maybe it also works in EPEL 6 but I've never
> tried it. I guess it depends mostly on the level of lua support in EL6 rpm
> and rpm-related tools now the forge macro code is lua only. I'm pretty sure
> many of the problems in the early versions of the macro were due to non-lua
> code and its interactions with lua code once the lua-ification started.
>
> It's a good idea to let people play with it in fedora-devel maybe a month, in
> case I missed something, but from a technical POW I'm prety sure it could be
> merged up to EL7 now. I'll submit fedora-devel specific tweaks later (just
> like I submitted bitkeeper.org support today), right now the code is
> distro-agnostic. For my part I doubt I'll ever use it in EL6 since I did it
> for Go and the EL6 Go stack is really too old for a merge to be interesting.
> Anyway I'll certainly let you know when I feel the time is right (but do not
> block on me!)

If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its upstream support), although Go packaging macros are missing.

JC
> > Finally, we should also talk about whether there is any integration or
> > automation possible between fedpkg and specfiles configured with these
> > macros.
>
> I'm afraid my knowledge of recent fedpkg enhancements is too sparse to be of
> any use there. Though I'm not opposed to the idea at all.
>
> Thanks again!
>
> --
> Nicolas Mailhot
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

----- Mail original -----
De: "Jakub Cajka"

Hi Jakub,
> > For my part I doubt I'll ever use it in EL6 since I did it
> > for Go and the EL6 Go stack is really too old for a merge to be interesting.
> > Anyway I'll certainly let you know when I feel the time is right (but do not
> > block on me!)
> If we are talking about EPEL6 stack, it is fairly fresh(1.9.2) and stable(it will be on 1.9 for whole of its 
> upstream support), although Go packaging macros are missing.

Yes the core golang package is there but first the other Go macros are not available in EL6 and second the software level of many common Go packages is very old in EL6.

So, assuming the forge macros just work in EL6, because the little bit of rpm lua they need works the same as in EL7 and devel, updating EL6 to the level needed for a Go spec to be shared mostly unchanged between EL6/EL7 and devel would require:

— making the other Go macros available in EL6 (and I'm far from sure they would work unchanged in EL6, the forge part is easy since it's 100% lua but the other parts exercise the shell and rpm and other stuff that may behave in a slightly different way in such an ancient codebase. Not necessarily a show-stopper but definitely something that requires testing and adjusting time by someone)

— updating all the common Go software packages to the same level as in devel. I don't say it can't be done (after all I'm doing it for EL7 on hundreds of packages) but that's a *lot* more work than just updating a few macro files. Especially if it hits bootstraping issues not existing in EL7 and devel.

Therefore, it requires someone with time and motivation, and should really not be attempted before EL7 and devel are done and things have settled a bit, and by that point will there still be enough interest in upgrading the Go state in EL6 for it to be worth the pain?

Of course one could limit oneself to making the macros available (which still requires some testing and eventually some code porting), that would enable using the same Go spec coding style in EL6 EL7 and devel, if not sharing the spec themselves, but the interest of autodeps is severely limited, if you do not have a large baseline of packages providing the deps (and the dep version) other packages need to build.

That's why I wrote the forge part could be made available in EL6, it depends on little except the lua built in rpm, and can be useful as-is, while the Go part is something else entirely, as its utility is directly linked to the quantity and freshness of Go software packages in the distro.

Regards,

[Nicolas Mailhot]

Hi Jason,
> > "nm" == nicolas mailhot wrote:
>
> Jason L Tibbitts wrote:
> > nm> And the forge macros are now available since
> > nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to
> > upstream
> > nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
> > Please don't forget to let me know when it's time to start thinking
> > about pushing this down to F27.  And maybe F26.  And as far as I can
> > tell it should work with only minor modification in EPEL7 (via
> > epel-rpm-macros).
>
> I don't know about EPEL6, but we use it as-is in EL7 and it works just
> as well (except maybe for the %autosetup bits but IIRC that's
> autosetup which is broken in EL7).
>
> It's a good idea to let people play with it in fedora-devel maybe a
> month, in case I missed something, but from a technical POW I'm prety
> sure it could be merged up to EL7 now.

Now that the appropriate observation time has passed, and people have
not find new problems (in fact they found scenarii that worked I had not
thought about) can you please merge

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/18

and push it to stable?

The Go macros need it and Fedora Go maintainers now want to fork
forgemeta if redhat-rpm-config does not make forgemeta available in the
Fedora stable versions they target, which would all make a huge mess. I
guess it's been too successful ;).

Regards,

[Jakub Cajka]

----- Original Message -----
> From: "Jason L Tibbitts III" 
> To: "nicolas mailhot" 
> Cc: [e-mail], "Development discussions related to Fedora" ,
> "Discussion of RPM packaging standards and practices for Fedora" 
> Sent: Tuesday, January 23, 2018 6:00:24 PM
> Subject: Re: Proposed Fedora packaging guideline: More Go packaging
>
> I wish this message wasn't crossposted everywhere, but I don't want to
> lose any discussion by trimming the CC list.  Sorry if replies generate
> bounces for some.
> > > > > > "nm" == nicolas mailhot  writes:
>
> nm> And the forge macros are now available since
> nm> redhat-rpm-config-73-1.fc28 (I had missed the push due to upstream
> nm> renaming the file). Heartfelt thanks to Jason Tibbitts !
>
> Please don't forget to let me know when it's time to start thinking
> about pushing this down to F27.  And maybe F26.  And as far as I can
> tell it should work with only minor modification in EPEL7 (via
> epel-rpm-macros).  I don't know about EPEL6, but we really should look
> at it given some of the other discussions about specfile compatibility.
> Some packagers wouldn't ever use it if it doesn't work everywhere.

I think that it would be great to land it also in the EPEL6/7.

JC
> Finally, we should also talk about whether there is any integration or
> automation possible between fedpkg and specfiles configured with these
> macros.
> J<
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

<DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

[Neal Gompa]

On Tue, Jan 23, 2018 at 5:45 AM,   wrote:
> ----- Mail original -----
> De: "Neal Gompa"
> > On Mon, Jan 22, 2018 at 8:33 AM, Dridi Boukelmoune
> > > > I really do like this. There are only two issues I have with it:
> > > > This seems to mandate that all packages must be named by their
> > > >
> > > > import path. My golang package (snapd) is not, intentionally so. I
> > > > don't want to change this.
> > > > Mandating a forge is going to be tricky for self-hosted stuff, or
> > > >
> > > > people who release Go code as tarballs (it's rare, but it happens).
> > > > How do you deal with that?
> > >
> > > By not using the macros for packages not fitting the model?
> > The issue is that the new Go macros are tightly wound into the forge
> > macros. I just want to be sure that we can leverage things like the
> > dependency generators without all the other stuff.
>
> Hi Neal,
>
> I should probably not let this pass without clarifying:
> if your concern is that the *forge* macros will block packaging of projects not hosted on known forges, they won't:
>
> — they include code to disable transparently and without exploding the associated bits if the hosting site is unknown
>   — the packager can pre-define the variables that would have been computed if the site was known and all the rest of the automation will just work
>   — baring that the packager can ignore the site-related computations and use whatever he wants instead in Source: and *setup, which are the only parts that depend on hosting structure knowledge,

This was my main concern. There are plenty of people who don't like
any of the known forges. And Go ecosystem incompetence for mandating
URI-style deps and build paths doesn't really stop people from wanting
to do that.
> if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different rpm tools and they are rock solid for me

I don't consider them defective. They make me a bit squeamish because
I don't see the spec preamble and such because they're autogenerated,
but you've mentioned that it's possible to selectively use these
things with Go and forge macros.

SUSE does something similar with Python for their "singlespec" part,
and it's fantastic[1][2]. But I'm slightly uncomfortable with the idea
that there's no preamble *at all* in the spec using the Go macros and
it's generated fully by macros.

[1]: https://github.com/openSUSE/python-rpm-macros
[2]: https://build.opensuse.org/package/view_file/openSUSE:Factory/python-inipars...
> if your concern is that they won't ever be merged due to some people non-constructive obstruction, well I share it to some point but I won't spend more time arguing unicorns and what-if-maybe I-don't-like-commits-let's-make-them-awful-to-use. The people blocking there are free to take up the maintenance of my ~400 specs and redo them as they want, we'll see how they love to waste tens of minutes per spec to position setup args manually when they have to do it for hundreds of them, every time there is a package update or a forge relayout.

Technically, no one has to "accept" anything for the macros to be
available. It's whether the macros would be broadly used. I could, in
fact, tomorrow, make various extra useful macros that I use available
in a package for people to pull in. It doesn't mean anyone has to use
them.
> The main difficulty with the proposal is not the *forge part, it's that automated autodeps are well, automated. As every automation they are strict and unyielding and do not take approximations well. They will fail spectacularly in the following cases:
>
> the packager does not package all the deps of its code → the resulting package is uninstallable because autodeps add requires on missing bits
>
> the packager packages code with garbage imports (very common in example or test code) → the resulting package is uninstallable because autodeps add require
>
> the packager does not own properly the directories where its Go code is installed → given that Go deps depend on directories, Go autodeps also depend on proper directory ownership. Positioning autodeps on Go files themselves would make package build times increase many many times as rpm would try to compute the same autodeps for every single .go file in a directory. (and Go autodeps are already sloooow as snails because invoking the go command is slow)
>
> All those things are mitigated by the use of %goinstall that filters usual example/test/vendor dirs so they don't trigger autodeps, and tries very hard to own all relevant directories.
>
> For those reasons I don't propose to activate autodeps in old-style golang packages. They need conversion  (and review by a human to check no problem code is deployed) first.

As long as I can do Obsoletes/Provides for the old name for the devel,
unit-test, etc. stuff so the old names work, it won't bother me too
much. I want people to be able to do "snapd-devel" or whatnot and get
the right thing. Though I'm not sure why package name enforcement is
required for autodeps, since that's based on file lists passed to a
dep generator...

<DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Neal Gompa"
> > if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
> >
> > first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different
> > rpm tools and they are rock solid for me
> I don't consider them defective. They make me a bit squeamish because
> I don't see the spec preamble and such because they're autogenerated,
> but you've mentioned that it's possible to selectively use these
> things with Go and forge macros.

I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.

For example, what are you missing there?

%<---
%global goipath        github.com/hashicorp/consul
Version:               1.0.2

%gometa

%global common_description %{expand:
Consul is a tool for service discovery […]}

Name:    consul
Release: 5%{?dist}
Summary: A tool for easy service discovery, monitoring and configuration
License: MPLv2.0
URL:     https://www.consul.io/
Source:  %{gosource}
%<---

Or here ?

%<---
%global goipath        github.com/hashicorp/go-sockaddr
%global commit         9b4c5fa5b10a683339a270d664474b9f4aee62fc

%gometa

%global common_description %{expand:
Socket address convenience functions for Go. go-sockaddr is a convenience
library…}

Name:    %{goname}
Version: 0
Release: 0.14%{?dist}
Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets
License: MPLv2.0
URL:     %{gourl}
Source:  %{gosource}
%<---

Regards,

<DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

[Neal Gompa]

On Tue, Jan 23, 2018 at 8:54 AM,   wrote:
> ----- Mail original -----
> De: "Neal Gompa"
> > > if your concern is that the *forge* macros are defective somewhere I'd be curious where as you'd be the
> > >
> > > first to report an actual technical problem. I've used them intensively in rawhide and el7 with many different
> > > rpm tools and they are rock solid for me
> > I don't consider them defective. They make me a bit squeamish because
> > I don't see the spec preamble and such because they're autogenerated,
> > but you've mentioned that it's possible to selectively use these
> > things with Go and forge macros.
>
> I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all converted to variable use now to avoid packager surprise and permit customization.
>
> For example, what are you missing there?
>
> %<---
> %global goipath        github.com/hashicorp/consul
> Version:               1.0.2
>
> %gometa
>
> %global common_description %{expand:
> Consul is a tool for service discovery […]}
>
> Name:    consul
> Release: 5%{?dist}
> Summary: A tool for easy service discovery, monitoring and configuration
> License: MPLv2.0
> URL:     https://www.consul.io/
> Source:  %{gosource}
> %<---
>
> Or here ?
>
> %<---
> %global goipath        github.com/hashicorp/go-sockaddr
> %global commit         9b4c5fa5b10a683339a270d664474b9f4aee62fc
>
> %gometa
>
> %global common_description %{expand:
> Socket address convenience functions for Go. go-sockaddr is a convenience
> library…}
>
> Name:    %{goname}
> Version: 0
> Release: 0.14%{?dist}
> Summary: A Go convenience library to manipulate IP Addresses and UNIX sockets
> License: MPLv2.0
> URL:     %{gourl}
> Source:  %{gosource}
> %<---

Actually, this is fine with me. This is what disturbed me a bit:
https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...

<DKIM> Re: Re: <DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

> ----- Mail original -----
> De: "Neal Gompa"
> > I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values
> > set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all
> > converted to variable use now to avoid packager surprise and permit customization.
> Actually, this is fine with me. This is what disturbed me a bit:
> https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...

Ah, yes the examples snip the lines that do not need specific changes for readability
Is the … not clear enough?
I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in

<DKIM> Re: Re: <DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

[Neal Gompa]

On Tue, Jan 23, 2018 at 9:40 AM,   wrote:
> > ----- Mail original -----
> > De: "Neal Gompa"
> > > I'm curious, what are you missing in the preamble ? As far as I can see it's all there (even though some values
> > > set to variables %gometa precomputes). I had it's right autogenerated some parts of it in the past but it's all
> > > converted to variable use now to avoid packager surprise and permit customization.
> > Actually, this is fine with me. This is what disturbed me a bit:
> > https://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation#Pa...
>
> Ah, yes the examples snip the lines that do not need specific changes for readability
> Is the … not clear enough?
> I fear that putting a full preamble would be even more confusing for some readers. Though if people disagree, I'l try to find some time to add the skipped lines back in

For snipping, use "[...]" notation to indicate skipped stuff. It's
hard to tell otherwise.

<DKIM> Re: <DKIM> Re: Re: <DKIM> Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Neal Gompa"
> For snipping, use "[...]" notation to indicate skipped stuff. It's
> hard to tell otherwise.

Ok, that was easy to fix :)

<DKIM> Re: Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Neal Gompa"
> As long as I can do Obsoletes/Provides for the old name for the devel,
> unit-test,

BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)

To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?

Regards,

<DKIM> Re: Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

[Neal Gompa]

On Tue, Jan 23, 2018 at 9:00 AM,   wrote:
> ----- Mail original -----
> De: "Neal Gompa"
> > As long as I can do Obsoletes/Provides for the old name for the devel,
> > unit-test,
>
> BTW is anyone using the unit-test packages? Right now I do not generate them, I don't need them, and making them work with autodeps would be hairy (deploying without autodeps should be trivial however)
>
> To be honest, given all the parts current packages fail to install, I'd expect many of the current unit test packages to fail in mysterious ways, so I'm curious: what use has been found for them?

No idea, but I just checked, and I don't even have the unit-test
subpackage enabled on snapd, so I don't particularly care too much
there...

<DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Neal Gompa"

Hi,

Thanks for the review !
> I really do like this. There are only two issues I have with it:
> This seems to mandate that all packages must be named by their
>
> import path. My golang package (snapd) is not, intentionally so. I
> don't want to change this.

Well that was not the intent, I probably need to clarify things a bit more. The *devel* package must absolutely be named after the import path (for sanity sake). The core package can be named something else (usually because it's the packaging of an app)

For example in my etcd spec

Name:    etcd ← well-known app name
%package   -n %{goname}-devel ← go naming for go devel stuff
> Mandating a forge is going to be tricky for self-hosted stuff, or
>
> people who release Go code as tarballs (it's rare, but it happens).
> How do you deal with that?

The proposal automates integration with forges, it does not mandate it

You can declare manually archivename archiveurl and forgesetup before calling the %gometa macro and 
you should benefit from the rest of the automation even on an hosting site it does not know anything about. The whole code tries very hard to let the packager pre-define everything it may guess wrong or not know about to avoid giving up on all the automation just because a little part is wrong.

And even after that you can still declare Source manually and pass arguments to setup the old way it that's what you want

Not calling %gometa at all will kill stuff like goname which is kind of mandatory for consistency.

Regards,

Hi,

Now the technical PR is submitted
https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1

and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:

https://fedoraproject.org/wiki/More_Go_packaging

I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.

Then I will push it FPC side again.

Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.

Regards,

[Neal Gompa]

On Tue, Jan 30, 2018 at 10:11 AM,   wrote:
> Hi,
>
> Now the technical PR is submitted
> https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
>
> and waiting for action from the go-srpm-macros maintainers, I took (quite a long) time to refresh and flesh out the corresponding packaging guidelines proposal. It should be fairly complete now:
>
> https://fedoraproject.org/wiki/More_Go_packaging
>
> I'd appreciate review to check if I have forgotten an important case, if people understand the text, if they have enhancements or corrections to propose, and so on.
>
> Then I will push it FPC side again.
>
> Actual practice should be fairly simple and self-explanatory, the proposal length can be scary but that's because it documents all kinds of corner cases that required digging through specs and mailing lists to find resolution examples before. The basic Go packaging skeleton will be sufficient is most cases without requiring to read any documentation.

The only thing I see that might be missing is autogenerating
bundled(golang()) Provides when a vendor tree exists (with the
appropriate automatic filters on Requires).

<DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

> De: "Neal Gompa"
> The only thing I see that might be missing is autogenerating
> bundled(golang()) Provides when a vendor tree exists (with the
> appropriate automatic filters on Requires).

I had though a little about doing it but first, as many Go elements, vendoring relies on conventions not standards. The nasty thing about conventions is that they are not applied 100% the same way by everyone, making automation a PITA. And second interactions with autodeps can be nasty: you can filter out provides, but do you filter out requires? What about all the junk code many projects ship as testing and examples and which is vendored with the rest?

I don't say it can't be done, or that it would be difficult to do once the rest is merged, but I'll live it to someone that absolutely want to ship a Go package with vendored parts.

Right now we sidestep the issue in our packages by rm -fr ing anything that looks like vendored code in %prep. Unbundling takes time but it has a positive cumulative effect: the more you unbundle the less you need to worry about in other packages with the same requirements. And unbundling reveals code/legal problems that it would take about as much work to solve by auditing the vendored code manually.

Regards,

[Robert-André Mauchin]

On mardi 30 janvier 2018 16:11:49 CET [e-mail] wrote:
> Hi,
>
> Now the technical PR is submitted
> https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
>
> and waiting for action from the go-srpm-macros maintainers, I took (quite a
> long) time to refresh and flesh out the corresponding packaging guidelines
> proposal. It should be fairly complete now:
> https://fedoraproject.org/wiki/More_Go_packaging
>
> I'd appreciate review to check if I have forgotten an important case, if
> people understand the text, if they have enhancements or corrections to
> propose, and so on.
> Then I will push it FPC side again.
>
> Actual practice should be fairly simple and self-explanatory, the proposal
> length can be scary but that's because it documents all kinds of corner
> cases that required digging through specs and mailing lists to find
> resolution examples before. The basic Go packaging skeleton will be
> sufficient is most cases without requiring to read any documentation.
> Regards,
>
> -- 
> Nicolas Mailhot
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

Hello Nicolas,

Could you add two full examples, one for binary package, one for library 
package, like in https://fedoraproject.org/wiki/PackagingDrafts/
Go#Sample_RPM_Spec

Putting it all together (https://fedoraproject.org/wiki/
More_Go_packaging#Putting_it_all_together) is nice but don't understand if it 
is for a binary or a library.

Also, is there any plan to update Gofed with these new guidelines? If it's not 
automated like today, we will lose time instead of gaining some in the end.

Best regards,

Robert-André

[Nicolas Mailhot]

Hi Robert André

That's an interesting request

I guess you can't figure if the example is for building binaries or Go libs, because there is no hard frontier between both cases in the proposed guidelines

In Go, everything is effectively a code library that can be reused elsewhere. So the approach taken in the proposed guidelines, is that everything is a code library, some of which happen to build binaries, or setup compatibility symlinks, or need bootstrapping

It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The example was chosen to include as many blocks as possible, with the walkthrough explaining their respective functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old guidelines

<DKIM> Re: Proposed Fedora packaging guideline: More Go packaging

----- Mail original -----
De: "Nicolas Mailhot"
> It's a bit of a Lego guideline, you assemble the spec blocs you need, and ignore those you don't need. The
> example was chosen to include as many blocks as possible, with the walkthrough explaining their respective
> functions. All the blocks are very short and succinct, don't search for long runs of shell code like in the old
> guidelines

That being said if people feel it's useful I can add a barebones lib-only example with none of the optional blocks in. However, I fear that the more ways the page explains the same stuff, the more confusing and difficult to maintain it will be.

So, can people chime in to say whether they feel a supplementary example is useful of counter productive?

Regards,

[Jakub Cajka]

----- Original Message -----
> From: "Robert-André Mauchin" 
> To: [e-mail]
> Cc: [e-mail], "Discussion of RPM packaging standards and practices for Fedora"
>
> Sent: Friday, February 2, 2018 4:54:10 PM
> Subject: Re: Proposed Fedora packaging guideline:  More Go packaging
>
> On mardi 30 janvier 2018 16:11:49 CET [e-mail] wrote:
> > Hi,
> >
> > Now the technical PR is submitted
> > https://src.fedoraproject.org/rpms/go-srpm-macros/pull-request/1
> >
> > and waiting for action from the go-srpm-macros maintainers, I took (quite a
> > long) time to refresh and flesh out the corresponding packaging guidelines
> > proposal. It should be fairly complete now:
> > https://fedoraproject.org/wiki/More_Go_packaging
> >
> > I'd appreciate review to check if I have forgotten an important case, if
> > people understand the text, if they have enhancements or corrections to
> > propose, and so on.
> > Then I will push it FPC side again.
> >
> > Actual practice should be fairly simple and self-explanatory, the proposal
> > length can be scary but that's because it documents all kinds of corner
> > cases that required digging through specs and mailing lists to find
> > resolution examples before. The basic Go packaging skeleton will be
> > sufficient is most cases without requiring to read any documentation.
> > Regards,
> >
> > --
> > Nicolas Mailhot
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
>
> Hello Nicolas,
>
> Could you add two full examples, one for binary package, one for library
> package, like in https://fedoraproject.org/wiki/PackagingDrafts/
> Go#Sample_RPM_Spec
>
> Putting it all together (https://fedoraproject.org/wiki/
> More_Go_packaging#Putting_it_all_together) is nice but don't understand if it
> is for a binary or a library.
>
> Also, is there any plan to update Gofed with these new guidelines? If it's
> not
> automated like today, we will lose time instead of gaining some in the end.
>
> Best regards,
>
> Robert-André

I think that it would be best if Nicolas could fold his proposal in to the original draft as optional part for simple library/binary packages.

As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they have no place in devel package as they artificially inflate build root size) and shipping pre-built shared libraries.

Just my opinion,

JC
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

----- Mail original -----
De: "Jakub Cajka"

Hi Jakub
> I think that it would be best if Nicolas could fold his proposal in to the original draft as
> optional part for simple library/binary packages.

Frankly, that's a lot of work and churn, I don't want the new parts to be refused because of something in the old parts, and I certainly do not want to spend weeks replacing bits only to put them back in and so on while people made up their mind on the things they want replaced or not. The new parts are pretty much autonomous and complete, they are sufficient to create working Fedora Go packages, they are ready for FPC review.

If someone wants to extract the ready non discussion parts of the old page and get them approved I can work with him to merge both elements

I didn't want to write about the old page, but since you put it on the table.

It is full of digressions and elements of no evident applied value while packaging Go software. It's not an operational "how to do stuff" document it's a "here are various things you may like to know about Go that may or may not help you create Fedora Go packages, if they do not help you forget about it and run gofed". There are too many WIP non operational bits in the old pages to allow merging while in an approval process. And I'm sure any attempt to strip the WIP bits from my side will be met with energetic protests.

People, if you want that page to be ever approved by FPC make it more focused and accurate. Remove anything that does not explain to a Fedora Go packager how to write a Fedora Go spec file and what to ask upstream Go projects in a Fedora packager role. And make sure what remains is succinct, easy to read, and applicable without undocumented holes or gofed magic.

I know that's a lot of non-fun work. I did this work on the new page. I don't particularly *like* writing documentation. For every line I put in the new page, I had to ask myself "is the value to the Fedora packager sufficient to justify the time spend writing the text, formatting it, linking in the right place, proofing the result". I didn't put in stuff I could not justify cleaning up myself. If it was too much work to document it was either of insufficient value or better automated rather than explained in a manual process.

Any part of the text that can not be finished or serves another role has no place in a guideline submitted to FPC. It should be nuked or moved to a separate wiki page. All the half-finished and non operational stuff in there is the reason the draft has been stuck in pre-approval stage for four years. Just put yourself in FPC's place, its mission is to confirm a guideline is ready to be used by the average Fedora packager, not to produce it from half-finished half-related messy notes of the domain experts.
> As his proposal doesn't cover at least two major use cases, i.e. separate packaging of tests(they
> have no place in devel package as they artificially inflate build root size)

At this point of time my mind is pretty much set. I won't do separate packaging of tests because:
1. that raises complex dependency handling questions 
2. the average Go project test code is full of crufty junk
3. the average Go test depends on assets not tracked within Go
4. Go is not designed to separate elements shipped in the same directory
5. no one could answer when I asked the *three* Fedora lists what they were using the existing test packages for
6. from what I've seen many of the existing test packages simply can not work because they fail to include elements they need
7. even core Go people refuse to touch the subject with a 10 foot pole. Sam Boyer tried to demo a very small part of what would be necessary this week end at FOSDEM and this part of his demo *failed*. I don't have the level of Go understanding Sam Boyer Has.

I am ready to work with people who want to make separate test packages and I tried very hard to make them possible in the future but I won't spend time trying to ship half-baked stuff that does not work and that no one has a need for.

The proposed patterns strip test code from devel packages so build root sizes are safe (except for the original package build root, since I *do* execute unit tests in check, and they pull their deps in the build root).

You want to test one of my packages, fine, just rebuild it. That will run unit tests *and* build the project binaries (which are also a form of code testing, and a very thorough one at that).
> and shipping pre-built shared libraries.

The pre-build shared libraries the old guidelines refuse to build or document? Why exactly are you stating it's a major use case, Fedora is not doing it today.

"By default, the standard golang compiler produces static libraries. There is little value in shipping these prebuilt, especially since these libraries are very specifically tied to the exact minor release of the golang compiler."

"Presently the shared object libraries produced by GCC-Go are not usable"

Regards,

[Jakub Cajka]

----- Original Message -----
> From: "nicolas mailhot" 
> To: [e-mail]
> Cc: "Development discussions related to Fedora" , "Discussion of RPM packaging
> standards and practices for Fedora" 
> Sent: Monday, February 5, 2018 4:48:31 PM
> Subject: Re: Proposed Fedora packaging guideline:  More Go packaging
>
> ----- Mail original -----
> De: "Jakub Cajka"
>
> Hi Jakub
> > I think that it would be best if Nicolas could fold his proposal in to the
> > original draft as
> > optional part for simple library/binary packages.
>
> Frankly, that's a lot of work and churn, I don't want the new parts to be
> refused because of something in the old parts, and I certainly do not want
> to spend weeks replacing bits only to put them back in and so on while
> people made up their mind on the things they want replaced or not. The new
> parts are pretty much autonomous and complete, they are sufficient to create
> working Fedora Go packages, they are ready for FPC review.
>
> If someone wants to extract the ready non discussion parts of the old page
> and get them approved I can work with him to merge both elements
>
> I didn't want to write about the old page, but since you put it on the table.
>
> It is full of digressions and elements of no evident applied value while
> packaging Go software. It's not an operational "how to do stuff" document
> it's a "here are various things you may like to know about Go that may or
> may not help you create Fedora Go packages, if they do not help you forget
> about it and run gofed". There are too many WIP non operational bits in the
> old pages to allow merging while in an approval process. And I'm sure any
> attempt to strip the WIP bits from my side will be met with energetic
> protests.

Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.
> People, if you want that page to be ever approved by FPC make it more focused
> and accurate. Remove anything that does not explain to a Fedora Go packager
> how to write a Fedora Go spec file and what to ask upstream Go projects in a
> Fedora packager role. And make sure what remains is succinct, easy to read,
> and applicable without undocumented holes or gofed magic.

I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.
> I know that's a lot of non-fun work. I did this work on the new page. I don't
> particularly *like* writing documentation. For every line I put in the new
> page, I had to ask myself "is the value to the Fedora packager sufficient to
> justify the time spend writing the text, formatting it, linking in the right
> place, proofing the result". I didn't put in stuff I could not justify
> cleaning up myself. If it was too much work to document it was either of
> insufficient value or better automated rather than explained in a manual
> process.
>
> Any part of the text that can not be finished or serves another role has no
> place in a guideline submitted to FPC. It should be nuked or moved to a
> separate wiki page. All the half-finished and non operational stuff in there
> is the reason the draft has been stuck in pre-approval stage for four years.
> Just put yourself in FPC's place, its mission is to confirm a guideline is
> ready to be used by the average Fedora packager, not to produce it from
> half-finished half-related messy notes of the domain experts.

I believe I can see myself in their shoes(sure not 100% accurately) and I don't see anything that you are describing from their POV, it would be good to have the guidelines sooner, but I will have, as you, rather something complete than something half baked and semi broken. Please note that they are WIP and everybody is welcomed to contribute in constructive way. I don't think that there is any request for FPC to create the guidelines on their own and honestly it would rather bad idea IMHO.
> > As his proposal doesn't cover at least two major use cases, i.e. separate
> > packaging of tests(they
> > have no place in devel package as they artificially inflate build root
> > size)
>
> At this point of time my mind is pretty much set. I won't do separate
> packaging of tests because:
> that raises complex dependency handling questions
> the average Go project test code is full of crufty junk
> the average Go test depends on assets not tracked within Go
> Go is not designed to separate elements shipped in the same directory
> no one could answer when I asked the *three* Fedora lists what they were
>
> using the existing test packages for

They are used primarily to minimize build root size, by reducing the deps size and code size.
> from what I've seen many of the existing test packages simply can not work
>
> because they fail to include elements they need

Please file bugs or pull requests. We need to work together to improve the experience. I'm sure that you are including all the dependencies in your devel sub-package(that contain the tests) to be able to run them locally.
> even core Go people refuse to touch the subject with a 10 foot pole. Sam
>
> Boyer tried to demo a very small part of what would be necessary this week
> end at FOSDEM and this part of his demo *failed*. I don't have the level of
> Go understanding Sam Boyer Has.

What has been demoed?? Unfortunately I could not make it there. Have you met with Jan there?
> I am ready to work with people who want to make separate test packages and I
> tried very hard to make them possible in the future but I won't spend time
> trying to ship half-baked stuff that does not work and that no one has a
> need for.
>
> The proposed patterns strip test code from devel packages so build root sizes
> are safe (except for the original package build root, since I *do* execute
> unit tests in check, and they pull their deps in the build root).

Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.
> You want to test one of my packages, fine, just rebuild it. That will run
> unit tests *and* build the project binaries (which are also a form of code
> testing, and a very thorough one at that).

I don't think that is great user experience nor packagers/maintainers experience.
> > and shipping pre-built shared libraries.
>
> The pre-build shared libraries the old guidelines refuse to build or
> document? Why exactly are you stating it's a major use case, Fedora is not
> doing it today.

?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible and added there.
> "By default, the standard golang compiler produces static libraries. There is
> little value in shipping these prebuilt, especially since these libraries
> are very specifically tied to the exact minor release of the golang
> compiler."
>
> "Presently the shared object libraries produced by GCC-Go are not usable"

You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is now feasible. I believe that any Go packaging guideline that doesn't implement this side in working fashion is not FPC ready IMNHO.

JC
> Regards,
>
> --
> Nicolas Mailhot
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

----- Mail original -----
De: "Jakub Cajka"

Hi Jakub,
> > And I'm sure any
> > attempt to strip the WIP bits from my side will be met with energetic
> > protests.
> Have you tried that? What make you assume that? I'm sure that if you do it in constructive way, they will be accepted.

My experience with humans is that they get attached to their bits of text even when they have no time to finish them. But yes, I haven't tried, I don't want to see a wiki page for a month after writing and formatting and revising two separate packaging drafts.
> I believe that technically exhausting document is needed as Go packaging is far from trivial. Sure it would be great to have
> (trivial) quick start guide. I think that your proposal fits that bill more than full documentation.

IMHO that's exactly what FPC expects: a quick start document, not an in-depth encyclopædia. In-depth is too hardcore to be validated by domain laymen, is useless to onboard new packagers (which is the primary objective of Fedora guidelines), is never finished because the state of the art changes, and can't be applied to check whether a spec is good or not because in-depth concerns *complex* cases where the "right" choice is not obvious, depends on many factors that change over time, and usually requires asking the domain experts directly.

So you have the simple common cases, which are sufficient for most packagers, and can be addressed by a formal document FPC can review and enforce, and "other things" which are valuable but can not ever attain the formalness and strictness level expected of Fedora guidelines, require continuous freedom to edit, and are hosted in other Fedora wiki pages. At least that's how real-life law is written (law text + expert commentaries) and how other Fedora packaging SIGs function and got their guidelines approved.
> They are used primarily to minimize build root size, by reducing the deps size and code size.

Stripping example material from gopath installation filelists achieves the same objective without all the complexity of managing packages that share directories Go considers indivisible.
> > even core Go people refuse to touch the subject with a 10 foot pole. Sam
> >
> > Boyer tried to demo a very small part of what would be necessary this week
> > end at FOSDEM and this part of his demo *failed*. I don't have the level of
> > Go understanding Sam Boyer Has.
> What has been demoed?? Unfortunately I could not make it there

Sam Boyer presented the current state of his go dep prototype, and explained his design choices (for example no nested projects which means the go import path root becomes a core concept and it's useless to invest in splitting projects as that usually requires nesting). He ended up stating go dep will ignore tests by default, tried to demo optional test handling, and that failed. He had another conference the next day I couldn't get to as I was stuck in another part of the campus.
> Have you met with Jan there?

Unfortunately FOSDEM was its usual crazy stuff with multiple interesting conferences at any given time and no hope of being admitted in a room if you're late. So I've just been running from conf room to conf room without any hope of meeting anyone :(
> Your packaging proposal seems fairly monolithic and disregarding any prior art, so extensions will be very painful.

Well, the creation of the proposal was a long suite of extending to cover more cases (that's why some macros are rather long and convoluted), so I'm quite sure extension is possible. Of course I may miss something, if you identify a roadblock please point it out.
> > You want to test one of my packages, fine, just rebuild it. That will run
> > unit tests *and* build the project binaries (which are also a form of code
> > testing, and a very thorough one at that).
> I don't think that is great user experience nor packagers/maintainers experience.

That's pretty much how rpm prefers it to be (use %check and Builddeps) and also how Go prefers it to be (no provision for splitting directories). Of course anyone interested can try to fight the design of both of them to achieve something else, and would try to help, but I'm not interested in being this person.
> > > and shipping pre-built shared libraries.
> >
> > The pre-build shared libraries the old guidelines refuse to build or
> > document? Why exactly are you stating it's a major use case, Fedora is not
> > doing it today.
> ?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been working on stabilizing APIs, so it will be feasible
> and added there.

My maybe naïve POW is that shared libraries are just another container for the elements currently installed in %gopath/src, so doing them is mostly adding a separate stage in %build and making %goinstall install the result instead of installing raw files.

The hardest part is to get compatible code bases, defining what needs stripping in production, and defining shared libraries numbering.

The proposal helps for the two first points and I have no idea or opinion on the third.

The rest is macro plumbing that should not require a lot of spec surgery once specs are mostly free of custom shell code.
> You have mentioned that the API breakage is not that big issue, so this led me to conclude that shared lib/bin packaging is
> now feasible.

API breakage is not a showstopper with the static linking Go prefers because careful BuildDep construction can ensure only one (and the right one) dependency code state ends up in the buildroot, and BuildDeps of the next app can select another code state. So it's mostly a mater of creating the necessary -devel compat packages and defining BuildDep constrains.

It is a showstopper for shared libs because you can't install several versions of the same shared lib without separate identifiers/release ids to disambiguate them, Go upstreams do not provide those identifiers today and I doubt creating a Fedora-specific identifier namespace at the lib level would be a good idea. Though, the few projects that version their import paths, and make sure to only extend their APIS within a major release, could be libified today (theoretically, I've not checked how to do it). I'm not sure there are enough of them to be worth multiple packaging style

Regards,

[Jakub Cajka]

----- Original Message -----
> From: "nicolas mailhot" 
> To: [e-mail]
> Cc: "Development discussions related to Fedora" , "Discussion of RPM packaging
> standards and practices for Fedora" 
> Sent: Tuesday, February 6, 2018 7:34:03 PM
> Subject: Re: Proposed Fedora packaging guideline:  More Go packaging
>
> ----- Mail original -----
> De: "Jakub Cajka"
>
> Hi Jakub,
> > > And I'm sure any
> > > attempt to strip the WIP bits from my side will be met with energetic
> > > protests.
> > Have you tried that? What make you assume that? I'm sure that if you do it
> > in constructive way, they will be accepted.
>
> My experience with humans is that they get attached to their bits of text
> even when they have no time to finish them. But yes, I haven't tried, I
> don't want to see a wiki page for a month after writing and formatting and
> revising two separate packaging drafts.
> > I believe that technically exhausting document is needed as Go packaging is
> > far from trivial. Sure it would be great to have
> > (trivial) quick start guide. I think that your proposal fits that bill more
> > than full documentation.
>
> IMHO that's exactly what FPC expects: a quick start document, not an in-depth
> encyclopædia. In-depth is too hardcore to be validated by domain laymen, is
> useless to onboard new packagers (which is the primary objective of Fedora
> guidelines), is never finished because the state of the art changes, and
> can't be applied to check whether a spec is good or not because in-depth
> concerns *complex* cases where the "right" choice is not obvious, depends on
> many factors that change over time, and usually requires asking the domain
> experts directly.
>
> So you have the simple common cases, which are sufficient for most packagers,
> and can be addressed by a formal document FPC can review and enforce, and
> "other things" which are valuable but can not ever attain the formalness and
> strictness level expected of Fedora guidelines, require continuous freedom
> to edit, and are hosted in other Fedora wiki pages. At least that's how
> real-life law is written (law text + expert commentaries) and how other
> Fedora packaging SIGs function and got their guidelines approved.
> > They are used primarily to minimize build root size, by reducing the deps
> > size and code size.
>
> Stripping example material from gopath installation filelists achieves the
> same objective without all the complexity of managing packages that share
> directories Go considers indivisible.

??, I have been talking about tests. From Go perspective source files and test source file, testdata are two distinctive sets.
From my POV if your code depends for regular builds on tests code it means that there is something horribly wrong with your project(regardless of language used).

Docs and examples would be great too, it should be in guidelines that packager must use reasonable effort to create subpackages with them.
> > > even core Go people refuse to touch the subject with a 10 foot pole.
> > >
> > > Sam
> > > Boyer tried to demo a very small part of what would be necessary this week
> > > end at FOSDEM and this part of his demo *failed*. I don't have the level
> > > of
> > > Go understanding Sam Boyer Has.
> > What has been demoed?? Unfortunately I could not make it there
>
> Sam Boyer presented the current state of his go dep prototype, and explained
> his design choices (for example no nested projects which means the go import
> path root becomes a core concept and it's useless to invest in splitting
> projects as that usually requires nesting). He ended up stating go dep will
> ignore tests by default, tried to demo optional test handling, and that
> failed. He had another conference the next day I couldn't get to as I was
> stuck in another part of the campus.

Cool, there is recording for anyone interested https://fosdem.org/2018/schedule/event/dep/.
> > Have you met with Jan there?
>
> Unfortunately FOSDEM was its usual crazy stuff with multiple interesting
> conferences at any given time and no hope of being admitted in a room if
> you're late. So I've just been running from conf room to conf room without
> any hope of meeting anyone :(
> > Your packaging proposal seems fairly monolithic and disregarding any prior
> > art, so extensions will be very painful.
>
> Well, the creation of the proposal was a long suite of extending to cover
> more cases (that's why some macros are rather long and convoluted), so I'm
> quite sure extension is possible. Of course I may miss something, if you
> identify a roadblock please point it out.
> > > You want to test one of my packages, fine, just rebuild it. That will run
> > > unit tests *and* build the project binaries (which are also a form of code
> > > testing, and a very thorough one at that).
> > I don't think that is great user experience nor packagers/maintainers
> > experience.
>
> That's pretty much how rpm prefers it to be (use %check and Builddeps) and
> also how Go prefers it to be (no provision for splitting directories). Of
> course anyone interested can try to fight the design of both of them to
> achieve something else, and would try to help, but I'm not interested in
> being this person.
> > > > and shipping pre-built shared libraries.
> > >
> > > The pre-build shared libraries the old guidelines refuse to build or
> > > document? Why exactly are you stating it's a major use case, Fedora is not
> > > doing it today.
> > ?? I'm bit confused. Yes, it is not mentioned there yet as Jan have been
> > working on stabilizing APIs, so it will be feasible
> > and added there.
>
> My maybe naïve POW is that shared libraries are just another container for
> the elements currently installed in %gopath/src, so doing them is mostly
> adding a separate stage in %build and making %goinstall install the result
> instead of installing raw files.
>
> The hardest part is to get compatible code bases, defining what needs
> stripping in production, and defining shared libraries numbering.
>
> The proposal helps for the two first points and I have no idea or opinion on
> the third.
>
> The rest is macro plumbing that should not require a lot of spec surgery once
> specs are mostly free of custom shell code.
> > You have mentioned that the API breakage is not that big issue, so this led
> > me to conclude that shared lib/bin packaging is
> > now feasible.
>
> API breakage is not a showstopper with the static linking Go prefers because
> careful BuildDep construction can ensure only one (and the right one)
> dependency code state ends up in the buildroot, and BuildDeps of the next
> app can select another code state. So it's mostly a mater of creating the
> necessary -devel compat packages and defining BuildDep constrains.

Not really, mostly with static linking you are fine if you change deps as long as you don't touch basic runtime(glibc,...) and binaries will keep on working. In dynamic linking when you change the deps in incompatible way(break API/ABI) you break the runtime for all built binaries and you need to do rebuild to just keep distro working.
> It is a showstopper for shared libs because you can't install several
> versions of the same shared lib without separate identifiers/release ids to
> disambiguate them, Go upstreams do not provide those identifiers today and I
> doubt creating a Fedora-specific identifier namespace at the lib level would
> be a good idea. Though, the few projects that version their import paths,
> and make sure to only extend their APIS within a major release, could be
> libified today (theoretically, I've not checked how to do it). I'm not sure
> there are enough of them to be worth multiple packaging style
>
> Regards,
>
> --
> Nicolas Mailhot

You can version the so's arbitrarily for example and have explicit non general require on the devel, I think this really minor implementation detail from packaging discussion POV.

JC
> golang mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]