Golang packaging in Fedora

[Jan Chaloupka]

Hey,

sending summary of the current state of golang packaging in Fedora from 
my POV:

# Packaging guidelines (draft), fpc ticket

The current packaging guidelines are still in a form of a draft [1]. 
Corresponding discussion w.r.t. guidelines is held in fpc ticket [2]. 
There are still some issues that needs to be solved before the 
guidelines can be accepted (mentioned in [3]). The most severe is the 
api backward compatibility problem. In short, two or more Go projects 
can import the same Go package. As it has been proved many times, those 
projects require different snapshot (commit) of the package. All 
snapshots are either compatible, or not. In the former case, the most 
recent snapshot can be taken as an update (of a Fedora package). In the 
latter case, one of the snapshots has to be taken as a reference one. 
Causing some of the projects to be patched in order to be compatible 
with the reference snapshot.

As there is more than 250 Go projects packaged in Fedora, it is not 
trivial to update one Go package without breaking another one. In some 
cases, it is no longer possible to update to a new release without 
breaking dependencies of other package. For that reason, it is more 
feasible to concentrate on api backward incompatibilities detection in a 
distribution rather then to concentrate on actually solving the backward 
incompatibility problem.

Other aspects of the guidelines is to concentrate on support of dynamic 
go libraries, extending support to other architectures, or improving go 
spec file so it is more transparent for developers (e.g. better support 
for CI, providing more data for analysis, etc.).

# Current problems

This is not a complete list of all problems. Still:

- api backward incompatibility (described above): at the moment, it is 
possible to detect missing dependencies, however, it is not possible to 
detect missing API symbols of imported dependencies
- lack of reviewers for new go packages
- testing of Go projects: in most cases, go projects are shipped with 
unit-tests, however, it does not cover regression tests, or testing 
support for various (e.g. does a given secondary architecture has all 
dependencies; e.g. there is no fuse FS for arm) architectures, or for CD 
(continuous deployment)
- minimizing a number or bundled dependencies some projects are built 
with: in some cases, it is not possible to built from debundled 
dependencies, a projects must be built from bundled
- lack of maintainers of Go projects packaged in Fedora
- development of tooling for task automation and analysis of Go projects
- others: communication with upstream to improve the code or fixing 
issues, writing articles about Go projects (e.g. which tools are more 
suitable for which use cases, available tooling, the best practices for 
development, etc.),

# Tooling

As a lot of tasks can be automated (spec file generator, spec file lint, 
dependency analysis, etc.), it is critical to have a tooling saving each 
contributor/maintainer's time. To support the packaging effort, gofed 
[4] has been developed. In order to support analysis of source code and 
detection of bad behavior, the tool is continuously being extended with 
other tasks such as distribution scan, dependency graph construction, 
etc. One of the current goals is to improve:

- detection of api backward incompatibilities: only detection of missing 
packages is implemented, detection of specific symbols from API of go a 
package is not (removal of exported symbols from API that are not used 
by any project does not break backward compatibility)
- analysis of code and discovery of unhealthy behavior: supported 
architectures, periodically running code coverage, unit-tests and other 
checks
- collecting data about code: list of dependencies, list of defined 
symbols, tests, upstream changes, api backward compatibility violations, 
etc.
- more improved updating and checking of spec files: better spec file 
updates (automatic update of dependencies, tests, decomposition into 
smaller devel subpackages or dynamic libraries, etc.)
- more automated update of Fedora packages: in the best case "import 
srpm -> scratch build -> push -> build -> override -> update" in one 
command (already implemented up to the import srpm), at the same time 
extend the work-flow with running smoke tests, CI and other tests before 
the raw build is created (or put into updates-testing).
- running govet, gofmt and other tools collecting errors, warnings and 
other useful information about state of go source code
- others at [5]

There are tools for Go packaging and other automated tasks. Let's put 
the best from all tools and make our life easier.

# Joined effort, community

There is already a lot of effort in packaging of new Go projects into 
Fedora. Or updating the current ones with newer releases. In each case, 
there is always a need for reviews and review requests. The more 
projects we get into distribution, the better support we get for users 
(and other developers). On the other hand, more projects will increase 
the number of packages to maintain. Everyone's time is limited and 
should not be spent on dealing with routine tasks. Let's discuss about 
the most common issues we deal with, the way we solve go problems and 
ideas how to improve the current work-flows. Either by starting threads, 
each one for a new package into Fedora (e.g. snapd, notary, grafana, 
vault, etc.), about improvements of guildelines, or just about looking 
for reviewers.

This is not a complete reference of all problems the go packaging deals 
with in Fedora. Let me know if you need more information about specific 
problems, want to join gofed effort, or anything else related to Go in 
Fedora.

[1] https://fedoraproject.org/wiki/PackagingDrafts/Go
[2] https://fedorahosted.org/fpc/ticket/382
[3] https://fedorahosted.org/fpc/ticket/382#comment:38
[4] https://github.com/gofed/gofed
[5] https://github.com/gofed/gofed/issues

Happy to hear from ya!!!

Regards
Jan