The Docker Stack and Go Vendoring

[Maxwell G]

Hi everyone,

Intro

There have been on-and-off discussions within the Go SIG about vendoring 
dependencies. To put my thoughts into words, I wrote a blog post [0] 
about the issues with Go dependency de-vendoring that Fedora currently does.

Docker stack vendoring

I worry that the current approach to dependency management is 
unfeasible—especially for complex package stacks like 
Containerd/Docker/Moby. These packages and the underlying libraries have 
huge dependency trees and circular dependencies and have been 
out-of-date for months or longer. moby-engine already takes a 
complicated, hybrid approach to vendoring (part of the package uses 
vendored dependencies, part does not), while containerd is all 
un-bundled. Getting everything up-to-date will require a significant 
amount of work that nobody has stepped up to do, and in my view, is 
ultimately unsustainable. Last year, I onboarded new contributors, as I 
no longer could dedicate time to maintain these packages, but it was 
also difficult for them to keep up with the web of dependencies.

I propose we start with fully vendoring the Docker stack. As I said, 
parts of moby-engine are already bundled, and so are podman, kubernetes, 
cri-o, containernetworking-plugins, and other applications in the 
written-in-Go containerization stack. I have been working on revamped 
Docker stack packages at [1]. I believe that the simplified packaging 
approach will entice new maintainers to come onboard—I have already 
reached out to one. I also wrote specfiles for Docker Buildx and Docker 
Compose v2 that were not feasible to package with the previous approach.

Overall Go ecosystem

Then, we can re-evaluate the overall state of the Go library ecosystem. 
I am not proposing that we immediately mass retire all Go libraries and 
vendor everything, but I think we need to consider the overall health of 
Go applications in Fedora and consider vendoring in at least some cases.

I will also note the Go Packaging Guidelines' current stance on the 
vendoring[2]:
> At the moment golang projects packaged in Fedora SHOULD be unbundled 
> by default. It means projects are built from dependencies packaged in 
> Fedora.
>
> For some project it can be reasonable to build from bundled 
> dependencies. Every bundling needs a proper justification.

Vendoring the Docker stack is allowed under this guideline. Any more 
drastic steps to mass de-vendor certain packages or use vendoring for 
any new packages would obviously require guidelines changes—but again, 
we are not there yet. There is more tooling work to be done and more 
discussion to be had.

go-vendor-tools

I have been working on go-vendor-tools [3], a tool to enable packaging 
vendored Go applications in a Guidelines-compliant way, for the past 
couple weeks. go-vendor-tools aims to make creating fully reproducible 
vendor archives and handling licensing a relatively frictionless 
process. The tool also natively supports regenerating vendor tarballs to 
apply security updates[4].

See [5] for instructions to test the latest go-vendor-tools and the 
current iteration of the go2rpm code to allow automatically generating 
vendored Go package specfiles. I look forward to your feedback.

***

If you have read this far, thank you! Any feedback about the Docker 
stack or Go vendoring overall is welcome.

Best,

Maxwell

[0] https://gtmx.me/blog/fedora-go-unbundling-is-broken/

[1] https://git.sr.ht/~gotmax23/docker-ng

[2] 
https://docs.fedoraproject.org/en-US/packaging-guidelines/Golang/#_bundled_o...

[3] https://fedora.gitlab.io/sigs/go/go-vendor-tools/

[4] 
https://fedora.gitlab.io/sigs/go/go-vendor-tools/scenarios/#security-updates

[5] https://gitlab.com/fedora/sigs/go/go2rpm/-/merge_requests/4

[Sérgio Basto]

On Mon, 2024-03-25 at 18:17 -0500, Maxwell G wrote:
> I propose we start with fully vendoring the Docker stack. As I said,
> parts of moby-engine are already bundled, and so are podman,
> kubernetes, cri-o,

TL DR; 
yeah I was think about do that to containerd , use vendor,  instead of
hell of dependencies

[Sérgio Basto]

On Mon, 2024-03-25 at 23:34 +0000, Sérgio Basto wrote:
> On Mon, 2024-03-25 at 18:17 -0500, Maxwell G wrote:
> > I propose we start with fully vendoring the Docker stack. As I
> > said, parts of moby-engine are already bundled, and so are podman,
> > kubernetes, cri-o,
>
> TL DR; 
> yeah I was think about do that to containerd , use vendor,  instead
> of hell of dependencies

one propose 
https://github.com/terrapkg/packages/blob/frawhide/anda/docker/containerd/co...

*****SPAM***** Re: The Docker Stack and Go Vendoring

[Maxwell G]

On Tue Mar 26, 2024 at 21:28 +0000, Sérgio Basto wrote:
> On Mon, 2024-03-25 at 23:34 +0000, Sérgio Basto wrote:
> > On Mon, 2024-03-25 at 18:17 -0500, Maxwell G wrote:
> > > I propose we start with fully vendoring the Docker stack. As I
> > > said, parts of moby-engine are already bundled, and so are podman,
> > > kubernetes, cri-o,
> >
> > TL DR; 
> > yeah I was think about do that to containerd , use vendor,  instead
> > of hell of dependencies
>
> one propose 
> https://github.com/terrapkg/packages/blob/frawhide/anda/docker/containerd/co...

I already put together
https://git.sr.ht/~gotmax23/docker-ng/tree/main/item/containerd/containerd.s....
It uses the upstream Makefile and regenerates the vendor archive with
go-vendor-tools instead of using the built-in vendor directory.
The latter is not strictly necessary but allows us to take advantage of
go-vendor-tools' dependency override tooling if we need to bump
dependencies independently of upstream (e.g., for security or Go
compatibility purposes).
Also, I want these specfiles to serve as an example of go-vendor-tools'
functionality.

[Dave Dykstra]

Hi Maxwell,

FYI the singularity-ce & apptainer packages have been using vendoring all along.  I don't know if there's anything from our tooling that would help you or not.  Maybe we should switch to yours when you are finished.

go mod vendor is run here:
    https://github.com/apptainer/apptainer/blob/main/mlocal/frags/Makefile.stub#...

Here's the script that updates license dependencies:
    https://github.com/apptainer/apptainer/blob/main/scripts/update-license-depe...
There's an automated check for every PR to make sure it is up to date:
    https://github.com/apptainer/apptainer/blob/main/.github/workflows/ci.yml#L1...

Bundled provides are inserted here:
    https://github.com/apptainer/apptainer/blob/main/mconfig#L890

Dave

On Mon, Mar 25, 2024 at 06:17:55PM -0500, Maxwell G wrote:
> Hi everyone,
>
> Intro
>
> There have been on-and-off discussions within the Go SIG about vendoring dependencies. To put my thoughts into words, I wrote a blog post [0] about the issues with Go dependency de-vendoring that Fedora currently does.
>
> Docker stack vendoring
>
> I worry that the current approach to dependency management is unfeasible—especially for complex package stacks like Containerd/Docker/Moby. These packages and the underlying libraries have huge dependency trees and circular dependencies and have been out-of-date for months or longer. moby-engine already takes a complicated, hybrid approach to vendoring (part of the package uses vendored dependencies, part does not), while containerd is all un-bundled. Getting everything up-to-date will require a significant amount of work that nobody has stepped up to do, and in my view, is ultimately unsustainable. Last year, I onboarded new contributors, as I no longer could dedicate time to maintain these packages, but it was also difficult for them to keep up with the web of dependencies.
>
> I propose we start with fully vendoring the Docker stack. As I said, parts of moby-engine are already bundled, and so are podman, kubernetes, cri-o, containernetworking-plugins, and other applications in the written-in-Go containerization stack. I have been working on revamped Docker stack packages at [1]. I believe that the simplified packaging approach will entice new maintainers to come onboard—I have already reached out to one. I also wrote specfiles for Docker Buildx and Docker Compose v2 that were not feasible to package with the previous approach.
>
> Overall Go ecosystem
>
> Then, we can re-evaluate the overall state of the Go library ecosystem. I am not proposing that we immediately mass retire all Go libraries and vendor everything, but I think we need to consider the overall health of Go applications in Fedora and consider vendoring in at least some cases.
>
> I will also note the Go Packaging Guidelines' current stance on the vendoring[2]:
>
> At the moment golang projects packaged in Fedora SHOULD be unbundled by default. It means projects are built from dependencies packaged in Fedora.
>
> For some project it can be reasonable to build from bundled dependencies. Every bundling needs a proper justification.
>
> Vendoring the Docker stack is allowed under this guideline. Any more drastic steps to mass de-vendor certain packages or use vendoring for any new packages would obviously require guidelines changes—but again, we are not there yet. There is more tooling work to be done and more discussion to be had.
>
> go-vendor-tools
>
> I have been working on go-vendor-tools [3], a tool to enable packaging vendored Go applications in a Guidelines-compliant way, for the past couple weeks. go-vendor-tools aims to make creating fully reproducible vendor archives and handling licensing a relatively frictionless process. The tool also natively supports regenerating vendor tarballs to apply security updates[4].
>
> See [5] for instructions to test the latest go-vendor-tools and the current iteration of the go2rpm code to allow automatically generating vendored Go package specfiles. I look forward to your feedback.
>
> If you have read this far, thank you! Any feedback about the Docker stack or Go vendoring overall is welcome.
>
> Best,
>
> Maxwell
>
> [0] https://gtmx.me/blog/fedora-go-unbundling-is-broken/
>
> [1] https://git.sr.ht/~gotmax23/docker-ng
>
> [2] https://docs.fedoraproject.org/en-US/packaging-guidelines/Golang/#_bundled_o...
>
> [3] https://fedora.gitlab.io/sigs/go/go-vendor-tools/
>
> [4] https://fedora.gitlab.io/sigs/go/go-vendor-tools/scenarios/#security-updates
>
> [5] https://gitlab.com/fedora/sigs/go/go2rpm/-/merge_requests/4
>
> --
> Maxwell G (@gotmax23)
> Pronouns: He/They
> --
> _______________________________________________
> golang mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ 
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines 
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail] 
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Mikel Olasagasti]

Hi Maxwell,

One doubt about other packages that currently depend on any of the
packages you list to be vendored.

Checking your containerd spec[1] I see the only the binary package is
installed, no -devel package is created.

If that's correct I undrestand that all the packages that depend on it
will fail?

# fedrq whatrequires-src -X -F source  containerd
golang-github-containerd-aufs
golang-github-containerd-fuse-overlayfs-snapshotter
golang-github-containerd-imgcrypt
golang-github-containerd-nri
golang-github-containerd-stargz-snapshotter
golang-github-containerd-zfs
golang-github-docker
golang-github-docker-cli
golang-github-moby-buildkit
golang-github-tonistiigi-fsutil
golang-gvisor
golang-helm-3
golang-oras-1
kubernetes
moby-engine

If so, golang-github-docker would break other packages/binaries like `doctl`.

# fedrq whatrequires-src -X -F source  golang-github-docker
cadvisor
doctl
golang-github-docker-cli
golang-github-fsouza-dockerclient
golang-github-moby-buildkit
golang-github-moby-swarmkit-2
golang-github-prometheus
golang-github-tonistiigi-fsutil
golang-oras-1
migrate
moby-engine
reg

And the chain of broken deps could be quite large.

Have you checked the impact this could have?

Regards,
Mikel

[1] https://git.sr.ht/~gotmax23/docker-ng/tree/main/item/containerd/containerd.s...

Hau idatzi du Maxwell G ([e-mail]) erabiltzaileak (2024 mar.
26(a), ar. (00:18)):
> Hi everyone,
>
> Intro
>
> There have been on-and-off discussions within the Go SIG about vendoring dependencies. To put my thoughts into words, I wrote a blog post [0] about the issues with Go dependency de-vendoring that Fedora currently does.
>
> Docker stack vendoring
>
> I worry that the current approach to dependency management is unfeasible—especially for complex package stacks like Containerd/Docker/Moby. These packages and the underlying libraries have huge dependency trees and circular dependencies and have been out-of-date for months or longer. moby-engine already takes a complicated, hybrid approach to vendoring (part of the package uses vendored dependencies, part does not), while containerd is all un-bundled. Getting everything up-to-date will require a significant amount of work that nobody has stepped up to do, and in my view, is ultimately unsustainable. Last year, I onboarded new contributors, as I no longer could dedicate time to maintain these packages, but it was also difficult for them to keep up with the web of dependencies.
>
> I propose we start with fully vendoring the Docker stack. As I said, parts of moby-engine are already bundled, and so are podman, kubernetes, cri-o, containernetworking-plugins, and other applications in the written-in-Go containerization stack. I have been working on revamped Docker stack packages at [1]. I believe that the simplified packaging approach will entice new maintainers to come onboard—I have already reached out to one. I also wrote specfiles for Docker Buildx and Docker Compose v2 that were not feasible to package with the previous approach.
>
> Overall Go ecosystem
>
> Then, we can re-evaluate the overall state of the Go library ecosystem. I am not proposing that we immediately mass retire all Go libraries and vendor everything, but I think we need to consider the overall health of Go applications in Fedora and consider vendoring in at least some cases.
>
> I will also note the Go Packaging Guidelines' current stance on the vendoring[2]:
>
> At the moment golang projects packaged in Fedora SHOULD be unbundled by default. It means projects are built from dependencies packaged in Fedora.
>
> For some project it can be reasonable to build from bundled dependencies. Every bundling needs a proper justification.
>
> Vendoring the Docker stack is allowed under this guideline. Any more drastic steps to mass de-vendor certain packages or use vendoring for any new packages would obviously require guidelines changes—but again, we are not there yet. There is more tooling work to be done and more discussion to be had.
>
> go-vendor-tools
>
> I have been working on go-vendor-tools [3], a tool to enable packaging vendored Go applications in a Guidelines-compliant way, for the past couple weeks. go-vendor-tools aims to make creating fully reproducible vendor archives and handling licensing a relatively frictionless process. The tool also natively supports regenerating vendor tarballs to apply security updates[4].
>
> See [5] for instructions to test the latest go-vendor-tools and the current iteration of the go2rpm code to allow automatically generating vendored Go package specfiles. I look forward to your feedback.
>
> If you have read this far, thank you! Any feedback about the Docker stack or Go vendoring overall is welcome.
>
> Best,
>
> Maxwell
>
> [0] https://gtmx.me/blog/fedora-go-unbundling-is-broken/
>
> [1] https://git.sr.ht/~gotmax23/docker-ng
>
> [2] https://docs.fedoraproject.org/en-US/packaging-guidelines/Golang/#_bundled_o...
>
> [3] https://fedora.gitlab.io/sigs/go/go-vendor-tools/
>
> [4] https://fedora.gitlab.io/sigs/go/go-vendor-tools/scenarios/#security-updates
>
> [5] https://gitlab.com/fedora/sigs/go/go2rpm/-/merge_requests/4
>
> --
> Maxwell G (@gotmax23)
> Pronouns: He/They
>
> --
> _______________________________________________
> golang mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Maxwell G]

On Wed Mar 27, 2024 at 16:53 +0100, Mikel Olasagasti wrote:
> Hi Maxwell,

Hi Mikel!
> One doubt about other packages that currently depend on any of the
> packages you list to be vendored.

Yes, that's a good point. Thank you for raising it!
> Checking your containerd spec[1] I see the only the binary package is
> installed, no -devel package is created.
>
> If that's correct I undrestand that all the packages that depend on it
> will fail?

I plan to create a transitional golang-github-containerd package[1] that
will only provide the -devel subpackage. This Provides `deprecated()`
(meaning that nothing new is allowed to depend on it), and I plan to
eventually remove it once other packages are handled.

[1] https://git.sr.ht/~gotmax23/docker-ng/tree/main/item/golang-github-container...
> # fedrq whatrequires-src -X -F source  containerd

You probably want to use `fedrq whatrequires containerd-devel` instead.
Some of these depend on the main containerd application and not the
-devel subpackage,
> golang-github-containerd-aufs
> golang-github-containerd-fuse-overlayfs-snapshotter
> golang-github-containerd-imgcrypt
> golang-github-containerd-nri
> golang-github-containerd-stargz-snapshotter
> golang-github-containerd-zfs

The main users of these libraries are containerd itself.
> golang-github-docker
> golang-github-docker-cli

These will be kept around for now (they'll depend on the transitional
-devel package I mentioned).
> golang-github-moby-buildkit

This will kept around until we can get rid of golang-github-docker* and
golang-github-containerd-* packages.
> golang-github-tonistiigi-fsutil

Only golang-github-docker and containerd-devel depend on this.
> golang-gvisor

This package has been FTBFS since Fedora 37 and is up for retirement—
well, the retirement has been pushed off for a release cycle to avoid
breaking a bunch of things, but that cannot happen forever.
> golang-helm-3

This package will probably end up getting vendored for the same reasons
as containerd and docker.
> golang-oras-1

Only helm uses this.
> kubernetes

kubernetes only depends on containerd.
> moby-engine

Ditto.
> If so, golang-github-docker would break other packages/binaries like `doctl`.

<snip>
> And the chain of broken deps could be quite large.
>
> Have you checked the impact this could have?

The plan is to add the transitional golang-github-containerd package and
keep the current versions of the golang-github-docker-* libraries until
we can go through the dependency tree.
Yes, this will have some impact on the other packages in the ecosystem.
That is the idea behind the two-step plan:
I do not want containerd and moby-engine to rely on the broken,
out-of-date container library ecosystem anymore; but I also want to take
a careful approach to the packages outside the Docker ecosystem that
removing the library package will affect.
The fedrq and Go Package Data tooling [2] that we already have in place
will assist this effort.

[2] https://gitlab.com/fedora/sigs/go/package-data

[Maxwell G]

On 3/25/24 18:17, Maxwell G wrote:
> I propose we start with fully vendoring the Docker stack. As I said, 
> parts of moby-engine are already bundled, and so are podman, 
> kubernetes, cri-o, containernetworking-plugins, and other applications 
> in the written-in-Go containerization stack. I have been working on 
> revamped Docker stack packages at [1]. I believe that the simplified 
> packaging approach will entice new maintainers to come onboard—I have 
> already reached out to one. I also wrote specfiles for Docker Buildx 
> and Docker Compose v2 that were not feasible to package with the 
> previous approach.
>
> [1] https://git.sr.ht/~gotmax23/docker-ng

I have started submitting reviews:

For docker:

1. *moby-engine *—
    https://src.fedoraproject.org/rpms/moby-engine/pull-request/21
    (fixes https://bugzilla.redhat.com/show_bug.cgi?id=2237925)
 2. *docker-cli* (split out from moby-engine) —
    https://bugzilla.redhat.com/show_bug.cgi?id=2274561
 3. *docker-compose* —
    https://src.fedoraproject.org/rpms/docker-compose/pull-request/5
    (fixes https://bugzilla.redhat.com/show_bug.cgi?id=2000794)
 4. *docker-compose-switch* —
    https://bugzilla.redhat.com/show_bug.cgi?id=2274654
 5. *docker-buildx* — https://bugzilla.redhat.com/show_bug.cgi?id=2274656

For containerd:

1. *containerd* —
    https://src.fedoraproject.org/rpms/containerd/pull-request/16 (fixes
    https://bugzilla.redhat.com/show_bug.cgi?id=2177352)
 2. *golang-github-containerd* (new independent containerd-devel
    package) — https://bugzilla.redhat.com/show_bug.cgi?id=2274601
    https://bugzilla.redhat.com/show_bug.cgi?id=2274601