[resending to keep the golang-dev listserver happy]
Hi Jakub,
The notary part of Go modules, like the rest of the module
implementation, suffers from a lack of understanding of integration and
QA workflows, and a simplistic dev-centric worldview.
As designed, it will have to be patched out by every single entity
trying to perform large scale Go code integration and QA, starting with
the usual suspects, Linux distributions. Where that will leave upstream
Go, since its main target platform is Linux, I have no idea (I suspect
there will be some angst @Google about it).
I write this from the POW of the person that tries to update the Fedora
Linux Go tooling so we can ship Go with module mode on in august.
Whatever we do at that time will be mirrored by Fedora downstreams like
RHEL and Centos and all the other Linux variants that get their
inspiration Fedora-side.
It's not an armchair analysis, I already wrote way too much module-
oriented custom code because the upstream tooling is deficient, and I've
read my share of upstream issue reports.
CONTINUOUS TRANSPARENT INTERNET LOOKUPS ARE NOT ACCEPTABLE POST−DEV
When you integrate large volumes of code, targeting multiple hardware
architectures, it’s not acceptable to have the result of the Arm QA run
differ from the result of the x86_64 run just because it was scheduled
some minutes later on the QA farm, the internet state changed in the
meanwhile, and the Go tools decided to “help” you by looking at the
internet behind your back and updated transparently the state being
QAed. That's nothing new of Go specific, and that’s why our build system
disables network access in build containers after populating them with
the code to be built and tested, and already did so before someone
decided to invent Go.
But, that QA reproducibility constrain was not understood by Go
developers, and pretty much all the Go tools that were ported to module
mode, will attempt Internet accesses at the slightest occasion, and
change the project state based on those access results, with no ability
to control or disable it, and nasty failure modes if the internet access
fails.
The Go issue tracker is filling up with reports of people expressing
their incredulity (or more) after hitting a variation of this problem on
their QA systems.
QA−ED CODE IS NOT PRISTINE DEV CODE
No code is ever perfect.
When you integrate large volumes of code, targeting multiple hardware
architectures, you *will* hit problems missed by the original upstream
dev team. And you *will* have to patch them up, and you *will* have to
build the result without waiting for upstream to look at it because, in
production, the show must go on, some of those will eventually have
zero-day security implications, and the upstreaming lag is not
acceptable (assuming upstream is available and friendly, which is not
necessarily the case).
So no serious shop is ever going to build its information system from
pristine upstream dev code or pristine upstream Go modules. Pristine
upstream code building is a nice ideal, but in the real world, it’s
utopia.
If you don’t believe me, take your favorite app, and try to build a
system containing it from genuine unadultered upstream code (that means,
without relying on Fedora, Debian or whatever, since we all know they
are full of “nasty” patched code).
Again, that technical real-world-is-not-perfect was not understood by Go
developers. Sure they gave us the replace directive in Go module. But
that is opt-in. So it only works on a small scale, when fixing third
party code is an exception, and you only have a handful of projects that
use this third-party code.
On a large scale everyone will just preventively rename everything by
default just to retain the ability to perform fixes easily. You'll end
up with blanket replacement of every "upstream" module name with
"qaed/upstream" forks. And probably easier to rewrite all the imports to
point to qaed/upstream by default in the source code.
That will result in a huge mess, much larger than the current GOPATH
vendoring mess.
All because the design does not take into account the last mile QA
patching that occurs before putting code in production.
Blindly enforcing renaming rules because you don't understand or accept
the existence of QA middlemen produced things like Iceweasel. And I
don't think Mozilla or Debian were ever happy about it.
PUBLISHING QA-ED CODE CAN NOT DEPEND ON A THIRD PARTY
At least, not if you want the result to work in a free software
ecosystem. Freedom 4 “The freedom to distribute copies of your modified
versions to others” is not “The freedom to distribute copies of your
modified versions to others, but only if you tell this third party
first”.
And Google is free to choose not to target a free software ecosystem,
but that means all the entities that do target a free software ecosystem
(like Linux distributions) or have happily built their infrastructure on
free software products (like every single major cloud operator out
there), will have fork Go, or reorient their software investments
somewhere else, or some combination of those. And I don't think Google
wants that, or will be happy about it.
But, that's the situation the current notary design will create.
Because in trying to enforce a dev utopia, it is going to break the
workflows of a large proportion of the current Go ecosystem.
Any working system will need a way to declare trust in a local authority
(the shop QA team) or a set of authorities (the shop QA team, and
trusted third parties), and not have this trust rely on continuous
access to a server, regardless of who controls it (so detached digital
signatures, not direct server hash lookups).
And I’ll stop here before I write things I will regret later.
Regards,
--
Nicolas Mailhot