cmd/go: 'go build' sets arbitrary requirement of module name containing dots

[shoenig]

All of my dependent modules are in order, and none of them have a hostname prefix that contain dots. The convention we have been using for years with GOPATH is "company/foo/bar", rather than "our.company.com/foo/bar".

What version of Go are you using (go version)?

$ go version
go version go1.11 linux/amd64

What did you do?

Built a proxy which can be used to serve up modules which reflect our real world, which is a world with custom import paths with no "hostname" prefix. Trying to use go build with this proxy and our modules results in the go build command complaining that our modules do not start with a "hostname" prefix, because the first path element in our modules do not contain any dot characters.

First, a view of the modules being served by our proxy

[k9 modprox-modules] $ tree
.
└── indeed
    └── gophers
        ├── 3rdparty
        │   └── p
        │       └── github.com
        │           ├── pkg
        │           │   └── errors
        │           │       ├── v0.0.1.info
        │           │       ├── v0.0.1.mod
        │           │       └── v0.0.1.zip
        │           └── stretchr
        │               └── testify
        │                   ├── assert
        │                   │   ├── v0.0.1.info
        │                   │   ├── v0.0.1.mod
        │                   │   └── v0.0.1.zip
        │                   ├── require
        │                   │   ├── v0.0.1.info
        │                   │   ├── v0.0.1.mod
        │                   │   └── v0.0.1.zip
        │                   └── suite
        │                       ├── v0.0.1.info
        │                       ├── v0.0.1.mod
        │                       └── v0.0.1.zip
        ├── libconfig
        │   ├── v0.0.1.info
        │   ├── v0.0.1.mod
        │   └── v0.0.1.zip
        └── libtest
            ├── v0.0.1.info
            ├── v0.0.1.mod
            └── v0.0.1.zip

14 directories, 18 files

This seems like a reasonable module structure.

A peek at the go.mod file we are about to use:

$ cat go.mod
module indeed/gophers/libconfig

require (
	indeed/gophers/3rdparty/p/github.com/pkg/errors v0.0.1
	indeed/gophers/3rdparty/p/github.com/stretchr/testify/assert v0.0.1 // indirect
	indeed/gophers/3rdparty/p/github.com/stretchr/testify/require v0.0.1
	indeed/gophers/3rdparty/p/github.com/stretchr/testify/suite v0.0.1 // indirect
	indeed/gophers/libtest v0.0.1
)

Lines up fine with the modules being served from the proxy.

Okay let's build it

 $ echo $GOPROXY
http://localhost:12505

$ go build
go: indeed/gophers/3rdparty/p/github.com/stretchr/testify/require@v0.0.1: unrecognized import path "indeed/gophers/3rdparty/p/github.com/stretchr/testify/require" (import path does not begin with hostname)
go: indeed/gophers/libtest@v0.0.1: unrecognized import path "indeed/gophers/libtest" (import path does not begin with hostname)
go: indeed/gophers/3rdparty/p/github.com/pkg/errors@v0.0.1: unrecognized import path "indeed/gophers/3rdparty/p/github.com/pkg/errors" (import path does not begin with hostname)
go: indeed/gophers/3rdparty/p/github.com/stretchr/testify/suite@v0.0.1: unrecognized import path "indeed/gophers/3rdparty/p/github.com/stretchr/testify/suite" (import path does not begin with hostname)
go: indeed/gophers/3rdparty/p/github.com/stretchr/testify/assert@v0.0.1: unrecognized import path "indeed/gophers/3rdparty/p/github.com/stretchr/testify/assert" (import path does not begin with hostname)
go: error loading module requirements

Hmm, that's not useful. The go build command is enforcing import paths to begin with a path element that contains dot characters. This seems totally arbitrary (at least in the context of GOPROXY) and breaks at least this one real world use case (a fix on our end would be "atomically" updating hundreds of repositories and interrupting dozens of developer workflows).

It seems to me at least with GOPROXY set, go build should just trust that the module will be resolvable.

[andybons]

@rsc @bcmills

[bcmills]

This seems totally arbitrary (at least in the context of GOPROXY)

It's not totally arbitrary. Dotless paths in general are reserved for the standard library; go get has (to my knowledge) never worked with them, but go get is also the main entry point for working with versioned modules.

(It's also kind of weird to use names like indeed/gophers/3rdparty/p/github.com/stretchr/testify/require to refer to what is presumably a vendored copy of github.com/stretchr/testify/require: vendoring should no longer be necessary with modules, although you could certainly still maintain forks in order to apply local edits.)

a fix on our end would be "atomically" updating hundreds of repositories and interrupting dozens of developer workflows

It may be the case that you'd need to update lots of repositories, but it's not obvious to me why that would have to be atomic. Now that Go has type aliases, it is generally straightforward to define one package as a thin forwarding shim for another; with semantic import paths, we should probably even write a tool to automate construction of those shims, because I expect it will be common to define vN packages as forwarding shims for vN+1.

The general migration process would be:

1. Define indeed/…/somepkg as a forwarding shim for indeed.com/…/somepkg (using type aliases and forwarding functions).
2. Fix up callers to import from the new indeed.com/…/somepkg path (using go fix or fiximports or a similar tool).
3. Remove the forwarding shim.

[shoenig]

@bcmills @rsc Thinking about this some more, the assertion that a hostname must contain dots is not even correct. https://tools.ietf.org/html/rfc952

A "name" (Net, Host, Gateway, or Domain name) is a text string up
to 24 characters drawn from the alphabet (A-Z), digits (0-9), minus
sign (-), and period (.). Note that periods are only allowed when
they serve to delimit components of "domain style names".

go get may have never worked with non addressable names. I don't see how that is relevant, except to say there is code out there that does not abide by its requirements. Until now, go get has not been a required component of the go compiler.

Furthermore, with this assertion, the go1.X compiler will no longer be compliant with the Go1 BNF
https://golang.org/ref/spec#Import_declarations
which does not stipulate that a non-stdlib package must contain dots in the first path element.

t's also kind of weird to use names like ...

Welcome to about 6 years ago, to a huge source tree that predates the GOVENDOR experiment, and an internal package system that was usable with private and authenticated VCSs. We are very much looking forward to relieving years of pain with Go modules, but understandably the transition isn't painless, due to quirks like this one.

[bcmills]

with this assertion, the go1.X compiler will no longer be compliant with the Go1 BNF

The BNF isn't the entire spec. The relevant clause is this one (emphasis mine):

The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.

In this case, you can think of the universe of modules as the “repository of installed packages”, and the dotted-domain restriction ensures that module paths do not collide.

(It is possible that we'll revisit that requirement in light of the local-proxy use case, but I'm not sure either way. For what it's worth, Google also has a substantial body of code using non-domain-prefixed import paths.)

[shoenig]

FWIW we've got this patched version of go1.11 that we're using internally now which removes the "hostname" check. Our internal GOPROXY is the source of truth for managing namespaces, not go get.

As a side note we do plan on migrating away from the dot-less indeed/ prefix as well as the gross 3rdparty-vendoring-into-a-single-monolithic-nightmare-repository, but that's easier to do after migrating to Go modules, which gives us semver semantics and the ability to make those changes without disrupting (too much) people's workflow.

[rsc]

We want to allow experimentation, so we're not categorically cutting off all module statements in all contexts without dots, but anything that has to get resolved through downloading needs a dot and more generally should have a fully qualified domain name. That's how we've carved the name space between standard library and external things. I understand that pedantically the RFCs don't require dots but that's the separation we've used from very very early on with goinstall and later go get.