cmd/go: non-root modules not fetched from a repository hosted on GitHub Enterprise 2.16.12

[augustoroman]

What version of Go are you using (go version)?

$ go version
go version go1.13.7 darwin/amd64

Does this issue reproduce with the latest release?

Yes, and with go1.14beta1.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/augusto.roman/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.7/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOMOD=""

The GHE server version is 2.16.12

What did you do?

I created a test repository on the company's internal Github Enterprise (GHE) that had a module that was not at the root of the repo. I was unable to import the module from elsewhere using go get:

ghe.company.com/user/repo/packages/lib
  with:  go.mod  --> "module ghe.company.com/user/repo/packages/lib"
         lib.go

+ git tag: packages/lib/v1.0.0

Then elsewhere I tried:

GOPRIVATE=ghe.company.com go get ghe.company.com/user/repo/packages/lib@master

(or with tags of the format: packages/lib/v1.0.0)

go get Output

$ GOPRIVATE=ghe.company.com go get -x -v ghe.company.com/user/repo/packages/[email protected]
# get https://ghe.company.com/?go-get=1
# get https://ghe.company.com/user/repo/packages/lib?go-get=1
# get https://ghe.company.com/user/repo/packages?go-get=1
# get https://ghe.company.com/user/repo?go-get=1
# get https://ghe.company.com/user?go-get=1
# get //ghe.company.com/user/repo?go-get=1: 200 OK (0.364s)
get "ghe.company.com/user/repo": found meta tag get.metaImport{Prefix:"ghe.company.com/user/repo", VCS:"git", RepoRoot:"https://ghe.company.com/user/repo.git"} at //ghe.company.com/user/repo?go-get=1
mkdir -p /Users/augusto.roman/go/pkg/mod/cache/vcs # git3 https://ghe.company.com/user/repo.git
# lock /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447.lock# /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447 for git3 https://ghe.company.com/user/repo.git
go: finding ghe.company.com/user/repo v1.0.0
cd /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447; git tag -l
0.011s # cd /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447; git tag -l
cd /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447; git ls-remote -q https://ghe.company.com/user/repo.git
# get //ghe.company.com/?go-get=1: 200 OK (0.427s)
# get //ghe.company.com/user?go-get=1: 200 OK (0.435s)
# get //ghe.company.com/user/repo/packages?go-get=1: 200 OK (0.466s)
# get //ghe.company.com/user/repo/packages/lib?go-get=1: 200 OK (0.469s)
0.538s # cd /Users/augusto.roman/go/pkg/mod/cache/vcs/6c97662eddd96ff8454c4d00d298aa628735acaad52e65b5a3eb02a6df0b1447; git ls-remote -q https://ghe.company.com/user/repo.git
# get https://ghe.company.com/user/repo?go-get=1
# get https://ghe.company.com/?go-get=1
# get https://ghe.company.com/user/repo/packages?go-get=1
# get https://ghe.company.com/user?go-get=1
# get https://ghe.company.com/user/repo/packages/lib?go-get=1
# get //ghe.company.com/user/repo?go-get=1: 200 OK (0.076s)
get "ghe.company.com/user/repo": found meta tag get.metaImport{Prefix:"ghe.company.com/user/repo", VCS:"git", RepoRoot:"https://ghe.company.com/user/repo.git"} at //ghe.company.com/user/repo?go-get=1
go: finding ghe.company.com/user/repo v1.0.0
# get //ghe.company.com/?go-get=1: 200 OK (0.180s)
# get //ghe.company.com/user/repo/packages?go-get=1: 200 OK (0.198s)
# get //ghe.company.com/user?go-get=1: 200 OK (0.209s)
# get //ghe.company.com/user/repo/packages/lib?go-get=1: 200 OK (0.251s)
go get ghe.company.com/user/repo/packages/[email protected]: unrecognized import path "ghe.company.com/user/repo/packages/lib" (parse https://ghe.company.com/user/repo/packages/lib?go-get=1: no go-import meta tags ())

Workaround

It works if I name the module with repo.git instead of repo as:

ghe.company.com/user/repo/packages/lib
  with:  go.mod  --> "module ghe.company.com/user/repo.git/packages/lib"
         lib.go

And elsewhere I can do:

GOPRIVATE=ghe.company.com go get ghe.company.com/user/repo.git/packages/lib@master

or I can use tags of the format: packages/lib/v1.2.3

However, this only works if the go.mod file itself names the module with the .git suffix.

---

This is in contrast to importing a module at the root which works fine with or without the .git suffix.

(related #17898)

[bcmills]

The protocol that the go command uses to resolve import paths (other than from a handful of unfortunately hard-coded services) is described at https://golang.org/cmd/go/#hdr-Remote_import_paths. Most likely the server you are using does not fully implement that protocol.

So, what is the output of curl -sL https://ghe.company.com/user/repo/packages/lib?go-get=1 | grep go-import?

If the server implements the protocol correctly, it should be something like:

$ curl -sL https://ghe.company.com/user/repo/packages/lib?go-get=1 | grep go-import
	<meta name="go-import" content="ghe.company.com/user/repo git https://ghe.company.com/user/repo.git">

[bcmills]

According to https://enterprise.github.com/releases, the current GHE release is 2.19.6. Does this issue reproduce with that version of the server?

[bcmills]

(Note that per https://enterprise.github.com/releases/series/2.16, “GitHub Enterprise Server 2.16 [is] deprecated as of January 22, 2020[.]”)

[augustoroman]

Thanks @bcmills, the output for checking the subdirectory is nothing, only the root repo responds with a go-import meta tag:

$ curl -sL https://ghe.company.com/user/repo.git/packages/lib?go-get=1 | grep go-import
$ curl -sL https://ghe.company.com/user/repo.git/?go-get=1 | grep go-import
  <meta name="go-import" content="ghe.company.com/user/repo git https://ghe.company.com/user/repo.git">
$

I don't know what the current release response is -- I'll look into upgrading our server.

[bcmills]

Argh, I typo'd the command. Should be curl -sL https://ghe.company.com/user/repo/packages/lib?go-get=1 | grep go-import, omitting the internal .git suffix.

[augustoroman]

Same output:

$ curl -sL https://ghe.company.com/user/repo/packages/lib?go-get=1 | grep go-import
$ curl -sL https://ghe.company.com/user/repo/?go-get=1 | grep go-import
  <meta name="go-import" content="ghe.company.com/user/repo git https://ghe.company.com/user/repo.git">
$

[augustoroman]

Still seems buggy to me. If only the exact import path is allowed, why does it bother checking all the parent paths? Public github also doesn't respond to subpath imports

$ curl -sL https://github.com/augustoroman/test-go-modules/packages/lib?go-get=1 | grep go-import
$ curl -sL https://github.com/augustoroman/test-go-modules/?go-get=1 | grep go-import
  <meta name="go-import" content="github.com/augustoroman/test-go-modules git https://github.com/augustoroman/test-go-modules.git">

...but the import works fine. I realize there may be hardcoded exceptions, but it seems to me that if a git repo responds at the root then go get should check the git repo for appropriate sub-packages in the same manner that adding the .git suffix to the repo works.

[bcmills]

go get accepts any package import path, not (only) an exact module path. It is checking the parent paths in order to attempt to locate the modules corresponding to those paths. (The package with import path ghe.company.com/user/repo/packages/lib could be provided by a module with any path that is a prefix of that import path.)

In contrast, other subcommands (such as go mod download) do know the exact module path, so if the server tells one of those subcommands that the module does not exist, that's it — no other paths are attempted.

So it is important that go get only resolve a module using the same path that any other subcommand would use.

[bcmills]

Public GitHub is (unfortunately) hard-coded, so any bugs in its implementation of the go-import protocol are masked. (It is not a reliable example to emulate.)

[augustoroman]

I've tested this on GHE 2.19.3 and it works as expected, with the subdirectories returning a proper go-import meta tag.

[bcmills]

Great! Closing this issue, since the go command seems to be working as documented and we don't appear to need to follow up with the GHE maintainers.