x/vgo: internal directories inconsistently handled from package main in repo root

[philpennock]

What version of Go are you using (go version)?

go version go1.10 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/pdp/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/pdp/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/vs/7q9nz5955_z7b7zxfz437zrm0000gn/T/go-build533554610=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I tried to set up an existing dummy application to be vgo-compatible.
The application is go get go.pennock.tech/dummyapp cloneable directly as https://github.com/PennockTech/dummyapp (a mutable implementation detail).

The application does not use an import declaration.

First attempt was a setup within my existing $GOPATH directory, for maximum compatibility:

% pwd
/Users/pdp/go/src/go.pennock.tech/dummyapp
% vgo build
vgo: finding golang.org/x/sys v0.0.0-20180202135801-37707fdb30a5
vgo: finding golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4
vgo: finding github.com/sirupsen/logrus v1.0.4
vgo: finding github.com/heroku/x v0.0.0-20171004170240-705849e307dd
vgo: resolving import "go.pennock.tech/dummyapp/internal/logging"
vgo: finding go.pennock.tech/dummyapp (latest)
vgo: adding go.pennock.tech/dummyapp v0.0.1
vgo: finding go.pennock.tech/dummyapp v0.0.1
vgo: downloading go.pennock.tech/dummyapp v0.0.1
vgo: downloading github.com/sirupsen/logrus v1.0.4
vgo: downloading golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4
vgo: downloading golang.org/x/sys v0.0.0-20180202135801-37707fdb30a5
main.go:30:2: use of internal package not allowed
main.go:31:2: use of internal package not allowed
main.go:32:2: use of internal package not allowed
% touch go.mod
% vgo build
main.go:30:2: use of internal package not allowed
main.go:31:2: use of internal package not allowed
main.go:32:2: use of internal package not allowed

After confirming that the app is go get compatible, and double-checking https://golang.org/cmd/go/#hdr-Internal_Directories to be sure that my usage is correct per normal go rules, I tried a build from a place without $GOPATH set; this at least got me much better diagnostics, so that after making one change I finally got a working build.

% touch go.mod
% vgo build
vgo: cannot determine module path for legacy source directory /Users/pdp/tmp/gv/dummyapp (outside GOROOT, no import comments)

then:

% vi main.go #...
% git diff -U0 |cat
diff --git a/main.go b/main.go
index 726fae5..bf430d8 100644
--- a/main.go
+++ b/main.go
@@ -15 +15 @@
-package main
+package main // import "go.pennock.tech/dummyapp"
% vgo build
vgo: resolving import "github.com/sirupsen/logrus"
vgo: finding github.com/sirupsen/logrus (latest)
vgo: adding github.com/sirupsen/logrus v1.0.4
vgo: resolving import "golang.org/x/crypto/ssh/terminal"
vgo: finding golang.org/x/crypto (latest)
vgo: adding golang.org/x/crypto v0.0.0-20180222182404-49796115aa4b
vgo: resolving import "golang.org/x/sys/unix"
vgo: finding golang.org/x/sys (latest)
vgo: adding golang.org/x/sys v0.0.0-20180222210305-c1138c84af3a
vgo: finding golang.org/x/sys v0.0.0-20180222210305-c1138c84af3a
vgo: finding golang.org/x/crypto v0.0.0-20180222182404-49796115aa4b
vgo: downloading golang.org/x/crypto v0.0.0-20180222182404-49796115aa4b
vgo: downloading golang.org/x/sys v0.0.0-20180222210305-c1138c84af3a
vgo build  18.33s user 5.41s system 178% cpu 13.314 total

Applying the same import path constraint to the version inside $GOPATH did not help and still errored:

% pwd
/Users/pdp/go/src/go.pennock.tech/dummyapp
% git diff -U0 | cat
diff --git a/main.go b/main.go
index 726fae5..bf430d8 100644
--- a/main.go
+++ b/main.go
@@ -15 +15 @@
-package main
+package main // import "go.pennock.tech/dummyapp"
% vgo build
main.go:30:2: use of internal package not allowed
main.go:31:2: use of internal package not allowed
main.go:32:2: use of internal package not allowed

What did you expect to see?

1. Within a normal $GOPATH setup, the use of internal package not allowed error should not have happened, or the language specification needs refining to clarify that this usage is wrong.
2. The error message (outside GOROOT, no import comments) looks buggy; behavior differed from that seen when inside GOPATH, I suspect that the wrong variable is being mentioned in the error.
3. The hard-requirement for a package import declaration for vgo to work outside of any tree listed in $GOPATH should be documented.
4. The prohibition on access to internal, even with an import declaration, but only if currently inside $GOPATH, should not have happened

What did you see instead?

No way to have this go get compliant code work with vgo when inside $GOPATH, though I can at least make it work outside.

[philpennock]

I found issue #23970 while searching for an existing bug before filing this one, but it appeared to me to be a distinct issue which just happens to affect internal, so I opened a new bug.

[rsc]

I tried this and reproduced your failure. In my case, it happened because vgo picked the wrong module path, which made it go look elsewhere for the go.pennock.tech imports, which then were not importable from the github path. That is, after the failure I had:

$ cat go.mod
module "github.com/PennockTech/dummyapp"

require (
	"go.pennock.tech/dummyapp" v0.1.0
)
$ 

If I start with the module path into the go.mod file, everything is fine:

echo 'module "go.pennock.tech/dummyapp"' >go.mod
vgo build

The auto-initializer is not perfect, unfortunately.