cmd/go: modules do not work with Gitlab.com/Company/Team/Repo Path format

[Marvalero]

What version of Go are you using (go version)?

root@670ea5e97a56:/app# go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

root@670ea5e97a56:/app# go version
go version go1.11.2 linux/amd64
root@670ea5e97a56:/app# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/app/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build057206232=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I was testing if go modules made sense for my company. We want to use it to create private libraries. So I wrote a simple test to check I can import submodules:

// File: main.go 
package main

import (
	"gitlab.com/Beamery/DevOps/gotalker"
	"gitlab.com/Beamery/gozzlightyear"
	"gitlab.com/maria.campana/sounds"
)

func main() {
	sounds.Run() // This just calls fmt.Println("Muuu")
	gozzlightyear.Run() // This just calls fmt.Println("to infinity and beyond")
	gotalker.Run()  // This just calls fmt.Println("there is a snake in my boot")
}

// File: go.mod 
module gitlab.com/Beamery/DevOps/maria-modules-test

Then I try to build the code. I am actually running my code on a container. This is the setup I do:

$ docker run -it --entrypoint bash -v /tmp/maria-modules-test:/app -v /home/maria/.ssh/:/root/.ssh/ golang:1.11.2-stretch
root@670ea5e97a56# cd /app
root@670ea5e97a56# git config --global url."ssh://[email protected]/".insteadOf "https://gitlab.com/"
root@670ea5e97a56:/app# go build
build gitlab.com/Beamery/DevOps/maria-modules-test: cannot find module for path gitlab.com/Beamery/DevOps/gotalker

I think the problem is that the repo name is server.com/Company/Team/Repo and go expects to see server.com/Company/Repo. I tried removing the path "gitlab.com/Beamery/DevOps/gotalker" the app gets built correctly

What did you expect to see?

I expect Go Modules to support longer paths like server.com/Company/Team/Repo

What did you see instead?

I see build gitlab.com/Beamery/DevOps/maria-modules-test: cannot find module for path gitlab.com/Beamery/DevOps/gotalker

And If I manually add a require for it on go.mod

module gitlab.com/Beamery/DevOps/maria-modules-test

require (
	gitlab.com/Beamery/gozzlightyear v1.0.0
	gitlab.com/maria.campana/sounds v1.0.0
	gitlab.com/Beamery/DevOps/gotalker v1.0.0
)

I see I try to import server.com/Company/Team/Repo and it will try to clone server.com/Company/Team and look for the tag version Repo/v1.0.0

root@670ea5e97a56:/app# go build
go: finding gitlab.com/Beamery/DevOps/gotalker v1.0.0
go: gitlab.com/Beamery/DevOps/[email protected]: unknown revision gotalker/v1.0.0
go: error loading module requirements

[agnivade]

I think you have to add a go.mod to the gotalker repo. Without that, the module path does not match.

Roping in @myitcv for a deeper look.

[Marvalero]

I tried adding a go.mod to my "gitlab.com/Beamery/DevOps/gotalker" library but I cannot fetch it from my other repo. Thanks for looking into it tho 😄

[JAicewizard]

https://gitlab.com/antipy/antibuild/api works. maybe see if there is any diferences (or check if you can import that?)

[myitcv]

@Marvalero - please include the output from go build -x -v. Although from first glance it appears to suggest that gitlab.com/Beamery/DevOps/gotalker has not been tagged with v1.0.0.

[Marvalero]

I created gotalker, gozzlightyear and sounds and they all have a file with a Run() function and tags. If there was something missing in gotalker, it would be missing in the other 2 repos.

This is the output of go build -x -v (Also: thanks! nice command)

root@fd4caec26a46:/app# go build -x -v 
WORK=/tmp/go-build291933567
Fetching https://gitlab.com/Beamery/DevOps/gotalker?go-get=1
Parsing meta tags from https://gitlab.com/Beamery/DevOps/gotalker?go-get=1 (status code 200)
get "gitlab.com/Beamery/DevOps/gotalker": found meta tag get.metaImport{Prefix:"gitlab.com/Beamery/DevOps", VCS:"git", RepoRoot:"https://gitlab.com/Beamery/DevOps.git"} at https://gitlab.com/Beamery/DevOps/gotalker?go-get=1
get "gitlab.com/Beamery/DevOps/gotalker": verifying non-authoritative meta tag
Fetching https://gitlab.com/Beamery/DevOps?go-get=1
Parsing meta tags from https://gitlab.com/Beamery/DevOps?go-get=1 (status code 200)
# /go/pkg/mod/cache/vcs/0001a1afefa10f4157c4164eafd56dab52e8d34658da16e33602533f22c91849 for git2 https://gitlab.com/Beamery/DevOps.git
cd /go/pkg/mod/cache/vcs/0001a1afefa10f4157c4164eafd56dab52e8d34658da16e33602533f22c91849; git ls-remote -q https://gitlab.com/Beamery/DevOps.git
2.255s # cd /go/pkg/mod/cache/vcs/0001a1afefa10f4157c4164eafd56dab52e8d34658da16e33602533f22c91849; git ls-remote -q https://gitlab.com/Beamery/DevOps.git
Fetching https://gitlab.com/Beamery/DevOps?go-get=1
Parsing meta tags from https://gitlab.com/Beamery/DevOps?go-get=1 (status code 200)
get "gitlab.com/Beamery/DevOps": found meta tag get.metaImport{Prefix:"gitlab.com/Beamery/DevOps", VCS:"git", RepoRoot:"https://gitlab.com/Beamery/DevOps.git"} at https://gitlab.com/Beamery/DevOps?go-get=1
Fetching https://gitlab.com/Beamery?go-get=1
Parsing meta tags from https://gitlab.com/Beamery?go-get=1 (status code 200)
Fetching https://gitlab.com?go-get=1
Parsing meta tags from https://gitlab.com?go-get=1 (status code 200)
build gitlab.com/Beamery/DevOps/maria-modules-test: cannot find module for path gitlab.com/Beamery/DevOps/gotalker

[Marvalero]

Also, just in case you had doubts:

1. Beamery/Devops repo does not exists

root@fd4caec26a46:/app# git clone https://gitlab.com/Beamery/DevOps.git
Cloning into 'DevOps'...
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

2. Gotalker exists (but it is private):

root@fd4caec26a46:/# git clone https://gitlab.com/Beamery/DevOps/gotalker.git
Cloning into 'gotalker'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 11 (delta 1), reused 0 (delta 0)
Receiving objects: 100% (11/11), done.
Resolving deltas: 100% (1/1), done.

And it has many tags (I created them to see if I could fix this import problem):

root@fd4caec26a46:/gotalker# git tag --list
gotalker/v1.0.0
gotalker/v1.0.1
v0.4.0
v1.0.0
v1.0.1

[bcmills]

In general it is up to the server to tell the go tool where to find a given repository. We have a few hosting services hard-coded, but gitlab.com is not one of them.

The protocol for other services uses an HTTPS request to resolve the repo root, and is documented here. So in this case the go command will issue a request for https://gitlab.com/Beamery/DevOps/gotalker?go-get=1.

When I do that (using curl https://gitlab.com/Beamery/DevOps/gotalker?go-get=1), I get the following response:

<html><head><meta name="go-import" content="gitlab.com/Beamery/DevOps git https://gitlab.com/Beamery/DevOps.git" /></head></html>

If I understand your description correctly, the actual repo location should instead be https://gitlab.com/Beamery/DevOps/gotalker.git.

So it seems that the issue here is in @gitlabhq's server, and you'll need to escalate to them.

[JAicewizard]

gitlab is not willing to propperly support the protocol for repo privacy reasons, would it be an option to add a hardcoded solution for gitlab with a special indicator indicating the repo(and the rest the path from the root) or something like that?
the fact that gitlab doesnt work propperly with go is frustating a lot of people and pretty much marked as no-fix on gitlabs end(privacy reasons)

[bcmills]

Sorry, but “privacy reasons” — without any sort of detail as to what those reasons are — is not a sufficient justification for hard-coding provider details into the go command.

We do need some sort of general authentication support for HTTPS fetches, but that's already tracked in #26232. Is there reason to believe that that won't work for GitLab?

[JAicewizard]

the issue with gitlab is documented here(https://gitlab.com/gitlab-org/gitlab-ce/issues/1337)
this is indeed discussed in #26232 and if that would come to live it could difinitly fix the issue.