cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./...

[AlexRouSg]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/chotepud/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/.local/go"
GOTMPDIR=""
GOTOOLDIR="~/.local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build941063065=/tmp/go-build -gno-record-gcc-switches"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

package test

import "fmt"

func Foo() {
	fmt.Println("Hello World")
}

package test_test

import (
	"testing"

	"github.com/AlexRouSg/test"
)

func TestFoo(t *testing.T) {
	test.Foo()
}

Run both go test -coverpkg=all and go test -coverpkg=all ./...

What did you expect to see?

Same coverage value for go test -coverpkg=all vs go test -coverpkg=all ./...

What did you see instead?

go test -coverpkg=all ./...

ok github.com/AlexRouSg/test 0.002s coverage: 16.1% of statements in all

go test -coverpkg=all

Hello World
PASS
coverage: 14.8% of statements in all
ok github.com/AlexRouSg/test 0.002s

In actual packages the difference could be as big as go test -coverpkg=all ./... 47.3% vs go test -coverpkg=all 94.4% when comparing the same package.

[ianlancetaylor]

It's not clear to me that it has anything to do with whether you specify ./... or not. If I use go test -c -coverpkg=all to generate a test binary, I see different coverage numbers each time I run it.

One issue is that -coverpkg=all really means all. All packages, including the standard library, are recompiled for coverage. It's not clear that this is a useful or meaningful thing to do. It may not even be possible to get consistent coverage results in this case.

[ianlancetaylor]

I'm marking this for 1.11 since as far as I can tell -coverpkg=all in 1.9 tried to recompile everything in GOPATH. At least we're doing better than that.

[AlexRouSg]

@ianlancetaylor
I think I finally figured out what's the deal with the different numbers when I ran test with ./... and without.

To make this easier to explain, I have a test only package containing some non test files to run cgo functions.

When running test without ./... -coverpkg only matches my test package for some reason, which is why it gave 94.4%. However running with ./... includes my other packages that I'm actually interested in, which then gave a lower value of 47.3%.

Modifiying the test case such that func Foo is in a separate folder than the test function should show 0% coverage vs 100% coverage with running test -coverpkg=foopackage/... with ./... and without.

[jdef]

I'm having a related problem where I'm using go test -c ... to generate a test-binary. The test-binary is generated from package ./e2e and this package ONLY contains test code. I've tried generating the binary with -coverpkg set to ./... or all and I consistently get 0% coverage, even though the executed tests leverage tons of other packages in the project. If I specify a fully-qualified package name for coverpkg (for a package other than e2e) then go test -c ... complains that I've specified a package that's not exercised by the test -- which is crazy talk

warning: no packages being tested depend on matches for pattern github.com/myorg/myproject/pkg/...

[AlexRouSg]

@jdef Try running go test -c ./... on the root package or passing a list of all the package you want coverage for. I have a hunch that coverage only checks against packages you're testing and maybe the std.

[jdef]

data point: I created a coverage_imports.go file in my project's e2e/ (test-only) package. it anonymously imports every other package in the project, besides itself. when I pre-compile the e2e/ test binary (via go test -c -coverpkg ./... ./e2e) then the subsequent execution of said binary FINALLY emits a coverage report for the other packages in the project. this feels pretty hacky, but I'm not aware of a better workaround.

[rsc]

/cc @bcmills

[founderio]

Description of my experiences, which led me here: https://groups.google.com/forum/#!topic/golang-nuts/63zlrBhnNw8

My use case: One package with tests I want to run isolated from the other tests but report coverage on all dependent sibling packages.

[maxxgl]

run isolated from the other tests but report coverage on all dependent sibling packages.

I have a similar use case. I have it working under windows with:

go test my_test_package -coverpkg "sibling_pkg"

even though it throws the "no packages being tested depend on match for pattern" warning.

I absolutely cannot make it work under Ubuntu 18.04.

[bcmills]

Likely the same underlying cause as #23910.

[thanm]

Not completely clear I understand all aspects of this bug, but I agree with @bcmills that this is most likely the same bug as #23910. FWIW we've moved away from the model in which all -coverpkg packages are force-imported into every test, this was done as part of the coverage redesign in Go 1.20. Closing this bug out, please reopen if needed.