testing: early break in a testing.B.Loop benchmark loop should be reported. #72933
Description
Go version
go version go1.24.1 darwin/arm64
Output of go env in your module/workspace:
AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/shaojunyang/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/shaojunyang/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/6t/d5y6mrc90rzd1gm2q2k99lb400zg0c/T/go-build3077995788=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/Users/shaojunyang/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/shaojunyang/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/shaojunyang/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'What did you do?
For benchmark that breaks with its own logic, -benchtime won't work and the users should be notified.
What did you see happen?
If we run the following benchmark with go test -bench=. -benchtime=1s:
func BenchmarkFoo(b *testing.B) {
i := 0
for b.Loop() {
fmt.Printf("iter %d\n", i)
i += 1
if i > 5 {
break
}
time.Sleep(100 * time.Millisecond)
}
}
The output would be
iter 0
iter 1
iter 2
iter 3
iter 4
iter 5
goos: darwin
goarch: arm64
cpu: Apple M1 Pro
BenchmarkFoo-10 10 50555750 ns/op
PASS
ok command-line-arguments 0.732s
What did you expect to see?
Instead of allowing the benchmark to pass with bogus numbers, we should let the user know that the benchmark failed.