Flaky test incorrectly determined

[ag321123]

I am not sure how a flaky test is determined. It seems like a failed test is run again, and the stack tracebacks are compared. I have a very simple example without any non-determinism which is marked flaky, and the tracebacks look the same.

// property based testing used rapid
func TestAbs6(t *testing.T) {
	rapid.Check(t, func(t *rapid.T) {
    value := rapid.Int().Draw(t, "value").(int)
    result := Abs(value)
    if (value * value) != (result * result) {
      t.Errorf("square of result not equal to square of input: %d, result %d\n", value, result)
    }
	})
}

// function to be tested
func Abs(x int) int {
  if x < 0 {
	    return -x
  }
  if x > 15 {
    return x - 1
  }
  return x - 1
}

abs_test.go:89: [rapid] flaky test, can not reproduce a failure
        To try to reproduce, specify -run="TestAbs6" -rapid.seed=1579796179831259001
        Traceback (square of result not equal to square of input: 25541, result 25540
        ):
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:188 in github.com/flyingmutant/rapid.checkOnce
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:142 in github.com/flyingmutant/rapid.doCheck
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:100 in github.com/flyingmutant/rapid.checkTB
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:71 in github.com/flyingmutant/rapid.Check
            /Users/allangold/go/src/pbt1/abs_test.go:89 in pbt1.TestAbs6
            /usr/local/go/src/testing/testing.go:827 in testing.tRunner
            /usr/local/go/src/runtime/asm_amd64.s:1333 in runtime.goexit
        Original traceback (square of result not equal to square of input: 25541, result 25540
        ):
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:188 in github.com/flyingmutant/rapid.checkOnce
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:163 in github.com/flyingmutant/rapid.findBug
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:134 in github.com/flyingmutant/rapid.doCheck
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:100 in github.com/flyingmutant/rapid.checkTB
            /Users/allangold/go/src/github.com/flyingmutant/rapid/engine.go:71 in github.com/flyingmutant/rapid.Check
            /Users/allangold/go/src/pbt1/abs_test.go:89 in pbt1.TestAbs6
            /usr/local/go/src/testing/testing.go:827 in testing.tRunner
            /usr/local/go/src/runtime/asm_amd64.s:1333 in runtime.goexit
        Failed test output:
    abs_test.go:90: [rapid] draw value: 25541
    abs_test.go:93: square of result not equal to square of input: 25541, result 25540

[flyingmutant]

Can you please try the latest master (import path is now pgregory.net/rapid)? . This is what I get (on both Windows and Linux, can't test on OS X right now):

λ go test -run=Abs6
--- FAIL: TestAbs6 (0.00s)
    failure_external_test.go:133: [rapid] failed after 1 tests: square of result not equal to square of input: 0, result -1

        To reproduce, specify -run="TestAbs6" -rapid.seed=1579804795669640702
        Failed test output:
    failure_external_test.go:134: [rapid] draw value: 0
    failure_external_test.go:137: square of result not equal to square of input: 0, result -1
FAIL
exit status 1
FAIL    pgregory.net/rapid      0.547s

[ag321123]

Yes, that did it. It eliminate the flaky test message. Thanks, Allan

…

On Jan 23, 2020, at 2:15 PM, Gregory Petrosyan ***@***.***> wrote: Can you please try the latest master (import path is now pgregory.net/rapid)? . This is what I get (on both Windows and Linux, can't test on OS X right now): λ go test -run=Abs6 --- FAIL: TestAbs6 (0.00s) failure_external_test.go:133: [rapid] failed after 1 tests: square of result not equal to square of input: 0, result -1 To reproduce, specify -run="TestAbs6" -rapid.seed=1579804795669640702 Failed test output: failure_external_test.go:134: [rapid] draw value: 0 failure_external_test.go:137: square of result not equal to square of input: 0, result -1 FAIL exit status 1 FAIL pgregory.net/rapid 0.547s — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#8?email_source=notifications&email_token=AK6HSUBKHOOU4NS2JPL7FGTQ7HUDVA5CNFSM4KKZJZUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYQDHI#issuecomment-577831325>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK6HSUFE2PTAPNLOCIWOUCTQ7HUDVANCNFSM4KKZJZUA>.

[flyingmutant]

Great!

I've found go module system to be a little flaky when it comes to changes of import path. E.g. for me, https://play.golang.org/p/ZdNjIGzT_EX says

go: finding pgregory.net/rapid v0.2.2
go: downloading pgregory.net/rapid v0.2.2
go: extracting pgregory.net/rapid v0.2.2
go: pgregory.net/rapid: pgregory.net/rapid@v0.2.2: parsing go.mod:
	module declares its path as: github.com/flyingmutant/rapid
	        but was required as: pgregory.net/rapid

which is bogus, because the only published version of pgregory.net/rapid is v0.3.0. This may be related to the issue you've reported: change of import path affects tracebacks, which are used to check for flakiness.

[pgaskin]

The playground import should work properly now. The issue is that the playground uses modules from proxy.golang.org, which only gets updated when someone tries to get a version from it. In addition, the playground caches results, including errors.

To fix this kind of thing in the future, just run go get pgregory.net/rapid@v0.3.0 from within a Go module to force the release to be cached. To force a Go playground run to bypass the cache, just add whitespace or some other trivial change.

https://play.golang.org/p/IZ4XfquYYFG

https://pkg.go.dev/pgregory.net/rapid@v0.3.0?tab=versions

[flyingmutant]

Thanks for the information; it is unfortunate to have to work around the playground cache.

I did try the v0.3.0 in a local Go module immediately after pushing the tag to ensure that everything is working.

The weird thing is, sometimes playground did work, so playground cache is not always used; also, the cache is not shared between different playground URLs, which is unexpected.