cmd/coordinator: use 'go tool dist test -k' on release branches

Right now, on release branches, testing is stopped very soon after
encountering the first package with a failing test. For example:

	[...]
	FAIL	cmd/go/internal/modfetch	17.181s
	ok  	cmd/go/internal/modfetch/codehost	5.474s
	FAIL
	2019/11/07 02:35:34 Failed: exit status 1

Ideally we should not have failing tests on release branches for long,
but that is not the current state. Until we reach that state, make
coordinator run 'go tool dist test' with -k flag so that it keeps
testing all remaining packages even if one fails, and report complete
test results in the log.

That way, a package that fails early doesn't make it completely
impossible to see if other package tests pass or fail.

Updates golang/go#14305
Updates golang/go#36181

Change-Id: Ia674005ae45c6b9dbffa6f5b56d60b7ed38f6b34
Reviewed-on: https://go-review.googlesource.com/c/build/+/211678
Run-TryBot: Dmitri Shuralyov <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
Reviewed-by: Alexander Rakoczy <[email protected]>

Author: dmitshur

cmd/coordinator/coordinator.go

@@ -2059,6 +2059,19 @@ func (st *buildStatus) build() error {
 
 func (st *buildStatus) HasBuildlet() bool { return atomic.LoadInt32(&st.hasBuildlet) != 0 }
 
+// useKeepGoingFlag reports whether this build should use -k flag of 'go tool
+// dist test', which makes it keep going even when some tests have failed.
+func (st *buildStatus) useKeepGoingFlag() bool {
+	// For now, keep going for post-submit builders on release branches,
+	// because we prioritize seeing more complete test results over failing fast.
+	// Later on, we may start doing this all post-submit builders on all branches.
+	// See golang.org/issue/14305.
+	//
+	// TODO(golang.org/issue/36181): A more ideal long term solution is one that reports
+	// a failure fast, but still keeps going to make all other test results available.
+	return !st.isTry() && strings.HasPrefix(st.branch, "release-branch.go")
+}
+
 func (st *buildStatus) isTry() bool { return st.trySet != nil }
 
 func (st *buildStatus) isSlowBot() bool {
@@ -3236,6 +3249,9 @@ func (st *buildStatus) runTestsOnBuildlet(bc *buildlet.Client, tis []*testItem,
 	if st.conf.CompileOnly {
 		args = append(args, "--compile-only")
 	}
+	if st.useKeepGoingFlag() {
+		args = append(args, "-k")
+	}
 	args = append(args, names...)
 	var buf bytes.Buffer
 	t0 := time.Now()