-
Notifications
You must be signed in to change notification settings - Fork 18k
x/build: darwin-amd64
LUCI builders are sometimes too slow
#65040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
The "darwin-amd64-13" builder on the old dashboard doesn't have Another possibility is that we're running more tests on Mac builders in LUCI than we are on the old dashboard. For one, the old dashboard uses the |
We discussed and looked into this a bit. This turns out not to be a consistent slowness: looking at https://ci.chromium.org/ui/p/golang/builders/ci/gotip-darwin-amd64_13 briefly shows many builds complete successfully in 15-20 min, with some taking 10 or 30 min, but the failures are taking an hour or longer and reaching timeout. This builder at this time has two different providers. The two recent timeouts happened on the same |
darwin-amd64_13
LUCI builder is too slowdarwin-amd64_13
LUCI builder is sometimes too slow or stalls
Update on this. I took a look at median, worst case, etc build times for the main repo builder by bot hostname and there definitely seems to be a lot of variance between different hosts: nearly 4x difference in median runtime between the fastest and slowest hosts. My first thought was slow network on some hosts, so I eyeballed the difference between a run on the fastest and slowest hosts and found that the slowdown seems to impact pretty much all packages. e.g., cmd/go 139s -> 540s Even small packages, like cmd/gofmt 0.064s -> 0.79s Some are similar though crypto/md5 0.021s -> 0.024s So it seems unrelated to network. That slowest machine is https://chromium-swarm.appspot.com/bot?id=darwin-amd64-13--ac2a2248-1530-4adb-bb91-ae854aa7c79c.golang.ca.macservice.goog. Interestingly, it seems to be much faster today. Previously x/tools was timing out, now it is ~15m. x/build is faster, etc. This definitely still needs more investigation. |
Found new dashboard test flakes for:
2023-12-08 00:50 gotip-darwin-amd64-nocgo go@4bf1ca4b cmd/fix.TestRewrite/jni.ArrayElement (log)
2024-01-08 16:00 gotip-darwin-amd64_13 go@881869dd cmd/go.TestScript (log)
2024-01-09 15:29 gotip-darwin-amd64_13 go@aba18d5b cmd/fix.TestRewrite (log)
2024-01-09 15:29 gotip-darwin-amd64_13 go@aba18d5b cmd/go.TestScript (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/cgo/internal/testplugin.TestIssue22175 (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/compile/internal/loopvar.TestLoopVarHashes (log)
2024-01-10 17:35 gotip-darwin-amd64_11 go@e9b3ff15 cmd/compile/internal/ssa.TestFmaHash (log)
2024-01-17 19:47 gotip-darwin-amd64_13 go@92d7169a cmd/fix.TestRewrite (log)
2024-01-17 19:47 gotip-darwin-amd64_13 go@92d7169a cmd/go.TestScript (log)
|
There seem to be timeouts on other LUCI darwin builders (other than 13). I put them all here for now. Feel free to split if this is really specific to 13. Thanks. |
Found new dashboard test flakes for:
2024-01-22 21:47 gotip-darwin-amd64_13 go@b7ae16e0 cmd/fix.TestRewrite (log)
2024-01-22 21:47 gotip-darwin-amd64_13 go@b7ae16e0 cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2024-01-04 17:31 go1.22-darwin-amd64_13 release-branch.go1.22@15dcdeb5 cmd/fix.TestRewrite [ABORT] (log)
2024-01-04 17:31 go1.22-darwin-amd64_13 release-branch.go1.22@15dcdeb5 cmd/go.TestScript (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/cgo/internal/testplugin.TestIssue22175 [ABORT] (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
2024-01-08 18:52 go1.22-darwin-amd64_11 release-branch.go1.22@8eaa7935 cmd/compile/internal/ssa.TestFmaHash (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/cgo/internal/testplugin.TestIssue22295 [ABORT] (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log)
2024-01-10 20:16 go1.21-darwin-amd64_11 release-branch.go1.21@2540b143 cmd/compile/internal/ssa.TestFmaHash (log)
|
Found new dashboard test flakes for:
2024-01-18 23:21 x_build-gotip-darwin-amd64_13 build@911ff433 go@66d34c7d x/build/internal/workflow.TestWatchdog/success (log)
|
Found new dashboard test flakes for:
2024-01-08 18:41 x_tools-gotip-darwin-amd64_13 tools@25a0e9d3 go@b702e043 x/tools/gopls/internal/test/integration/misc.TestStaticcheckRelatedInfo/default [ABORT] (log)
|
darwin-amd64_13
LUCI builder is sometimes too slow or stallsdarwin-amd64
LUCI builders are sometimes too slow
Found new dashboard test flakes for:
2024-01-30 18:58 gotip-darwin-amd64_13 go@0b1b6c15 runtime/pprof.TestVMInfo (log)
2024-01-30 23:37 gotip-darwin-amd64_12 go@ae457e81 runtime/pprof.TestVMInfo (log)
2024-01-30 23:37 gotip-darwin-amd64_14 go@ae457e81 runtime/pprof.TestVMInfo (log)
2024-01-31 14:20 gotip-darwin-amd64-nocgo go@13766fe7 runtime/pprof.TestVMInfo (log)
2024-01-31 16:50 gotip-darwin-amd64_13 go@1bb947b2 runtime/pprof.TestVMInfo (log)
2024-01-31 19:56 gotip-darwin-amd64_12 go@79738217 runtime/pprof.TestVMInfo (log)
2024-01-31 20:51 gotip-darwin-amd64_12 go@5c0d0929 runtime/pprof.TestVMInfo (log)
2024-01-31 20:51 gotip-darwin-amd64_13 go@5c0d0929 runtime/pprof.TestVMInfo (log)
2024-01-31 23:21 gotip-darwin-amd64_12 go@2f6a25f4 runtime/pprof.TestVMInfo (log)
2024-01-31 23:21 gotip-darwin-amd64_14 go@2f6a25f4 runtime/pprof.TestVMInfo (log)
2024-02-01 15:34 gotip-darwin-amd64_13 go@5b6cd3d0 runtime/pprof.TestVMInfo (log)
2024-02-01 18:09 gotip-darwin-amd64_12 go@18840865 runtime/pprof.TestVMInfo (log)
2024-02-01 19:29 gotip-darwin-amd64_13 go@29746b48 runtime/pprof.TestVMInfo (log)
2024-02-01 20:09 gotip-darwin-amd64_12 go@d864eac2 runtime/pprof.TestVMInfo (log)
2024-02-01 21:02 gotip-darwin-amd64_12 go@829f2ce3 runtime/pprof.TestVMInfo (log)
2024-02-01 21:42 gotip-darwin-amd64_13 go@c9d88ea2 runtime/pprof.TestVMInfo (log)
2024-02-01 21:42 gotip-darwin-amd64_14 go@c9d88ea2 runtime/pprof.TestVMInfo (log)
2024-02-01 22:47 gotip-darwin-amd64_13 go@ac08c05d runtime/pprof.TestVMInfo (log)
2024-02-01 22:57 gotip-darwin-amd64-nocgo go@6d3c1ce8 runtime/pprof.TestVMInfo (log)
2024-02-01 22:57 gotip-darwin-amd64_13 go@6d3c1ce8 runtime/pprof.TestVMInfo (log)
2024-02-01 22:59 gotip-darwin-amd64-nocgo go@117164f9 runtime/pprof.TestVMInfo (log)
2024-02-01 22:59 gotip-darwin-amd64_12 go@117164f9 runtime/pprof.TestVMInfo (log)
2024-02-02 15:13 gotip-darwin-amd64_12 go@244ccf47 runtime/pprof.TestVMInfo (log)
2024-02-02 15:22 gotip-darwin-amd64_13 go@149db960 runtime/pprof.TestVMInfo (log)
2024-02-02 17:06 gotip-darwin-amd64_12 go@3b83560b runtime/pprof.TestVMInfo (log)
|
Filed #65468 for |
Found new dashboard test flakes for:
2024-02-02 18:35 gotip-darwin-amd64_12 go@3d20a327 runtime/pprof.TestVMInfo (log)
2024-02-02 22:08 gotip-darwin-amd64_12 go@9819d826 runtime/pprof.TestVMInfo (log)
|
TestVMInfo failure is #62352. It seems that it matches this because it keeps retrying until it timed out, so |
I changed the pattern to "default", so the other issue should take priority. |
Found new dashboard test flakes for:
2024-02-07 19:42 x_tools-gotip-darwin-amd64-longtest tools@76ef6b6a go@1400b268 x/tools/gopls/internal/test/integration/misc.TestWindowsVendoring_Issue56291/default [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-07 21:58 x_tools-go1.22-darwin-amd64-longtest tools@5fcc6273 release-branch.go1.22@dd31ad7e x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckError/default [ABORT] (log)
2024-02-07 21:58 x_tools-go1.22-darwin-amd64-longtest tools@5fcc6273 release-branch.go1.22@dd31ad7e x/tools/gopls/internal/test/integration/misc.TestWindowsVendoring_Issue56291/default (log)
2024-02-08 13:20 x_tools-go1.22-darwin-amd64-longtest tools@f4fa7a75 release-branch.go1.22@20107e05 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd/experimental [ABORT] (log)
2024-02-08 13:20 x_tools-gotip-darwin-amd64-longtest tools@f4fa7a75 go@58fea5df x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckError2/experimental [ABORT] (log)
|
Found new dashboard test flakes for:
2024-02-09 17:28 x_tools-gotip-darwin-amd64-longtest tools@a5af84e3 go@4a7f3ac8 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd/experimental [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-05 16:17 x_tools-gotip-darwin-amd64-longtest tools@07219402 go@cad4dca5 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-05 16:17 x_tools-gotip-darwin-amd64-longtest tools@07219402 go@cad4dca5 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-05 20:18 x_tools-gotip-darwin-amd64-longtest tools@6a5b66be go@938b6c15 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-05 20:18 x_tools-gotip-darwin-amd64-longtest tools@6a5b66be go@938b6c15 x/tools/go/loader.TestLoad2 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@1cf6b502 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@1cf6b502 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@24d22352 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@24d22352 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@9bd21607 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@9bd21607 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@f55bb135 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@f55bb135 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@8a7742e7 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@8a7742e7 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@232dfd22 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@232dfd22 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@66b76403 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@66b76403 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@3aef7161 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@3aef7161 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@5e0dbec7 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@5e0dbec7 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@e7f9e760 x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-darwin-amd64-longtest tools@b08c7a26 go@e7f9e760 x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-07 17:40 x_tools-go1.24-darwin-amd64-longtest tools@7435a814 release-branch.go1.24@0ace2d8a x/tools/gopls/internal/test/marker.Test/callhierarchy/issue64451.txt [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-13 00:52 gotip-darwin-amd64-longtest go@b10c3594 cmd/go.TestScript/test_fuzz_deadline (log)
|
Found new dashboard test flakes for:
2025-03-20 14:07 x_tools-go1.23-darwin-amd64-longtest tools@3a64d744 release-branch.go1.23@15e01a2e x/tools/gopls/internal/test/integration/workspace.TestStdWorkspace/default [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-21 14:54 x_tools-go1.23-darwin-amd64_14 tools@c2768b73 release-branch.go1.23@15e01a2e x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd/legacy=false/default [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-04 20:31 go1.23-darwin-amd64-longtest release-branch.go1.23@45aade7f cmd/go.TestScript/gotoolchain_net (log)
|
Found new dashboard test flakes for:
2025-03-24 17:38 x_tools-gotip-darwin-amd64-longtest tools@961631ad go@6bf95d40 x/tools/internal/gcimporter.TestStdlib/gotypesalias=0 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-25 21:09 x_tools-go1.24-darwin-amd64-longtest tools@b75dab25 release-branch.go1.24@dae59b59 x/tools/gopls/internal/test/integration/workspace.TestStdWorkspace/default [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-25 21:39 x_tools-go1.24-darwin-amd64-longtest tools@8be0d5f6 release-branch.go1.24@e9162e7e x/tools/go/loader.TestLoad1 [ABORT] (log)
2025-03-25 21:39 x_tools-go1.24-darwin-amd64-longtest tools@8be0d5f6 release-branch.go1.24@e9162e7e x/tools/go/loader.TestLoad2 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-26 14:12 x_tools-gotip-darwin-amd64-longtest tools@b3ce3e13 go@9ce47e66 x/tools/gopls/internal/telemetry/cmd/stacks.TestUpdateIssues/open_issue [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-27 22:39 gotip-darwin-amd64-longtest go@b17a99d6 cmd/go.TestScript/mod_get_direct (log)
|
Found new dashboard test flakes for:
2025-04-04 19:13 x_tools-go1.24-darwin-amd64_13 tools@3e7f74d0 release-branch.go1.24@fa7217f7 x/tools/gopls/internal/test/integration/misc.TestWorkspaceSymbolMissingMetadata/default [ABORT] (log)
|
Found new dashboard test flakes for:
2025-04-07 20:36 gotip-darwin-amd64-longtest go@8054d2da cmd/go.TestScript/test_fuzz_fuzztime (log)
|
Found new dashboard test flakes for:
2025-04-08 20:26 gotip-darwin-amd64-longtest go@8433412b [build] (log)
2025-04-08 21:04 gotip-darwin-amd64-longtest go@ec4a9fb3 [build] (log)
|
Found new dashboard test flakes for:
2025-04-08 21:27 gotip-darwin-amd64-longtest go@53badd48 [build] (log)
|
Found new dashboard test flakes for:
2025-04-14 16:26 gotip-darwin-amd64_13 go@adf21a9b cmd/internal/testdir.Test/fixedbugs/issue46234.go (log)
|
Found new dashboard test flakes for:
2025-04-16 12:14 x_tools-gotip-darwin-amd64-nocgo tools@ee8f1384 go@2cb9e7f6 x/tools/go/packages.TestInvalidFilesBeforeOverlayContains/Modules/main [ABORT] (log)
|
Found new dashboard test flakes for:
2025-05-07 20:14 gotip-darwin-amd64_13 go@d6c8bedc cmd/internal/testdir.Test/fixedbugs/issue46234.go (log)
|
Found new dashboard test flakes for:
2025-05-07 18:43 gotip-darwin-amd64_12 go@86101b08 cmd/internal/testdir.Test/fixedbugs/issue46234.go (log)
2025-05-07 19:22 gotip-darwin-amd64_13 go@591c4b50 cmd/internal/testdir.Test/fixedbugs/issue46234.go (log)
|
Go version
aba18d5
Output of
go env
in your module/workspace:What did you do?
Check status of https://ci.chromium.org/p/golang/g/go-gotip/console.
What did you see happen?
Multiple failures involving timeouts on the
darwin-amd64_13
builder:What did you expect to see?
No timeouts:
go test -short cmd/go
takes <20s to run locally, so either the builders should be faster, or they should set aGO_TEST_TIMEOUT_SCALE
that is long enough to reliably run the tests.The only timeouts on the builders should be for (a) true deadlocks, and (b) tests that take a similarly long time (approaching 3 minutes) when run locally.
The text was updated successfully, but these errors were encountered: