Repository navigation
GO/TESTS: Retry epoll_wait on EINTR in worker tests - #12273
Conversation
epoll_wait() is never restarted after a signal handler, so a signal delivered to the thread that waits in it, such as the Go runtime preemption signal, fails TestUcpWorkerEfd with EINTR. The helper goroutine then called t.Fatalf(), which exits only that goroutine, so nothing was sent on eventsChan and the test goroutine waited until the 10 minute go test timeout. Retry epoll_wait() on EINTR, and send errors from the helper goroutine back to the test goroutine instead of calling t.Fatalf() there. Wait for the result in TestUcpWorkerEfd and TestUcpWorkerWakeup for at most 10 seconds, and check the ucpWorker.Wait() error, which was ignored.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The gtest stage failed with exactly one failing test, Full analysisSummary: Root cause: Evidence from the log: the whole suite ran to completion (no hang — the largest inter-line gap in the fetched tail is ~1 s, and the run finished with a normal teardown and TOP-20 timing report), and only the Crucially, the commit under test ( Note: the exact gtest assertion text for the failing case is earlier in the job log than the ~500-line tail the log API returns; only the final Implicated commit: Not the PR commit. The failure lies in the EP-flush path last substantially changed by File: Suggested fix:
Related: PR #12273 (the PR under test, Go tests only), PR #12266 (UCP/FLUSH: Flush the other lanes after the pending lane — likely fix), PR #11923 (topology-safe selective endpoint flush)
|
If TestUcpWorkerWakeup times out, its goroutine is still in ucpWorker.Wait(), and the deferred Close() destroyed the worker and the context under it. The same happened in TestUcpWorkerEfd when the goroutine did not start in 500 ms, while it may still call ucpWorker.Arm() or Progress(). Leak the worker and the context in these cases instead. Also make eventsChan buffered, so the TestUcpWorkerEfd goroutine can exit after the test stopped waiting for it.
|
🤖 Starting review — findings will be posted here when done. |
TestUcpWorkerEfd destroyed the worker when epoll_wait() timed out, while its goroutine still waited on the worker efd. Leak the worker there as well, so that a late event can still wake the goroutine, which then exits through the buffered eventsChan. Also wait wakeupTimeout instead of 500 ms for the goroutine to start, and pass the leak flag to closeWorker() by value from a deferred closure.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Azure job "roce on worker 0" (build 143447) failed at Root cause: For the Implicated commit: unknown — no recent commit on the implicated files ( File: test/gtest/uct/test_md.cc:559 (threshold check), with the failing allocation at src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2174-2180 ( Suggested fix:
Related: PR #12273 (the triggering PR, unrelated to the failure); no existing issue found for |
TestUcpWorkerEfd and TestUcpWorkerWakeup ignored the ucpWorker.Signal() error, so a failed signal showed up only as a 10 second wait timeout. Fail the test with that error instead, and leak the worker there as well, since the helper goroutine still waits on it. Also let the callers decide whether to leak the worker, and make closeWorker() always destroy the worker and the context.
|
🤖 Starting review — findings will be posted here when done. |
TestUcpWorkerEfd and TestUcpWorkerWakeup ignored the NewUcpContext() error. NewUcpContext() returns a nil context on failure, so the tests then crashed with a nil pointer dereference in NewWorker() instead of failing with that error.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The ASAN "roce on worker 1" job failed because a single UCT gtest, Full analysisSummary: Root cause: Evidence from the log: the run completed normally (no timeout, no SIGTERM, no ASAN report in the tail, continuous output up to teardown, 2433112 ms total) with Implicated commit: unknown — not the PR commit [REDACTED:Hex High Entropy String] (Go test-only change); the failure lies in the master-side IB/mlx5 atomic-mkey code. File: test/gtest/uct/test_amo_fadd_fxor.cc:36-50 (assertions Suggested fix: Treat this as unrelated to PR #12273 and re-trigger the "roce on worker 1" ASAN job; do not block the Go EINTR fix on it. For the real defect: re-run the single test in a loop ( Related: PR #12273 (the PR under test, #12273), candidate fix PR #12265 (#12265)
|
Both worker tests created the context and the worker and guarded their cleanup the same way. Move this to prepareWakeupWorker(), which reuses prepareContext() and TestEntity.Close() from endpoint_test.go and registers the cleanup with t.Cleanup(). Also buffer the started channel of TestUcpWorkerEfd, so its goroutine does not block on the send after the test gave up waiting for it.
|
🤖 Starting review — findings will be posted here when done. |
Five places in the worker tests set the leak flag and then failed the
test. Do both in wakeupEntity.leakFatalf(), which marks itself as a
test helper, so the failure still points to the caller.
It takes ...interface{} rather than ...any: bindings/go/tests/go.mod
has no go directive, so the tests are built with the Go 1.16 language
version, which has no any.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
What?
Fix a hang of
TestUcpWorkerEfdin the Go bindings tests: retryepoll_wait()onEINTR, and fail the test from the test goroutine instead of callingt.Fatalf()in the helper goroutine. Also bound the waits inTestUcpWorkerEfdandTestUcpWorkerWakeup.Why?
TestUcpWorkerEfdcan fail and then hang the go CI job until the 10-minutego testtimeout, e.g. the "go gpu" job of build 143332 on #12241. Two bugs in the test cause it:epoll_wait()fails withEINTRwhen a signal arrives at the thread that waits in it, and it is never restarted, even withSA_RESTART(see signal(7)). A Go process gets signals all the time, e.g. the runtime preemption signal (SIGURG). The test treatedEINTRas an error (worker_test.go:61in the log below).t.Fatalf(), which exits only the calling goroutine, not the test. Nothing was sent oneventsChan, and the test goroutine waited for it without a timeout (worker_test.go:74) untilgo testkilled the binary."go gpu" job log, build 143332
How?
syscall.EpollWait()while it returnsEINTR.TestUcpWorkerEfd, and forWait()inTestUcpWorkerWakeup, for at most 10 seconds. Also check theNewUcpContext(),Signal()andWait()errors, which were ignored.ucp_worker_wait()already retriespoll()onEINTR, soTestUcpWorkerWakeupneeds no retry.