Skip to content

GO/TESTS: Retry epoll_wait on EINTR in worker tests - #12273

Merged
brminich merged 7 commits into
openucx:masterfrom
dmitrygx:topic/ucx/go-worker-efd-eintr
Nov 9, 2026
Merged

brminich merged 7 commits into
openucx:masterfrom
dmitrygx:topic/ucx/go-worker-efd-eintr

Conversation

@dmitrygx

@dmitrygx dmitrygx commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

What?

Fix a hang of TestUcpWorkerEfd in the Go bindings tests: retry epoll_wait() on EINTR, and fail the test from the test goroutine instead of calling t.Fatalf() in the helper goroutine. Also bound the waits in TestUcpWorkerEfd and TestUcpWorkerWakeup.

Why?

TestUcpWorkerEfd can fail and then hang the go CI job until the 10-minute go test timeout, e.g. the "go gpu" job of build 143332 on #12241. Two bugs in the test cause it:

  1. epoll_wait() fails with EINTR when a signal arrives at the thread that waits in it, and it is never restarted, even with SA_RESTART (see signal(7)). A Go process gets signals all the time, e.g. the runtime preemption signal (SIGURG). The test treated EINTR as an error (worker_test.go:61 in the log below).
  2. The helper goroutine then called t.Fatalf(), which exits only the calling goroutine, not the test. Nothing was sent on eventsChan, and the test goroutine waited for it without a timeout (worker_test.go:74) until go test killed the binary.
"go gpu" job log, build 143332
04:18:06.989 === RUN   TestUcpWorkerEfd
04:18:07.105     worker_test.go:61: Failed to call epoll_wait interrupted system call
04:28:02.236 panic: test timed out after 10m0s
...
04:28:02.237 goroutine 24 [chan receive, 9 minutes]:
04:28:02.238 github.com/openucx/ucx/bindings/go/tests.TestUcpWorkerEfd(0xc000086680)
04:28:02.238 	bindings/go/tests/worker_test.go:74 +0x2d4
04:28:02.238 testing.tRunner(0xc000086680, 0x58b490)
04:28:02.238 	$GOROOT/src/testing/testing.go:1439 +0x102
04:28:02.238 created by testing.(*T).Run
04:28:02.238 	$GOROOT/src/testing/testing.go:1486 +0x35f
04:28:02.310 exit status 2
04:28:02.310 FAIL	github.com/openucx/ucx/bindings/go/tests	600.544s

How?

  • Retry syscall.EpollWait() while it returns EINTR.
  • Send errors from the helper goroutine over a channel and fail the test from the test goroutine.
  • Wait for the epoll result in TestUcpWorkerEfd, and for Wait() in TestUcpWorkerWakeup, for at most 10 seconds. Also check the NewUcpContext(), Signal() and Wait() errors, which were ignored. ucp_worker_wait() already retries poll() on EINTR, so TestUcpWorkerWakeup needs no retry.
  • If the test gives up while the helper goroutine may still use the worker, leak the worker and the context instead of destroying them under it.

epoll_wait() is never restarted after a signal handler, so a signal
delivered to the thread that waits in it, such as the Go runtime
preemption signal, fails TestUcpWorkerEfd with EINTR. The helper
goroutine then called t.Fatalf(), which exits only that goroutine, so
nothing was sent on eventsChan and the test goroutine waited until the
10 minute go test timeout.

Retry epoll_wait() on EINTR, and send errors from the helper goroutine
back to the test goroutine instead of calling t.Fatalf() there. Wait for
the result in TestUcpWorkerEfd and TestUcpWorkerWakeup for at most 10
seconds, and check the ucpWorker.Wait() error, which was ignored.
@dmitrygx
dmitrygx requested a review from a team as a code owner November 7, 2026 11:45
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
Comment thread bindings/go/tests/worker_test.go Resolved
@svc-ucx

svc-ucx commented Nov 7, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 0) · commit 4d9f316c

TL;DR: The gtest stage failed with exactly one failing test, tcp/test_ucp_rma.put_nonblocking/1 (tcp,cuda_copy,rocm_copy/flush_ep), in UCP RMA/EP-flush code that the PR does not touch — PR #12273 only changes Go binding tests, so this is a pre-existing/flaky failure in the flush_ep RMA path, not a regression from this commit.

Full analysis

Summary: ./contrib/test_jenkins.sh → make test in build-test/test/gtest exited with code 2 because 1 of 8918 gtests failed: tcp/test_ucp_rma.put_nonblocking/1, GetParam() = tcp,cuda_copy,rocm_copy/flush_ep.

Root cause: Evidence from the log: the whole suite ran to completion (no hang — the largest inter-line gap in the fetched tail is ~1 s, and the run finished with a normal teardown and TOP-20 timing report), and only the flush_ep variant of test_ucp_rma.put_nonblocking failed; the flush_worker and user_memh variants of the same test passed. That variant is the only one that relies on ucp_ep_flush_nbx alone (test_ucp_rma::is_ep_flush() → flush_ep(sender()) in test_ucp_memheap::test_xfer) to make the remote memheap coherent before mem_buffer::compare(). The test also forces MAX_RMA_LANES=2, so a multi-lane TCP endpoint flush is exercised, where uct_ep_flush frequently returns UCS_ERR_NO_RESOURCE and the flush request is parked on one lane's pending queue (src/ucp/rma/flush.c:189-209), while the remaining lanes must still be flushed afterwards. A premature flush completion there lets the validation compare before all lanes drained. This code path is actively being fixed in the open PR #12266 ("UCP/FLUSH: Flush the other lanes after the pending lane").

Crucially, the commit under test ([REDACTED:Hex High Entropy String], "GO/TESTS: Retry epoll_wait on EINTR in worker tests", Dmitrii) touches only bindings/go test code — it cannot influence the C test_ucp_rma gtest. So this failure is not caused by the PR.

Note: the exact gtest assertion text for the failing case is earlier in the job log than the ~500-line tail the log API returns; only the final [ FAILED ] summary line was retrievable, so the specific failure message (most likely data validation failed from test_ucp_memheap.cc:115) could not be quoted verbatim.

Implicated commit: Not the PR commit. The failure lies in the EP-flush path last substantially changed by d513f98f (nbellalou, "UCP/FLUSH: Add topology-safe selective endpoint flush", #11923) and [REDACTED:Hex High Entropy String] (Evgeny Leksikov, #12091); the tested commit [REDACTED:Hex High Entropy String] (Dmitrii) is unrelated.

File: src/ucp/rma/flush.c:156-213 (multi-lane flush / UCS_ERR_NO_RESOURCE pending path), surfaced by test/gtest/ucp/test_ucp_memheap.cc:110-116 via test/gtest/ucp/test_ucp_rma.cc:283

Suggested fix:

  1. Do not block PR GO/TESTS: Retry epoll_wait on EINTR in worker tests #12273 on this — re-run the "Tests new on worker 0" job; the Go-only change cannot affect test_ucp_rma.
  2. For the real defect, track it against the EP-flush fix in PR UCP/FLUSH: Flush the other lanes after the pending lane #12266 (ensure all remaining lanes are flushed after the lane that went pending, and that ucp_flush_check_completion() cannot report completion while all_lanes & lane_mask & ~started_lanes != 0).
  3. To make this diagnosable next time, have the gtest stage upload the full gtest output as a build artifact (the tail-only log loses the assertion text), and optionally re-run just --gtest_filter=tcp/test_ucp_rma.put_nonblocking/1 with UCX_LOG_LEVEL=req to capture the flush trace.

Related: PR #12273 (the PR under test, Go tests only), PR #12266 (UCP/FLUSH: Flush the other lanes after the pending lane — likely fix), PR #11923 (topology-safe selective endpoint flush)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 00f8e58c-cd09-42ec-9b93-fc63d2255008 in the triage console for the audit trail.

If TestUcpWorkerWakeup times out, its goroutine is still in
ucpWorker.Wait(), and the deferred Close() destroyed the worker and the
context under it. The same happened in TestUcpWorkerEfd when the
goroutine did not start in 500 ms, while it may still call
ucpWorker.Arm() or Progress(). Leak the worker and the context in these
cases instead.

Also make eventsChan buffered, so the TestUcpWorkerEfd goroutine can exit
after the test stopped waiting for it.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
Comment thread bindings/go/tests/worker_test.go Resolved
Comment thread bindings/go/tests/worker_test.go Resolved
TestUcpWorkerEfd destroyed the worker when epoll_wait() timed out, while
its goroutine still waited on the worker efd. Leak the worker there as
well, so that a late event can still wake the goroutine, which then exits
through the buffered eventsChan.

Also wait wakeupTimeout instead of 500 ms for the goroutine to start, and
pass the leak flag to closeWorker() by value from a deferred closure.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
Comment thread bindings/go/tests/worker_test.go Resolved
@svc-ucx

svc-ucx commented Nov 7, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit d8bb377f

TL;DR: ib/test_md.alloc/0 (mlx5_0) was the single failing test out of 9030 — it is the IB device-memory (ibv_alloc_dm) allocation test, which fails when more than 50% of DM allocations return UCS_ERR_NO_MEMORY; this is a resource-contention flake unrelated to PR #12273 (Go worker eventfd/EINTR), so re-run the job and harden the test to skip instead of fail when device memory is exhausted.

Full analysis

Summary: Azure job "roce on worker 0" (build 143447) failed at make test in build-test/test/gtest: [ FAILED ] ib/test_md.alloc/0, where GetParam() = mlx5_0 — 1 failed / 9029 passed, make: *** [Makefile:4992: test] Error 1, Bash exited with code '2'.

Root cause: For the ib component, UCT_MD_FLAG_ALLOC is only set when KSM-over-device-memory works (uct_ib_mlx5dv_check_dm_ksm_reg, ib_mlx5dv_md.c:2275-2315), and uct_ib_mlx5_devx_md_query reports alloc_mem_types = UCS_MEMORY_TYPE_RDMA with max_alloc = dev_attr.max_dm_size (ib_mlx5dv_md.c:3183-3188). So test_md.alloc does 300 uct_mem_alloc() rounds of on-NIC device memory via ibv_alloc_dm(). Device memory is a tiny, device-global resource shared by every process on the host; when it is exhausted ibv_alloc_dm fails and uct_ib_mlx5_devx_device_mem_alloc returns UCS_ERR_NO_MEMORY (ib_mlx5dv_md.c:2156-2180). The test only tolerates 50% such failures: EXPECT_LT((double)num_alloc_failures / iterations, 0.5) << "Too many OUT_OF_RESOURCE failures" (test/gtest/uct/test_md.cc:559). The job name ("roce on worker 0") and the host (swx-rain04) show multiple gtest workers running concurrently on the same node, each hammering the same NIC's DM pool, which is exactly the condition that pushes this ratio over 0.5. Supporting evidence that this is an environment/resource flake rather than a code regression: the run shows generic resource pressure elsewhere too (SKIP (Can't fill UCT resources in the given time.), 54 (out of 200) eps were not disconnected), the log shows continuous activity with no multi-minute gaps (no hang), and PR #12273 / branch topic/ucx/go-worker-efd-eintr concerns the Go bindings worker eventfd EINTR handling — it does not touch src/uct/ib/** or the MD alloc path exercised by this test. Caveat: Azure only returns the last ~500 log lines, so the exact gtest assertion line for test_md.alloc was truncated; the diagnosis above is the only failure mode in that test reachable on an IB MD (all other paths are ASSERT_UCS_OK on uct_mem_alloc/uct_md_mkey_pack_v2, which would equally indicate DM exhaustion/KSM-on-DM failure).

Implicated commit: unknown — no recent commit on the implicated files (test/gtest/uct/test_md.cc last touched by 4cdc171, Tristan Rice, 2026-10-30; src/uct/ib/mlx5/dv/ib_mlx5dv_md.c / ib_md.c last by deaa328, Thomas Vegas, 2026-10-06) correlates with this PR, which is Go-bindings-only.

File: test/gtest/uct/test_md.cc:559 (threshold check), with the failing allocation at src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2174-2180 (ibv_alloc_dm)

Suggested fix:

  1. Immediate: re-run the "roce on worker 0" job — this is not a regression from PR GO/TESTS: Retry epoll_wait on EINTR in worker tests #12273; nothing in the Go eventfd/EINTR change can affect uct_ib_mlx5_devx_device_mem_alloc.
  2. Durable de-flake of the test, in test_md.cc:516-561:
    • Reset num_alloc_failures inside the ucs_for_each_bit(mem_type, ...) loop (it is currently declared outside and accumulates across memory types — a latent bug if a second alloc-capable mem type is ever added).
    • For UCS_MEMORY_TYPE_RDMA, treat sustained UCS_ERR_NO_MEMORY as a skip rather than a failure, e.g. replace the EXPECT_LT(...) with if ((double)num_alloc_failures / iterations >= 0.5) { UCS_TEST_SKIP_R("device memory is exhausted by concurrent users"); }, since on-NIC DM is a host-global resource the test cannot reserve.
    • Alternatively scale the request size down for UCS_MEMORY_TYPE_RDMA (current max_size is max(max_alloc/8, min(max_alloc,1024)) capped at 63356, i.e. up to 16 KB per iteration on a 128 KB DM) so a couple of parallel workers cannot starve each other.

Related: PR #12273 (the triggering PR, unrelated to the failure); no existing issue found for ib/test_md.alloc device-memory flakiness — worth opening one so future hits are not re-triaged.

TestUcpWorkerEfd and TestUcpWorkerWakeup ignored the ucpWorker.Signal()
error, so a failed signal showed up only as a 10 second wait timeout.
Fail the test with that error instead, and leak the worker there as
well, since the helper goroutine still waits on it.

Also let the callers decide whether to leak the worker, and make
closeWorker() always destroy the worker and the context.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
TestUcpWorkerEfd and TestUcpWorkerWakeup ignored the NewUcpContext()
error. NewUcpContext() returns a nil context on failure, so the tests
then crashed with a nil pointer dereference in NewWorker() instead of
failing with that error.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
Comment thread bindings/go/tests/worker_test.go Resolved
@svc-ucx

svc-ucx commented Nov 8, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 1) · commit c3ec0a67

TL;DR: The ASAN "roce on worker 1" job failed because a single UCT gtest, dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1, failed its atomic fetch-add validation (8959/8960 passed); this is unrelated to PR #12273, which only changes Go worker test code, so it is a pre-existing/intermittent failure in the IB atomic-mkey path for RDMA-memory-type buffers.

Full analysis

Summary: make test exited non-zero (Makefile:4992: test Error 1) after exactly one gtest failure: dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1, where GetParam() = dc_mlx5/mlx5_1:1.

Root cause: Evidence from the log: the run completed normally (no timeout, no SIGTERM, no ASAN report in the tail, continuous output up to teardown, 2433112 ms total) with [ FAILED ] 1 test — dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1. That test is a derivative of the fadd64 concurrent fetch-and-add test (test/gtest/uct/test_amo_fadd_fxor.cc:48 → test_fop<uint64_t, UCT_ATOMIC_OP_ADD>), which runs 4 sender threads × 1000 uct_ep_atomic64_fetch ops against one buffer and then checks (a) every reply appears exactly once (uct_amo_test::validate_replies) and (b) the final remote value equals the expected sum. The failure is therefore a wrong/duplicated atomic reply or wrong final value on a buffer registered as UCS_MEMORY_TYPE_RDMA over dc_mlx5 — i.e. in the IB/mlx5 atomic memory-key (indirect atomic MR / atomic rkey offset) path, not in anything touched by this PR. PR #12273 (GO/TESTS: Retry epoll_wait on EINTR in worker tests, commit c3ec0a6) modifies only Go binding test code and is not even linked into the gtest binary, so it cannot produce this failure. Note the gtest assertion text itself is earlier in the log than the retrievable tail, so the exact failing expectation (reply histogram vs. final value) could not be read.

Implicated commit: unknown — not the PR commit [REDACTED:Hex High Entropy String] (Go test-only change); the failure lies in the master-side IB/mlx5 atomic-mkey code.

File: test/gtest/uct/test_amo_fadd_fxor.cc:36-50 (assertions validate_replies / EXPECT_EQ(value, *(T*)recvbuf.ptr())); implementation side: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c (atomic mkey / atomic rkey offset for UCS_MEMORY_TYPE_RDMA registrations)

Suggested fix: Treat this as unrelated to PR #12273 and re-trigger the "roce on worker 1" ASAN job; do not block the Go EINTR fix on it. For the real defect: re-run the single test in a loop (--gtest_filter=dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1 --gtest_repeat=50 with UCX_LOG_LEVEL=debug) on an mlx5_1 RoCE node to confirm reproducibility, and check whether open PR #12265 ("UCT/IB/MLX5: Fix atomic MR offset without a flush memory key") fixes it — that change covers exactly the atomic-MR-offset path this test exercises. Separately, make CI archive the full gtest log (or add --gtest_output=xml) so the failing assertion text survives log truncation; right now only the summary line is retrievable.

Related: PR #12273 (the PR under test, #12273), candidate fix PR #12265 (#12265)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f135c052-5c8a-4341-bf39-2f821c445363 in the triage console for the audit trail.

Both worker tests created the context and the worker and guarded their
cleanup the same way. Move this to prepareWakeupWorker(), which reuses
prepareContext() and TestEntity.Close() from endpoint_test.go and
registers the cleanup with t.Cleanup().

Also buffer the started channel of TestUcpWorkerEfd, so its goroutine
does not block on the send after the test gave up waiting for it.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread bindings/go/tests/worker_test.go Resolved
Five places in the worker tests set the leak flag and then failed the
test. Do both in wakeupEntity.leakFatalf(), which marks itself as a
test helper, so the failure still points to the caller.

It takes ...interface{} rather than ...any: bindings/go/tests/go.mod
has no go directive, so the tests are built with the Go 1.16 language
version, which has no any.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@brminich
brminich merged commit 91c4ffc into openucx:master Nov 9, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants