Skip to content

UCT/GTEST: Hardening of UD stale private EP tests - v1.23 - #11948

Merged
iyastreb merged 1 commit into
openucx:v1.23.xfrom
iyastreb:iyastreb/ud-test-hardening-1.23
Sep 15, 2026
Merged

iyastreb merged 1 commit into
openucx:v1.23.xfrom
iyastreb:iyastreb/ud-test-hardening-1.23

Conversation

@iyastreb

Copy link
Copy Markdown
Contributor

What?

Backport #11920 to 1.23

Raise UD_TIMEOUT from 1s to 3s in stale_private_ep_reuse and private_ep_keepalive.

Why?

CI hit ud_ep.c: Fatal: UD endpoint ... unhandled timeout error in stale_private_ep_reuse on the ASAN stage. The failing job stalled for ~2s (other UD tests in it ran 2s slower than in every other ASAN job). With a 1s UD timeout, a stall while a CREQ is in flight makes the UD timer declare a peer timeout on a user endpoint before the already arrived CREP is processed. The UD gtest fixture has no error handler, so that is fatal and kills the whole shard. 3s matches test_ud_peer_failure and survives such stalls; the keepalive idle window grows accordingly.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/uct/ib/test_ud.cc Resolved
@svc-ucx

svc-ucx commented Sep 14, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit 027a92f7

TL;DR: The AddressSanitizer/roce worker-3 job did not fail on a test — the Azure agent on swx-rain03 was killed ~20 s into the gtest run ("The agent has received a shutdown signal... The operation was canceled"), so this is an infrastructure/agent-loss failure unrelated to PR #11948; just re-run the job.

Full analysis

Summary: Job "AddressSanitizer roce on worker 3" (build 135997) was cancelled mid-gtest because the Azure Pipelines agent received a shutdown signal, not because of a test assertion or sanitizer report.

Root cause: Evidence from the log:

  • 14:43:39–14:43:40 — memtrack sanity test and test_memtrack_limit both pass as expected (SUCCESS, then the intentional Fatal: reached memtrack memory limit 412M which is the expected negative case).
  • 14:43:40 — timeout 200m make -C test/gtest test starts shard 4 of 4 (8627 tests), builds are all "Nothing to be done".
  • 14:43:47–14:43:50 — tests run normally and quickly (test_socket, tcp/test_ucp_proto, ud_mlx5/uct_p2p_rma_test_xfer, rc_verbs/uct_amo_swap_test, shm_gga/test_ucp_fault_tolerance — all OK/SKIP).
  • 14:43:50.56 — [ RUN ] rcx/test_ucp_tag_nbx.rndv_bcopy/5 <rc_x/prereg/iov>
  • 14:43:54.64 — ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled. followed by The Operation will be canceled and The operation was canceled.

There is no gap in the timestamps longer than ~4 seconds anywhere in the log, so nothing hung and nothing timed out (the 200-minute timeout was nowhere near expiring — total elapsed gtest time was ~14 s). No AddressSanitizer report, no assertion failure, no non-zero test exit is present. The build was terminated externally: the self-hosted agent process/host (swx-rain03) went away — agent service stopped, host rebooted/drained, or the agent was OOM-killed at the machine level. The last in-flight test, rcx/test_ucp_tag_nbx.rndv_bcopy/5, is a UCP tag rendezvous test with no relation to the UD stale-private-EP gtest hardening in PR #11948, and it had only been running 4 s (these tests normally complete in a few hundred ms, but 4 s is not long enough to conclude a hang before an external kill).

Implicated commit: none — commit [REDACTED:Hex High Entropy String] (PR #11948, "UCT/GTEST: Hardening of UD stale private EP tests - v1.23") is not implicated; the cancellation is environmental.

File: n/a (no source file implicated; failure occurred at the agent level, last log lines at 14:43:54 of the "Run ./contrib/test_jenkins.sh" step)

Suggested fix: Re-trigger the "AddressSanitizer roce on worker 3" job — no code change is warranted. In parallel, have CI admins check the swx-rain03 agent host for the cause of the shutdown (agent service restart/drain, host reboot, or kernel OOM-killer entries in dmesg/journalctl around 2026-09-14 14:43:54 UTC); ASAN jobs have a large memory footprint, so if the host log shows an OOM kill, reduce concurrent ASAN workers on that node or raise its memory/swap. Do not raise the step timeout — the log shows continuous activity and the 200 m timeout never fired. If this recurs specifically on worker 3 across unrelated PRs, mark the node offline pending investigation.

Related: PR #11948 (#11948) — the PR under test, not the cause. No existing issue found tracking this agent-shutdown signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a0600a1c-28ea-48d1-8181-380159ab33aa in the triage console for the audit trail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants