Skip to content

GTEST: Restore and check the current CUDA device after each test - #12073

Merged
brminich merged 4 commits into
openucx:masterfrom
guy-ealey-morag:gtest-cuda-device-check
Oct 19, 2026
Merged

brminich merged 4 commits into
openucx:masterfrom
guy-ealey-morag:gtest-cuda-device-check

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What?

Restore the current CUDA device at every device switch in the gtests, and fail any test that leaves a non-default device current.

Why?

detect_mem_type_cuda_fabric switches to another GPU before allocating. Where fabric memory is unsupported, as on the CI GPU agents, the allocation skips the test before it switches back, so later tests run on the wrong GPU. With shuffled test order, this makes test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance fail intermittently, as seen on many PRs before: #11858, #11319, #11990, #11676, #11866 and #12041.

How?

  • A new scoped_cuda_device guard saves the current device and restores it when the scope ends, including on a failed assertion or a skip. It replaces the manual save/restore pairs in test_switch_cuda_device.cc and test_ucp_mmap.cc.
  • After cleanup(), TearDownProxy() restores the default device if a test left another device current, and fails that test. A future unguarded switch then fails in the test that caused it, instead of breaking a random later one.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage gap worth noting: this change is only verifiable on a multi-GPU runner (both fixtures skip with fewer than 2 CUDA devices), so CI may not exercise the restore path at all.

@svc-ucx

svc-ucx commented Oct 12, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 3eb900c1

TL;DR: The gtest binary hung in rc/test_ucp_tag_limits.check_max_short_rndv_thresh_zero/0 <rc_v> — a UCP request never completed, then EP close spun forever in close_all_eps, so the 900 s watchdog fired SIGABRT and aborted the whole test run. This is an RC/RoCE transport hang unrelated to the PR's CUDA device-restore change, most likely an infra/transport flake, but the close_all_eps inner loop is a real unbounded-wait bug worth fixing.

Full analysis

Summary: make test aborted (core dumped) after rc/test_ucp_tag_limits.check_max_short_rndv_thresh_zero/0 <rc_v> stalled for ~16 minutes and tripped the gtest watchdog ("Connection timed out - abort testing", signal 6).

Root cause: Timeline from the log shows a clear hang, not slowness:

  • 10:07:04.089 — last normal test completes (dc_mlx5/test_uct_ib_pkey).
  • 10:07:04.089 — [ RUN ] rc/test_ucp_tag_limits.check_max_short_rndv_thresh_zero/0 <rc_v> starts. No output at all after this.
  • 10:10:04.665 — 3-minute test_timeout_in_sec deadline expires: ucp_test.cc:300 UCX ERROR request 0x55abf3c3cfd8 did not complete on time.
  • 10:22:04.226 — 900 s watchdog fires: test_helpers.cc:57: Failure / Connection timed out - abort testing, Caught signal 6.

The ~12-minute gap between the request-timeout error and the watchdog abort is the smoking gun. The backtrace shows the thread was stuck in ucp_test_base::entity::close_all_eps() → ucp_test::progress() → sched_yield(). In test/gtest/ucp/ucp_test.cc:924-931, the outer while loop checks ucs_get_deadline(), but the inner while (!is_request_completed(req)) { test.progress(worker_idx); } has no deadline check at all. Once the rc_v endpoint failed to drain (the RMA/rndv request never completed over RoCE), the EP-close request stayed UCS_INPROGRESS and the inner loop spun indefinitely until the watchdog killed the process — taking down the entire remaining test suite rather than failing just this one test.

The underlying transport stall is in the RC verbs (rc_v) path on RoCE; the PR under test (#12073, "GTEST/UCT/CUDA: Restore the current device after each test") touches only CUDA device state in UCT gtests and has no bearing on test_ucp_tag_limits / rc_v. Recent commits to ucp_test.cc (most recent: abf206ae, Guy Ealey Morag, "UCP/PROTO: Apply GPU-NIC assignment to RMA and rendezvous lanes (#11985)") did change RMA/rendezvous lane selection, which is the subsystem exercised by the stuck request — worth a look if this recurs.

Implicated commit: Not caused by the PR commit 3eb900c1. Closest related change to the hung code path: abf206ae (Guy Ealey Morag) — "UCP/PROTO: Apply GPU-NIC assignment to RMA and rendezvous lanes (#11985)". The unbounded-loop bug itself is pre-existing.

File: test/gtest/ucp/ucp_test.cc:924-931 (close_all_eps, inner loop without deadline); failing test is rc/test_ucp_tag_limits.check_max_short_rndv_thresh_zero/0 <rc_v>

Suggested fix:

  1. Re-run the job to confirm the flake — this failure is independent of PR GTEST: Restore and check the current CUDA device after each test #12073 and should not block it.
  2. Bound the inner wait in close_all_eps so a stuck EP fails one test instead of aborting the whole binary:
ucs_time_t deadline = ucs::get_deadline();
while (!m_close_ep_reqs.empty() && (ucs_get_time() < deadline)) {
    void *req = m_close_ep_reqs.front();
    while (!is_request_completed(req) && (ucs_get_time() < deadline)) {
        test.progress(worker_idx);
    }
    if (!is_request_completed(req)) {
        ADD_FAILURE() << "EP close request " << req << " did not complete";
        break;   // let the existing EXPECT_TRUE report leftover EPs
    }
    close_ep_req_free(req);
}

Note close_ep_req_free currently ASSERT_NE(UCS_INPROGRESS, status), so it must not be called on a timed-out request — hence the break.
3. For the transport stall itself, re-run with UCX_LOG_LEVEL=debug / UCX_PROTO_INFO=y on the roce worker and check whether RMA/rendezvous lane assignment from #11985 picks an unusable lane for rc_v with RNDV_THRESH=0.

Related: PR #12073 (the PR under test — unrelated to this failure); PR #11985 / commit abf206ae (RMA & rendezvous lane assignment, same code path as the stuck request). No existing issue found for check_max_short_rndv_thresh_zero hangs — consider opening one to track this flake.

@guy-ealey-morag
guy-ealey-morag marked this pull request as draft October 14, 2026 09:13
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@guy-ealey-morag guy-ealey-morag changed the title GTEST/UCT/CUDA: Restore the current device after each test GTEST: Restore and check the current CUDA device after each test Oct 15, 2026
@guy-ealey-morag
guy-ealey-morag marked this pull request as ready for review October 15, 2026 17:16
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@openucx openucx deleted a comment from svc-nvidia-pr-review Oct 16, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@brminich
brminich merged commit f931b38 into openucx:master Oct 19, 2026
165 checks passed
amastbaum pushed a commit to amastbaum/ucx that referenced this pull request Oct 26, 2026
…nucx#12073)

* GTEST/UCT/CUDA: Restore the current device after each test

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>

* GTEST: Restore and check the current CUDA device after each test

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>

* GTEST: Code cleanup

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>

---------

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
ofirfarjun7 pushed a commit that referenced this pull request Oct 29, 2026
#12189)

GTEST: Restore and check the current CUDA device after each test (#12073)

* GTEST/UCT/CUDA: Restore the current device after each test



* GTEST: Restore and check the current CUDA device after each test



* GTEST: Code cleanup



---------

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Co-authored-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants