Repository navigation
GTEST: Restore and check the current CUDA device after each test - #12073
Conversation
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Coverage gap worth noting: this change is only verifiable on a multi-GPU runner (both fixtures skip with fewer than 2 CUDA devices), so CI may not exercise the restore path at all. |
|
🤖 CI Triage Agent — TL;DR: The gtest binary hung in Full analysisSummary: Root cause: Timeline from the log shows a clear hang, not slowness:
The ~12-minute gap between the request-timeout error and the watchdog abort is the smoking gun. The backtrace shows the thread was stuck in The underlying transport stall is in the RC verbs ( Implicated commit: Not caused by the PR commit File: Suggested fix:
ucs_time_t deadline = ucs::get_deadline();
while (!m_close_ep_reqs.empty() && (ucs_get_time() < deadline)) {
void *req = m_close_ep_reqs.front();
while (!is_request_completed(req) && (ucs_get_time() < deadline)) {
test.progress(worker_idx);
}
if (!is_request_completed(req)) {
ADD_FAILURE() << "EP close request " << req << " did not complete";
break; // let the existing EXPECT_TRUE report leftover EPs
}
close_ep_req_free(req);
}Note Related: PR #12073 (the PR under test — unrelated to this failure); PR #11985 / commit |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
…nucx#12073) * GTEST/UCT/CUDA: Restore the current device after each test Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * GTEST: Restore and check the current CUDA device after each test Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> * GTEST: Code cleanup Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> --------- Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
#12189) GTEST: Restore and check the current CUDA device after each test (#12073) * GTEST/UCT/CUDA: Restore the current device after each test * GTEST: Restore and check the current CUDA device after each test * GTEST: Code cleanup --------- Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com> Co-authored-by: Guy Ealey Morag <gealeymorag@nvidia.com>
What?
Restore the current CUDA device at every device switch in the gtests, and fail any test that leaves a non-default device current.
Why?
detect_mem_type_cuda_fabricswitches to another GPU before allocating. Where fabric memory is unsupported, as on the CI GPU agents, the allocation skips the test before it switches back, so later tests run on the wrong GPU. With shuffled test order, this makestest_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenancefail intermittently, as seen on many PRs before: #11858, #11319, #11990, #11676, #11866 and #12041.How?
scoped_cuda_deviceguard saves the current device and restores it when the scope ends, including on a failed assertion or a skip. It replaces the manual save/restore pairs intest_switch_cuda_device.ccandtest_ucp_mmap.cc.cleanup(),TearDownProxy()restores the default device if a test left another device current, and fails that test. A future unguarded switch then fails in the test that caused it, instead of breaking a random later one.