Repository navigation
TOOLS/PERF: Separate device mode from memory placement - #12003
Conversation
Move optional device IDs to the -m allocator syntax and make -a only enable device API mode. Preserve group-index placement by default and honor explicit IDs for CUDA, ROCm, and ZE.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The gtest job failed on a single test, Full analysisSummary: Root cause: From the log, the test suite completed normally (no SIGTERM, no timeout, no hang — the failing test ran to completion in 44478 ms and the log shows continuous per-test output with no gaps > ~47 s of legitimate stress-test work), and exactly one assertion-level test failure caused the non-zero make exit. The failing test repeatedly connects a sender/receiver pair with Implicated commit: Not this PR's commit. File: Suggested fix: Re-run the failed Azure job to confirm flakiness — do not change the PR. To capture the actual assertion, reproduce locally/on the worker with just that case: Related: #7886 (same test family reported as intermittently failing), #11843, #11563
|
| } | ||
|
|
||
| if ((device_id < 0) || ((unsigned)device_id >= device_count)) { | ||
| ucs_error("device index %d is invalid for %u available devices", |
There was a problem hiding this comment.
minor: can the error messages name the allocator/device type? no accelerator devices available / device index %d is invalid for %u available devices don't say which backend failed, which makes failures harder to identify when send and receive use different accelerators. The previous CUDA message included it.
|
Please qualify " |
| case 'a': | ||
| return parse_accel_device_params(opt_arg, ¶ms->super.send_device, | ||
| ¶ms->super.recv_device); | ||
| params->super.flags |= UCX_PERF_TEST_FLAG_DEVICE; |
There was a problem hiding this comment.
Since -a no longer takes an argument, the old -a cuda:0 host syntax now makes cuda:0 the server hostname and silently ignores host, so the user gets a name-resolution error instead of an argument error. Can we reject extra positional arguments in parse_opts()? That covers the client form; an old-syntax server command (-a cuda:0 with no host) would still turn into a client connecting to cuda:0, so a note about the syntax change may be worth adding too.
There was a problem hiding this comment.
added one check to narrow it.
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
Coverage gap: the new |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: the |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Gtest crashed with SIGSEGV (address nil) during Root cause: Race in the new perftest allocator registry. ucx_perf_allocators[ucx_perf_num_allocators++] = allocator; /* libperf_int.h:132 */The counter can be incremented/visible before the slot pointer is stored, so the second thread iterating the registry reads a NULL if (!strcmp(name, ucx_perf_allocators[i]->name)) /* libperf_int.h:121 <-- crash */This matches the backtrace exactly: Implicated commit: c4fb300 "TOOLS/PERF: Separate device mode from memory placement" — Thomas Vegas (the head commit of this PR branch; it introduced the dynamic File: Suggested fix: Make global init idempotent and thread-safe, e.g. in #include <ucs/type/init_once.h>
void ucx_perf_global_init()
{
static ucs_init_once_t init_once = UCS_INIT_ONCE_INITIALIZER;
static ucx_perf_allocator_t host_allocator = {...};
static ucx_perf_allocator_t rdma_allocator = {...};
UCS_INIT_ONCE(&init_once) {
UCS_MODULE_FRAMEWORK_DECLARE(ucx_perftest);
ucx_perf_allocator_register(&host_allocator);
ucx_perf_allocator_register(&rdma_allocator);
UCS_MODULE_FRAMEWORK_LOAD(ucx_perftest, UCS_MODULE_LOAD_FLAG_GLOBAL);
}
}Additionally harden the registry itself since module allocators (cuda/rocm/ze) also register at load time: protect Related: PR #12003 (this change) — #12003; same-area changes in #11993 and #11880 add more allocators through the same registry and will be exposed to the identical race. |
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: the rocm and ze positive |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The ASan job aborted on a heap-use-after-free in Full analysisSummary: Root cause: The freed 144-byte block is the IB MD stats node allocated in The ordering is wrong because this test instance was skipped from inside PR #12003 ("TOOLS/PERF: Separate device mode from memory placement") touches only Implicated commit: File: Suggested fix: Destroy the UCT entities (and hence close the MDs and stop their async threads) before tearing down statistics. Concretely, stop calling // test/gtest/uct/test_stats.cc
~test_uct_stats() {
m_entities.clear(); /* entities may survive if init() threw a skip */
stats_restore();
}
virtual void cleanup() {
uct_p2p_test::cleanup(); /* destroys eps + entities -> closes MD, stops async thread */
stats_restore(); /* now safe: nothing references md->stats */
}A more general hardening (worth a separate PR) is to make Related: #11201 (added
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — Summary: Unable to fetch logs — cannot diagnose without them.
|
|
🤖 Starting review — findings will be posted here when done. |
tomerg-nvidia
left a comment
There was a problem hiding this comment.
Update copyright years
| * do not work. Need to use bistro for memtype hooks to fix it. | ||
| */ | ||
| UCS_MODULE_FRAMEWORK_LOAD(ucx_perftest, UCS_MODULE_LOAD_FLAG_GLOBAL); | ||
| UCS_INIT_ONCE(&init_once) { |
There was a problem hiding this comment.
What the init_once change? How is it related to the PR?
There was a problem hiding this comment.
there was a race found by CI where multiple init were run in parallel leading to crash/sanitizer issue
| ${ucx_inst}/bin/ucx_info -d) | ||
| check_ze_devices "${null_ze_info}" "null-driver" | ||
| run_perftest_device_id_smoke ze-device \ | ||
| ZE_ENABLE_ALT_DRIVERS=/usr/local/lib/libze_null.so.1 UCX_TLS=self,ze_copy |
There was a problem hiding this comment.
Why set ZE_ENABLE_ALT_DRIVERS? Where did this path come from?
There was a problem hiding this comment.
without it, the run fails as it uses the real/non-null ze driver that fails to initialized properly and ends-up crashing.
There was a problem hiding this comment.
path from CI image: rdmz-harbor.rdmz.labs.mlnx/ucx/x86_64/ubuntu24.04/builder:ze-oneapi-2026.0.0-l0-1.27.0
|
🤖 Starting review — findings will be posted here when done. |
Why?
Device API mode and accelerator memory placement are independent settings and should not share one option.
Needed for future cuda-localized memory placement and #11993.
What?
-m <allocator>[:device-id]selects allocation and placement. For UCP with ZE/ROCm, the device ID selects the device context for operations but does not control placement of buffers allocated byUCP_MEM_MAP_ALLOCATE.-a/--deviceenables device API mode without an argument.How?
UCX_PERF_TEST_FLAG_DEVICE.group_index % device_countplacement.