Skip to content

TOOLS/PERF: Separate device mode from memory placement - #12003

Merged
tvegas1 merged 11 commits into
openucx:masterfrom
tvegas1:perftest_memory_placement
Oct 5, 2026
Merged

tvegas1 merged 11 commits into
openucx:masterfrom
tvegas1:perftest_memory_placement

Conversation

@tvegas1

@tvegas1 tvegas1 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Why?

Device API mode and accelerator memory placement are independent settings and should not share one option.

Needed for future cuda-localized memory placement and #11993.

What?

  • -m <allocator>[:device-id] selects allocation and placement. For UCP with ZE/ROCm, the device ID selects the device context for operations but does not control placement of buffers allocated by UCP_MEM_MAP_ALLOCATE.
  • -a/--device enables device API mode without an argument.
  • Explicit device IDs are supported for CUDA, ROCm, and ZE.

How?

  • Gate UCP/UCT device behavior with UCX_PERF_TEST_FLAG_DEVICE.
  • Preserve default group_index % device_count placement.
  • Use receive placement for group 0 and send placement for other groups.
  • Preserve existing allocation behavior and CUDA batch configuration.

Move optional device IDs to the -m allocator syntax and make -a only enable device API mode. Preserve group-index placement by default and honor explicit IDs for CUDA, ROCm, and ZE.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread src/tools/perf/lib/libperf_int.h Resolved
@svc-ucx

svc-ucx commented Sep 30, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 3) · commit c4fb3006

TL;DR: The gtest job failed on a single test, tcp/test_ucp_wireup_errh_peer.stress_connect_force_disconnect/2 (tcp/rma,no_ep_match) — a connect/force-disconnect stress loop in the UCP wireup + peer-error-handling path that is unrelated to this PR's changes (which touch only src/tools/perf); treat it as a flaky/pre-existing failure, rerun, and if it reproduces bisect against the recent wireup lane-exchange commits.

Full analysis

Summary: make test in build-test/test/gtest exited with error 1: 8772/8773 tests passed, with the only failure being tcp/test_ucp_wireup_errh_peer.stress_connect_force_disconnect/2, where GetParam() = tcp/rma,no_ep_match.

Root cause: From the log, the test suite completed normally (no SIGTERM, no timeout, no hang — the failing test ran to completion in 44478 ms and the log shows continuous per-test output with no gaps > ~47 s of legitimate stress-test work), and exactly one assertion-level test failure caused the non-zero make exit. The failing test repeatedly connects a sender/receiver pair with UCP_ERR_HANDLING_MODE_PEER (err_mode = UCP_ERR_HANDLING_MODE_PEER, err_cb installed), does a send/recv, flushes both workers and then force-closes the endpoints, up to ~1000 iterations (test/gtest/ucp/test_ucp_wireup.cc:984-1007), with NO_EP_MATCH forcing max_conn_sn = 0 so connection matching is disabled (test_ucp_wireup.cc:178-181). This is a stress path over TCP that is entirely independent of the PR under test: commit c4fb3006 ("TOOLS/PERF: Separate device mode from memory placement", Thomas Vegas) only touches src/tools/perf, which is not exercised by test_ucp_wireup_errh_peer. Note: the assertion text for the failing test is in an earlier portion of the Azure log that is not retained in the fetchable tail, so the precise failed expectation could not be read — the evidence supports "unrelated single-test failure in the wireup/errh stress path", not a defect introduced by this commit.

Implicated commit: Not this PR's commit. [REDACTED:Hex High Entropy String] (Thomas Vegas, src/tools/perf only) cannot affect this test. Nearest plausible suspects if it reproduces: fb88400f (Evgeny Leksikov, "UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange", #11843) and db208ee4 (Evgeny Leksikov, "UCP/FT: probe-gated lane recovery via aux uct_ep_check", #11563), both of which changed the wireup/error-handling lane exchange.

File: test/gtest/ucp/test_ucp_wireup.cc:984 (test body), class defined at test/gtest/ucp/test_ucp_wireup.cc:943-962

Suggested fix: Re-run the failed Azure job to confirm flakiness — do not change the PR. To capture the actual assertion, reproduce locally/on the worker with just that case:
GTEST_FILTER='tcp/test_ucp_wireup_errh_peer.stress_connect_force_disconnect/2' UCX_LOG_LEVEL=debug GTEST_REPEAT=20 ./gtest and grab the full stdout (the Azure tail truncates it). If it reproduces consistently, bisect the UCP wireup lane-address-exchange changes (fb88400f, db208ee4) rather than the perftest commit; if it is intermittent, open a flaky-test tracking issue referencing the historical report below.

Related: #7886 (same test family reported as intermittently failing), #11843, #11563

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e765b46d-c3bc-469a-a318-82fe4c7a2e43 in the triage console for the audit trail.

Comment thread src/tools/perf/lib/libperf_int.h Outdated
}

if ((device_id < 0) || ((unsigned)device_id >= device_count)) {
ucs_error("device index %d is invalid for %u available devices",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can the error messages name the allocator/device type? no accelerator devices available / device index %d is invalid for %u available devices don't say which backend failed, which makes failures harder to identify when send and receive use different accelerators. The previous CUDA message included it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@yafshar

yafshar commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Please qualify "-m <allocator>[:device-id] selects allocation and placement" in the description: for UCP with ZE/ROCm, the buffers are allocated by UCP_MEM_MAP_ALLOCATE (those allocators define no .mem_alloc), so the specified id does not control buffer placement there.

case 'a':
return parse_accel_device_params(opt_arg, &params->super.send_device,
&params->super.recv_device);
params->super.flags |= UCX_PERF_TEST_FLAG_DEVICE;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since -a no longer takes an argument, the old -a cuda:0 host syntax now makes cuda:0 the server hostname and silently ignores host, so the user gets a name-resolution error instead of an argument error. Can we reject extra positional arguments in parse_opts()? That covers the client form; an old-syntax server command (-a cuda:0 with no host) would still turn into a client connecting to cuda:0, so a note about the syntax change may be worth adding too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added one check to narrow it.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread src/tools/perf/perftest_params.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/tools/perf/lib/libperf_int.h Resolved
Comment thread contrib/test_jenkins.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread contrib/test_jenkins.sh Resolved
Comment thread src/tools/perf/perftest_run.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Coverage gap: the new -m <mem>:<dev-id> parsing and the adjust_test_params rejection of -a for memory types without a device dispatcher have no unit test; the only validation is the GPU-gated run_ucx_perftest_cuda_device / run_ucx_perftest jobs. The negative paths (-m host:0, -a -m host, out-of-range id) are not exercised anywhere and would be cheap to cover in contrib/test_jenkins.sh since they do not need a GPU.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread contrib/test_jenkins.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread contrib/test_jenkins.sh Resolved
Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread src/tools/perf/lib/libperf_int.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the -m <mem>:<dev-id> positive path on rocm and ze is untested — ucx_perf_rocm_init/ucx_perf_ze_init previously ignored the device id entirely, and the new CI checks only cover cuda plus host-only negative paths.

@svc-ucx

svc-ucx commented Oct 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit 41e3ca02

TL;DR: shm_ib/test_ucp_perf.envelope/18 segfaulted in ucx_perf_allocator_register() (strcmp against a NULL entry) because PR #12003's new dynamic allocator registry is called concurrently and unguarded from the two gtest perf threads; wrap ucx_perf_global_init() in UCS_INIT_ONCE (or a mutex) so registration happens exactly once.

Full analysis

Summary: Gtest crashed with SIGSEGV (address nil) during shm_ib/test_ucp_perf.envelope/18, aborting make test with "Segmentation fault" and failing the Azure job.

Root cause: Race in the new perftest allocator registry. test_perf::run_multi_threaded() spawns two threads that each call ucx_perf_run() (test_perf.cc:288 → test_perf.cc:194), and ucx_perf_run() unconditionally calls ucx_perf_global_init() on every invocation (libperf.c:2239). ucx_perf_global_init() registers host_allocator and rdma_allocator and loads the perftest module framework with no once-guard and no lock (libperf_memory.c:415-422). The registration itself is non-atomic:

ucx_perf_allocators[ucx_perf_num_allocators++] = allocator;   /* libperf_int.h:132 */

The counter can be incremented/visible before the slot pointer is stored, so the second thread iterating the registry reads a NULL ucx_perf_allocators[i] and dereferences ->name:

if (!strcmp(name, ucx_perf_allocators[i]->name))   /* libperf_int.h:121  <-- crash */

This matches the backtrace exactly: ucx_perf_allocator_register() libperf_int.h:121 ← ucx_perf_global_init() libperf_memory.c:415 ← ucx_perf_run() libperf.c:2239 ← test_perf::test_func() test_perf.cc:194, in thread p-0, with "address not mapped to object at address (nil)". It is inherently flaky — it only fires when the two perf threads interleave inside the registration window, which is why only one perf test instance crashed.

Implicated commit: c4fb300 "TOOLS/PERF: Separate device mode from memory placement" — Thomas Vegas (the head commit of this PR branch; it introduced the dynamic ucx_perf_allocator_register()/ucx_perf_num_allocators registry replacing the previously static, compile-time allocator table). Related follow-up touching the same header: bbd7c85.

File: src/tools/perf/lib/libperf_int.h:121 (crash site; unsafe publish at :132), triggered from src/tools/perf/lib/libperf_memory.c:415 via src/tools/perf/lib/libperf.c:2239

Suggested fix: Make global init idempotent and thread-safe, e.g. in ucx_perf_global_init():

#include <ucs/type/init_once.h>
void ucx_perf_global_init()
{
    static ucs_init_once_t init_once = UCS_INIT_ONCE_INITIALIZER;
    static ucx_perf_allocator_t host_allocator = {...};
    static ucx_perf_allocator_t rdma_allocator = {...};

    UCS_INIT_ONCE(&init_once) {
        UCS_MODULE_FRAMEWORK_DECLARE(ucx_perftest);
        ucx_perf_allocator_register(&host_allocator);
        ucx_perf_allocator_register(&rdma_allocator);
        UCS_MODULE_FRAMEWORK_LOAD(ucx_perftest, UCS_MODULE_LOAD_FLAG_GLOBAL);
    }
}

Additionally harden the registry itself since module allocators (cuda/rocm/ze) also register at load time: protect ucx_perf_allocator_register()/_unregister() with a static mutex, and publish the slot before bumping the counter (ucx_perf_allocators[n] = allocator; ucs_memory_cpu_store_fence(); ucx_perf_num_allocators = n + 1;) so a concurrent reader can never observe a NULL entry. Optionally also skip NULL entries in the compare loop as a belt-and-braces guard, and consider hoisting the init out of the per-run ucx_perf_run() path.

Related: PR #12003 (this change) — #12003; same-area changes in #11993 and #11880 add more allocators through the same registry and will be exposed to the identical race.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/tools/perf/perftest_params.c Resolved
Comment thread buildlib/tools/builds.sh Resolved
Comment thread buildlib/tools/builds.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the rocm and ze positive -m <mem>:<dev-id> paths are still only smoke-tested on hardware-gated jobs, and the -m cuda:0,host -l style mixed-memory loopback combination is not exercised anywhere — which is why the loopback device-id finding would not be caught by CI.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer new on worker 1) · commit 864c5a0e

TL;DR: The ASan job aborted on a heap-use-after-free in uct_ib_reg_mr_trace because test_uct_stats's destructor calls stats_restore() (which frees the IB MD's stats node) while the still-alive UCT entity's UD async-progress thread keeps registering memory and bumping md->stats; this is a pre-existing test teardown-ordering bug (exposed when the test is SKIPPED from inside init()), not caused by PR #12003.

Full analysis

Summary: gtest crashed with AddressSanitizer: heap-use-after-free at src/uct/ib/base/ib_log.inl:27 during ud_verbs/test_uct_stats.atomic_fetch_SWAP64/0, failing the make test step (Bash exited with code '2').

Root cause: The freed 144-byte block is the IB MD stats node allocated in uct_ib_md_open_common (ib_md.c:1395, ucs_stats_node_alloc). It is freed by thread T0 via ucs_stats_cleanup → ucs_stats_clean_node_recurs → ucs_stats_node_clean_children (stats.c:824) called from test_base::stats_restore() (test/gtest/common/test.cc:249), invoked from ~test_uct_stats() (test/gtest/uct/test_stats.cc:40). Meanwhile the IB async thread T5220 — created by uct_ib_device_init (ib_device.c:682) when the MD was opened — is still running uct_ud_iface_timer → uct_ud_verbs_iface_async_progress → uct_ib_iface_prepare_rx_wrs → ucs_mpool_grow → uct_ib_mlx5_devx_reg_mr → uct_ib_reg_mr, which dereferences md->stats in UCS_STATS_UPDATE_COUNTER / md->pd in uct_ib_reg_mr_trace.

The ordering is wrong because this test instance was skipped from inside init(): uct_p2p_test::init() creates entity e1 and only then calls check_skip_test() (test/gtest/uct/uct_p2p_test.cc:70-73). The thrown skip exception leaves m_initialized == false (test.cc:412-419), so TearDownProxy() never calls cleanup() (test.cc:430-432) and therefore uct_test::cleanup()'s m_entities.clear() (uct_test.cc:483-488) never runs. The MD/iface (and its async thread) thus outlive the test body and are only destroyed by the entities_storage base destructor — i.e. after ~test_uct_stats() has already freed the stats tree. Hence the UAF. Note the log shows [ SKIP ] (!check_atomics(UCS_BIT(UCT_ATOMIC_OP_SWAP), FOP64)) immediately before the ASan report, confirming this path.

PR #12003 ("TOOLS/PERF: Separate device mode from memory placement") touches only tools/perf, so this is a pre-existing flaky/latent failure unrelated to the change under test.

Implicated commit: 2ee1df53 — "TEST/STATS: Make sure statistics are disabled after test is finished (#11201)", Yossi Itigin (introduced stats_restore() in the ~test_uct_stats() destructor); the dereference site itself comes from [REDACTED:Hex High Entropy String] "UCP/IB/CUDA: Direct NIC data path support", Raul Akhmetshin (ib_log.inl).

File: test/gtest/uct/test_stats.cc:38-41 (and src/uct/ib/base/ib_log.inl:27, test/gtest/common/test.cc:246-253)

Suggested fix: Destroy the UCT entities (and hence close the MDs and stop their async threads) before tearing down statistics. Concretely, stop calling stats_restore() from the destructor and do it in a cleanup() override that runs after the entities are released, plus guard the skipped-in-init() case:

// test/gtest/uct/test_stats.cc
~test_uct_stats() {
    m_entities.clear();   /* entities may survive if init() threw a skip */
    stats_restore();
}

virtual void cleanup() {
    uct_p2p_test::cleanup();   /* destroys eps + entities -> closes MD, stops async thread */
    stats_restore();           /* now safe: nothing references md->stats */
}

A more general hardening (worth a separate PR) is to make test_base::TearDownProxy() call cleanup() whenever init() created any state — e.g. when m_state == SKIPPED and initialization had already begun — so partially initialized fixtures always release their resources before per-test global teardown. Re-running the ASan job should let PR #12003 pass, but the race will keep recurring until the teardown order is fixed.

Related: #11201 (added stats_restore() to the destructor), #12003 (the innocent PR under test)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b74f8d7b-8d12-4fe3-9848-6f555851cf03 in the triage console for the audit trail.

Comment thread buildlib/tools/builds.sh Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR · commit 797057fc

Summary: Unable to fetch logs — cannot diagnose without them.
Root cause: Unknown — log access failed.
Implicated commit: unknown
File: unknown
Suggested fix: Provide a direct link to the failed build log in a follow-up comment.
Log fetch attempts:

  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=137893'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 137893]
  • fetch_jenkins_stages({'job': 'UCX PR', 'build': 137893}) → [fetch_jenkins_stages ERROR] build URL names Jenkins controller 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=137893', but this deployment is configured for 'https://nbuprod.blsm.nvidia.com/nb
  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=137893&view=logs'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 137893]
  • fetch_jenkins_logs({'job': 'UCX PR', 'build': 137893}) → [fetch_jenkins_logs ERROR] build URL names Jenkins controller 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=137893', but this deployment is configured for 'https://nbuprod.blsm.nvidia.com/nbu-
  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=137893'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 137893]
  • fetch_azure_logs({'build_url': 'https://dev.azure.com/ucfconsort/0b36e3f0-8ab9-4a48-b68b-[REDACTED:Hex High Entropy String]/_build/results?buildId=137893&view=results', 'job_id': '0b36e3f0-8ab9-4a48-b68b-4b2350e02c88'}) → [fetch_azure_logs ERROR: no failed job found in timeline for build 137893]

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7d968052-7c7f-4347-b9b6-840420105cda in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Artemy-Mellanox
Artemy-Mellanox previously approved these changes Oct 5, 2026

@tomerg-nvidia tomerg-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update copyright years

* do not work. Need to use bistro for memtype hooks to fix it.
*/
UCS_MODULE_FRAMEWORK_LOAD(ucx_perftest, UCS_MODULE_LOAD_FLAG_GLOBAL);
UCS_INIT_ONCE(&init_once) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What the init_once change? How is it related to the PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there was a race found by CI where multiple init were run in parallel leading to crash/sanitizer issue

Comment thread buildlib/tools/builds.sh
${ucx_inst}/bin/ucx_info -d)
check_ze_devices "${null_ze_info}" "null-driver"
run_perftest_device_id_smoke ze-device \
ZE_ENABLE_ALT_DRIVERS=/usr/local/lib/libze_null.so.1 UCX_TLS=self,ze_copy

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why set ZE_ENABLE_ALT_DRIVERS? Where did this path come from?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without it, the run fails as it uses the real/non-null ze driver that fails to initialized properly and ends-up crashing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

path from CI image: rdmz-harbor.rdmz.labs.mlnx/ucx/x86_64/ubuntu24.04/builder:ze-oneapi-2026.0.0-l0-1.27.0

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@tvegas1
tvegas1 enabled auto-merge (squash) October 5, 2026 10:41
@tvegas1
tvegas1 merged commit 40a33dc into openucx:master Oct 5, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants