Skip to content

UCP/CUDA: Prepare dormant async GDRCopy short copies - #12157

Open
nbellalou wants to merge 1 commit into
masterfrom
nv4741160-async-gdr-short-copies
Open

nbellalou wants to merge 1 commit into
masterfrom
nv4741160-async-gdr-short-copies

Conversation

@nbellalou

@nbellalou nbellalou commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

What?

Second of three PRs to enable gdr_copy for buffers allocated with cudaMallocAsync, following merged #12092.

This PR adds the complete short-copy routing and registration path, but leaves it dormant in normal use. The third PR will connect allocation detection, runtime qualification and the enable/disable setting.

Why?

Stream-ordered CUDA allocations need a targeted route to GDRCopy's DMA-BUF mapping path without making other transports newly eligible or changing the buffer's outward memory type. Preparing the complete path before activating it keeps this PR safe to merge on its own.

How?

  • Introduce the CUDA-async memory flag and the matching MD capability together with their readers. Select an async candidate only for GET/PUT short operations, and retain all required memory flags except legacy GDR_CAPABLE on that qualified route.
  • Keep async detection and registration in a private helper, with a shared preparation function for pack/unpack. Register an aligned region quietly, presenting it as CUDA only for this local registration. If registration fails, exclude the failed MD and use the original endpoint before any copy starts.
  • Preserve fatal/no-retry handling once a GET/PUT short operation has been issued; do not risk retrying a partially completed copy.
  • Use the operation-specific lane for qualified async short-copy cost estimates. Preserve the existing ordinary-buffer, zero-copy and unknown-remote-memory decisions.
  • Add routing/isolation and cost-model tests. Document why the local CUDA registration override is safe with GDRCopy's current shared-cache contract.

Scope

No production allocation detector emits the new marker, and no MD advertises the new capability in this PR. The qualification helper from #12092 still has no production caller. There is no new configuration setting, global reclassification, or change to other transports' eligibility.

The path can be exercised by tests that inject the marker and capability, but it does not enable async GDRCopy for applications. No measured zero-overhead or latency/throughput claim is made.

Validation

  • Native A100 developer build with GDRCopy 2.6: passed on Funk32.
  • Focused GDRCopy, async routing, datatype, registration, memory-flag and CPU-accessible eager-protocol suite: 83 executed passes, 1 C2C hardware skip.
  • Both added tests, cuda_async_short_route and cuda_async_short_perf, executed and passed.
  • Full-PR custom formatter and whitespace checks passed.
  • One signed feature commit (parent 59ca7e6d2e); 11 files, +452/-43.
  • No new R620 enabled-copy or performance acceptance run is claimed for this dormant PR.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 25, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 0) · commit b5b6bc1b

TL;DR: The build (not the tests) failed while linking test/mpi/shmem_pingpong — the HPC-X OpenMPI liboshmem.la on the BlueField worker drags in -liberty from a custom binutils path that doesn't exist on that machine, so ld fails. This is an agent/toolchain environment problem, unrelated to PR #12157; fix it by making the binutils/libiberty path available (or disabling the BFD dependency) on the BlueField agent.

Full analysis

Summary: BlueField on worker 0 failed in ./contrib/test_jenkins.sh during make all in build-test/test/mpi with /bin/ld: cannot find -liberty: No such file or directory → make[2]: *** [Makefile:727: shmem_pingpong] Error 1.

Root cause: Pure link-time environment breakage, not a code defect:

  • At 14:44:25 libtool emitted libtool: warning: library '/hpc/local/benchmarks/daily/ga/2026-10-06/hpcx-gcc-ubuntu22.04/ompi/lib/liboshmem.la' was moved. — the stale/moved .la file still carries hard-coded dependency_libs.
  • The resulting link line for shmem_pingpong therefore contains -L/hpc/local/oss/binutils/2.37/ubuntu22.04/lib -L/hpc/local/oss/binutils/2.37/ubuntu22.04/lib64 -L/tmp/binutils-gdb/zlib -lpthread -lz -liberty ....
  • Those are paths from a custom binutils build (the -lbfd/-liberty/-lz chain used by UCS backtrace support, see commit 29aa1f2 "UCS/BUILD: Support custom path for binutils dependency"). libiberty.a is not present/mounted on this BlueField agent (note -L/tmp/binutils-gdb/zlib, a temp dir that no longer exists), so -liberty cannot be resolved.
  • Everything before this point compiled and linked cleanly (libucs, libuct, libucp, ucx_info, ucx_perftest, io_demo all built), and the log shows continuous activity with no stalls — so this is not a hang or a timeout, and no test ever ran. Nothing in the failure touches the GDR short-copy code in this PR.

Implicated commit: unknown — not caused by commit [REDACTED:Hex High Entropy String] (nv4741160-async-gdr-short-copies). The relevant historical change is 29aa1f2 (Yossi Itigin, "UCS/BUILD: Support custom path for binutils dependency"), which introduced the custom-binutils link dependency, but the trigger is the HPC-X daily install ga/2026-10-06 on the BlueField agent.

File: build-test/test/mpi/Makefile:727 (generated from test/mpi/Makefile.am, target shmem_pingpong); dependency comes from /hpc/local/benchmarks/daily/ga/2026-10-06/hpcx-gcc-ubuntu22.04/ompi/lib/liboshmem.la

Suggested fix:

  1. Infra first: on the BlueField agent (agent-02), restore/mount /hpc/local/oss/binutils/2.37/ubuntu22.04/lib{,64} containing libiberty.a, or install the distro binutils-dev/libiberty-dev package; and refresh the HPC-X daily module so the relocated liboshmem.la no longer points at /tmp/binutils-gdb/zlib.
  2. Then re-run the PR build — it should pass without any code change.
  3. Hardening (optional, to stop a moved HPC-X .la from breaking every PR on BlueField): configure the BlueField job with the BFD/backtrace dependency off (e.g. --without-bfd/--disable-backtrace-detail, whichever the local configure.ac exposes) so the -lbfd -liberty -lz chain is not propagated into MPI test links, or skip building test/mpi in the BlueField flow since it only needs the MPI pingpong helpers.

Related: none found — no existing issue/PR in this repo matches cannot find -liberty / shmem_pingpong link failure.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id db9b2ac9-c855-485e-92ac-ffd8b05e3ab8 in the triage console for the audit trail.

UCS_MEM_FLAG_GDR_CAPABLE = UCS_BIT(2),

/** Memory originates from a CUDA stream-ordered allocation. */
UCS_MEM_FLAG_CUDA_ASYNC = UCS_BIT(3)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing in the tree sets this flag, and no MD sets UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY either, so the whole new route is unreachable except in the gtest that patches md_attr directly. are the producers (cuda_copy mem-flag detection and gdr_copy md_query using the existing uct_gdr_copy_cuda_async_detect()) planned for a follow-up PR?

Comment thread src/ucp/dt/dt.c Outdated
ucs_status_t status;

if ((mem_info == NULL) &&
(ucp_worker_cuda_async_ep_for_short(worker, &detected, operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this probe (the mem_info == NULL probe) passes only the fabricated UCS_MEM_FLAG_CUDA_ASYNC, so an MD whose required_mem_flags also contain e.g. UCS_MEM_FLAG_REGISTRABLE never passes ucs_test_all_flags() here, and detection (hence the async route) is silently skipped for it. why not probe with all mem flags set and let the second call do the real flag matching?

Comment thread src/ucp/dt/dt.c Outdated
}

async_ep = ((mem_info == NULL) ||
!(mem_info->flags & UCS_MEM_FLAG_CUDA_ASYNC)) ? NULL :

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: ucp_worker_cuda_async_ep_for_short() already filters by mem type and by UCS_MEM_FLAG_CUDA_ASYNC, so the extra flag test here is redundant - (mem_info == NULL) ? NULL : ucp_worker_cuda_async_ep_for_short(...) is enough.

Comment thread src/ucp/dt/dt.c Outdated
ucs_align_ptr_range(&reg_address, &reg_length,
ucp_ep_md_attr(async_ep, lane)->reg_alignment);
status = ucp_mem_type_reg_buffers(
worker, reg_address, reg_length, UCS_MEMORY_TYPE_CUDA,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why register as UCS_MEMORY_TYPE_CUDA when the buffer may be CUDA_MANAGED? if this md is in context->cache_md_map[], ucp_mem_type_reg_buffers() goes through ucp_memh_get(), and ucp_memh_find_slow() asserts the cached region mem_type equals the requested one - a managed buffer already cached as CUDA_MANAGED would trip that assert.


if (worker->mem_type_ep[local_mem_type] != NULL) {
ep_config = ucp_ep_config(worker->mem_type_ep[local_mem_type]);
copy_mem_info = local_mem_info;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy_mem_info follows whichever side happens to have a mem_type ep, so for the mtype "local copy" (cuda frag vs cuda user buffer) the frag flags win and the user buffer's async flag is dropped: the estimate keeps rma_lanes[0] while the runtime copy takes the async lane. should the async check consider both mem_infos?

m_counts{0, 0}
{
ucs_assert(m_active == nullptr);
ucs_assert(m_ifaces[0]->internal_ops != m_ifaces[1]->internal_ops);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucs_assert() is compiled out in release builds, and then m_orig[1] gets the hook itself if both ifaces share internal_ops, so estimate_perf() recurses forever. pls make this a real check in the test (skip/fail) rather than an assert, and consider copying internal_ops and swapping the pointer instead of writing into the per-TL table shared by all ifaces of that transport.

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
auto saved = ucs_global_opts.handle_errors;
ucs_global_opts.handle_errors = 0;
if (operation == UCT_EP_OP_GET_SHORT) {
EXPECT_DEATH(ucp_mem_type_pack(worker(), buffer, buffer, 64,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the first death test in the gtest suite; EXPECT_DEATH forks a child that already has an initialized CUDA context and open UCT ifaces, which is not supported and tends to hang or die with an unrelated message. can we assert the "no retry" behavior without killing the process (e.g. check lane/ep selection and registration), or drop these two cases?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: No test exercises the async route end to end with a real stream-ordered allocation, because no MD advertises the new capability; the new tests only validate the selection helper and the perf model with hand-patched md_attr.

Residual coverage gap: test_short_copy_perf() feeds a remote_mem_info carrying UCS_MEM_FLAG_CUDA_ASYNC, but ucp_proto_init_add_buffer_perf() now forces recv_mem_info.flags = 0, so that input cannot occur in production.

Note (not flagged): rndv_rtr.c now derives the "unpack copy" local type from params.super.reg_mem_info instead of frag_mem_type; this is equivalent only because UCS_MEMORY_TYPE_HOST is bit 0 and is the one type that may skip ucp_mm_get_alloc_md_index() success - a hidden dependency on iteration order.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/dt/dt.c Outdated
ucs_align_ptr_range(&reg_address, &reg_length,
ucp_ep_md_attr(async_ep, lane)->reg_alignment);
status = ucp_mem_type_reg_buffers(
worker, reg_address, reg_length, UCS_MEMORY_TYPE_CUDA,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why register as UCS_MEMORY_TYPE_CUDA and not mem_type? The async branch passes UCS_MEMORY_TYPE_CUDA while the buffer the caller described is mem_type (typically UCS_MEMORY_TYPE_CUDA_MANAGED for stream-ordered allocations). Inside ucp_mem_type_reg_buffers() that value drives both reg_params.mem_type and the context->cache_md_map[mem_type] branch, so for a CUDA_MANAGED buffer it also flips the cache_md_map[CUDA] decision and would call ucp_memh_get() with a type that mismatches an existing rcache region for the same range, hitting the ucs_assertv(mem_type == memh->mem_type) assertion in ucp_memh_find_slow(). It works today only because gdr_copy leaves cache_mem_types == 0, which is an undocumented dependency. Pls use mem_type, or add a comment on why the override is safe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CUDA override is intentional: the buffer keeps its CUDA_MANAGED classification to preserve existing UCX behavior, while this registration uses GDRCopy’s CUDA-only MD. GDRCopy has cache_mem_types == 0, so the shared-cache mismatch doesn’t occur here. I agree that this dependency deserves a comment, but changing the argument to mem_type would not match the MD’s declared registration type.

Comment thread src/ucp/core/ucp_ep.c Outdated
/* DMA-BUF CPU mapping waives only legacy peer-pinning eligibility. */
if (ucs_test_all_flags(mem_info->flags,
md_attr->required_mem_flags &
~UCS_MEM_FLAG_GDR_CAPABLE)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bakes gdr_copy-specific policy into UCP core: md_attr->required_mem_flags & ~UCS_MEM_FLAG_GDR_CAPABLE silently drops that requirement for any MD that sets UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY, duplicating the generic required_mem_flags mechanism. src/AGENTS.md asks UCP to compose UCT capabilities rather than bake in backend behavior; the MD is the one that knows its DMA-BUF path no longer needs peer pinning. Why not have the MD report the relaxed required_mem_flags when it can use the DMA-BUF path, so UCP keeps the generic check?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

required_mem_flags applies to all buffers. Clearing GDR_CAPABLE there would also change eligibility for ordinary buffers, which we want to preserve. Here the exception applies only to async buffers, qualified MDs and short-copy operations. That is why the relaxation is local to this selection path.

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
auto saved = ucs_global_opts.handle_errors;
ucs_global_opts.handle_errors = 0;
if (operation == UCT_EP_OP_GET_SHORT) {
EXPECT_DEATH(ucp_mem_type_pack(worker(), buffer, buffer, 64,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EXPECT_DEATH is not used anywhere else in the UCX gtest suite, and it forks a process that already has the watchdog thread and a CUDA context initialized by main() (which calls mem_buffer::set_device_context() before any test), so the child executes CUDA/gdr_copy code in a forked multi-threaded process — gtest documents this as unsafe and it is a likely source of CI hangs/flakes. Can we assert the failure another way (e.g. intercept the log handler) instead of a death test?

Comment thread src/ucp/dt/dt.c Outdated

/* Preserve the original lane walk except for an already failed async MD. */
lane = ucp_mem_type_ep_find_lane(ep, address, length, mem_info,
skip_md_index);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continuation indent is off by one; .clang-format uses AlignAfterOpenBracket: true and these lines were not re-aligned after the signatures changed:

Suggested change
skip_md_index);
lane = ucp_mem_type_ep_find_lane(ep, address, length, mem_info,
skip_md_index);

Pls run git-clang-format; same off-by-one continuation indent in ucp_ep.c:843,855 and in the new test helpers (test_ucp_proto.cc:950,964,1023), otherwise the format code job (buildlib/pr/codestyle.yml runs git-clang-format --diff on the PR range) will fail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: nothing exercises the registration-failure fallback in ucp_mem_type_lane_reg() (async ucp_mem_type_reg_buffers() returning non-OK, then re-running the lane walk with skip_md_index), which is the path most likely to regress once an MD advertises the new capability.

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 98fa2aef

TL;DR: The ASAN/RoCE gtest job didn't time out globally — one test, dcx/test_ucp_tag_match_rndv.sync_send_unexp/6 <dc_x/rndv_put_flush,proto_v1>, hung for exactly 900 s with zero output until the gtest watchdog aborted the process, i.e. a rendezvous PUT+force-flush request never completed; investigate the rndv completion/fragment-wakeup path rather than raising any timeout.

Full analysis

Summary: gtest shard 1/4 (AddressSanitizer, roce, worker 0) aborted with SIGABRT because the gtest watchdog killed the hung test dcx/test_ucp_tag_match_rndv.sync_send_unexp/6 <dc_x/rndv_put_flush,proto_v1>; make -C test/gtest test then failed with "Aborted (core dumped)".

Root cause: A hang, not slowness. Timeline from the log: the test started at 11:23:08.708 ([ RUN ] ... sync_send_unexp/6 <dc_x/rndv_put_flush,proto_v1>) and the next line in the whole job is at 11:38:08.662 — a single 15:00 gap with no intervening output. 900 s is exactly watchdog_timeout in test/gtest/common/test_helpers.cc:31, and the failure text Connection timed out - abort testing is ADD_FAILURE() << strerror(ETIMEDOUT) from test_helpers.cc:57, after which the watchdog signals the watched thread (backtrace frame 5 is __sched_yield inside the gtest progress/wait loop). So the main thread was spinning in a progress loop waiting for a UCP request completion that never arrived — a lost completion/wakeup in the proto-v1 rendezvous PUT with forced flush over dc_x. No UCX error, assertion, ASAN report or transport error was printed, which is consistent with a silently stalled request (e.g. a request parked on a pending/flow-control queue that is never rescheduled, or a flush completion that is never invoked), not with a crash or a genuinely long transfer.

Implicated commit: Not provable from this single log. Two candidates, both touching the exact code the hung variant exercises:

  • 1acfdd61 (amastbaum, 2026-11-03) "UCP/RNDV: Throttle rndv fragment requests (UCP/RNDV: Throttle rndv fragment requests #11062)" — merged 6 days before this run and the most recent change to every rndv file (rndv.c, rndv_put.c, rndv_get.c, proto_rndv.c/.inl, rndv_mtype.inl) plus ucp_mm.h/ucp_context.c. It introduced worker-level fragment quotas with pending queues and hand-off of "wakeup ownership"; a dropped wakeup there stalls a request forever with no log output.
  • 98fa2aef (Nathan Bellalou, PR UCP/CUDA: Prepare dormant async GDRCopy short copies #12157 head) "UCP/CUDA: Prepare dormant async GDRCopy short copies" — touches src/ucp/core/ucp_mm.c, src/ucp/core/ucp_mm.h (which holds the UCP_MEMH_FLAG_SEND_FLUSH_CHECKED/NEEDED sender-flush caching flags) and src/ucp/rndv. The fact that the one variant that hung is precisely the rndv_put_flush one makes a wrong/stale cached "send flush needed" decision worth ruling out, even though the change is advertised as dormant/CUDA-only while this test is host memory over RoCE.

File: test/gtest/common/test_helpers.cc:57 (watchdog that reported it); real suspects: src/ucp/rndv/rndv_mtype.inl:58 (ucp_proto_rndv_mtype_fc_reschedule_pending, single dequeue per released fragment + wakeup hand-off) and src/ucp/core/ucp_mm.h:48-53 (cached sender-flush flags).

Suggested fix:

  1. Do not raise the watchdog/timeout 200m limit — the process made no progress for 15 min.
  2. Reproduce in isolation and capture the stuck state: GTEST_FILTER='dcx/test_ucp_tag_match_rndv.sync_send_unexp/6' UCX_LOG_LEVEL=req UCX_PROTO_ENABLE=n on the roce node, and when it hangs attach gdb / dump ucp_worker state to see whether the request is sitting on worker->rndv_mtype_fc.pending_q[], on a UCT pending queue, or waiting on an un-invoked uct_ep_flush completion.
  3. Audit the fragment flow-control wakeup logic added in 1acfdd61: ucp_proto_rndv_mtype_fc_reschedule_pending() wakes only one waiter per released fragment and relies on FC_RESCHED requests passing their wakeup on (request_init/fc_cancel paths); verify no path releases an mdesc without a reschedule and that proto-v1 callers of the new ucp_rndv_mpool_get(..., POOL_*) handle UCS_ERR_NO_RESOURCE (proto v1 has no pending queue to retry from) — add a stall watchdog/assert there.
  4. In parallel, confirm whether the hang reproduces on the merge-base without 98fa2aef; if it does, the regression is master-side (UCP/RNDV: Throttle rndv fragment requests #11062) and the PR should be re-run, otherwise re-check the memh sender-flush caching introduced/modified by the PR against the UCX_RNDV_PUT_FORCE_FLUSH path.

Related: PR #12157 (this PR), PR #11062 / #12276 "UCP/RNDV: Throttle rndv fragment requests", PR #11673 "UCP/RNDV: Skip PUT fence for ordered transports" (origin of the rndv_put_flush variant).

@nbellalou
nbellalou force-pushed the nv4741160-async-gdr-short-copies branch from 98fa2ae to de686df Compare November 10, 2026 15:15
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/dt/dt.c
* CUDA_MANAGED. Its cache_mem_types == 0 keeps this local type override out
* of UCP's shared registration cache, avoiding conflicting cached types.
*/
status = ucp_mem_type_reg_buffers(worker, reg_address, reg_length,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GDR_CAPABLE waiver in ucp_worker_cuda_async_ep_for_short() is not mirrored in ucp_memh_register_internal(), which still filters MDs by required_mem_flags against the detected mem_flags; this only works because gdr_copy has cache_mem_types == 0 so we take the direct uct_md_mem_reg_v2() branch here. pls add an ucs_assertv() that md_index is not in context->cache_md_map[], otherwise an async-capable cached MD makes ucp_memh_get() return UCS_OK with the MD filtered out and pack_context->uct_memh is then read from an unregistered slot.

Comment thread src/ucp/dt/dt.c
.flags = UCS_MEM_FLAG_CUDA_ASYNC
};
if ((mem_info == NULL) &&
(ucp_worker_cuda_async_ep_for_short(worker, detected, operation,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once any MD advertises the new capability, every ucp_mem_type_pack()/unpack() on CUDA memory pays this out-of-line lane walk plus ucp_memory_detect(), also for buffers that are not stream-ordered - before, find_lane() returned on the first lane with required_mem_flags == 0 without any detection. can we resolve the async ep/lane once per mem_type ep config instead of per copy? This is the short-message data path (docs/OptimizationStyle.md).

Comment thread src/ucp/dt/dt.c
ucp_md_index_t md_index;
ucs_status_t status;

*detected = (ucp_memory_info_t){

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls align the field assignments, and set sys_dev explicitly - the compound literal zeroes it, which is device id 0 rather than unknown.

Suggested change
*detected = (ucp_memory_info_t){
*detected = (ucp_memory_info_t){
.type = mem_type,
.sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN,
.flags = UCS_MEM_FLAG_CUDA_ASYNC
};

@@ -350,8 +354,10 @@ ucp_proto_init_add_buffer_copy_time(ucp_worker_h worker, const char *title,

if (worker->mem_type_ep[local_mem_type] != NULL) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: consecutive assignments should align on the column (docs/CodeStyle.md).

Suggested change
if (worker->mem_type_ep[local_mem_type] != NULL) {
if (worker->mem_type_ep[local_mem_type] != NULL) {
ep_config = ucp_ep_config(worker->mem_type_ep[local_mem_type]);
copy_mem_info = local_mem_info;
} else if (worker->mem_type_ep[remote_mem_type] != NULL) {
ep_config = ucp_ep_config(worker->mem_type_ep[remote_mem_type]);
copy_mem_info = remote_mem_info;

Comment thread src/uct/api/v2/uct_v2.h
UCT_MD_FLAG_IPC_MEMTYPE_COPY = UCS_BIT(13),

/** MD supports short copies for CUDA stream-ordered allocations. */
UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY = UCS_BIT(14)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A CUDA-specific MD capability in the generic v2 flag enum is what forces UCP core to hardcode mem_type_ep[UCS_MEMORY_TYPE_CUDA] in ucp_worker_cuda_async_ep_for_short(). can we express the capability generically (MD can short-copy memory that does not satisfy its required_mem_flags) so UCP stays memory-type agnostic, per src/AGENTS.md?

m_active = this;
for (unsigned i = 0; i < 2; ++i) {
m_orig[i] = m_ifaces[i]->internal_ops->iface_estimate_perf;
m_ifaces[i]->internal_ops->iface_estimate_perf = estimate_perf;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: internal_ops points to one file-static table shared by all ifaces of the TL (e.g. uct_gdr_copy_iface_internal_ops), so this patches the op process-wide and needs the identity check in estimate_perf(). can we use ucs::mock from test_helpers.h on &internal_ops->iface_estimate_perf instead? Same RAII restore without the static m_active singleton (test/AGENTS.md: reuse common helpers).

@svc-nvidia-pr-review

Copy link
Copy Markdown

Notes on what I checked and did not flag: PR size is ~440 added lines, under the 500 limit; the uct_v2.h re-alignment is required by the longer enum name, not unrelated refactoring. The ep == NULL path into ucp_mem_type_lane_reg() is unchanged from the pre-PR behavior, so not a new NULL deref. Registering the page-aligned range and then packing/using the unaligned remote_addr is correct for gdr_copy, since bar_offset = remote_addr - key->vaddr. ucp_ep.h including ucp/dt/dt.h introduces no include cycle (dt.h only reaches ucp_types.h). copy_mem_info cannot be used uninitialized; the third branch returns. The four test_short_copy_perf() expectations match the code: mem_type_ep[HOST] is always NULL, so copy_mem_info is the remote one, and ucp_worker_iface_estimate_perf() does no caching.

Residual coverage gaps: the async path is only reachable with a hand-patched md_attr; nothing in the tree advertises UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY or sets UCS_MEM_FLAG_CUDA_ASYNC (already reported in earlier rounds, not repeated). The latent cached-MD registration path described in the first comment is untestable today for the same reason, which is exactly why an assertion rather than a test is the right ask.

@svc-ucx

svc-ucx commented Nov 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 0) · commit de686df3

TL;DR: The only failure in the whole 8966-test run is ib/test_md.alloc/0 <mlx5_0>, which exercises HCA device-memory (DM) allocation plus an atomic/invalidating mkey pack — a path the PR (UCP/CUDA GDRCopy) does not touch; the suspect is the just-merged atomic-MR-offset change (e4d1e59 / PR #12265) combined with the MD-open ordering that probes DM atomic keys before the flush mkey exists. Re-run the single test and move uct_ib_mlx5dv_check_dm_ksm_reg() after uct_ib_mlx5_devx_init_flush_mr() so probe and runtime use the same atomic_mr_id.

Full analysis

Summary: make test exited 2 (Makefile:5014) because 1 of 8966 gtests failed: [ FAILED ] ib/test_md.alloc/0, where GetParam() = mlx5_0; all other tests passed.

Root cause: From the log we know only the failing test identity (the per-assertion output is earlier in the step log than the ~500-line tail that is retrievable; the tail contains just the gtest summary). From the source, ib/test_md.alloc (test/gtest/uct/test_md.cc:479-562) is the only test that uses the IB MD's UCT_MD_FLAG_ALLOC capability, which for mlx5 DEVX means HCA device memory (UCS_MEMORY_TYPE_RDMA, ibv_alloc_dm, uct_ib_mlx5_devx_device_mem_alloc() in src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2123). It does 300 alloc/pack/free iterations of up to ~63 KB out of the device's very small, node-global max_dm_size, and it can fail in exactly three places:

  • ASSERT_UCS_OK(status) on uct_mem_alloc (test_md.cc:541) for any non-NO_MEMORY error,
  • EXPECT_LT(num_alloc_failures/300, 0.5) "Too many OUT_OF_RESOURCE failures" (test_md.cc:559) if DM is exhausted/shared with the other parallel gtest workers,
  • ASSERT_UCS_OK on uct_md_mkey_pack_v2(... INVALIDATE_RMA|INVALIDATE_AMO) (test_md.cc:551-553), which returns UCS_ERR_INVALID_PARAM from uct_ib_mlx5_devx_mkey_pack() (ib_mlx5dv_md.c:3022: "cannot invalidate AMO without creating an atomic key") whenever the indirect atomic KSM key could not be created.

The last path is directly affected by the newly merged atomic-MR-offset change: uct_ib_mlx5_devx_reg_atomic_key() (ib_mlx5dv_md.c:598) uses atomic_offset = 8 * uct_ib_md_get_atomic_mr_id(md) = 8 * (flush_rkey >> 8), while the DM capability probe uct_ib_mlx5dv_check_dm_ksm_reg() runs at ib_mlx5dv_md.c:2649 — before uct_ib_mlx5_devx_init_flush_mr() at :2652 — i.e. with flush_rkey == 0 and offset 0. So the probe can enable UCT_MD_FLAG_ALLOC with offset 0 while the test later creates the DM atomic key with a non-zero IOVA offset, which can fail CREATE_MKEY and make mkey_pack return UCS_ERR_INVALID_PARAM. Nothing in PR #12157 (UCP/CUDA async GDRCopy short copies) touches the IB MD, DM allocation, or KSM atomic keys, so the PR is not the cause.

Implicated commit: e4d1e59 — "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key (#12265)", dmitrygx, 2026-11-08 (merged 3 days before this run); not the PR under test (de686df).

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:598-621 (uct_ib_mlx5_devx_reg_atomic_key) and :2648-2652 (DM probe ordered before flush-MR init); test side: test/gtest/uct/test_md.cc:541,551-559.

Suggested fix:

  1. Re-run just the failing test to capture the assertion line (GTEST_FILTER=ib/test_md.alloc UCX_LOG_LEVEL=debug ./gtest/gtest --gtest_filter='ib/test_md.alloc*') — this distinguishes "too many OUT_OF_RESOURCE" (environment/DM exhaustion from parallel workers) from the mkey_pack/atomic-key error.
  2. In uct_ib_mlx5_devx_md_open_common(), call uct_ib_mlx5_devx_init_flush_mr(md) before uct_ib_mlx5dv_check_dm_ksm_reg(md), so the DM atomic-key probe uses the same atomic_mr_id/offset as runtime; alternatively force atomic_offset = 0 for DM memhs (uct_ib_mlx5_devx_has_dm()), since the DM key is zero-based.
  3. If the log shows the OUT_OF_RESOURCE budget instead, make the test robust to a node-shared max_dm_size (skip when max_dm_size is tiny, or serialize the DM test across gtest workers) rather than treating it as a PR regression.

Related: PR under test: #12157 · suspect merged change: #12265 · adjacent IB MD work: #12271

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants