Repository navigation
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The build (not the tests) failed while linking Full analysisSummary: Root cause: Pure link-time environment breakage, not a code defect:
Implicated commit: unknown — not caused by commit [REDACTED:Hex High Entropy String] ( File: Suggested fix:
Related: none found — no existing issue/PR in this repo matches
|
| UCS_MEM_FLAG_GDR_CAPABLE = UCS_BIT(2), | ||
|
|
||
| /** Memory originates from a CUDA stream-ordered allocation. */ | ||
| UCS_MEM_FLAG_CUDA_ASYNC = UCS_BIT(3) |
There was a problem hiding this comment.
nothing in the tree sets this flag, and no MD sets UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY either, so the whole new route is unreachable except in the gtest that patches md_attr directly. are the producers (cuda_copy mem-flag detection and gdr_copy md_query using the existing uct_gdr_copy_cuda_async_detect()) planned for a follow-up PR?
| ucs_status_t status; | ||
|
|
||
| if ((mem_info == NULL) && | ||
| (ucp_worker_cuda_async_ep_for_short(worker, &detected, operation, |
There was a problem hiding this comment.
this probe (the mem_info == NULL probe) passes only the fabricated UCS_MEM_FLAG_CUDA_ASYNC, so an MD whose required_mem_flags also contain e.g. UCS_MEM_FLAG_REGISTRABLE never passes ucs_test_all_flags() here, and detection (hence the async route) is silently skipped for it. why not probe with all mem flags set and let the second call do the real flag matching?
| } | ||
|
|
||
| async_ep = ((mem_info == NULL) || | ||
| !(mem_info->flags & UCS_MEM_FLAG_CUDA_ASYNC)) ? NULL : |
There was a problem hiding this comment.
minor: ucp_worker_cuda_async_ep_for_short() already filters by mem type and by UCS_MEM_FLAG_CUDA_ASYNC, so the extra flag test here is redundant - (mem_info == NULL) ? NULL : ucp_worker_cuda_async_ep_for_short(...) is enough.
| ucs_align_ptr_range(®_address, ®_length, | ||
| ucp_ep_md_attr(async_ep, lane)->reg_alignment); | ||
| status = ucp_mem_type_reg_buffers( | ||
| worker, reg_address, reg_length, UCS_MEMORY_TYPE_CUDA, |
There was a problem hiding this comment.
why register as UCS_MEMORY_TYPE_CUDA when the buffer may be CUDA_MANAGED? if this md is in context->cache_md_map[], ucp_mem_type_reg_buffers() goes through ucp_memh_get(), and ucp_memh_find_slow() asserts the cached region mem_type equals the requested one - a managed buffer already cached as CUDA_MANAGED would trip that assert.
|
|
||
| if (worker->mem_type_ep[local_mem_type] != NULL) { | ||
| ep_config = ucp_ep_config(worker->mem_type_ep[local_mem_type]); | ||
| copy_mem_info = local_mem_info; |
There was a problem hiding this comment.
copy_mem_info follows whichever side happens to have a mem_type ep, so for the mtype "local copy" (cuda frag vs cuda user buffer) the frag flags win and the user buffer's async flag is dropped: the estimate keeps rma_lanes[0] while the runtime copy takes the async lane. should the async check consider both mem_infos?
| m_counts{0, 0} | ||
| { | ||
| ucs_assert(m_active == nullptr); | ||
| ucs_assert(m_ifaces[0]->internal_ops != m_ifaces[1]->internal_ops); |
There was a problem hiding this comment.
ucs_assert() is compiled out in release builds, and then m_orig[1] gets the hook itself if both ifaces share internal_ops, so estimate_perf() recurses forever. pls make this a real check in the test (skip/fail) rather than an assert, and consider copying internal_ops and swapping the pointer instead of writing into the per-TL table shared by all ifaces of that transport.
| auto saved = ucs_global_opts.handle_errors; | ||
| ucs_global_opts.handle_errors = 0; | ||
| if (operation == UCT_EP_OP_GET_SHORT) { | ||
| EXPECT_DEATH(ucp_mem_type_pack(worker(), buffer, buffer, 64, |
There was a problem hiding this comment.
this is the first death test in the gtest suite; EXPECT_DEATH forks a child that already has an initialized CUDA context and open UCT ifaces, which is not supported and tends to hang or die with an unrelated message. can we assert the "no retry" behavior without killing the process (e.g. check lane/ep selection and registration), or drop these two cases?
|
Residual coverage gap: No test exercises the async route end to end with a real stream-ordered allocation, because no MD advertises the new capability; the new tests only validate the selection helper and the perf model with hand-patched Residual coverage gap: Note (not flagged): |
b5b6bc1 to
98fa2ae
Compare
|
🤖 Starting review — findings will be posted here when done. |
| ucs_align_ptr_range(®_address, ®_length, | ||
| ucp_ep_md_attr(async_ep, lane)->reg_alignment); | ||
| status = ucp_mem_type_reg_buffers( | ||
| worker, reg_address, reg_length, UCS_MEMORY_TYPE_CUDA, |
There was a problem hiding this comment.
why register as UCS_MEMORY_TYPE_CUDA and not mem_type? The async branch passes UCS_MEMORY_TYPE_CUDA while the buffer the caller described is mem_type (typically UCS_MEMORY_TYPE_CUDA_MANAGED for stream-ordered allocations). Inside ucp_mem_type_reg_buffers() that value drives both reg_params.mem_type and the context->cache_md_map[mem_type] branch, so for a CUDA_MANAGED buffer it also flips the cache_md_map[CUDA] decision and would call ucp_memh_get() with a type that mismatches an existing rcache region for the same range, hitting the ucs_assertv(mem_type == memh->mem_type) assertion in ucp_memh_find_slow(). It works today only because gdr_copy leaves cache_mem_types == 0, which is an undocumented dependency. Pls use mem_type, or add a comment on why the override is safe.
There was a problem hiding this comment.
The CUDA override is intentional: the buffer keeps its CUDA_MANAGED classification to preserve existing UCX behavior, while this registration uses GDRCopy’s CUDA-only MD. GDRCopy has cache_mem_types == 0, so the shared-cache mismatch doesn’t occur here. I agree that this dependency deserves a comment, but changing the argument to mem_type would not match the MD’s declared registration type.
| /* DMA-BUF CPU mapping waives only legacy peer-pinning eligibility. */ | ||
| if (ucs_test_all_flags(mem_info->flags, | ||
| md_attr->required_mem_flags & | ||
| ~UCS_MEM_FLAG_GDR_CAPABLE)) { |
There was a problem hiding this comment.
This bakes gdr_copy-specific policy into UCP core: md_attr->required_mem_flags & ~UCS_MEM_FLAG_GDR_CAPABLE silently drops that requirement for any MD that sets UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY, duplicating the generic required_mem_flags mechanism. src/AGENTS.md asks UCP to compose UCT capabilities rather than bake in backend behavior; the MD is the one that knows its DMA-BUF path no longer needs peer pinning. Why not have the MD report the relaxed required_mem_flags when it can use the DMA-BUF path, so UCP keeps the generic check?
There was a problem hiding this comment.
required_mem_flags applies to all buffers. Clearing GDR_CAPABLE there would also change eligibility for ordinary buffers, which we want to preserve. Here the exception applies only to async buffers, qualified MDs and short-copy operations. That is why the relaxation is local to this selection path.
| auto saved = ucs_global_opts.handle_errors; | ||
| ucs_global_opts.handle_errors = 0; | ||
| if (operation == UCT_EP_OP_GET_SHORT) { | ||
| EXPECT_DEATH(ucp_mem_type_pack(worker(), buffer, buffer, 64, |
There was a problem hiding this comment.
EXPECT_DEATH is not used anywhere else in the UCX gtest suite, and it forks a process that already has the watchdog thread and a CUDA context initialized by main() (which calls mem_buffer::set_device_context() before any test), so the child executes CUDA/gdr_copy code in a forked multi-threaded process — gtest documents this as unsafe and it is a likely source of CI hangs/flakes. Can we assert the failure another way (e.g. intercept the log handler) instead of a death test?
|
|
||
| /* Preserve the original lane walk except for an already failed async MD. */ | ||
| lane = ucp_mem_type_ep_find_lane(ep, address, length, mem_info, | ||
| skip_md_index); |
There was a problem hiding this comment.
Continuation indent is off by one; .clang-format uses AlignAfterOpenBracket: true and these lines were not re-aligned after the signatures changed:
| skip_md_index); | |
| lane = ucp_mem_type_ep_find_lane(ep, address, length, mem_info, | |
| skip_md_index); |
Pls run git-clang-format; same off-by-one continuation indent in ucp_ep.c:843,855 and in the new test helpers (test_ucp_proto.cc:950,964,1023), otherwise the format code job (buildlib/pr/codestyle.yml runs git-clang-format --diff on the PR range) will fail.
|
Residual coverage gap: nothing exercises the registration-failure fallback in |
|
🤖 CI Triage Agent — TL;DR: The ASAN/RoCE gtest job didn't time out globally — one test, Full analysisSummary: gtest shard 1/4 (AddressSanitizer, roce, worker 0) aborted with SIGABRT because the gtest watchdog killed the hung test Root cause: A hang, not slowness. Timeline from the log: the test started at Implicated commit: Not provable from this single log. Two candidates, both touching the exact code the hung variant exercises:
File: Suggested fix:
Related: PR #12157 (this PR), PR #11062 / #12276 "UCP/RNDV: Throttle rndv fragment requests", PR #11673 "UCP/RNDV: Skip PUT fence for ordered transports" (origin of the |
98fa2ae to
de686df
Compare
|
🤖 Starting review — findings will be posted here when done. |
| * CUDA_MANAGED. Its cache_mem_types == 0 keeps this local type override out | ||
| * of UCP's shared registration cache, avoiding conflicting cached types. | ||
| */ | ||
| status = ucp_mem_type_reg_buffers(worker, reg_address, reg_length, |
There was a problem hiding this comment.
The GDR_CAPABLE waiver in ucp_worker_cuda_async_ep_for_short() is not mirrored in ucp_memh_register_internal(), which still filters MDs by required_mem_flags against the detected mem_flags; this only works because gdr_copy has cache_mem_types == 0 so we take the direct uct_md_mem_reg_v2() branch here. pls add an ucs_assertv() that md_index is not in context->cache_md_map[], otherwise an async-capable cached MD makes ucp_memh_get() return UCS_OK with the MD filtered out and pack_context->uct_memh is then read from an unregistered slot.
| .flags = UCS_MEM_FLAG_CUDA_ASYNC | ||
| }; | ||
| if ((mem_info == NULL) && | ||
| (ucp_worker_cuda_async_ep_for_short(worker, detected, operation, |
There was a problem hiding this comment.
Once any MD advertises the new capability, every ucp_mem_type_pack()/unpack() on CUDA memory pays this out-of-line lane walk plus ucp_memory_detect(), also for buffers that are not stream-ordered - before, find_lane() returned on the first lane with required_mem_flags == 0 without any detection. can we resolve the async ep/lane once per mem_type ep config instead of per copy? This is the short-message data path (docs/OptimizationStyle.md).
| ucp_md_index_t md_index; | ||
| ucs_status_t status; | ||
|
|
||
| *detected = (ucp_memory_info_t){ |
There was a problem hiding this comment.
minor: pls align the field assignments, and set sys_dev explicitly - the compound literal zeroes it, which is device id 0 rather than unknown.
| *detected = (ucp_memory_info_t){ | |
| *detected = (ucp_memory_info_t){ | |
| .type = mem_type, | |
| .sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN, | |
| .flags = UCS_MEM_FLAG_CUDA_ASYNC | |
| }; |
| @@ -350,8 +354,10 @@ ucp_proto_init_add_buffer_copy_time(ucp_worker_h worker, const char *title, | |||
|
|
|||
| if (worker->mem_type_ep[local_mem_type] != NULL) { | |||
There was a problem hiding this comment.
minor: consecutive assignments should align on the column (docs/CodeStyle.md).
| if (worker->mem_type_ep[local_mem_type] != NULL) { | |
| if (worker->mem_type_ep[local_mem_type] != NULL) { | |
| ep_config = ucp_ep_config(worker->mem_type_ep[local_mem_type]); | |
| copy_mem_info = local_mem_info; | |
| } else if (worker->mem_type_ep[remote_mem_type] != NULL) { | |
| ep_config = ucp_ep_config(worker->mem_type_ep[remote_mem_type]); | |
| copy_mem_info = remote_mem_info; |
| UCT_MD_FLAG_IPC_MEMTYPE_COPY = UCS_BIT(13), | ||
|
|
||
| /** MD supports short copies for CUDA stream-ordered allocations. */ | ||
| UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY = UCS_BIT(14) |
There was a problem hiding this comment.
A CUDA-specific MD capability in the generic v2 flag enum is what forces UCP core to hardcode mem_type_ep[UCS_MEMORY_TYPE_CUDA] in ucp_worker_cuda_async_ep_for_short(). can we express the capability generically (MD can short-copy memory that does not satisfy its required_mem_flags) so UCP stays memory-type agnostic, per src/AGENTS.md?
| m_active = this; | ||
| for (unsigned i = 0; i < 2; ++i) { | ||
| m_orig[i] = m_ifaces[i]->internal_ops->iface_estimate_perf; | ||
| m_ifaces[i]->internal_ops->iface_estimate_perf = estimate_perf; |
There was a problem hiding this comment.
minor: internal_ops points to one file-static table shared by all ifaces of the TL (e.g. uct_gdr_copy_iface_internal_ops), so this patches the op process-wide and needs the identity check in estimate_perf(). can we use ucs::mock from test_helpers.h on &internal_ops->iface_estimate_perf instead? Same RAII restore without the static m_active singleton (test/AGENTS.md: reuse common helpers).
|
Notes on what I checked and did not flag: PR size is ~440 added lines, under the 500 limit; the uct_v2.h re-alignment is required by the longer enum name, not unrelated refactoring. The Residual coverage gaps: the async path is only reachable with a hand-patched md_attr; nothing in the tree advertises UCT_MD_FLAG_CUDA_ASYNC_MEMTYPE_COPY or sets UCS_MEM_FLAG_CUDA_ASYNC (already reported in earlier rounds, not repeated). The latent cached-MD registration path described in the first comment is untestable today for the same reason, which is exactly why an assertion rather than a test is the right ask. |
|
🤖 CI Triage Agent — TL;DR: The only failure in the whole 8966-test run is Full analysisSummary: Root cause: From the log we know only the failing test identity (the per-assertion output is earlier in the step log than the ~500-line tail that is retrievable; the tail contains just the gtest summary). From the source,
The last path is directly affected by the newly merged atomic-MR-offset change: Implicated commit: e4d1e59 — "UCT/IB/MLX5: Fix atomic MR offset without a flush memory key (#12265)", dmitrygx, 2026-11-08 (merged 3 days before this run); not the PR under test (de686df). File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:598-621 ( Suggested fix:
Related: PR under test: #12157 · suspect merged change: #12265 · adjacent IB MD work: #12271 |
What?
Second of three PRs to enable
gdr_copyfor buffers allocated withcudaMallocAsync, following merged #12092.This PR adds the complete short-copy routing and registration path, but leaves it dormant in normal use. The third PR will connect allocation detection, runtime qualification and the enable/disable setting.
Why?
Stream-ordered CUDA allocations need a targeted route to GDRCopy's DMA-BUF mapping path without making other transports newly eligible or changing the buffer's outward memory type. Preparing the complete path before activating it keeps this PR safe to merge on its own.
How?
GDR_CAPABLEon that qualified route.Scope
No production allocation detector emits the new marker, and no MD advertises the new capability in this PR. The qualification helper from #12092 still has no production caller. There is no new configuration setting, global reclassification, or change to other transports' eligibility.
The path can be exercised by tests that inject the marker and capability, but it does not enable async GDRCopy for applications. No measured zero-overhead or latency/throughput claim is made.
Validation
cuda_async_short_routeandcuda_async_short_perf, executed and passed.59ca7e6d2e); 11 files, +452/-43.