Skip to content

UCP/WIREUP: Select device lanes for CUDA memory only - #12281

Draft
amastbaum wants to merge 5 commits into
openucx:masterfrom
amastbaum:device_api_cuda_ipc_lanes
Draft

amastbaum wants to merge 5 commits into
openucx:masterfrom
amastbaum:device_api_cuda_ipc_lanes

Conversation

@amastbaum

Copy link
Copy Markdown
Contributor

What?

Select device lanes for CUDA memory only: drop the CUDA|HOST and ROCM iterations in ucp_wireup_add_device_lanes.

Why?

The extra passes were needed when we couldn't rely on the cuda_ipc reachability check. They let wireup fall back to a non-cuda_ipc lane at a later stage if cuda_ipc turned out to be unusable. Now that cuda_ipc correctly reports unreachable peers (e.g. a different NVLink domain), that fallback is no longer needed.

How?

mem_type_bitmaps now holds only UCS_BIT(UCS_MEMORY_TYPE_CUDA). Tested NIXL EP elastic.py on 2 GB300 nodes. Same NVLink domain: device(cuda_ipc/cuda). Different domains: device(rc_gda/cuda0-mlx5_*). Both pass.

@amastbaum
amastbaum requested a review from a team as a code owner November 9, 2026 09:17
@amastbaum
amastbaum requested a review from ofirfarjun7 November 9, 2026 09:18
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/wireup/select.c Outdated
UCS_BIT(UCS_MEMORY_TYPE_CUDA) |
UCS_BIT(UCS_MEMORY_TYPE_HOST),
UCS_BIT(UCS_MEMORY_TYPE_ROCM)};
const uint64_t mem_type_bitmaps[] = {UCS_BIT(UCS_MEMORY_TYPE_CUDA)};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is ROCM removed? rocm_ipc advertises UCT_IFACE_FLAG_DEVICE_EP and implements ep_get_device_ep, and its MD mem types are ROCM only, so with CUDA alone in the list it never passes ucs_test_all_flags() and device lanes are no longer selected for ROCm. There is no UCP-level ROCm device test, so CI would not show this.

Comment thread src/ucp/wireup/select.c Outdated
UCS_BIT(UCS_MEMORY_TYPE_CUDA) |
UCS_BIT(UCS_MEMORY_TYPE_HOST),
UCS_BIT(UCS_MEMORY_TYPE_ROCM)};
const uint64_t mem_type_bitmaps[] = {UCS_BIT(UCS_MEMORY_TYPE_CUDA)};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: if the list stays with a single memory type, maybe drop the array, the loop and i, and call ucp_wireup_memaccess_bitmap(context, UCS_BIT(UCS_MEMORY_TYPE_CUDA), ...) once?

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 491a35a0

TL;DR: test_ucp_am_nbx_eager_data_release.multi/0 <rc_x> never received its 2‑fragment eager AM (10 s wait_receives timeout, m_recv_counter=0 vs m_send_counter=1), and the test then called ucp_am_data_release() on a still‑NULL m_data_ptr, which segfaulted and killed the whole gtest shard. Fix the test to bail out instead of releasing a NULL descriptor, and investigate the multi‑fragment AM reassembly/PSN‑drop path that swallowed the message.

Full analysis

Summary: Azure job "roce on worker 0" failed with make: *** [Makefile:4992: test] Segmentation fault (core dumped) during gtest shard 1/4; the crash came from rcx/test_ucp_am_nbx_eager_data_release.multi/0.

Root cause:

  • At 10:06:33 the first test of the shard starts; at 10:06:43.59 (≈10.2 s later, exactly the wait_for_value / DEFAULT_TIMEOUT_SEC default in test/gtest/ucp/ucp_test.h:375) it reports test_ucp_am.cc:594: Failure — m_recv_counter (0) != m_send_counter (1). So test_am_send_recv(fragment_size()*2, 0, 0, UCP_AM_FLAG_PERSISTENT_DATA) sent the message but the receiver's AM callback was never invoked — the multi-fragment eager AM (UCP_AM_ID_AM_FIRST / UCP_AM_ID_AM_MIDDLE) was never reassembled and delivered on rc_x. Both silent-drop paths live in src/ucp/core/ucp_am.c: the interval-tree completeness check in ucp_am_handle_unfinished() (lines 1455–1472, returns silently while "not all fragments arrived yet") and ucp_am_is_duplicate_psn() (line 1758) which drops fragments with only a ucs_debug message. Neither produces an error, which matches the observed silent 10 s stall.
  • The segfault that actually failed the build is a secondary test bug: test_data_release() (test/gtest/ucp/test_ucp_am.cc:1598) calls ucp_am_data_release(receiver().worker(), m_data_ptr) unconditionally. Because the callback never ran, m_data_ptr is still nullptr, and ucp_am.c:173 computes rdesc = (ucp_recv_desc_t*)NULL - 1 and reads rdesc->flags — exactly the faulting address reported, 0xfffffffffffffff0 (-16 = -sizeof(ucp_recv_desc_t)).
  • The 10:06:43 → 10:08:52 gap in the log is just the UCS signal handler resolving the backtrace after the SIGSEGV, not a hang; no time-limit change is warranted.

Implicated commit: unknown for the delivery stall (PR head under test: 491a35a, branch device_api_cuda_ipc_lanes). Prime suspects in the AM multi-fragment/PSN path: ef265cdc (shasson5, "UCP/PROTO: Introduce Multi Eager PSN Protocol" #11160) and 2bafbb02 (Evgeny Leksikov, "UCP/AM/FT: introduce resend flag for first fragment retransmissions" #11452); the NULL-release crash path is pre-existing test code.

File: test/gtest/ucp/test_ucp_am.cc:1598 (crash trigger) and src/ucp/core/ucp_am.c:173 / src/ucp/core/ucp_am.c:1455-1472, src/ucp/core/ucp_am.c:1758 (missing delivery)

Suggested fix:

  1. Harden the test so a lost message produces a readable failure instead of a core dump — in test_data_release():
    test_am_send_recv(size, 0, 0, UCP_AM_FLAG_PERSISTENT_DATA);
    ASSERT_NE(nullptr, m_data_ptr);   // or: if (m_data_ptr == nullptr) return;
    ucp_am_data_release(receiver().worker(), m_data_ptr);
    (same for the second send at line 1601), and reset m_data_ptr = nullptr in reset_counters()/before each send.
  2. Optionally add a defensive if (data == NULL) return; in ucp_am_data_release() so a user NULL pointer cannot crash the library.
  3. Re-run the rcx/test_ucp_am_nbx_eager_data_release.* family with UCX_LOG_LEVEL=trace_data (and --gtest_filter=*eager_data_release* on rc_x, repeated) to determine whether a fragment was dropped by ucp_am_is_duplicate_psn() or whether the interval tree never reports the full [payload_offset, payload_offset+total_size-1] range; consider upgrading that debug drop message to a warning so such losses are visible in CI logs.

Related: PR #12281 (build under test); no existing issue found for this signature.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Outdated
ucs_offsetof(ucp_context_config_t, multi_path_ratio),
UCS_CONFIG_TYPE_POS_DOUBLE},

{"DEVICE_MEM_TYPES", "cuda",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default keeps only the cuda set, while today lanes are also selected for cuda|host and rocm - this silently drops ROCm device lanes and the host+cuda-capable transports. Was that intended?

Suggested change
{"DEVICE_MEM_TYPES", "cuda",
{"DEVICE_MEM_TYPES", "cuda,cuda|host,rocm",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure whether we need cuda|host, but removing rocm does not seem correct indeed, since we did add support for some device APIs in rocm_ipc.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I reverted rocm lanes removal.
Originally it was a POC to allocate the minimum resources during the connection establishment, but I changed the way it's configured.

Comment thread src/ucp/core/ucp_context.c Outdated

ucs_string_buffer_rtrim(&strb, "|");
return 1;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: this is what ucs_string_buffer_append_flags() already does.

Suggested change
}
static int ucp_config_sprintf_mem_type_set(char *buf, size_t max,
const void *src, const void *arg)
{
ucs_string_buffer_t strb;
ucs_string_buffer_init_fixed(&strb, buf, max);
ucs_string_buffer_append_flags(&strb, *(const uint64_t*)src,
ucs_memory_type_names);
return 1;
}

Comment thread src/ucp/core/ucp_context.c Outdated
UCS_CONFIG_TYPE_MEMUNITS);

/* Parse a '|'-separated set of memory types into a bitmap */
static int ucp_config_sscanf_mem_type_set(const char *buf, void *dest,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates ucs_config_sscanf_bitmap(), only with '|' instead of ','. Why not add a delimiter-aware _impl in ucs/config/parser.c (like ucs_config_sscanf_array_impl) and reuse it here?

Comment thread src/ucp/core/ucp_context.c Outdated
return 1;
}

static void ucp_config_help_mem_type_set(char *buf, size_t max, const void *arg)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: pls list the allowed names like ucs_config_help_bitmap does, otherwise the error for a typo prints "Expected: comma-separated list of: '|'-separated set of memory types" without showing what is accepted.

Comment thread src/ucp/core/ucp_context.c Outdated
goto err_free_config_ext;
}

if (context->config.ext.device_mem_types.count == 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why fail ucp_init() here? An empty DEVICE_MEM_TYPES could simply mean "select no device lanes"; as written there is no way to disable them, and the failure also breaks contexts that do not use UCP_FEATURE_DEVICE.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Error-path check: the new goto err_free_env_prefix is correct at that point — context->config.env_prefix is allocated just above (base ucp_context.c:2700), and err_free_env_prefix falls through to err_free_config_ext.

unsigned i matches UCS_CONFIG_ARRAY_FIELD's unsigned count, and rejecting mem_type >= UCS_MEMORY_TYPE_LAST correctly filters the "unknown" entry in ucs_memory_type_names.

ucs_config_clone_ulong on a sizeof(uint64_t) element is a type mismatch, but harmless in practice (array data is ucs_calloc'd and all memory-type bits are < 32), and it matches what UCS_CONFIG_TYPE_BITMAP already does — not raised.

Coverage gap worth noting to the author if they want: no test sets UCX_DEVICE_MEM_TYPES, so the new parser (sets, invalid names, empty value) and the lane-selection change are only covered indirectly by test/gtest/ucp/test_ucp_device.cc running with the default.

@svc-ucx

svc-ucx commented Nov 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 6233fb3a

TL;DR: The Coverity release stage failed on one new OVERRUN defect: the new ucp_config_sprintf_mem_type_set() iterates all 64 bits of a uint64_t memory-type bitmap and indexes the 12-element ucs_memory_type_names[] with it. Mask the bitmap with UCS_MASK(UCS_MEMORY_TYPE_LAST) (and/or assert) before the ucs_for_each_bit loop.

Full analysis

Summary: coverity release on coverity_rh7 failed with ##[error]Coverity found 1 issues: /src/ucp/core/ucp_context.c:221 Type: Out-of-bounds read (OVERRUN) — the build and cov-build (559 units, 100%) succeeded; only the defect gate failed.

Root cause: ucp_config_sprintf_mem_type_set() does ucs_for_each_bit(mem_type, *(const uint64_t*)src), so Coverity derives mem_type may be up to 63 ("cond_at_most: Checking mem_type < 64U implies that mem_type may be up to 63"). It is then used to index ucs_memory_type_names, which has only UCS_MEMORY_TYPE_LAST + 1 = 12 8-byte elements → "Overrunning array ucs_memory_type_names of 12 8-byte elements at element index 63 (byte offset 511)". Nothing in that function constrains the bitmap to valid memory-type bits; the matching parser ucp_config_sscanf_mem_type_set() validates mem_type < UCS_MEMORY_TYPE_LAST (line 201), but the sprintf path has no equivalent bound, and the config value can also be cloned/round-tripped from any uint64_t.

Implicated commit: [REDACTED:Hex High Entropy String] — Alma Mastbaum, "UCP/WIREUP: Make device lane memory types configurable" (the PR head; it introduced the mem_type_sets config array and both sscanf/sprintf callbacks)

File: src/ucp/core/ucp_context.c:220-222 (defect reported at line 221)

Suggested fix: Bound the iteration to valid memory types so Coverity can prove the index is in range:

static int ucp_config_sprintf_mem_type_set(char *buf, size_t max,
                                           const void *src, const void *arg)
{
    ucs_string_buffer_t strb;
    ucs_memory_type_t mem_type;

    ucs_string_buffer_init_fixed(&strb, buf, max);
    ucs_for_each_bit(mem_type, *(const uint64_t*)src &
                               UCS_MASK(UCS_MEMORY_TYPE_LAST)) {
        ucs_assert(mem_type < UCS_MEMORY_TYPE_LAST);
        ucs_string_buffer_appendf(&strb, "%s|",
                                  ucs_memory_type_names[mem_type]);
    }

    ucs_string_buffer_rtrim(&strb, "|");
    return 1;
}

The UCS_MASK(UCS_MEMORY_TYPE_LAST) term is what silences the checker; the assert alone may not be enough under release (NDEBUG) analysis. Consider applying the same mask in ucs_config_clone_ulong's consumers for this field if the value can arrive from outside the sscanf path.

Related: PR #12281 (this PR); similar pattern previously touched in #11902.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 6563e667-ffe4-42e7-a4a9-b99b2f246c74 in the triage console for the audit trail.

Comment thread src/ucp/wireup/select.c Outdated
for (i = 0; i < ucs_static_array_size(mem_type_bitmaps); ++i) {
ucp_wireup_memaccess_bitmap(context, mem_type_bitmaps[i],
&mem_type_tl_bitmap);
for (i = 0; i < context->config.ext.device_mem_types.count; ++i) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The extra passes were needed when we couldn't rely on the cuda_ipc reachability check. They let wireup fall back to a non-cuda_ipc lane at a later stage if cuda_ipc turned out to be unusable. Now that cuda_ipc correctly reports unreachable peers (e.g. a different NVLink domain), that fallback is no longer needed.

I think we still need the fallback if we are in the same NVLink domain but cannot use MNNVL: e.g., the imex channel has not been created or is unavailable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I changed the configuration to create the fallback lanes by default.
This is a POC (I changed it to draft) to enable the allocation of minimum resources during wireup.

@amastbaum
amastbaum marked this pull request as draft November 10, 2026 14:47
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_context.c Outdated
UCS_CONFIG_TYPE_POS_DOUBLE},

{"DEVICE_LANES_FALLBACK", "y",
"Select fallback device API lanes from the transports which can access both\n"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: first line is 81 columns, over the 80-column limit (docs/CodeStyle.md), so the clang-format CI job may re-wrap it.

Suggested change
"Select fallback device API lanes from the transports which can access both\n"
"Select fallback device API lanes from the transports which can access\n"
"both CUDA and host memory, in addition to the lanes selected for CUDA or\n"
"ROCm memory only. The fallback lanes are used when the remote memory is\n"
"not reachable by the other lanes. If disabled, fewer resources are\n"
"allocated during connection establishment, but no fallback is available.",

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

"both CUDA and host memory, in addition to the lanes selected for CUDA or\n"
"ROCm memory only. The fallback lanes are used when the remote memory is\n"
"not reachable by the other lanes. If disabled, fewer resources are\n"
"allocated during connection establishment, but no fallback is available.",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: transports which support both CUDA and host memory are also eligible in the CUDA-only pass, so a fallback lane may still be selected when this is disabled - it is just not guaranteed.

Suggested change
"allocated during connection establishment, but no fallback is available.",
"not reachable by the other lanes. If disabled, fewer resources are\n"
"allocated during connection establishment, but a fallback is not\n"
"guaranteed.",

ucs_offsetof(ucp_context_config_t, multi_path_ratio),
UCS_CONFIG_TYPE_POS_DOUBLE},

{"DEVICE_LANES_FALLBACK", "y",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOST_DEVICE_LANES
Set default to "n".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants