Repository navigation
Conversation
|
🤖 Starting review — findings will be posted here when done. |
| UCS_BIT(UCS_MEMORY_TYPE_CUDA) | | ||
| UCS_BIT(UCS_MEMORY_TYPE_HOST), | ||
| UCS_BIT(UCS_MEMORY_TYPE_ROCM)}; | ||
| const uint64_t mem_type_bitmaps[] = {UCS_BIT(UCS_MEMORY_TYPE_CUDA)}; |
There was a problem hiding this comment.
why is ROCM removed? rocm_ipc advertises UCT_IFACE_FLAG_DEVICE_EP and implements ep_get_device_ep, and its MD mem types are ROCM only, so with CUDA alone in the list it never passes ucs_test_all_flags() and device lanes are no longer selected for ROCm. There is no UCP-level ROCm device test, so CI would not show this.
| UCS_BIT(UCS_MEMORY_TYPE_CUDA) | | ||
| UCS_BIT(UCS_MEMORY_TYPE_HOST), | ||
| UCS_BIT(UCS_MEMORY_TYPE_ROCM)}; | ||
| const uint64_t mem_type_bitmaps[] = {UCS_BIT(UCS_MEMORY_TYPE_CUDA)}; |
There was a problem hiding this comment.
minor: if the list stays with a single memory type, maybe drop the array, the loop and i, and call ucp_wireup_memaccess_bitmap(context, UCS_BIT(UCS_MEMORY_TYPE_CUDA), ...) once?
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: Azure job "roce on worker 0" failed with Root cause:
Implicated commit: unknown for the delivery stall (PR head under test: 491a35a, branch File: Suggested fix:
Related: PR #12281 (build under test); no existing issue found for this signature. |
|
🤖 Starting review — findings will be posted here when done. |
| ucs_offsetof(ucp_context_config_t, multi_path_ratio), | ||
| UCS_CONFIG_TYPE_POS_DOUBLE}, | ||
|
|
||
| {"DEVICE_MEM_TYPES", "cuda", |
There was a problem hiding this comment.
The default keeps only the cuda set, while today lanes are also selected for cuda|host and rocm - this silently drops ROCm device lanes and the host+cuda-capable transports. Was that intended?
| {"DEVICE_MEM_TYPES", "cuda", | |
| {"DEVICE_MEM_TYPES", "cuda,cuda|host,rocm", |
There was a problem hiding this comment.
I am not sure whether we need cuda|host, but removing rocm does not seem correct indeed, since we did add support for some device APIs in rocm_ipc.
There was a problem hiding this comment.
Agreed, I reverted rocm lanes removal.
Originally it was a POC to allocate the minimum resources during the connection establishment, but I changed the way it's configured.
|
|
||
| ucs_string_buffer_rtrim(&strb, "|"); | ||
| return 1; | ||
| } |
There was a problem hiding this comment.
Minor: this is what ucs_string_buffer_append_flags() already does.
| } | |
| static int ucp_config_sprintf_mem_type_set(char *buf, size_t max, | |
| const void *src, const void *arg) | |
| { | |
| ucs_string_buffer_t strb; | |
| ucs_string_buffer_init_fixed(&strb, buf, max); | |
| ucs_string_buffer_append_flags(&strb, *(const uint64_t*)src, | |
| ucs_memory_type_names); | |
| return 1; | |
| } |
| UCS_CONFIG_TYPE_MEMUNITS); | ||
|
|
||
| /* Parse a '|'-separated set of memory types into a bitmap */ | ||
| static int ucp_config_sscanf_mem_type_set(const char *buf, void *dest, |
There was a problem hiding this comment.
This duplicates ucs_config_sscanf_bitmap(), only with '|' instead of ','. Why not add a delimiter-aware _impl in ucs/config/parser.c (like ucs_config_sscanf_array_impl) and reuse it here?
| return 1; | ||
| } | ||
|
|
||
| static void ucp_config_help_mem_type_set(char *buf, size_t max, const void *arg) |
There was a problem hiding this comment.
Minor: pls list the allowed names like ucs_config_help_bitmap does, otherwise the error for a typo prints "Expected: comma-separated list of: '|'-separated set of memory types" without showing what is accepted.
| goto err_free_config_ext; | ||
| } | ||
|
|
||
| if (context->config.ext.device_mem_types.count == 0) { |
There was a problem hiding this comment.
Why fail ucp_init() here? An empty DEVICE_MEM_TYPES could simply mean "select no device lanes"; as written there is no way to disable them, and the failure also breaks contexts that do not use UCP_FEATURE_DEVICE.
|
Error-path check: the new
Coverage gap worth noting to the author if they want: no test sets |
|
🤖 CI Triage Agent — TL;DR: The Coverity Full analysisSummary: Root cause: Implicated commit: File: Suggested fix: Bound the iteration to valid memory types so Coverity can prove the index is in range: static int ucp_config_sprintf_mem_type_set(char *buf, size_t max,
const void *src, const void *arg)
{
ucs_string_buffer_t strb;
ucs_memory_type_t mem_type;
ucs_string_buffer_init_fixed(&strb, buf, max);
ucs_for_each_bit(mem_type, *(const uint64_t*)src &
UCS_MASK(UCS_MEMORY_TYPE_LAST)) {
ucs_assert(mem_type < UCS_MEMORY_TYPE_LAST);
ucs_string_buffer_appendf(&strb, "%s|",
ucs_memory_type_names[mem_type]);
}
ucs_string_buffer_rtrim(&strb, "|");
return 1;
}The Related: PR #12281 (this PR); similar pattern previously touched in #11902.
|
| for (i = 0; i < ucs_static_array_size(mem_type_bitmaps); ++i) { | ||
| ucp_wireup_memaccess_bitmap(context, mem_type_bitmaps[i], | ||
| &mem_type_tl_bitmap); | ||
| for (i = 0; i < context->config.ext.device_mem_types.count; ++i) { |
There was a problem hiding this comment.
The extra passes were needed when we couldn't rely on the cuda_ipc reachability check. They let wireup fall back to a non-cuda_ipc lane at a later stage if cuda_ipc turned out to be unusable. Now that cuda_ipc correctly reports unreachable peers (e.g. a different NVLink domain), that fallback is no longer needed.
I think we still need the fallback if we are in the same NVLink domain but cannot use MNNVL: e.g., the imex channel has not been created or is unavailable.
There was a problem hiding this comment.
You're right, I changed the configuration to create the fallback lanes by default.
This is a POC (I changed it to draft) to enable the allocation of minimum resources during wireup.
|
🤖 Starting review — findings will be posted here when done. |
| UCS_CONFIG_TYPE_POS_DOUBLE}, | ||
|
|
||
| {"DEVICE_LANES_FALLBACK", "y", | ||
| "Select fallback device API lanes from the transports which can access both\n" |
There was a problem hiding this comment.
minor: first line is 81 columns, over the 80-column limit (docs/CodeStyle.md), so the clang-format CI job may re-wrap it.
| "Select fallback device API lanes from the transports which can access both\n" | |
| "Select fallback device API lanes from the transports which can access\n" | |
| "both CUDA and host memory, in addition to the lanes selected for CUDA or\n" | |
| "ROCm memory only. The fallback lanes are used when the remote memory is\n" | |
| "not reachable by the other lanes. If disabled, fewer resources are\n" | |
| "allocated during connection establishment, but no fallback is available.", |
|
🤖 Starting review — findings will be posted here when done. |
| "both CUDA and host memory, in addition to the lanes selected for CUDA or\n" | ||
| "ROCm memory only. The fallback lanes are used when the remote memory is\n" | ||
| "not reachable by the other lanes. If disabled, fewer resources are\n" | ||
| "allocated during connection establishment, but no fallback is available.", |
There was a problem hiding this comment.
minor: transports which support both CUDA and host memory are also eligible in the CUDA-only pass, so a fallback lane may still be selected when this is disabled - it is just not guaranteed.
| "allocated during connection establishment, but no fallback is available.", | |
| "not reachable by the other lanes. If disabled, fewer resources are\n" | |
| "allocated during connection establishment, but a fallback is not\n" | |
| "guaranteed.", |
| ucs_offsetof(ucp_context_config_t, multi_path_ratio), | ||
| UCS_CONFIG_TYPE_POS_DOUBLE}, | ||
|
|
||
| {"DEVICE_LANES_FALLBACK", "y", |
There was a problem hiding this comment.
HOST_DEVICE_LANES
Set default to "n".
What?
Select device lanes for CUDA memory only: drop the
CUDA|HOSTandROCMiterations inucp_wireup_add_device_lanes.Why?
The extra passes were needed when we couldn't rely on the cuda_ipc reachability check. They let wireup fall back to a non-cuda_ipc lane at a later stage if cuda_ipc turned out to be unusable. Now that cuda_ipc correctly reports unreachable peers (e.g. a different NVLink domain), that fallback is no longer needed.
How?
mem_type_bitmapsnow holds onlyUCS_BIT(UCS_MEMORY_TYPE_CUDA). Tested NIXL EPelastic.pyon 2 GB300 nodes. Same NVLink domain:device(cuda_ipc/cuda). Different domains:device(rc_gda/cuda0-mlx5_*). Both pass.