Repository navigation
UCP/CORE: Implement GPU to NIC assignment logic - #11868
Conversation
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Scope note (not a blocker): the Test gap: no test exercises a GPU with no NIC (empty-NIC group / unassigned GPU), so the |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Compilation of Root cause: ucs_assertv((policy >= 0) && (policy < UCP_GPU_NIC_POLICY_LAST),
"invalid gpu-nic assignment policy %d", policy);makes Implicated commit: File: src/ucp/core/ucp_gpu_nic_assignment.c:145 Suggested fix: Remove the redundant lower-bound comparison, since the enum type cannot hold a negative value: ucs_assertv(policy < UCP_GPU_NIC_POLICY_LAST,
"invalid gpu-nic assignment policy %d", policy);If a defensive lower bound is desired for callers passing raw ints, cast explicitly instead (e.g. Related: PR #11868 (branch
|
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
| ASSERT_LE(config.num_gpu_devices, UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT); | ||
| ASSERT_LE(config.num_nic_ports, UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT); | ||
|
|
||
| /* Leave (UCS_SYS_DEVICE_ID_UNKNOWN - 1) unmapped for the ungrouped lookup. */ |
There was a problem hiding this comment.
minor: this line is 86 columns; ReflowComments: false in .clang-format means clang-format will not fix it for you.
Suggestion:
/* Leave (UCS_SYS_DEVICE_ID_UNKNOWN - 1) unmapped for the ungrouped
* lookup. */
| BEGIN_C_DECLS | ||
|
|
||
|
|
||
| #define UCP_GPU_NIC_BITMAP_INDEX_INVALID UCS_SYS_DEVICE_ID_COUNT |
There was a problem hiding this comment.
minor: can we add UCS_STATIC_ASSERT(UCP_GPU_NIC_BITMAP_INDEX_INVALID == UINT8_MAX) here? bitmap_idx_by_gpu_sys_dev is uint8_t and is filled with memset(..., UCP_GPU_NIC_BITMAP_INDEX_INVALID, ...), so the sentinel would silently stop working if UCS_SYS_DEVICE_ID_COUNT ever changes.
|
🤖 CI Triage Agent — TL;DR: The BlueField gtest job failed on a single test, Full analysisSummary: Root cause: Evidence from the log: Implicated commit: db208ee "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)" — Evgeny Leksikov (added File: test/gtest/ucp/test_ucp_fault_tolerance.cc:941 (test body 913-943) Suggested fix:
Related: PR #11868 (this PR, #11868); test/probe logic from #11563; prior FT test flakiness precedent: #11397 ("Disable failing fault-tolerance test"). |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
@tvegas1 @brminich Refactored gpu-nic assignment logging, example output: |
|
🤖 Starting review — findings will be posted here when done. |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gaps (not blocking; already raised in earlier rounds):
|
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "Create UCX source tarball" job failed in Full analysisSummary: Azure job Root cause: Evidence from the log:
Implicated commit: Not a code regression from this PR. The fragile pattern was introduced by d9b582e "UCT/GDA: Use gpunetio submodule (#11122)" (Artemy Kovalyov) and the error-tolerant clone by eae153e "BUILD: Log an error if GPUNetIO cannot be updated (#11378)" (Thomas Vegas). File: Suggested fix:
Related: none found (no matching issues/PRs in this repo for the gpunetio submodule / artifact upload failure). |
What?
Implement GPU to NIC assignment logic as part of the effort to allow more tailored assignments on certain types of hardware (e.g. Vera Rubin).
This implementation assumed exclusive assignments for now (only 1 GPU assigned to each NIC), and only
FLIPandROUND_ROBINpolicies.How?
The groups data structure is built outside of this module, and each group represents equally distant GPUs and NICs.
The assignment logic pairs every NIC with a GPU to spread out the bandwidth within the group.
The devices are assumed to be sorted by BDF within the arrays.