Skip to content

UCP/CORE: Implement GPU to NIC assignment logic - #11868

Merged
tvegas1 merged 51 commits into
openucx:masterfrom
guy-ealey-morag:topo-gpu-nic-assign
Sep 18, 2026
Merged

tvegas1 merged 51 commits into
openucx:masterfrom
guy-ealey-morag:topo-gpu-nic-assign

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

What?

Implement GPU to NIC assignment logic as part of the effort to allow more tailored assignments on certain types of hardware (e.g. Vera Rubin).
This implementation assumed exclusive assignments for now (only 1 GPU assigned to each NIC), and only FLIP and ROUND_ROBIN policies.

How?

The groups data structure is built outside of this module, and each group represents equally distant GPUs and NICs.
The assignment logic pairs every NIC with a GPU to spread out the bandwidth within the group.
The devices are assumed to be sorted by BDF within the arrays.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_gpu_nic_assignment.h Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Scope note (not a blocker): the ucp_gpu_nic_assignment_* API is not consumed anywhere in src/ yet; it is a standalone building block with its own gtest. That is fine for a foundational PR, but worth confirming it matches the intended scope.

Test gap: no test exercises a GPU with no NIC (empty-NIC group / unassigned GPU), so the lookup-returns-NULL path — the one broken by the sentinel bug — has no coverage. Adding a test where a GPU-only group leaves some GPU unassigned so lookup returns NULL would harden the API and catch this regression.

@svc-ucx

svc-ucx commented Sep 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_icc_pgi centos7_x86_64) · commit b00e9043

TL;DR: The build_icc_pgi centos7_x86_64 build failed compiling src/ucp/core/ucp_gpu_nic_assignment.c because NVIDIA's nvc treats the ucp_gpu_nic_policy_t enum as unsigned, so policy >= 0 in an assertion triggers error #186 ("pointless comparison of unsigned integer with zero") under -Werror; drop the >= 0 half of the assertion.

Full analysis

Summary: Compilation of core/ucp_gpu_nic_assignment.c with nvc (HPC SDK 21.2) aborted with error #186: pointless comparison of unsigned integer with zero, failing make (make[2]: *** [core/libucp_la-ucp_gpu_nic_assignment.lo] Error 1).

Root cause: ucp_gpu_nic_policy_t (src/ucp/core/ucp_gpu_nic_assignment.h:36-40) contains no negative enumerators, so nvc gives it an unsigned underlying type. The assertion at line 145 of ucp_gpu_nic_assignment.c:

ucs_assertv((policy >= 0) && (policy < UCP_GPU_NIC_POLICY_LAST),
            "invalid gpu-nic assignment policy %d", policy);

makes policy >= 0 tautologically true. nvc diagnoses this as #186; the build uses -Wall -Werror --display_error_number and the suppression list in contrib/configure-* (--diag_suppress 1 68 111 167 181 188 381 1215 1626 1901 1902) does not include 186, so it becomes a hard error. GCC/Clang jobs pass because -Wtype-limits/-Wtautological-compare isn't fatal there, so this is compiler-specific and only shows up in the ICC/PGI pipeline.

Implicated commit: 218a96ad — "UCP/CORE: Add GPU to NIC assignment logic based on groups", Guy Ealey Morag (file introduced; [REDACTED:Hex High Entropy String] by the same author is the follow-up on this file)

File: src/ucp/core/ucp_gpu_nic_assignment.c:145

Suggested fix: Remove the redundant lower-bound comparison, since the enum type cannot hold a negative value:

ucs_assertv(policy < UCP_GPU_NIC_POLICY_LAST,
            "invalid gpu-nic assignment policy %d", policy);

If a defensive lower bound is desired for callers passing raw ints, cast explicitly instead (e.g. ucs_assertv((unsigned)policy < UCP_GPU_NIC_POLICY_LAST, ...)), which is portable across gcc/clang/icc/nvc. Also consider changing the %d format specifier accordingly. Prefer this over adding --diag_suppress 186 to contrib/configure-*, which would mask genuine tautological comparisons elsewhere.

Related: PR #11868 (branch topo-gpu-nic-assign); no pre-existing issue found for error #186 in this repo.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id d0fa34a4-d8c1-44ac-bbf4-3e3c7e29108e in the triage console for the audit trail.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
ASSERT_LE(config.num_gpu_devices, UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT);
ASSERT_LE(config.num_nic_ports, UCS_TOPO_MAX_SYS_DEVS_PER_ELEMENT);

/* Leave (UCS_SYS_DEVICE_ID_UNKNOWN - 1) unmapped for the ungrouped lookup. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: this line is 86 columns; ReflowComments: false in .clang-format means clang-format will not fix it for you.

Suggestion:

        /* Leave (UCS_SYS_DEVICE_ID_UNKNOWN - 1) unmapped for the ungrouped
         * lookup. */

BEGIN_C_DECLS


#define UCP_GPU_NIC_BITMAP_INDEX_INVALID UCS_SYS_DEVICE_ID_COUNT

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: can we add UCS_STATIC_ASSERT(UCP_GPU_NIC_BITMAP_INDEX_INVALID == UINT8_MAX) here? bitmap_idx_by_gpu_sys_dev is uint8_t and is filled with memset(..., UCP_GPU_NIC_BITMAP_INDEX_INVALID, ...), so the sentinel would silently stop working if UCS_SYS_DEVICE_ID_COUNT ever changes.

@svc-ucx

svc-ucx commented Sep 17, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 0) · commit a894ca7d

TL;DR: The BlueField gtest job failed on a single test, rcx/test_ucp_fault_tolerance.probe_gated_recovery/6 (rc_x/AM), whose EXPECT_TRUE(probe_armed) check depends on polling and catching a transient internal recovery state — it is a racy/flaky assertion unrelated to this PR's GPU→NIC assignment logic (the node has no CUDA). Fix by making the probe-arming observation deterministic (mock/count ep_check calls or use a sticky counter) instead of sampling recovery_arg->probe[].comp.func.

Full analysis

Summary: make test in build-test/test/gtest exited with error because exactly 1 of 8681 gtest cases failed: rcx/test_ucp_fault_tolerance.probe_gated_recovery/6, where GetParam() = rc_x/AM.

Root cause: Evidence from the log: [ PASSED ] 8680 tests, [ FAILED ] 1 test ... probe_gated_recovery/6, then make: *** [Makefile:4713: test] Error 1 / Bash exited with code '2'. (The per-test assertion text is no longer in the retained log tail — the test ran early in a ~3046 s run — but the test body has only one test-local check.) In test/gtest/ucp/test_ucp_fault_tolerance.cc:913-943, after test_am_with_injected_failure(FAILURE_SIDE_TARGET, TEST_OP_AM), the test polls with wait_for_cond() and sets probe_armed only if a poll iteration happens to observe ep->ext->recovery_arg != NULL and arg->probe[lane].comp.func != NULL; the loop terminates as soon as ucp_ep_get_failed_lanes(ep) == 0. Since the aux probe (uct_ep_check) can be issued and completed — and recovery_arg freed (it is a union member with close_req, cleared once recovery reaches UCP_EP_RECOVERY_STATE_PROBE_OK, see ucp_ep_recovery_arg_t in src/ucp/core/ucp_ep.h:502-528) — within a single short_progress_loop(), the transient state can be missed entirely and EXPECT_TRUE(probe_armed) fires with "RC p2p lane recovery completed without arming an aux probe". Nothing in the log points at the PR's topology/GPU-NIC code: this BlueField ARM worker has no GPU (tcp/test_ucp_perf_cuda.am_recv_copy/0 [SKIP] (CUDA is not supported)), so the GPU→NIC assignment path of PR #11868 is not exercised there. There are also no hangs/timeouts — timestamps show continuous progress up to the summary.

Implicated commit: db208ee "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)" — Evgeny Leksikov (added probe_gated_recovery and the recovery probe state machine); not commit a894ca7 of this PR.

File: test/gtest/ucp/test_ucp_fault_tolerance.cc:941 (test body 913-943)

Suggested fix:

  1. Re-run the BlueField job to confirm the flake (a single unrelated FT test failing on a GPU-less worker).
  2. Make the probe-arming check deterministic instead of state sampling, e.g.:
    • Wrap iface->ops.ep_check with ucs::mock (as teardown_with_outstanding_probe/recovery_retries_exhausted_live_lanes already do) and count invocations in the mock, asserting count > 0 after recovery; or
    • Call the existing helper wait_for_recovery_probe_in_flight(ep, deadline) and assert on it before waiting for failed_lanes == 0; or
    • Add a sticky counter (e.g. probes_issued in ucp_ep_recovery_arg_t / an EP-level stat) that is not cleared when the probe completes, and assert on that.
  3. Until then, consider marking probe_gated_recovery as non-blocking/flaky so an unrelated timing race does not fail unrelated PR CI.

Related: PR #11868 (this PR, #11868); test/probe logic from #11563; prior FT test flakiness precedent: #11397 ("Disable failing fault-tolerance test").

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@guy-ealey-morag

guy-ealey-morag commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor Author

@tvegas1 @brminich Refactored gpu-nic assignment logging, example output:

gpu GPU0 (bdf 0000:01:00.0 sys_dev 0) is assigned 2 nics: [mlx5_0/mlx5_1, mlx5_2/mlx5_3] sys_devs [2/3, 4/5]
gpu GPU1 (bdf 0000:41:00.0 sys_dev 1) is assigned 2 nics: [mlx5_4/mlx5_5, mlx5_6/mlx5_7] sys_devs [6/7, 8/9]
gpu GPU2 (bdf 0000:81:00.0 sys_dev 14) is assigned 0 nics
2 nics are unassigned: [mlx5_8/mlx5_9, mlx5_10/mlx5_11] sys_devs [10/11, 12/13]

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps (not blocking; already raised in earlier rounds):

  • No consumer in src/ yet — ucs_topo_groups_build_groups() is still a TODO, so there is no end-to-end coverage on a real topology.
  • UCS_ERR_EXCEEDS_LIMIT / UCS_ERR_NO_MEMORY paths untested; the limit path is in fact unreachable (≤255 sys devs, ≥1 per GPU element).
  • Only uniform group shapes are exercised; no single ucs_topo_groups_t mixing a GPU-only group with a GPU+NIC group.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_gpu_nic_assignment.h Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_gpu_nic_assignment.h Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Sep 17, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Cuda Create UCX source tarball) · commit 82e35f54

TL;DR: The "Create UCX source tarball" job failed in PublishBuildArtifacts because autogen.sh couldn't clone the gpunetio submodule (transient Empty reply from server from github.com), leaving the checked-in symlink src/uct/ib/mlx5/gdaki/gpunetio -> external/gpunetio dangling, and the Azure artifact uploader aborts with FileNotFoundException on broken symlinks. Fix by retrying the submodule clone in autogen.sh and pruning dangling symlinks (or tarring the tree) before publishing.

Full analysis

Summary: Azure job Create UCX source tarball (build 136576) failed at the PublishBuildArtifacts step with ##[error]File upload failed even after retry. — unrelated to the PR's code changes.

Root cause: Evidence from the log:

  • 13:56:12 ./autogen.sh → Cloning into 'external/gpunetio'... fatal: unable to access 'https://github.com/NVIDIA-DOCA/gpunetio/': Empty reply from server / WARNING: Failed to update GPUNetIO submodule, continuing... — a transient network/registry failure. autogen.sh:28-29 deliberately swallows this error, so the build continued with an empty external/gpunetio directory.
  • 14:00:01 and again on retry at 14:01:18: Fail to upload '/scrap/.../s/src/uct/ib/mlx5/gdaki/gpunetio' due to 'Could not find file ...' → System.IO.FileNotFoundException.

src/uct/ib/mlx5/gdaki/gpunetio is a git-tracked symlink into the submodule path external/gpunetio (see .gitmodules, which maps submodule name src/uct/ib/mlx5/gdaki/gpunetio to path external/gpunetio). When the submodule clone fails the symlink target doesn't exist, and PublishBuildArtifacts@1 with pathToPublish: $(Build.Repository.LocalPath) tries to open every file it walks — including the dangling symlink — and hard-fails the whole 1628-file upload. Note the job also has no retryCountOnTaskFailure on the publish task, and the retry re-hits the same missing file, so it can never recover.

Implicated commit: Not a code regression from this PR. The fragile pattern was introduced by d9b582e "UCT/GDA: Use gpunetio submodule (#11122)" (Artemy Kovalyov) and the error-tolerant clone by eae153e "BUILD: Log an error if GPUNetIO cannot be updated (#11378)" (Thomas Vegas).

File: autogen.sh:28-29 (tolerated submodule clone failure) and buildlib/pr/cuda/cuda.yml:21-24 (publishes raw workspace containing the dangling symlink)

Suggested fix:

  1. Retry the submodule clone in autogen.sh instead of giving up on the first transient failure:
    for i in 1 2 3; do
        git submodule update --init external/gpunetio && break
        echo "WARNING: GPUNetIO submodule update failed (attempt $i), retrying..."
        sleep 10
    done
  2. Make the publish step immune to a missing submodule — in buildlib/pr/cuda/cuda.yml, add a cleanup step before publishing and a retry on the task:
    - bash: find . -xtype l -print -delete   # drop dangling symlinks (e.g. gdaki/gpunetio)
      displayName: Prune broken symlinks
    - task: PublishBuildArtifacts@1
      retryCountOnTaskFailure: 3
      inputs: ...
    Better still, publish an actual tarball (tar czf ucx_src.tar.gz --exclude=.git .) rather than the raw 1628-file tree — that also removes the ~4 minutes spent uploading the .git pack file.
  3. Re-run the failed job; the underlying GitHub fetch error is transient.

Related: none found (no matching issues/PRs in this repo for the gpunetio submodule / artifact upload failure).

Comment thread src/ucp/core/ucp_gpu_nic_assignment.h Resolved
Comment thread src/ucp/core/ucp_gpu_nic_assignment.c Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants