Skip to content

UCP/CORE: Support up to 255 TL resources - #12054

Merged
guy-ealey-morag merged 8 commits into
openucx:masterfrom
guy-ealey-morag:topic/ucp-max-resources-255
Oct 14, 2026
Merged

guy-ealey-morag merged 8 commits into
openucx:masterfrom
guy-ealey-morag:topic/ucp-max-resources-255

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What?

  • Raise UCP_MAX_RESOURCES from 128 to 255.
  • The TL bitmap grows from 2 to 4 words
  • ucp_rsc_index_t stays 8 bits (UCP_NULL_RESOURCE remains 255).

Why?

Systems with many NICs/GPUs exceed 128 transport resources, and UCP context creation fails with "exceeded transports/devices limit" with the default UCX_TLS=all. Increasing the limits allows running on these systems with the default UCX_TLS.

How?

  • Iterate TL bitmaps with a new UCP_TL_BITMAP_FOR_EACH_BIT, built on ucp_tl_bitmap_ffs(), which caps the search at UCP_MAX_RESOURCES.
  • ucp_tl_bitmap_max sets exactly the valid indices 0-254.

Compatibility

The wire format is unchanged. Peers with the old limit reject a worker address with more than 128 entries (more than 64 for versions before v1.11). With mixed versions, limit resources with UCX_TLS/UCX_NET_DEVICES.

Tests

  • test_ucp_tl: contexts with 50 and 255 resources, including a worker address pack/unpack; 256 resources fail with UCS_ERR_EXCEEDS_LIMIT.
  • test_ucp_tl_bitmap: empty bitmap, word boundaries and the last valid index, the min/max values, and the unused rounding bit.

Verification

Tested on two VR nodes:

  • This branch on both nodes, over 128 resources in each: transfers pass, with the same lanes and bandwidth as below the limit.
  • This branch against UCX 1.18.1, both directions: up to 128 resources works as normal; above 128, the 1.18.1 side rejects the address (number of addresses exceeds 128).

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@tomerg-nvidia
tomerg-nvidia self-requested a review October 9, 2026 10:10
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@tomerg-nvidia tomerg-nvidia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note there's also ucp_address_iface_flags_ptr which packs some flags to the higher bits of the resource index if using unified mode. I think it was already partly problematic because UCP_ADDRESS_FLAG_HAS_EP_ADDR was already in the representable range.

Comment thread src/ucp/rma/amo_sw.c Resolved
Comment thread src/ucs/datastruct/static_bitmap.h Resolved
@guy-ealey-morag

Copy link
Copy Markdown
Contributor Author

@tomerg-nvidia The thing with unified mode is an existing issue on master already, it was implemented when the limit was only 64 so this issue exists from the last resources limit increase.
I can implement a specific resource limit for unified mode to fix this issue, but it should not block this PR.

@tomerg-nvidia

Copy link
Copy Markdown
Contributor

@tomerg-nvidia The thing with unified mode is an existing issue on master already, it was implemented when the limit was only 64 so this issue exists from the last resources limit increase. I can implement a specific resource limit for unified mode to fix this issue, but it should not block this PR.

Agreed, as I said, it was already an issue before with UCP_ADDRESS_FLAG_HAS_EP_ADDR. I'm not sure how used this option is given it's been an issue for so long.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
tomerg-nvidia
tomerg-nvidia previously approved these changes Oct 9, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 8d93495e

TL;DR: rcx/test_ucp_stream.send_recv_64/0 <rc_x> hung for ~11 minutes with zero output and then aborted with "request did not complete on time", i.e. a stream transfer over rc_x never progressed — consistent with the resource-index widening in this PR, where UCP_MAX_RESOURCES (255) now collides with the UCP_NULL_RESOURCE sentinel ((uint8_t)-1 == 255). Fix by making the sentinel distinct from the valid index range (e.g. UCP_MAX_RESOURCES 254, or widen ucp_rsc_index_t/reserve 255 exclusively for NULL) and auditing all UCP_NULL_RESOURCE comparisons and resource-iteration bounds.

Full analysis

Summary: gtest stage "roce on worker 0" aborted: rcx/test_ucp_stream.send_recv_64/0 <rc_x> hung and timed out ("request 0x62e000286418 did not complete on time"), causing Connection timed out - abort testing and make: *** [Makefile:4925: test] Aborted (core dumped).

Root cause: This is a hang, not a slow test. The log shows the test suite running continuously at ~200ms per test right up to 18:04:32.92, where [ RUN ] rcx/test_ucp_stream.send_recv_64/0 <rc_x> is printed — then 11 minutes and 17 seconds of complete silence (18:04:32.9 → 18:15:49.8), the single largest gap in the entire log, ending in ucp_test.cc:300 UCX ERROR request ... did not complete on time. A further ~3.7 min of silence follows before the test_helpers.cc:57 abort. So the stream request over the rc_x transport was issued and never completed — the progress loop spun (backtrace shows __sched_yield under the gtest wait loop) without the operation ever making forward progress.

The hang points at the change under test. PR #12054 / commit d24d5c6d ("UCP/CORE: Support up to 255 TL resources") is the only recent commit touching all of the implicated core files (ucp_types.h, ucp_context.c, ucp_worker.c, wireup/address.c). In src/ucp/core/ucp_types.h the new limit creates a sentinel collision:

#define UCP_MAX_RESOURCES   255
#define UCP_NULL_RESOURCE   ((ucp_rsc_index_t)-1)   /* == 255 */
typedef uint8_t             ucp_rsc_index_t;

With ucp_rsc_index_t being uint8_t, resource index 255 is simultaneously a valid resource index (the bitmap ucs_static_bitmap_s(255) rounds up to 4 words / 256 bits, so bit 255 is addressable) and the "no resource" sentinel. Any lane or iface that lands on index 255 — or any loop bounded by <= UCP_MAX_RESOURCES rather than < — will be silently treated as UCP_NULL_RESOURCE, so the lane is never selected/progressed and the request stalls forever. This matches the observed symptom exactly: wireup completes (the test got far enough to post a send) but the data transfer never advances. Note the related UCT_TL_BITMAP_FMT/ARG macros in the same file hardcode exactly 4 words, confirming the 256-bit width while only 255 indices are nominally legal.

Implicated commit: d24d5c6d — Guy Ealey Morag, "UCP/CORE: Support up to 255 TL resources" (2026-10-08), the PR #12054 change under test at commit 8d93495e.

File: src/ucp/core/ucp_types.h:29-31 (and :92, :107-110 for the bitmap width); failure observed at test/gtest/ucp/test_ucp_stream.cc via test/gtest/ucp/ucp_test.cc:300.

Suggested fix:

  1. Break the sentinel/valid-index collision. Either set #define UCP_MAX_RESOURCES 254 so that 255 remains reserved exclusively for UCP_NULL_RESOURCE, or widen ucp_rsc_index_t to uint16_t and define UCP_NULL_RESOURCE as a value outside the 0..255 range. Add a compile-time guard, e.g. UCS_STATIC_ASSERT(UCP_MAX_RESOURCES < UCP_NULL_RESOURCE);
  2. Audit every resource-index loop bound in the diff for <= vs < against UCP_MAX_RESOURCES, and every == UCP_NULL_RESOURCE / != UCP_NULL_RESOURCE comparison in ucp_context.c, ucp_worker.c and ucp_ep.c for indices that can now legitimately reach 255.
  3. Re-check the address pack/unpack path: ucp_address_pack_byte_extended() / ucp_address_unpack_byte_extended() (src/ucp/wireup/address.c:571-603) use a mask value as the "escape to next byte" marker; verify that a resource/MD index of 254 or 255 round-trips correctly rather than aliasing onto the escape marker, since a mis-unpacked iface index would produce exactly this silent no-progress hang.
  4. To reproduce locally, run the single case under the same config: ./gtest --gtest_filter='rcx/test_ucp_stream.send_recv_64/0' on an rc_x device, with a ucp_rsc_index_t assertion enabled.

Do not raise the job time limit — the 11-minute gap with no output proves the process was hung, not legitimately slow.

Related: PR #12054 — UCP/CORE: Support up to 255 TL resources; likely interacts with PR #10700 — UCP/TYPES: increase max number of lanes to 128 - use STATIC_BITMAP, which touches the same index/bitmap widening area.

Comment thread src/ucp/rma/amo_sw.c Outdated
UCP_WORKER_GET_EP_BY_ID(&ep, worker, atomicreqh->req.ep_id, return UCS_OK,
"SW AMO request");
if (ucs_unlikely((amo_rsc_idx != UCP_MAX_RESOURCES) &&
if (ucs_unlikely((amo_rsc_idx < UCP_MAX_RESOURCES) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why changed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an empty bitmap amo_rsc_idx would be 256, so a more robust check is < UCP_MAX_RESOURCES (UCP_MAX_RESOURCES is 255).

Comment thread src/ucp/core/ucp_types.h Outdated
Comment on lines +31 to +32
typedef uint8_t ucp_rsc_index_t;
typedef unsigned ucp_rsc_iter_t; /* For iterating ucp_tl_bitmap_t bits */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need both? in across the diff ucp_rsc_iter_t replaces ucp_rsc_index_t

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_rsc_index_t is the type used for the wire and it should not be changed, ucp_rsc_iter_t is needed for iteration because looping on a bitmap of size 255 would reach the value of 256 on the last iteration (without the new type it would wrap back to 0).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so it means we still cannot use > 128 resources connected to a peer.
maybe rename ucp_rsc_iter_t -> ucp_local_rsc_index_t and ucp_rsc_index_t -> ucp_addr_rsc_index_t?

@guy-ealey-morag guy-ealey-morag Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@evgeny-leksikov Why would it be limited by 128 resources connected to a peer? All 8 bits can be used for resource indices. Values 0-254 can be used, with the special value UCP_NULL_RESOURCE=255.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, but why do we need the new type then?

@guy-ealey-morag guy-ealey-morag Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_rsc_index_t wraps back to 0 when iterating over the bitmap, it can't hold the value 256

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we cannot safely iterate it even with larger type if UCP_NULL_RESOURCE=255, IMO need to add a helper macro to iterate the bitmap and use ucp_rsc_index_t for that, new type adds confusing

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented it using a new macro UCP_TL_BITMAP_FOR_EACH_BIT

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 12, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 3f975ca0

TL;DR: The build was killed mid-compile by an Azure agent shutdown ("The agent has received a shutdown signal"), not by any UCX code or test failure — this is an infrastructure/cancellation event, so just re-run the roce on worker 0 job.

Full analysis

Summary: The "Tests roce on worker 0" job (step Run ./contrib/test_jenkins.sh) was canceled while still in the compile phase of src/ucp; no test ever ran.

Root cause: Infrastructure cancellation, not a code defect. The log shows continuous, healthy build activity with no gap larger than ~2 seconds (every line is a libtool: compile / mv -f .deps/... at 0.1–1.5 s intervals) right up to 10:31:51.2647807Z ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled. followed by ##[error]The Operation will be canceled. and ##[error]The operation was canceled. at 10:32:01. So there is no hang and no slowness — the self-hosted agent (agent-09, workspace [REDACTED:Base64 High Entropy String]) was stopped out from under the job. Typical triggers: the agent service was restarted/drained on that host, the host was rebooted/reimaged, or the pipeline run was superseded (new push to topic/ucp-max-resources-255) and Azure DevOps canceled the in-flight run. Zero compiler warnings/errors appear before the shutdown, and the only remaining work at kill time was linking libucp.so (which actually completed, including ranlib .libs/libucp.a), confirming the source tree at [REDACTED:Hex High Entropy String] builds fine.

Implicated commit: none — the failure is unrelated to [REDACTED:Hex High Entropy String] or any source change.

File: n/a (build/agent infrastructure; step Run ./contrib/test_jenkins.sh on agent-09)

Suggested fix: Re-queue the roce on worker 0 job for build 138761 (or push an empty commit / "re-run failed jobs"). If it reproduces on the same agent, have CI ops check agent-09's health — specifically whether the Azure Pipelines agent service is being restarted or the node drained during runs, and whether /scrap/azure/agent-09 is running out of disk (a common cause of agent auto-restart on these self-hosted ROCE nodes). No UCX code change is warranted, and do not raise any timeout: the log shows uninterrupted progress with no stall before the kill.

Related: none — searches for this signature returned only unrelated open PRs (#11932, #11903, #12033, #11870, #11923); no existing issue tracks this agent-shutdown flake.

🛡️ This comment had 2 potential secret(s) redacted (Base64 High Entropy String, Hex High Entropy String). See request_id bdd128ca-33f4-406b-a90a-244e9f085b4c in the triage console for the audit trail.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UCP changes LGTM, can we revert UCS changes now?

@guy-ealey-morag

Copy link
Copy Markdown
Contributor Author

UCP changes LGTM, can we revert UCS changes now?

The new static assert added in UCS helps prevent misuse of UCS_STATIC_BITMAP_FOR_EACH_BIT with the wrong type (e.g. using UCS_STATIC_BITMAP_FOR_EACH_BIT on the tl bitmap with ucp_rsc_index_t)

@evgeny-leksikov

Copy link
Copy Markdown
Contributor

The new static assert added in UCS helps prevent misuse of UCS_STATIC_BITMAP_FOR_EACH_BIT with the wrong type (e.g. using UCS_STATIC_BITMAP_FOR_EACH_BIT on the tl bitmap with ucp_rsc_index_t)

I see, but passing (void)0 to check param looks confusing and it works only for fatal cases, let's move the check to the loop condition statement to make macro API clear, constant positive check should be compiled out anyway, negative will be triggered as well + option for runtime check. WDYT?

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@guy-ealey-morag

guy-ealey-morag commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

@evgeny-leksikov Moved that static assert to #12080

@svc-ucx

svc-ucx commented Oct 13, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 2) · commit 8058a945

TL;DR: One gtest, shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0, failed an expectation (not a crash) out of 10429 tests, making make test exit 1; the failure is in config/topology-sensitive RNDV protocol-selection assertions, not in the PR's TL-resource-count change. Re-run with the full gtest output captured (or run that single test with -v) to get the exact EXPECT_* line, and harden the test's global count_rkey_configs_with_flag() assertion.

Full analysis

Summary: Azure job "Tests gpu on worker 2" (build 138850) failed because the gtest binary reported 1 FAILED TEST — shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 (GetParam() = shm,ib,cuda_copy,rocm_copy) — causing make: *** [Makefile:4926: test] Error 1 and ##[error]Bash exited with code '2'.

Root cause: Evidence from the log:

  • 10408 passed, 20 skipped, exactly 1 failed. No ucs_assert, no ucs_fatal, no SIGSEGV/SIGABRT, no stack trace, no timeout/SIGTERM anywhere in the log; the suite ran to Global test environment tear-down and the process exited normally. So this is a C++ EXPECT_*/ASSERT_* mismatch inside the test body, not a crash, hang, or resource-limit error.
  • The whole-build log is only a ~500-line tail, and gtest prints detailed failure text at the moment the test runs (tests execute in shuffled order — this test ran long before the tail window). The failing expectation text is therefore not present in any log I could retrieve, so the specific failing assertion cannot be named from evidence.
  • What the log does establish is that it is not the PR's subject matter: there is no "too many components"/UCS_ERR_EXCEEDS_LIMIT message (src/ucp/core/ucp_context.c:2282), no TL-bitmap or resource-index diagnostic, and every other shm_ib / rndv / rkey-compare / proto test in the run passed.
  • Reading test/gtest/ucp/test_ucp_proto.cc:829-837, this test runs with RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y plus RNDV_SCHEME=get_zcopy and calls check_am_rndv_remote_proto_config(0, 0) (line 449), which asserts three things: that am/rndv is the selected protocol at 1 MB for CUDA memory (line 439), that remote_proto_config->select_param.op_id_flags == UCP_OP_ID_RNDV_RECV (line 456), and that count_rkey_configs_with_flag(worker(), UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION) == 0 (line 458). The two sibling tests with the same assertions but without RNDV_SCHEME=get_zcopy (am_default_remote_model_preserves_legacy_key, am_force_remote_model_remains_untagged) both passed, so the divergence is tied to the explicit-scheme config path (ucp_proto_rndv_shm_pipeline_force_enabled() requires rndv_mode == UCP_RNDV_MODE_AUTO, src/ucp/rndv/proto_rndv.inl:20-28).
  • Supporting context: this exact test name is the subject of three prior throwaway diagnosis PRs ([TEMP][DO NOT REVIEW] Diagnose PR 11685 Azure protocol selection #11858 "Diagnose PR 11685 Azure protocol selection", [TEMP][DO NOT REVIEW] Isolate PR 11685 mock topology state #11861, [TEMP][DO NOT REVIEW] Isolate PR 11685 mtype topology state #11864 "Isolate PR 11685 mtype topology state"), i.e. it has a documented history of environment/topology-dependent failures specifically on the Azure GPU workers.

Implicated commit: unknown — not attributable from the available log. Candidates recently touching the exercised code, all merged into this branch within days of the failure: [REDACTED:Hex High Entropy String] (Guy Ealey Morag, UCP/RNDV: Use local memory detection for RTR_REQ responder, #12041), b60d87b0 (jeffnvidia, UCP/RKEY: Keep configurations at stable addresses, #11676), 0311ed79 (jeffnvidia, UCP/PROTO/RNDV: Isolate remote system device namespace, #11685). The PR's own commits (d24d5c6d, 72386916, Guy Ealey Morag) show no corroborating signature in the log.

File: test/gtest/ucp/test_ucp_proto.cc:829 (failing test), assertions at test/gtest/ucp/test_ucp_proto.cc:439, :456, :458-459; helper test/gtest/ucp/test_ucp_proto.cc:59-70

Suggested fix:

  1. Get the actual assertion before changing product code — re-run just this case to capture gtest's failure block, e.g. GTEST_FILTER='shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0' UCX_LOG_LEVEL=debug ./gtest on the same worker. The three candidate assertions above produce very different fixes, so pick based on which one fires.
  2. Make the third assertion non-global regardless. count_rkey_configs_with_flag() (line 59) walks every rkey config in the worker via ucs_array_for_each(&worker->rkey_config) and dereferences (*rkey_config_p)->key.flags. Since b60d87b0 moved rkey configs to an array of separately-allocated pointers, a count over all slots is sensitive to any unrelated config created during the run. Scope the check to the rkey_cfg_index actually returned by select_am_rndv_remote_proto_config() instead of counting worker-wide, which removes a whole class of order/state-dependent false failures.
  3. If the failing expectation turns out to be EXPECT_STREQ("am/rndv", ...) at line 439, add a UCS_TEST_SKIP_R guard when am/rndv is not selectable for CUDA on the local topology — matching the existing guard style used at line 724 ("stub protocol" → skip) in rndv_force_cuda_send_child_uses_tag_provenance.
  4. Do not raise any timeout or retry count — there is no timing component to this failure.

Related: PR #12054 (this PR), #11858, #11861, #11864 (prior diagnosis of this same test on Azure), #11676, #12041, #11685

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f417aa9b-e12d-4f2d-92ac-804526bbb562 in the triage console for the audit trail.

@guy-ealey-morag
guy-ealey-morag enabled auto-merge (squash) October 13, 2026 13:18
@guy-ealey-morag
guy-ealey-morag merged commit e57ea1c into openucx:master Oct 14, 2026
162 checks passed
guy-ealey-morag added a commit that referenced this pull request Oct 15, 2026
UCP/CORE: Support up to 255 TL resources - v1.23.x (#12054)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants