Repository navigation
UCP/CORE: Support up to 255 TL resources - #12054
Conversation
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
tomerg-nvidia
left a comment
There was a problem hiding this comment.
Note there's also ucp_address_iface_flags_ptr which packs some flags to the higher bits of the resource index if using unified mode. I think it was already partly problematic because UCP_ADDRESS_FLAG_HAS_EP_ADDR was already in the representable range.
|
@tomerg-nvidia The thing with unified mode is an existing issue on master already, it was implemented when the limit was only 64 so this issue exists from the last resources limit increase. |
Agreed, as I said, it was already an issue before with |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: gtest stage "roce on worker 0" aborted: Root cause: This is a hang, not a slow test. The log shows the test suite running continuously at ~200ms per test right up to The hang points at the change under test. PR #12054 / commit #define UCP_MAX_RESOURCES 255
#define UCP_NULL_RESOURCE ((ucp_rsc_index_t)-1) /* == 255 */
typedef uint8_t ucp_rsc_index_t;With Implicated commit: File: Suggested fix:
Do not raise the job time limit — the 11-minute gap with no output proves the process was hung, not legitimately slow. Related: PR #12054 — UCP/CORE: Support up to 255 TL resources; likely interacts with PR #10700 — UCP/TYPES: increase max number of lanes to 128 - use STATIC_BITMAP, which touches the same index/bitmap widening area. |
| UCP_WORKER_GET_EP_BY_ID(&ep, worker, atomicreqh->req.ep_id, return UCS_OK, | ||
| "SW AMO request"); | ||
| if (ucs_unlikely((amo_rsc_idx != UCP_MAX_RESOURCES) && | ||
| if (ucs_unlikely((amo_rsc_idx < UCP_MAX_RESOURCES) && |
There was a problem hiding this comment.
For an empty bitmap amo_rsc_idx would be 256, so a more robust check is < UCP_MAX_RESOURCES (UCP_MAX_RESOURCES is 255).
| typedef uint8_t ucp_rsc_index_t; | ||
| typedef unsigned ucp_rsc_iter_t; /* For iterating ucp_tl_bitmap_t bits */ |
There was a problem hiding this comment.
why do we need both? in across the diff ucp_rsc_iter_t replaces ucp_rsc_index_t
There was a problem hiding this comment.
ucp_rsc_index_t is the type used for the wire and it should not be changed, ucp_rsc_iter_t is needed for iteration because looping on a bitmap of size 255 would reach the value of 256 on the last iteration (without the new type it would wrap back to 0).
There was a problem hiding this comment.
so it means we still cannot use > 128 resources connected to a peer.
maybe rename ucp_rsc_iter_t -> ucp_local_rsc_index_t and ucp_rsc_index_t -> ucp_addr_rsc_index_t?
There was a problem hiding this comment.
@evgeny-leksikov Why would it be limited by 128 resources connected to a peer? All 8 bits can be used for resource indices. Values 0-254 can be used, with the special value UCP_NULL_RESOURCE=255.
There was a problem hiding this comment.
right, but why do we need the new type then?
There was a problem hiding this comment.
ucp_rsc_index_t wraps back to 0 when iterating over the bitmap, it can't hold the value 256
There was a problem hiding this comment.
we cannot safely iterate it even with larger type if UCP_NULL_RESOURCE=255, IMO need to add a helper macro to iterate the bitmap and use ucp_rsc_index_t for that, new type adds confusing
There was a problem hiding this comment.
I implemented it using a new macro UCP_TL_BITMAP_FOR_EACH_BIT
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The build was killed mid-compile by an Azure agent shutdown ("The agent has received a shutdown signal"), not by any UCX code or test failure — this is an infrastructure/cancellation event, so just re-run the Full analysisSummary: The "Tests roce on worker 0" job (step Root cause: Infrastructure cancellation, not a code defect. The log shows continuous, healthy build activity with no gap larger than ~2 seconds (every line is a Implicated commit: none — the failure is unrelated to File: n/a (build/agent infrastructure; step Suggested fix: Re-queue the Related: none — searches for this signature returned only unrelated open PRs (#11932, #11903, #12033, #11870, #11923); no existing issue tracks this agent-shutdown flake.
|
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
evgeny-leksikov
left a comment
There was a problem hiding this comment.
UCP changes LGTM, can we revert UCS changes now?
The new static assert added in UCS helps prevent misuse of |
I see, but passing |
Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
|
🤖 Starting review — findings will be posted here when done. |
|
@evgeny-leksikov Moved that static assert to #12080 |
|
🤖 CI Triage Agent — TL;DR: One gtest, Full analysisSummary: Azure job "Tests gpu on worker 2" (build 138850) failed because the gtest binary reported Root cause: Evidence from the log:
Implicated commit: unknown — not attributable from the available log. Candidates recently touching the exercised code, all merged into this branch within days of the failure: File: Suggested fix:
Related: PR #12054 (this PR), #11858, #11861, #11864 (prior diagnosis of this same test on Azure), #11676, #12041, #11685
|
UCP/CORE: Support up to 255 TL resources - v1.23.x (#12054)
What?
UCP_MAX_RESOURCESfrom 128 to 255.ucp_rsc_index_tstays 8 bits (UCP_NULL_RESOURCEremains 255).Why?
Systems with many NICs/GPUs exceed 128 transport resources, and UCP context creation fails with "exceeded transports/devices limit" with the default
UCX_TLS=all. Increasing the limits allows running on these systems with the defaultUCX_TLS.How?
UCP_TL_BITMAP_FOR_EACH_BIT, built onucp_tl_bitmap_ffs(), which caps the search atUCP_MAX_RESOURCES.ucp_tl_bitmap_maxsets exactly the valid indices 0-254.Compatibility
The wire format is unchanged. Peers with the old limit reject a worker address with more than 128 entries (more than 64 for versions before v1.11). With mixed versions, limit resources with
UCX_TLS/UCX_NET_DEVICES.Tests
test_ucp_tl: contexts with 50 and 255 resources, including a worker address pack/unpack; 256 resources fail withUCS_ERR_EXCEEDS_LIMIT.test_ucp_tl_bitmap: empty bitmap, word boundaries and the last valid index, the min/max values, and the unused rounding bit.Verification
Tested on two VR nodes:
number of addresses exceeds 128).