Skip to content

UCP: Keep measured path bandwidth per endpoint - #12334

Merged
dmitrygx merged 9 commits into
openucx:masterfrom
dmitrygx:topic/ucx/dlb-path-bw-samples
Nov 17, 2026
Merged

dmitrygx merged 9 commits into
openucx:masterfrom
dmitrygx:topic/ucx/dlb-path-bw-samples

Conversation

@dmitrygx

@dmitrygx dmitrygx commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

What?

Keep the bandwidth that the transport measures on each RMA lane of an endpoint. This is infrastructure for path bandwidth balancing, part 1 of 3 (whole series: #12335): nothing in this PR reads the samples or UCP_EP_FLAG_PATH_DEGRADED yet; the RMA protocols do it in part 2 (#12362). No in-tree transport raises UCT_EP_EVENT_PATH_SPEED_CHANGE; an RC plugin raises it through the hooks of #12162.

Why?

On systems with two NICs per GPU, the RMA lanes of an endpoint go over both NICs, and the static lane model splits every operation between them. Other traffic on one NIC then slows every operation down to its lane on that NIC. With the RC plugin hooks of #12162, a plugin measures each path and raises UCT_EP_EVENT_PATH_SPEED_CHANGE; this PR keeps the result in UCP. With the whole series, nixlbench READ/WRITE gets 1.56x and vLLM disaggregated serving 1.44x tokens/s per GPU under such load.

How?

  • Set the endpoint event handler on every iface; UCS_ERR_UNSUPPORTED is ignored.
  • Handle the path events of an iface together: one oneshot worker progress callback per iface queries UCT_EP_ATTR_FIELD_BANDWIDTH of the lanes of all endpoints on it, so an event needs no endpoint lookup.
  • Compare a sample with the path estimate (UCT_PERF_ATTR_FIELD_PATH_BANDWIDTH), which a path without samples reports, and keep only the samples below it, in a worker hash table keyed by the endpoint, so ucp_ep_t and the endpoint configuration do not change. UCP_EP_PATH_BW_INVALID marks a path which is not degraded.
  • A failed query keeps the previous sample, a configuration or lane change drops all samples, and below 1 B/s the path is stalled (0).
  • UCP_EP_FLAG_PATH_DEGRADED is set while a lane has a sample, so part 2 changes the send path only for such endpoints. Endpoint destruction frees the samples without touching the endpoint flags, since it can run without the async lock.

Tests: test_ucp_proto_mock_path_bw* on rc_x and posix,cma: samples follow events, per-process samples, unusable samples, reset on reconfiguration, destroy with a pending update, recovery of a multi-path device, and close of a single-thread worker endpoint while a worker flush holds its last reference (rc_x).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Register the endpoint event handler on transports which support it. On UCT_EP_EVENT_PATH_SPEED_CHANGE, query the bandwidth of the RMA bandwidth lanes from worker progress and keep the samples in a worker hash table, so the endpoint size and its configuration do not change. A failed query keeps the previous sample, a configuration or lane change drops the samples, and a rate below 1 B/s marks a stalled path. An endpoint flag marks endpoints which have a lane below the iface bandwidth, and it is cleared when the lanes recover.
@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-path-bw-samples branch from 44eeeba to 7a2d5f9 Compare November 14, 2026 19:23
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.h
uint32_t lane_generation; /* Lanes of the samples */
ucp_lane_map_t valid; /* Lanes which have a sample */
double bandwidth[]; /* Lane bandwidths */
} ucp_ep_path_bw_t;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new ucp_ep_path_bw_t: nothing reads state->bandwidth[]/valid or UCP_EP_FLAG_PATH_DEGRADED in this PR, and no
transport raises UCT_EP_EVENT_PATH_SPEED_CHANGE yet - is the consumer coming in a
follow-up PR?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: No transport in-tree invokes ep_event_cb, so the real event path (rc_mlx5 port speed change) is only exercised through the mock; the rc_x variant relies on rc_mlx5's real iface_set_ep_event_handler while the posix,cma variant mocks it.

Residual coverage gaps: No test covers dev_num_paths > 1 (IB_NUM_PATHS is forced to 1), which is exactly the case where the degraded comparison above misfires.

Note on process: no gh/network access in this environment, so the PR's existing inline comments, reviews and replies could not be fetched. Per the review workflow, duplicate detection could not be completed — these are draft comments, not submitted, and some may already be covered by existing threads.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.h Resolved
Comment thread src/ucp/core/ucp_ep.h Resolved
@svc-ucx

svc-ucx commented Nov 14, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests new on worker 0) · commit 7a2d5f94

TL;DR: The gtest binary aborted on rcache.c:405 Assertion '!(region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE)' failed while unmapping a user memh in tcp/test_ucp_rma.put_nonblocking/2 <user_memh> — an rcache region's refcount reached 0 while it was still in the page table, because ucs_rcache_region_put_unsafe()/ucs_rcache_lookup_unsafe() in rcache.inl update region->refcount with plain non‑atomic --/++ while the rest of the rcache uses ucs_atomic_fsub32/add32, so a reference can be lost and the still‑cached region gets destroyed inline.

Full analysis

Summary: Azure job "Tests new on worker 0" (build 145150) failed with make: *** [Makefile:5015: test] Aborted (core dumped) — assertion failure + core dump in ucs_mem_region_destroy_internal() during ucp_mem_unmap() in tcp/test_ucp_rma.put_nonblocking/2 <tcp,cuda_copy,rocm_copy/user_memh> at 20:33:50.

Root cause: Backtrace from the log:
ucp_mem_unmap (ucp_mm.c:1298) → ucp_memh_cleanup (ucp_mm.c:473) → ucp_memh_put_rcache (ucp_mm.c:444) → ucs_rcache_region_put_unsafe (rcache.inl:128) → ucs_mem_region_destroy_internal (rcache.c:405).
A cached region's refcount is documented as "including +1 if it's in the page table" (rcache.h:196; rcache.c:1108: region->refcount = 2; /* Page-table + user */). Therefore the user's final put should take it 2→1, never 0. The assert firing means one reference was already lost, and the "unsafe" put then destroyed a region that is still linked in the page table (the next lookup of that address would use freed memory).
The lost reference is explained by inconsistent refcount access: rcache.inl does non-atomic read‑modify‑write — region->refcount++ (line 103, ucs_rcache_lookup_unsafe) and --region->refcount (line 127, ucs_rcache_region_put_unsafe) — while rcache.c manipulates the same volatile uint32_t with atomics: ucs_atomic_add32 in ucs_rcache_region_hold (line 1135) and ucs_atomic_fsub32 in ucs_rcache_region_put_internal (line 459), and ucs_rcache_purge (line 676). The rcache async handler thread (ucs_rcache_invalidate_handler → ucs_rcache_clean → ucs_rcache_invalidate_range → put_internal) and UCM unmapped callbacks run concurrently with UCP memh get/put, so a non-atomic increment/decrement can drop a concurrent atomic update. Additionally, put_unsafe is called holding only the UCP mt_lock (a no-op in ST builds), yet it destroys the region and mutates the page table/LRU, which should only happen under pgt_lock write mode.
Note the failing branch (topic/ucx/dlb-path-bw-samples, PR #12334 — path bandwidth samples) does not touch memory/rcache code, so this is a pre-existing race in master surfacing as a flaky abort, not a regression from the PR.

Implicated commit: unknown (not reproducible from the log alone). Most recent changes in this area, worth bisecting: e536ef03 "UCS/RCACHE: Adds capability for merging adjacent rcache regions (#11958)", william-gallagher-nv, 2026-11-04 (touches both rcache.c and ucp_mm.c) and 311e3506 "UCS/RCACHE: Change inv_q type to interval tree (#11248)", nbellalou (last change to rcache.inl).

File: src/ucs/memory/rcache.inl:121-132 (put: line 127) and src/ucs/memory/rcache.inl:103; assertion site src/ucs/memory/rcache.c:405.

Suggested fix:

  1. Make the "unsafe" fast paths use the same atomics as the rest of the rcache:
    • rcache.inl:103: ucs_atomic_add32(&region->refcount, +1); (or call ucs_rcache_region_hold).
    • rcache.inl:127: if (ucs_unlikely(ucs_atomic_fsub32(&region->refcount, 1) == 1)) { ... }.
  2. Harden the destroy path so a cached region can never be freed from an unlocked context: in ucs_rcache_region_put_unsafe, when the refcount reaches 0 and region->flags & UCS_RCACHE_REGION_FLAG_PGTABLE is still set, take pgt_lock in write mode and go through ucs_rcache_region_invalidate_internal() (pgtable removal) or push the region to the GC list, instead of calling ucs_mem_region_destroy_internal() inline. Add an ucs_assertv printing refcount, flags and the region range to make future occurrences diagnosable.
  3. Re-run the tcp/test_ucp_rma suite in a loop with UCX_RCACHE_MERGE_ADJACENT=y/n and asserts enabled to confirm, and consider a short-term CI retry/skip only if the fix cannot land quickly.

Related: #11958 (adjacent-region merging, recent rcache refactor), #11248 (rcache invalidation queue → interval tree), #12209 (recent ucp_mm.c changes merged one day before this build)

@roiedanino
roiedanino self-requested a review November 15, 2026 08:26
Comment thread src/ucp/core/ucp_worker.h Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
A healthy path of a multi-path device reports the path estimate, a share
of the device bandwidth, so the endpoint must not stay degraded after it
recovers. A worker flush which holds the last reference of a closed
endpoint destroys it from the progress of a single-thread worker, without
the async lock.
- Handle the path events of an interface together from worker progress,
  for the lanes of all its endpoints, instead of looking up the endpoint
  of each event.
- Compare a sample with the path bandwidth estimate, which a path without
  samples reports, instead of the interface bandwidth, so a healthy path
  of a multi-path device is not degraded.
- Keep only the samples below the estimate, with a negative bandwidth for
  a path which is not degraded, instead of a map of valid lanes.
- Do not update the endpoint flags on its destruction, which can run
  without the async lock.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_proto_mock.cc Outdated
}

ASSERT_UCS_OK(request_wait(flush_req));
EXPECT_EQ(nullptr, path_bw(ep));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: use-after-free. The ep is already freed at this point (the flush releases the last reference), so path_bw() reads a clobbered ep->worker — ucp_ep_destroy_base() → ucp_ep_deallocate() → ucs_strided_alloc_put() overwrites the first word of ucp_ep_t (which is worker) with the freelist next pointer. It only appears to pass because the freelist head happens to point at zero-filled chunk memory (n_buckets == 0); under valgrind (VALGRIND_MEMPOOL_FREE in ucs_strided_alloc_put) this is an invalid read. Please use ep only as a hash key, e.g.:

Suggested change
EXPECT_EQ(nullptr, path_bw(ep));
EXPECT_EQ(kh_end(&worker->ep_path_bw),
kh_get(ucp_worker_ep_path_bw, &worker->ep_path_bw, ep));

Comment thread src/ucp/core/ucp_worker.c
state = ucs_malloc(sizeof(*state) +
ucp_ep_num_lanes(ep) * sizeof(*state->bandwidth),
"ep_path_bw");
if (state == NULL) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor (ucp_worker_ep_path_bw_get): please log the allocation failure here; otherwise an OOM silently stops sampling and clears UCP_EP_FLAG_PATH_DEGRADED with no trace (docs/CodeStyle.md: a function which returns an error should print a log message).

Comment thread src/ucp/core/ucp_ep.h
UCP_EP_FLAG_CONNECT_PRE_REQ_QUEUED = UCS_BIT(9), /* Pre-Connection request was queued */
UCP_EP_FLAG_CLOSED = UCS_BIT(10),/* EP was closed */
/* 11 bit is vacant for a flag */
UCP_EP_FLAG_PATH_DEGRADED = UCS_BIT(11),/* Lane path is degraded */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: which flow consumes UCP_EP_FLAG_PATH_DEGRADED / state->bandwidth[]? Right now both are write-only, so every ep on such an iface keeps a permanent allocation for state nobody reads — is the consumer coming in a follow-up PR?

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual gap: PR size is ~630 added lines (≈200 src, ≈420 test), above the 500-line limit in REVIEW.md. Not turned into a comment because the bulk is the test for a ~200-line feature and splitting tests from code conflicts with the test expectations in the same doc — flagging for your call.

Residual gap: no test covers the OOM path in ucp_worker_ep_path_bw_get() (flag is cleared, sampling stops).

@svc-ucx

svc-ucx commented Nov 15, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 0) · commit 1dca972b

TL;DR: The BlueField gtest run failed on exactly one test — rcx/test_ucp_proto_mock_path_bw.reconfig_resets_samples/0, a new test added by this PR's HEAD commit ([REDACTED:Hex High Entropy String]) — because the test's lane bookkeeping (m_lane_ids / the ep_query hooks installed once in init()) goes stale after the endpoint reconfiguration the test itself provokes, so lane_ep() no longer resolves the RMA lanes and the sample assertions fail. Re-resolve the lane map (re-call install_hooks()/key the map by device name) after the port-speed reconfiguration, or make ucp_worker_ep_path_bw_update() re-sample the re-created lanes.

Full analysis

Summary: make test in "BlueField on worker 0" exited 2: 9048/9049 gtests passed, the single failure being rcx/test_ucp_proto_mock_path_bw.reconfig_resets_samples/0 (GetParam() = rc_x); the mm_cma instantiation of the same test passed.

Root cause: The test is brand-new in this PR (commit [REDACTED:Hex High Entropy String], Dmitrii, which touches both test/gtest/ucp/test_ucp_proto_mock.cc and src/ucp/core/ucp_worker.c). test_ucp_proto_mock_path_bw::init() installs the uct_ep_query hook and builds m_lane_ids keyed by the current uct_ep_h pointers of the endpoint's lanes (test_ucp_proto_mock.cc:4230-4243). reconfig_resets_samples then deliberately bumps ucp_worker_iface(...)->port_speed and worker->epoch and issues a PUT so the endpoint moves to a new configuration (ASSERT_NE(cfg_index, ep->cfg_index)), which on rc_x re-wires/replaces the lane transport endpoints. After that, lane_ep(ep, 0/1) (test_ucp_proto_mock.cc:4245-4255) can no longer match any current lane in m_lane_ids, so set_samples(ep, 8G, 16G) hits ASSERT_NE(nullptr, tl_eps[...]) and/or sample(ep, 1) returns NaN instead of 16 GB/s. The sibling class test_ucp_proto_mock_path_bw_single_thread already acknowledges this hazard — it re-calls install_hooks(sender().ep()) after flush_workers() "because wireup completes and replaces the lanes later" (test_ucp_proto_mock.cc:4465-4472) — but reconfig_resets_samples never refreshes the map after its own reconfiguration. The posix/cma variant passes because those mock devices keep (or have no) re-wired RMA lanes. A secondary suspect in product code is the new filter in ucp_worker_ep_path_bw_update() (src/ucp/core/ucp_worker.c:1039-1048), which skips a lane whose key->lanes[lane].port_speed != wiface->port_speed; if the reconfigured key does not carry the updated port speed, the lane is never re-sampled after the event.

Note: the Azure log tail only contains the gtest summary ([ FAILED ] rcx/test_ucp_proto_mock_path_bw.reconfig_resets_samples/0); the individual assertion text scrolled out of the retrievable 500-line window, so the exact failing EXPECT/ASSERT line could not be quoted.

Implicated commit: [REDACTED:Hex High Entropy String] "UCP: Sample paths per interface against the path estimate" (Dmitrii), stacked on 0f401ed "UCP: Keep measured path bandwidth per endpoint"

File: test/gtest/ucp/test_ucp_proto_mock.cc:4419-4444 (test body), :4230-4255 (install_hooks/lane_ep); src/ucp/core/ucp_worker.c:1029-1091 (ucp_worker_ep_path_bw_update)

Suggested fix:

  1. In reconfig_resets_samples, after the second send_recv_rma() that triggers reconfiguration, refresh the lane bookkeeping before sampling again, e.g.:
    ASSERT_NE(cfg_index, ep->cfg_index);
    m_lane_ids.clear();
    install_hooks(ep);          /* lanes were replaced by the reconfiguration */
    set_samples(ep, 8.0 * UCS_GBYTE, 16.0 * UCS_GBYTE);
    Better still, stop keying m_lane_ids by uct_ep_h and resolve lane IDs on demand from the lane's dev_name (ucp_ep_get_tl_rsc(ep, lane)->dev_name == "mock_1:1"), so the helper is immune to lane replacement.
  2. While there, add a debug print/SCOPED_TRACE of path_bw(ep)->cfg_index, ep->cfg_index and ep->ext->lane_generation so the next CI failure is diagnosable from the log.
  3. Verify in product code that after a port-speed-driven reconfiguration the new ucp_ep_config_key_t::lanes[].port_speed equals wiface->port_speed; otherwise the port_speed mismatch check at src/ucp/core/ucp_worker.c:1045 permanently suppresses sampling of that lane (which would be a real bug, not just a test issue).

Related: PR #12334 (this PR), stacked PR #12335 "[WIP-DNM] UCP/RMA: Balance PUT and GET by measured path bandwidth"

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 594f44cb-62a9-4ba6-8d79-23fa814dd3e6 in the triage console for the audit trail.

roiedanino
roiedanino previously approved these changes Nov 16, 2026
send_recv_rma checked the target buffer right after the PUT request
completed, but a PUT completes locally. On rc_x a 1 KB PUT is sent as
short, which completes before the data reaches the target, so the pattern
check of reconfig_resets_samples failed. Flush the endpoint first, as the
RMA tests do.
The worker flush releases the last reference of the endpoint, so the last
check read ep->worker of a released endpoint, whose memory the strided
allocator reuses for its free list. Look the endpoint up in the hash of
the worker as a key only.
A failure to allocate the state, or to add it to the hash, silently
stopped the sampling of the endpoint.
@dmitrygx
dmitrygx force-pushed the topic/ucx/dlb-path-bw-samples branch from 631e3ba to 06aa8e3 Compare November 16, 2026 10:56
Comment thread src/ucp/core/ucp_worker.c Resolved
Use UCP_EP_PATH_BW_INVALID instead of -1 and the sign checks of the path
samples.
roiedanino
roiedanino previously approved these changes Nov 16, 2026
@svc-ucx

svc-ucx commented Nov 16, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests BlueField on worker 1) · commit 34a0b61e

TL;DR: The new PR test rcx/test_ucp_proto_mock_path_bw.reconfig_resets_samples/0 failed on BlueField because it relies on a 1 KB ucp_put_nbx() going through protocol selection to trigger the endpoint reconfiguration after a port-speed/epoch bump — but ucp_put_send_short() short-circuits before the worker-epoch check whenever the message fits the real device's cap.put.max_short, which this mock class never pins; fix the test to force the non-short path (zero put.max_short in the mock or use a message larger than max_short).

Full analysis

Summary: gtest run on "BlueField on worker 1" ended with [ FAILED ] rcx/test_ucp_proto_mock_path_bw.reconfig_resets_samples/0, where GetParam() = rc_x → make: *** [Makefile:5015: test] Error 1 (9057/9058 passed; no hangs, no timeout — total runtime 2407 s with continuous output).

Root cause: reconfig_resets_samples (test/gtest/ucp/test_ucp_proto_mock.cc:4421) bumps wiface->port_speed and worker->epoch, then expects the next send_recv_rma(UCS_KBYTE, UCP_OP_ID_PUT) to move the endpoint to a new configuration (ASSERT_NE(cfg_index, ep->cfg_index) at line 4439) and thus to drop the previously recorded path-bandwidth sample. The epoch check that performs that reconfiguration lives only in the slow path, ucp_proto_request_send_op_rma() (src/ucp/proto/proto_common.inl:353). ucp_put_nbx() first calls ucp_put_send_short() (src/ucp/rma/rma_send.c:219/154), which uses the cached rkey_config->put_short lane and issues uct_ep_put_short() without consulting worker->epoch or rkey_config->key.ep_cfg_index. Unlike the other mock classes in this file (which explicitly set iface_attr.cap.put.max_short = 2048, e.g. lines 952, 1491, 2596), test_ucp_proto_mock_path_bw::init() (lines 4200‑4214) only overrides bandwidth, latency and dev_num_paths, so put.max_short is whatever the real rc_mlx5 device/driver reports. On the BlueField worker that inline limit covers 1 KB, so the PUT took the short fast path, no reconfiguration happened, the ep cfg_index (and hence path_bw(ep)->cfg_index/samples) stayed the same, and the assertions failed. This is why the failure is host-specific and appears only in this one instantiation (the mm_cma variant and the sibling tests in the class don't issue RMA).

Implicated commit: [REDACTED:Hex High Entropy String] (branch HEAD, Dmitrii) — test introduced in this PR series by Dmitrii ([REDACTED:Hex High Entropy String] "GTEST/UCP: Cover endpoint path bandwidth samples" / eb19e05 "GTEST/UCP: Cover multi-path samples…"), supported by 0f401ed "UCP: Keep measured path bandwidth per endpoint".

File: test/gtest/ucp/test_ucp_proto_mock.cc:4421‑4446 (assert at :4439); interacting product code src/ucp/rma/rma_send.c:154‑224 and src/ucp/proto/proto_common.inl:353.

Suggested fix: Make the test independent of the host's inline size so the PUT always goes through protocol selection:

  • in test_ucp_proto_mock_path_bw::init()'s add_mock_iface iface-attr lambda add attr.cap.put.max_short = 0; (and/or clear UCT_IFACE_FLAG_PUT_SHORT), or
  • use a size that cannot be short, e.g. send_recv_rma(64 * UCS_KBYTE, UCP_OP_ID_PUT) as the other RMA mock tests do, or pass UCP_OP_ATTR_FLAG_NO_IMM_CMPL for these two PUTs.

Separately worth a product follow-up: ucp_put_send_short() silently keeps using a stale rkey_config->put_short lane after a worker-epoch/port-speed change — consider validating proto_select->worker_epoch/ep_cfg_index in the short fast path too.

Related: PR #12334 (this PR), PR #12335 ([WIP-DNM] UCP/RMA: Balance PUT and GET by measured path bandwidth)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 692357f0-fe9a-477a-99dc-ce90574767ad in the triage console for the audit trail.

On rc_x a 1 KB PUT goes through the short fast path of ucp_put_nbx, which
does not check the worker epoch. So the endpoint kept its configuration
after the port speed change, and the configuration check failed. Send
64 KB, which goes through the protocol send and its epoch check, as on
cma, which has no PUT short.
@dmitrygx
dmitrygx merged commit 48795b8 into openucx:master Nov 17, 2026
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants