Repository navigation
UCP: Keep measured path bandwidth per endpoint - #12334
Conversation
|
🤖 Starting review — findings will be posted here when done. |
Register the endpoint event handler on transports which support it. On UCT_EP_EVENT_PATH_SPEED_CHANGE, query the bandwidth of the RMA bandwidth lanes from worker progress and keep the samples in a worker hash table, so the endpoint size and its configuration do not change. A failed query keeps the previous sample, a configuration or lane change drops the samples, and a rate below 1 B/s marks a stalled path. An endpoint flag marks endpoints which have a lane below the iface bandwidth, and it is cleared when the lanes recover.
44eeeba to
7a2d5f9
Compare
| uint32_t lane_generation; /* Lanes of the samples */ | ||
| ucp_lane_map_t valid; /* Lanes which have a sample */ | ||
| double bandwidth[]; /* Lane bandwidths */ | ||
| } ucp_ep_path_bw_t; |
There was a problem hiding this comment.
new ucp_ep_path_bw_t: nothing reads state->bandwidth[]/valid or UCP_EP_FLAG_PATH_DEGRADED in this PR, and no
transport raises UCT_EP_EVENT_PATH_SPEED_CHANGE yet - is the consumer coming in a
follow-up PR?
|
Residual coverage gaps: No transport in-tree invokes Residual coverage gaps: No test covers Note on process: no |
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The gtest binary aborted on Full analysisSummary: Azure job "Tests new on worker 0" (build 145150) failed with Root cause: Backtrace from the log: Implicated commit: unknown (not reproducible from the log alone). Most recent changes in this area, worth bisecting: File: Suggested fix:
Related: #11958 (adjacent-region merging, recent rcache refactor), #11248 (rcache invalidation queue → interval tree), #12209 (recent |
A healthy path of a multi-path device reports the path estimate, a share of the device bandwidth, so the endpoint must not stay degraded after it recovers. A worker flush which holds the last reference of a closed endpoint destroys it from the progress of a single-thread worker, without the async lock.
- Handle the path events of an interface together from worker progress, for the lanes of all its endpoints, instead of looking up the endpoint of each event. - Compare a sample with the path bandwidth estimate, which a path without samples reports, instead of the interface bandwidth, so a healthy path of a multi-path device is not degraded. - Keep only the samples below the estimate, with a negative bandwidth for a path which is not degraded, instead of a map of valid lanes. - Do not update the endpoint flags on its destruction, which can run without the async lock.
|
🤖 Starting review — findings will be posted here when done. |
| } | ||
|
|
||
| ASSERT_UCS_OK(request_wait(flush_req)); | ||
| EXPECT_EQ(nullptr, path_bw(ep)); |
There was a problem hiding this comment.
blocker: use-after-free. The ep is already freed at this point (the flush releases the last reference), so path_bw() reads a clobbered ep->worker — ucp_ep_destroy_base() → ucp_ep_deallocate() → ucs_strided_alloc_put() overwrites the first word of ucp_ep_t (which is worker) with the freelist next pointer. It only appears to pass because the freelist head happens to point at zero-filled chunk memory (n_buckets == 0); under valgrind (VALGRIND_MEMPOOL_FREE in ucs_strided_alloc_put) this is an invalid read. Please use ep only as a hash key, e.g.:
| EXPECT_EQ(nullptr, path_bw(ep)); | |
| EXPECT_EQ(kh_end(&worker->ep_path_bw), | |
| kh_get(ucp_worker_ep_path_bw, &worker->ep_path_bw, ep)); |
| state = ucs_malloc(sizeof(*state) + | ||
| ucp_ep_num_lanes(ep) * sizeof(*state->bandwidth), | ||
| "ep_path_bw"); | ||
| if (state == NULL) { |
There was a problem hiding this comment.
minor (ucp_worker_ep_path_bw_get): please log the allocation failure here; otherwise an OOM silently stops sampling and clears UCP_EP_FLAG_PATH_DEGRADED with no trace (docs/CodeStyle.md: a function which returns an error should print a log message).
| UCP_EP_FLAG_CONNECT_PRE_REQ_QUEUED = UCS_BIT(9), /* Pre-Connection request was queued */ | ||
| UCP_EP_FLAG_CLOSED = UCS_BIT(10),/* EP was closed */ | ||
| /* 11 bit is vacant for a flag */ | ||
| UCP_EP_FLAG_PATH_DEGRADED = UCS_BIT(11),/* Lane path is degraded */ |
There was a problem hiding this comment.
question: which flow consumes UCP_EP_FLAG_PATH_DEGRADED / state->bandwidth[]? Right now both are write-only, so every ep on such an iface keeps a permanent allocation for state nobody reads — is the consumer coming in a follow-up PR?
|
Residual gap: PR size is ~630 added lines (≈200 src, ≈420 test), above the 500-line limit in Residual gap: no test covers the OOM path in |
|
🤖 CI Triage Agent — TL;DR: The BlueField gtest run failed on exactly one test — Full analysisSummary: Root cause: The test is brand-new in this PR (commit [REDACTED:Hex High Entropy String], Dmitrii, which touches both Note: the Azure log tail only contains the gtest summary ( Implicated commit: [REDACTED:Hex High Entropy String] "UCP: Sample paths per interface against the path estimate" (Dmitrii), stacked on 0f401ed "UCP: Keep measured path bandwidth per endpoint" File: test/gtest/ucp/test_ucp_proto_mock.cc:4419-4444 (test body), :4230-4255 ( Suggested fix:
Related: PR #12334 (this PR), stacked PR #12335 "[WIP-DNM] UCP/RMA: Balance PUT and GET by measured path bandwidth"
|
send_recv_rma checked the target buffer right after the PUT request completed, but a PUT completes locally. On rc_x a 1 KB PUT is sent as short, which completes before the data reaches the target, so the pattern check of reconfig_resets_samples failed. Flush the endpoint first, as the RMA tests do.
The worker flush releases the last reference of the endpoint, so the last check read ep->worker of a released endpoint, whose memory the strided allocator reuses for its free list. Look the endpoint up in the hash of the worker as a key only.
A failure to allocate the state, or to add it to the hash, silently stopped the sampling of the endpoint.
631e3ba to
06aa8e3
Compare
Use UCP_EP_PATH_BW_INVALID instead of -1 and the sign checks of the path samples.
|
🤖 CI Triage Agent — TL;DR: The new PR test Full analysisSummary: gtest run on "BlueField on worker 1" ended with Root cause: Implicated commit: [REDACTED:Hex High Entropy String] (branch HEAD, Dmitrii) — test introduced in this PR series by Dmitrii ([REDACTED:Hex High Entropy String] "GTEST/UCP: Cover endpoint path bandwidth samples" / eb19e05 "GTEST/UCP: Cover multi-path samples…"), supported by 0f401ed "UCP: Keep measured path bandwidth per endpoint". File: test/gtest/ucp/test_ucp_proto_mock.cc:4421‑4446 (assert at :4439); interacting product code src/ucp/rma/rma_send.c:154‑224 and src/ucp/proto/proto_common.inl:353. Suggested fix: Make the test independent of the host's inline size so the PUT always goes through protocol selection:
Separately worth a product follow-up: Related: PR #12334 (this PR), PR #12335 ([WIP-DNM] UCP/RMA: Balance PUT and GET by measured path bandwidth)
|
On rc_x a 1 KB PUT goes through the short fast path of ucp_put_nbx, which does not check the worker epoch. So the endpoint kept its configuration after the port speed change, and the configuration check failed. Send 64 KB, which goes through the protocol send and its epoch check, as on cma, which has no PUT short.
What?
Keep the bandwidth that the transport measures on each RMA lane of an endpoint. This is infrastructure for path bandwidth balancing, part 1 of 3 (whole series: #12335): nothing in this PR reads the samples or
UCP_EP_FLAG_PATH_DEGRADEDyet; the RMA protocols do it in part 2 (#12362). No in-tree transport raisesUCT_EP_EVENT_PATH_SPEED_CHANGE; an RC plugin raises it through the hooks of #12162.Why?
On systems with two NICs per GPU, the RMA lanes of an endpoint go over both NICs, and the static lane model splits every operation between them. Other traffic on one NIC then slows every operation down to its lane on that NIC. With the RC plugin hooks of #12162, a plugin measures each path and raises
UCT_EP_EVENT_PATH_SPEED_CHANGE; this PR keeps the result in UCP. With the whole series, nixlbench READ/WRITE gets 1.56x and vLLM disaggregated serving 1.44x tokens/s per GPU under such load.How?
UCS_ERR_UNSUPPORTEDis ignored.UCT_EP_ATTR_FIELD_BANDWIDTHof the lanes of all endpoints on it, so an event needs no endpoint lookup.UCT_PERF_ATTR_FIELD_PATH_BANDWIDTH), which a path without samples reports, and keep only the samples below it, in a worker hash table keyed by the endpoint, soucp_ep_tand the endpoint configuration do not change.UCP_EP_PATH_BW_INVALIDmarks a path which is not degraded.UCP_EP_FLAG_PATH_DEGRADEDis set while a lane has a sample, so part 2 changes the send path only for such endpoints. Endpoint destruction frees the samples without touching the endpoint flags, since it can run without the async lock.Tests:
test_ucp_proto_mock_path_bw*on rc_x and posix,cma: samples follow events, per-process samples, unusable samples, reset on reconfiguration, destroy with a pending update, recovery of a multi-path device, and close of a single-thread worker endpoint while a worker flush holds its last reference (rc_x).