Repository navigation
Conversation
|
🤖 Starting review — findings will be posted here when done. |
f17305e to
dbe6b8b
Compare
| } | ||
| } | ||
|
|
||
| return ucp_proto_common_iface_bandwidth(context, &wiface->attr.bandwidth); |
There was a problem hiding this comment.
In ucp_worker_ep_path_bw_model, the UCP_EP_PATH_BW_OP_PUT fallback uses the full iface bandwidth, but uct_ep_query() reports a single path bandwidth (uct_ep_attr::bandwidth defaults to uct_perf_attr_t::path_bandwidth, and uct_ib_iface_estimate_path_bw() scales it by 1/dev_num_paths). So with dev_num_paths > 1 every lane is permanently degraded, and the ratio sent to the peer is ~1/num_paths, which the peer then applies again on top of its own get path bandwidth. Can we query UCT_PERF_ATTR_FIELD_PATH_BANDWIDTH with UCT_EP_OP_PUT_ZCOPY here too, like the peer-put branch does? The IB_NUM_PATHS=1 in the new test suggests this is already known.
| goto out; | ||
| } | ||
|
|
||
| lane_map = ucp_worker_ep_path_bw_lanes(ep) & |
There was a problem hiding this comment.
In ucp_worker_ep_path_bw_pack: the pack callback recomputes lane_map from the current state->valid[PUT], but the size was validated against max_bcopy back in ucp_worker_ep_path_bw_send(). If the request waits on the pending queue, another progress round can sample more lanes and add bits to valid[PUT], so the pack can write more than the validated size. pls bound the number of packed paths here (and use ucp_ep_get_max_bcopy() for the limit, since it also caps by the peer's seg_size).
| return lpriv; | ||
| } | ||
|
|
||
| for (i = 0; i < mpriv->num_lanes; ++i) { |
There was a problem hiding this comment.
In ucp_proto_rma_adjust_lane: maybe we could cache total_bw/kept_bw/max_frag_ratio (and the per-lane weight/max_frag) in ucp_ep_path_bw_t when the samples change? Right now every fragment of every RMA request on a degraded endpoint walks all lanes twice, while the inputs only change on a sample update (docs/OptimizationStyle.md: cache results that are stable).
| { | ||
| const auto &tls = GetParam().transports; | ||
| ucs_assert(m_self == nullptr); | ||
| m_self = this; |
There was a problem hiding this comment.
| m_self = this; | |
| m_path_bw_self = this; |
minor: test_ucp_proto_mock_path_bw::m_self hides mock_iface::m_self, so this init()/dtor pair looks like it manages the base pointer but actually manages a second one; pls rename.
|
The diff adds well over 500 lines (≈500 in Residual coverage gap: no test exercises Residual coverage gap: no coverage for the bcopy RMA protocols ( Residual coverage gap: no test for the endpoint-close path while a |
|
🤖 Starting review — findings will be posted here when done. |
| return state; | ||
| } | ||
|
|
||
| ucp_worker_ep_path_bw_cleanup(ep); |
There was a problem hiding this comment.
In ucp_worker_ep_path_bw_get: when the state is stale, ucp_worker_ep_path_bw_cleanup() also removes the queued oneshot and the new state starts with valid[PUT] == 0 and pending == 0. On the AM-handler path nothing re-arms sampling: ucp_proto_rma_adjust_lane() calls refresh() only while the state is stale, and for op == PUT it returns early on !(state->valid[op] & mpriv->lane_map). Impact: if the peer's UCP_AM_ID_EP_PATH_BW is handled after the local reconfiguration (very likely, since both sides react to the same link slowdown), the local put samples are dropped and the degraded put path is used at full weight until the next port-speed event.
Can we call ucp_worker_ep_path_bw_refresh() for the newly allocated state here? The cleanup above also cancels a pending refresh, and in the AM handler path nothing re-arms sampling, so the local put samples are lost until the next port speed event.
| return lpriv; | ||
| } | ||
|
|
||
| for (i = 0; i < mpriv->num_lanes; ++i) { |
There was a problem hiding this comment.
In ucp_proto_rma_adjust_lane: both lane loops (total_bw/max_frag_ratio, then kept_bw) plus the kh_get() run per fragment, per lane, for as long as UCP_EP_FLAG_PATH_DEGRADED is set — including when only the opposite direction is degraded, where the function ends up returning lpriv unchanged after the full O(num_lanes) FP loop. Maybe cache the per-lane weight/max_frag (and kept_bw/max_frag_ratio) in ucp_ep_path_bw_t when the samples change? docs/OptimizationStyle.md asks to keep this path free of O(n) work and cache stable results.
| goto out; | ||
| } | ||
|
|
||
| lane_map = ucp_worker_ep_path_bw_lanes(ep) & |
There was a problem hiding this comment.
In ucp_worker_ep_path_bw_pack: pack recomputes the path list from the current state, while the max_bcopy bound was checked in ucp_worker_ep_path_bw_send() for a possibly smaller put_lanes. A request that waits on the pending queue can therefore pack more paths than were validated — e.g. the endpoint is reconfigured with more rma_bw lanes, the new _send() size check fails and returns, and the queued request then packs the larger list into the bcopy buffer. Maybe store the checked lane map in the request and pack only it?
|
|
||
| req = ucp_request_get(ep->worker); | ||
| if (req == NULL) { | ||
| ucs_error("ep %p: failed to allocate path bandwidth request", ep); |
There was a problem hiding this comment.
Minor: request-pool exhaustion here only skips one best-effort sample message; the flow continues as usual, so error is the wrong level per docs/LoggingStyle.md (error = "could not continue as usual"). Suggestion:
| ucs_error("ep %p: failed to allocate path bandwidth request", ep); | |
| ucs_debug("ep %p: failed to allocate path bandwidth request", ep); |
|
Residual coverage gaps: (1) No test covers the interleaving in finding 1 (peer |
|
🤖 CI Triage Agent — Summary: Unable to fetch logs — cannot diagnose without them.
|
eaafe9a to
c98f548
Compare
|
🤖 CI Triage Agent — TL;DR: The gtest job failed on a single hung test — Full analysisSummary: Root cause: From the log: all other Implicated commit: [REDACTED:Hex High Entropy String] — "UCP: Split GET by the put samples of the peer" (Dmitrii), on top of 0f401ed / [REDACTED:Hex High Entropy String] of the same series. File: src/ucp/core/ucp_worker.c:1228-1268 ( Suggested fix:
Related: PR #12335 (this change), PR #12334 "UCP: Keep measured path bandwidth per endpoint", prior reverted attempt: #12002 / #12015, reverted by #12082.
|
3c466ab to
d2feb04
Compare
|
🤖 CI Triage Agent — TL;DR: The gtest run aborted on a debug assertion Full analysisSummary: Root cause: Sequence from the log: the test injected a target failure on lane 1/2, then entered "Checking recovery status..."; during recovery the EP was reconfigured and marked In Implicated commit: d513f98 "UCP/FLUSH: Add topology-safe selective endpoint flush (#11923)" (nbellalou) introduced the partial File: src/ucp/rma/flush.c:371-373 (assertion; reported as Suggested fix: Make the reschedule path tolerant of an already-started flush when the EP is blocked due to failover rather than initial wireup. Concretely, in Related: #12266 (UCP/FLUSH: Flush the other lanes after the pending lane — same area, open), #11923, #12274, #12183
|
d2feb04 to
ac17d46
Compare
ac17d46 to
c5b4e5c
Compare
Keep the samples of each lane per source, so that a source other than the put of this side can split other operations. The put of this side, which splits put, is the only source for now. Setting a sample and the endpoint flag of a degraded path go to helpers shared by the sources.
c5b4e5c to
9c16b4f
Compare
|
🤖 CI Triage Agent — TL;DR: The ASan Full analysisSummary: Build of the AddressSanitizer Root cause: The log shows a normal, continuously progressing ASan build ( Implicated commit: none — not a code failure. The tested commit [REDACTED:Hex High Entropy String] (branch File: n/a (failure originates in the Azure DevOps agent lifecycle, during Suggested fix: Simply re-trigger the Related: none — searches for "agent has received a shutdown signal" and related terms returned only unrelated open PRs (#12317, #12272, #12325, #12194), no existing issue tracking this agent cancellation.
|
|
🤖 Starting review — findings will be posted here when done. |
| !(ep->flags & UCP_EP_FLAG_REMOTE_ID) || | ||
| ((sizeof(ucp_ep_path_bw_hdr_t) + | ||
| (ucs_popcount(lane_map) * sizeof(ucp_ep_path_bw_path_t))) > | ||
| ucp_ep_get_iface_attr(ep, key->am_lane)->cap.am.max_bcopy)) { |
There was a problem hiding this comment.
pls bound the message by ucp_ep_get_max_bcopy(); the local cap.am.max_bcopy ignores the peer's seg_size (key->lanes[].seg_size is the remote limit, see ucp_ep_get_max_bcopy), so with a smaller remote segment size we would post an AM bigger than the receiver's buffer. All other AM bcopy senders use it, and proto_am.inl even asserts against it.
| ucp_ep_get_iface_attr(ep, key->am_lane)->cap.am.max_bcopy)) { | |
| ((sizeof(ucp_ep_path_bw_hdr_t) + | |
| (ucs_popcount(lane_map) * sizeof(ucp_ep_path_bw_path_t))) > | |
| ucp_ep_get_max_bcopy(ep, key->am_lane))) { |
| req->send.ep = ep; | ||
| req->send.datatype = ucp_dt_make_contig(1); | ||
| req->send.uct.func = ucp_worker_ep_path_bw_progress_send; | ||
| state->sync_peer = degraded; |
There was a problem hiding this comment.
sync_peer is cleared here, before the send can fail: if uct_ep_am_bcopy() returns an error other than UCS_ERR_NO_RESOURCE, the recovery message is dropped and the peer keeps applying the stale degraded samples to its get until the next degradation. This also contradicts the comment in ucp_worker_ep_path_bw_progress_send() that the next update sends them again - can we clear it only once the message is actually packed?
|
Residual coverage gaps (beyond those already reported): no test exercises a peer whose AM-lane segment size is smaller than the packed |
9c16b4f to
8ec967c
Compare
|
🤖 Starting review — findings will be posted here when done. |
| ucp_proto_common_lane_priv_init(¶ms->super, mpriv->reg_md_map, lane, | ||
| &lpriv->super); | ||
| /* Path samples are compared with the bandwidth of one path */ | ||
| lpriv->bandwidth = lane_perf->bandwidth * lane_perf->path_ratio; |
There was a problem hiding this comment.
lane_perf->bandwidth is already capped by the local/remote distance in ucp_proto_common_update_lane_perf_by_distance(), while path_ratio was computed from the uncapped iface bandwidth, so this double-counts the cap. The samples it is compared with come from uct_ep_query() and are not capped, so on a PCIe-limited lane (GPU buffers) a degraded path keeps a larger share than it should. Can we keep the absolute path bandwidth in ucp_proto_common_tl_perf_t instead of a ratio?
| return; | ||
| } | ||
|
|
||
| req = ucp_request_get(ep->worker); |
There was a problem hiding this comment.
No guard against stacking path-bandwidth requests: ucp_worker_ep_path_bw_send() allocates and submits a new request on every update while the path is degraded. Please skip the send while a previous request is still outstanding (e.g. a flag in state); uct_ep_am_bcopy() returns UCS_ERR_NO_RESOURCE while the AM lane is a wireup proxy ep (ucp_wireup_ep_t rejects every id except UCP_AM_ID_WIREUP), and then every following path event adds another request to the lane pending queue, with nothing bounding it since these requests are not counted in worker->flush_ops_count, and they all pack identical state once resources free up.
| double bandwidth, selected_bw, min_bw; | ||
| ucp_lane_index_t i; | ||
|
|
||
| for (i = 0; i < mpriv->num_lanes; ++i) { |
There was a problem hiding this comment.
Minor: per-fragment O(num_lanes) recompute. ucp_proto_rma_split_lane() runs two passes over all lanes plus floating-point divisions for every fragment, but total_bw and max_frag_ratio do not depend on length and change only when the samples or the endpoint configuration change, so they could be computed once per samples update; docs/OptimizationStyle.md asks to keep scheduling O(1) in the large-message path.
|
Residual coverage gaps: no test uses a non-host (GPU) buffer, which is the configuration where the distance cap in finding 1 changes the split; and no test drives |
8ec967c to
5f7f7a7
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| ucp_ep_update_flags(ep, 0, UCP_EP_FLAG_PATH_DEGRADED); | ||
| ucp_worker_ep_path_bw_update_flag(ep, state); | ||
| ucp_worker_ep_path_bw_send(ep, state); |
There was a problem hiding this comment.
this sends while ucp_worker_iface_path_bw_progress() iterates worker->all_eps, and uct_ep_am_bcopy() can report a failure inline (e.g. uct_tcp_ep_handle_send_err() -> uct_iface_handle_ep_err()), which ends in ucp_ep_invoke_err_cb(); a user callback that closes the endpoint removes it from all_eps and the loop then walks freed memory. keepalive avoids this with worker->keepalive.iter + ucp_worker_keepalive_remove_ep() - can we collect the endpoints and send after the loop instead?
Impact: potential use-after-free / list corruption on the worker's endpoint list during a path-bandwidth update.
|
Residual coverage gap: no test closes or fails an endpoint from the error callback while a path-bandwidth update is being delivered, which is exactly the interleaving described in the finding above. Residual coverage gap: no test exercises an AM lane on a different transport than the one reporting the path event (the suite uses |
5f7f7a7 to
10b5f94
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| for (lane = 0; lane < ucp_ep_num_lanes(ep); ++lane) { | ||
| for (op = 0; op < UCP_EP_PATH_BW_OP_LAST; ++op) { | ||
| if (state->bandwidth[lane][op] != UCP_EP_PATH_BW_INVALID) { |
There was a problem hiding this comment.
maybe we could keep the degraded state per source instead of one endpoint flag? as it is, an endpoint degraded only in UCP_EP_PATH_BW_OP_PEER_PUT makes every PUT fragment enter ucp_proto_rma_adjust_lane() (hash lookup + two loops over the lanes) only to find degraded == 0 and return lpriv, and that path never clears the flag, so the cost stays as long as the other direction is degraded (and vice versa for GET).
| rsc = &context->tl_rscs[key->lanes[lane].rsc_index]; | ||
| bandwidth = state->bandwidth[lane][UCP_EP_PATH_BW_OP_PUT]; | ||
| if ((bandwidth == UCP_EP_PATH_BW_INVALID) || | ||
| (ucp_worker_iface_path_bw_estimate( |
There was a problem hiding this comment.
minor: ucp_worker_iface_path_bw_estimate() is re-queried here per degraded lane per endpoint, so one path event now does O(eps x degraded lanes) uct_iface_estimate_perf() calls under the async lock instead of the single call ucp_worker_iface_path_bw_progress() already made. The value is stable per iface and operation, so can we cache it on ucp_worker_iface_t (or pass down the estimate the caller already computed)? See "Cache expensive capability queries when the result is stable" in docs/OptimizationStyle.md. The same applies to the call in ucp_worker_ep_path_bw_peer_update().
|
Residual coverage gap: no test covers an endpoint whose two directions are degraded independently (peer-put degraded while local put is healthy and the reverse within one test), which is the state that keeps Residual coverage gap: no test has an endpoint with RMA BW lanes spread over more than one |
10b5f94 to
a6210be
Compare
|
🤖 Starting review — findings will be posted here when done. |
| hdr->ep_id, hdr->num_paths); | ||
| } | ||
|
|
||
| UCP_DEFINE_AM_WITH_PROXY(UINT64_MAX, UCP_AM_ID_EP_PATH_BW, |
There was a problem hiding this comment.
why WITH_PROXY? the counting proxy makes this message count as interface activity, so receiving it activates and progress-enables an interface that was deactivated by adaptive progress, although it carries no data for that interface. UCP_AM_ID_WIREUP, the other UINT64_MAX control AM, uses UCP_DEFINE_AM.
| rsc = &context->tl_rscs[key->lanes[lane].rsc_index]; | ||
| bandwidth = state->bandwidth[lane][UCP_EP_PATH_BW_OP_PUT]; | ||
| if ((bandwidth == UCP_EP_PATH_BW_INVALID) || | ||
| (ucp_worker_iface_path_bw_estimate( |
There was a problem hiding this comment.
maybe compute the path estimate per iface in ucp_worker_ep_path_bw_send() and pass it in the request? here it is a uct_iface_estimate_perf() call per lane from inside the uct_ep_am_bcopy() pack callback, and lanes on the same iface repeat the same query.
|
Residual coverage gap: no test covers the AM-id/dst-version compatibility gate itself ( Residual coverage gap: no test exercises an endpoint whose AM lane is on a transport without Residual coverage gap: |
|
🤖 CI Triage Agent — TL;DR: The ASAN "roce on worker 3" job failed on a single gtest, Full analysisSummary: Root cause: The Azure job log (only the tail is retrievable; the per-test Implicated commit: No regression commit identified; the PR head under test is [REDACTED:Hex High Entropy String] "UCP: Split GET by the put samples of the peer" (Dmitrii), which touches src/ucp/proto/proto_multi.c and src/ucp/core/ucp_worker.c but is gated off for this test's single/degraded-less lane configuration. File: test/gtest/ucp/test_ucp_peer_failure.cc:395 and :421-432 (test body at :601-610); PR code under suspicion: src/ucp/proto/proto_multi.c:114-126 Suggested fix: 1) Re-run the "AddressSanitizer roce" job to confirm flakiness (a single failure out of 9130 with no sanitizer report strongly suggests a race, not a build regression). 2) If it reproduces on this branch, capture the Related: #6762 (udx/test_ucp_peer_failure.zcopy/3 failure), #6203 (dcx/test_ucp_peer_failure.zcopy/3 failure), #12335 (PR under test), #12334, #12362
|
a6210be to
8b37b28
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
In Residual coverage gaps: no test covers losing the peer's |
Split the next PUT fragment by the static shares of the lanes, each scaled down by the ratio of the path bandwidth sample of its lane to the path estimate, so a slower path gets a smaller share, a recovered path gets it back without endpoint reconfiguration, and healthy lanes keep their shares of the static lane model. A lane whose share of the message or fragment per round is below the minimal fragment is skipped. After the configuration or lanes of the endpoint change, the static lane model applies until the next path events, and the first fragment which finds no samples of the current lanes clears the endpoint flag of a degraded path under the async lock, so the next fragments do not look up the samples. The endpoint flag keeps the send path of other endpoints unchanged. The lane private data keeps the bandwidth of one path of the lane, limited by the system distances like the bandwidth of the lane, since the samples are compared with the path estimate, which is a share of the device when it has several paths. The maximal fragment and the weight are computed by the same helpers as in the static lane model; the weight helper gets the share of the lane as a fraction of the weight range, since the shares of nearly stalled paths are far below one. A protocol passes its multi-lane private data to the adjustment, since the private data of other protocols, such as rendezvous, has another layout.
The PUT reconfiguration check runs with a single-thread worker, whose send does not hold the async lock; the force close test moves to a fixture of its own on top of the single-thread one. The number of paths of a mock device can differ between devices, so a split checks that a sample scales the share of its lane only, also when a system distance limits the bandwidth of the device. Samples of a few B/s check the split of nearly stalled paths.
The data of GET from an endpoint crosses the paths on which the peer endpoint puts, so the put samples of the peer must split GET of this side. Check the split and its recovery, the split with two paths per device, a stalled path of the peer, that the put samples of this side do not split GET and the two directions degrade apart, that the samples go to the peer after all endpoints update, and that the path estimates are queried once per port speed.
The data of GET from an endpoint crosses the paths on which the peer puts. The GET side samples these paths poorly: its probes find no room in send queues full of its reads, which wait for read slots, and where they run, they also see its own reads. The peer samples the same paths from endpoints without application work requests. After the put samples of an endpoint are updated, send them to the peer in a UCP_AM_ID_EP_PATH_BW message: per path, the ratio of the sample to the path estimate, with the memory domains and system devices of both sides. The peer scales the path estimate of get on its lanes of these paths by the ratio, keeps the result as a second source of samples, and splits GET fragments by it with the same lane adjustment as PUT. A degraded path is sent on every update, so the peer gets it again after its endpoint reconfiguration, and its recovery once. Only peers of a release which handles the message get it. The samples are sent from progress, after the update of all endpoints, since a failed send can invoke the error callback of the user, which may close endpoints. An endpoint keeps the sources of its degraded samples next to its flag, so an operation looks up the samples only when its own source is degraded. The path estimates of an interface are queried once per port speed.
8b37b28 to
9655ef9
Compare
|
🤖 Starting review — findings will be posted here when done. |
| double path_ratio; | ||
|
|
||
| /* Single path bandwidth, without the limits of the system distances */ | ||
| double path_bandwidth; |
There was a problem hiding this comment.
minor: pls also set path_bandwidth in ucp_proto_common_tl_perf_reset(). That function is the NULL-lane path of ucp_proto_common_get_lane_perf() and already misses path_ratio; now ucp_proto_multi_init_priv() reads path_bandwidth into lpriv->bandwidth, so the field should not be left uninitialized.
| uct_iov_t iov; | ||
| ucs_status_t status; | ||
|
|
||
| lpriv = ucp_proto_rma_lane_priv(req, req->send.proto_config->priv, lpriv, |
There was a problem hiding this comment.
the same adjust-and-skip block now appears 4 times (here, ucp_proto_put_sgl_offload_send_func, ucp_proto_put_sgl_offload_sw_send_func, ucp_proto_get_offload_zcopy_send_func). can we fold it into one helper/macro next to ucp_proto_rma_lane_priv()? the root AGENTS.md asks not to duplicate similar implementation patterns.
What?
Split GET between the RMA lanes of an endpoint by the put samples of the peer. Part 3 of 3 of path bandwidth balancing: #12334 keeps the measured path bandwidth of each endpoint, #12362 splits PUT by it.
Based on #12362: its three commits come first, and this part is the last two commits (+654 / -41, of them +231 / -12 tests). It will be rebased on master after #12362 is merged.
Why?
On systems with two NICs per GPU, other traffic on one NIC slows down every GET of an endpoint to its lane on that NIC, as with PUT. But the GET side samples its paths poorly: its probes find no room in send queues full of its reads, which wait for read slots, and where they run, they also see its own reads. The data of GET crosses the paths on which the peer puts, and the peer samples these paths from endpoints without such work requests.
Two nodes, other traffic on one of the two NICs of the GPU (~75% of its bandwidth), balancing on vs off, single runs of the revision before the first review of #12334:
How?
UCP_AM_ID_EP_PATH_BW: per path, the ratio of the sample to the path estimate, with the memory domain and system device of both sides. The send runs from progress, after the update of all endpoints, since a failed send can invoke the error callback of the user, which may close endpoints. A degraded path is sent on every update, so the peer gets it again after its endpoint reconfiguration, and its recovery once. Only peers which handle the message get it (UCP_MIN_DST_VERSION_EP_PATH_BW).UCT_EP_OP_GET_ZCOPY) by the mean ratio of the matching paths, and keeps the result below the estimate as a second source of samples next to its own put samples. The path estimates of an interface are queried once per port speed.Not covered yet: RMA bcopy and rendezvous protocols.
Tests:
test_ucp_proto_mock_path_bw*on rc_x and posix,cma: GET split by the put samples of the peer and its recovery, the split with two paths per device, GET without samples of the peer, a zero sample of the peer skips its lane, PUT and GET degraded apart, the samples sent after the update of all endpoints, and the path estimates queried once.