Repository navigation
UCP/RMA: Estimate per-lane throughput from multi-rail samples - #12015
Conversation
|
🤖 Starting review — findings will be posted here when done. |
| lane->smoothed = lane->nominal; | ||
| } | ||
|
|
||
| /* A single low observation can reduce the previous estimate by |
There was a problem hiding this comment.
| /* A single low observation can reduce the previous estimate by | |
| /* A single low observation can reduce the previous estimate by | |
| * at most 27%; the EWMA uses the DOCA model's alpha of 0.3. */ |
the 0.1 floor applies to the observation, so the worst case is 0.7 + 0.3*0.1 = 0.73 of the previous value, as the test comment at test_ucp_rma.cc:1835 says.
|
|
||
| rsc_index = ucp_ep_config(ep)->key.lanes[lane_id].rsc_index; | ||
| wiface = ucp_worker_iface(worker, rsc_index); | ||
| perf_attr.field_mask = UCT_PERF_ATTR_FIELD_BANDWIDTH; |
There was a problem hiding this comment.
pls use ucp_worker_iface_estimate_perf() and set UCT_PERF_ATTR_FIELD_OPERATION to PUT_ZCOPY/GET_ZCOPY according to dir. As is, the nominal is direction-agnostic (GET and PUT bandwidth differ on IB/rocm_copy), and uct_iface_estimate_perf() skips the memory-distance adjustment that the ucp_worker_iface_bandwidth() fallback below does include, so the two paths are not comparable.
| struct ucp_rma_bw_estimator { | ||
| uint64_t epoch; | ||
| uint32_t generation; | ||
| ucp_worker_cfg_index_t cfg_index; |
There was a problem hiding this comment.
cfg_index looks redundant: rma_bw_generation is incremented inside ucp_ep_set_cfg_index(), which is the only writer of ep->cfg_index, so generation already covers every cfg change. IMO we can keep epoch + generation only.
| for (i = 0; i < sample->num_lanes; ++i) { | ||
| ucp_trace_req(req, "rma bw sample lane %u/%u: %zu bytes / " | ||
| "%.3f us", i, sample->num_lanes, | ||
| ucs_debug("ep %p req %p: rma bw %s lane %u/%u ep_lane %u: " |
There was a problem hiding this comment.
why switch from ucp_trace_req() to ucs_debug()? per-sample per-lane messages are trace volume (docs/LoggingStyle.md), and ucp_trace_req() already prints the request and ep.
| return estimator; | ||
| } | ||
|
|
||
| void ucp_rma_bw_estimator_free(ucp_ep_h ep) |
There was a problem hiding this comment.
minor: the function frees ucp_rma_bw_ep_state_t (both directions), not one estimator - pls rename to ucp_rma_bw_ep_state_cleanup() and set ep->ext->rma_bw_state = NULL, so it is safe to call outside ucp_ep_deallocate().
| uint32_t generation; | ||
| ucp_worker_cfg_index_t cfg_index; | ||
| ucp_lane_index_t num_lanes; | ||
| ucp_rma_bw_dir_t dir; |
There was a problem hiding this comment.
| ucp_rma_bw_dir_t dir; | |
| ucp_rma_bw_dir_t dir; | |
| unsigned accepted; | |
| unsigned rejected[UCP_RMA_BW_REJECT_LAST]; |
minor: align fields on the column with the rest of the struct.
| { | ||
| ucp_request_t req = {}; | ||
| ucp_rma_bw_sample_t sample = {}; | ||
| std::vector<ucp_rma_bw_sample_t> samples(1); |
There was a problem hiding this comment.
minor: why replace ucp_rma_bw_sample_t sample = {} with a 1-element vector here? = {} already zero-initializes, and this is an unrelated change to an existing test.
|
nothing in src/ calls ucp_rma_bw_estimator_is_valid(), only the gtest. can we add it together with the consumer that will use the estimate? Residual coverage gap: Residual coverage gap: |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: That is exactly +16 bytes (216 → 232), matching the observed mismatch. This is a real failure caused by the PR, not flakiness or infrastructure: the test is deterministic, ran in release config (no debug/stats/assert skip), and the same build's earlier Implicated commit: [REDACTED:Hex High Entropy String] — Leonid Genkin, "UCP/RMA: Estimate per-lane throughput from multi-rail samples" File: Suggested fix: Preferred: avoid growing the hot per-EP structure. Since If the 16-byte increase is genuinely required and reviewed as acceptable, update Related: PR #12015 (branch
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "Codestyle (commit title)" check failed because one commit in PR #12015 has a typo'd title — Full analysisSummary: Azure Pipelines build 137939, job "commit title" (Codestyle), exited with code 1 after reporting Root cause: Not a product or infrastructure bug — a commit message formatting violation. Implicated commit: File: Suggested fix: Reword the offending commit title to remove the duplicated colon, then force-push the branch to re-trigger the PR pipeline: If it is the tip commit, Optionally, consider having the check emit the expected pattern alongside "Bad commit title" so the required format is self-evident from the log; as written, the error message gives no hint about what rule was violated. Related: PR #12015; check implemented by
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: This isn't a code failure — the commit-title codestyle gate rejected one of PR #12015's commits because its title is Full analysisSummary: The Root cause:
The bad title has two colons. The character class Secondary (CI script) issue visible in the log: the step runs with Implicated commit: The PR commit titled File: Suggested fix:
Related: PR #12015; style guideline referenced by the check: https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines
|
8acd658 to
91b19b6
Compare
|
🤖 Starting review — findings will be posted here when done. |
| ucp_rma_bw_sample_complete); | ||
| } | ||
|
|
||
| if (ucs_unlikely(req->send.ep->worker->context->config.ext.rma_bw_measure)) { |
There was a problem hiding this comment.
can we set a request flag in ucp_rma_bw_sample_try_start() instead? this adds a 3-level deref (req->send.ep->worker->context->config.ext) plus a branch to every progress call of the non-measuring path, while the flag is already available at init time. (same pattern in put_offload.c:347)
| request_rate = observation->bytes / | ||
| ucs_time_to_sec(observation->last_comp - | ||
| observation->first_post); | ||
| stream_rate = (observation->posted_total - lane->marker_posted) / |
There was a problem hiding this comment.
posted_total counts bytes submitted to UCT, but the denominator is a completion-time window (last_comp - marker_time), so bytes still in flight at last_comp inflate the rate. a backlogged lane then looks faster than it completes, which is the opposite of what the estimate should show - shouldn't we accumulate completed bytes per lane instead?
| * structure. TODO allocate this array dynamically. | ||
| */ | ||
| uct_ep_h *uct_eps; | ||
| struct ucp_rma_bw_ep_state *rma_bw_state; |
There was a problem hiding this comment.
can we keep this state in a worker-level hash keyed by ep (like ucp_worker_remote_flush_hash) instead of growing ucp_ep_ext_t? RMA_BW_MEASURE is experimental and off by default, so every endpoint pays 8 bytes for it (see docs/OptimizationStyle.md: avoid enlarging the endpoint structure).
| return UCP_RMA_BW_REJECT_NONE; | ||
| } | ||
|
|
||
| static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id, |
There was a problem hiding this comment.
| static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id, | |
| static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id, | |
| ucp_rma_bw_dir_t dir) |
minor: arguments should be indented on column; same for the perf_attr.operation ternary and the lane->nominal assignment below.
| estimator->lanes[i].smoothed / UCS_MBYTE); | ||
| } | ||
| } else { | ||
| ucp_trace_req(req, "rma bw %s sample rejected: reason %u, " |
There was a problem hiding this comment.
minor: pls print the reject reason name instead of %u - this trace is the only user-visible output of the feature, and a bare enum value requires opening the header to decode.
|
Notes not posted: Residual gap: Notes not posted: Notes not posted: Verified the new |
|
🤖 CI Triage Agent — TL;DR: The build never got past Full analysisSummary: Azure Pipelines "Check" job (build 137987) failed in the "Get sources" task; all 5 retries × 3 attempts of Root cause: The agent machine The agent correctly passes its own credential via
Note: the agent's GitHub credential (and/or the pipeline's GitHub service connection PAT) appears to be expired or revoked and should be rotated, not just re-read. Implicated commit: unknown — no repo commit is implicated; the failure precedes checkout. ( File: N/A — failure is in the Azure Pipelines Suggested fix: Infrastructure action on the agent host, then re-run the build:
Related: none — no matching issues/PRs found for this checkout-authentication signature in openucx/ucx.
|
|
🤖 CI Triage Agent — TL;DR: The job never built anything — it died in the Full analysisSummary: Root cause: Authentication for the GitHub fetch did not work on this agent. Every attempt ran: Implicated commit: unknown — not a source regression. Commit File: none in the repo — failure is in the Azure Pipelines Suggested fix:
Related: none — searches for this checkout/credential signature in openucx/ucx issues and PRs returned no matching reports.
|
|
🤖 Starting review — findings will be posted here when done. |
evgeny-leksikov
left a comment
There was a problem hiding this comment.
The main concern is that the tracker works per EP and do not utilizes shared capabilities of EP config.
- code style is not accurate across the PR
| ucp_worker_rma_bw_key_is_equal(ucp_ep_h ep1, ucp_ep_h ep2) | ||
| { | ||
| return ep1 == ep2; | ||
| } |
There was a problem hiding this comment.
- maybe per EP config? I think EPs that share the same config should reflect the same way
- if perf has low granularity then it could be part of config
There was a problem hiding this comment.
I tried grouping by ep config and remote worker. Two EPs w same key benefited under contention, but NIXL creates one EP per such group, so it added state w/o benefit. I suggest keeping measurements per EP in this PR.
There was a problem hiding this comment.
does it add complexity or overhead?
I suggest keeping measurements per EP in this PR
sounds good as an initial approach
| uct_completion_t *comp = &req->send.state.uct_comp; | ||
| const ucp_proto_multi_priv_t *mpriv; | ||
| ucp_rma_bw_frag_t *frag; | ||
| ucp_rma_bw_frag_t *frag = NULL; |
There was a problem hiding this comment.
initiated var goes first
| ucp_rma_bw_frag_posted(frag, iov.length, status); | ||
| } | ||
| if (mode != UCP_RMA_BW_POST_NONE) { | ||
| const ucp_proto_multi_priv_t *mpriv = req->send.proto_config->priv; |
There was a problem hiding this comment.
pls move var definition to begin of the function
|
🤖 Starting review — findings will be posted here when done. |
evgeny-leksikov
left a comment
There was a problem hiding this comment.
LGTM but code style fixes are required
|
🤖 CI Triage Agent — TL;DR: The ASan/roce gtest job failed on a single test, Full analysisSummary: Root cause: This is a hang, not a slow test. The build log's "TOP-20 longest tests" shows the failing test consumed 678 217 ms, while the #2 longest test took only 18 667 ms — a ~36x outlier. 678 s is almost exactly 3–4 multiples of the gtest request-wait deadline ( The failing variant is the worst-case combination: A contributing test-infra weakness: Evidence this is not the PR's fault: PR #12015 ("UCP/RMA: Estimate per-lane throughput from multi-rail samples") touches only Implicated commit: unknown — no commit in this PR touches the tag/peer-failure path exercised by the test; the failure matches a pre-existing flake. File: Suggested fix: Re-run the job to confirm it's the known flake — do not raise the job time limit, since the log shows a hang rather than steady progress. To make this diagnosable and stop it blocking unrelated PRs: (1) bound the spin in Related: #7025, #6341, #6320 (same test hanging/failing on rc, tcp, ud); PR under test: #12015, predecessor #12002
|
|
🤖 Starting review — findings will be posted here when done. |
What?
Estimate per-lane throughput from multi-rail samples.
Why?
Dynamic load-balancing.