Skip to content

UCP/RMA: Estimate per-lane throughput from multi-rail samples - #12015

Merged
gleon99 merged 6 commits into
openucx:masterfrom
gleon99:rma-measured-estimator-step2
Oct 6, 2026
Merged

gleon99 merged 6 commits into
openucx:masterfrom
gleon99:rma-measured-estimator-step2

Conversation

@gleon99

@gleon99 gleon99 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What?

Estimate per-lane throughput from multi-rail samples.

Why?

Dynamic load-balancing.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/rma_bw.c
lane->smoothed = lane->nominal;
}

/* A single low observation can reduce the previous estimate by

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/* A single low observation can reduce the previous estimate by
/* A single low observation can reduce the previous estimate by
* at most 27%; the EWMA uses the DOCA model's alpha of 0.3. */

the 0.1 floor applies to the observation, so the worst case is 0.7 + 0.3*0.1 = 0.73 of the previous value, as the test comment at test_ucp_rma.cc:1835 says.

Comment thread src/ucp/rma/rma_bw.c Outdated

rsc_index = ucp_ep_config(ep)->key.lanes[lane_id].rsc_index;
wiface = ucp_worker_iface(worker, rsc_index);
perf_attr.field_mask = UCT_PERF_ATTR_FIELD_BANDWIDTH;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls use ucp_worker_iface_estimate_perf() and set UCT_PERF_ATTR_FIELD_OPERATION to PUT_ZCOPY/GET_ZCOPY according to dir. As is, the nominal is direction-agnostic (GET and PUT bandwidth differ on IB/rocm_copy), and uct_iface_estimate_perf() skips the memory-distance adjustment that the ucp_worker_iface_bandwidth() fallback below does include, so the two paths are not comparable.

Comment thread src/ucp/rma/rma_bw.h Outdated
struct ucp_rma_bw_estimator {
uint64_t epoch;
uint32_t generation;
ucp_worker_cfg_index_t cfg_index;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cfg_index looks redundant: rma_bw_generation is incremented inside ucp_ep_set_cfg_index(), which is the only writer of ep->cfg_index, so generation already covers every cfg change. IMO we can keep epoch + generation only.

Comment thread src/ucp/rma/rma_bw.c Outdated
for (i = 0; i < sample->num_lanes; ++i) {
ucp_trace_req(req, "rma bw sample lane %u/%u: %zu bytes / "
"%.3f us", i, sample->num_lanes,
ucs_debug("ep %p req %p: rma bw %s lane %u/%u ep_lane %u: "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why switch from ucp_trace_req() to ucs_debug()? per-sample per-lane messages are trace volume (docs/LoggingStyle.md), and ucp_trace_req() already prints the request and ep.

Comment thread src/ucp/rma/rma_bw.c Outdated
return estimator;
}

void ucp_rma_bw_estimator_free(ucp_ep_h ep)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the function frees ucp_rma_bw_ep_state_t (both directions), not one estimator - pls rename to ucp_rma_bw_ep_state_cleanup() and set ep->ext->rma_bw_state = NULL, so it is safe to call outside ucp_ep_deallocate().

Comment thread src/ucp/rma/rma_bw.h Outdated
uint32_t generation;
ucp_worker_cfg_index_t cfg_index;
ucp_lane_index_t num_lanes;
ucp_rma_bw_dir_t dir;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ucp_rma_bw_dir_t dir;
ucp_rma_bw_dir_t dir;
unsigned accepted;
unsigned rejected[UCP_RMA_BW_REJECT_LAST];

minor: align fields on the column with the rest of the struct.

Comment thread test/gtest/ucp/test_ucp_rma.cc Outdated
{
ucp_request_t req = {};
ucp_rma_bw_sample_t sample = {};
std::vector<ucp_rma_bw_sample_t> samples(1);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: why replace ucp_rma_bw_sample_t sample = {} with a 1-element vector here? = {} already zero-initializes, and this is an unrelated change to an existing test.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucp/rma/rma_bw.c

nothing in src/ calls ucp_rma_bw_estimator_is_valid(), only the gtest. can we add it together with the consumer that will use the estimate?

Residual coverage gap: ucp_rma_bw_estimator_prepare() / ucp_rma_bw_nominal() are not covered by any test; the new gtests exercise only ucp_rma_bw_estimator_update() / _is_valid() on hand-built structs. The epoch/generation/cfg_index invalidation and the realloc-on-num_lanes-change path are therefore untested against a real endpoint.

Residual coverage gap: sample->concurrent requires every lane to have an in-flight fragment simultaneously; only the 3-lane helper test covers it. Worth confirming with real hardware runs (test_ucp_rma_bw.put_get_enabled) that samples are actually accepted and not always rejected as UNSATURATED.

@svc-ucx

svc-ucx commented Oct 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit 480ac70d

TL;DR: The test_obj_size.size gtest failed because PR #12015's commit 480ac70 added two new fields (rma_bw_state pointer + rma_bw_generation) to ucp_ep_ext_t, growing it from 216 to 232 bytes while the hard-coded expected size in the test was not updated. Either pack the new state into the existing structure/allocate it elsewhere, or bump the expected size constant.

Full analysis

Summary: test_obj_size.size failed in the release gtest run with Expected equality of these values: (size_t)216 ... sizeof(ucp_ep_ext_t) Which is: 232, causing make -C test/gtest test to exit 1 and the "althca on worker 0" job to fail.

Root cause: test/gtest/common/test_obj_size.cc:53 asserts sizeof(ucp_ep_ext_t) == 216 as a guard against unintended growth of the per-endpoint structure. The PR commit 480ac70 ("UCP/RMA: Estimate per-lane throughput from multi-rail samples") added to ucp_ep_ext_t:

struct ucp_rma_bw_ep_state   *rma_bw_state;      /* 8 bytes */
uint32_t                     rma_bw_generation;  /* 4 bytes + 4 padding */

That is exactly +16 bytes (216 → 232), matching the observed mismatch. This is a real failure caused by the PR, not flakiness or infrastructure: the test is deterministic, ran in release config (no debug/stats/assert skip), and the same build's earlier test_ucp_config step error (gcc: error: test_ucp_config-test_ucp_config.o: No such file or directory) was non-fatal and unrelated to this assertion.

Implicated commit: [REDACTED:Hex High Entropy String] — Leonid Genkin, "UCP/RMA: Estimate per-lane throughput from multi-rail samples"

File: src/ucp/core/ucp_ep.h:610-611 (new fields); assertion at test/gtest/common/test_obj_size.cc:53

Suggested fix: Preferred: avoid growing the hot per-EP structure. Since rma_bw_state is already an out-of-line pointer, move rma_bw_generation into the pointed-to ucp_rma_bw_ep_state struct, which removes 8 bytes (4 + 4 padding) of growth. Better still, if rma_bw_generation is only meaningful when rma_bw_state != NULL, it belongs there unconditionally. Alternatively, reclaim space by packing rma_bw_generation into an existing padding hole near the smaller fields (e.g. adjacent to ucp_lane_map_t unflushed_lanes / flush_sys_dev_map) rather than between two 8-byte-aligned members.

If the 16-byte increase is genuinely required and reviewed as acceptable, update test/gtest/common/test_obj_size.cc:53 to EXPECTED_SIZE(ucp_ep_ext_t, 232) in the same commit, and note the justification in the commit message — this test exists specifically to force that conversation, so it should not be bumped silently.

Related: PR #12015 (branch rma-measured-estimator-step2). The expected-size constants were last touched in #11647 (Aug 2026), so no pending PR already adjusts this value — the test and the new fields are out of sync in this branch alone.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 51e31a3f-97d7-4027-b988-8aea5e0e6cb7 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle commit title) · commit 9e8e0cba

TL;DR: The "Codestyle (commit title)" check failed because one commit in PR #12015 has a typo'd title — UCP/RMA:: refine measured RMA bandwidth estimator and review fixes — with a doubled colon, which fails the required PREFIX: Text pattern. Amend/reword that commit title to a single colon (e.g. UCP/RMA: Refine measured RMA bandwidth estimator and review fixes) and force-push.

Full analysis

Summary: Azure Pipelines build 137939, job "commit title" (Codestyle), exited with code 1 after reporting Bad commit title: 'UCP/RMA:: refine measured RMA bandwidth estimator and review fixes'.

Root cause: Not a product or infrastructure bug — a commit message formatting violation. codestyle_check_commit_title() validates every non-merge, first-parent commit in the PR range against the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_], i.e. an uppercase/slash-separated prefix, exactly one colon, then a space, then an alphanumeric character. The commit title contains a doubled colon (UCP/RMA::): the character class consumes UCP/RMA, the literal : consumes the first colon, and the regex then requires a space but encounters the second colon, so the match fails and err=1 is returned. The log confirms the other commit in the same PR (UCP/RMA: Estimate per-lane throughput from multi-rail samples) passed, isolating the failure to the single malformed title. Build 137939 ran against merge commit 40175c8b0 (PR head [REDACTED:Hex High Entropy String], branch rma-measured-estimator-step2).

Implicated commit: [REDACTED:Hex High Entropy String] (PR #12015 head, branch rma-measured-estimator-step2) — the commit whose title is UCP/RMA:: refine measured RMA bandwidth estimator and review fixes

File: buildlib/tools/codestyle.sh:14 (the enforcing regex); the defect itself is in the commit message, not in any tracked source file

Suggested fix: Reword the offending commit title to remove the duplicated colon, then force-push the branch to re-trigger the PR pipeline:

git rebase -i [REDACTED:Hex High Entropy String]
# mark the offending commit as 'reword', then set the title to:
#   UCP/RMA: Refine measured RMA bandwidth estimator and review fixes
git push --force-with-lease origin rma-measured-estimator-step2

If it is the tip commit, git commit --amend is sufficient. Note that the check iterates over all commits in the range, so every commit must satisfy PREFIX: Text — verify locally before pushing with:

git log --first-parent --no-merges origin/master..HEAD --format="%s"

Optionally, consider having the check emit the expected pattern alongside "Bad commit title" so the required format is self-evident from the log; as written, the error message gives no hint about what rule was violated.

Related: PR #12015; check implemented by codestyle_check_commit_title in buildlib/tools/codestyle.sh, with the PR commit range scoping added in cfd9cd1 "BUILD/CI: Limit PR commit range checks (#11527)" by Yossi Itigin

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3ca983c3-d376-4a22-a429-9efa63960826 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Oct 3, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle commit title) · commit 8acd6580

TL;DR: This isn't a code failure — the commit-title codestyle gate rejected one of PR #12015's commits because its title is UCP/RMA:: refine measured RMA bandwidth estimator and review fixes with a doubled colon (::); reword that commit to UCP/RMA: Refine ... and the job passes.

Full analysis

Summary: The commit title job of "UCX PR (Codestyle)" failed (Bash exited with code '1') because one commit in the PR has a malformed title prefix.

Root cause: codestyle_check_commit_title validates each non-merge commit title in base..head against the regex ^[0-9A-Z/_-]+:\ [[:alnum:]_]. The log shows three commits checked:

  • OK: UCP/RMA: Refine measured RMA bandwidth estimator and review
  • BAD: UCP/RMA:: refine measured RMA bandwidth estimator and review fixes
  • OK: UCP/RMA: Estimate per-lane throughput from multi-rail samples

The bad title has two colons. The character class [0-9A-Z/_-]+ does not include :, so the prefix UCP/RMA: followed by a second : never matches : + alphanumeric, and the function returns 1.

Secondary (CI script) issue visible in the log: the step runs with set -eE, so the non-zero return from codestyle_check_commit_title aborts the script immediately. The following if [[ $? -ne 0 ]] block never executes, which is why the log shows a bare ##[error]Bash exited with code '1' instead of the intended "Bad commit title(s), see " annotation (note the blank line right before the error at 05:34:38.2235).

Implicated commit: The PR commit titled UCP/RMA:: refine measured RMA bandwidth estimator and review fixes on branch rma-measured-estimator-step2 (PR head [REDACTED:Hex High Entropy String], merge commit 874d2794e). Exact SHA not printed by the log — the check only echoes titles.

File: buildlib/tools/codestyle.sh:14 (the validating regex); failing input is the commit title itself. CI wrapper: buildlib/pr/codestyle.yml:18-30.

Suggested fix:

  1. Primary — fix the commit title. Interactive rebase and reword, e.g.:
    git rebase -i [REDACTED:Hex High Entropy String]
    # mark the offending commit as 'reword', change title to:
    #   UCP/RMA: Refine measured RMA bandwidth estimator and review fixes
    git push --force-with-lease
    
    Since this and the first commit are near-duplicates ("...and review" / "...and review fixes"), consider squashing them into one commit instead.
  2. Optional CI hardening — in buildlib/pr/codestyle.yml, the if [[ $? -ne 0 ]] check is dead code under set -eE. Change line 24 to capture the failure, e.g.:
    if ! codestyle_check_commit_title "$base_revision..$head_revision"; then
        url="https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines"
        echo "##vso[task.logissue type=error]Bad commit title(s), see $url for more info."
        echo "##vso[task.complete result=Failed;]"
    fi
    so contributors get the actionable message and wiki link instead of an opaque exit code.

Related: PR #12015; style guideline referenced by the check: https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 41d92d41-0e9c-4bc9-a985-b7477c56f7d1 in the triage console for the audit trail.

@gleon99
gleon99 force-pushed the rma-measured-estimator-step2 branch from 8acd658 to 91b19b6 Compare October 3, 2026 12:02
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/rma/get_offload.c Outdated
ucp_rma_bw_sample_complete);
}

if (ucs_unlikely(req->send.ep->worker->context->config.ext.rma_bw_measure)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we set a request flag in ucp_rma_bw_sample_try_start() instead? this adds a 3-level deref (req->send.ep->worker->context->config.ext) plus a branch to every progress call of the non-measuring path, while the flag is already available at init time. (same pattern in put_offload.c:347)

Comment thread src/ucp/rma/rma_bw.c
request_rate = observation->bytes /
ucs_time_to_sec(observation->last_comp -
observation->first_post);
stream_rate = (observation->posted_total - lane->marker_posted) /

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

posted_total counts bytes submitted to UCT, but the denominator is a completion-time window (last_comp - marker_time), so bytes still in flight at last_comp inflate the rate. a backlogged lane then looks faster than it completes, which is the opposite of what the estimate should show - shouldn't we accumulate completed bytes per lane instead?

Comment thread src/ucp/core/ucp_ep.h Outdated
* structure. TODO allocate this array dynamically.
*/
uct_ep_h *uct_eps;
struct ucp_rma_bw_ep_state *rma_bw_state;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we keep this state in a worker-level hash keyed by ep (like ucp_worker_remote_flush_hash) instead of growing ucp_ep_ext_t? RMA_BW_MEASURE is experimental and off by default, so every endpoint pays 8 bytes for it (see docs/OptimizationStyle.md: avoid enlarging the endpoint structure).

Comment thread src/ucp/rma/rma_bw.c Outdated
return UCP_RMA_BW_REJECT_NONE;
}

static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id,
static double ucp_rma_bw_nominal(ucp_ep_h ep, ucp_lane_index_t lane_id,
ucp_rma_bw_dir_t dir)

minor: arguments should be indented on column; same for the perf_attr.operation ternary and the lane->nominal assignment below.

Comment thread src/ucp/rma/rma_bw.c Outdated
estimator->lanes[i].smoothed / UCS_MBYTE);
}
} else {
ucp_trace_req(req, "rma bw %s sample rejected: reason %u, "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls print the reject reason name instead of %u - this trace is the only user-visible output of the feature, and a bare enum value requires opening the header to decode.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Notes not posted: Residual gap: ucp_rma_bw_estimator_prepare() / ucp_rma_bw_nominal() / the realloc-on-num_lanes-change path are still exercised only indirectly; the new gtest covers ucp_rma_bw_estimator_update() on hand-built structs. (Already raised in an earlier round, so only noted here.)

Notes not posted: ucp_rma_bw_estimator_prepare() frees the estimator whenever num_lanes changes, so a workload mixing message sizes (different selected lane count) drops all accumulated history. Left out as it overlaps the existing "add it with the consumer" thread.

Notes not posted: Verified the new frag != NULL guards cover the UCP_RMA_BW_MAX_FRAGS overflow path, lane->pending/active_lanes bookkeeping matches frag_posted/frag_complete, UCS_MASK(64) comparison against ucp_lane_map_t is correct, cleanup in ucp_ep_deallocate() runs before ucs_free(ep->ext), and the hard-coded EWMA expectations in the new gtest are exact rather than approximate.

@svc-ucx

svc-ucx commented Oct 3, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX snapshot · commit 130ede38

TL;DR: The build never got past Checkout openucx/ucx@refs/pull/12015/merge — git fetch failed 15 times with Authentication failed for 'https://github.com/openucx/ucx/' because the Azure agent swx-rdmz-instinct02-02 has a stale gh auth git-credential credential helper in its global git config that hijacks the credential request. This is a CI infrastructure/agent problem, not a regression in PR #12015.

Full analysis

Summary: Azure Pipelines "Check" job (build 137987) failed in the "Get sources" task; all 5 retries × 3 attempts of git fetch against https://github.com/openucx/ucx returned exit code 128 with Invalid username or token.

Root cause: The agent machine swx-rdmz-instinct02 has the GitHub CLI configured as a global git credential helper. Every fetch logs:

gh auth git-credential: "erase" operation not supported
remote: Invalid username or token. Password authentication is not supported for Git operations.
fatal: Authentication failed for 'https://github.com/openucx/ucx/'

The agent correctly passes its own credential via --config-env=http.extraheader=..., but the gh credential helper supplies an expired/invalid GitHub CLI token that wins over (or conflicts with) the pipeline-supplied header, so GitHub rejects the request. Evidence it is environmental, not code-related:

  • The repository was never cloned (fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree on every retry), so zero code from commit 130ede38 was ever compiled or executed.
  • Failure is 100 % deterministic across all 15 fetch attempts over ~2 minutes, with no large time gaps (so not a hang/timeout).
  • The error originates from the remote (remote: Invalid username or token), i.e. the credential presented is bad — independent of branch or PR content.

Note: the agent's GitHub credential (and/or the pipeline's GitHub service connection PAT) appears to be expired or revoked and should be rotated, not just re-read.

Implicated commit: unknown — no repo commit is implicated; the failure precedes checkout. ([REDACTED:Hex High Entropy String] on rma-measured-estimator-step2 is incidental.)

File: N/A — failure is in the Azure Pipelines Checkout openucx/ucx@refs/pull/12015/merge (Get sources) task on agent swx-rdmz-instinct02-02, workspace [REDACTED:Base64 High Entropy String]; no repository file involved.

Suggested fix: Infrastructure action on the agent host, then re-run the build:

  1. On swx-rdmz-instinct02 (user swx-azure-svc), remove the interfering GitHub CLI credential helper so the pipeline-injected http.extraheader is used:
    git config --global --unset-all credential.helper
    # verify nothing references gh:
    git config --global --list | grep -i credential
    
    (Check /home/swx-azure-svc/.gitconfig and any /etc/gitconfig entry such as helper = !gh auth git-credential / credential.https://github.1485827954.workers.dev.helper.)
  2. Rotate the credentials: refresh/renew the expired gh token (gh auth logout && gh auth login, or delete ~/.config/gh/hosts.yml) and regenerate the PAT behind the pipeline's GitHub service connection, since GitHub is reporting it as invalid.
  3. As a belt-and-braces guard, set credential.helper= (empty) for the agent's git invocations or add GIT_CONFIG_NOSYSTEM=1/GCM_* isolation in the agent environment so host credential helpers can never shadow Azure's extraheader.
  4. If other jobs on the same pool are green, take swx-rdmz-instinct02-02 out of rotation until step 1–2 are done, then re-queue build 137987 to get a real signal for PR UCP/RMA: Estimate per-lane throughput from multi-rail samples #12015.

Related: none — no matching issues/PRs found for this checkout-authentication signature in openucx/ucx.

🛡️ This comment had 2 potential secret(s) redacted (Base64 High Entropy String, Hex High Entropy String). See request_id 22fc1dec-06e6-47e5-8890-4f251b1c709b in the triage console for the audit trail.

@svc-ucx

svc-ucx commented Oct 3, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_icc_pgi centos7_x86_64) · commit 130ede38

TL;DR: The job never built anything — it died in the Checkout openucx/ucx@refs/pull/12015/merge step because git fetch could not authenticate (all 5 retries failed with exit 128, falling back to a missing gh credential helper on agent swx-rdmz-instinct01). This is a CI agent/infrastructure problem, unrelated to PR #12015; re-run the job (ideally after fixing the agent's git credential config).

Full analysis

Summary: build_icc_pgi centos7_x86_64 failed during source checkout: fatal: could not read Username for 'https://github.com': terminal prompts disabled → ##[error]Git fetch failed with exit code: 128, after 5 full RetryHelper attempts (21:52:01 → 21:54:10).

Root cause: Authentication for the GitHub fetch did not work on this agent. Every attempt ran:
git --config-env=http.extraheader=env_var_http.extraheader fetch ... origin --depth=100 +[REDACTED:Hex High Entropy String]:refs/remotes/origin/1a5b947...
and each time git immediately fell through to a credential helper that does not exist on the agent:
/usr/bin/gh auth git-credential get: 1: /usr/bin/gh: not found
followed by fatal: could not read Username for 'https://github.com'. So the injected http.extraheader auth was not accepted/used, git fell back to the stale credential.helper pointing at /usr/bin/gh (gh CLI absent on swx-rdmz-instinct01), and with GIT_TERMINAL_PROMPT=0 the fetch aborted. No compile step ever started — there is zero UCX source code involvement, and nothing in the log touches the rma-measured-estimator-step2 changes. (A secondary, pre-existing and non-fatal issue appears earlier: the centos7:4 image's glibc is too old for Node 20, so the agent logged ##[warning]Can not run Node 20 in container. Falling back to Node 16.)

Implicated commit: unknown — not a source regression. Commit 130ede38 / PR #12015 is not implicated; the failure precedes checkout completion. Recent buildlib/ commits (latest d5b6b278, Alexey Rivkin) do not touch checkout/credentials.

File: none in the repo — failure is in the Azure Pipelines Get sources task on agent swx-rdmz-instinct01 (job a26acaf5-8863-5daa-3ca4-2ecc9e8a2c22, build 137990).

Suggested fix:

  1. Simply re-run the build_icc_pgi centos7_x86_64 job — if it lands on a healthy agent it should pass, which also confirms the diagnosis.
  2. Fix the agent: remove the stale global credential.helper entry referencing /usr/bin/gh (e.g. git config --global --unset-all credential.helper in the agent's and container image's /etc/gitconfig / ~/.gitconfig), or install the gh CLI. With no bogus helper, the http.extraheader token path is used directly.
  3. Have the CI admin re-authorize/rotate the GitHub service connection credential used by this pipeline (do not paste the token value anywhere) — an expired/invalid token is the most likely reason git skipped the extraheader and fell back to the helper at all.
  4. Optional hardening: bump the centos7 image (or pin the agent's Node version) so the Can not run Node 20 in container glibc warning stops masking real container-init issues.

Related: none — searches for this checkout/credential signature in openucx/ucx issues and PRs returned no matching reports.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1bae5709-6d79-498e-a0d2-7b4171bca00f in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main concern is that the tracker works per EP and do not utilizes shared capabilities of EP config.

  • code style is not accurate across the PR

Comment on lines +88 to +91
ucp_worker_rma_bw_key_is_equal(ucp_ep_h ep1, ucp_ep_h ep2)
{
return ep1 == ep2;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. maybe per EP config? I think EPs that share the same config should reflect the same way
  2. if perf has low granularity then it could be part of config

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried grouping by ep config and remote worker. Two EPs w same key benefited under contention, but NIXL creates one EP per such group, so it added state w/o benefit. I suggest keeping measurements per EP in this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it add complexity or overhead?

I suggest keeping measurements per EP in this PR

sounds good as an initial approach

Comment thread src/ucp/rma/get_offload.c
uct_completion_t *comp = &req->send.state.uct_comp;
const ucp_proto_multi_priv_t *mpriv;
ucp_rma_bw_frag_t *frag;
ucp_rma_bw_frag_t *frag = NULL;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initiated var goes first

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread src/ucp/rma/put_offload.c Outdated
ucp_rma_bw_frag_posted(frag, iov.length, status);
}
if (mode != UCP_RMA_BW_POST_NONE) {
const ucp_proto_multi_priv_t *mpriv = req->send.proto_config->priv;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls move var definition to begin of the function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

evgeny-leksikov
evgeny-leksikov previously approved these changes Oct 6, 2026

@evgeny-leksikov evgeny-leksikov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but code style fixes are required

@svc-ucx

svc-ucx commented Oct 6, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit 254980bf

TL;DR: The ASan/roce gtest job failed on a single test, ud/test_ucp_sockaddr_protocols_err.tag_zcopy_64k_unexp_sync/17, which hung for 678 s (vs. 18 s for the next-longest test) while waiting for sync-tag-send requests to complete after a forced bidirectional disconnect — a known pre-existing flake in this test family, not caused by the PR's RMA estimator changes.

Full analysis

Summary: make test exited with code 1 because 1 of 8789 gtests failed: ud/test_ucp_sockaddr_protocols_err.tag_zcopy_64k_unexp_sync/17, with GetParam() = ud_v,cuda_copy,rocm_copy/bidi_stop,not_all_devs,mt.

Root cause: This is a hang, not a slow test. The build log's "TOP-20 longest tests" shows the failing test consumed 678 217 ms, while the #2 longest test took only 18 667 ms — a ~36x outlier. 678 s is almost exactly 3–4 multiples of the gtest request-wait deadline (ucs::test_timeout_in_sec = 180. in test/gtest/common/test_helpers.cc:29), meaning the test repeatedly burned full 180 s deadlines in request_wait()/requests_wait() (test/gtest/ucp/ucp_test.cc:289-301, which logs request %p did not complete on time) rather than being killed by the 900 s watchdog (watchdog_timeout = 900., line 31).

The failing variant is the worst-case combination: bidi_stop (SEND_STOP | RECV_STOP, test_ucp_sockaddr.cc:3049-3051) force-closes both sender and receiver EPs mid-transfer, with a 64 KB sync tag send over ud_v (ZCOPY_THRESH=2k, RNDV_THRESH=inf, so eager zcopy), in multi-threaded mode, GPU-aware (cuda_copy,rocm_copy), under AddressSanitizer. After both endpoints are force-closed, the ucp_tag_send_sync_nbx request and/or the error callback never complete, so requests_wait(reqs) (test_ucp_sockaddr.cc:2402), wait_for_flag(&m_err_count) (line 2407) and extra_send_before_disconnect() (line 2331) each stall to their deadlines.

A contributing test-infra weakness: test_ucp_sockaddr_protocols::disconnect() at test_ucp_sockaddr.cc:2590-2595 spins while (m_err_count == 0) { test.short_progress_loop(); } with no deadline at all, so a missing error callback produces an unbounded spin instead of a prompt, diagnosable failure.

Evidence this is not the PR's fault: PR #12015 ("UCP/RMA: Estimate per-lane throughput from multi-rail samples") touches only src/ucp/rma/* and src/ucp/core/ucp_ep.{c,h} (commits [REDACTED:Hex High Entropy String], 130ede38, 91b19b6e, d896a54a, 480ac70d, all by Leonid Genkin). The failing test registers only UCP_FEATURE_TAG (test_ucp_sockaddr.cc:3044) and performs no RMA operation whatsoever — it exercises the tag eager-zcopy + peer-failure path. Identical failures of this exact test on other transports are long-standing open issues (#7025, #6341, #6320).

Implicated commit: unknown — no commit in this PR touches the tag/peer-failure path exercised by the test; the failure matches a pre-existing flake.

File: test/gtest/ucp/test_ucp_sockaddr.cc:3103 (the failing test), with the unbounded wait at test/gtest/ucp/test_ucp_sockaddr.cc:2590-2595

Suggested fix: Re-run the job to confirm it's the known flake — do not raise the job time limit, since the log shows a hang rather than steady progress. To make this diagnosable and stop it blocking unrelated PRs: (1) bound the spin in disconnect() with ucs::get_deadline() and UCS_TEST_ABORT/skip on expiry instead of looping on m_err_count == 0 forever; (2) reduce the per-request wait in the err_handling_test branch of test_tag_send_recv() so a stuck request fails in seconds rather than accumulating 180 s deadlines; (3) given this test has hung repeatedly across transports (#7025, #6341, #6320), consider gating the bidi_stop + GPU-aware + MT variant under ASan until the underlying issue — sync 64 KB tag-zcopy requests on ud_v not being purged/completed after bidirectional UCP_EP_CLOSE_FLAG_FORCE — is fixed in the UCP peer-failure request-purge logic.

Related: #7025, #6341, #6320 (same test hanging/failing on rc, tcp, ud); PR under test: #12015, predecessor #12002

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b3b82f12-a3d7-443b-9c15-24cd8039ef57 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@gleon99
gleon99 enabled auto-merge (squash) October 6, 2026 20:08
@gleon99
gleon99 merged commit e2ac29d into openucx:master Oct 6, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants