Skip to content

UCP/FT: TX/RX tokens based protocols - #11810

Closed
evgeny-leksikov wants to merge 78 commits into
openucx:masterfrom
evgeny-leksikov:ucp_ft_on_tokens
Closed

evgeny-leksikov wants to merge 78 commits into
openucx:masterfrom
evgeny-leksikov:ucp_ft_on_tokens

Conversation

@evgeny-leksikov

Copy link
Copy Markdown
Contributor

@evgeny-leksikov evgeny-leksikov changed the title Ucp ft on tokens UCP/FT: TX/RX tokens based protocols Aug 24, 2026
@svc-ucx

svc-ucx commented Aug 24, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR · commit da029a76

TL;DR: The BlueField gtest job failed with 22 real test failures — all in this PR's own failover/PSN area (test_ucp_fault_tolerance and test_ucp_am_psn on rc_x and shm,ib) — so the PSN/token failover code on branch ucp_ft_on_tokens (head commit da029a76, which touched ucp_ep_failover.c/proto_failover.c/wireup.c while claiming to only remove traces) is functionally broken, not a CI infra issue. Fix the token/PSN handling (verify da029a76 removed only trace statements and no side-effecting code) rather than re-running.

Full analysis

Summary: make test in the "BlueField on worker 1" job exited non-zero: 8642 tests ran, 22 FAILED — 3 × test_ucp_am_psn (no_duplicates, some_duplicates, all_duplicates) and 19 × test_ucp_fault_tolerance (initiator_failure*, target_failure*, incl. pure GET and GET|FLUSH variants) under rcx/ and shm_ib/; make: *** [Makefile:4714: test] Error 1 → ##[error]Bash exited with code '2'.

Root cause: Genuine functional regressions in the PR's fault-tolerance/PSN work, not flakiness or a timeout (the log shows continuous test progress up to the summary, no stalls). The failure set maps exactly onto the code this branch changes:

  • test_ucp_am_psn.{no,some,all}_duplicates failing means the multi-eager PSN duplicate-detection path is now wrong for rc_x/shm,ib — i.e. the PSN/RX-token state is not being maintained as the receiver expects.
  • test_ucp_fault_tolerance.initiator_failure/target_failure fail even for the RMA-only variants (rc_x/GET, rc_x/GET|FLUSH, shm,ib/GET), which only exercise lane invalidation + recovery, pointing at the lane/token recovery handshake (ucp_ep_failover_apply_rx_tokens / pending_rx_lanes) rather than the AM replay logic alone.
  • Suspicion falls on the two most recent commits: e43fa0a3 added a new post-failover pending-AM count assertion (the failing initiator_failure_am_egr_single_bcopy_queue_outstanding/4 uses test_outstanding_queue()), and head commit da029a76 "remove temporary PSN debug traces" modified src/ucp/core/ucp_ep_failover.c, src/ucp/proto/proto_failover.c and src/ucp/wireup/wireup.c — a "traces only" commit should not need to touch three core failover files, so it plausibly dropped side-effecting code along with the traces.

Note: the other 53 "failed" jobs (all build_*, Static checks, Setup Server/Client) are the Azure matrix being cancelled after this first failure — a symptom, not additional root causes.

Implicated commit: [REDACTED:Hex High Entropy String] — Evgeny Leksikov, "UCP/UCT/GTEST: remove temporary PSN debug traces." (immediately preceded by e43fa0a3, same author, which added the new pending-AM assertion and the traces)

File: src/ucp/core/ucp_ep_failover.c (RX-token install/extract path, declared at src/ucp/core/ucp_ep_failover.h:53); test drivers at test/gtest/ucp/test_ucp_fault_tolerance.cc:858 (test_outstanding_queue) and :329 (test_am_with_injected_failure)

Suggested fix:

  1. Diff da029a76 against e43fa0a3 for ucp_ep_failover.c, proto_failover.c, wireup.c and confirm only ucs_trace/ucs_debug statements were deleted — any PSN/token assignment, flags |= UCP_EP_FAILOVER_LANE_FLAG_RX_TOKEN, or counter update removed together with a trace line must be restored.
  2. If the diff is clean, the bug predates it: re-run only the failing subset locally, e.g. UCX_TLS=rc_x GTEST_FILTER='*test_ucp_am_psn*:*test_ucp_fault_tolerance.initiator_failure*:*target_failure*' ./gtest with UCX_LOG_LEVEL=debug, and audit ucp_ep_failover_apply_rx_tokens() / ucp_ep_failover_pending_rx_lanes() — the 199d7aef change ("apply ADDR RX tokens only from peer reply/ACK") can leave lanes armed for failover without RX tokens, which would both stall recovery in the GET variants and desynchronize the receive PSN window that test_ucp_am_psn checks.
  3. Do not treat the 53 cancelled build jobs as separate failures; they will pass once the gtest regression is fixed.

Related: PR #11810 (branch ucp_ft_on_tokens); prior FT/token PRs in this series: #11563, #11562, #11545, #11451, and the PSN protocol origin #11160. No matching open issue found.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 789e50b4-7b01-41ca-b596-61ca89abdd29 in the triage console for the audit trail.

@svc-ucx

svc-ucx commented Aug 24, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (ucx_perftest over MAD RTE Setup Server) · commit 28775092

TL;DR: The build failed to compile, not a test failure: ucp_rkey_retain() in src/ucp/core/ucp_rkey.c:1163 declares old_refcount that is only consumed by ucs_assertv(), which compiles out in this release (non-debug, -Werror) build → -Werror=unused-variable. Mark the variable UCS_V_UNUSED (or wrap the assert usage) to fix.

Full analysis

Summary: "ucx_perftest over MAD RTE / Setup Server" job failed during make -s -j48 with core/ucp_rkey.c:1163:14: error: unused variable 'old_refcount' [-Werror=unused-variable], aborting the pipeline before any perftest ran.

Root cause: This job configures UCX without debug/assertions (log shows ASAN check: no, CFLAGS: -O3 -g -Wall -Werror ... -Werror=declaration-after-statement), so ucs_assertv() expands to nothing. In ucp_rkey_retain() the only consumer of old_refcount is the ucs_assertv(old_refcount > 0, ...) call, so with assertions disabled the variable becomes genuinely unused and -Wall -Werror turns the warning into a hard error. The neighbouring ucp_rkey_release() (line 1170) escapes the error only because it also uses old_refcount in a real if (old_refcount == 1) statement. Consequently make[2]: *** [Makefile:1303: core/libucp_la-ucp_rkey.lo] Error 1 → Bash exited with code '2'. This is a pure compile-time regression on the PR branch, unrelated to MAD/RTE or the test environment.

Implicated commit: 3478631 — "UCP/RKEY: Retain keys for failover replay", Zihao Zhao (the commit that introduced ucp_rkey_retain(), on branch ucp_ft_on_tokens / PR #11810)

File: src/ucp/core/ucp_rkey.c:1163 (function ucp_rkey_retain, lines 1161–1166)

Suggested fix: Annotate the assertion-only variable so release builds accept it:

void ucp_rkey_retain(ucp_rkey_h rkey)
{
    uint32_t UCS_V_UNUSED old_refcount = ucs_atomic_fadd32(&rkey->refcount, 1);

    ucs_assertv(old_refcount > 0, "rkey %p refcount %u", rkey, old_refcount);
}

UCS_V_UNUSED (from ucs/sys/compiler_def.h) is the idiomatic UCX marker for this pattern and keeps the assert meaningful in debug builds. Alternatively, drop the temporary and use ucs_assertv(ucs_atomic_fadd32(&rkey->refcount, 1) > 0, ...) — but avoid that, since the atomic increment would then vanish entirely in non-debug builds and break refcounting. Before re-pushing, validate locally with a release-style configure (./configure --disable-debug --enable-optimizations plus -Werror), since a debug-enabled local build will not reproduce this error.

Related: PR #11810 (openucx/ucx) — the failing PR itself; no pre-existing issue found for this warning.

@svc-ucx

svc-ucx commented Aug 25, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_icc_pgi centos7_x86_64) · commit cc18efac

TL;DR: The ICC/ICX build of centos7_x86_64 failed on a -Werror compile error: redundant double parentheses around ctx == NULL in ucp_ep_failover.c:1066 trigger clang/icx's -Wparentheses-equality. Remove the extra parentheses.

Full analysis

Summary: build_icc_pgi centos7_x86_64 failed compiling src/ucp/core/ucp_ep_failover.c with icx; make aborted with Error 2.

Root cause: The build uses icx with -Wall -Werror. icx (clang-based) emits -Wparentheses-equality for an equality comparison wrapped in extraneous parentheses, and -Werror promotes it to a hard error:

/__w/1/s/contrib/../src/ucp/core/ucp_ep_failover.c:1066:14: error: equality comparison with extraneous parentheses [-Werror,-Wparentheses-equality]
    if ((ctx == NULL)) {
         ~~~~^~~~~~~
1 error generated.
compilation aborted for .../src/ucp/core/ucp_ep_failover.c (code 1)
make[2]: *** [core/libucp_la-ucp_ep_failover.lo] Error 1

Line 1066 in ucp_ep_failover_lanes_complete() is indeed if ((ctx == NULL)) {. GCC does not warn about this pattern, so the issue only appears on the icc/icx job — this is a PR-introduced code-style defect, not an infrastructure problem. Note the failure is a real compile stop, not a timeout: log timestamps are continuous (11:04:20 → 11:05:17) with no gaps.

Implicated commit: The failing branch is ucp_ft_on_tokens (PR #11810, head [REDACTED:Hex High Entropy String]). The file is new/heavily modified in this series by Evgeny Leksikov; most recent touching commits: da029a76, e43fa0a3, fe4a23e0, caebdaf1, 1c1c1b66.

File: src/ucp/core/ucp_ep_failover.c:1066

Suggested fix: Drop the redundant parentheses:

-    if ((ctx == NULL)) {
+    if (ctx == NULL) {

Then re-push. Optionally grep the rest of the PR's new code for the same pattern, since GCC-only local builds won't catch it:
grep -rnE 'if \(\([A-Za-z_][A-Za-z0-9_.\->]* (==|!=) ' src/

Related: PR #11810 (ucp_ft_on_tokens); no pre-existing issue found for this specific warning.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e4ac7ef9-a04f-48a4-8771-ab7788c387c5 in the triage console for the audit trail.

@svc-ucx

svc-ucx commented Aug 25, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 2) · commit 0a390ee9

TL;DR: The new gtest test_uct_ib_mlx5_ext_outstanding_purge.invalid_params deliberately passes bad parameters to uct_ib_mlx5_ext_ep_outstanding_purge(), which emits ucs_error() messages; because the test body has no scoped_log_handler(wrap_errors_logger), ucx's gtest harness fails the test in TearDownProxy() ("Got N errors during the test"). Wrap the invalid-param calls in a scoped_log_handler wrap_err(wrap_errors_logger);.

Full analysis

Summary: gtest run on the "althca on worker 2" job failed with 1 of 3975 tests failing: test_uct_ib_mlx5_ext_outstanding_purge.invalid_params, causing make: *** [test] Error 1 (Bash exit code 2).

Root cause: The test calls uct_ib_mlx5_ext_ep_outstanding_purge() five times with intentionally invalid parameters (NULL params, missing UCT_EP_OUTSTANDING_FIELD_RX_TOKEN/_CB, NULL rx_token, NULL cb). Each of those calls goes through uct_ib_mlx5_ext_ep_outstanding_purge_check_param(), which logs at error level:

  • src/uct/ib/mlx5/ib_mlx5_ext.c:257 — "ib mlx5 ext: outstanding purge parameters are NULL"
  • src/uct/ib/mlx5/ib_mlx5_ext.c:264 — "ib mlx5 ext: rx token or callback is not set or is NULL"

The ucx gtest base class counts every unhidden UCS_LOG_LEVEL_ERROR (count_warns_logger) and unconditionally fails the test in test_base::TearDownProxy() (test/gtest/common/test.cc:450-457, ADD_FAILURE() << "Got N errors ..."). Unlike the analogous negative test in test/gtest/uct/ib/test_ib_mlx5_ext.cc:236-240, which installs scoped_log_handler wrap_err(wrap_errors_logger);, the new test body has no such handler, so the expected error logs are treated as unexpected. The EXPECT_EQ(UCS_ERR_INVALID_PARAM, ...) assertions themselves pass — the failure comes purely from the error-log accounting, so it is deterministic on any node built with HAVE_MLX5_DV (not node- or timing-specific).

Implicated commit: 1c1c1b6 "UCP/FT: resolve post-rebase extract→purge conflicts; drop leftover rc_mlx5 TLS" — Evgeny Leksikov (the only commit touching test/gtest/uct/v2/test_uct_outstanding_purge.cc; the error logging it now triggers was added in [REDACTED:Hex High Entropy String] "UCT: outstanding purge implementation (#11668)", Zihao Zhao)

File: test/gtest/uct/v2/test_uct_outstanding_purge.cc:54-82 (error sources: src/uct/ib/mlx5/ib_mlx5_ext.c:257 and :264)

Suggested fix: Suppress the expected error logs in the negative test, mirroring the existing pattern:

UCS_TEST_F(test_uct_ib_mlx5_ext_outstanding_purge, invalid_params) {
    uct_ep_outstanding_purge_params_t params = {};
    uint64_t rx_token = 0;
    scoped_log_handler wrap_err(wrap_errors_logger);   /* <-- add this */

    EXPECT_EQ(UCS_ERR_INVALID_PARAM,
              uct_ib_mlx5_ext_ep_outstanding_purge(NULL, NULL));
    ...
}

(Alternatively demote the parameter-validation messages in uct_ib_mlx5_ext_ep_outstanding_purge_check_param() to ucs_diag/ucs_debug, but adding the scoped log handler is the minimal, convention-consistent change. Also double-check test_uct_outstanding_purge.unsupported_on_self, which can reach the ucs_error("ib mlx5 ext: ep outstanding purge is not supported") path at ib_mlx5_ext.c:298 on mlx5-capable workers and would need the same wrapping.)

Related: PR #11810 (UCP/FT: TX/RX tokens based protocols); prior work #11668, #11580

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3e393c72-2142-4afa-b736-d1f77c99c322 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

PR scope: this PR is far over the 500-added-line limit set in REVIEW.md, and feature PRs must not bundle unrelated refactoring. It adds two ~1150/~415-line new source files plus large changes across wireup.c, ucp_ep.c, the UCT v2 API, and mlx5 transport, and mixes in unrelated refactors (e.g. ucp_wireup_msg_prepare/ucp_wireup_msg_send signature rework, put_offload RESUME/remote_flush gating, test_ib_mlx5_ext init restructure, AGENTS.md/UCX_HANDLE_ERRORS doc change). Please split into reviewable pieces: at minimum the UCT API + mlx5 token/defer-completion support, the wireup token wire-format, and the UCP failover engine as separate PRs, with pure refactors extracted on their own.

src/uct/ib/mlx5/rc/rc_mlx5.inl

Hot path: uct_rc_mlx5_common_post_send now calls uct_rc_mlx5_txwq_record_token (an uct_ib_mtu_value lookup + ucs_div_round_up) for every non-NOP RC WQE, regardless of whether failover was ever enabled on the endpoint. This adds a division to the innermost TX path for all RC users. Can we gate the token bookkeeping on ep->super.failover_flags & UCT_RC_EP_FAILOVER_FLAG_ENABLED (or precompute packets-per-MTU) so non-failover traffic is unaffected?

src/uct/api/v2/uct_v2.h

uct_ep_failover_enable is a permanent, non-revertible arm with no disable. uct_rc_mlx5_ep_failover_enable only sets UCT_RC_EP_FAILOVER_FLAG_ENABLED and there is no matching disable entry point. Combined with the always-on token recording, once any lane enables it the ep pays the cost for its lifetime. Is a disable path needed, and should the new public UCT function document that it cannot be undone?

src/uct/ib/mlx5/rc/rc_mlx5_ep.c

uct_rc_mlx5_ep_failover_enable is a permanent, non-revertible arm with no disable. Combined with the always-on token recording, once any lane enables it the ep pays the cost for its lifetime. Is a disable path needed?

src/ucp/core/ucp_ep.c

Minor: confusing nested version gate in ucp_ep_recovery_arm. The outer if (dst_version < UCP_WIREUP_ADDR_TOKEN_MIN_VERSION) has no else-body, so for dst_version == 22 it silently falls through. This reads as dead nesting. Flattening to a single if (dst_version < 22) { ...return; } with the token-vs-address distinction handled where tokens are actually used would be clearer.

src/uct/api/v2/uct_v2.h

Minor: UCT_EP_INVALIDATE_PARAM_FIELD_FLAGS documented but flags is unsigned while the flags enum uses UCS_BIT. The new uct_ep_invalidate_flags_t and field_mask handling look consistent, but please confirm the uct_ep_invalidate doc update matches the existing "Must currently be equal to zero" callers elsewhere — old callers passing field_mask=0 (and tests passing 0/NULL params) must keep working. From the code it does; calling it out to verify the ABI-compat claim.

src/ucp/core/ucp_ep.c

Test coverage: ucp_ep_recovery_send_request reuses worker->am_message_id as the recovery generation id with a do { } while (request_id == 0) guard. The correlation logic in ucp_ep_failover_apply_rx_tokens (mismatched request_id ignored) is a wire-protocol contract; the new gtests exercise single-generation flows but there is no test that drives a stale/mismatched request_id. If practical, add a focused check, otherwise note the coverage gap.

test/gtest/ucp/test_ucp_fault_tolerance.cc

Test coverage: no test drives a stale/mismatched request_id for the ucp_ep_failover_apply_rx_tokens correlation logic (mismatched request_id ignored). If practical, add a focused check, otherwise note the coverage gap.

@svc-ucx

svc-ucx commented Aug 26, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 2) · commit 564d70df

TL;DR: test_uct_ib_mlx5_ext_outstanding_purge.invalid_params fails because it intentionally calls uct_ib_mlx5_ext_ep_outstanding_purge() with bad parameters, which emits ucs_error() messages, and the test does not install a scoped_log_handler — so ucs::test's teardown turns those 5 error logs into a test failure. Wrap the negative calls in scoped_log_handler wrap_err(wrap_errors_logger);.

Full analysis

Summary: gtest run on "althca on worker 2" ended with 1 FAILED TEST — test_uct_ib_mlx5_ext_outstanding_purge.invalid_params, causing make: *** [test] Error 1 (Bash exit 2); all other 3982 tests passed.

Root cause: UCS_TEST_F(test_uct_ib_mlx5_ext_outstanding_purge, invalid_params) makes five deliberately-invalid calls to uct_ib_mlx5_ext_ep_outstanding_purge(). Each of those hits uct_ib_mlx5_ext_ep_outstanding_purge_check_param(), which logs at error level ("ib mlx5 ext: outstanding purge parameters are NULL" / "rx token or callback is not set or is NULL", src/uct/ib/mlx5/ib_mlx5_ext.c:257,264) before returning UCS_ERR_INVALID_PARAM. The gtest base class counts every error/warning emitted during a test (count_warns_logger, test/gtest/common/test.cc:262) and unconditionally fails the test in TearDownProxy() (test/gtest/common/test.cc:450-457: "Got N errors and 0 warnings during the test") unless the calls are wrapped by scoped_log_handler wrap_err(wrap_errors_logger) / hide_errors_logger. The EXPECT_EQ assertions themselves pass — the failure comes purely from the unsuppressed error logs. The sibling tests in test/gtest/uct/ib/test_ib_mlx5_ext.cc (e.g. iface_query:191, ep_query:221, ep_outstanding_purge:237) do exactly this wrapping for their negative cases; the new test omits it.

Implicated commit: 1c1c1b6 — Evgeny Leksikov, "UCP/FT: resolve post-rebase extract→purge conflicts; drop leftover rc_mlx5 TLS" (the only commit touching this test file, on the PR branch)

File: test/gtest/uct/v2/test_uct_outstanding_purge.cc:54-82 (error source: src/uct/ib/mlx5/ib_mlx5_ext.c:253-269)

Suggested fix: Suppress the expected error logs in the negative test, e.g.:

UCS_TEST_F(test_uct_ib_mlx5_ext_outstanding_purge, invalid_params) {
    uct_ep_outstanding_purge_params_t params = {};
    uint64_t rx_token = 0;
    scoped_log_handler wrap_err(wrap_errors_logger);   /* <-- add this */
    ...
}

(scope it to cover all five EXPECT_EQ(UCS_ERR_INVALID_PARAM, ...) calls, matching the style used in test/gtest/uct/ib/test_ib_mlx5_ext.cc). Optionally also demote the parameter-validation messages in uct_ib_mlx5_ext_ep_outstanding_purge_check_param() from ucs_error to ucs_diag/ucs_debug, since caller-side validation failures are already reported via the returned status.

Related: PR #11810 (UCP/FT: TX/RX tokens based protocols) — the branch introducing this test; prior purge work: #11668, #11580.

evgeny-leksikov and others added 20 commits August 27, 2026 17:56
…s exchange

Append an optional trailer to the LANES_ADDR messages, holding the
generation of the recovery round and a token section per direction: the TX
tokens of the lanes the message provides, and the RX tokens of the lanes it
answers. Both sections carry one length per lane and are empty until the
lanes provide tokens, which lands in a follow-up patch together with their
consumer, as does matching the generation against the tokens of an answer.

Add LANES_ADDR_ACK, the third leg the exchange needs so that the peer which
replies also receives the tokens derived from its own. The trailer is sent
only towards peers of UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION and above, and
the ACK, which exists only to carry it, is withheld from older peers.
The token sections are parsed ahead of the packed addresses, so a malformed
section must be rejected rather than shift the addresses. Feed crafted
sections to the parser, including truncated ones, which the empty sections
of the wire itself do not exercise.
Master bumped UCP_API_MINOR to 24. A gate of 23 would send the trailer
to peers that still parse addresses immediately after lanes_info.

Keep the original meaning of the LANES_ADDR lane maps in comments, and
drop the claim that the trailer carries a length of its own.
Adopt the dst_version 24 tokens_info layout and skip_token_section
parser, and keep FT TX collect, RX derive, and apply_rx_tokens on
that handshake so extract stays compatible with the upstreamable
wire protocol.
Take master's UCT on overlapping mlx5/token files so this PR can stay
UCP-focused. Keep failover extract, INPROGRESS freeze, and FT tests.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c
return UCS_OK;
}

if (ucp_ep_failover_is_uct_ep(ucp_ep, lane, uct_ep)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead code: the first ucp_ep_failover_is_uct_ep check already returns UCS_INPROGRESS for this uct_ep, so the second identical check inside the wireup_ep branch (around L484) looks unreachable. Since ucp_ep_set_lanes_failed() between them does not turn a lane into a failover uct_ep, the second check never returns UCS_INPROGRESS. Can we drop it, or is a state change between them intended?

Comment thread test/gtest/AGENTS.md
- most fixtures support memtype skipping via
`check_skip_test()` patterns; prefer `UCS_TEST_SKIP_R(...)` over silent
short-circuits so the runner reports skipped cases.
- New tests must be listed in the relevant `Makefile.am` source list.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test-default behavior change bundled into a feature PR: changing the default UCX_HANDLE_ERRORS from freeze to bt affects the whole gtest suite, not just failover tests, and is unrelated to the failover feature. Per REVIEW.md ("Feature PRs must not refactor code unrelated to the feature"), this should be split out.

Comment thread test/gtest/Makefile.am
uct/test_pending.cc \
uct/test_progress.cc \
uct/test_uct_ep.cc \
uct/test_uct_iface.cc \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test-default behavior change bundled into a feature PR: changing the default UCX_HANDLE_ERRORS from freeze to bt affects the whole gtest suite, not just failover tests, and is unrelated to the failover feature. Per REVIEW.md ("Feature PRs must not refactor code unrelated to the feature"), this should be split out.

Comment thread src/ucp/rma/put_offload.c
@@ -164,7 +164,8 @@ ucp_proto_put_offload_bcopy_send_func(ucp_request_t *req,
return status;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated behavioral change: adding UCP_PROTO_COMMON_INIT_FLAG_RESUME and a new ucp_rkey_need_remote_flush gate to ucp_proto_put_offload_bcopy_probe is a real functional change to PUT offload protocol selection that will affect non-failover PUT users. It should be justified in the description or split out; as-is it is hard to review in the middle of a failover PR.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the token wire-format parsing has a focused gtest (test_ucp_wireup_token_section), but the cross-version path (old peer without trailer, ucp_wireup_ep_supports_tokens false) is only implicitly covered.

Residual coverage gap: no test drives the asymmetric-failure LANES_ADDR_REQ/dual-REQ tie-break paths (ucp_ep_local_id(ep) < msg->src_ep_id) added in ucp_wireup_process_lanes_addr_request.

Dominant recommendation remains splitting the PR; the individual failover-engine correctness paths are difficult to review with confidence at this size.

@svc-ucx

svc-ucx commented Sep 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Basic_compile Basic compile) · commit ab9c9a38

TL;DR: The "Basic compile" job failed with -Werror compile errors in src/uct/ib/mlx5/rc/ because the merge commit ab9c9a38 ("Merge upstream/master into ucp_ft_on_tokens") kept the new .c code but lost the corresponding header-side additions (uct_ib_mlx5_txwq_t::ft_ci/next_token, uct_rc_mlx5_txwq_record_token(), and the UCT invalidate-flags API). Re-apply those header changes in the merge resolution.

Full analysis

Summary: make -j4 failed compiling rc_mlx5_iface.c and rc_mlx5_ep.c — undeclared struct members, missing inline helper, and missing UCT API enums/fields (all fatal via -Werror).

Root cause: The branch's C code references symbols that do not exist in the merged tree:

  • uct_rc_mlx5_txwq_record_token() — not declared anywhere reachable (rc_mlx5_ep.c:363, "implicit declaration" + "nested extern").
  • uct_ib_mlx5_txwq_t.ft_ci and .next_token — the struct in src/uct/ib/mlx5/ib_mlx5.h:685-705 contains sw_pi, prev_sw_pi, next_wqe_psn, reg, curr, dbrec, qstart, qend, bb_max, sig_pi, path_mtu_mask, path_mtu_shift, hw_ci/flags (only under #if UCS_ENABLE_ASSERT), fi — no ft_ci, no next_token (rc_mlx5_iface.c:227,229, rc_mlx5_ep.c:874,876,887,890).
  • UCT_EP_INVALIDATE_PARAM_FIELD_FLAGS, UCT_EP_INVALIDATE_FLAG_DEFER_COMPLETIONS, and uct_ep_invalidate_params_t.flags — not present in the UCT API (rc_mlx5_ep.c:862-864).

This is a bad merge resolution: the head commit is a merge of upstream master, and upstream commit 8282944a "UCT/IB/MLX5: Track RC send PSN with path MTU (#11821)" rewrote exactly the uct_ib_mlx5_txwq_t struct region that the feature branch also modified. The conflict was resolved by taking upstream's version of ib_mlx5.h (and evidently the API/inline headers too), discarding the branch's field/helper/API additions while the .c users survived.

Implicated commit: [REDACTED:Hex High Entropy String] — Evgeny Leksikov, "UCP/FT: Merge upstream/master into ucp_ft_on_tokens" (conflicting upstream commit: 8282944a, Roie Danino, #11821)

File: src/uct/ib/mlx5/ib_mlx5.h:685-705 (missing ft_ci/next_token); users at src/uct/ib/mlx5/rc/rc_mlx5_ep.c:363,862-864,874-890 and src/uct/ib/mlx5/rc/rc_mlx5_iface.c:227-229

Suggested fix:

  1. Redo the merge resolution and restore the branch-side header changes on top of upstream's new struct layout:
    • Add back uint16_t ft_ci; and the token counter next_token to uct_ib_mlx5_txwq_t in src/uct/ib/mlx5/ib_mlx5.h, keeping upstream's next_wqe_psn/path_mtu_mask/path_mtu_shift fields.
    • Restore uct_rc_mlx5_txwq_record_token() in the header/inline file where it was defined (e.g. src/uct/ib/mlx5/rc/rc_mlx5.inl / rc_mlx5_common.h) so it is visible from rc_mlx5_ep.c.
    • Restore the UCT API additions in src/uct/api/uct.h: UCT_EP_INVALIDATE_PARAM_FIELD_FLAGS in the invalidate field-mask enum, the UCT_EP_INVALIDATE_FLAG_DEFER_COMPLETIONS flag, and the flags member of uct_ep_invalidate_params_t.
  2. While fixing: note hw_ci is guarded by #if UCS_ENABLE_ASSERT upstream, but rc_mlx5_ep.c:874,884 use txwq->hw_ci unconditionally. This build happened to have asserts enabled; a release/--disable-assertions build will fail. Either move hw_ci out of the assert guard or make the new failover code use an unconditionally-available index.
  3. Recommend running a local ./contrib/configure-devel && make -j (which is what "Basic compile" does with -Werror) after any master merge on this branch.

Related: PR #11810 (UCP/FT: TX/RX tokens based protocols); upstream conflict source #11821 (UCT/IB/MLX5: Track RC send PSN with path MTU); #11804 (UCT: add token query support) also touched ib_mlx5.h.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 9835f3cc-c682-467e-a74b-bad33f651d92 in the triage console for the audit trail.

Take master's remaining mlx5/iface files so this PR stays UCP-first
and builds against the token purge API without DEFER UCT bits.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

uct_invoke_completion is declared there; dropping the include broke
the devel -Werror build after taking master's UCT headers.
Comment thread src/ucp/wireup/wireup.c
@@ -2389,10 +2949,9 @@ static void ucp_wireup_msg_dump(ucp_worker_h worker, uct_am_trace_type_t type,
char *p, *end;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor — ucp_wireup_msg_dump skips only ucp_wireup_msg_lanes_info_t before calling ucp_address_unpack. When the peer is >= UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION, a ucp_wireup_msg_tokens_info_t + variable-length token sections sit between the lanes_info and the packed address (as ucp_wireup_parse_lanes_addr correctly accounts for). The dump therefore points addr_ptr into the token trailer and will misparse / log "failed to unpack address" whenever UCX_LOG_LEVEL=data AM tracing is enabled on a token-capable connection. Impact is limited to the debug/trace path (no data-path effect), but the dump should mirror the trailer-skipping logic (e.g. reuse ucp_wireup_skip_token_section) or bail out for token-carrying messages.

Comment thread src/ucp/core/ucp_ep.c
/* Re-evaluate am_lane from scratch so a recovered AM lane can become the
* short-protocol lane again; keeping a post-failover fallback here races
* with proto short selection (lane mismatch assert). */
cfg_key.am_lane = UCP_NULL_LANE;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — In ucp_ep_reconfig_clear_failed_lanes, cfg_key.am_lane = UCP_NULL_LANE; is dead code: ucp_ep_config_key_find_am_lane() takes a const ucp_ep_config_key_t * and does not read am_lane, so the first assignment is immediately overwritten with no observable effect. Drop the redundant first assignment (or, if the intent is to guard a future reader of am_lane, add a comment clarifying why the reset must precede the call).

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.h
@@ -349,7 +349,7 @@ typedef struct ucp_worker {
uct_worker_cb_id_t rkey_ptr_cb_id; /* RKEY PTR worker callback queue ID */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: am_message_id isn't used by failover — recovery uses recovery_arg->request_id. The comment change from /* For matching long AMs */ to /* AM/failover message ID generator */ is misleading since no failover code path uses worker->am_message_id (only long-AM matching in proto_common.inl and ucp_request.c consumes it). Please revert the comment to /* For matching long AMs */ to avoid implying a coupling that doesn't exist.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucp/core/ucp_ep_failover.c

Latent path worth confirming: in ucp_ep_failover_apply_rx_tokens, when rx_token_lengths[lane] > 0 but the wire tokens pointer is NULL, the RX token buffer is allocated but left uninitialized and still installed as a valid token (only a NULL malloc result triggers fallback). Please confirm the parser guarantees tokens != NULL whenever any RX length is non-zero.

The leftover mlx5_ext checks do not compile against master's
uct_rc_mlx5_ep flags.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_ep.c
/* Re-evaluate am_lane from scratch so a recovered AM lane can become the
* short-protocol lane again; keeping a post-failover fallback here races
* with proto short selection (lane mismatch assert). */
cfg_key.am_lane = UCP_NULL_LANE;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the first assignment cfg_key.am_lane = UCP_NULL_LANE; is dead — ucp_ep_config_key_find_am_lane() scans lane types and does not read key->am_lane, so it is immediately overwritten. The comment about "re-evaluate from scratch" is already satisfied by the call alone; pls drop the redundant = UCP_NULL_LANE line.

Comment thread src/ucp/proto/proto.c
[UCP_OP_ID_RNDV_SEND] = "rndv_send",
[UCP_OP_ID_RNDV_RECV] = "rndv_recv",
[UCP_OP_ID_RNDV_RECV_DROP] = "rndv_recv_drop",
[UCP_OP_ID_FAILOVER_AM_BCOPY] = "failover_am_bcopy",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the new [UCP_OP_ID_FAILOVER_AM_BCOPY] = "..." entry breaks the = column alignment used by the rest of the ucp_operation_names table (it sits one column left of [UCP_OP_ID_RNDV_RECV_DROP] =). pls re-align for consistency with the surrounding designated initializers.

Comment thread src/ucp/proto/proto.c
[UCP_OP_ID_RNDV_RECV] = "rendezvous data fetch",
[UCP_OP_ID_RNDV_RECV_DROP] = "rendezvous data drop",
[UCP_OP_ID_LAST] = NULL
[UCP_OP_ID_FAILOVER_AM_BCOPY] = "failover replay active message",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the new ucp_operation_descs failover entry breaks the = column alignment used by the rest of the table. pls re-align for consistency with the surrounding designated initializers.

@svc-ucx

svc-ucx commented Sep 1, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit aec9a186

TL;DR: The Coverity release job failed on a single new REVERSE_INULL defect in src/ucp/wireup/wireup.c:1456 — ep->ext is unconditionally dereferenced by ucp_ep_update_remote_id() on line 1452, so the later ep->ext != NULL guards are dead null checks; drop those redundant checks.

Full analysis

Summary: coverity release on coverity_rh7 build/analysis succeeded but the gate failed: ##[error]Coverity found 1 issues: /__w/14/s/src/ucp/wireup/wireup.c:1456 Type: Dereference before null check (REVERSE_INULL).

Root cause: In ucp_wireup_process_lanes_addr_request(), line 1452 calls ucp_ep_update_remote_id(ep, msg->src_ep_id), which in src/ucp/core/ucp_ep.inl dereferences ep->ext on every path (ep->ext->remote_ep_id = remote_id; at line 234, and also ep->ext->remote_ep_id inside the assert at line 225 when UCP_EP_FLAG_REMOTE_ID is set). Coverity therefore flags the subsequent (ep->ext != NULL) conditions at lines 1456 and 1465 as null checks after an unconditional dereference. ep->ext is in fact always allocated together with the endpoint, so the checks are logically dead — but Coverity treats the pattern as a defect and the pipeline gate fails on any non-zero defect count (nerrors=1 → job exit 1). This code is new in this PR branch (ucp_ft_on_tokens, PR #11810), which introduced the recovery_arg / dual-REQ handling block; nothing else in the 555-unit build or the analysis (94 pre-existing occurrences, only this one gated) is at fault.

Implicated commit: PR #11810 branch ucp_ft_on_tokens at [REDACTED:Hex High Entropy String]; most recent related change to this file is 1127ba97 "UCP/WIREUP: Merge PR #11843 LANES_ADDR token trailer framing" by Evgeny Leksikov.

File: src/ucp/wireup/wireup.c:1456 (and the duplicate check at :1465); dereference site src/ucp/core/ucp_ep.inl:225,234

Suggested fix: Remove the redundant ep->ext != NULL tests, since ep->ext is guaranteed non-NULL after ucp_ep_update_remote_id(). Also factor the repeated condition to avoid a second copy of the same pattern, e.g.:

    ucp_ep_update_remote_id(ep, msg->src_ep_id);

    /* Dual-REQ: lower ep_id is the initiator. */
    if ((ep->ext->recovery_arg != NULL) &&
        (ep->ext->recovery_arg->state == UCP_EP_RECOVERY_STATE_WAIT_REPLY)) {
        if (ucp_ep_local_id(ep) < msg->src_ep_id) {
            ucs_debug("ep %p: ignore LANES_ADDR_REQ from higher ep id 0x%" PRIx64
                      " while we are initiator", ep, msg->src_ep_id);
            return;
        }
        if (ucp_ep_local_id(ep) > msg->src_ep_id) {
            ep->ext->recovery_arg->state = UCP_EP_RECOVERY_STATE_IDLE;
        }
    }

If a NULL ep->ext is genuinely believed possible here, the check must instead be moved before the ucp_ep_update_remote_id() call (and ucs_assert(ep->ext != NULL) used to document the invariant) — but the existing code in ucp_ep.inl implies it is not. After the change, re-run the Coverity stage; the gate passes only when the reported error count is 0.

Related: PR #11810 (#11810); prior fix-up commits on the same file/branch: 1127ba9, 6b7c7bf ("Fix maybe-uninitialized lanes_info"), c2dd315.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 0e25b794-c37c-49c3-afd1-bf5c24e567f0 in the triage console for the audit trail.

Keep the PR UCP-only: remove unused enable_lanes, restore master's
gtest HANDLE_ERRORS default, and drop the outstanding-purge UCT test.
Coverity REVERSE_INULL: ucp_ep_update_remote_id already uses ep->ext.
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/wireup/wireup.c
@@ -2389,10 +2948,9 @@ static void ucp_wireup_msg_dump(ucp_worker_h worker, uct_am_trace_type_t type,
char *p, *end;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_wireup_msg_dump skips lanes_info but not the token trailer added here, so with a token-capable peer the address dump starts mid-trailer. When the token trailer is present (peer dst_version >= UCP_WIREUP_ADDR_TOKEN_MIN_DST_VERSION), the dump interprets the token bytes as the start of the packed address and prints garbage / possibly log-spams an unpack error. Can we reuse the same trailer-skip logic (ucp_wireup_parse_lanes_addr / ucp_wireup_skip_token_section) here? Debug-only, so minor.

@svc-nvidia-pr-review

Copy link
Copy Markdown

src/ucp/rma/put_offload.c

The put_offload.c change adds UCP_PROTO_COMMON_INIT_FLAG_RESUME to bcopy probe and a new ucp_rkey_need_remote_flush gate. The remote_flush gate is a behavior change for the RMA path affecting non-failover configs. This is part of the scope concern: the PR should be split so each correctness path can be reviewed with confidence.

The dominant recommendation stands: this PR is far over the 500-added-line limit and bundles the UCT API/mlx5 token support, the wireup token wire-format, the UCP failover engine, and unrelated refactors (the put_offload RESUME flag + ucp_rkey_need_remote_flush gate, the ucp_wireup_msg_prepare/msg_send signature rework, and test restructuring). Splitting into separate reviewable PRs would let each correctness path be reviewed with confidence — the individual failover state-machine transitions (abort vs. replay-completion ownership of super_req, flush_ops_count accounting) are hard to verify at this size.

Coverage gap: No test drives a stale/mismatched request_id through ucp_ep_failover_apply_rx_tokens.

Coverage gap: No test exercises the dual-REQ tie-break (ucp_ep_local_id(ep) < msg->src_ep_id) in ucp_wireup_process_lanes_addr_request.

Coverage gap: The old-peer/no-trailer cross-version path (ucp_wireup_ep_supports_tokens false) is only implicitly covered.

@svc-ucx

svc-ucx commented Sep 2, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 8cca7818

TL;DR: The ASan/roce gtest shard didn't time out on the CI step — the very first test (rc/test_ucp_tag_mt.send_recv/0 <rc_v/req_int,mt_context>) hung for the full 900 s gtest watchdog window and got aborted, i.e. a blocking UCP tag send/recv over rc_verbs never completed on this PR branch. The fix belongs in PR #11810's new wireup/FT-token path (head commit 8cca7818), not in any timeout setting.

Full analysis

Summary: make -C test/gtest test aborted (core dumped) because the gtest watchdog killed a hung test: the first test of shard 1/4 never completed.

Root cause: A hang, not slowness. Log evidence:

  • 21:03:36.0500 [ RUN ] rc/test_ucp_tag_mt.send_recv/0 <rc_v/req_int,mt_context>
  • …complete silence for 14 min 59 s (the largest gap in the log, and essentially the whole runtime of the step)…
  • 21:18:35.9459 test/gtest/common/test_helpers.cc:57: Failure / Failed / Connection timed out - abort testing — this is ADD_FAILURE() << strerror(ETIMEDOUT) from the watchdog thread (watchdog_timeout = 900. at test/gtest/common/test_helpers.cc:31), followed by pthread_kill → signal 6.
  • The backtrace shows the watched thread inside the OpenMP region of the test (libgomp GOMP_parallel → gtest frames → __sched_yield), i.e. it was spinning in the blocking send/recv progress loop of test_ucp_tag_mt.send_recv (test/gtest/ucp/test_ucp_tag_mt.cc:61-68), waiting for a request that never completed.
  • Zero UCX error/warn/diag output was produced, and only 1 of 8758 tests in the shard ran — so this is a silent, deterministic stall of plain tag send/recv over rc_v, not a device/fabric error and not a flaky slow test.

The stall is in code introduced by this PR branch (ucp_ft_on_tokens), which rewrites the wireup/lane-recovery path: the LANES_ADDR REQ/REP/ACK token-trailer handshake and the failover pending/replay queues. Note that these paths silently drop or park messages/requests with no retry when something doesn't line up (ucp_wireup_parse_lanes_addr() → goto out in ucp_wireup_msg_handler, src/ucp/wireup/wireup.c:1694-1700; requests parked on ctx->pending_queue in src/ucp/core/ucp_ep_failover.c), which matches exactly the observed "no output, nothing ever completes" signature. The branch head also introduced an inconsistency worth fixing regardless: 8cca7818 removed the ep->ext NULL guards, so src/ucp/wireup/wireup.c:1456 dereferences ep->ext->recovery_arg unconditionally, while every failover helper still treats ep->ext == NULL as legal (src/ucp/core/ucp_ep_failover.c:206, 220, 249, 480, 490, 516).

Implicated commit: [REDACTED:Hex High Entropy String] — "UCP/WIREUP: Drop dead ext null checks after remote_id update" (Evgeny Leksikov), on top of 1127ba97 "LANES_ADDR token trailer framing" and 05f011f7 "fold token exchange into ADDR REQ/REP/ACK handshake" (same author, same PR).

File: src/ucp/wireup/wireup.c:1442-1510 (LANES_ADDR request handling; ep->ext->recovery_arg deref at :1456) and src/ucp/wireup/wireup.c:1659-1757 (ucp_wireup_msg_handler, silent drop path); symptom surfaces at test/gtest/common/test_helpers.cc:57 / test/gtest/ucp/test_ucp_tag_mt.cc:61

Suggested fix:

  1. Do not raise the watchdog/step timeout — the process was hung, not slow.
  2. Reproduce deterministically (it hung on the first test, so it should be reliable): UCX_TLS=rc_v UCX_LOG_LEVEL=debug ./test/gtest/gtest --gtest_filter='rc/test_ucp_tag_mt.send_recv/0' under ASan, and check whether the debug log stops at wireup (ep %p: remote connected never reached) or after wireup in the tag send path. Attach gdb -p/ucx_info -e on the hung process to see which request/lane is stuck.
  3. Restore the ep->ext != NULL guard dropped by 8cca7818 in ucp_wireup_process_lanes_addr_request() (wireup.c:1456) — the surrounding code and all ucp_ep_failover_* helpers still assume ep->ext may be NULL, so the "dead check" removal is not safe.
  4. Make the silent failure modes loud so this can't hang for 15 minutes again: in ucp_wireup_msg_handler() the goto out paths after ucp_wireup_parse_lanes_addr() failure, and requests parked on the failover pending_queue, should log at diag/error and/or fail the EP instead of dropping the handshake with no retry.

Related: PR #11810 (this PR, "UCP/FT: TX/RX tokens based protocols"), PR #11843 ("UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange", merged into this branch as 1127ba97).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3e17a517-8526-4b71-a17c-15c3fb61d24d in the triage console for the audit trail.

@evgeny-leksikov

Copy link
Copy Markdown
Contributor Author

out of date

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants