Skip to content

UCP/RKEY: Keep configurations at stable addresses - #11676

Merged
tvegas1 merged 8 commits into
openucx:masterfrom
jeffnvidia:jmahou/fix-rkey-config-stability
Sep 30, 2026
Merged

tvegas1 merged 8 commits into
openucx:masterfrom
jeffnvidia:jmahou/fix-rkey-config-stability

Conversation

@jeffnvidia

@jeffnvidia jeffnvidia commented Jul 19, 2026 •

Copy link
Copy Markdown
Contributor

What?

Store each rkey configuration separately from the resizable index array, and
update its users and tests for pointer-based storage.

This PR now contains only the stable-storage change. It depends on #11817 to
keep protocol selection out of async context; it does not add async locking.

Why?

Protocol selection can recursively create another rkey configuration. With
inline storage, growing the array copies a partially initialized configuration
while the outer selection continues updating the old address. Later lookups
then use an incomplete copy.

This is reachable without an injected configuration: on Gaia, normal
default-reserve traffic naturally moved the array from 32 to 64 while an outer
rkey configuration was live. A diagnostic consistency check observed
put_short diverging between the old and current copies in 4/5 runs.

How?

The resizable array stores pointers to separately allocated configurations.
Growing the array can move the pointer slots, but it cannot move a live
configuration. Worker cleanup destroys and frees each configuration.

Validation

@jeffnvidia
jeffnvidia marked this pull request as ready for review July 19, 2026 18:38
@jeffnvidia
jeffnvidia force-pushed the jmahou/fix-rkey-config-stability branch 3 times, most recently from c465ff9 to 08f5005 Compare July 20, 2026 08:39
@brminich
brminich requested review from evgeny-leksikov and iyastreb and removed request for iyastreb August 23, 2026 09:11
@iyastreb

Copy link
Copy Markdown
Contributor

Is it related fix? #11802

@jeffnvidia

Copy link
Copy Markdown
Contributor Author

Yes, they are related. After comparing both current heads, I think they address the same underlying rkey-config address-instability, but with different coverage.

#11802 keeps worker->rkey_config as an inline object array and reloads proto_select / rkey_config from the config indexes after nested calls that may grow the worker arrays. That addresses the synchronous recursive-allocation path while avoiding extra indirection. It also includes analogous ep_config reloads.

#11676 instead makes each rkey configuration a separate allocation. Growing the index array can then no longer move a live rkey config, its key, or its proto_select; this makes pointer stability an invariant rather than requiring every nested-call boundary to be audited. It additionally serializes rkey_config_hash lookup/create with the worker async context. #11802 does not change that hash synchronization, and ucp_rkey_proto_resolve() still has the raw lookup/direct-add path there.

So I see the rkey relocation changes as alternative strategies, with a tradeoff between allocation/indirection and maintaining a complete reload audit. The async hash serialization is complementary and should be retained whichever storage strategy is selected. If inline storage is preferred, I would evaluate the reloads from #11802 together with the async serialization from #11676. Conversely, the ep_config reload work in #11802 is complementary to #11676, since #11676 only stabilizes rkey configs.

Comment thread src/ucp/core/ucp_rkey.c Resolved
@svc-ucx

svc-ucx commented Aug 26, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Build build_aarch64 ctyunos2507_aarch64) · commit 371a5d83

TL;DR: The aarch64 (and every) build fails compiling test/gtest/ucp/test_ucp_proto.cc because PR #11676 changed worker->rkey_config from an array of ucp_rkey_config_t values to an array of ucp_rkey_config_t* pointers, but the gtest code still dereferences elements as values; update the test to use pointer semantics.

Full analysis

Summary: make failed with -Werror compile errors in ucp/gtest-test_ucp_proto.o ("cannot convert ucp_rkey_config_t** to ucp_rkey_config_t*", "request for member 'proto_select' in ... which is of pointer type"), aborting the Build on aarch64 step (make[3]: *** [Makefile:2698: ucp/gtest-test_ucp_proto.o] Error 1).

Root cause: The PR's own change (UCS_ARRAY_DECLARE_TYPE(ucp_rkey_config_arr_t, unsigned, ucp_rkey_config_t*) in src/ucp/core/ucp_worker.h:296, so ucs_array_elem(&worker->rkey_config, i) now yields a ucp_rkey_config_t*, see src/ucp/core/ucp_rkey.inl:44) was not propagated to the gtest. In test/gtest/ucp/test_ucp_proto.cc (PR head):

  • line 65 ucs_array_for_each(rkey_config, &worker->rkey_config) with rkey_config typed as ucp_rkey_config_t * — the iterator is now ucp_rkey_config_t **, causing the assignment/comparison errors.
  • lines 179, 248, 346, 390 use ucs_array_elem(&worker()->rkey_config, idx).proto_select — must be ->proto_select.
  • The "proto_select is not captured" errors at lines 200–202 are cascade fallout from the failed type deduction at line 178.

This is not aarch64-specific; the file simply wasn't updated with the container-type change.

Implicated commit: 9ebccd4 — "UCP/RKEY: Keep configurations at stable addresses" (jeffnvidia), the head commit series of PR #11676 (build commit 371a5d8)

File: test/gtest/ucp/test_ucp_proto.cc:65 (also lines 178–179, 200–202, 248, 346, 390); type change at src/ucp/core/ucp_worker.h:296

Suggested fix: In test/gtest/ucp/test_ucp_proto.cc, adapt to the new pointer-element array:

  • Change the loop variable to ucp_rkey_config_t **rkey_config and access fields via (*rkey_config)->... (or keep ucp_rkey_config_t *cfg and assign cfg = *it inside the loop).
  • Replace ucs_array_elem(&worker()->rkey_config, idx).proto_select with ucs_array_elem(&worker()->rkey_config, idx)->proto_select at lines 179, 248, 346, 390 (keeping the & where a pointer to proto_select is taken).
  • Optionally introduce a small helper (e.g. ucp_worker_rkey_config(worker, idx)) used by both src and tests so future container changes don't break the gtest. Then re-run the build; grep the whole tree for rkey_config value-semantics uses (ucs_array_elem(&...rkey_config / ucs_array_for_each(...rkey_config) to catch any remaining sites.

Related: PR #11676

@jeffnvidia
jeffnvidia force-pushed the jmahou/fix-rkey-config-stability branch from 371a5d8 to 04d582b Compare August 26, 2026 12:12
@iyastreb

Copy link
Copy Markdown
Contributor

Do we need all 3 PRs fixing related things?
This one + #11817, #11676
If so, is there interdependency on them?

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_worker.cc Resolved
@jeffnvidia

Copy link
Copy Markdown
Contributor Author

Assuming the last link meant #11802: we do not need all three rkey approaches.

#11817 addresses the async-context race by keeping protocol selection out of async context, but it does not prevent synchronous recursive rkey-array relocation.

For that remaining issue, #11676 and the rkey portion of #11802 are alternative strategies: #11676 provides stable object addresses, while #11802 reloads array-derived pointers after nested allocation. The ep-config reloads in #11802 are complementary because #11676 only changes rkey storage.

So the complete choices are:

The rkey portions of #11676 and #11802 are not both required.

@jeffnvidia

Copy link
Copy Markdown
Contributor Author

@iyastreb @tvegas1 All feedback on this PR has been addressed. The lock changes were removed, the PR now contains only the stable rkey-config storage change and its tests, and all 162 CI checks pass.

When you have a chance, could you please review the current version? This PR still depends on #11817 for keeping protocol selection out of async context.

@tvegas1

tvegas1 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

@iyastreb @tvegas1 All feedback on this PR has been addressed. The lock changes were removed, the PR now contains only the stable rkey-config storage change and its tests, and all 162 CI checks pass.

When you have a chance, could you please review the current version? This PR still depends on #11817 for keeping protocol selection out of async context.

related question: #11817 (comment)

@jeffnvidia
jeffnvidia force-pushed the jmahou/fix-rkey-config-stability branch from 04d582b to e19fc05 Compare September 9, 2026 10:46
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@jeffnvidia

Copy link
Copy Markdown
Contributor Author

@yosefe @tvegas1 @iyastreb #11817 is now merged. I rebased #11676 onto current master (e19fc051a); the three PR commits remain patch-identical.

A fresh Gaia build (job 68106) completed successfully and passed all 31/31 focused rkey/protocol tests. Refreshed Azure CI is now running on the rebased head.

The #11817 review explicitly deferred the rkey-config pointer-invalidation fix to another PR. Should #11676 be that follow-up, or do you prefer the #11802 contiguous-storage/index-relookup approach?

Comment thread test/gtest/ucp/test_ucp_worker.cc Resolved
Comment thread src/ucp/core/ucp_rkey.inl Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread test/gtest/ucp/test_ucp_worker.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gaps: no functional coverage that put_short/proto_select of an already-created rkey config stays correct after the array grows (i.e. the actual failure mode being fixed); existing test_ucp_proto_mock rkey-config checks only look at counts and hash contents. Also, ucp_worker_print_info() iterates all rkey configs; the new test leaves 33 fake configs (zeroed keys, no hash entries) in the worker, so ucx_info-style dumps on that worker would walk them. Harmless for gtest, but another reason to build the test on real configs.

@svc-ucx

svc-ucx commented Sep 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 1) · commit e19fc051

TL;DR: The ASan/RoCE gtest job didn't time out from slowness — it hung: all/test_ucp_sockaddr.listen/0 <all/tag> posted a tag request that never completed while the worker kept polling for ~11 minutes, and the only change on this branch is the UCP config-storage rework (ba871b37, "UCP/RKEY: Keep configurations at stable addresses"), which leaves raw pointers into the still-by-value ep_config array live across reallocation. Fix by giving ep_config the same pointer-stability treatment (or re-resolving configs by index after any call that can append a config) instead of caching ucp_ep_config_t*/ep_config_key pointers.

Full analysis

Summary: gtest aborted ("Connection timed out - abort testing") because all/test_ucp_sockaddr.listen/0 <all/tag> hung: request 0x62e0002f2418 never completed, make test died with SIGABRT (core dumped).

Root cause: Log timestamps show continuous test activity (one test every ~0.2–1.2 s) up to 13:55:41.006 server listening on 10.224.36.27:36937, then a single 11m16s gap with zero output, ending in ucp_test.cc:300 UCX ERROR request 0x62e0002f2418 did not complete on time (14:06:57) and the gtest abort at 14:10:41. The backtrace shows the main thread inside the gtest polling loop (__sched_yield under ucp_test::request_process, ucp_test.cc:290-295), so the process was actively progressing the worker — this is a lost/never-progressed request, not a lock deadlock and not a genuinely slow test. The only code change on this branch is the UCP configuration-storage rework: worker->rkey_config became an array of heap pointers (ucp_worker.h:301, ucp_worker.c:2306-2422, ucp_rkey.inl:40-45) and a new realloc-with-deferred-free scheme was added (ucp_worker_config_array_append / ucp_worker_config_array_release_old, ucp_worker.c:2205-2233). ep_config entries are still stored by value in that same array, and protocol selection still keeps raw pointers into it — init_params.ep_config_key = &ucs_array_elem(&worker->ep_config, ep_cfg_index).key (proto_select.c:202-203) is captured before the probe loop (proto_select.c:222-237), and proto_select->cache.value is cached with the still-present /* FIXME proto_select could get invalidated by adding rkey config */ (proto_select.inl:111). Under the new scheme the old buffer is memcpy'd and freed asynchronously, so a selection made against a stale ep_config/cached select element picks lanes/thresholds from an outdated configuration — the send is posted on a lane that is not the one actually wired up/registered for progress (ucp_proto_select_wiface_activate / ep_config->proto_lane_map, proto_select.c:454-469), producing exactly the observed symptom: no error, no completion, endless progress. This is the highest-probability cause given the evidence; the hang is single-occurrence, so a pre-existing flake cannot be fully excluded until the test is looped on master.

Implicated commit: ba871b37 — jeffnvidia, "UCP/RKEY: Keep configurations at stable addresses" (branch head [REDACTED:Hex High Entropy String], PR #11676)

File: src/ucp/core/ucp_worker.c:2205-2233 (and src/ucp/proto/proto_select.c:202-210, src/ucp/proto/proto_select.inl:111)

Suggested fix:

  1. Extend the stability fix to ep_config: store ucp_ep_config_t* in ucp_ep_config_arr_t exactly as was done for ucp_rkey_config_arr_t, so init_params.ep_config_key and any ucp_ep_config_t* held across proto->probe() stay valid.
  2. Until then, never hold ucp_ep_config_t*/&...->key across a call that can append a config — re-resolve via ucp_worker_ep_config(worker, ep_cfg_index) after ucp_proto_select_init_protocols()/probe() returns.
  3. Resolve the FIXME at proto_select.inl:111 by calling ucp_proto_select_cache_reset() for affected proto_select objects whenever a config array grows, and add a debug-only generation counter assert that a cached select_elem/ep_config_key belongs to the current buffer.
  4. To confirm attribution before merging, run all/test_ucp_sockaddr.listen/0 (and the full all/test_ucp_sockaddr suite) in a loop on the ASan RoCE config on both master and this branch; the hang is reproducible-by-repetition rather than deterministic.

Related: PR #11676 (this PR); commit [REDACTED:Hex High Entropy String] "UCP/WORKER: Allow storing up to 1024 rkey_config" (#11171) — earlier change to the same rkey-config storage.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 611d973c-7322-4519-8f71-1680d2213158 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage note: rkey_config_stable_after_growth creates 65 rkey configs through the production path, and each one runs a full ucp_proto_select_short_init() pass for UCP_OP_ID_PUT; worth watching the runtime of this test on slower CI machines.

Residual note: the stale-pointer workarounds that this fix makes unnecessary (re-fetching ucp_rkey_config() after ucp_ep_update_rkey_config() in proto_common.inl and ucp_ep.c:4842) are left in place; fine to keep out of this PR, but they are now dead defensiveness.

@svc-ucx

svc-ucx commented Sep 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer gpu on worker 0) · commit f4f50c5a

TL;DR: The ASAN GPU job failed on a single gtest expectation — shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 — which forces RNDV_SCHEME=get_zcopy and then insists am/rndv is still the protocol chosen for a 1 MB CUDA buffer; that assumption only holds if a network MD can register CUDA memory, so the check is host/topology dependent, not a failure of the rkey-config-stability change itself. Fix: guard the test with require_cuda_net_md() (as the sibling CUDA tests do) or drop the protocol-name assertion and keep only the "no PROTO_ESTIMATION rkey config" assertion.

Full analysis

Summary: make test in the AddressSanitizer gpu on worker 0 job exited 2 because 1 of 10222 gtests failed: shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 (GetParam() = shm,ib,cuda_copy,rocm_copy). All other tests passed; no ASAN report, no crash, no timeout (test suite ran to completion and printed the TOP-20 timing table).

Root cause: The failing test is defined at test/gtest/ucp/test_ucp_proto.cc:830-838 and runs check_am_rndv_remote_proto_config(0, 0) (test_ucp_proto.cc:453-464) with "RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y", "RNDV_SCHEME=get_zcopy". Its only guard is require_cuda_rndv_support() (mem-type + rndv supported), unlike the neighbouring CUDA tests which additionally call require_cuda_net_md() (test_ucp_proto.cc:102-108).
With an explicit RNDV_SCHEME, ucp_proto_rndv_cfg_thresh() (src/ucp/rndv/proto_rndv.inl:131-150) returns UCS_MEMUNITS_INF for every rendezvous variant except the forced get_zcopy one, and ucp_proto_rndv_ctrl_variant_cfg_thresh() propagates that INF to the am/rndv ctrl variants (proto_rndv.inl:168-182). So am/rndv is only selected at 1 MB if the rndv/get/zcopy remote variant is actually viable for the CUDA buffer — i.e. if an MD reachable from the ep lanes can register CUDA memory (see the md filtering in ucp_proto_rndv_ctrl_get_md_map(), src/ucp/rndv/proto_rndv.c:57-125). When it is not, eager protocols cover 1 MB, EXPECT_STREQ("am/rndv", ...) fails, select_am_rndv_remote_proto_config() returns NULL and ASSERT_NE(nullptr, remote_proto_config) fires.
Note the two sibling tests that use the same helper but default RNDV_SCHEME (am_default_remote_model_preserves_legacy_key, am_force_remote_model_remains_untagged) passed, and the RMA analogue passes because it restricts PROTOS to put/rndv,get/rndv,rndv/* (test_ucp_proto.cc:319) so a rndv proto wins by elimination — exactly the asymmetry you'd expect from this mechanism rather than from the rkey-config change.
Caveat: the Azure log I could retrieve is only the tail of the job, so the gtest failure text/line for this test is not in it; the above is derived from the test source + selection logic, and the remaining possibility (EXPECT_EQ(0, count_rkey_configs_with_flag(..., UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION))) is unlikely because with rndv_mode != AUTO ucp_proto_rndv_shm_pipeline_force_enabled() (proto_rndv.inl:20-28) short-circuits and the flag at proto_rndv.c:245-249 is never set.

Implicated commit: Test introduced by the CUDA-staging-force series (9cc08081 / merge d9c844ae, Nathan Bellalou — “use tag-only signal for CUDA staging force”, PR #11641); recent selection changes in master that can flip this range are fe0ea858 (Thomas Vegas, “UCP/RMA: Make put/rndv and get/rndv a fallback of zcopy”, #11940) and the topology assignment commits fd1eb5e0 / [REDACTED:Hex High Entropy String]. The PR under test (f4f50c5a, jeffnvidia — “UCP: Address stable rkey config review feedback”, PR #11676) only touched rkey-config storage (src/ucp/core/ucp_worker.c:2305-2420, src/ucp/core/ucp_worker.inl:372-378, src/ucp/proto/proto_select.c:606-666) and the gtest accessors; nothing there changes the forced-provenance decision.

File: test/gtest/ucp/test_ucp_proto.cc:830 (test body) and test/gtest/ucp/test_ucp_proto.cc:443 (EXPECT_STREQ("am/rndv", ...))

Suggested fix:

  1. Make the test environment-independent: add require_cuda_net_md(); (or skip when rndv/get/zcopy is not viable for CUDA) to am_explicit_scheme_disables_force_provenance, mirroring test_ucp_proto_mock_rcx_cuda::require_cuda_net_md() usage in the other CUDA tests. Alternatively, restrict PROTOS to the rndv set as test_ucp_proto_rma_rndv::init() does (test_ucp_proto.cc:317-321), or assert only the thing the test actually targets — remote_proto_config->select_param.op_id_flags == UCP_OP_ID_RNDV_RECV and `count_rkey_configs_with_flag(..., UCP_RKEY_CONFIG_FLAG_

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e56ee184-b6f1-414e-ad1b-108699c5536b in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_rkey.inl Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread test/gtest/ucp/test_ucp_peer_failure.cc Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_rkey.inl Outdated
{
ucs_assert(rkey->cfg_index != UCP_WORKER_CFG_INDEX_NULL);
return &ucs_array_elem(&worker->rkey_config, rkey->cfg_index);
return ucs_array_elem(&worker->rkey_config, rkey->cfg_index);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_rkey_config() is on the ucp_put_nbx short path (rma_send.c:169 reads rkey_config->put_short), and the indirection adds one extra dependent load there. did you measure small-put latency?

Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.inl Resolved
Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
static size_t count_rkey_configs_with_flag(ucp_worker_h worker,
uint8_t flag)
{
ucp_worker_cfg_index_t rkey_cfg_index;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: pls keep ucs_array_for_each(), same as in ucp_worker_trace_configs(); the manual index loop is longer for no gain. Same in test_ucp_peer_failure.cc:794.

Suggested change
ucp_worker_cfg_index_t rkey_cfg_index;
size_t count = 0;
ucp_rkey_config_t **rkey_config_p;
ucs_array_for_each(rkey_config_p, &worker->rkey_config) {
count += !!((*rkey_config_p)->key.flags & flag);
}

@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual note: rkey_config_stable_after_growth asserts ASSERT_NE(old_buffer, ucs_array_begin(...)), which couples the test to the ucs_array_reserve(&worker->rkey_config, 32) constant in ucp_worker.c:2764 — bumping that reserve above 65 would make the test fail spuriously rather than silently lose coverage. Acceptable, but worth knowing.

Residual note: the test verifies pointer identity, not that put_short/proto_select of an already-created config stay functionally correct after growth; and the 65-config loop runs 65 full UCP_OP_ID_PUT selection passes. Both were raised in earlier rounds.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_rkey.inl Outdated
{
ucs_assert(rkey->cfg_index != UCP_WORKER_CFG_INDEX_NULL);
return &ucs_array_elem(&worker->rkey_config, rkey->cfg_index);
return ucs_array_elem(&worker->rkey_config, rkey->cfg_index);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ucp_rkey_config() is on the short data path (ucp_put_send_short() in rma_send.c, and ucp_amo_send_nbx()), so every put_short/AMO now pays an extra dependent load. docs/OptimizationStyle.md asks to avoid extra pointer dereferences there — can you share ucx_perftest put latency / message-rate numbers before and after?

Comment thread src/ucp/core/ucp_worker.c Outdated
sizeof(buf)));
}

rkey_config_p = ucs_array_append(&worker->rkey_config,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops the deferred old-buffer release that ucp_worker_config_array_append() gave (ucp_worker_config_array_release_old()), so ep_config and rkey_config now use two different strategies for the same stale-pointer problem. Can we apply the same indirection to ep_config, or note why it is not affected, so the deferred-release mechanism can eventually be dropped?

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
static size_t count_rkey_configs_with_flag(ucp_worker_h worker,
uint8_t flag)
{
ucp_worker_cfg_index_t rkey_cfg_index;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor:

Suggested change
ucp_worker_cfg_index_t rkey_cfg_index;
ucp_rkey_config_t **rkey_config_p;
size_t count = 0;
ucs_array_for_each(rkey_config_p, &worker->rkey_config) {
count += !!((*rkey_config_p)->key.flags & flag);

ucs_array_for_each still works with the pointer element type, as used in ucp_worker_trace_configs()/ucp_worker_destroy_configs() in this PR; same applies to setup_progress_mock() in test_ucp_peer_failure.cc.

@svc-ucx

svc-ucx commented Sep 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit 485e4d60

TL;DR: The UCT test dc_mlx5/uct_p2p_mix_test.mix_10000/2 <dc_mlx5/mlx5_1:1> hung for 15 minutes waiting for a zcopy completion on the RoCE DC transport, and the gtest watchdog aborted the run (SIGTERM→signal 6); this is a UCT/dc_mlx5 completion hang, not something PR #11676 (a UCP-level rkey-config change) can cause — re-run the job and, if it reproduces, investigate the recent mlx5 completion-handling change (#11915).

Full analysis

Summary: roce on worker 0 job failed because dc_mlx5/uct_p2p_mix_test.mix_10000/2 never completed; the gtest watchdog fired ("Connection timed out - abort testing", test_helpers.cc:57), aborting gtest and failing make test with "Aborted (core dumped)".

Root cause: A hang, not slowness. The test started at 17:09:52.24 and the next log line is at 17:24:52.23 — a single ~15-minute gap of total silence, exactly the watchdog timeout. The backtrace shows the watched thread was spinning in uct_p2p_mix_test::random_op() → uct_test::progress() → uct_dc_mlx5_iface_progress_ll() (src/uct/ib/mlx5/dc/dc_mlx5.c:418), i.e. it was stuck in the unbounded loop at test/gtest/uct/test_p2p_mix.cc:162-168:

while (comp.count > 0) { progress(); }

So a posted DC operation (zcopy/atomic on mlx5_1:1, RoCE) never generated its completion, so comp.count never reached 0. Note the test's own time limit (test_p2p_mix.cc:210) is only checked between operations, so this inner wait can never time out — hence the 15-minute wall-clock stall and watchdog kill.

This is entirely within the UCT DC/mlx5 path. PR #11676 ("UCP/RKEY: Keep configurations at stable addresses", commits 6d6b0731 / f4f50c5a by jeffnvidia) only touches UCP rkey-configuration storage; uct_p2p_mix_test does not use UCP at all, so the PR is not implicated by any evidence in the log. The most plausible code-side suspect for a lost UCT completion is the recent mlx5 completion-tracking change 68cfcbc5 "UCT/IB/MXL5: uct comp marker (#11915)" (merged 5 days before this run), otherwise a dropped RoCE packet / stuck DCI on the test host swx-rain04.

Implicated commit: Not PR #11676. Candidate on the merge base: 68cfcbc5 "UCT/IB/MXL5: uct comp marker (#11915)", Zihao Zhao (also relevant: [REDACTED:Hex High Entropy String] "UCT: outstanding purge implementation", Zihao Zhao). Otherwise environment/flake — unknown.

File: src/uct/ib/mlx5/dc/dc_mlx5.c:418 (hang site in progress loop); test wait loop test/gtest/uct/test_p2p_mix.cc:166

Suggested fix:

  1. Re-trigger the roce on worker 0 job on PR UCP/RKEY: Keep configurations at stable addresses #11676 — nothing in the log links the failure to the rkey-config change; treat it as a merge-base/flaky failure unless it reproduces.
  2. If it reproduces, debug the dc_mlx5 lost-completion: run dc_mlx5/uct_p2p_mix_test.mix_10000 with UCX_LOG_LEVEL=trace_data/UCX_IB_TX_CQE_ZIPPING/DC pool settings on swx-rain04, and review 68cfcbc5 (UCT/IB/MXL5: uct comp marker #11915) for cases where a TX CQE is consumed without invoking uct_completion_t::func (e.g. comp-marker descriptors skipped in uct_rc_mlx5_txqp_process_tx_cqe / uct_dc_mlx5_poll_tx).
  3. Independently, make the failure diagnosable instead of a 15-minute blackout: bound the wait in uct_p2p_mix_test::random_op() (e.g. ucs_get_accurate_time() deadline around the while (comp.count > 0) loop, then UCS_TEST_ABORT/dump iface state) so a lost completion reports the stuck operation rather than tripping the global watchdog.

Related: PR #11676 (#11676), PR #11915 "UCT/IB/MXL5: uct comp marker" (#11915), PR #11668 "UCT: outstanding purge implementation"

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 4b9b5bbf-4806-40ba-9fa6-75e949b3f937 in the triage console for the audit trail.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.h Resolved
Comment thread src/ucp/core/ucp_worker.h Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
Comment thread src/ucp/core/ucp_worker.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.


return &ucs_array_elem(&worker->rkey_config, *new_rkey_cfg_index)
.proto_select;
return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index)->proto_select;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the joined line is 82 columns, over the 80-column limit (docs/CodeStyle.md, .clang-format ColumnLimit: 80); the git-clang-format codestyle job will emit a patch for it.

Suggested change
return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index)->proto_select;
return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index)
->proto_select;

Comment thread src/ucp/core/ucp_worker.c
status = UCS_ERR_NO_MEMORY;
goto err;);
rkey_cfg_index = ucs_array_length(&worker->rkey_config);
rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: the new allocation failure returns UCS_ERR_NO_MEMORY without any log; the path it replaced logged from ucs_array_grow(), so this is the first place deciding the error (docs/CodeStyle.md, error handling).

Suggested change
rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config");
rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config");
if (rkey_config == NULL) {
ucs_error("failed to allocate rkey configuration");
status = UCS_ERR_NO_MEMORY;
goto err;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nearby functions already do the same

Comment thread src/ucp/core/ucp_worker.h Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual note: the fix adds one dependent load to ucp_rkey_config(), which is on the ucp_put_short()/AMO short path ("avoid extra pointer dereference" in docs/OptimizationStyle.md). Acceptable for a correctness fix, but worth a latency sanity check with ucx_perftest.

Residual note: ucp_worker_rkey_config() lives in ucp_rkey.inl while its sibling ucp_worker_ep_config() is in ucp_worker.inl; both work, only a placement/consistency observation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants