Repository navigation
UCP/RKEY: Keep configurations at stable addresses - #11676
Conversation
c465ff9 to
08f5005
Compare
|
Is it related fix? #11802 |
|
Yes, they are related. After comparing both current heads, I think they address the same underlying rkey-config address-instability, but with different coverage. #11802 keeps worker->rkey_config as an inline object array and reloads proto_select / rkey_config from the config indexes after nested calls that may grow the worker arrays. That addresses the synchronous recursive-allocation path while avoiding extra indirection. It also includes analogous ep_config reloads. #11676 instead makes each rkey configuration a separate allocation. Growing the index array can then no longer move a live rkey config, its key, or its proto_select; this makes pointer stability an invariant rather than requiring every nested-call boundary to be audited. It additionally serializes rkey_config_hash lookup/create with the worker async context. #11802 does not change that hash synchronization, and ucp_rkey_proto_resolve() still has the raw lookup/direct-add path there. So I see the rkey relocation changes as alternative strategies, with a tradeoff between allocation/indirection and maintaining a complete reload audit. The async hash serialization is complementary and should be retained whichever storage strategy is selected. If inline storage is preferred, I would evaluate the reloads from #11802 together with the async serialization from #11676. Conversely, the ep_config reload work in #11802 is complementary to #11676, since #11676 only stabilizes rkey configs. |
5f70c6c to
371a5d8
Compare
|
🤖 CI Triage Agent — TL;DR: The aarch64 (and every) build fails compiling Full analysisSummary: Root cause: The PR's own change (
This is not aarch64-specific; the file simply wasn't updated with the container-type change. Implicated commit: 9ebccd4 — "UCP/RKEY: Keep configurations at stable addresses" (jeffnvidia), the head commit series of PR #11676 (build commit 371a5d8) File: Suggested fix: In
Related: PR #11676 |
371a5d8 to
04d582b
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Assuming the last link meant #11802: we do not need all three rkey approaches. #11817 addresses the async-context race by keeping protocol selection out of async context, but it does not prevent synchronous recursive rkey-array relocation. For that remaining issue, #11676 and the rkey portion of #11802 are alternative strategies: #11676 provides stable object addresses, while #11802 reloads array-derived pointers after nested allocation. The ep-config reloads in #11802 are complementary because #11676 only changes rkey storage. So the complete choices are:
The rkey portions of #11676 and #11802 are not both required. |
|
@iyastreb @tvegas1 All feedback on this PR has been addressed. The lock changes were removed, the PR now contains only the stable rkey-config storage change and its tests, and all 162 CI checks pass. When you have a chance, could you please review the current version? This PR still depends on #11817 for keeping protocol selection out of async context. |
related question: #11817 (comment) |
04d582b to
e19fc05
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
@yosefe @tvegas1 @iyastreb #11817 is now merged. I rebased #11676 onto current master ( A fresh Gaia build (job 68106) completed successfully and passed all 31/31 focused rkey/protocol tests. Refreshed Azure CI is now running on the rebased head. The #11817 review explicitly deferred the rkey-config pointer-invalidation fix to another PR. Should #11676 be that follow-up, or do you prefer the #11802 contiguous-storage/index-relookup approach? |
|
Residual coverage gaps: no functional coverage that |
|
🤖 CI Triage Agent — TL;DR: The ASan/RoCE gtest job didn't time out from slowness — it hung: Full analysisSummary: gtest aborted ("Connection timed out - abort testing") because Root cause: Log timestamps show continuous test activity (one test every ~0.2–1.2 s) up to Implicated commit: File: Suggested fix:
Related: PR #11676 (this PR); commit
|
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage note: Residual note: the stale-pointer workarounds that this fix makes unnecessary (re-fetching |
|
🤖 CI Triage Agent — TL;DR: The ASAN GPU job failed on a single gtest expectation — Full analysisSummary: Root cause: The failing test is defined at Implicated commit: Test introduced by the CUDA-staging-force series ( File: test/gtest/ucp/test_ucp_proto.cc:830 (test body) and test/gtest/ucp/test_ucp_proto.cc:443 ( Suggested fix:
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
| { | ||
| ucs_assert(rkey->cfg_index != UCP_WORKER_CFG_INDEX_NULL); | ||
| return &ucs_array_elem(&worker->rkey_config, rkey->cfg_index); | ||
| return ucs_array_elem(&worker->rkey_config, rkey->cfg_index); |
There was a problem hiding this comment.
ucp_rkey_config() is on the ucp_put_nbx short path (rma_send.c:169 reads rkey_config->put_short), and the indirection adds one extra dependent load there. did you measure small-put latency?
| static size_t count_rkey_configs_with_flag(ucp_worker_h worker, | ||
| uint8_t flag) | ||
| { | ||
| ucp_worker_cfg_index_t rkey_cfg_index; |
There was a problem hiding this comment.
minor: pls keep ucs_array_for_each(), same as in ucp_worker_trace_configs(); the manual index loop is longer for no gain. Same in test_ucp_peer_failure.cc:794.
| ucp_worker_cfg_index_t rkey_cfg_index; | |
| size_t count = 0; | |
| ucp_rkey_config_t **rkey_config_p; | |
| ucs_array_for_each(rkey_config_p, &worker->rkey_config) { | |
| count += !!((*rkey_config_p)->key.flags & flag); | |
| } |
|
Residual note: Residual note: the test verifies pointer identity, not that |
|
🤖 Starting review — findings will be posted here when done. |
| { | ||
| ucs_assert(rkey->cfg_index != UCP_WORKER_CFG_INDEX_NULL); | ||
| return &ucs_array_elem(&worker->rkey_config, rkey->cfg_index); | ||
| return ucs_array_elem(&worker->rkey_config, rkey->cfg_index); |
There was a problem hiding this comment.
ucp_rkey_config() is on the short data path (ucp_put_send_short() in rma_send.c, and ucp_amo_send_nbx()), so every put_short/AMO now pays an extra dependent load. docs/OptimizationStyle.md asks to avoid extra pointer dereferences there — can you share ucx_perftest put latency / message-rate numbers before and after?
| sizeof(buf))); | ||
| } | ||
|
|
||
| rkey_config_p = ucs_array_append(&worker->rkey_config, |
There was a problem hiding this comment.
This drops the deferred old-buffer release that ucp_worker_config_array_append() gave (ucp_worker_config_array_release_old()), so ep_config and rkey_config now use two different strategies for the same stale-pointer problem. Can we apply the same indirection to ep_config, or note why it is not affected, so the deferred-release mechanism can eventually be dropped?
| static size_t count_rkey_configs_with_flag(ucp_worker_h worker, | ||
| uint8_t flag) | ||
| { | ||
| ucp_worker_cfg_index_t rkey_cfg_index; |
There was a problem hiding this comment.
minor:
| ucp_worker_cfg_index_t rkey_cfg_index; | |
| ucp_rkey_config_t **rkey_config_p; | |
| size_t count = 0; | |
| ucs_array_for_each(rkey_config_p, &worker->rkey_config) { | |
| count += !!((*rkey_config_p)->key.flags & flag); |
ucs_array_for_each still works with the pointer element type, as used in ucp_worker_trace_configs()/ucp_worker_destroy_configs() in this PR; same applies to setup_progress_mock() in test_ucp_peer_failure.cc.
|
🤖 CI Triage Agent — TL;DR: The UCT test Full analysisSummary: Root cause: A hang, not slowness. The test started at So a posted DC operation (zcopy/atomic on This is entirely within the UCT DC/mlx5 path. PR #11676 ("UCP/RKEY: Keep configurations at stable addresses", commits Implicated commit: Not PR #11676. Candidate on the merge base: File: Suggested fix:
Related: PR #11676 (#11676), PR #11915 "UCT/IB/MXL5: uct comp marker" (#11915), PR #11668 "UCT: outstanding purge implementation"
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Starting review — findings will be posted here when done. |
|
|
||
| return &ucs_array_elem(&worker->rkey_config, *new_rkey_cfg_index) | ||
| .proto_select; | ||
| return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index)->proto_select; |
There was a problem hiding this comment.
minor: the joined line is 82 columns, over the 80-column limit (docs/CodeStyle.md, .clang-format ColumnLimit: 80); the git-clang-format codestyle job will emit a patch for it.
| return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index)->proto_select; | |
| return &ucp_worker_rkey_config(worker, *new_rkey_cfg_index) | |
| ->proto_select; |
| status = UCS_ERR_NO_MEMORY; | ||
| goto err;); | ||
| rkey_cfg_index = ucs_array_length(&worker->rkey_config); | ||
| rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config"); |
There was a problem hiding this comment.
minor: the new allocation failure returns UCS_ERR_NO_MEMORY without any log; the path it replaced logged from ucs_array_grow(), so this is the first place deciding the error (docs/CodeStyle.md, error handling).
| rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config"); | |
| rkey_config = ucs_malloc(sizeof(*rkey_config), "rkey_config"); | |
| if (rkey_config == NULL) { | |
| ucs_error("failed to allocate rkey configuration"); | |
| status = UCS_ERR_NO_MEMORY; | |
| goto err; | |
| } |
There was a problem hiding this comment.
nearby functions already do the same
|
Residual note: the fix adds one dependent load to Residual note: |
What?
Store each rkey configuration separately from the resizable index array, and
update its users and tests for pointer-based storage.
This PR now contains only the stable-storage change. It depends on #11817 to
keep protocol selection out of async context; it does not add async locking.
Why?
Protocol selection can recursively create another rkey configuration. With
inline storage, growing the array copies a partially initialized configuration
while the outer selection continues updating the old address. Later lookups
then use an incomplete copy.
This is reachable without an injected configuration: on Gaia, normal
default-reserve traffic naturally moved the array from 32 to 64 while an outer
rkey configuration was live. A diagnostic consistency check observed
put_shortdiverging between the old and current copies in 4/5 runs.How?
The resizable array stores pointers to separately allocated configurations.
Growing the array can move the pointer slots, but it cannot move a live
configuration. Worker cleanup destroys and frees each configuration.
Validation
masterand updated all current-master rkey-configtest accesses.
relocation check detected stale/current state divergence in 4/5 runs.
master+ UCP/PROTO: Initialize short thresholds from protocol selection #11817 + this PR: clean build, 31/31 selected tests,20/20 natural two-node runs, and 5/5 forced 32-to-33 pointer-array
relocations. Every forced marker reported synchronous context.