Skip to content

[TEMP][DO NOT REVIEW] Isolate PR 11685 mtype topology state - #11864

Closed
jeffnvidia wants to merge 16 commits into
openucx:masterfrom
jeffnvidia:agent/pr11685-mtype-cleanup-candidate
Closed

jeffnvidia wants to merge 16 commits into
openucx:masterfrom
jeffnvidia:agent/pr11685-mtype-cleanup-candidate

Conversation

@jeffnvidia

Copy link
Copy Markdown
Contributor

Diagnostic companion to #11685 and #11861.

This keeps the PR 11685 production changes and all of its tests. It changes only test_ucp_proto_mock_mtype_sys_dev to snapshot and restore the process-global topology registry, following the existing neighboring NUMA mock fixture pattern. Existing PR11685_DIAG instrumentation is retained.

Purpose: distinguish mtype-fixture topology leakage from a production protocol-selection regression. Do not review or merge.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
min_msg_length = thresh->max_msg_length + 1;
++thresh;
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: leftover PR11685_DIAG debugging output. The entire dump_am_rndv_selection() function (lines 403-479) only emits UCS_TEST_MESSAGE << "PR11685_DIAG ..." diagnostics. This is leftover debug instrumentation: it spams verbose messages on every call (used by all three test_ucp_proto_am_rndv tests), and the tag PR11685_DIAG references the wrong PR (this is #11864). Please remove dump_am_rndv_selection() before merge.

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
return nullptr;
}

dump_am_rndv_selection(select_elem);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: dump_am_rndv_selection() is unconditionally invoked here from select_am_rndv_remote_proto_config(). Remove this call site along with the debugging function.

Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
"RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y",
"RNDV_SCHEME=get_zcopy")
"RNDV_SCHEME=get_zcopy",
"PROTO_INFO=y")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: the paired "PROTO_INFO=y" config added to am_explicit_scheme_disables_force_provenance makes the test dump full protocol-selection info to a directory. This is debugging-only leftover; please remove it before merge.

"IB_NUM_PATHS?=1", "RNDV_SCHEME=put_zcopy")
{
constexpr size_t select_size = UCS_MBYTE;
uint8_t remote = 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: in set_frag_sys_dev(), md_index and mem_info are written by ucp_mm_get_alloc_md_index() but never read afterwards (the function directly overwrites alloc_md[...].sys_dev). This may trigger -Wunused-but-set-variable. Consider passing throwaway locals with UCS_V_UNUSED or documenting that only the side effect matters.

Comment thread src/ucp/rndv/proto_rndv.c Outdated
* protocol.
*/
mem_info.sys_dev = init_params->rkey_config_key->sys_dev;
mem_info.sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: flipping mem_info.sys_dev to UCS_SYS_DEVICE_ID_UNKNOWN in ucp_proto_rndv_remote_select_param_init aligns with the assertion in ucp_proto_common_get_lane_perf (sys_dev == select_param->sys_dev for SEND_ZCOPY), and the mtype prepare keeps them consistent for the local side. No correctness issue found.

@jeffnvidia
jeffnvidia force-pushed the agent/pr11685-mtype-cleanup-candidate branch from 83de702 to 6eb6bb5 Compare August 31, 2026 18:27
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Review complete — no issues found.

@svc-ucx

svc-ucx commented Aug 31, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 3) · commit 6eb6bb5b

TL;DR: The only failing test, rcx/test_ucp_proto_mock_rcx.rndv_remote_sys_dev_namespace/0, fails because commit 6c7abeee ("TEST: Restore base RNDV production code for control") reverted the remote-sys_dev isolation fix in src/ucp/rndv/proto_rndv.c while the gtest asserting that behavior was kept — restore the fix (set the remote select_param's sys_dev to UCS_SYS_DEVICE_ID_UNKNOWN) or drop/skip the test.

Full analysis

Summary: gtest run on "roce on worker 3" ended with 1 FAILED TEST — rcx/test_ucp_proto_mock_rcx.rndv_remote_sys_dev_namespace/0 (make: *** [Makefile:4713: test] Error 1, Bash exit code 2). All 8718 other tests passed; no hang, timeout or crash (run completed normally with per-test timings printed).

Root cause: The test asserts that a remote peer's process-local sys_dev value must not leak into the locally-computed remote protocol selection parameters:

EXPECT_EQ(remote_sys_dev, rkey_config->key.sys_dev);
EXPECT_EQ(UCS_SYS_DEVICE_ID_UNKNOWN,
          rpriv->remote_proto_config.select_param.sys_dev);

(test/gtest/ucp/test_ucp_proto_mock.cc:930-932)

But the current production code copies the rkey's sys_dev straight into the remote select param:

mem_info.sys_dev = init_params->rkey_config_key->sys_dev;   /* proto_rndv.c:346 */

so remote_proto_config.select_param.sys_dev equals remote_sys_dev (the mocked mock_0:1 device), not UCS_SYS_DEVICE_ID_UNKNOWN, and the second EXPECT_EQ fails. Git history shows why: the sys_dev namespace isolation was introduced in 5fa905a7 "UCP/PROTO/RNDV: Isolate remote system device namespace" (PR #11685) and then reverted on this branch by 6c7abeee "TEST: Restore base RNDV production code for control" (jeffnvidia, 2026-08-31T17:50Z, ~2h before this build), while the test added in 8e90a98b / touched by 0555af15 remained. The test is therefore correctly detecting that the production fix is missing — a self-inflicted branch inconsistency, not a flake or an infra problem.

Implicated commit: 6c7abeee — jeffnvidia, "TEST: Restore base RNDV production code for control" (reverts 5fa905a7 by Jeffrey Mahou)

File: src/ucp/rndv/proto_rndv.c:346 (assertion in test/gtest/ucp/test_ucp_proto_mock.cc:931)

Suggested fix: Re-apply the 5fa905a7 production change on this branch: in ucp_proto_rndv_remote_select_param_init(), when rkey_config_key != NULL, do not propagate the peer's sys_dev — set mem_info.sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN (keeping mem_info.type/flags as-is), since rkey_config_key->sys_dev is only meaningful in the remote process's device namespace. If commit 6c7abeee was intentionally a temporary control experiment, then drop 6c7abeee before pushing, or gate rndv_remote_sys_dev_namespace so it isn't run without the fix — otherwise this job will keep failing deterministically on every rerun.

Related: PR #11685 "UCP/PROTO/RNDV: Isolate remote system device namespace" (#11685); this build is PR #11864 (branch agent/pr11685-mtype-cleanup-candidate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants