Skip to content

UCS/TOPO: Fix off-by-one error in max device id check - #11834

Merged
brminich merged 1 commit into
openucx:masterfrom
guy-ealey-morag:max-device-id-fix
Aug 27, 2026
Merged

brminich merged 1 commit into
openucx:masterfrom
guy-ealey-morag:max-device-id-fix

Conversation

@guy-ealey-morag

@guy-ealey-morag guy-ealey-morag commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

What?

Fix off-by-one error that could potentially assign a device with id UCS_SYS_DEVICE_ID_UNKNOWN.

Why?

Previously UCS_TOPO_MAX_SYS_DEVICES was set to 256 which also included the special value of UCS_SYS_DEVICE_ID_UNKNOWN.
This means that a real device could be allocated an id that is equal to UCS_SYS_DEVICE_ID_UNKNOWN.
The correct value is 255, but instead of just changing the value I made the macros depend on each other so it would be more robust.

Signed-off-by: Guy Ealey Morag <gealeymorag@nvidia.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@openucx openucx deleted a comment from svc-nvidia-pr-review Aug 26, 2026
@openucx openucx deleted a comment from svc-nvidia-pr-review Aug 26, 2026
Comment thread test/gtest/ucs/test_topo.cc Resolved
Comment thread test/gtest/ucs/test_topo.cc Resolved
Comment thread test/gtest/ucs/test_topo.cc Resolved
Comment thread src/ucp/core/ucp_rkey.c Resolved
Comment thread src/ucp/core/ucp_rkey.c Resolved
@svc-nvidia-pr-review

Copy link
Copy Markdown

No blocking issues. Boundary regression-test gap (registering up to num_devices == UCS_SYS_DEVICE_ID_COUNT to prove id 255 is never handed out and stays distinct from UCS_SYS_DEVICE_ID_UNKNOWN) remains a reasonable nice-to-have but is low risk given the ucs_assert_always(num_devices < UCS_SYS_DEVICE_ID_COUNT) guard. This duplicates an existing unresolved comment rather than warranting a new thread.

@svc-ucx

svc-ucx commented Aug 26, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 810f53b7

TL;DR: The ASan/roce gtest job died while rcx/test_ucp_fault_tolerance.target_failure/1 <rc_x/PUT|FLUSH> was hung for ~11.3 minutes with zero output (a UCP flush/request that never completed), and the Azure agent shutdown/cancel is only the symptom; this hang is in the UCP fault-tolerance/recovery path and is unrelated to the topo assert change in this PR, so re-run and track the FT test hang separately.

Full analysis

Summary: Job "AddressSanitizer roce on worker 0" (build 132669) was cancelled ("The agent has received a shutdown signal" / "The operation was canceled") while shard 1/4 of the gtest run was stuck in rcx/test_ucp_fault_tolerance.target_failure/1 <rc_x/PUT|FLUSH>.

Root cause: Log timeline shows the test started at 11:19:44.28 and produced no output at all until 11:31:00 — a single ~676 s gap that is essentially the whole remaining runtime, i.e. the process was hung, not slow. The only application line emitted at the end is ucp_test.cc:300 UCX ERROR request 0x62e000022418 did not complete on time, which comes from ucp_test::request_process() when a waited-on request never completes within ucs::get_deadline(). Since no UCS_TEST_MESSAGE ([ INFO ] Attempting PUT|FLUSH operation before failure injection...) was printed, the hang occurred before the first RMA op — i.e. in init()/flush_workers() at test_ucp_fault_tolerance.cc:382, where the wireup-completing flush never finished on rc_x (test config uses configure_peer_failure_settings() + KEEPALIVE_INTERVAL=0.3s, which under ASan slowdown can trigger peer-failure/recovery handling during wireup and leave the flush request pending). The gtest watchdog (900 s) and the per-request deadline had not yet fired, so the Azure agent cancel/shutdown ended the job first.

The PR head commit [REDACTED:Hex High Entropy String] ("UCS/TOPO: Fix off-by-one error in max device id check") only touches the ucs_assert_always(ucs_topo_global_ctx.num_devices < UCS_SYS_DEVICE_ID_COUNT) bound in ucs_topo_find_device_by_bus_id_value(); with UCS_SYS_DEVICE_ID_COUNT = 255 and a handful of devices on this node, it cannot affect RC_X wireup/flush behavior. The failure is not attributable to this PR.

Implicated commit: Not the PR commit ([REDACTED:Hex High Entropy String], Guy Ealey Morag — topo assert bound only). Most likely area of the hang: recent UCP fault-tolerance work, db208ee4 "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)" and 9db2e488 "UCP/FT: rebuild failed lanes on recovery via LANES_ADDR exchange (#11562)" (Evgeny Leksikov) — unconfirmed, no stack trace was captured.

File: test/gtest/ucp/test_ucp_fault_tolerance.cc:382 (flush_workers() in test_rma_with_injected_failure); timeout reported from test/gtest/ucp/ucp_test.cc:300

Suggested fix:

  1. Re-run the ASan roce job — this PR's change is inert with respect to the hung test; do not treat it as a blocker for UCS/TOPO: Fix off-by-one error in max device id check #11834.
  2. For the real bug: make the hang diagnosable instead of silent. Lower/verify the watchdog and per-request deadlines under ASan so the gtest watchdog fires before the CI agent kills the job, and dump ucp_ep/lane state (e.g. ucp_ep_print_info / ucs_debug_print_backtrace) when request_process() hits the "did not complete on time" path, so the stuck lane and pending flush op are identified.
  3. Investigate the FT flush path: test_ucp_fault_tolerance sets KEEPALIVE_INTERVAL=0.3s together with peer-failure settings (test_ucp_fault_tolerance.cc:50); on rc_x under ASan this interval is very aggressive and can start keepalive/lane-recovery while wireup is still in flight, leaving the flush request never completing. Consider scaling KEEPALIVE_INTERVAL by ucs::test_time_multiplier() and auditing the recovery state machine for a flush request that is never progressed when a lane enters recovery.

Related: PR #11834 (this PR, UCS/TOPO off-by-one); FT-area PRs #11563, #11562, #11810; no existing issue found for this specific hang — worth opening one.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 5c8e7622-cf3f-453c-8ba7-eb81177bb4a7 in the triage console for the audit trail.

Comment thread src/ucs/sys/topo/base/topo.h Resolved
@brminich
brminich merged commit 45a9372 into openucx:master Aug 27, 2026
162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants