Skip to content

CUDA_IPC: Don't show CUDA error on exit, from static destroy - v1.23 - #11852

Merged
brminich merged 1 commit into
openucx:v1.23.xfrom
iyastreb:iyastreb/cuda-error-exit-1.23
Aug 29, 2026
Merged

brminich merged 1 commit into
openucx:v1.23.xfrom
iyastreb:iyastreb/cuda-error-exit-1.23

Conversation

@iyastreb

Copy link
Copy Markdown
Contributor

What?

Backport of #11846
Address ai-dynamo/nixl#2171

Why?

We should ignore this diagnostic on cleanup, because it's not a real issue, as all this mem handles will be destroyed anyway, as the cuda driver is shutting down

How?

Add log_level param

@iyastreb
iyastreb requested review from rakhmets and tvegas1 August 28, 2026 12:46
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-nvidia-pr-review

Copy link
Copy Markdown

Test coverage gap: this is a static-cleanup / driver-shutdown ordering behavior that is hard to unit test, and there is no new test exercising the close_handles == 0 path. Understandable given the shutdown-race nature, but the change is validated only by existing CI and manual observation. If practical, note in the PR description how the deinitialized path was verified.

@svc-ucx

svc-ucx commented Aug 28, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 0) · commit b3e19d05

TL;DR: The build failed because test/apps/test_hooks -n 26 segfaulted on iteration 13/30 of the "UCM Bistro hook test" — a known race between UCM bistro patching of mmap in libucs.so and the many threads calling mmap concurrently. It is unrelated to PR #11852 (a CUDA_IPC error-message backport); re-run the job and consider backporting the bistro grace-time/syscall-wait fixes to the v1.23 branch.

Full analysis

Summary: althca on worker 0 stage aborted with Bash exited with code '1' after contrib/test_jenkins.sh: line 1039: 12473 Segmentation fault timeout 10 ./test/apps/test_hooks -n $threads during iteration 13/30 (26 threads) of test_ucm_hooks.

Root cause: test_hooks starts N threads (random 1..2*NPROC; 26 in this run) that hammer mmap() in a tight loop, then dlopen()s libucs.so, which makes UCM install its bistro hooks by live-patching the first bytes of mmap in the loaded library. The patching sequence in ucm_bistro_apply_patch_atomic() (src/ucm/bistro/bistro.c:70-103) writes a 2-byte self-jump "lock", waits only ucm_global_opts.bistro_grace_duration, then memcpys the rest of the jump payload. A thread that had already prefetched/entered the function past the lock bytes can execute a half-written instruction stream and crash — exactly the failure mode this stress test exists to expose. With 26 threads racing, the grace period was insufficient and one thread SIGSEGV'd. Supporting evidence: the crash left test_hooks.log empty (cat printed nothing) because the process's block-buffered stdout was lost on the segfault, i.e. the crash happened during/right after patching rather than at a clean failure point. Nothing in this PR's area (CUDA_IPC static destructor error suppression, per PR #11852 / #11846) touches UCM, bistro, or the mmap hooks, so this is a pre-existing flake on the branch, not a regression from commit b3e19d0.

Also visible earlier in the same log, unrelated but worth noting: gcc: error: test_ucp_config-test_ucp_config.o: No such file or directory from the test_ucp_config step — a stale/incremental build artifact issue in the same workspace that did not fail the script.

Implicated commit: unknown (not caused by [REDACTED:Hex High Entropy String]). The relevant mitigations in master are c77bb48 "UCM/BISTRO: Increase grace time to reduce patch race" (#11112, shasson5) and #11088 "UCM/BISTRO: Wait for syscall completion before patching"; the v1.23 branch under test may predate them.

File: src/ucm/bistro/bistro.c:70-103 (ucm_bistro_apply_patch_atomic, grace-time window); test trigger at contrib/test_jenkins.sh:1039-1048; test at test/apps/test_hooks.c:82-89

Suggested fix:

  1. Re-run the althca job — this is a flaky, load-dependent race, not a PR regression.
  2. Backport c77bb48 (UCM/BISTRO: Increase grace time to reduce patch race #11112, increased UCX_BISTRO_GRACE_DURATION default) and UCM/BISTRO: Wait for syscall completion before patching #11088 (wait for syscall completion before patching) into the v1.23.x branch if they are not already there; verify with git log v1.23.x -- src/ucm/bistro.
  3. Improve triage-ability of this test: add setbuf(stdout, NULL) (or setvbuf(..., _IONBF, 0)) at the top of main() in test/apps/test_hooks.c so the diagnostic output survives a crash, and in test_ucm_hooks() (contrib/test_jenkins.sh:1043) capture the exit code and dump a core/backtrace (e.g. run under ulimit -c unlimited + gdb -batch -ex bt) instead of just cat test_hooks.log, which was empty here.
  4. Separately, investigate the stale-object gcc: error: test_ucp_config-test_ucp_config.o: No such file or directory on this agent's workspace (/scrap/azure/agent-02/...) — it suggests incremental-build state leaking between runs.

Related: #11112 (bistro grace time), #11088 (wait for syscall completion before patching), #11852 (this PR), #11846 (master counterpart)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f3bca380-1080-433d-89a7-6c31dbc2b731 in the triage console for the audit trail.

@brminich
brminich enabled auto-merge (squash) August 28, 2026 15:56
@brminich
brminich merged commit 224e064 into openucx:v1.23.x Aug 29, 2026
162 checks passed
@iyastreb
iyastreb deleted the iyastreb/cuda-error-exit-1.23 branch August 29, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants