Skip to content

[CI][KV Connector] Start NIXL accuracy decoders after prefillers on shared GPUs - #62879

Merged
NickLucche merged 2 commits into
mainfrom
claude/nixl-accuracy-serialize-shared-gpu-startup
Oct 21, 2026
Merged

NickLucche merged 2 commits into
mainfrom
claude/nixl-accuracy-serialize-shared-gpu-startup

Conversation

@khluu

@khluu khluu commented Oct 21, 2026

Copy link
Copy Markdown
Member

Purpose

Fix :nvidia: (L4) NixlConnector PD accuracy core. It has failed on every main build since 96279 and also hits PR builds, e.g. 96345. It failed intermittently in an earlier stretch too (96041 to 96139).

The failing config is DP_EP=1 GPU_MEMORY_UTILIZATION=0.4 PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 MODEL_NAMES=deepseek-ai/deepseek-vl2-tiny:

(EngineCore) Model loading took 6.29 GiB memory
(EngineCore) Available KV cache memory: -2.34 GiB
ValueError: No available memory for the cache blocks.
(Worker_DP1_EP1) AssertionError: Error in memory profiling. Initial free memory 14.91 GiB, current free memory 17.29 GiB.

Root cause: a startup race on a shared GPU. The L4 step has 2 GPUs. The prefiller gets GPU 0, and the decoder (DP=2) gets CUDA_VISIBLE_DEVICES=1,0, so decoder rank 1 wraps onto GPU 0. All servers launch at once. If the decoder loads its weights while the prefiller is profiling, the prefiller counts the decoder's 4 GiB as its own usage, and its 0.4 × 22 GiB budget goes negative. The decoder's "Error in memory profiling" is a follow-on: the prefiller dies and frees GPU 0 in the middle of the decoder's own profiling.

Nothing changed in the memory footprint between pass and fail. Prefiller weights are 6.29 GiB and the decoder takes 4.04 GiB per rank in both. Only the timing changed: the prefiller load took 10.7 s in passing 96249 and 17.5 s in failing 96279.

96249 (pass) 96279 (fail)
Prefiller model load 6.29 GiB, 10.7 s 6.29 GiB, 17.5 s
Prefiller available KV cache 1.92 GiB −2.34 GiB
Decoder model load per rank 4.04 GiB 4.04 GiB

Fix: when the instances need more GPUs than the host has (so some GPUs are shared), wait for the prefillers to be ready before launching the decoders. Configs with dedicated GPUs keep launching everything in parallel. The prefiller is then steady when the decoder profiles, so the decoder correctly sees the prefiller's reservation as already used, and 0.4 × 22 GiB still fits in the remaining free memory.

Not a duplicate: #62806 changes how a worker reacts when another process frees memory during profiling, which is the decoder-side symptom. It doesn't stop the prefiller's budget going negative. #62568 restructures the sweep into pytest but keeps concurrent startup.

Test Plan

  • bash -n and shellcheck (pre-commit) pass.
  • Not run on a GPU locally. The (L4) NixlConnector PD accuracy core step will confirm.

Notes

AI assistance (Claude Code) was used; reviewed by the submitter.

🤖 Generated with Claude Code

…hared GPUs

With 2 GPUs, the DP_EP=1 config (prefiller TP=1 + decoder DP=2) puts decoder
rank 1 on the prefiller's GPU, and both servers start at once. When the
decoder loads weights while the prefiller is profiling, the prefiller counts
them against its gpu_memory_utilization budget and gets negative KV cache
memory. Wait for prefillers first whenever instances share GPUs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: khluu <khluu000@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@mergify mergify Bot added the kv-connector label Oct 21, 2026
@khluu

khluu commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #96398 for commit 427d540157ab.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as 427d540157ab before starting CI. Pull the branch before pushing more changes.

@NickLucche NickLucche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@NickLucche
NickLucche merged commit a0a645a into main Oct 21, 2026
36 checks passed
@NickLucche
NickLucche deleted the claude/nixl-accuracy-serialize-shared-gpu-startup branch October 21, 2026 14:25
bantianyue pushed a commit to bantianyue/vllm that referenced this pull request Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants