Repository navigation
[CI][KV Connector] Start NIXL accuracy decoders after prefillers on shared GPUs - #62879
Merged
NickLucche merged 2 commits intoOct 21, 2026
Merged
Conversation
…hared GPUs With 2 GPUs, the DP_EP=1 config (prefiller TP=1 + decoder DP=2) puts decoder rank 1 on the prefiller's GPU, and both servers start at once. When the decoder loads weights while the prefiller is profiling, the prefiller counts them against its gpu_memory_utilization budget and gets negative KV cache memory. Wait for prefillers first whenever instances share GPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: khluu <khluu000@gmail.com>
khluu
requested review from
ApostaC,
ivanium,
orozery and
wzhao18
as code owners
October 21, 2026 10:18
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #96398 for commit ℹ️ This PR is 1 commit behind upstream |
6 tasks done
NickLucche
deleted the
claude/nixl-accuracy-serialize-shared-gpu-startup
branch
October 21, 2026 14:25
7 tasks done
bantianyue
pushed a commit
to bantianyue/vllm
that referenced
this pull request
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix
:nvidia: (L4) NixlConnector PD accuracy core. It has failed on every main build since 96279 and also hits PR builds, e.g. 96345. It failed intermittently in an earlier stretch too (96041 to 96139).The failing config is
DP_EP=1 GPU_MEMORY_UTILIZATION=0.4 PREFILLER_TP_SIZE=1 DECODER_TP_SIZE=2 MODEL_NAMES=deepseek-ai/deepseek-vl2-tiny:Root cause: a startup race on a shared GPU. The L4 step has 2 GPUs. The prefiller gets GPU 0, and the decoder (DP=2) gets
CUDA_VISIBLE_DEVICES=1,0, so decoder rank 1 wraps onto GPU 0. All servers launch at once. If the decoder loads its weights while the prefiller is profiling, the prefiller counts the decoder's 4 GiB as its own usage, and its 0.4 × 22 GiB budget goes negative. The decoder's "Error in memory profiling" is a follow-on: the prefiller dies and frees GPU 0 in the middle of the decoder's own profiling.Nothing changed in the memory footprint between pass and fail. Prefiller weights are 6.29 GiB and the decoder takes 4.04 GiB per rank in both. Only the timing changed: the prefiller load took 10.7 s in passing 96249 and 17.5 s in failing 96279.
Fix: when the instances need more GPUs than the host has (so some GPUs are shared), wait for the prefillers to be ready before launching the decoders. Configs with dedicated GPUs keep launching everything in parallel. The prefiller is then steady when the decoder profiles, so the decoder correctly sees the prefiller's reservation as already used, and 0.4 × 22 GiB still fits in the remaining free memory.
Not a duplicate: #62806 changes how a worker reacts when another process frees memory during profiling, which is the decoder-side symptom. It doesn't stop the prefiller's budget going negative. #62568 restructures the sweep into pytest but keeps concurrent startup.
Test Plan
bash -nand shellcheck (pre-commit) pass.(L4) NixlConnector PD accuracy corestep will confirm.Notes
AI assistance (Claude Code) was used; reviewed by the submitter.
🤖 Generated with Claude Code