Repository navigation
[CI][Bugfix][Core] Size KV cache from torch reserved when profiling sees a release on any platform - #62806
Closed
wenjinhust wants to merge 1 commit into
Closed
[CI][Bugfix][Core] Size KV cache from torch reserved when profiling sees a release on any platform#62806wenjinhust wants to merge 1 commit into
wenjinhust wants to merge 1 commit into
Conversation
…e on any platform Signed-off-by: wenjinhust <wenjin.sh@huawei.com>
6 tasks done
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Generalize the ROCm-only startup-profiling fallback from #53591 to every
platform: when profiling measures a release of device memory by other
processes, size the KV cache from this process's torch reserved memory
instead of aborting. Fixes #56830 and the deterministic NVIDIA NIXL PD CI
failure (#62579's 2-GPU accuracy core job).
Claims
AssertionError: Error in memory profiling.when other processes on the GPU release memory during profiling —
previously only ROCm had the fallback; CUDA and others kept the hard assert.
nvidia-l4 nixlconnector-pd-accuracy-coreCI job, whichfailed 3/3 deterministically (device free memory 14.91 → 17.29 GiB while a
decode worker profiled).
the fallback exists (its trigger condition is exactly the fallback
condition).
ROCm-only parametrization.
Validation
Unit tests (pure function, no accelerator required — passes locally):
test_profiling_fallback_declines_when_free_memory_droppedtest_profiling_fallback_replaces_a_released_measurementtest_profiling_fallback_never_returns_a_negative_amountLint:
ruff check vllm/v1/worker/gpu_worker.py tests/v1/worker/test_gpu_worker.pyis clean.
Real-world regression: without the patch the NIXL PD job fails on every
retry with:
(build 96123, 3/3 retries with identical values). With the patch the
worker logs
Free memory grew by 2.38 GiB during profiling (...) Sizing the KV cache from the ... torch reserved instead of the profiling difference.andstartup proceeds.
CI: <fill in Buildkite build link(s) showing the nixlconnector job green>.
Details
total_consumedis the drop in device-global free memoryacross startup. It goes negative when another process on the same GPU
releases memory inside the profiling window — a co-tenant, or the paired
instance of a disaggregated deployment finishing its own startup. In the
failing NIXL PD config (2-GPU L4, prefill + decode sharing GPUs), a decode
DP1_EP1 worker profiles while the prefill instance finishes
deepseek-vl2-tiny's multimodal warmup and releases ~2.4 GiB.
which is process-local, unaffected by other processes, and a lower bound on
this instance's usage — is lifted from ROCm to all platforms. The warning
is retained and the result is clamped so it can never be negative.
conservatively (smaller than the device could actually support at that
moment). This is preferable to refusing to start, and consistent with
documented GPU co-tenancy ([Bug]: Startup memory-profiling assertion aborts whenever free memory grows, contradicting documented support for GPU co-tenancy #56830).
VLLM_MEMORY_PROFILER_ALLOW_FREE_MEMORY_INCREASEescape hatch (unmerged).This PR makes the safe path the default on every platform instead.
Worker.sleep()),fix(memory): resolve UMA concurrent profiling race condition and page cache leak #35929 (declined on co-tenancy-scope grounds, challenged by [Bug]: Startup memory-profiling assertion aborts whenever free memory grows, contradicting documented support for GPU co-tenancy #56830).
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)