Skip to content

[CI][Bugfix][Core] Size KV cache from torch reserved when profiling sees a release on any platform - #62806

Closed
wenjinhust wants to merge 1 commit into
vllm-project:mainfrom
wenjinhust:dev2610/fix_ci/fix_rocm
Closed

wenjinhust wants to merge 1 commit into
vllm-project:mainfrom
wenjinhust:dev2610/fix_ci/fix_rocm

Conversation

@wenjinhust

@wenjinhust wenjinhust commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Generalize the ROCm-only startup-profiling fallback from #53591 to every
platform: when profiling measures a release of device memory by other
processes, size the KV cache from this process's torch reserved memory
instead of aborting. Fixes #56830 and the deterministic NVIDIA NIXL PD CI
failure (#62579's 2-GPU accuracy core job).

Claims

  • Startup no longer aborts with AssertionError: Error in memory profiling.
    when other processes on the GPU release memory during profiling —
    previously only ROCm had the fallback; CUDA and others kept the hard assert.
  • Unblocks the nvidia-l4 nixlconnector-pd-accuracy-core CI job, which
    failed 3/3 deterministically (device free memory 14.91 → 17.29 GiB while a
    decode worker profiled).
  • Existing ROCm behavior is unchanged; the removed assert is dead code once
    the fallback exists (its trigger condition is exactly the fallback
    condition).
  • Unit tests now pin the fallback contract for all platforms instead of a
    ROCm-only parametrization.

Validation

  • Unit tests (pure function, no accelerator required — passes locally):

    pytest tests/v1/worker/test_gpu_worker.py
    
    • test_profiling_fallback_declines_when_free_memory_dropped
    • test_profiling_fallback_replaces_a_released_measurement
    • test_profiling_fallback_never_returns_a_negative_amount
  • Lint:
    ruff check vllm/v1/worker/gpu_worker.py tests/v1/worker/test_gpu_worker.py
    is clean.

  • Real-world regression: without the patch the NIXL PD job fails on every
    retry with:

    AssertionError: Error in memory profiling. Initial free memory 14.91 GiB,
    current free memory 17.29 GiB. This happens when other processes sharing
    the same container release GPU memory while vLLM is profiling during
    initialization. ...
    

    (build 96123, 3/3 retries with identical values). With the patch the
    worker logs
    Free memory grew by 2.38 GiB during profiling (...) Sizing the KV cache from the ... torch reserved instead of the profiling difference. and
    startup proceeds.
    CI: <fill in Buildkite build link(s) showing the nixlconnector job green>.

Details


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

…e on any platform

Signed-off-by: wenjinhust <wenjin.sh@huawei.com>
@wenjinhust
wenjinhust requested a review from njhill as a code owner October 21, 2026 03:20

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@wenjinhust wenjinhust changed the title [Core] Size KV cache from torch reserved when profiling sees a release on any platform [CI][Bugfix][Core] Size KV cache from torch reserved when profiling sees a release on any platform Oct 21, 2026
@mergify mergify Bot added the bug Something isn't working label Oct 21, 2026
@wenjinhust wenjinhust closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Startup memory-profiling assertion aborts whenever free memory grows, contradicting documented support for GPU co-tenancy

1 participant