Skip to content

[Startup] Skip the CUDA graph memory profile when its estimate is disabled - #64622

Merged
njhill merged 4 commits into
vllm-project:mainfrom
emmericp:skip-cudagraph-memory-profile-when-disabled
Nov 6, 2026
Merged

njhill merged 4 commits into
vllm-project:mainfrom
emmericp:skip-cudagraph-memory-profile-when-disabled

Conversation

@emmericp

@emmericp emmericp commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Cleanup: with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, determine_available_memory() still runs profile_cudagraph_memory() and then ignores the result. Skip the call when the flag is 0. AI assistance (Claude Code) was used to prepare this PR; I reviewed every changed line.

Since #56492 (extensible KV cache), MRV2's profile_cudagraph_memory() returns 0 when the extensible cache is on. The startup saving therefore applies only to fallback configurations (MRV1, elastic EP, LMCacheMPConnector, etc.) with the flag at 0.

Claims

Fallback configurations with the flag at 0 no longer run the throwaway capture pass. On current main the pass costs:

Fallback configuration Estimate pass on main Startup, main → this PR
MRV2, --no-enable-extensible-kv-cache 89 s, 90 s 260.8 s → 171.0 s
MRV1 (VLLM_USE_V2_MODEL_RUNNER=0) 3 s, 3 s within run-to-run noise

Setup: Qwen3.5-35B-A3B, TP1, one GB300, --compilation-config '{"max_cudagraph_capture_size": 4096}', VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, warm caches, main at c96e5f6. The estimate pass is the span from vLLM's Profiling CUDA graph memory log line to Estimated CUDA graph memory (MRV1), and the first Graph capturing finished line (MRV2). The real capture takes the same time with and without this PR (MRV2: 95 s and 96 s). The earlier 95.1 s → 72.5 s figure was measured before #56492 and is superseded.

There is no behaviour change with the flag at 1 (the default).

  • The "profiling is disabled" warning now fires only when the flag is 0, graphs may be captured, and the extensible cache is off. It no longer suggests a --gpu-memory-utilization value, since that needed the skipped estimate.

Validation

test_cudagraph_memory_profile_runs_only_when_applied covers MRV1, MRV2 with and without the extensible cache, and MRV2 without graphs. For each case it checks the estimate call, the KV cache budget and the warning.

$ pytest tests/v1/worker/test_gpu_worker.py -k cudagraph_memory_profile_runs_only_when_applied
7 passed                    # this PR (CPU)
3 failed, 4 passed          # main: all three flag-0 cases still run the estimate
3 failed, 4 passed          # previous revision of this PR: warns with the extensible cache on, and for MRV2 without graphs

CI runs this file in the H200 "V1 Executor + Worker" step (pytest -v -s v1/worker, .buildkite/test_areas/misc.yaml).


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

…abled

With VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 the estimate was still
computed (a full capture pass) and then ignored. Skip the pass and keep
the warning without the suggested utilization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Emmerich <pemmerich@nvidia.com>
@emmericp
emmericp requested a review from njhill as a code owner October 30, 2026 00:46

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@aijanai aijanai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works and simplifies the code

@njhill njhill left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the "compute it, then ignore it" logic is a worthwhile cleanup, but #56492 (extensible KV cache) landed after this branched, and that changes a few things. MRV2's profile_cudagraph_memory() now returns 0 when the extensible cache is on, so this skip only saves time in fallback configurations (MRV1, elastic EP, LMCacheMPConnector, etc.) with the flag set to 0. The branch merges cleanly, but needs these changes first:

  1. Rebase onto main and fix the warning condition. elif graphs_captured: doesn't check the flag, and graphs_captured is true for any MRV2 run on CUDA. After a rebase, a default startup (extensible cache on, flag at 1) has an estimate of 0, so it would warn that the flag is 0 and that graph memory isn't accounted for. This also happens on the current branch with MRV2, cudagraph_mode=NONE, no encoder graphs and the flag at 1. Suggested condition:

    elif (
        graphs_captured
        and not envs.VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS
        and not self.cache_config.enable_extensible_kv_cache
    ):

    This still warns for MRV2 with the flag at 0 when no graphs will be captured, because the runner makes that decision. Renaming graphs_captured to something like may_capture_graphs would make that clearer.

  2. Repair the test stub and assert on the warning. From reading the merged code (I haven't run it), main's determine_available_memory reads cache_config.resolved_gpu_memory_utilization in an eagerly evaluated logger.debug call, and the stub only sets gpu_memory_utilization, so both cases should raise AttributeError after a rebase. Please also cover the warning, including the extensible and flag-at-1 cases, since the current test only covers the MRV1 path and wouldn't catch the problem in (1). Also, the only explicit CI reference to this file I found runs just ::test_cpu_worker_without_triton (.buildkite/test_areas/misc.yaml), so please check that the new test actually runs in CI.

  3. Reword the PR as a cleanup. The startup saving now only applies to fallback configurations with the flag at 0, and the description should say so and mention #56492. The 95.1 s → 72.5 s figure predates #56492, so please either re-measure it on a fallback configuration or label it that way.

  4. Correct the new comment, or cut it back. "Captures every graph once" is wrong for MRV2 FULL graphs, which capture _FULL_GRAPH_PROFILING_SAMPLES (2) and extrapolate. "Can take minutes" also goes beyond the roughly 23 s measured here.

emmericp and others added 2 commits November 6, 2026 07:56
Warn about the disabled estimate only when the flag is 0 and the
extensible KV cache is off; with the extensible cache the capture
footprint is measured after warmup. Rename graphs_captured to
may_capture_graphs, cut the comment back, and extend the test to MRV2
with and without the extensible cache, asserting on the warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Paul Emmerich <pemmerich@nvidia.com>
@emmericp

emmericp commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I've reframed this as a cleanup: it removes the "compute the estimate, then ignore it" path, and it only saves startup time in fallback configurations with the flag at 0. I merged main and addressed all four points:

  1. Renamed graphs_captured to may_capture_graphs. The warning now uses your condition (flag at 0 and extensible cache off).
  2. The test stub now sets resolved_gpu_memory_utilization and enable_extensible_kv_cache and asserts on the warning. It has 7 cases: MRV1, MRV2 with and without the extensible cache at both flag values, and MRV2 without graphs at flag 1. The previous revision of this PR fails 3 of them (extensible cache at flag 1 and flag 0, and MRV2 without graphs), and main fails the 3 flag-0 cases. On CI: .buildkite/test_areas/misc.yaml now runs pytest -v -s v1/worker in the H200 "V1 Executor + Worker" step, so the whole file runs there.
  3. The description now says this is a cleanup and mentions [Core] Extensible KV cache #56492. I re-measured on current main with Qwen3.5-35B-A3B, TP1 on GB300, and max_cudagraph_capture_size 4096. With the extensible cache off on MRV2, the skipped pass costs 89–90 s, and startup drops from 260.8 s to 171.0 s. On MRV1 it costs 3 s. The old 95.1 s → 72.5 s figure is gone from the description.
  4. I cut the comment back to "The estimate is a throwaway capture pass; skip it when the flag says its result is not applied."

@njhill njhill left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @emmericp

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Nov 6, 2026
@njhill

njhill commented Nov 6, 2026

Copy link
Copy Markdown
Member

/ci run

@njhill njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 6, 2026
@njhill
njhill enabled auto-merge (squash) November 6, 2026 21:36
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #100675 for commit 0501cab9e082.

ℹ️ This PR is 67 commits behind upstream main. Merged upstream main into this branch as 0501cab9e082 before starting CI. Pull the branch before pushing more changes.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

✅ @emmericp, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@njhill
njhill merged commit 74bb766 into vllm-project:main Nov 6, 2026
248 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Nov 6, 2026
yeshihai pushed a commit to yeshihai/vllm that referenced this pull request Nov 8, 2026
…abled (vllm-project#64622)

Signed-off-by: Paul Emmerich <pemmerich@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
twu3202 added a commit to twu3202/vllm that referenced this pull request Nov 9, 2026
test_cudagraph_memory_profile_runs_only_when_applied (vllm-project#64622) replaces
memory_profiling with a SimpleNamespace that lacks the field this PR adds.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Tianyao Wu <rayroy31@gmail.com>
lesj0610 added a commit to lesj0610/vllm that referenced this pull request Nov 9, 2026
…counting-20261006

Resolve the conflict with vllm-project#64622 in determine_available_memory: keep
subtracting the KV cache init memory next to the CUDA graph estimate.

vllm-project#64622 skips profile_cudagraph_memory() when
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0, and that pass is where the
init memory is measured, so the test drops its flag-0 case. It also sets
the cache config fields determine_available_memory now reads.

vllm-project#63237 already bumped the startup plan schema to 2, so bump it to 3 to
keep ignoring plans recorded without this accounting.

Signed-off-by: lesj0610 <lesj0610@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants