Skip to content

[Core][MRV2] Warm up the sampler after CUDA graph capture to stop memory growth under load - #62560

Merged
dllehr-amd merged 17 commits into
vllm-project:mainfrom
GodRishUniverse:fix_sampler
Nov 2, 2026
Merged

dllehr-amd merged 17 commits into
vllm-project:mainfrom
GodRishUniverse:fix_sampler

Conversation

@GodRishUniverse

@GodRishUniverse GodRishUniverse commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Overview

On Model Runner V2, GPU memory keeps growing after startup while load ramps up. The sampler makes an fp32 copy of the [num_reqs, vocab] logits on every step. capture_model() ends with empty_cache(), so serving starts with no cached block of that size, and each new largest batch reserves a slightly bigger block while the smaller ones stay cached.

V1 already avoids this: compile_or_warm_up_model runs the sampler once at the largest batch after capture_model() ("on purpose to prevent memory buffers from being cleared by torch.accelerator.empty_cache"). This PR adds the same step for MRV2.

Depends on #57926. That PR makes _dummy_sampler_run take the fp32 path and counts it in profile_run. Without it, the dummy run skips the copy and this warm-up keeps nothing.

Claims

Validation

Qwen3-4B, TP1, --gpu-memory-utilization 0.9

main #57926 #57926 + this PR
KV cache (tokens) 106,912 105,680 105,680
Memory growth under load +1.78 GB (18 steps) +1.77 GB (18 steps) +0.11 GB (1 step)
Peak GPU memory under load 25.72 GB 25.54 GB 24.11 GB
Output tok/s, c=16 / 64 / 160 / 256 608 / 1279 / 1562 / 1519 607 / 1277 / 1563 / 1503 608 / 1276 / 1561 / 1499

Gemma-4-31B-it, TP4, --gpu-memory-utilization 0.95 --kv-cache-memory 3GB

#57926 #57926 + this PR
KV cache (tokens) 18,806 18,806
Memory growth under load (each GPU) +0.24 GB (3 steps) +0.003 GB
Output tok/s, c=8 / 32 / 64 / 128 133 / 496 / 520 / 536 133 / 496 / 521 / 537

Setup:

Run on main, #57926 and #57926 + this PR. Requests carry no sampling params, so each model's default top-k/top-p applies.

Qwen3-4B, 1 GPU

vllm serve Qwen/Qwen3-4B-Instruct-2507 --tensor-parallel-size 1 \
  --max-model-len 8192 --gpu-memory-utilization 0.9 --port 8000

vllm bench serve --backend openai --endpoint /v1/completions \
  --model Qwen/Qwen3-4B-Instruct-2507 --base-url http://localhost:8000 \
  --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos \
  --num-prompts 1280 --max-concurrency 256 --request-rate inf --seed 1256

Gemma-4-31B, 4 GPUs (fixed KV size so all builds get the same KV cache)

vllm serve google/gemma-4-31B-it --tensor-parallel-size 4 \
  --max-model-len 8192 --gpu-memory-utilization 0.95 \
  --kv-cache-memory 3221225472 --port 8000

vllm bench serve --backend openai --endpoint /v1/completions \
  --model google/gemma-4-31B-it --base-url http://localhost:8000 \
  --dataset-name random --random-input-len 256 --random-output-len 128 --ignore-eos \
  --num-prompts 640 --max-concurrency 128 --request-rate inf --seed 1128

Details

  • Not tested on NVIDIA; the code path is the same. I can test on it.

Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

AI assistance was used. I have reviewed it.

Signed-off-by: GodRishUniverse <risagarw@amd.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added nvidia mrv2 Model Runner V2 specific labels Oct 19, 2026
@GodRishUniverse
GodRishUniverse marked this pull request as draft October 19, 2026 21:45
…mment

Signed-off-by: GodRishUniverse <risagarw@amd.com>
@GodRishUniverse
GodRishUniverse marked this pull request as ready for review October 19, 2026 21:50

@dllehr-amd dllehr-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with this @yewentao256 are you good if we land this after #57926? Any issues with this PR?

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Oct 20, 2026
@dllehr-amd dllehr-amd added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 20, 2026
@github-actions

Copy link
Copy Markdown

✅ @GodRishUniverse, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@yewentao256 yewentao256 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!

Please do the following:

  • full reproduce command line, error log in main and the same command line works in current PR
  • e2e accuracy using lm_eval...
  • e2e benchmark using vllm bench serve...

When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
  | (EngineCore pid=1580)     super().__init__(
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
  | (EngineCore pid=1580)     self.scheduler: SchedulerInterface = Scheduler(
  | (EngineCore pid=1580)                                          ^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
  | (EngineCore pid=1580)     super().__init__(*args, **kwargs)
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
  | (EngineCore pid=1580)     self.kv_cache_manager = KVCacheManager(
  | (EngineCore pid=1580)                             ^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
  | (EngineCore pid=1580)     self.coordinator = get_kv_cache_coordinator(
  | (EngineCore pid=1580)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
  | (EngineCore pid=1580)     return HybridKVCacheCoordinator(
  | (EngineCore pid=1580)            ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
  | (EngineCore pid=1580)     assert all(
  | (EngineCore pid=1580)            ^^^^
  | (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
  | FAILED
  | v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2

# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) ==================
"lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"

# Now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9212|±  |0.0074|
|     |       |strict-match    |     5|exact_match|↑  |0.9212|±  |0.0074|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9151|±  |0.0077|
|     |       |strict-match    |     5|exact_match|↑  |0.9158|±  |0.0076|

`lm_eval run --model local-chat-completions --model_args base_url=http://127.0.0.1:9256/v1/chat/completions,model=$MODEL,num_concurrent=1024,seed=256 --tasks aime25 --trust_remote_code --apply_chat_template`
# now
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|
# main
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.7333|±  |0.0821|
vllm bench serve --model zai-org/GLM-5.2-FP8  --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16

# Now

============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.08      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              41.50     
Output token throughput (tok/s):         41.50     
Peak output token throughput (tok/s):    76.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          340041.81 
---------------Time to First Token----------------
Mean TTFT (ms):                          1669.15   
Median TTFT (ms):                        1669.02   
P99 TTFT (ms):                           2724.38   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     128       
Failed requests:                         0         
Benchmark duration (s):                  3.23      
Total input tokens:                      1048576   
Total generated tokens:                  128       
Request throughput (req/s):              39.63     
Output token throughput (tok/s):         39.63     
Peak output token throughput (tok/s):    50.00     
Peak concurrent requests:                128.00    
Total token throughput (tok/s):          324671.20 
---------------Time to First Token----------------
Mean TTFT (ms):                          1668.69   
Median TTFT (ms):                        1649.00   
P99 TTFT (ms):                           3126.43   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
---------------Inter-token Latency----------------
Mean ITL (ms):                           0.00      
Median ITL (ms):                         0.00      
P99 ITL (ms):                            0.00      
==================================================

Comment thread vllm/v1/worker/gpu_worker.py Resolved
@GodRishUniverse

GodRishUniverse commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

@yewentao256 here are the snippets and the files attached (please let me know if you need anything else). These are with the updated dummy tensor shape:

Stress test leads to OOM on main:

vllm serve Qwen/Qwen3-4B-Instruct-2507 --tensor-parallel-size 1 --max-model-len 8192 --kv-cache-memory 21045000000 --port 8000

vllm bench serve --backend openai --endpoint /v1/completions --model Qwen/Qwen3-4B-Instruct-2507 --base-url http://localhost:8000 --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos --num-prompts 1280 --max-concurrency 256 --request-rate inf --seed 1256

# now
$ grep -c 'allocation failed' pr_vllm_serve.log
0

# main
[rank0]:[W1026 18:14:01.745758963 HIPCachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 104857600 bytes (free: 0, total: 32061259776).
[rank0]:[W1026 18:14:03.486826687 HIPCachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 123731968 bytes (free: 0, total: 32061259776).
[rank0]:[W1026 18:14:05.287041872 HIPCachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 144703488 bytes (free: 0, total: 32061259776).  

Lm-eval:

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:8000/v1/completions,model=Qwen/Qwen3-4B-Instruct-2507,num_concurrent=256" --tasks gsm8k


# now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8552|±  |0.0097|
|     |       |strict-match    |     5|exact_match|↑  |0.8613|±  |0.0095|


# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8552|±  |0.0097|
|     |       |strict-match    |     5|exact_match|↑  |0.8613|±  |0.0095|


vLLM bench (not the stress test)

vllm bench serve --backend openai --endpoint /v1/completions --model Qwen/Qwen3-4B-Instruct-2507 --base-url http://localhost:8000 --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos --num-prompts 1280 --max-concurrency 256 --request-rate inf --seed 1256

# now
============ Serving Benchmark Result ============
Successful requests:                     1280      
Failed requests:                         0         
Maximum request concurrency:             256       
Benchmark duration (s):                  108.30    
Total input tokens:                      655360    
Total generated tokens:                  163840    
Request throughput (req/s):              11.82     
Output token throughput (tok/s):         1512.84   
Peak output token throughput (tok/s):    3099.00   
Peak concurrent requests:                280.00    
Total token throughput (tok/s):          7564.18   
---------------Time to First Token----------------
Mean TTFT (ms):                          6940.22   
Median TTFT (ms):                        4949.50   
P99 TTFT (ms):                           17719.45  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          108.25    
Median TPOT (ms):                        109.30    
P99 TPOT (ms):                           152.36    
---------------Inter-token Latency----------------
Mean ITL (ms):                           108.26    
Median ITL (ms):                         62.32     
P99 ITL (ms):                            195.97    
==================================================

# main
============ Serving Benchmark Result ============
Successful requests:                     1280      
Failed requests:                         0         
Maximum request concurrency:             256       
Benchmark duration (s):                  108.08    
Total input tokens:                      655360    
Total generated tokens:                  163840    
Request throughput (req/s):              11.84     
Output token throughput (tok/s):         1515.93   
Peak output token throughput (tok/s):    3073.00   
Peak concurrent requests:                279.00    
Total token throughput (tok/s):          7579.67   
---------------Time to First Token----------------
Mean TTFT (ms):                          6918.76   
Median TTFT (ms):                        4932.72   
P99 TTFT (ms):                           17666.86  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          108.07    
Median TPOT (ms):                        108.90    
P99 TPOT (ms):                           151.42    
---------------Inter-token Latency----------------
Mean ITL (ms):                           108.07    
Median ITL (ms):                         62.50     
P99 ITL (ms):                            195.17    
==================================================

Stress tests:
main_vllm_bench_serve_stress.log
pr_vllm_bench_serve_stress.log

The serve files for above are:
main_vllm_serve_stress.log
pr_vllm_serve_stress.log

Main LM eval and vllm bench
main_lm_eval_gsm8k.log
main_vllm_bench_serve.log

PR LM eval and vllm bench
pr_lm_eval_gsm8k.log
pr_vllm_bench_serve.log

@dllehr-amd

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #97649 for commit afe76bb6e814.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #97960 for commit 9917deb1f0e0.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98325 for commit bfd1c5b78108.

ℹ️ This PR is 2 commits behind upstream main. Merged upstream main into this branch as bfd1c5b78108 before starting CI. Pull the branch before pushing more changes.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 2 failed job(s) for retry in Buildkite CI #98325.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98719 for commit 81109f9e4cbf.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #98719.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #98719.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98822 for commit 003d40b2eed0.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

@yewentao256 CI passes. Can we merge?

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #99215 for commit d6670f8171a9.

@GodRishUniverse

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

✅ Queued 1 failed job(s) for retry in Buildkite CI #99215.

@dllehr-amd
dllehr-amd merged commit b18775f into vllm-project:main Nov 2, 2026
238 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Nov 2, 2026
AndreasKaratzas added a commit to AndreasKaratzas/vllm that referenced this pull request Nov 8, 2026
…code step

The mono decode runner (its symmetric peer memory included) was built
lazily by the first decode step that took the mono path. When that was a
real step, after warmup, under VLLM_GPU_SYNC_CHECK=error the blocking H2D
copy of the peer addresses in PeerBuffer.__init__
(torch.tensor(addresses, device=...)) raised "GPU<->CPU sync detected".
The same build also allocates the IPC heap, opens the peers' handles and
runs gloo collectives (handle exchange, tp_agree) in the step.

Whether a warmup step happened to build it first depends on the tree and
the configuration. The AMD CI DSv4-Pro Mono Decode Numerics job ran a tree
from before vllm-project#62560: an eager MRV2 run with enable_jit_warmup=False and no
extensible KV cache then ran no warmup forward with attention metadata,
so the first real decode step built the runner. Since vllm-project#62560 the MRV2
sampler warmup's dummy batch (max_num_seqs one-token requests) builds it
when max_num_seqs is within the mono kernels' row limit (48 for DSv4.1,
64 for DSv4-Pro); with more, the first real decode step still does.
The DSv4.1 mono CI jobs failed earlier, at the replay builders, so their
mono-runner sync was reproduced only locally.

The runner depends only on the TP group and the device, so build it
after loading, where every TP rank processes the same layers in order:
- DSv4.1: DeepseekV4DecoderLayer.process_weights_after_loading builds
  the target runner, or the DSpark draft runner for draft layers.
- DSv4-Pro: DeepseekV4ForCausalLM.process_weights_after_loading, after
  the preshuffle, builds the runner and takes each layer's weights, whose
  fp32 scale conversion also syncs (torch.equal) on the first step.

The lazy build in _mono_runner stays as the fallback for direct callers.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mrv2 Model Runner V2 specific nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants