Repository navigation
[Core][MRV2] Warm up the sampler after CUDA graph capture to stop memory growth under load - #62560
Conversation
Signed-off-by: GodRishUniverse <risagarw@amd.com>
…mment Signed-off-by: GodRishUniverse <risagarw@amd.com>
dllehr-amd
left a comment
There was a problem hiding this comment.
I'm good with this @yewentao256 are you good if we land this after #57926? Any issues with this PR?
|
✅ @GodRishUniverse, CI is now available for this PR.
|
yewentao256
left a comment
There was a problem hiding this comment.
Thanks for the work!
Please do the following:
- full reproduce command line, error log in main and the same command line works in current PR
- e2e accuracy using
lm_eval... - e2e benchmark using
vllm bench serve...
When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here
For example:
pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
| (EngineCore pid=1580) super().__init__(
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
| (EngineCore pid=1580) self.scheduler: SchedulerInterface = Scheduler(
| (EngineCore pid=1580) ^^^^^^^^^^
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
| (EngineCore pid=1580) super().__init__(*args, **kwargs)
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
| (EngineCore pid=1580) self.kv_cache_manager = KVCacheManager(
| (EngineCore pid=1580) ^^^^^^^^^^^^^^^
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
| (EngineCore pid=1580) self.coordinator = get_kv_cache_coordinator(
| (EngineCore pid=1580) ^^^^^^^^^^^^^^^^^^^^^^^^^
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
| (EngineCore pid=1580) return HybridKVCacheCoordinator(
| (EngineCore pid=1580) ^^^^^^^^^^^^^^^^^^^^^^^^^
| (EngineCore pid=1580) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
| (EngineCore pid=1580) assert all(
| (EngineCore pid=1580) ^^^^
| (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
| FAILED
| v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) =================="lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k --trust_remote_code"
# Now
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9212|± |0.0074|
| | |strict-match | 5|exact_match|↑ |0.9212|± |0.0074|
# main
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.9151|± |0.0077|
| | |strict-match | 5|exact_match|↑ |0.9158|± |0.0076|
`lm_eval run --model local-chat-completions --model_args base_url=http://127.0.0.1:9256/v1/chat/completions,model=$MODEL,num_concurrent=1024,seed=256 --tasks aime25 --trust_remote_code --apply_chat_template`
# now
|Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25| 0|none | 0|exact_match|↑ |0.7333|± |0.0821|
# main
|Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25| 0|none | 0|exact_match|↑ |0.7333|± |0.0821|vllm bench serve --model zai-org/GLM-5.2-FP8 --dataset-name random --host 127.0.0.1 --port 9256 --random-input-len 8192 --random-output-len 1 --request-rate inf --num-prompts 128 --num-warmups 16
# Now
============ Serving Benchmark Result ============
Successful requests: 128
Failed requests: 0
Benchmark duration (s): 3.08
Total input tokens: 1048576
Total generated tokens: 128
Request throughput (req/s): 41.50
Output token throughput (tok/s): 41.50
Peak output token throughput (tok/s): 76.00
Peak concurrent requests: 128.00
Total token throughput (tok/s): 340041.81
---------------Time to First Token----------------
Mean TTFT (ms): 1669.15
Median TTFT (ms): 1669.02
P99 TTFT (ms): 2724.38
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 0.00
Median TPOT (ms): 0.00
P99 TPOT (ms): 0.00
---------------Inter-token Latency----------------
Mean ITL (ms): 0.00
Median ITL (ms): 0.00
P99 ITL (ms): 0.00
==================================================
# main
============ Serving Benchmark Result ============
Successful requests: 128
Failed requests: 0
Benchmark duration (s): 3.23
Total input tokens: 1048576
Total generated tokens: 128
Request throughput (req/s): 39.63
Output token throughput (tok/s): 39.63
Peak output token throughput (tok/s): 50.00
Peak concurrent requests: 128.00
Total token throughput (tok/s): 324671.20
---------------Time to First Token----------------
Mean TTFT (ms): 1668.69
Median TTFT (ms): 1649.00
P99 TTFT (ms): 3126.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 0.00
Median TPOT (ms): 0.00
P99 TPOT (ms): 0.00
---------------Inter-token Latency----------------
Mean ITL (ms): 0.00
Median ITL (ms): 0.00
P99 ITL (ms): 0.00
==================================================Signed-off-by: GodRishUniverse <risagarw@amd.com>
|
@yewentao256 here are the snippets and the files attached (please let me know if you need anything else). These are with the updated dummy tensor shape: Stress test leads to OOM on main: Lm-eval: vLLM bench (not the stress test) Stress tests: The serve files for above are: Main LM eval and vllm bench PR LM eval and vllm bench |
|
/ci run |
|
✅ Triggered Buildkite CI #97649 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #97960 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #98325 for commit ℹ️ This PR is 2 commits behind upstream |
|
/ci retry |
|
✅ Queued 2 failed job(s) for retry in Buildkite CI #98325. |
|
/ci run |
|
✅ Triggered Buildkite CI #98719 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #98719. |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #98719. |
|
/ci run |
|
✅ Triggered Buildkite CI #98822 for commit |
|
@yewentao256 CI passes. Can we merge? |
|
/ci run |
|
✅ Triggered Buildkite CI #99215 for commit |
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #99215. |
…code step The mono decode runner (its symmetric peer memory included) was built lazily by the first decode step that took the mono path. When that was a real step, after warmup, under VLLM_GPU_SYNC_CHECK=error the blocking H2D copy of the peer addresses in PeerBuffer.__init__ (torch.tensor(addresses, device=...)) raised "GPU<->CPU sync detected". The same build also allocates the IPC heap, opens the peers' handles and runs gloo collectives (handle exchange, tp_agree) in the step. Whether a warmup step happened to build it first depends on the tree and the configuration. The AMD CI DSv4-Pro Mono Decode Numerics job ran a tree from before vllm-project#62560: an eager MRV2 run with enable_jit_warmup=False and no extensible KV cache then ran no warmup forward with attention metadata, so the first real decode step built the runner. Since vllm-project#62560 the MRV2 sampler warmup's dummy batch (max_num_seqs one-token requests) builds it when max_num_seqs is within the mono kernels' row limit (48 for DSv4.1, 64 for DSv4-Pro); with more, the first real decode step still does. The DSv4.1 mono CI jobs failed earlier, at the replay builders, so their mono-runner sync was reproduced only locally. The runner depends only on the TP group and the device, so build it after loading, where every TP rank processes the same layers in order: - DSv4.1: DeepseekV4DecoderLayer.process_weights_after_loading builds the target runner, or the DSpark draft runner for draft layers. - DSv4-Pro: DeepseekV4ForCausalLM.process_weights_after_loading, after the preshuffle, builds the runner and takes each layer's weights, whose fp32 scale conversion also syncs (torch.equal) on the first step. The lazy build in _mono_runner stays as the fallback for direct callers. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Overview
On Model Runner V2, GPU memory keeps growing after startup while load ramps up. The sampler makes an fp32 copy of the
[num_reqs, vocab]logits on every step.capture_model()ends withempty_cache(), so serving starts with no cached block of that size, and each new largest batch reserves a slightly bigger block while the smaller ones stay cached.V1 already avoids this:
compile_or_warm_up_modelruns the sampler once at the largest batch aftercapture_model()("on purpose to prevent memory buffers from being cleared bytorch.accelerator.empty_cache"). This PR adds the same step for MRV2.Depends on #57926. That PR makes
_dummy_sampler_runtake the fp32 path and counts it inprofile_run. Without it, the dummy run skips the copy and this warm-up keeps nothing.Claims
Validation
Qwen3-4B, TP1,
--gpu-memory-utilization 0.9Gemma-4-31B-it, TP4,
--gpu-memory-utilization 0.95 --kv-cache-memory 3GBSetup:
8a6c0449b3with [Bugfix][Core][MRV2] Profile the sampler with the params the warm-up actually uses #57926 and this PR.Run on
main, #57926 and #57926 + this PR. Requests carry no sampling params, so each model's default top-k/top-p applies.Qwen3-4B, 1 GPU
Gemma-4-31B, 4 GPUs (fixed KV size so all builds get the same KV cache)
Details
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance was used. I have reviewed it.