Repository navigation
[Core] Add prefill step cadence for better non-PD DP balancing - #44558
Merged
Merged
Conversation
njhill
changed the title
Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
marked this pull request as ready for review
June 4, 2026 19:45
njhill
requested review from
ApostaC,
ProExpertProg,
WoosukKwon,
alexm-redhat,
heheda12345,
hmellor,
houseroad,
mgoin,
orozery,
robertgshaw2-redhat,
tlrmchlsmth,
yewentao256,
youkaichao and
ywang96
as code owners
June 4, 2026 19:45
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Signed-off-by: Nick Hill <nickhill123@gmail.com>
WoosukKwon
reviewed
Jun 5, 2026
WoosukKwon
left a comment
WoosukKwon
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the work!
Member
|
This definitely helps balance relatively rare prefills. Thank you for the PR. I’m curious whether, at least in theory, there is a better solution for balancing DP. Let’s take PD with a prefill node and many prefill requests, using long inputs such as 32K. With chunked prefill, we split it into 8K + 8K + 8K + 8K. However, due to the quadratic dependency on input length, the last chunk takes much longer than the first one. Frequently, one rank processes the first chunk while another rank processes the fourth chunk. As a result, all DP ranks effectively run at the speed of the slowest fourth chunk. This is not just theoretical reasoning. I saw this with Qwen3.5 when feeding it many 32K prompts: full attention was 3–4x slower on the last chunk, and other ranks often had to wait. Maybe we need some estimation of step duration? |
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Member
Author
|
Thanks @vadiklyutiy this is a good point. Actually a similar imbalance can occur with chunked prefill + pipeline parallel. @WoosukKwon and I have discussed using a weighted max-batch-tokens as a more general solution for these and I am experimenting with that next. The complication is how to model/parameterize the latency estimate as a function of tokens (including positions), which could depend on a bunch of factors including model/gpu/attn-impl. I think that is complementary to this PR though. This one is particularly beneficial given decode-only batches are so much faster due to using full CG with faster FlashMLA kernel. @WoosukKwon FYI I also introduced an additional check to temporarily disable the throttling if the engine is saturated with new requests (i.e. if waiting queue was not drained in the last prefill step). |
Member
Sure, I didn’t mean that this should be included in this PR. I just thought this might be the right place to discuss it. |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
MengqingCao
pushed a commit
to vllm-project/vllm-ascend
that referenced
this pull request
Jul 4, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
slokesha
pushed a commit
to slokesha/vllm-gaudi
that referenced
this pull request
Jul 6, 2026
…HPU scheduler, ngram proposer and offloading connector tests to upstream API drift (vllm-project#1556) ## Bug 1: Forward throttle_prefills in HPUAsyncScheduler.schedule - **State machine id**: hpu_async_scheduler_schedule_positional_arg - **Commit**: 957ba4d ### Root cause vLLM PR #44558 added a throttle_prefills positional arg to Scheduler.schedule(); EngineCore calls it positionally but the HPU override only accepted self. ### Upstream PR vllm-project/vllm#44558 ### Fix Accept throttle_prefills (default False) on the HPUAsyncScheduler.schedule override and forward it to super().schedule(). ## Bug 2: Pass num_speculative_tokens to NgramProposer.propose - **State machine id**: ngram_proposer_propose_missing_positional_arg - **Commit**: 82155ea ### Root cause vLLM PR #32374 (Dynamic SD) added a leading num_speculative_tokens positional arg to NgramProposer.propose(). ### Upstream PR vllm-project/vllm#32374 ### Fix Prepend self.speculative_config.num_speculative_tokens in propose_ngram_draft_token_ids to match the new upstream signature. ## Bug 3: Align OffloadingConnector stats tests with upstream flat-metrics API - **State machine id**: offloading_connector_cpu_to_gpu_metrics_missing - **Commit**: c1eb9e3 ### Root cause vLLM PR #35669 rewrote OffloadingConnectorStats to a self-describing {types, data} flat-metric payload, dropping the per-direction CPU_to_GPU/GPU_to_CPU list shape the tests still asserted. ### Upstream PR vllm-project/vllm#35669 ### Fix Rewrite test_metrics.py to exercise increase_counter/observe_histogram/aggregate/reduce/reset against the new self-describing stats contract. ## Bug 4: Align OffloadingConnector scheduler flush assertions with upstream defer-on-finish - **State machine id**: offloading_connector_flush_on_finish_deferred - **Commit**: 575a178 ### Root cause vLLM commit f428718 (PR #45823, "Defer on_request_finished until in-flight transfers drain") changed OffloadingConnectorScheduler: a finishing request with in-flight store jobs no longer flushes those stores immediately — finalization is deferred until transfers drain, and flush now fires only on preemption or block reuse. test_concurrent_lookups_of_the_same_prefix and test_abort_loading_requests still asserted flush-on-finish, so they failed once the target vLLM SHA picked up #45823. ### Upstream PR vllm-project/vllm#45823 ### Fix Drop the stale expected_flushed_gpu_block_indexes assertions in the two affected tests (matching upstream's own equivalents, which assert no flush in these scenarios). test_request_preemption keeps its flush-on-preemption assertion, which upstream still honors. --------- Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com> Signed-off-by: slokesha <slokeshappa@habana.ai>
wangyichao1999
pushed a commit
to wangyichao1999/vllm-ascend
that referenced
this pull request
Jul 9, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
xqchen7
pushed a commit
to nv-action/vllm-benchmarks
that referenced
this pull request
Jul 15, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
…project#44558) Signed-off-by: Nick Hill <nickhill123@gmail.com>
Alex-stack-hub
pushed a commit
to 0moyi0-2024/vllm-ascend_tp
that referenced
this pull request
Jul 27, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
mgehre-amd
pushed a commit
to ROCm/vllm
that referenced
this pull request
Aug 13, 2026
…project#44558) Signed-off-by: Nick Hill <nickhill123@gmail.com>
MmMmaru
pushed a commit
to jiaqi-lee/vllm-ascend
that referenced
this pull request
Aug 19, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
shiqiangA
pushed a commit
to shiqiangA/vllm-ascend
that referenced
this pull request
Aug 20, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
Leetrytry
pushed a commit
to Leetrytry/vllm-ascend
that referenced
this pull request
Sep 11, 2026
### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to `b9a7cd464c9ae9b1b450f8982b76d7be4de73724` and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. `vllm_ascend/__init__.py` - Set `VLLM_USE_V2_MODEL_RUNNER=0` for non-`0.23.0`. - Reason: this branch keeps the main compatibility path on V1 model runner. V2 still has partial compatibility updates, but it is not enabled as the default path here. - Related upstream changes: vLLM Model Runner V2 migration work, including [vllm#42667](vllm-project/vllm#42667) and [vllm#44446](vllm-project/vllm#44446). `tests/e2e/pull_request/one_card/test_guided_decoding.py` - For non-`0.23.0` and `xgrammar` / `outlines`, set `VLLM_REGEX_COMPILATION_TIMEOUT_S=30` during the test. - Keep `0.23.0` behavior unchanged. - Reason: upstream added a structured-output regex compilation timeout. The existing complex JSON schema can exceed the default 5s timeout during grammar compilation. - Upstream source: [vllm#45118](vllm-project/vllm#45118). `vllm_ascend/core/scheduler_profiling_chunk.py` - Add `throttle_prefills: bool = False` to `ProfilingChunkScheduler.schedule`. - Reason: upstream scheduler `schedule()` now accepts `throttle_prefills`. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_balance_schedule.py` - Add `throttle_prefills` to `BalanceScheduler.schedule`. - For `0.23.0`, continue calling `super().schedule()` with no argument. - For non-`0.23.0`, forward `throttle_prefills` to upstream scheduler. - Reason: keep the balance scheduler wrapper compatible with both old and new upstream scheduler signatures. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/patch/platform/patch_profiling_chunk.py` - Change the schedule wrapper to accept `*args, **kwargs`. - Reason: profiling chunk patch wraps scheduler implementations whose signatures now differ between `0.23.0` and upstream main. - Upstream source: [vllm#44558](vllm-project/vllm#44558). `vllm_ascend/spec_decode/ngram_proposer.py` - Split `AscendNgramProposer.propose` by `vllm_version_is("0.23.0")`. - `0.23.0` keeps the old Ascend behavior using `runner.input_batch`. - Non-`0.23.0` follows upstream Dynamic SD signature: `propose(num_speculative_tokens, sampled_token_ids, num_tokens_no_spec, token_ids_cpu, ...)`. - Pass `num_speculative_tokens` into `batch_propose`. - Reason: upstream Dynamic Speculative Decoding changed ngram proposer signatures and made the per-step draft-token budget explicit. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/spec_decode/suffix_proposer.py` - Add version-aware wrapper for suffix decoding proposer. - `0.23.0` keeps calling upstream as: `super().propose(input_batch, sampled_token_ids)`. - Non-`0.23.0` calls upstream as: `super().propose(num_speculative_tokens, input_batch, sampled_token_ids, ...)`. - Reason: upstream Dynamic SD added `num_speculative_tokens` to suffix decoding proposer. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/model_runner_v1.py` - For `AscendNgramProposer`, pass `scheduler_output.num_spec_tokens_to_schedule`, `self.input_batch.num_tokens_no_spec`, and `self.input_batch.token_ids_cpu` on non-`0.23.0`. - For `AscendSuffixDecodingProposer`, pass `num_speculative_tokens=scheduler_output.num_spec_tokens_to_schedule` on non-`0.23.0`. - For Eagle-style proposers, pass `self.speculative_config.num_speculative_tokens` on non-`0.23.0`. - Reason: upstream Dynamic SD changed proposer call signatures and scheduler output now carries the dynamic speculative token budget. - Upstream source: [vllm#32374](vllm-project/vllm#32374). `vllm_ascend/worker/encoder_acl_graph.py` - For `0.23.0`, continue using `self.budget_graphs[token_budget]`. - For non-`0.23.0`, use upstream graph-set storage through `self._get_graph_set("default")`. - Reason: upstream encoder CUDA graph manager changed from a flat budget graph map to graph sets for multiple capture paths. - Upstream source: [vllm#43586](vllm-project/vllm#43586). `vllm_ascend/ops/fused_moe/fused_moe.py` - Keep `0.23.0` routed-experts capture config lookup from `layer.vllm_config.model_config`. - For non-`0.23.0`, use `get_current_vllm_config()` to obtain `model_config`. - Reason: upstream routed experts capture is now driven by current vLLM config context instead of relying only on the layer-local config path. - Upstream source: [vllm#28284](vllm-project/vllm#28284). `vllm_ascend/worker/v2/aclgraph_utils.py` - Add `lora_capture_cases` to `ModelAclGraphManager.__init__` and forward it upstream. - Add `lora_capture_hook` to `capture_model` and forward it by keyword. - Reason: upstream MRV2 LoRA warmup/capture now passes LoRA capture cases and hook through CUDA graph manager APIs. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/model_runner.py` - Update the graph manager factory to accept `lora_capture_cases`. - Forward `lora_capture_cases` into `ModelAclGraphManager`. - Reason: without this factory-level argument, upstream MRV2 LoRA warmup fails before reaching Ascend ACL graph manager. - Upstream source: [vllm#35536](vllm-project/vllm#35536). `vllm_ascend/worker/v2/spec_decode/eagle/aclgraph.py` - Replace the misleading `AttentionStatePair as CapturedAttentionState` alias with direct `AttentionStatePair` usage. - Rename `full_cg_attn_states` to `attn_states`. - Forward `progress_bar_desc` by keyword. - Reason: upstream Eagle graph capture split/renamed captured attention state flow. - Upstream source: [vllm#44253](vllm-project/vllm#44253). `vllm_ascend/patch/platform/__init__.py` - Only import GLM4.7 / MiniMax M2 parser patches and MiniMax usage accounting patch on `0.23.0`. - Reason: upstream main has refactored these parser internals, so the old Ascend patches are only valid for the `0.23.0` parser implementation. - Upstream sources: [vllm#45701](vllm-project/vllm#45701), [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_glm47_tool_call_parser.py` - Skip the module on non-`0.23.0` before importing incompatible upstream parser internals. - Reason: upstream GLM parser internals were migrated/refactored and `_extract_tool_call_regions` is no longer available in the old form. - Upstream source: [vllm#45915](vllm-project/vllm#45915). `tests/ut/patch/platform/test_patch_minimax_m2_tool_call_parser.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 tool parser moved to the new streaming parser engine and old attributes such as `tool_call_start_token` are no longer available. - Upstream source: [vllm#45701](vllm-project/vllm#45701). `tests/ut/patch/platform/test_patch_minimax_usage_accounting.py` - Skip on non-`0.23.0`. - Reason: upstream MiniMax M2 reasoning/parser internals changed and old usage accounting patch assumptions no longer hold on main. - Upstream source: [vllm#45701](vllm-project/vllm#45701). - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@a30addc --------- Signed-off-by: MrZ20 <2609716663@qq.com> Signed-off-by: Mrz20 <2609716663@qq.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 13, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 13, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: lifulu <fululi12@amd.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 13, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: lifulu <fululi12@amd.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 13, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: lifulu <fululi12@amd.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 14, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: lifulu <fululi12@amd.com>
fululi12
pushed a commit
to fululi12/vllm
that referenced
this pull request
Oct 14, 2026
…escape Review feedback from @okorzh-amd on vllm-project#54627. Each item below was reproduced before changing anything. DROP THE LIVENESS BOUND. `empty_schedules_since_executed` was added for a reported stall that never reproduced locally, and it cannot be the fix: at PP=1, where the stall was reported, there are no consecutive empty schedule() calls in steady state, so the clause never fires. Under PP it fires constantly (measured by the reviewer at 199/200 steps for PP=3, I=2), collapsing the cadence it is supposed to protect. Removed along with its test rather than patched. RESTORE THE CAPACITY ESCAPE. The DP gate is `(throttle_prefills and not self.prefill_capacity_bound) and ...`; this gate never read prefill_capacity_bound. Without it a backed-up engine keeps throttling once a release step cannot drain the queue, capping prefill at about one budget per interval and growing the waiting queue without bound -- the TTFT avalanche vllm-project#44558 added the guard for. Added `and not self.prefill_capacity_bound`. ALIGN THE CADENCE STAMP WITH THE GATE. The gate defers on `num_computed_tokens < num_tokens - 1`; the stamp fired on `< num_prompt_tokens`. A request with exactly one token left therefore passed through a held step and still restarted the interval -- which is every full prefix-cache hit, and every remote-KV resume, since _update_waiting_for_remote_kv sets num_computed_tokens = num_tokens - 1. Both now use the gate's predicate. DOCUMENT THE THREE DP REGIMES AND LOG THE CADENCE. `reconfigure_for_independent_dp_rank()` sets data_parallel_size=1 for every dense DP rank before the scheduler is built, so the local cadence now applies there where the option used to be a no-op. That is intended but was undocumented and invisible; the three regimes are spelled out and a logger.info_once reports which clock is active. Help text 177 -> 54 words, and the init comments trimmed. Net -13 lines in vllm/ while fixing three correctness issues. Full scheduler suite: 211 passed, including the pre-existing test_throttle_capacity_bound_guard_admits. NOT changing `break` -> `skip_request()` in the waiting loop, although I agreed to it in review. The prefix-cache lookup (get_num_new_matched_tokens, get_computed_blocks) runs BEFORE the gate, so `break` costs one lookup per held step while skip-and-continue would cost one per waiting request per held step -- multiplying the connector accounting problem (increase_num_stored_blocks on every hit) rather than fixing it. The right fix is to move the gate above the lookup, which restructures the waiting loop and belongs in its own PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: lifulu <fululi12@amd.com>
tu10ng
added a commit
to vllm-data-comm/vllm
that referenced
this pull request
Oct 21, 2026
Upstream added a throttle_prefills parameter to SchedulerInterface.schedule() (vllm-project#44558, DP prefill step cadence), and the engine core now calls scheduler.schedule(self._should_throttle_prefills()). LwdCloudScheduler and LwdEdgeScheduler still overrode schedule() without it, so the first cloud step raised TypeError, and the edge would on its first step too. Take the parameter in both overrides. Only DPEngineCoreProc throttles prefills, and LWD engines are plain EngineCoreProc subclasses whose _should_throttle_prefills() is always False; LWD also gates its prefills itself. A shared base-scheduler check therefore rejects True as a wiring error instead of silently ignoring it. Signed-off-by: tu10ng <2059734099@qq.com>
8 tasks
6 tasks done
jbelofamd
added a commit
to jbelofamd/vllm
that referenced
this pull request
Nov 4, 2026
Address review on the DP-global prefill gate: - Fail open for a backlogged rank. Under the gate a capacity-bound rank defers like the others, so with skewed load it could get one token budget per interval indefinitely. The scheduler now counts releases in a row that leave its queue backed up; after 8 (at most 8 intervals, since every cadence step is a release) the per-rank capacity escape from vllm-project#44558 applies again until a release drains the queue. - Clamp prefill_release_min_dp_ranks to the current DP group size, which an elastic EP scale-down shrinks, so the gate cannot stop releasing. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Jon Belof <jonathan.belof@amd.com>
jbelofamd
added a commit
to jbelofamd/vllm
that referenced
this pull request
Nov 5, 2026
Address review on the DP-global prefill gate: - Fail open for a backlogged rank. Under the gate a capacity-bound rank defers like the others, so with skewed load it could get one token budget per interval indefinitely. The scheduler now counts releases in a row that leave its queue backed up; after 8 (at most 8 intervals, since every cadence step is a release) the per-rank capacity escape from vllm-project#44558 applies again until a release drains the queue. - Clamp prefill_release_min_dp_ranks to the current DP group size, which an elastic EP scale-down shrinks, so the gate cannot stop releasing. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Jon Belof <jonathan.belof@amd.com>
jbelofamd
added a commit
to jbelofamd/vllm
that referenced
this pull request
Nov 9, 2026
Address review on the DP-global prefill gate: - Fail open for a backlogged rank. Under the gate a capacity-bound rank defers like the others, so with skewed load it could get one token budget per interval indefinitely. The scheduler now counts releases in a row that leave its queue backed up; after 8 (at most 8 intervals, since every cadence step is a release) the per-rank capacity escape from vllm-project#44558 applies again until a release drains the queue. - Clamp prefill_release_min_dp_ranks to the current DP group size, which an elastic EP scale-down shrinks, so the gate cannot stop releasing. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Jon Belof <jonathan.belof@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found that interval=2 was optimal, but that a guard is needed to disable the throttling when saturated, to avoid overall perf degradation in that case.
Setup: DeepSeek-R1-0528 FP4 (671B, MLA, FP8 KV), DP=8 / TP=1 /
--enable-expert-parallel, 2× GB200 (4 GPU each), internal LB, cudagraph mode.vllm-bench(Rust) random dataset, Poisson (burstiness=1.0),--ignore-eos. interval=1 = baseline (feature off).The feature now includes a parameter-free capacity-bound saturation guard: when an admitting step can't drain the waiting prefill queue, throttling is suppressed so a saturated rank never avalanches.
1. Headline benefit — decode-bound workload, below saturation
Workload A = 1k input / 200 output (single-chunk prefill), rate 24 req/s (sub-saturation). The throttle removes prefill interference from most decode steps, so median decode ITL collapses toward the cudagraph decode floor (~32 ms).
Decode latency drops sharply, throughput stays pinned to offered load, and TTFT improves — the win is free in this regime.
2. Interval sweep (1 / 2 / 3) — interval=2 is the sweet spot
Same workload A, rate 24, sweeping the interval. TPOT keeps falling as the interval grows (decodes get cleaner steps), but TTFT degrades past interval=2 because prefill admission is throttled too hard — classic U-shape on the tail.
interval=2 gives the best balance: near-floor ITL and the best TTFT. interval=3 squeezes a little more off TPOT/ITL but the prefill backlog pushes TTFT tail back up.
3. Rate ramp — interval=2 wins across the load curve
Workload A, single server, ramping request rate. Mean ITL is consistently 30–44% lower with interval=2, and the TTFT tail is much better through the mid-range.
interval=2 dominates up to ~rate 40; at the top (rate 56, capacity-limited) its TTFT starts to regress — the expected edge of the sustainable band.
4. Saturation safety — the guard holds
Workload A at rate 40 (at/near saturation). No avalanche: the capacity-bound guard keeps TTFT ≈ baseline while still delivering the median-ITL win.
5. Known trade-off — very prefill-heavy workloads
Workload B = 16k input / 200 output (prefill:decode ≈ 80:1). Throttling halves prefill admission capacity, so at rate 4–8 the system is already at/over prefill capacity → interval=2 gives no ITL win and regresses TTFT.
The win lives in the decode-bound, sub-saturation regime.