Repository navigation
[Bugfix] Flashinfer attention warmup deadlock under PP>1 - #64338
Conversation
| @@ -0,0 +1,108 @@ | |||
| # SPDX-License-Identifier: Apache-2.0 | |||
| # SPDX-FileCopyrightText: Copyright contributors to the vLLM project | |||
| """Tests for the PP-consistent FlashInfer attention warmup gate.""" | |||
There was a problem hiding this comment.
I would fold this into tests/model_executor/test_attention_warmup.py
| # NOTE: This should be `any` instead of `all` but other hybrid attention | ||
| # backends don't support this dummy run. Once we remove | ||
| # `build_for_cudagraph_capture`, we can change it to `any`. | ||
| and all( |
There was a problem hiding this comment.
One concern: this fixes the hang, but now all ranks skip, so with a non-FlashInfer drafter the FlashInfer attention kernels are never warmed and the first request pays the JIT cost, right?
On V2 I don’t think the all restriction is needed: run_mixed_prefill_decode_warmup runs a real scheduler step through execute_model/sample_tokens (drafter included), with no build_for_cudagraph_capture, which is what the NOTE is guarding against.
Could the gate be any(FlashInfer) when use_v2_model_runner, keeping all for V1 and keeping the PP sync (i.e., _all_pp_ranks_eligible) for both? i.e.,
check = any if worker.use_v2_model_runner else all
local_flashinfer_warmup = (
not worker.model_runner.is_pooling_model
and bool(worker.model_runner.attn_groups)
and check(_is_flashinfer_backend(g.backend)
for groups in worker.model_runner.attn_groups for g in groups)
)That should warm FlashInfer in your repro instead of skipping it. Can you rerun the same PP=2 dflash repro and confirm, and add a V2 any case to the test?
|
Thanks @LopezCastroRoberto, addr your comments and re-validated. starts up fine on this branch, so I think we are good |
7be2f04 to
7a6d343
Compare
|
/ci run |
|
✅ @yzong-rh, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #98720 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #98748 for commit |
|
Looks to be a pre-existing failure: #60873 |
|
/ci retry |
|
✅ Triggered Buildkite CI #98818 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #98934 for commit |
Overview
Fix a deadlock during Flashinfer attention warmup under PP>1.
Claims
Fix a deadlock that occurs when one PP rank contains only Flashinfer attention groups while another PP rank contains a mix of Flashinfer and non-Flashinfer attention groups.
Most commonly affects models with dflash / dspark drafters on Blackwell, where Flashsinfer is often the default attention backend for the target model.
Validation
Ran on B200:
On this PR, we get
On main, we get stuck after
--enforce-eagerdoesn't resolve this because Flashinfer attention warmup is run regardless of enforce eager and compilation mode.--no-enable-flashinfer-autotunedoesn't resolve this because it controls Flashinfer autotuning, not Flashinfer attention warmup.Details
During
kernel_warmup, we may callmixed_batch_attention_warmupto run "a mixed prefill+decode batch to warm up both attention paths".This is currently gated under "all attention groups use the Flashinfer backend" because hybrid attention backends may not support the dummy run.
With spec decode, the drafter may choose a different attention backend than the target model. Since the drafter is present on the last PP rank only, different ranks will disagree on whether they should participate in the Flashinfer warmup, causing deadlock.
The current fix synchronizes across PP ranks and only runs the Flashinfer attention warmup if all ranks agree. In effect, warmup behavior is consistent between PP=1 and PP>1.
Related:
mixed_batch_attention_warmupbut it's separate from Flashinfer warmup.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.