Skip to content

[Bugfix] Flashinfer attention warmup deadlock under PP>1 - #64338

Merged
LucasWilkinson merged 5 commits into
vllm-project:mainfrom
yzong-rh:yzong-rh/flashinfer-warmup-pp
Oct 31, 2026
Merged

LucasWilkinson merged 5 commits into
vllm-project:mainfrom
yzong-rh:yzong-rh/flashinfer-warmup-pp

Conversation

@yzong-rh

@yzong-rh yzong-rh commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Fix a deadlock during Flashinfer attention warmup under PP>1.

Claims

Fix a deadlock that occurs when one PP rank contains only Flashinfer attention groups while another PP rank contains a mix of Flashinfer and non-Flashinfer attention groups.

Most commonly affects models with dflash / dspark drafters on Blackwell, where Flashsinfer is often the default attention backend for the target model.

Validation

Ran on B200:

vllm serve Qwen/Qwen3-4B \
  --pipeline-parallel-size 2 \
  --attention-backend FLASHINFER \
  --enforce-eager \
  --no-enable-flashinfer-autotune \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3-4B-DFlash-b16","num_speculative_tokens":2,"attention_backend":"FLASH_ATTN"}'

On this PR, we get

INFO:     Started server process [771742]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

On main, we get stuck after

EngineCore pid=813035) INFO 10-28 17:46:40 [shm_broadcast.py:878] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=813035) INFO 10-28 17:47:40 [shm_broadcast.py:878] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

--enforce-eager doesn't resolve this because Flashinfer attention warmup is run regardless of enforce eager and compilation mode.

--no-enable-flashinfer-autotune doesn't resolve this because it controls Flashinfer autotuning, not Flashinfer attention warmup.

pytest -v -s \
  tests/model_executor/test_attention_warmup.py \
  tests/model_executor/test_flashinfer_autotune_warmup.py \
  tests/v1/worker/test_mixed_warmup_gate.py

99 passed, 2 warnings

Details

During kernel_warmup, we may call mixed_batch_attention_warmup to run "a mixed prefill+decode batch to warm up both attention paths".

This is currently gated under "all attention groups use the Flashinfer backend" because hybrid attention backends may not support the dummy run.

With spec decode, the drafter may choose a different attention backend than the target model. Since the drafter is present on the last PP rank only, different ranks will disagree on whether they should participate in the Flashinfer warmup, causing deadlock.

The current fix synchronizes across PP ranks and only runs the Flashinfer attention warmup if all ranks agree. In effect, warmup behavior is consistent between PP=1 and PP>1.

Related:


Pull Request Checklist
  • I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

  • AI assistance was used during the creation of this PR.

  • Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

  • Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

  • Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

  • Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

@mergify mergify Bot added nvidia bug Something isn't working labels Oct 28, 2026
@yzong-rh
yzong-rh marked this pull request as ready for review October 28, 2026 18:43

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@LopezCastroRoberto LopezCastroRoberto self-assigned this Oct 29, 2026
@@ -0,0 +1,108 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Tests for the PP-consistent FlashInfer attention warmup gate."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would fold this into tests/model_executor/test_attention_warmup.py

# NOTE: This should be `any` instead of `all` but other hybrid attention
# backends don't support this dummy run. Once we remove
# `build_for_cudagraph_capture`, we can change it to `any`.
and all(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concern: this fixes the hang, but now all ranks skip, so with a non-FlashInfer drafter the FlashInfer attention kernels are never warmed and the first request pays the JIT cost, right?

On V2 I don’t think the all restriction is needed: run_mixed_prefill_decode_warmup runs a real scheduler step through execute_model/sample_tokens (drafter included), with no build_for_cudagraph_capture, which is what the NOTE is guarding against.

Could the gate be any(FlashInfer) when use_v2_model_runner, keeping all for V1 and keeping the PP sync (i.e., _all_pp_ranks_eligible) for both? i.e.,

check = any if worker.use_v2_model_runner else all
local_flashinfer_warmup = (
    not worker.model_runner.is_pooling_model
    and bool(worker.model_runner.attn_groups)
    and check(_is_flashinfer_backend(g.backend)
              for groups in worker.model_runner.attn_groups for g in groups)
)

That should warm FlashInfer in your repro instead of skipping it. Can you rerun the same PP=2 dflash repro and confirm, and add a V2 any case to the test?

@yzong-rh

yzong-rh commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @LopezCastroRoberto, addr your comments and re-validated.

vllm serve Qwen/Qwen3-4B \
  --pipeline-parallel-size 2 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3-4B-DFlash-b16","num_speculative_tokens":2,"attention_backend":"FLASH_ATTN"}'

starts up fine on this branch, so I think we are good

Signed-off-by: Yifan Zong <yzong@redhat.com>
@yzong-rh
yzong-rh force-pushed the yzong-rh/flashinfer-warmup-pp branch from 7be2f04 to 7a6d343 Compare October 30, 2026 14:02
@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Oct 30, 2026
@tlrmchlsmth tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 30, 2026
@tlrmchlsmth

Copy link
Copy Markdown
Member

/ci run

@github-actions

Copy link
Copy Markdown

✅ @yzong-rh, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • Run commands test your branch with every commit currently on its upstream target branch. If it is behind, the bot merges the target branch into it first (pull before pushing again). Append --allow-stale to a run command to test an outdated branch as is, at your own risk.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98720 for commit b2e7c728a9f4.

@LopezCastroRoberto

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98748 for commit 463ac306be9d.

@yzong-rh

yzong-rh commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Looks to be a pre-existing failure: #60873
There's a thread in sig-ci.

@yzong-rh

Copy link
Copy Markdown
Contributor Author

/ci retry

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98818 for commit 8595749ee8d3, running 2 failed step(s) from Buildkite CI #98748.

@LopezCastroRoberto

Copy link
Copy Markdown
Contributor

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #98934 for commit cf232a26d229.

@LucasWilkinson
LucasWilkinson merged commit 4aba9bf into vllm-project:main Oct 31, 2026
222 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants