[Bug]: Since #18437 can't serve any Dual Chunked attention model #20484
Description
Your current environment
The output of python collect_env.py
Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 22.04.5 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : 14.0.0-1ubuntu1.1
CMake version : version 4.0.2
Libc version : glibc-2.35
==============================
PyTorch Info
==============================
PyTorch version : 2.9.0.dev20250702+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0] (64-bit runtime)
Python platform : Linux-5.15.0-105-generic-x86_64-with-glibc2.35
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: GRID A100-40C
GPU 1: GRID A100-40C
GPU 2: GRID A100-40C
Nvidia driver version : 570.133.20
cuDNN version : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.5
/usr/local/cuda-12.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.5
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7282 16-Core Processor
CPU family: 23
Model: 49
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 4
Stepping: 0
BogoMIPS: 5599.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero wbnoinvd arat umip rdpid overflow_recov succor
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 1 MiB (32 instances)
L1i cache: 1 MiB (32 instances)
L2 cache: 16 MiB (32 instances)
L3 cache: 64 MiB (4 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled
Vulnerability Spec rstack overflow: Mitigation; SMT disabled
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.27.3
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.2.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] onnxruntime==1.22.0
[pip3] pynvml==12.0.0
[pip3] pytorch-triton==3.3.1+gitc8757738
[pip3] pyzmq==27.0.0
[pip3] sentence-transformers==3.4.1
[pip3] torch==2.9.0.dev20250702+cu128
[pip3] torchaudio==2.8.0.dev20250702+cu128
[pip3] torchvision==0.24.0.dev20250702+cu128
[pip3] transformers==4.53.0
[pip3] triton==3.3.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.1.dev7393+gb1c1fe3.d20250703 (git sha: b1c1fe3, date: 20250703)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PIX PIX 0-31 0-1 N/A
GPU1 PIX X PIX 0-31 0-1 N/A
GPU2 PIX PIX X 0-31 0-1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
LD_LIBRARY_PATH=:/usr/local/cuda-12.8/lib64
OMP_NUM_THREADS=1
CUDA_HOME=/usr/local/cuda-12.8/
CUDA_HOME=/usr/local/cuda-12.8/
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
Since commit 5d6d1ad and PR #18437
Using Dual Chunked Attention with Qwen 1M makes the first prompt request hangs and crashes the vllm serve server.
INFO 07-04 11:32:50 [engine.py:317] Added request chatcmpl-0ff807c9e8744e56b2e69fb3687499a6.
ERROR 07-04 11:33:12 [client.py:307] RuntimeError('Engine process (pid 3607156) died.')
ERROR 07-04 11:33:12 [client.py:307] NoneType: None
INFO: Shutting down
INFO: Waiting for connections to close. (CTRL+C to force quit)
ERROR 07-04 11:33:15 [serving_chat.py:897] Error in chat completion stream generator.
ERROR 07-04 11:33:15 [serving_chat.py:897] Traceback (most recent call last):
ERROR 07-04 11:33:15 [serving_chat.py:897] File "/home/pierre/idextend/vllm_repo/vllm/entrypoints/openai/serving_chat.py", line 481, in chat_completion_stream_generator
ERROR 07-04 11:33:15 [serving_chat.py:897] async for res in result_generator:
ERROR 07-04 11:33:15 [serving_chat.py:897] File "/home/pierre/idextend/vllm_repo/vllm/engine/multiprocessing/client.py", line 671, in _process_request
ERROR 07-04 11:33:15 [serving_chat.py:897] raise request_output
ERROR 07-04 11:33:15 [serving_chat.py:897] vllm.engine.multiprocessing.MQEngineDeadError: Engine loop is not running. Inspect the stacktrace to find the original error: RuntimeError('Engine process (pid 3607156) died.').
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [3607080]
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 CUDA_VISIBLE_DEVICES=1,2 VLLM_ATTENTION_BACKEND=DUAL_CHUNK_FLASH_ATTN vllm serve Qwen/Qwen2.5-7B-Instruct-1M --max-model-len 140000 --max-num-seqs 1 --port 2483 --enforce-eager --gpu-memory-utilization 0.65 --enable-server-load-tracking --enable-prefix-caching
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.