bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWrapper._new_tokens. by pisceskkk · Pull Request #38895 · vllm-project/vllm

pisceskkk · 2026-04-03T08:35:34Z

Purpose

This PR fixes the compatibility issues with the HND kvcache layout when the FlashInfer backend is used with DCP enabled.

While adapting the nixl connector for DCP, I discovered that the FlashInfer backend would crash during the warmup process after specifying the HND layout. The reason is that the _new_tokens wrapper was initialized with a specific kvcache layout, but the actual key/value tensors received by the wrapper during runtime were being input in NHD format.

Test Plan

VLLM_KV_CACHE_LAYOUT="HND" vllm serve qwen/Qwen2.5-1.5B-Instruct -tp 4 -dcp 2 --attention_backend FLASHINFER

Test Result

main(58a249b):

Crash Log


(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:50 [monitor.py:76] Initial profiling/warmup run took 0.45 s
(Worker_TP3_DCP1 pid=500680) INFO 04-03 16:43:50 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP3_DCP1 pid=500680) INFO 04-03 16:43:50 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(Worker_TP1_DCP1 pid=500678) INFO 04-03 16:43:50 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP2_DCP0 pid=500679) INFO 04-03 16:43:50 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP1_DCP1 pid=500678) INFO 04-03 16:43:50 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(Worker_TP2_DCP0 pid=500679) INFO 04-03 16:43:50 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:50 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:50 [utils.py:73] `VLLM_KV_CACHE_LAYOUT` environment variable detected. Setting KV cache layout to HND.
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:50 [gpu_model_runner.py:5867] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(Worker_TP0_DCP0 pid=500677) WARNING 04-03 16:43:51 [flashinfer.py:342] Trtllm does not support returning LSE and as a result does not support DCP, reverting to FlashInfer
(Worker_TP2_DCP0 pid=500679) INFO 04-03 16:43:52 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 2.28 GiB total
(Worker_TP1_DCP1 pid=500678) INFO 04-03 16:43:52 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 2.28 GiB total
(Worker_TP3_DCP1 pid=500680) INFO 04-03 16:43:52 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 2.28 GiB total
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:52 [gpu_model_runner.py:5946] Estimated CUDA graph memory: 2.28 GiB total
(Worker_TP2_DCP0 pid=500679) INFO 04-03 16:43:52 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9728 to maintain the same effective KV cache size.
(Worker_TP1_DCP1 pid=500678) INFO 04-03 16:43:52 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9728 to maintain the same effective KV cache size.
(Worker_TP3_DCP1 pid=500680) INFO 04-03 16:43:52 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9728 to maintain the same effective KV cache size.
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:52 [gpu_worker.py:436] Available KV cache memory: 26.7 GiB
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:52 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9728 to maintain the same effective KV cache size.
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1311] Multiplying the GPU KV cache size by the cp_world_size 2 (pcp_world_size 1 * dcp_world_size 2).
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1319] GPU KV cache size: 3,999,392 tokens
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1324] Maximum concurrency for 32,768 tokens per request: 122.05x
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1311] Multiplying the GPU KV cache size by the cp_world_size 2 (pcp_world_size 1 * dcp_world_size 2).
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1311] Multiplying the GPU KV cache size by the cp_world_size 2 (pcp_world_size 1 * dcp_world_size 2).
(EngineCore pid=500414) INFO 04-03 16:43:52 [kv_cache_utils.py:1311] Multiplying the GPU KV cache size by the cp_world_size 2 (pcp_world_size 1 * dcp_world_size 2).
(Worker_TP0_DCP0 pid=500677) 2026-04-03 16:43:52,464 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP3_DCP1 pid=500680) 2026-04-03 16:43:52,464 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP1_DCP1 pid=500678) 2026-04-03 16:43:52,464 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP2_DCP0 pid=500679) 2026-04-03 16:43:52,464 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(Worker_TP0_DCP0 pid=500677) 2026-04-03 16:43:52,475 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP0_DCP0 pid=500677) INFO 04-03 16:43:52 [kernel_warmup.py:69] Warming up FlashInfer attention.
(Worker_TP3_DCP1 pid=500680) 2026-04-03 16:43:52,476 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP3_DCP1 pid=500680) INFO 04-03 16:43:52 [kernel_warmup.py:69] Warming up FlashInfer attention.
(Worker_TP1_DCP1 pid=500678) 2026-04-03 16:43:52,478 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP1_DCP1 pid=500678) INFO 04-03 16:43:52 [kernel_warmup.py:69] Warming up FlashInfer attention.
(Worker_TP2_DCP0 pid=500679) 2026-04-03 16:43:52,480 - INFO - autotuner.py:268 - flashinfer.jit: [Autotuner]: Autotuning process ends
(Worker_TP2_DCP0 pid=500679) INFO 04-03 16:43:52 [kernel_warmup.py:69] Warming up FlashInfer attention.
(EngineCore pid=500414) ERROR 04-03 16:43:52 [multiproc_executor.py:273] Worker proc VllmWorker-3 died unexpectedly, shutting down executor.
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108] EngineCore failed to start.
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108] Traceback (most recent call last):
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     super().__init__(
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     return func(*args, **kwargs)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 280, in _initialize_kv_caches
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 118, in initialize_from_config
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     compilation_times: list[float] = self.collective_rpc("compile_or_warm_up_model")
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 397, in collective_rpc
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     return aggregate(get_response())
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]                      ^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 376, in get_response
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     status, result = mq.dequeue(timeout=dequeue_timeout)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     with self.acquire_read(timeout, indefinite) as buf:
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     return next(self.gen)
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]            ^^^^^^^^^^^^^^
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 677, in acquire_read
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108]     raise RuntimeError("cancelled")
(EngineCore pid=500414) ERROR 04-03 16:43:53 [core.py:1108] RuntimeError: cancelled
(EngineCore pid=500414) Process EngineCore:
(EngineCore pid=500414) Traceback (most recent call last):
(EngineCore pid=500414)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=500414)     self.run()
(EngineCore pid=500414)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=500414)     self._target(*self._args, **self._kwargs)
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1112, in run_engine_core
(EngineCore pid=500414)     raise e
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core
(EngineCore pid=500414)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=500414)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=500414)     return func(*args, **kwargs)
(EngineCore pid=500414)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in __init__
(EngineCore pid=500414)     super().__init__(
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 124, in __init__
(EngineCore pid=500414)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=500414)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=500414)     return func(*args, **kwargs)
(EngineCore pid=500414)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 280, in _initialize_kv_caches
(EngineCore pid=500414)     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 118, in initialize_from_config
(EngineCore pid=500414)     compilation_times: list[float] = self.collective_rpc("compile_or_warm_up_model")
(EngineCore pid=500414)                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 397, in collective_rpc
(EngineCore pid=500414)     return aggregate(get_response())
(EngineCore pid=500414)                      ^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 376, in get_response
(EngineCore pid=500414)     status, result = mq.dequeue(timeout=dequeue_timeout)
(EngineCore pid=500414)                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue
(EngineCore pid=500414)     with self.acquire_read(timeout, indefinite) as buf:
(EngineCore pid=500414)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
(EngineCore pid=500414)     return next(self.gen)
(EngineCore pid=500414)            ^^^^^^^^^^^^^^
(EngineCore pid=500414)   File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 677, in acquire_read
(EngineCore pid=500414)     raise RuntimeError("cancelled")
(EngineCore pid=500414) RuntimeError: cancelled
(APIServer pid=500145) Traceback (most recent call last):
(APIServer pid=500145)   File "/usr/local/bin/vllm", line 10, in 
(APIServer pid=500145)     sys.exit(main())
(APIServer pid=500145)              ^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=500145)     args.dispatch_function(args)
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=500145)     uvloop.run(run_server(args))
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=500145)     return __asyncio.run(
(APIServer pid=500145)            ^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=500145)     return runner.run(main)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=500145)     return self._loop.run_until_complete(task)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=500145)     return await main
(APIServer pid=500145)            ^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server
(APIServer pid=500145)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 684, in run_server_worker
(APIServer pid=500145)     async with build_async_engine_client(
(APIServer pid=500145)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=500145)     return await anext(self.gen)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=500145)     async with build_async_engine_client_from_engine_args(
(APIServer pid=500145)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=500145)     return await anext(self.gen)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=500145)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=500145)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=500145)     return cls(
(APIServer pid=500145)            ^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=500145)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=500145)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=500145)     return func(*args, **kwargs)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=500145)     return AsyncMPClient(*client_args)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=500145)     return func(*args, **kwargs)
(APIServer pid=500145)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 887, in __init__
(APIServer pid=500145)     super().__init__(
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=500145)     with launch_core_engines(
(APIServer pid=500145)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=500145)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=500145)     next(self.gen)
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 998, in launch_core_engines
(APIServer pid=500145)     wait_for_engine_startup(
(APIServer pid=500145)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1057, in wait_for_engine_startup
(APIServer pid=500145)     raise RuntimeError(
(APIServer pid=500145) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

This PR:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.509	±	0.0158
		strict-match	5	exact_match	↑	0.297	±	0.0145

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

…pper._new_tokens. Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>

gemini-code-assist

Code Review

This pull request updates the initialization of BatchPrefillWithRaggedKVCacheWrapper in the FlashInfer backend by removing the get_kv_cache_layout() argument. I have no feedback to provide.

Copilot

Pull request overview

Fixes a FlashInfer+DCP compatibility issue when using the HND KV-cache layout by ensuring the _new_tokens ragged prefill wrapper no longer initializes with a potentially mismatched KV layout.

Changes:

Remove get_kv_cache_layout() argument from BatchDCPPrefillWrapper._new_tokens wrapper initialization to avoid crashes when runtime key/value tensors are in NHD format while global KV-cache layout is HND.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

LucasWilkinson

LGTM; thanks for the fix!

…pper._new_tokens. (vllm-project#38895) Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>

Cherry-pick 62 bugfix/security PRs from upstream vllm-project/vllm main (2026-05-03 to 2026-06-17), covering scheduler, engine core, model runner, worker, attention, KV cache, compilation, and structured output fixes. Security (4): vllm-project#43286 vllm-project#44744 vllm-project#45118 vllm-project#45252 Bugfix (56): vllm-project#35536 vllm-project#36616 vllm-project#38895 vllm-project#39155 vllm-project#39324 vllm-project#39562 vllm-project#39805 vllm-project#40398 vllm-project#40726 Runner fix (2): vllm-project#44568 vllm-project#44603 Skipped: vllm-project#43781 (ROCm-specific, not applicable to Ascend NPU) Conflict resolutions: - Manual merge: vllm-project#43286 vllm-project#45118 vllm-project#42112 vllm-project#43160 vllm-project#43719 vllm-project#44560 - Upstream-preferred (-X theirs): vllm-project#43808 vllm-project#43988 vllm-project#42967 vllm-project#35536 vllm-project#45195 - Test files (--theirs): vllm-project#44744 vllm-project#41895 vllm-project#42040 vllm-project#41233 vllm-project#45345 vllm-project#43982 Co-authored-by: GitHub Copilot Signed-off-by: MingqiWang-coder <mingqiwang@hust.edu.cn>

bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWra…

29be89d

…pper._new_tokens. Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>

pisceskkk requested review from mgoin, pavanimajety and vadiklyutiy as code owners April 3, 2026 08:35

Copilot AI review requested due to automatic review settings April 3, 2026 08:35

Copilot started reviewing on behalf of pisceskkk April 3, 2026 08:36

View session

mergify Bot added nvidia v1 bug Something isn't working labels Apr 3, 2026

github-project-automation Bot added this to NVIDIA Apr 3, 2026

gemini-code-assist Bot reviewed Apr 3, 2026

View reviewed changes

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread vllm/v1/attention/backends/flashinfer.py Resolved

Comment thread vllm/v1/attention/backends/flashinfer.py Resolved

pisceskkk mentioned this pull request May 8, 2026

feat(nixl,dcp): Supports MLA DCP for PD disaggregation with nixl connector #38433

Closed

LucasWilkinson approved these changes May 10, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA May 10, 2026

LucasWilkinson enabled auto-merge (squash) May 10, 2026 04:11

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label May 10, 2026

Merge branch 'main' into dcp/hnd

3fa6ee6

LucasWilkinson merged commit b1b5972 into vllm-project:main May 11, 2026
67 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA May 11, 2026

weifang231 pushed a commit to weifang231/eb-vllm that referenced this pull request May 13, 2026

bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWra…

c7568d5

…pper._new_tokens. (vllm-project#38895) Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>

mfylcek pushed a commit to mfylcek/vllm that referenced this pull request May 19, 2026

bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWra…

1297c9a

…pper._new_tokens. (vllm-project#38895) Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>

pisceskkk deleted the dcp/hnd branch June 23, 2026 06:14

MingqiWang-coder mentioned this pull request Jul 1, 2026

[Sync] Upstream V1 engine core — 89 PRs (bugfix, scheduler, runner, worker, hardware) intellistream/vllm-hust-legacy-20260831#82

Closed

Navigation Menu

Repository navigation

bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWrapper._new_tokens. - #38895

bugfix(flashinfer,dcp): remove kv_cache_layout for BatchDCPPrefillWrapper._new_tokens.#38895
LucasWilkinson merged 2 commits into
vllm-project:mainfrom
pisceskkk:dcp/hnd

pisceskkk commented Apr 3, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

LucasWilkinson left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pisceskkk commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

LucasWilkinson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pisceskkk commented Apr 3, 2026 •

edited

Loading