Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script by WoosukKwon · Pull Request #27 · vllm-project/vllm

WoosukKwon · 2023-04-05T06:00:21Z

This PR adds query stride to the multi_query_cached_kv_attention kernel so that it can support non-contiguous query tensors in our OPT and LLaMA models (after #20).

This PR also adds a benchmark script comparing our multi_query_cached_kv_attention and the optimized Flash attention implementation.

suquark

LGTM, thanks!

…hmark script (vllm-project#27) * Add query stride to multi_query_cached_kv_attention * Add kernel benchmark script

Add bitsandbytes to requirements and use fixed vllm version in the client

This fixes a miss where I had seen usages of `.labels` `**`a dictionary into kwargs, and I accidentally passed a raw dictionary as a value instead of using keyword arguments 🤦. This caused metrics to show eg. `method="{'method':'prefill'}"` instead of `method=prefill` Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

* Bucketing/Warmup WIP * Cleanup * Revert "Fix model_output_idx on HPU (vllm-project#27)" This reverts commit 90dfa92. * Rework selected_token_indices fix to also work with block_size padding * Simple prompt attention POC * Remove cumsum * MQA/GQA support for simple prompt_attention * Cleanup * Fix typo * Restore profiling runs

…t#27) SUMMARY: * initial set of "actions with a little a" that are the building blocks for eventual CI system * "build test" workflow * "remote push" workflow on `a10g` * update some requirement files to have packages listed in alphabetical order NOTE: this PR is still somewhat nebulas as i'm still working through building and testing "neuralmagic-vllm" in our automation environment. TEST: currently, i'm working through various workflow components, i.e. "actions with a little a". the bits making up the actions in this PR have been constructed from my notes along the way. we can do a "complete" run that includes: linting, building, installing, and running tests. GHA link ... https://github.com/neuralmagic/neuralmagic-vllm/actions/runs/7975058564 `testmo` ... https://neuralmagic.testmo.net/automation/runs/view/8097 Latest GHA link ... https://github.com/neuralmagic/neuralmagic-vllm/actions/runs/7992489982 --------- Co-authored-by: andy-neuma <andy@neuralmagic.com>

* [Kernel] Enable custome AR on ROCm * Install amdsmi in Docker in preparation for custom all reduce (cherry picked from commit f6cfb9bf31e9feeefbdedecf2165f80dd0564b75) * Fix for yapf * Linting and small fixes to vLLM syntax (cherry picked from commit 2cf8103bfb0afce59b28a06c5bbe905983c42728) --------- Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>

Signed-off-by: mayuyuace <qiming1.zhang@intel.com>

New Industry Use Cases (vllm-project#21-30): - vllm-project#21 Game Development: AI game testing + balance tuning - vllm-project#22 Construction: Vision AI safety inspection - vllm-project#23 Agriculture/Smart Farm: Crop monitoring + pest detection - vllm-project#24 Government/Public: Document automation + citizen services - vllm-project#25 Energy/Utilities: Grid monitoring + anomaly detection - vllm-project#26 Environment/Sustainability: Carbon tracking + ESG reporting - vllm-project#27 Fashion/Apparel: Trend analysis + inventory optimization - vllm-project#28 Sports/Fitness: Performance analytics + tactical analysis - vllm-project#29 Automotive/Mobility: Autonomous driving simulation - vllm-project#30 Space/Aerospace: Satellite image analysis Advanced Architecture Patterns: 1. Event-Driven Pattern: Webhook → Event Bus → Agent triggers 2. Streaming Pattern: Large dataset processing with chunking 3. Batch Processing Pattern: Celery-based parallel processing 4. Circuit Breaker Pattern: Fault tolerance + auto recovery 5. CQRS + Event Sourcing: Command/Query separation 6. Saga Pattern: Distributed transaction management Guide now covers: - 30+ industry-specific MCP implementations - 6 production-ready architecture patterns - Real-world scalability solutions - Enterprise integration strategies - Total: 8,672 lines (from 7,249)

Adds KVCacheManager.drain_request_blocks (single commit 00f9bfe) required by tqkv cold-tier (PR #27 in arbi-dev/tqkv) for the γ_merge attention path. Without this hook, TQKV_COLD_ENABLE_DRAIN=1 has no effect — pages are tracked as evicted but never returned to the block pool. No file overlap with the SM_89 fp8 GEMM stack on main; clean merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…aming-quant + H4 + TP-affinity) Big day on RTX: - Diagnose path validated H1 (TP-unaware MoE kernel) was wrong - Real bug stack: stub→filter→streaming-quant accounting → H4 grid-y limit → fp8/bf16 lm_head TP-affinity - 122B XFP TP=2 reaches API and serves, but output is gibberish: attention layer XFP packing produces avg cos=NaN, propagates to logits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… sync The decode sparse-MLA gather previously built slot->local index maps with a Python dict (.item() per slot) and assembled per-token indices with nested Python loops (~150 CPU-GPU syncs per layer per decode step). Rewritten with GPU ops (searchsorted-style match + argmax, vectorized scale broadcast): - case vllm-project#27 verified identical (H OK), smoke consistent - single-stream: 1.44 -> 2.12 tok/s (+47%) - concurrent-4 aggregate: 2.8 -> 5.1 tok/s (+82%) - vs original PyTorch-dequant baseline: 1.11 -> 2.12 tok/s (+91%) Also: pre-allocate QDQ eps constant (CUDA-graph capture compatibility) and add forward-level VLLM_PROFILE timing. Co-Authored-By: Claude <noreply@anthropic.com>

vllm-project#27 finished 10 passed / 1 failed and filled the duration store to ten steps - median total 522 minutes, so a full run is 8.7 hours with gpu_slots: 1 and the three kernel steps are 434 of those minutes. Re-proving them costs most of a working day, so this round narrows to what is still unknown. Twelve steps have never reached a verdict. Nine emit as auto after two temporary relaxations: auto_max_tier T2 -> T3 (eight of them spawn servers or sweep many models) and auto_max_minutes 20 -> 40 (Distributed DP Tests declares 35). The remaining three are TIER_UNKNOWN - V1 e2e and Weight Loading are marked optional upstream, Distributed Model Tests has targets the generator cannot resolve statically - and stay as block steps, which is the intended behaviour for costs we cannot estimate. pytorch-fullgraph-cudagraph-l4-compatibility is in the list because its vllm-project#27 failure was 80 errors of LocalEntryNotFoundError for one repo, Qwen/Qwen2-1.5B-Instruct. Staged since (2.9G with weights - that step runs real inference). The two Rust steps join the denylist instead. They only call run-rust-frontend-cargo-ci.sh, and the test image has no toolchain: `command -v cargo` and `rustc` both come back empty. Installing one into the shared image has the same problem as plugin-tests-2-gpus - it mutates state every concurrently queued step shares. fail_fast off for this round, per the note above it: a first failure would cancel the rest and hide whether the others work, which is the whole point. Co-authored-by: Claude Opus 5

sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. That is why it defaults ON only for compute capability 8.0 exactly (the card class that ships 40-64 GiB, where 1.2 GiB is cheap and load time is 9x), and OFF everywhere else. Override with VLLM_MARLIN_REPACK_STAGED=0/1. ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>

…umbing ci: keep build plumbing out of the change list

sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. That is why it defaults ON only for compute capability 8.0 exactly (the card class that ships 40-64 GiB, where 1.2 GiB is cheap and load time is 9x), and OFF everywhere else. Override with VLLM_MARLIN_REPACK_STAGED=0/1. ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>

sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>

Merge vllm-data-comm/vllm main at 928b990 while retaining the local gate fix b40d5cf as the first parent. Preserve all existing commits, original SHAs and ancestry without rebasing, reordering or squashing either history. Bring in PR vllm-project#54: remove the long-request exclusive prefill batch rule, its model-specific threshold argument and the associated upstream test and documentation updates. Keep the incoming resume-first, FCFS picker within its token budget and native request-slot limit. The scheduler auto-merges without textual conflicts. Retain the local explicit num_running_reqs argument to is_gated and compute free_seq from len(self.running). Waiting requests continue to affect routing scores but cannot reserve native slots before admission. Preserve shared edge slots across DP routes, the in-flight resume exemption, per-DP pending caps and the current Host Staging token budget. Relative to the incoming main, only the existing four-file gate-fix patch remains. Pair with vllm-ascend old-main-1026 fast-forwarded to 14602594f485da18702e57d8c10dade8c700b734. That upstream history includes our graph-placeholder storage fix through PR vllm-project#24, the cloud DSV4 Hash routing repair in PR vllm-project#26 and the dead FlashComm v1 sharding removal in PR vllm-project#27. Keep the runner-owned placeholder and both slice-based uses. Validation: source/diff review only. No tests, lint, pre-commit, CI, syntax checks, startup or benchmarks were run, as requested. NPU validation is pending. Backup branches were created before either update. Signed-off-by: zzyccccccccc <zzyccccccccc@noreply.gitcode.com>

sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>

…, DP and DCP (vllm-project#27) * [KV Cache] Keep a layer-compact layout for hybrid models with a replicated draft A DCP target next to a dense DSpark/DFlash draft that runs at DCP=1 (replicated KV) with a different page size resolved to a block-outer layout, packing the groups without equalizing pages. On hybrid models (attention + Mamba state) that layout does not work: the Mamba page inflates the manager block (e.g. 1280 tokens for Kimi-K3.1 at TP8), which packed MLA caches cannot split into their backends' kernel blocks (and Kimi-K3.1's merged DSA views need a uniform unit stride), and NIXL cannot transfer packed rows that hold Mamba state. For hybrid models, keep the layer-compact layout and let page unification pad the draft's pages to the target's, as it already does for Mamba pages. Non-hybrid models still pack blocks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> * [Spec Decode][Kimi-K3.1] Default GQA drafts to FLASH_ATTN on their own KV cache The Kimi config verifier forced FLASHINFER_MLA on an fp8 cache for every EAGLE3/DSpark draft of a KDA-hybrid DSA target, so a GQA draft such as a Qwen3 sliding-window DSpark (RedHatAI/Kimi-K3-speculator.dspark) was rejected. Unset, such a draft would also inherit the target's fp8_ds_mla cache dtype, which only the sparse MLA kernels read. Keep the MLA defaults for MLA drafts (configs with kv_lora_rank) and default a GQA draft to FLASH_ATTN on the model dtype ("auto"). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> * [Spec Decode][Kimi-K3] Group a GQA draft on its own and fit its block to the target's page A GQA draft next to Kimi-K3 (KDA + MLA) had two KV planning problems: - All groups share one page size and MLA pages cannot be padded, so the draft's page must fit under the target's. With all KV heads on one rank (attention TP1 under DP), a 16x64 bf16 draft needs 4096 B/token against Kimi-K3.1's ~789 B/token merged DSA page, and planning failed at startup. - The default grouping sizes every group by the smallest layer set, so a 5-layer draft split the target's 4 groups into 20. Kimi-K3 now owns its grouping (SupportsKVCacheGrouping, in gqa_draft.py next to dense_draft.py's MLA-draft support): non-MLA attention layers are the draft's; the draft's block shrinks to the largest 64-token multiple dividing the target's block whose page fits (1280 tokens at TP1), page unification pads it, and the target keeps its 4 groups next to one draft group. Without such a draft, or on a block-outer layout, the default grouping is used unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> * [Kimi-K3.1] Shorten the draft-defaults docstring; name the grouping protocol configure_dense_draft's docstring states the rule in three lines, and GQADraftKVCacheGrouping subclasses SupportsKVCacheGrouping explicitly, so the hook vLLM's KV planner calls is visible at the class. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> * [Kimi-K3.1] Pick the draft defaults side by side in configure_dense_draft One decision on kv_lora_rank sets both defaults: FLASHINFER_MLA on fp8 for an MLA draft (its rows live in the target's merged pages, where bf16 rows would not fit) and FLASH_ATTN on the model dtype for a GQA draft. The fp8 check still applies to MLA drafts only. No behavior change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> * [KV Cache] Spell out the linear-attention exception in resolve_kv_cache_layout Say which targets the DCP block-outer rule leaves out (linear-attention layers: Mamba, KDA, GDN), what they get instead (a layer-compact layout with the draft's pages padded), and why. Comment only. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: zixi-qi <zixi@inferact.ai> --------- Signed-off-by: zixi-qi <zixi@inferact.ai> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

WoosukKwon added 2 commits April 5, 2023 05:56

Add query stride to multi_query_cached_kv_attention

94378b7

Add kernel benchmark script

69ba6f0

WoosukKwon requested a review from suquark April 5, 2023 06:00

suquark approved these changes Apr 8, 2023

View reviewed changes

suquark merged commit c267b1a into main Apr 8, 2023

suquark deleted the kernel-fix branch April 8, 2023 20:36

shanshanpt mentioned this pull request Nov 17, 2023

Run long conetxt error : CUDA error: an illegal memory access was encountered #1700

Closed

junior-zsy mentioned this pull request Nov 20, 2023

Error with 32k Long Text in chatglm2-6b-32k Model #1725

Closed

hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024

Add query stride to multi_query_cached_kv_attention & Add kernel benc…

e80d156

…hmark script (vllm-project#27) * Add query stride to multi_query_cached_kv_attention * Add kernel benchmark script

slyalin pushed a commit to slyalin/vllm that referenced this pull request Apr 15, 2024

Merge pull request vllm-project#27 from mzegla/missing_req

948137a

Add bitsandbytes to requirements and use fixed vllm version in the client

tianyil1 pushed a commit to tianyil1/vllm that referenced this pull request Jun 5, 2024

Fix model_output_idx on HPU (vllm-project#27)

90dfa92

ZHJ19970917 mentioned this pull request Jul 14, 2024

[Bug]: When using qwen-32b-chat-awq with multi-threaded access, errors occur after approximately several hundred visits.”vllm.engine.async_llm_engine.AsyncEngineDeadError: Background loop has errored already.“ #6421

Closed

alixiaodi mentioned this pull request Aug 2, 2024

[Bug]: #7072

Closed

hao-cold mentioned this pull request May 13, 2025

[Bug]: CUDA error: an illegal instruction was encountered #18045

Closed

1 task done

markmc mentioned this pull request May 21, 2025

[Bug][Failing Test]: Distributed Comm Ops - distributed/test_shm_broadcast.py #18492

Closed

1 task done

zerosurplus mentioned this pull request Jun 16, 2025

[Bug]: torch.distributed.DistNetworkError: The client socket has timed out after 600000ms while trying to connect to (172.17.0.9, 46229). #19670

Closed

1 task done

xiaomofang mentioned this pull request Jul 31, 2025

[Bug]: There is an issue with speculative inference in Eagle mode, where the context length of vLLM inference is constrained by the draft model. #21986

Closed

1 task done

zyongye pushed a commit to zyongye/vllm that referenced this pull request Aug 5, 2025

fix pre-commit lint for rc1 (vllm-project#27)

076cfce

zyongye pushed a commit to zyongye/vllm that referenced this pull request Aug 6, 2025

fix pre-commit lint for rc1 (vllm-project#27)

6e3c145

Michel-debug mentioned this pull request Oct 23, 2025

[Bug]: qwen3-vl-2b after ms-swift fine-tuning lance errors #27405

Closed

1 task done

inkcherry pushed a commit to inkcherry/vllm that referenced this pull request Nov 6, 2025

feat: Add PTPC support in DE+EP. (vllm-project#27)

dc937c9

yma11 pushed a commit to yma11/vllm that referenced this pull request Nov 14, 2025

fix renormalize (vllm-project#27)

508253c

Signed-off-by: mayuyuace <qiming1.zhang@intel.com>

HervorTao mentioned this pull request Feb 3, 2026

[Bug]: [CPU Backend] AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' #33675

Closed

1 task done

SKPsanjeevi mentioned this pull request Mar 10, 2026

[ROCm] Utilize persistent MLA kernel from AITER #36574

Merged

3 of 5 tasks

LironKesem mentioned this pull request Mar 12, 2026

[Bug] DGX Spark (sm_121): CUTLASS can_implement() rejects sm_120f binaries #36835

Closed

1 task done

Copilot AI mentioned this pull request Mar 20, 2026

Fix XPU segfault when tensor_parallel_size exceeds available devices hongbolv/vllm#5

Closed

pawelniegowski-a2 mentioned this pull request Jul 31, 2026

Automatically configure KV cache size #6

Merged

grYe99 pushed a commit to grYe99/vllm that referenced this pull request Sep 17, 2026

Merge pull request vllm-project#27 from DaoCloud/ci/changelog-skip-pl…

23f507c

…umbing ci: keep build plumbing out of the change list

Navigation Menu

Repository navigation

Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script - #27

Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script#27
suquark merged 2 commits into
mainfrom
kernel-fix

WoosukKwon commented Apr 5, 2023

Uh oh!

suquark left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

WoosukKwon commented Apr 5, 2023

Uh oh!

suquark left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants