Repository navigation
Implement prefix sharing - #37
Closed
WoosukKwon wants to merge 4 commits into
Closed
WoosukKwon wants to merge 4 commits into
WoosukKwon wants to merge 4 commits into
Conversation
tianyil1
pushed a commit
to tianyil1/vllm
that referenced
this pull request
Jun 5, 2024
joerunde
added a commit
to joerunde/vllm
that referenced
this pull request
Jun 17, 2024
These are some changes we've been adding onto the last couple releases. Would be nice to have on main for the next round, and hopefully this is all moot once we hop over to the ODH fork anyway --------- Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by: Joe Runde <Joseph.Runde@ibm.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com>
yukavio
pushed a commit
to yukavio/vllm
that referenced
this pull request
Jul 3, 2024
Tested by making sure magic_wand was uninstalled and this code for a
dense model runs fine:
```python
from vllm import LLM, SamplingParams
model = LLM("nm-testing/opt-125m-pruned2.4", enforce_eager=True)
```
Then testing with a sparse model run:
```python
from vllm import LLM, SamplingParams
model = LLM("nm-testing/opt-125m-pruned2.4", sparsity="sparse_w16a16", enforce_eager=True)
```
output:
```
...
File "/home/michael/code/neuralmagic-vllm/vllm/model_executor/weight_utils.py", line 93, in get_sparse_config
from vllm.model_executor.layers.sparsity import get_sparsity_config
File "/home/michael/code/neuralmagic-vllm/vllm/model_executor/layers/sparsity/__init__.py", line 6, in <module>
raise ValueError(
ValueError: magic_wand is not available and required for sparsity support. Please install it with `pip install magic_wand`
```
yukavio
pushed a commit
to yukavio/vllm
that referenced
this pull request
Jul 3, 2024
Tested by making sure magic_wand was uninstalled and this code for a
dense model runs fine:
```python
from vllm import LLM, SamplingParams
model = LLM("nm-testing/opt-125m-pruned2.4", enforce_eager=True)
```
Then testing with a sparse model run:
```python
from vllm import LLM, SamplingParams
model = LLM("nm-testing/opt-125m-pruned2.4", sparsity="sparse_w16a16", enforce_eager=True)
```
output:
```
...
File "/home/michael/code/neuralmagic-vllm/vllm/model_executor/weight_utils.py", line 93, in get_sparse_config
from vllm.model_executor.layers.sparsity import get_sparsity_config
File "/home/michael/code/neuralmagic-vllm/vllm/model_executor/layers/sparsity/__init__.py", line 6, in <module>
raise ValueError(
ValueError: magic_wand is not available and required for sparsity support. Please install it with `pip install magic_wand`
```
bigPYJ1151
pushed a commit
to bigPYJ1151/vllm
that referenced
this pull request
Jul 30, 2024
flash attention for phi3
Closed
1 task done
1 task done
1 task done
1 task done
1 task done
1 task done
amd-xavierwang
pushed a commit
to amd-xavierwang/vllm
that referenced
this pull request
Apr 10, 2026
…project#37) Enables HybridW4A16LinearKernel for asymmetric quantized models (uint4 with per-group zero-points), which previously fell back to HipW4A16SkinnyLinearKernel. HipW4A16SkinnyLinearKernel is problematic because it stores a copy of pre-dequantized weights, which requires 4x more memory. Now both symmetric and asymmetric models get the Triton fused dequant GEMM for prefill and the HIP skinny kernel for decode, reading from a single weight copy. After this PR, I will remove the HipW4A16SkinnyLinearKernel. Unit tests: 20/20 pass (10 symmetric + 10 asymmetric). ``` Benchmark results (Strix Halo, gfx1151, --enforce-eager --max-num-seqs 1) Qwen/Qwen3-8B-AWQ (AWQ, asymmetric, group_size=128) ┌────────┬────────────────┬───────────┬─────────────┬─────────────┬──────────────┬──────────────┐ │ │ Kernel │ Memory │ TTFT in=128 │ TPOT in=128 │ TTFT in=2048 │ TPOT in=2048 │ ├────────┼────────────────┼───────────┼─────────────┼─────────────┼──────────────┼──────────────┤ │ Before │ HipW4A16Skinny │ 21.08 GiB │ 143ms │ 24.46ms │ 1060ms │ 25.70ms │ ├────────┼────────────────┼───────────┼─────────────┼─────────────┼──────────────┼──────────────┤ │ After │ HybridW4A16 │ 7.24 GiB │ 186ms │ 24.51ms │ 1499ms │ 25.72ms │ └────────┴────────────────┴───────────┴─────────────┴─────────────┴──────────────┴──────────────┘ RedHatAI/Qwen3-8B-quantized.w4a16 (CompressedTensors, asymmetric, group_size=128) ┌────────┬────────────────┬───────────┬─────────────┬─────────────┬──────────────┬──────────────┐ │ │ Kernel │ Memory │ TTFT in=128 │ TPOT in=128 │ TTFT in=2048 │ TPOT in=2048 │ ├────────┼────────────────┼───────────┼─────────────┼─────────────┼──────────────┼──────────────┤ │ Before │ HipW4A16Skinny │ 21.07 GiB │ 144ms │ 32.40ms │ 1008ms │ 33.70ms │ ├────────┼────────────────┼───────────┼─────────────┼─────────────┼──────────────┼──────────────┤ │ After │ HybridW4A16 │ 7.16 GiB │ 192ms │ 32.63ms │ 1592ms │ 33.87ms │ └────────┴────────────────┴───────────┴─────────────┴─────────────┴──────────────┴──────────────┘ Summary: ~3x memory reduction (21 GiB → 7 GiB) with no regression in decode TPOT. TTFT increases due to Triton kernel (prefill) being slower than hipBLASLt on small batch sizes, which is the expected trade-off — the memory savings enable running larger models or more concurrent requests. ```
flash7777
added a commit
to flash7777/vllm
that referenced
this pull request
Apr 16, 2026
Three changes that together enable loading a 244 GiB BF16 MoE model on a 119 GiB GB10 Unified Memory system via quant-on-load to XFP: 1. Layer-Grouped Safetensors Iterator (weight_utils.py) Pre-indexes all shard files by `.layers.N.` prefix, yields keys layer-by-layer instead of shard-by-shard. The existing per-layer quant trigger (`_sq_load_numel >= _sq_load_total`) now fires on time because each layer's weights arrive contiguously. Without this, safetensor shards interleave keys from many layers, causing all MoE layers to materialize simultaneously (peak = full model BF16). 2. Expert-wise MoE Packing (online_moe.py) `_expertwise_pack_and_repack` processes one expert at a time: float32 transient = N×K×4 bytes (~25 MiB) instead of E×N×K×4 (~9 GiB for 256 experts). The batched approach caused OOM at layer vllm-project#37/47 even with layer-grouped loading. 3. Supporting fixes: - MoE-scoped meta-context: thread-local flag in base_loader around initialize_model, create_weights allocates on meta device - Meta→Real materialize in initialize_streaming_quantload for FusedMoE params (plain nn.Parameter, no __torch_function__ issue) - Registry rebuild across process spawn (mq_dtype/mq_gs persisted as instance attrs on AutoRoundRTNConfig) - ViT/vision layers classified as "other" → skip XFP quantization - Outlier-scatter guard: early-return when outlier_col.numel() == 0 - Enhanced memory instrumentation: GC tensor inventory, memory_stats breakdown, cudaMemGetInfo driver view, /proc/meminfo full fields Qwen3.5-122B-A10B result (XFP auto + fp8 KV + fp8 LM Head): - 48 MoE layers: all xfp4, 256 experts, expertwise Lloyd (5 iters) - 120 Linear layers: all xfp3 (cos≈0.982, outliers 0.1–1.3%) - Final CUDA alloc: 70.2 GiB (vs 244 GiB BF16 original) - Peak during load: never exceeded 80 GiB, Avail never below 35 GiB - Math: 7*8=56 ✓, Application startup complete ✓ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jxmorris12
added a commit
to jxmorris12/vllm
that referenced
this pull request
Apr 21, 2026
* fix lgoprobs mem error with chunked prefill * fix lora! * merge upstream fixes for LoRA
nonameentername
pushed a commit
to nonameentername/vllm
that referenced
this pull request
Oct 7, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
cpuchip
pushed a commit
to cpuchip/vllm
that referenced
this pull request
Oct 9, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
cpuchip
pushed a commit
to cpuchip/vllm
that referenced
this pull request
Oct 19, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
cpuchip
pushed a commit
to cpuchip/vllm
that referenced
this pull request
Oct 20, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
cpuchip
pushed a commit
to cpuchip/vllm
that referenced
this pull request
Oct 20, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
cpuchip
pushed a commit
to cpuchip/vllm
that referenced
this pull request
Nov 5, 2026
sm80 Marlin repack: stage the per-layer transients through one grow-only buffer (vllm-project#27; allocation hygiene — the wedge theory below was DISPROVEN). History, kept honest: this patch was written against the theory that VMM map/unmap churn during Marlin repack wedged ahnguyen17's CMP 170HX (Xid 31, drifting faulting frames, wedge-until-reboot). The real root cause was found by ahnguyen17 afterwards and is not vLLM's at all: their CMP unlock driver registered the top reserved framebuffer region — which backs WPR2/GSP firmware — into the page allocator; it sorts last, so it is handed out only near full occupancy, which produced exactly the occupancy dependence and frame drift this header once attributed to churn. With the unlocker fixed, the stock un-staged repack path boots and serves clean on the same card (issue vllm-project#27, final comments). Do NOT cite this patch as evidence of an sm80 Marlin bug; stock-map sm80 (A100s) never had this failure mode. What survives, and why the patch stays: staging is measurably better allocation behavior during load. On the 170HX A/B (w8a16 target, full serving session, 8 streams): repack window 44.9 s against the CPU fallback's 403.7 s, engine ready ~90 s against ~540 s, decode identical to three decimals, outputs bit-exact end to end, buffer cost inside noise (≤0.15 GiB there). ahnguyen17 promoted it to their production after the driver fix on those grounds. This is the GPU-resident version of that workaround: pad+contiguous land in ONE persistent, grow-only staging buffer per device, and the source tensor is dropped before the repack output allocates. Per-layer allocation traffic falls from two transients + one output to just the output; the staging buffer reallocates only when a larger shape class first appears (a handful of times per load, instead of hundreds of alloc/free pairs). Bit-exactness is by construction — the repack kernel receives identical bytes — and verified on the 3090 across the checkpoint's real shape ladder (w4 and w8, padded and unpadded, contiguous and strided sources, buffer reuse and regrowth): kernel inputs and outputs torch.equal in all cases. Costs and gating: the buffer stays resident after load — up to ~1.2 GiB on this stack, because the quantized embedding/lm_head whales set its size. It is opt-in on every card: VLLM_MARLIN_REPACK_STAGED=1. It used to default ON for compute capability 8.0 exactly, on the 170HX's 9x load time, but that 9x was against the CPU fallback; on a real A100 40 GB against the stock GPU path, staging saves 0.85 s of load (9.30 against 10.15 s) and holds 1.19 GiB, 5.8% of the pool. The GA100 mining cards whose alternative is the CPU fallback turn it on (docs/gotchas.md vllm-project#37). ahnguyen17's CPU-fallback patch remains the zero-VRAM alternative. Status: validated on the 170HX (4 boot cycles + a full serving session, bit-exact, 0 Xid) and in production there — but as load-time hygiene, not as the wedge fix this header originally proposed; see the history above. Source: syv-ai/qwen38-27b-rtx3090 patches/marlin-repack-staged-sm80.patch (at d4c4cc2). Applied as-is from the patch file; the preamble above is the patch's own. Signed-off-by: Michael Stufflebeam <cpuchip@gmail.com>
triangleXIV
pushed a commit
to tokmo-infra/vllm
that referenced
this pull request
Nov 9, 2026
…cked + tuned GEMMs (vllm-project#37) * [Kimi K3.1] DSA: fused stock-path prologue, as DeepseekV32Attention The K3.1 DSA layers went through MultiHeadLatentAttentionWrapper: three input GEMMs, separate q_a / kv_a RMSNorms, the indexer key insert and the MLA cache insert as their own launches, and the output gate after attention. Follow DeepseekV32Attention instead when the layer allows it (fp8_ds_mla, NoPE, no DCP / PCP / hisparse, unquantized input projections): - One input GEMM over the packed [q_a | kv_a | wk | weights_proj] weight (packed after loading; the linears become views). - dsa_fused_prologue: q_a / kv_a RMSNorm, fp8_ds_mla insert of [kv_c_normed | k_pe], indexer key LayerNorm + FP8 insert and the top-k fill in one launch (fused_norm_rope without RoPE). The indexer key insert and top-k fill are shared with indexer_k_norm_quant_insert. - Attention through unified_mla_attention_with_output, skipping the wrapper's cache update. - The output-gate GEMM overlaps on the aux stream below 512 tokens, as in the dense Kimi-K3 MLA. Other configurations, dense drafts and the specialized decode (which packs the projections itself) keep the wrapper forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [Kimi K3.1] DSA: pack [q_b | wq_index] and tune the packed GEMMs Pack the query GEMMs as well, and add SM103 low-latency entries for the three packed shapes. Measured on GB300 (CUDA-graph replay, weights rotated past L2, median of 3), packed vs the separate GEMMs on their tuned plans: - [q_a | kv_a | wk | weights_proj] 2272x7168: 1.23-1.72x at M1..16; CuTe wins >=5% over packed cuBLAS at M1..5. - [q_b | wq_index] 22528x1536 (TP1): 1.16-1.38x; CuTe at M1. - [q_b | wq_index] 6400x1536 (TP8): 1.38-1.77x; CuTe at M1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> * [Kimi K3.1] DSA: review fixes for the fused forward - Warm up the CuTe configs of the packed GEMM shapes (no linear module owns them, so enable_kimi_k3_low_latency_gemm never requested them). - Make q contiguous on steps with prefills: it is a column slice of the packed query GEMM and some MHA prefill backends assume contiguous inputs. - Keep the wrapper forward with CPU / prefetch weight offload, and unless the three norm eps match (the fused prologue takes one). - Share low_latency_linear between the dense MLA and DSA paths. - Allocate the prologue's cache placeholders only when a cache is absent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> --------- Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.