Repository navigation
[Kernel][ROCm][Perf] FlyDSL decode-attention kernel for 4-bit TurboQuant KV cache - #47896
Conversation
fxmarty-amd
left a comment
There was a problem hiding this comment.
Great work, left a few comments!
Main questions are:
- mixed attention backend handling in vLLM / frontend API
- Should kernels be hosted in vLLM or elsewhere (e.g., aiter)?
- Yet new
VLLM_ROCM_*environment flags vs. abstracting turboquant backends (similar to--moe-backend,--linear-backend, etc.).
| # NOTE: ``aiter.ops.flydsl`` only ships in aiter >= v0.1.13.post1. Import | ||
| # it lazily inside the branches that actually need ``GateMode`` so that | ||
| # the common CK fused-MoE path (fp8 w8a8 / SILU, e.g. MiniMax) keeps | ||
| # working on older aiter builds that lack the flydsl submodule. |
There was a problem hiding this comment.
There was a problem hiding this comment.
The base image (v0.1.16.post3) has aiter.ops.flydsl, but some of our dev images don't ship that submodule at all, my env didnt have it. A top-level import would break those at import time. I will reword the comment to say "builds that don't ship aiter.ops.flydsl".
There was a problem hiding this comment.
some of our dev images don't ship that submodule at all
This import indeed fails on vllm/vllm-openai-rocm:v0.24.0-base from 11 days ago, which contains amd-aiter==0.1.13.post1.
This import passes on vllm/vllm-openai-rocm:nightly-95ed0feaa5cd7fb16d72c53ce04950aaf07c4698 which contains amd-aiter==0.1.16.post3.
This import passes as well on rocm/vllm-dev:nightly_main_20260709 which contains amd-aiter==0.1.16.post3.
I would suggest to revert this change.
There was a problem hiding this comment.
Reverted the lazy-import shim and the runtime swiglu_limit capability probe.
| logger.debug( | ||
| "Selected backend %s is incompatible with this layer " | ||
| "(%s); falling back to auto-selection. Reason: %s", | ||
| selected_backend.name, | ||
| attn_selector_config.attn_type, | ||
| invalid_reasons, | ||
| ) |
There was a problem hiding this comment.
This silently means that invalid_reasons = backend_class.validate_configuration(...) is ignored.
So you could specify --attention-backend FLASHINFER_MLA_SPARSE_SM120 and not get an error raised here? This does not sound right.
Instead of:
# specific layer (e.g. user passed ROCM_AITER_UNIFIED_ATTN but # this layer needs TURBOQUANT for a quantized KV cache dtype).
why would users not be required to pass --kv-cache-dtype turboquant_4bit_nc --attention-backend TURBOQUANT ?
For the mixed backend case:
mixed-backend setup (AITER for prefill / non-TQ layers
TURBOQUANT for turboquant_* KV-cache layers).
which probably corresponds to the blog post:
Skip boundary layers for full-attention models. The first and last layers are often the most sensitive to KV quantization and leaving them unquantized can recover substantial accuracy at modest compression cost. The same effect is documented in TurboQuant+ Boundary V experiments [2]. We include some analysis on the magnitude of this effect across different forms of attention.
Does vLLM already handle multi-backend in AttentionConfig ? If not, or if there is no abstraction supporting that, it seems that this would better be a standalone PR & that opening an RFC issue (if there is none yet) is a good idea.
How is --kv-cache-dtype-skip-layers currently playing along with --attention-backend?
There was a problem hiding this comment.
- Silent validate_configuration / wrong backend not raising: Agreed, I'll scope the fall-through to only turboquant_* kv-dtypes and re-raise for everything else, so FLASHINFER_MLA_SPARSE_SM120 errors again.
- Why not require --attention-backend TURBOQUANT / boundary-layer correspondence: Because the boundary (first/last) layers are kept unquantized, native KV dtype → AITER — while the rest go turboquant_* → TURBOQUANT as per the blog post snippet
- Multi-backend in AttentionConfig / RFC: AFAIK No abstraction exists; --attention-backend is a single global selector
- --kv-cache-dtype-skip-layers vs --attention-backend: skip-layers forces those layers to native dtype (→ AITER); non-skip turboquant_* layers fall to TURBOQUANT
I will do the following: Scope the fall-through to only turboquant_* kv-dtypes (re-raise for any other invalid --attention-backend), keep the info_once log, and reword the comment to cite boundary-layer skipping
There was a problem hiding this comment.
I still don't get how this change relates to flydsl kernel integration really. Is this not an existing issue, unrelated to this kernel integration? If so, could we submit a fix in a separate PR?
Why not require --attention-backend TURBOQUANT / boundary-layer correspondence: Because the boundary (first/last) layers are kept unquantized, native KV dtype → AITER — while the rest go turboquant_* → TURBOQUANT as per the blog post snippet
How is this different from existing behavior? e.g. https://vllm.ai/blog/2026-05-11-turboquant mentions:
so surely there is already some logic to handle mixed attention backend, if some attention backends are not compatible with --kv-cache-dtype turboquant_4bit_nc ?
There was a problem hiding this comment.
AFAIK, apart for the frontend at https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_flydsl_moe.py, AMD-specific fyldsl kernels are implemented in aiter.ops.flydsl.
@mgoin Is vLLM the good place to host these kernels or would we be better off using AMD-specific repos (aiter/flydsl/amd-quark) and importing from there, leaving vLLM to only handle the frontend part?
There was a problem hiding this comment.
Followed PR https://github.com/vllm-project/vllm/pull/44400 to keep flydsl kernel together in vllm
There was a problem hiding this comment.
Was told keeping AMD-only kernels in vllm-project/vllm is fine.
| VLLM_ROCM_TQ_FLYDSL_DECODE: bool = False | ||
| VLLM_TQ_FLYDSL_WHT_BUTTERFLY: bool = False |
There was a problem hiding this comment.
Do we really need to add these environment flags? Is it because of:
- Apply rotation and LUT-based quantization to K, and use standard uniform quantization for V. In our analysis, keys are more quantization-sensitive than values.
- Use Walsh-Hadamard rotation instead of random rotation. In practice, WHT is both more kernel-friendly and more accurate. TurboQuant+ also converges to the same choice for llama.cpp [2].
- Omit QJL in the default 4-bit path. In our experiments, QJL adds complexity and runtime overhead without delivering accuracy benefits in our experiments. This matches community consensus as some works drop QJL and D’Alberto et al. report similar conclusions [2, 3].
That is, VLLM_ROCM_TQ_FLYDSL_DECODE=1 is not numerically equivalent with the current other default backend?
Should we not have, similar to nvfp4/mxfp4 gemm/moe, say an automatic selection of turboquant backend, based on supported dtype, platform, etc?
So that e.g. flydsl is used by default as the turboquant implementation backend on gfx950 only, only in the specified case (4-bit, etc.).
If this implementation is not numerically equivalent, that is running --kv-cache-dtype turboquant_4bit_nc vs running VLLM_ROCM_TQ_FLYDSL_DECODE=1 --kv-cache-dtype turboquant_4bit_nc does not yield the same results, should this not be named differently?
Or --kv-cache-config backend=flydsl + warning?
There was a problem hiding this comment.
These are opt-in gates for a new, gfx950-only kernel, default-off keeps the existing TQ path unchanged/untouched. Numerically it's the same TurboQuant scheme, validated against an fp32 oracle and comparable to the Triton path within bf16 tolerance. Butterfly flag is more of a performance optimization flag which we could make default in the flydsl decode run to get rid of the opt-in flag. I agree the better long-term shape is a config field (--kv-cache-config backend=flydsl) and/or auto-selection on gfx950 rather than env vars. Any other suggestions on this before I make this change?
There was a problem hiding this comment.
Essentially wondering why this would be opt-in, if this is strictly better than current turboquant implementation and numerically equivalent.
Why not make it default on gfx950?
| if not _flydsl_v4_available(): | ||
| logger.warning( | ||
| "VLLM_ROCM_TQ_FLYDSL_DECODE=1 but FlyDSL is unavailable; the decode path " | ||
| "will fall back to the SoA Triton v3 kernel." | ||
| ) |
There was a problem hiding this comment.
If this is an opt-in and unavailable, an error should be raised.
There was a problem hiding this comment.
Agreed, now raises ValueError at init if VLLM_ROCM_TQ_FLYDSL_DECODE=1 but FlyDSL isn't importable. The per-layer fallback to Triton for FlyDSL-ineligible layers (GQA/head/dtype) stays, since no single kernel covers every layer.
| else: | ||
| _tq_full_dequant_kv[grid]( | ||
| kv_cache, | ||
| block_table, | ||
| centroids, | ||
| k_cached, | ||
| v_cached, | ||
| k_cached.stride(0), | ||
| k_cached.stride(1), | ||
| k_cached.stride(2), | ||
| v_cached.stride(0), | ||
| v_cached.stride(1), | ||
| v_cached.stride(2), | ||
| kv_cache.stride(0), | ||
| kv_cache.stride(1), | ||
| kv_cache.stride(2), | ||
| block_table.stride(0), | ||
| HEAD_DIM=D, | ||
| BLOCK_SIZE=block_size, | ||
| NUM_KV_HEADS=Hk, | ||
| MSE_BYTES=mse_bytes, | ||
| KPS=self.tq_config.key_packed_size, | ||
| VQB=self.tq_config.effective_value_quant_bits, | ||
| VAL_DATA_BYTES=val_data_bytes, | ||
| MSE_BITS=self.tq_config.key_mse_bits, | ||
| KEY_FP8=1 if self.tq_config.key_fp8 else 0, | ||
| BLOCK_D=BLOCK_D, | ||
| NORM_CORRECTION=1 if self.tq_config.norm_correction else 0, | ||
| FP8_E4B15=_use_fp8_e4b15(device.index or 0), | ||
| num_warps=4, | ||
| ) |
There was a problem hiding this comment.
Can we not inherit from TurboQuantAttentionImpl instead of modifying it with if/else branches?
And have a stateful dispatch to the correct TurboQuantAttentionImpl class/subclass?
There was a problem hiding this comment.
Yes, makes sense. So basically, keep TurboQuantAttentionImpl as the base (default Triton path untouched) and add a FlyDSLTurboQuantAttentionImpl subclass that overrides store/continuation/dequant/decode, so the if VLLM_ROCM_TQ_FLYDSL_DECODE branches disappear. I could do that once we have a decision on the env variable versus config field, maybe a follow up RFC/PR.
There was a problem hiding this comment.
This is just for readability purposes / easier review, yes
| logger.warning_once( | ||
| "TurboQuant v4 ineligible (key_fp8=%s mse_bits=%s vqb=%s " | ||
| "head_size=%s num_kv_groups=%s sinks=%s) -> SoA Triton v3", | ||
| self.tq_config.key_fp8, | ||
| self.tq_config.key_mse_bits, | ||
| self.tq_config.effective_value_quant_bits, | ||
| self.head_size, | ||
| self.num_kv_groups, | ||
| self.sinks is not None, | ||
| ) | ||
| return self._dispatch_decode_v3( | ||
| query=query, | ||
| kv_cache=kv_cache, | ||
| block_table=attn_metadata.block_table, | ||
| seq_lens=attn_metadata.seq_lens, | ||
| Pi=Pi, | ||
| centroids=centroids, | ||
| scale=self.scale, | ||
| mse_bits=self.tq_config.key_mse_bits, | ||
| key_packed_size=self.tq_config.key_packed_size, | ||
| value_quant_bits=self.tq_config.effective_value_quant_bits, | ||
| value_packed_size=self.tq_config.value_packed_size, | ||
| max_seq_len=attn_metadata.max_seq_len, | ||
| key_fp8=self.tq_config.key_fp8, | ||
| norm_correction=self.tq_config.norm_correction, | ||
| PiT=PiT, | ||
| mid_o_buf=mid_o_buf, | ||
| output_buf=output_buf, | ||
| lse_buf=lse_buf, | ||
| buf_holder=layer, | ||
| max_num_kv_splits=self.max_num_kv_splits, | ||
| sinks=self.sinks, | ||
| sliding_window=self.sliding_window, | ||
| ) |
There was a problem hiding this comment.
I find it slightly counter-intuitive that VLLM_ROCM_TQ_FLYDSL_DECODE=1 may dispatch on a triton kernel.
nit: given that this v4_eligible check can be done ahead of time, I would also have a subclass that overrides _decode_attention in case the flydsl kernel does not support the problem shape.
Generally - would it not make sense to only integrate one the best of flydsl/triton implementation? Instead of having the two under the same VLLM_ROCM_TQ_FLYDSL_DECODE=1 name?
Or alternatively, e.g. have a single entrypoint aiter that dispatches on the best supported available kernel for the problem shape?
e.g. as https://github.com/ROCm/aiter/blob/main/aiter/configs/bf16_tuned_gemm.csv
There was a problem hiding this comment.
Agreed the flag naming is misleading, but we can't ship only the best of the two: FlyDSL is a specialized fast path (gfx950, head_size=128, GQA {6,8,16}, 4-bit MSE keys), Triton v3 is the general fallback for every other shape/arch with SOA storage so both need to stay in some way, maybe another name? Not entirely sure about the aiter alternative, but can look into it.
| use_hw_v_transpose: bool = False, | ||
| tile_groups_per_partition: int = 1, |
There was a problem hiding this comment.
No hyperparameter tuning required here?
There was a problem hiding this comment.
use_hw_v_transpose isn't tuned, False is builder default. At launch it's set automatically from the GPU arch (_hw_tr_enabled() → True on gfx950, False elsewhere), so it turns on only where the hardware supports it. num_partitions and tile_groups_per_partition are computed automatically from the sequence length (same as Triton path), no autotuning needed here.
| import torch | ||
|
|
||
| from vllm.platforms import current_platform | ||
| from vllm.platforms.rocm import on_gfx950 |
There was a problem hiding this comment.
This import will fail on other platform, we have to guard with current_platform.is_rocm()
if current_platform.is_rocm():
from vllm.platforms.rocm import on_gfx950
...There was a problem hiding this comment.
Good catch, will fix.
| @@ -367,12 +367,16 @@ def rocm_aiter_fused_experts( | |||
| # `rocm_aiter_ops.shuffle_weight_a16w4` in `oracle/mxfp4.py`, | |||
| # which always sets `is_guinterleave=True`. | |||
| # Hence, we pass in GateMode.INTERLEAVE to match the weight shuffling. | |||
| from aiter.ops.flydsl.moe_common import GateMode | |||
|
|
|||
| # Import ``GateMode`` lazily: some aiter builds lack ``aiter.ops.flydsl``, | |||
There was a problem hiding this comment.
We are keeping support of a specific version of AITER on vLLM, we do not guarantee backwards nor forward version compatibility to avoid adding complex layer of code checks.
There was a problem hiding this comment.
So please use the AITER shipped on upstream docker image.
There was a problem hiding this comment.
Reverted, using AITER shipped on upstream.
| @@ -0,0 +1,143 @@ | |||
| # FlyDSL kernels for TurboQuant (vendored) | |||
There was a problem hiding this comment.
remove this file before final merge.
There was a problem hiding this comment.
Agreed, will remove.
| @@ -1843,6 +1833,23 @@ def fused_moe_supports_gate_mode(cls) -> bool: | |||
|
|
|||
| return "gate_mode" in inspect.signature(fused_moe).parameters | |||
|
|
|||
| @classmethod | |||
There was a problem hiding this comment.
We only support specific version of AITER, we do not guarantee backwards and forward compatibility.
| # whole SoA TurboQuant pipeline (SoA store + FlyDSL decode + SoA-aware | ||
| # continuation + SoA Triton v3 fallback) is enabled. Default off keeps the | ||
| # upstream TurboQuant v1 path unchanged. | ||
| "VLLM_ROCM_TQ_FLYDSL_DECODE": lambda: os.environ.get( |
There was a problem hiding this comment.
Let's not add flags for attention backend, all of those should go through python arguments system.
| @@ -70,6 +75,56 @@ | |||
| _CONTINUATION_DECODE_THRESHOLD = 128 | |||
|
|
|||
|
|
|||
| # --------------------------------------------------------------------------- # | |||
| # FlyDSL TurboQuant decode (opt-in; AMD MI355X / gfx950). # | |||
There was a problem hiding this comment.
Please move all of the flydsl into a turboquant_attn_flydsl.py .
we expect that --attention-backend turboquant_flydsl
There was a problem hiding this comment.
cc @mgoin Are we looking to contain turboquant related code within --attention-backend turboquant (as you discussed in vLLM SIG quantization call)?
If so, @mgoin suggested --attention-config.turboquant-backend=flydsl/xxx, is it acceptable @tjtanaa? There is a precedent for it with:
Lines 45 to 47 in 5810e88
| # Only tolerate the mismatch for turboquant_* KV-cache layers: | ||
| # boundary layers keep the native dtype (served by the selected | ||
| # backend) while turboquant_* layers need TURBOQUANT, so no single | ||
| # --attention-backend can serve every layer. For any other dtype | ||
| # the explicit selection is genuinely invalid -> fail loud. | ||
| kv_dtype = attn_selector_config.kv_cache_dtype | ||
| if not (kv_dtype is not None and str(kv_dtype).startswith("turboquant")): | ||
| raise ValueError( | ||
| f"Selected backend {selected_backend} is not valid for " | ||
| f"this configuration. Reason: {invalid_reasons}" | ||
| ) | ||
| # NOTE: pass a str (not the list) -- info_once hashes its args. | ||
| logger.info_once( | ||
| "Selected backend %s is incompatible with this turboquant " | ||
| "layer (%s); using the auto-selected per-layer backend. " | ||
| "Reason: %s", | ||
| selected_backend.name, | ||
| attn_selector_config.attn_type, | ||
| str(invalid_reasons), | ||
| ) |
There was a problem hiding this comment.
So as discussed offline, I understand this is in order to have --kv-cache-dtype turboquant_4bit_nc --attention-backend xyz not crashing, that is, being able to select the attention backend of non-turboquant layers (e.g. first and last skipped layers), e.g. in case vLLM's default is not the desired one or the best performing one.
If so, the behavior should be identical on all platforms I believe (allow --attention-backend xyz on nvidia as well, and verify it works).
@mgoin @tjtanaa Is --kv-cache-dtype turboquant_4bit_nc --attention-backend ROCM_AITER_UNIFIED_ATTN acceptable in order to select non-default attention backend for non-turboquant layers?
Or should --attention-backend expect only turboquant for --kv-cache-dtype turboquant_* cases, and there should be a configuration that goes along --kv-cache-dtype-skip-layer to precise the attention backend of non-turboquant layers?
Should this be done in an other PR or here is OK?
|
Hi @aditi-amd, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
Add the FlyDSL-based TurboQuant 4-bit KV-cache decode kernel (v4) for ROCm/MI355X, with butterfly (WHT) transform support and a GQA-6 sibling kernel for MiniMax-class models. Gated behind VLLM_ROCM_TQ_FLYDSL_DECODE with graceful fallback to the Triton v3 path when FlyDSL is unavailable. Also guards the ROCm aiter fused-MoE path for older aiter builds: capability-probe swiglu_limit/gate_mode kwargs and lazily import aiter.ops.flydsl only in the branches that need it. Signed-off-by: aditi <aditi.rana@amd.com> Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…t tests Reduce the user-facing env surface of the FlyDSL TurboQuant 4-bit KV decode kernel from 7 flags to 2 (VLLM_ROCM_TQ_FLYDSL_DECODE and VLLM_TQ_FLYDSL_WHT_BUTTERFLY). Internalize tuning defaults, auto-detect the hardware-transpose path, and drop the deprecated tuning/debug env vars and sys.path bootstraps from the kernel modules. Add a parametrized pytest (tests/kernels/turboquant_v4/) that checks the FlyDSL v4 decode output against a pure-PyTorch fp32 attention oracle across batch sizes, sequence lengths, GQA ratios, and block sizes. Also apply ruff lint/format fixes (import ordering, contextlib.suppress, line-length) with no behavioral changes. Signed-off-by: aditi <aditi.rana@amd.com> Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…aming Signed-off-by: aditi <aditi.rana@amd.com> Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…e otherwise Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
… vendored README Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…rfly off, clean up kernel comments Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…d WHT butterfly Replace VLLM_ROCM_TQ_FLYDSL_DECODE with capability gating (on_gfx950 + importable) and remove the hardwired-off WHT butterfly path from the launcher and both decode kernels. Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
nit: This link at PR description is broken. Could you please update to https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html ? |
Thanks for catching that! I've updated the URL in the PR description. |
|
/ci run |
|
/ci run |
|
✅ Triggered Buildkite CI #82799 for commit |
|
@mgoin I think these failures are known. |
Purpose
This PR upstreams the FlyDSL TurboQuant 4-bit KV-cache decode kernel developed and optimized by the AMD team. For background on the TurboQuant algorithm and its role in agentic vLLM serving, see our blog write-up [TurboQuant blog] (https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html)
This PR adds a custom FlyDSL TurboQuant 4-bit KV-cache decode kernel for ROCm / MI355X (gfx950) as an opt-in alternative to the default vLLM TurboQuant Triton decode, for its supported profile:
Selection is automatic (no env flag): on gfx950 with 4-bit TurboQuant KV and FlyDSL importable, eligible decode layers route to the FlyDSL kernel via capability gating (on_gfx950()). Layers with unsupported shapes — or all layers if FlyDSL isn't importable — fall back to the SoA Triton TurboQuant decode.
Test Plan
Test Result
Accuracy — GSM8K (exact match, 1319 samples)
| Model | BF16 | FlyDSL TQ44 |
|---|---|---|---|
| MiniMax-M2.5 | 88.17 | 87.95 (−0.22) |
| Qwen3-32B | 95.45 | 95.15 (−0.30) |
Accuracy is neutral (within ~0.4pp of BF16 across both models).
Perf — MiniMax-M2.5, 32K in / 1K out, TP=2, C=64, N=80, MI355X
Recommended serve command
Bench (identical for every leg):