Skip to content

[Kernel][ROCm][Perf] FlyDSL decode-attention kernel for 4-bit TurboQuant KV cache - #47896

Merged
vllm-bot merged 14 commits into
vllm-project:mainfrom
aditi-amd:feat/flydsl-tq-v4-pr
Aug 11, 2026
Merged

vllm-bot merged 14 commits into
vllm-project:mainfrom
aditi-amd:feat/flydsl-tq-v4-pr

Conversation

@aditi-amd

@aditi-amd aditi-amd commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

This PR upstreams the FlyDSL TurboQuant 4-bit KV-cache decode kernel developed and optimized by the AMD team. For background on the TurboQuant algorithm and its role in agentic vLLM serving, see our blog write-up [TurboQuant blog] (https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html)

This PR adds a custom FlyDSL TurboQuant 4-bit KV-cache decode kernel for ROCm / MI355X (gfx950) as an opt-in alternative to the default vLLM TurboQuant Triton decode, for its supported profile:

  1. KV cache quantized to 4-bit (MSE-key centroids + 4-bit value), HEAD_SIZE=128
  2. GQA group ∈ {8, 16} → canonical tq_decode kernel
  3. GQA group == 6 → tq_decode_gqa6 sibling kernel (MiniMax-M2.5)
  4. arch == gfx950 ( gfx950 / CDNA4; the kernel uses CDNA4-only intrinsics and is tuned and validated there)

Selection is automatic (no env flag): on gfx950 with 4-bit TurboQuant KV and FlyDSL importable, eligible decode layers route to the FlyDSL kernel via capability gating (on_gfx950()). Layers with unsupported shapes — or all layers if FlyDSL isn't importable — fall back to the SoA Triton TurboQuant decode.

Test Plan

  1. Kernel correctness (unit test): tests/kernels/turboquant/test_flydsl_turboquant_decode.py validates the flydsl decode kernel against a pure-PyTorch fp32 attention oracle built from the known dequantized KV values (both sides see identical quantized data; only bf16-vs-fp32 accumulation differs). Parametrized over batch × seq-len × {GQA-8, GQA-16, GQA-6} × block-size {16, 32}. Skips cleanly off ROCm/gfx950 or when FlyDSL is not importable.
  2. Accuracy (GSM8K, full 1319-sample test set) on MiniMax-M2.5 and Qwen3-32B: bf16 baseline vs TQ44v4.
  3. End-to-end perf runs: identical server + bench config, only the KV-cache dtype / decode kernel differs per leg.

Test Result

Accuracy — GSM8K (exact match, 1319 samples)

| Model | BF16 | FlyDSL TQ44 |
|---|---|---|---|
| MiniMax-M2.5 | 88.17 | 87.95 (−0.22) |
| Qwen3-32B | 95.45 | 95.15 (−0.30) |

Accuracy is neutral (within ~0.4pp of BF16 across both models).

Perf — MiniMax-M2.5, 32K in / 1K out, TP=2, C=64, N=80, MI355X

image The middle leg is the existing default vLLM TurboQuant 4-bit decode. TQ44 FlyDSL is +7% over BF16 and ~4.1× the default TurboQuant decode at 4-bit KV.

Recommended serve command

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1          # keep the fast aiter CK fused-MoE


vllm serve <MiniMax-M2.5> --port 8000 \
  --tensor-parallel-size=2 --enable-expert-parallel \
  --gpu-memory-utilization 0.85 --max-model-len 34816 \
  --kv-cache-dtype turboquant_4bit_nc --block-size=32 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  --no-enable-prefix-caching --no-enable-log-requests \
  --trust-remote-code --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'

Bench (identical for every leg):

vllm bench serve --backend openai --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions --model <MiniMax-M2.5> --dataset-name random \
  --random-input-len 32768 --random-output-len 1024 --random-range-ratio 0 \
  --max-concurrency 64 --num-prompts 80 --request-rate inf --ignore-eos \
  --num-warmups 8 --seed 42

@mergify mergify Bot added rocm Related to AMD ROCm v1 labels Jul 7, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Jul 7, 2026

@fxmarty-amd fxmarty-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, left a few comments!

Main questions are:

  • mixed attention backend handling in vLLM / frontend API
  • Should kernels be hosted in vLLM or elsewhere (e.g., aiter)?
  • Yet new VLLM_ROCM_* environment flags vs. abstracting turboquant backends (similar to --moe-backend, --linear-backend, etc.).

Comment on lines +370 to +373
# NOTE: ``aiter.ops.flydsl`` only ships in aiter >= v0.1.13.post1. Import
# it lazily inside the branches that actually need ``GateMode`` so that
# the common CK fused-MoE path (fp8 w8a8 / SILU, e.g. MiniMax) keeps
# working on older aiter builds that lack the flydsl submodule.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed?

As per

ARG AITER_BRANCH="v0.1.16.post3"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base image (v0.1.16.post3) has aiter.ops.flydsl, but some of our dev images don't ship that submodule at all, my env didnt have it. A top-level import would break those at import time. I will reword the comment to say "builds that don't ship aiter.ops.flydsl".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some of our dev images don't ship that submodule at all

This import indeed fails on vllm/vllm-openai-rocm:v0.24.0-base from 11 days ago, which contains amd-aiter==0.1.13.post1.

This import passes on vllm/vllm-openai-rocm:nightly-95ed0feaa5cd7fb16d72c53ce04950aaf07c4698 which contains amd-aiter==0.1.16.post3.

This import passes as well on rocm/vllm-dev:nightly_main_20260709 which contains amd-aiter==0.1.16.post3.

I would suggest to revert this change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted the lazy-import shim and the runtime swiglu_limit capability probe.

Comment thread vllm/platforms/rocm.py Outdated
Comment on lines +563 to +569
logger.debug(
"Selected backend %s is incompatible with this layer "
"(%s); falling back to auto-selection. Reason: %s",
selected_backend.name,
attn_selector_config.attn_type,
invalid_reasons,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This silently means that invalid_reasons = backend_class.validate_configuration(...) is ignored.

So you could specify --attention-backend FLASHINFER_MLA_SPARSE_SM120 and not get an error raised here? This does not sound right.

Instead of:

        # specific layer (e.g. user passed ROCM_AITER_UNIFIED_ATTN but
       # this layer needs TURBOQUANT for a quantized KV cache dtype).

why would users not be required to pass --kv-cache-dtype turboquant_4bit_nc --attention-backend TURBOQUANT ?


For the mixed backend case:

mixed-backend setup (AITER for prefill / non-TQ layers
TURBOQUANT for turboquant_* KV-cache layers).

which probably corresponds to the blog post:

Skip boundary layers for full-attention models. The first and last layers are often the most sensitive to KV quantization and leaving them unquantized can recover substantial accuracy at modest compression cost. The same effect is documented in TurboQuant+ Boundary V experiments [2]. We include some analysis on the magnitude of this effect across different forms of attention.

Does vLLM already handle multi-backend in AttentionConfig ? If not, or if there is no abstraction supporting that, it seems that this would better be a standalone PR & that opening an RFC issue (if there is none yet) is a good idea.

How is --kv-cache-dtype-skip-layers currently playing along with --attention-backend?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Silent validate_configuration / wrong backend not raising: Agreed, I'll scope the fall-through to only turboquant_* kv-dtypes and re-raise for everything else, so FLASHINFER_MLA_SPARSE_SM120 errors again.
  2. Why not require --attention-backend TURBOQUANT / boundary-layer correspondence: Because the boundary (first/last) layers are kept unquantized, native KV dtype → AITER — while the rest go turboquant_* → TURBOQUANT as per the blog post snippet
  3. Multi-backend in AttentionConfig / RFC: AFAIK No abstraction exists; --attention-backend is a single global selector
  4. --kv-cache-dtype-skip-layers vs --attention-backend: skip-layers forces those layers to native dtype (→ AITER); non-skip turboquant_* layers fall to TURBOQUANT

I will do the following: Scope the fall-through to only turboquant_* kv-dtypes (re-raise for any other invalid --attention-backend), keep the info_once log, and reword the comment to cite boundary-layer skipping

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't get how this change relates to flydsl kernel integration really. Is this not an existing issue, unrelated to this kernel integration? If so, could we submit a fix in a separate PR?

Why not require --attention-backend TURBOQUANT / boundary-layer correspondence: Because the boundary (first/last) layers are kept unquantized, native KV dtype → AITER — while the rest go turboquant_* → TURBOQUANT as per the blog post snippet

How is this different from existing behavior? e.g. https://vllm.ai/blog/2026-05-11-turboquant mentions:

image

so surely there is already some logic to handle mixed attention backend, if some attention backends are not compatible with --kv-cache-dtype turboquant_4bit_nc ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, apart for the frontend at https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_flydsl_moe.py, AMD-specific fyldsl kernels are implemented in aiter.ops.flydsl.

@mgoin Is vLLM the good place to host these kernels or would we be better off using AMD-specific repos (aiter/flydsl/amd-quark) and importing from there, leaving vLLM to only handle the frontend part?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followed PR https://github.com/vllm-project/vllm/pull/44400 to keep flydsl kernel together in vllm

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good, cc @mgoin for comments

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was told keeping AMD-only kernels in vllm-project/vllm is fine.

Comment thread vllm/envs.py Outdated
Comment on lines +38 to +39
VLLM_ROCM_TQ_FLYDSL_DECODE: bool = False
VLLM_TQ_FLYDSL_WHT_BUTTERFLY: bool = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to add these environment flags? Is it because of:

  1. Apply rotation and LUT-based quantization to K, and use standard uniform quantization for V. In our analysis, keys are more quantization-sensitive than values.
  2. Use Walsh-Hadamard rotation instead of random rotation. In practice, WHT is both more kernel-friendly and more accurate. TurboQuant+ also converges to the same choice for llama.cpp [2].
  3. Omit QJL in the default 4-bit path. In our experiments, QJL adds complexity and runtime overhead without delivering accuracy benefits in our experiments. This matches community consensus as some works drop QJL and D’Alberto et al. report similar conclusions [2, 3].

That is, VLLM_ROCM_TQ_FLYDSL_DECODE=1 is not numerically equivalent with the current other default backend?

Should we not have, similar to nvfp4/mxfp4 gemm/moe, say an automatic selection of turboquant backend, based on supported dtype, platform, etc?

So that e.g. flydsl is used by default as the turboquant implementation backend on gfx950 only, only in the specified case (4-bit, etc.).

If this implementation is not numerically equivalent, that is running --kv-cache-dtype turboquant_4bit_nc vs running VLLM_ROCM_TQ_FLYDSL_DECODE=1 --kv-cache-dtype turboquant_4bit_nc does not yield the same results, should this not be named differently?

Or --kv-cache-config backend=flydsl + warning?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are opt-in gates for a new, gfx950-only kernel, default-off keeps the existing TQ path unchanged/untouched. Numerically it's the same TurboQuant scheme, validated against an fp32 oracle and comparable to the Triton path within bf16 tolerance. Butterfly flag is more of a performance optimization flag which we could make default in the flydsl decode run to get rid of the opt-in flag. I agree the better long-term shape is a config field (--kv-cache-config backend=flydsl) and/or auto-selection on gfx950 rather than env vars. Any other suggestions on this before I make this change?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially wondering why this would be opt-in, if this is strictly better than current turboquant implementation and numerically equivalent.

Why not make it default on gfx950?

Comment on lines +103 to +107
if not _flydsl_v4_available():
logger.warning(
"VLLM_ROCM_TQ_FLYDSL_DECODE=1 but FlyDSL is unavailable; the decode path "
"will fall back to the SoA Triton v3 kernel."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is an opt-in and unavailable, an error should be raised.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, now raises ValueError at init if VLLM_ROCM_TQ_FLYDSL_DECODE=1 but FlyDSL isn't importable. The per-layer fallback to Triton for FlyDSL-ineligible layers (GQA/head/dtype) stays, since no single kernel covers every layer.

Comment on lines +1024 to +1054
else:
_tq_full_dequant_kv[grid](
kv_cache,
block_table,
centroids,
k_cached,
v_cached,
k_cached.stride(0),
k_cached.stride(1),
k_cached.stride(2),
v_cached.stride(0),
v_cached.stride(1),
v_cached.stride(2),
kv_cache.stride(0),
kv_cache.stride(1),
kv_cache.stride(2),
block_table.stride(0),
HEAD_DIM=D,
BLOCK_SIZE=block_size,
NUM_KV_HEADS=Hk,
MSE_BYTES=mse_bytes,
KPS=self.tq_config.key_packed_size,
VQB=self.tq_config.effective_value_quant_bits,
VAL_DATA_BYTES=val_data_bytes,
MSE_BITS=self.tq_config.key_mse_bits,
KEY_FP8=1 if self.tq_config.key_fp8 else 0,
BLOCK_D=BLOCK_D,
NORM_CORRECTION=1 if self.tq_config.norm_correction else 0,
FP8_E4B15=_use_fp8_e4b15(device.index or 0),
num_warps=4,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not inherit from TurboQuantAttentionImpl instead of modifying it with if/else branches?

And have a stateful dispatch to the correct TurboQuantAttentionImpl class/subclass?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, makes sense. So basically, keep TurboQuantAttentionImpl as the base (default Triton path untouched) and add a FlyDSLTurboQuantAttentionImpl subclass that overrides store/continuation/dequant/decode, so the if VLLM_ROCM_TQ_FLYDSL_DECODE branches disappear. I could do that once we have a decision on the env variable versus config field, maybe a follow up RFC/PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just for readability purposes / easier review, yes

Comment on lines +1193 to +1226
logger.warning_once(
"TurboQuant v4 ineligible (key_fp8=%s mse_bits=%s vqb=%s "
"head_size=%s num_kv_groups=%s sinks=%s) -> SoA Triton v3",
self.tq_config.key_fp8,
self.tq_config.key_mse_bits,
self.tq_config.effective_value_quant_bits,
self.head_size,
self.num_kv_groups,
self.sinks is not None,
)
return self._dispatch_decode_v3(
query=query,
kv_cache=kv_cache,
block_table=attn_metadata.block_table,
seq_lens=attn_metadata.seq_lens,
Pi=Pi,
centroids=centroids,
scale=self.scale,
mse_bits=self.tq_config.key_mse_bits,
key_packed_size=self.tq_config.key_packed_size,
value_quant_bits=self.tq_config.effective_value_quant_bits,
value_packed_size=self.tq_config.value_packed_size,
max_seq_len=attn_metadata.max_seq_len,
key_fp8=self.tq_config.key_fp8,
norm_correction=self.tq_config.norm_correction,
PiT=PiT,
mid_o_buf=mid_o_buf,
output_buf=output_buf,
lse_buf=lse_buf,
buf_holder=layer,
max_num_kv_splits=self.max_num_kv_splits,
sinks=self.sinks,
sliding_window=self.sliding_window,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it slightly counter-intuitive that VLLM_ROCM_TQ_FLYDSL_DECODE=1 may dispatch on a triton kernel.

nit: given that this v4_eligible check can be done ahead of time, I would also have a subclass that overrides _decode_attention in case the flydsl kernel does not support the problem shape.

Generally - would it not make sense to only integrate one the best of flydsl/triton implementation? Instead of having the two under the same VLLM_ROCM_TQ_FLYDSL_DECODE=1 name?

Or alternatively, e.g. have a single entrypoint aiter that dispatches on the best supported available kernel for the problem shape?

e.g. as https://github.com/ROCm/aiter/blob/main/aiter/configs/bf16_tuned_gemm.csv

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed the flag naming is misleading, but we can't ship only the best of the two: FlyDSL is a specialized fast path (gfx950, head_size=128, GQA {6,8,16}, 4-bit MSE keys), Triton v3 is the general fallback for every other shape/arch with SOA storage so both need to stay in some way, maybe another name? Not entirely sure about the aiter alternative, but can look into it.

Comment on lines +102 to +103
use_hw_v_transpose: bool = False,
tile_groups_per_partition: int = 1,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No hyperparameter tuning required here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use_hw_v_transpose isn't tuned, False is builder default. At launch it's set automatically from the GPU arch (_hw_tr_enabled() → True on gfx950, False elsewhere), so it turns on only where the hardware supports it. num_partitions and tile_groups_per_partition are computed automatically from the sequence length (same as Triton path), no autotuning needed here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

import torch

from vllm.platforms import current_platform
from vllm.platforms.rocm import on_gfx950

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import will fail on other platform, we have to guard with current_platform.is_rocm()

if current_platform.is_rocm():
    from vllm.platforms.rocm import on_gfx950
    ...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, will fix.

@@ -367,12 +367,16 @@ def rocm_aiter_fused_experts(
# `rocm_aiter_ops.shuffle_weight_a16w4` in `oracle/mxfp4.py`,
# which always sets `is_guinterleave=True`.
# Hence, we pass in GateMode.INTERLEAVE to match the weight shuffling.
from aiter.ops.flydsl.moe_common import GateMode

# Import ``GateMode`` lazily: some aiter builds lack ``aiter.ops.flydsl``,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are keeping support of a specific version of AITER on vLLM, we do not guarantee backwards nor forward version compatibility to avoid adding complex layer of code checks.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So please use the AITER shipped on upstream docker image.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted, using AITER shipped on upstream.

@@ -0,0 +1,143 @@
# FlyDSL kernels for TurboQuant (vendored)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this file before final merge.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, will remove.

Comment thread vllm/_aiter_ops.py Outdated
@@ -1843,6 +1833,23 @@ def fused_moe_supports_gate_mode(cls) -> bool:

return "gate_mode" in inspect.signature(fused_moe).parameters

@classmethod

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only support specific version of AITER, we do not guarantee backwards and forward compatibility.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, fixed

Comment thread vllm/envs.py Outdated
# whole SoA TurboQuant pipeline (SoA store + FlyDSL decode + SoA-aware
# continuation + SoA Triton v3 fallback) is enabled. Default off keeps the
# upstream TurboQuant v1 path unchanged.
"VLLM_ROCM_TQ_FLYDSL_DECODE": lambda: os.environ.get(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not add flags for attention backend, all of those should go through python arguments system.

@@ -70,6 +75,56 @@
_CONTINUATION_DECODE_THRESHOLD = 128


# --------------------------------------------------------------------------- #
# FlyDSL TurboQuant decode (opt-in; AMD MI355X / gfx950). #

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move all of the flydsl into a turboquant_attn_flydsl.py .

we expect that --attention-backend turboquant_flydsl

@fxmarty-amd fxmarty-amd Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @mgoin Are we looking to contain turboquant related code within --attention-backend turboquant (as you discussed in vLLM SIG quantization call)?

If so, @mgoin suggested --attention-config.turboquant-backend=flydsl/xxx, is it acceptable @tjtanaa? There is a precedent for it with:

mla_prefill_backend: MLAPrefillBackendEnum | None = None
"""MLA prefill backend to use. If None, will be selected automatically.
Valid options: FLASH_ATTN (FA3/FA4), FLASHINFER, TRTLLM_RAGGED."""

Comment thread vllm/platforms/rocm.py
Comment on lines +556 to +575
# Only tolerate the mismatch for turboquant_* KV-cache layers:
# boundary layers keep the native dtype (served by the selected
# backend) while turboquant_* layers need TURBOQUANT, so no single
# --attention-backend can serve every layer. For any other dtype
# the explicit selection is genuinely invalid -> fail loud.
kv_dtype = attn_selector_config.kv_cache_dtype
if not (kv_dtype is not None and str(kv_dtype).startswith("turboquant")):
raise ValueError(
f"Selected backend {selected_backend} is not valid for "
f"this configuration. Reason: {invalid_reasons}"
)
# NOTE: pass a str (not the list) -- info_once hashes its args.
logger.info_once(
"Selected backend %s is incompatible with this turboquant "
"layer (%s); using the auto-selected per-layer backend. "
"Reason: %s",
selected_backend.name,
attn_selector_config.attn_type,
str(invalid_reasons),
)

@fxmarty-amd fxmarty-amd Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So as discussed offline, I understand this is in order to have --kv-cache-dtype turboquant_4bit_nc --attention-backend xyz not crashing, that is, being able to select the attention backend of non-turboquant layers (e.g. first and last skipped layers), e.g. in case vLLM's default is not the desired one or the best performing one.

If so, the behavior should be identical on all platforms I believe (allow --attention-backend xyz on nvidia as well, and verify it works).

@mgoin @tjtanaa Is --kv-cache-dtype turboquant_4bit_nc --attention-backend ROCM_AITER_UNIFIED_ATTN acceptable in order to select non-default attention backend for non-turboquant layers?

Or should --attention-backend expect only turboquant for --kv-cache-dtype turboquant_* cases, and there should be a configuration that goes along --kv-cache-dtype-skip-layer to precise the attention backend of non-turboquant layers?

Should this be done in an other PR or here is OK?

@mergify mergify Bot added the quantization label Jul 23, 2026
@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 31, 2026
@mergify

mergify Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Hi @aditi-amd, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

@mergify mergify Bot added llama Related to Llama models multi-modality Related to multi-modality (#4194) mistral Related to Mistral models new-model Requests to new models labels Aug 4, 2026
aditi-amd and others added 11 commits August 4, 2026 20:52
Add the FlyDSL-based TurboQuant 4-bit KV-cache decode kernel (v4) for
ROCm/MI355X, with butterfly (WHT) transform support and a GQA-6 sibling
kernel for MiniMax-class models. Gated behind VLLM_ROCM_TQ_FLYDSL_DECODE
with graceful fallback to the Triton v3 path when FlyDSL is unavailable.
Also guards the ROCm aiter fused-MoE path for older aiter builds:
capability-probe swiglu_limit/gate_mode kwargs and lazily import
aiter.ops.flydsl only in the branches that need it.
Signed-off-by: aditi <aditi.rana@amd.com>

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…t tests

Reduce the user-facing env surface of the FlyDSL TurboQuant 4-bit KV decode
kernel from 7 flags to 2 (VLLM_ROCM_TQ_FLYDSL_DECODE and
VLLM_TQ_FLYDSL_WHT_BUTTERFLY). Internalize tuning defaults, auto-detect the
hardware-transpose path, and drop the deprecated tuning/debug env vars and
sys.path bootstraps from the kernel modules.
Add a parametrized pytest (tests/kernels/turboquant_v4/) that checks the
FlyDSL v4 decode output against a pure-PyTorch fp32 attention oracle across
batch sizes, sequence lengths, GQA ratios, and block sizes.
Also apply ruff lint/format fixes (import ordering, contextlib.suppress,
line-length) with no behavioral changes.
Signed-off-by: aditi <aditi.rana@amd.com>

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…aming

Signed-off-by: aditi <aditi.rana@amd.com>

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…e otherwise

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
… vendored README

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…rfly off, clean up kernel comments

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
…d WHT butterfly

Replace VLLM_ROCM_TQ_FLYDSL_DECODE with capability gating (on_gfx950 +
importable) and remove the hardwired-off WHT butterfly path from the launcher
and both decode kernels.

Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
Signed-off-by: root <root@smci355-ccs-aus-m02-09.cs-aus.dcgpu>
@namgyu-yun

namgyu-yun commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

For background on the TurboQuant algorithm and its role in agentic vLLM serving, see our blog write-up https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html

nit: This link at PR description is broken. Could you please update to https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html ?

@aditi-amd

Copy link
Copy Markdown
Contributor Author

For background on the TurboQuant algorithm and its role in agentic vLLM serving, see our blog write-up https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html

nit: This link at PR description is broken. Could you please update to https://rocm.blogs.amd.com/artificial-intelligence/turboquant-vllm-agentic/README.html ?

Thanks for catching that! I've updated the URL in the PR description.

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@AndreasKaratzas

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #82799 for commit 38dee557f369.

@AndreasKaratzas

Copy link
Copy Markdown
Member

@mgoin I think these failures are known.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build cpu Related to CPU backends deepseek Related to DeepSeek models documentation Improvements or additions to documentation frontend gpt-oss Related to GPT-OSS models intel-gpu Related to Intel GPU k3 kimi kv-connector llama Related to Llama models mistral Related to Mistral models mrv2 Model Runner V2 specific multi-modality Related to multi-modality (#4194) new-model Requests to new models nvidia performance Performance-related issues quantization qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm rust speculative-decoding structured-output tool-calling v1

Projects

Status: Done
Status: Done
Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants