Skip to content

UltraQuant MTP verify and FlyDSL T2 decode - #67327

Draft
amd-preetesh wants to merge 1 commit into
vllm-project:mainfrom
amd-preetesh:uq-mtp-pdv-hoist-k
Draft

amd-preetesh wants to merge 1 commit into
vllm-project:mainfrom
amd-preetesh:uq-mtp-pdv-hoist-k

Conversation

@amd-preetesh

@amd-preetesh amd-preetesh commented Nov 12, 2026 •

Copy link
Copy Markdown

Purpose

UltraQuant decode already ships. MTP verification did not reach it: on current main the TurboQuant metadata builder opts out of spec-as-decode, so a k-token verify is classified as a prefill and never enters the UltraQuant decode kernel.

This PR does two things.

  1. Keep an MTP verify on the UltraQuant decode path (supports_spec_as_decode=True).
  2. Add an opt-in FlyDSL verify kernel, FlyDSL Type 2 (T2, selected with VLLM_ULTRAQUANT_PDV_EXPERIMENT=hoist_k). The shipping single-token FlyDSL kernel scores one query row per request. FlyDSL T2 reads each KV tile once and scores all 2–8 verify tokens with that tile. Qwen3.8-27B is query group size 6, head size 256, MTP k=5, so a verify step is 6 tokens.

The 4-bit cache format is unchanged: FP4 E2M1 codes, UE8M0 per-group scales, Hadamard-rotated keys. See UltraQuant: 4-bit KV Caching for Context-Heavy Agents.

Design

  • UltraQuantMetadataBuilder subclasses TurboQuantMetadataBuilder. The parent sets supports_spec_as_decode=False. UltraQuant calls _init_reorder_batch_threshold(1, supports_spec_as_decode=True) after super().__init__, so a speculative verify stays a decode.
  • The shipping FlyDSL kernel is single-token. When spec-decode is on and FlyDSL T2 is off, _split_multi_query_decode rewrites a multi-token verify as one causal decode row per token. Row j of a request with q verify tokens attends to the first seq_len - q + 1 + j cached tokens.
  • When VLLM_ULTRAQUANT_PDV_EXPERIMENT=hoist_k, the query group size is 6, and every request has the same verify width in 2..8, the split is skipped. flydsl_ultraquant_decode_attention launches pdv_flydsl_decode_hd256_hoist_k with query_start_loc and one grid row per request.
  • The pre-warmed decode workspace is _max_capture_batch_size() * max_decode_query_len when a speculative config is set, so a verify step of 1 + k query rows fits the locked buffer.
  • Unset the env and the shipping FlyDSL kernel remains the default. VLLM_ULTRAQUANT_V3=1 also keeps T2 off.

Test plan

  • Accuracy: GPQA Diamond (198) and GSM8K (1319), seeds 7, 8, and 9. Sampling is the model-card setting (temperature 1.0, top_p 0.95, top_k 20, thinking on).
  • Performance: SemiAnalysis InferenceX replay, Qwen3.8-27B, TP=1, MTP k=5, prefix caching on, 300s at concurrency 8, 16, 24, 32, 40.

These accuracy and AgentX numbers were measured on the ROCm stack this PR ports (v0.1.dev21857+g536c4771c), not on a build of this branch against current main. This branch has not been GPU-tested.

Usage

Shipping UltraQuant MTP. The verify is split into single-token decode rows:

vllm serve /path/to/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --kv-cache-dtype ultraquant_4bit \
  --block-size 64 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

FlyDSL T2, one KV read for the whole verify block. Leave the env unset to keep the shipping kernel.

VLLM_ULTRAQUANT_PDV_EXPERIMENT=hoist_k \
vllm serve /path/to/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --kv-cache-dtype ultraquant_4bit \
  --block-size 64 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Requires query group size 6, head size 256, and a uniform verify width of 2–8 tokens. Qwen3.8-27B MTP k=5 verifies 6 tokens.

Test results

Accuracy

GPQA Diamond (198) and GSM8K (1319). Seeds 7 / 8 / 9. The mean is the unweighted average of the three seeds. UltraQuant stays with FP8. The arms trade places across seeds.

Arm GPQA 7 / 8 / 9 GPQA mean GSM8K 7 / 8 / 9 GSM8K mean
FP8 MTP 89.39 / 91.92 / 89.90 90.40 97.35 / 97.42 / 97.65 97.47
UQ prod FlyDSL 88.38 / 88.38 / 89.90 88.89 97.57 / 97.50 / 96.89 97.32
UQ FlyDSL T2 88.89 / 89.90 / 90.40 89.73 97.50 / 97.35 / 97.19 97.35

Performance, AgentX

SemiAnalysis InferenceX replay. Qwen3.8-27B, TP=1, MTP k=5, thinking on, prefix caching on, 300s windows. Throughput is total_output_tokens / benchmark_duration.

xychart-beta
    title "AgentX output throughput"
    x-axis "Concurrency" [c8, c16, c24, c32, c40]
    y-axis "Output tokens/s" 0 --> 200
    line "FP8 MTP" [158.2, 111.8, 139.2, 161.3, 76.8]
    line "UQ prod FlyDSL" [123.5, 113.3, 145.0, 144.6, 82.9]
    line "UQ FlyDSL T2" [158.5, 125.9, 158.4, 180.9, 129.4]
Loading
Arm c8 c16 c24 c32 c40
FP8 MTP 158.2 111.8 139.2 161.3 76.8
UQ prod FlyDSL 123.5 113.3 145.0 144.6 82.9
UQ FlyDSL T2 158.5 125.9 158.4 180.9 129.4

Inter-token latency p50, milliseconds, same cells.

Concurrency FP8 tok/s FP8 ITL p50 T2 tok/s T2 ITL p50 Prod tok/s Prod ITL p50
8 158.2 7.2 158.5 6.6 123.5 8.6
16 111.8 8.3 125.9 8.4 113.3 10.4
24 139.2 9.1 158.4 9.0 145.0 11.6
32 161.3 14.0 180.9 12.6 144.6 19.2
40 76.8 25.7 129.4 24.5 82.9 35.2

At c32 the acceptance lengths were 3.33 for FP8 MTP, 3.45 for UQ prod FlyDSL, and 3.68 for FlyDSL T2.

Notes

  • AgentX servers ran with --enable-prefix-caching.
  • FlyDSL T2 is off unless VLLM_ULTRAQUANT_PDV_EXPERIMENT=hoist_k. The env value stays hoist_k. T2 is the name used in this description.
  • Absolute tok/s should not be compared with the 4-bit KV PR's MI355X, TP=8, 262K-token Pareto. That run is a different model shape, a different parallel width, and no MTP. The relative claim here is MTP verify on UltraQuant, and T2 against the shipping FlyDSL kernel, on Qwen3.8-27B TP=1.

References

UltraQuant decode already ships, but MTP verify was classified as
prefill. Opt the builder into spec-as-decode and score a k-token
verify on the decode path. PDV 1.1 hoist_k is opt-in and reads each
KV tile once for the whole verify block.

Signed-off-by: amd-preetesh <330108521+amd-preetesh@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

@mergify mergify Bot added the quantization label Nov 12, 2026
@amd-preetesh amd-preetesh changed the title UltraQuant MTP verify and PDV 1.1 hoist_k UltraQuant MTP verify and FlyDSL Type 2 (T2) Nov 12, 2026
@amd-preetesh amd-preetesh changed the title UltraQuant MTP verify and FlyDSL Type 2 (T2) UltraQuant MTP verify and FlyDSL T2 decode Nov 12, 2026

@aditi-amd aditi-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks keeping MTP verify on the UltraQuant decode path shows good results but needs a few updates and should be the default flow.

For the default path to work on main, a few things come first. The kernel currently can't import on main's aiter pin, and it's forked from the UltraQuant kernel as it was before #61649 #61649, so it brings back the legacy APIs and duplicates the shipping kernel. There are two correctness issues inline, in the verify-width check and in how pure-verify batches are routed. And the new kernel doesn't have tests yet.

Small things: with the env var gone, the kernel could use one descriptive name, the commit says "PDV 1.1 hoist_k", the description says "T2", and the env value is hoist_k. The title needs the [Type] [Hardware] scope: title format to pass the check, e.g. [Feat] [AMD] ultraquant: Add MTP verify decode. And the numbers should be re-run on this branch against main before it's marked ready. For the format, please use the same as KV8, we dont want to introduce new names as we can just put the kernel for MTP under a condition (q_length >1 or soemthing similar used for KV8).


import flydsl.compiler as flyc
import flydsl.expr as fx
from aiter.ops.flydsl.kernels import buffer_ops, vector

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aiter.ops.flydsl.kernels.vector doesn't exist in aiter v0.1.29 (main's current pin) or 0.1.24.post1, so this module can't be imported on main. Today that only affects people who set the env var. If this becomes the default MTP verify kernel (see my comment on ultraquant_attn.py:304), every UltraQuant MTP deployment would hit ModuleNotFoundError on its first verify step. #61649 moved the UltraQuant kernel off aiter.ops.flydsl for this reason. Could this use the public FlyDSL API instead? Look at the PR #61649

# The build-local is handed to the launcher via a module attribute set AFTER
# the kernel is built (see end of function); that assignment is safe because
# no @flyc.jit function references the module global anymore.
allocator = SmemAllocator(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it was forked from the UltraQuant kernel before #61649 merged (Oct 23). That PR replaced SmemAllocator/SmemPtr, buffer_ops, raw scf, raw rocdl.mfma_*, magic s_waitcnt and arith/_mlir types with fx.SharedAllocator, buffer_view and fx.gemm after maintainer feedback on low-level APIs. Could you rebase onto main's ultraquant_decode_hd256.py? You'll also need to drop from future import annotations (line 20), because it breaks @fx.struct.

# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# PDV FlyDSL hoist-K experiment — gfx950 / CDNA4
#
# Copy of the shipping PDV kernel. The shipping file is not modified.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a 1205-line copy of the shipping kernel. On #61649, @okorzh-amd asked us to merge the TurboQuant GQA-6 copy back into the main kernel "so we don't need to fix the bugs twice". If this becomes the default verify kernel, could it live in ultraquant_decode_hd256.py as a num_verify_tokens compile-time parameter, with =1 giving today's single-token decode? Then there's only one kernel to maintain.

# T verify tokens in one pass. NT is how many 16-row instructions that is.
NUM_Q = int(num_verify_tokens)
assert 1 <= NUM_Q <= 8, f"PDV verifies 1..8 tokens; got {NUM_Q}"
PAIR_TWO_WAVE = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PAIR_TWO_WAVE, ROWPACK and PAIRS_PER_REQ are hardcoded, but the branches for the unused variants are still in the file, along with the identity helpers _kvi32/_kvi64/_sci (line 385). The header (line 14) also refers to production, 2-wave and row-pack sources that aren't in this PR. Can these be removed?

# allocation identical to the base decode for buffer reuse).
num_rows = query.shape[0]
num_reqs = int(attn_metadata.seq_lens.shape[0])
uniform = num_reqs > 0 and num_rows % num_reqs == 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

num_rows % num_reqs == 0 doesn't guarantee equal widths. With query lengths [2, 4], this gives qlen=3, and the kernel scores only NUM_Q rows per request, so the second request's 4th token is never written. The kernel already reads per-request bounds from query_start_loc and masks row < q_end. Could NUM_Q come from attn_metadata.max_query_len, and the uniformity requirement be dropped?

num_reqs = int(attn_metadata.seq_lens.shape[0])
uniform = num_reqs > 0 and num_rows % num_reqs == 0
qlen = num_rows // num_reqs if uniform else 1
hoist = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the env var, eligibility here could work like _flydsl_decode_eligible: compute the static parts (GQA 6, head size 256, FlyDSL available) once at init, and check only the verify width per batch. Batches that don't qualify can keep falling back to _split_multi_query_decode, which is the right behavior. A logger.info_once naming the active path would make it easy to confirm from serve logs.

)


def test_ragged_verify_batch_matches_causal_reference():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This covers _split_multi_query_decode through the Triton fallback, but nothing exercises the new FlyDSL kernel, the metadata builder, or the routing. That matters more if this becomes the default verify path. Could you add a kernel-vs-reference test (ragged widths, padded CUDA-graph rows, a >4 GiB cache) and one checking that a pure-verify batch routes as decode?

expected.float().reshape(-1),
dim=0,
).item()
assert cos > 0.98, (req, j, cos)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cos > 0.98 is looser than the 0.995 bar in test_flydsl_decode_d256.py. Could this match it? It's also worth running the new tests in a loop to rule out flakiness, AndreasKaratzas asked for that on #61649. Also please make sure all the ultraquant unit tests still pass

@@ -0,0 +1,35 @@
# SPDX-License-Identifier: Apache-2.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the kernel becomes the default (see my comment on ultraquant_attn.py:304), this selector isn't needed. _get_pdv_kernel can import the kernel module directly, the way _get_kernel does.

# 2 = nt (non-temporal: stream, do not pollute)
# 18 = nt|sc1 (also bypass the far cache level)
# VLLM_ULTRAQUANT_KV_AUX selects the experiment. 0 is the shipping path.
_kv_aux_env = int(os.environ.get("VLLM_ULTRAQUANT_KV_AUX", "0") or "0")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main's kernel hardcodes _KV_AUX = 0. Could this do the same instead of reading VLLM_ULTRAQUANT_KV_AUX? The value is also missing from the _get_pdv_kernel cache key, so changing it wouldn't trigger a rebuild.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants