Repository navigation
[Feat] [MI355X] qwen35_a17b: Serve block-FP8 decode at TP4 and TP8 - #58
Conversation
Decode widths 1-8 on gfx950 stay on the new GDN and MoE kernels, while wider batches and the full-attention KV update stay on the standard path.
… and fused GDN, norm and GEMV launches The first version ran SGLang's ops between about 110 grid barriers per step and an eager torch GDN, so decode was 6x slower than the standard path. Each step now has two barriers, a GDN layer is two Triton launches (the block-FP8 in_proj_qkvz with the conv, then the gated delta), the residual add, Gemma norm and FP8 quant are one launch, and the decode GEMVs split K across the warps of a workgroup. Decode at ISL 8192 / OSL 1024 is 21%, 18% and 16% faster than the standard path at concurrency 1, 4 and 8.
…p4-mono # Conflicts: # .github/CODEOWNERS # sgl_amd_mono/__init__.py
…path under qwen35_a17b
…of up to 8 requests at TP4 A verify step of 1 to 8 requests x 4 draft tokens runs the monokernel layer loop. Each GDN layer runs the draft chain in three launches plus out_proj and writes the per-token conv and SSM state to SGLang scratch. SGLang accept step commits from that scratch. The draft model, prefill and wider steps keep the standard forward. Adds the model README, the MTP server tests and the verify kernel tests.
…-tp4-mono-v2 # Conflicts: # .github/CODEOWNERS # README.md # sgl_amd_mono/__init__.py
… path SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in sgl-project/sglang#45389. Read it there first and fall back to sglang.srt.monokernels, as the other specs on main do, so the contract test passes on both SGLang versions.
The merge push reached the branch, but GitHub dropped the push event during its commit_refs_rules failure, so the pull request kept the old head.
29a40c2 to
c8f3265
Compare
GPQA DiamondQwen3.5-397B-A17B-FP8, TP4, MI355X, SGLang
Paired on the same 792 calls: 31 right only on the standard path, 25 right only on the monokernel. McNemar exact p = 0.504. Neither arm hit the token cap, left an answer unparsed, or returned an error. The monokernel arm needed one local compatibility change that is not on the branch yet. This SGLang moved SWE-bench Verified is not in this update. |
…-tp4-mono-v2 Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # README.md
Unit tests
These are the contract and forward-drift checks for Kimi and Muse Spark against an older SGLang than the one |
|
@yichiche Here are the test results. The qwen35 unit tests all pass at 3b9fec0 (merge of this branch with main at 56669ca). The run uses the image
This is the pytest output: |
|
Motivation
What
Add
sgl_amd_mono.qwen35_a17b, so that decode steps of 1 to 8 rows of Qwen3.5-397B-A17B-FP8 (128x128 block FP8) at TP4 or TP8 on MI355X run a fused decode forward under--monokernel-backend amd. The block-FP8 kernels live underqwen35_a17b/fp8. The family spec is the one registry entry forQwen3_5MoeForConditionalGeneration; it dispatches block FP8 and refuses MXFP4 until that precision moves underqwen35_a17b/mxfp4. At TP4, ISL 8192 / OSL 1024, output throughput is 21%, 18% and 16% above SGLang's standard path at concurrency 1, 4 and 8.Why
At these widths a standard decode step is bound by kernel count and per-kernel latency, not by bandwidth. At 8 rows a step is 983 kernels on each rank. A GDN layer runs seven of them for its projections and core (the block-scale GEMM,
in_proj_ba, a split, the conv, the gated delta, the gated norm and a quant), 42 µs a layer, and the preshuffled block-scale GEMM walks all of K in one workgroup, soin_proj_qkvz(K = 4096) takes about 11 µs whatever its N. This package runs a GDN layer's projections, conv, gated delta, gated norm and the FP8 quant ofout_proj's input as two Triton launches, takes each all-reduce, residual add, Gemma norm and FP8 quant as one aiter launch, and runs the decode GEMVs on the same preshuffled weights with K split across the warps of a workgroup. Attention (SGLang's q/k norm, rope, aiter paged attention and the KV write) and the MoE (SGLang's router and block-FP8 experts) keep SGLang's kernels.Modifications
qwen35_a17b, registered forQwen3_5MoeForConditionalGeneration. Dynamic 128x128 block FP8 delegates toqwen35_a17b/fp8. Any other quant method, including MXFP4, is refused by name. The small-M block-FP8 import stand-in runs on the FP8 config and build path, not at package import, so a later MXFP4 dispatch does not apply it.qwen35_a17b_fp8on gfx950, widths 1 to 8,tp_sizes=(4, 8), bf16, aiter attention,--moe-runner-backendauto or aiter._buildselects the specialization fromparallel.tp_size. TP4 keeps the previous local shapes, including projection K 2048. TP8 uses GDN K/V heads 2/8, QKVZ 2560, BA 16, conv 1536, projection K 1024, MoE intermediate 128, W13 N 256, and full-attention QKV N 2560. A checkpoint that is not 128x128 block FP8 (MXFP4 or Quark, per-channel FP8), a TP other than 4 or 8, a plain RMSNorm or a projection with a bias is refused with its reason. The forward opens each step with two guarded peer barriers, then runs each layer as K1 and K2. Wider steps, embedding requests and image inputs keep the standard forward.gfx950/layout.pyconstants, unchanged.block_fp8_gemv_wg, a Triton block-FP8 GEMV on aiter's (16, 16)-preshuffled weights. A workgroup owns 16 output columns and splits K into 8 slices over its 4 warps: one batched FP8 dot per 128-wide group, scaled by the activation and weight block scales, and the slices are added in fp32. Shapes it does not fit take aiter's block-scale GEMM.in_proj_qkvz(the GEMV above) with the causal conv and SiLU on the conv channels, storing z. The second is the bf16in_proj_ba, the gated delta rule on the mamba state, the gated RMSNorm and the 1x128 FP8 quant ofout_proj's input. The conv window, the recurrence and every rounding follow SGLang's decode kernels.gate_quant, the attention output gateattn * sigmoid(gate)with the FP8 quant ofo_proj's input. Both quantize as aiter's per-1x128 quant does, bit for bit._prepare_qkv_gatewithqkv_projfed the fused FP8 pair, thenattn.out_projando_projrun the GEMV above.SGLANG_AMD_MONO_CHECK=1runs SGLang's own layer next to each K1 and K2 in eager steps and logs cosine and relative error againstNUMERICS.QWEN35_FP8_TP=4or8), andsgl_amd_mono/qwen35_a17b/ @yichiche.Accuracy Tests
MI355X x 4,
rocm/sgl-dev:v0.5.22-rocm720-mi35x-20261021(ROCm 7.2.0, torch 2.9.1, Triton 3.7.0, FlyDSL 0.4.0), SGLangmainat sgl-project/sglang@8e3e2980db, aiter285579b621,Qwen/Qwen3.5-397B-A17B-FP8, TP4. Both servers use the flags oftest/e2e/qwen35_a17b_fp8_server.py; the monokernel server adds--monokernel-backend amd.GSM8K uses
run_sgl_eval, 200 questions, 12 threads so the batch steps past width 8 and back, thinking on, temperature 1.0, top_p 0.95, seed 42. The monokernel run cut no answer at the 32768-token limit; the standard path cut 7.GPQA Diamond follows openai/simple-evals: 198 questions,
Random(0)choice order, 4 repeats (792 calls), thinking mode temperature 0.6, top_p 0.95, top_k 20, min_p 0, max tokens 81920, concurrency 8. The model card reports 88.4. Paired on the same 792 calls: 31 right only on the standard path, 25 only on the monokernel, McNemar exact p = 0.504. Neither arm hit the token cap, left an answer unparsed, or returned an error. This run is on SGLang9691e711cdwith--enable-amd-monokernel. The monokernel arm used a local fix that is not on the branch yet:mamba2_layer_cachenow lives onMambaPool, and the kernel reads it throughmamba_map.SWE-bench Verified uses mini-swe-agent 2.4.6 and the SWE-bench harness on
SWE-bench/SWE-bench_Verified. Both arms ran the same 59 instances, the images that fit on the root disk after 3 astropy images returned 404. Sampling follows the model-card thinking mode: temperature 0.6, top_p 0.95, top_k 20, min_p 0, max tokens 32768, step limit 250, 4 workers. The model card reports 76.4 on the full verified set. On these 59 the standard arm resolved 40, left 17 unresolved, and had 2 harness errors (django__django-12209,django__django-16429), with 0 empty patches. The monokernel arm resolved 44, left 15 unresolved, and had 0 errors and 0 empty patches. Paired on the same 59: 39 resolved on both, 1 only on the standard path, 5 only on the monokernel (the 2 standard harness errors are in those 5), McNemar exact p = 0.219. Same SGLang9691e711cdand the same local mamba-cache fix on the monokernel arm.Speed Tests and Profiling
Same stack and flags as above,
--max-running-requests 16and decode graphs at batch sizes 1 to 8 on both servers.python3 -m sglang.bench_serving --backend sglang --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1 --num-prompts <10 x c> --max-concurrency <c> --warmup-requests 2 --seed 1. TPOT is the median per-user decode time.Median TTFT is within 1% at every concurrency, as prefill is SGLang's on both. At concurrency 16 most steps are wider than 8 rows and run SGLang's standard forward on both servers, eagerly, since no decode graph is captured above 8 rows; the hand-off costs nothing. With
--enable-aiter-allreduce-fusionon the standard path (193.78, 574.26 and 840.40 tok/s at concurrency 1, 4 and 8) the monokernel is +20.3%, +15.4% and +12.2%.One decode step at 8 rows, rank 0, kernel time from the torch profiler. Both servers decode the same 8 random 8192-token prompts.
in_proj_ba, split, conv, gated delta, gated norm, quant_proj_conv_kernel,_delta_kernelout_proj,qkv_proj,o_proj_wg_kernel_add_norm_quant_kernelat layer 0)_gate_quant_kernelE2E
The TP4 numbers above are unchanged and were measured before this layout move. After the move, on MI355X in
jacky-v0.5.20-rocm724-mi35x-qwen3.8-mxfp4-megakernel-1022, checkpoint/data/Qwen3.5-397B-A17B-FP8, flags oftest/e2e/qwen35_a17b_fp8_server.py:TestMonokernelCheck, GPUs 0-3, widths 1, 3 and 8): 1 passed in 85s, zero check violations.QWEN35_FP8_TP=8, GPUs 0-7, the same test): 1 passed in 94s, zero check violations. That covers GDN state, projections, MoE and the all-reduce at those widths.--tp 8. This is not the ISL 8192 / OSL 1024 sweep in the table above.On this SGLang the standard server does not load the monokernel spec, so it does not publish the small-M import stand-in. The standard arm of this TP8 run applied the same
_allow_block_fp8_linearstand-in through a process-localsitecustomizethat is not in the repo. The monokernel publishes that stand-in in the worker. Without it, the standard server dies oncannot import name '_check'fromsmallm_moe_gfx950before the first generate.Not covered
qwen35_a17b/mxfp4, so the two precisions share one architecture registration.smallm_moe_gfx950no longer defines, so every block-FP8 linear fails to import. When that is the case, the spec publishes stand-ins and setsSGLANG_ROCM_SMALLM_FP8_BS=0before the import, so the linears take aiter's block-scale GEMM. The standard-path server in these runs used the same stand-ins, so both sides compare the same prefill and linears.Checklist
pre-commit run --all-filespasses.test/cover the change..github/CODEOWNERSthat names its owner.Unit tests
The run uses the image
jacky-v0.5.23-rocm724-mi35x-wan2.2-1031, SGLang9691e711cd, on CPU, withPYTHONPATHset to this branch, from/home/yichiche/sgl-amd-mono-pr58:-k qwen35This is the pytest output: