Skip to content

[Feat] [MI355X] qwen35_a17b: Serve block-FP8 decode at TP4 and TP8 - #58

Merged
yichiche merged 11 commits into
sgl-project:mainfrom
yichiche:amd/qwen35-397b-fp8-tp4-mono-v2
Nov 16, 2026
Merged

yichiche merged 11 commits into
sgl-project:mainfrom
yichiche:amd/qwen35-397b-fp8-tp4-mono-v2

Conversation

@yichiche

@yichiche yichiche commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

What
Add sgl_amd_mono.qwen35_a17b, so that decode steps of 1 to 8 rows of Qwen3.5-397B-A17B-FP8 (128x128 block FP8) at TP4 or TP8 on MI355X run a fused decode forward under --monokernel-backend amd. The block-FP8 kernels live under qwen35_a17b/fp8. The family spec is the one registry entry for Qwen3_5MoeForConditionalGeneration; it dispatches block FP8 and refuses MXFP4 until that precision moves under qwen35_a17b/mxfp4. At TP4, ISL 8192 / OSL 1024, output throughput is 21%, 18% and 16% above SGLang's standard path at concurrency 1, 4 and 8.

Why
At these widths a standard decode step is bound by kernel count and per-kernel latency, not by bandwidth. At 8 rows a step is 983 kernels on each rank. A GDN layer runs seven of them for its projections and core (the block-scale GEMM, in_proj_ba, a split, the conv, the gated delta, the gated norm and a quant), 42 µs a layer, and the preshuffled block-scale GEMM walks all of K in one workgroup, so in_proj_qkvz (K = 4096) takes about 11 µs whatever its N. This package runs a GDN layer's projections, conv, gated delta, gated norm and the FP8 quant of out_proj's input as two Triton launches, takes each all-reduce, residual add, Gemma norm and FP8 quant as one aiter launch, and runs the decode GEMVs on the same preshuffled weights with K split across the warps of a workgroup. Attention (SGLang's q/k norm, rope, aiter paged attention and the KV write) and the MoE (SGLang's router and block-FP8 experts) keep SGLang's kernels.

Modifications

  • qwen35_a17b/spec.py: one family spec, qwen35_a17b, registered for Qwen3_5MoeForConditionalGeneration. Dynamic 128x128 block FP8 delegates to qwen35_a17b/fp8. Any other quant method, including MXFP4, is refused by name. The small-M block-FP8 import stand-in runs on the FP8 config and build path, not at package import, so a later MXFP4 dispatch does not apply it.
  • qwen35_a17b/fp8/spec.py: qwen35_a17b_fp8 on gfx950, widths 1 to 8, tp_sizes=(4, 8), bf16, aiter attention, --moe-runner-backend auto or aiter. _build selects the specialization from parallel.tp_size. TP4 keeps the previous local shapes, including projection K 2048. TP8 uses GDN K/V heads 2/8, QKVZ 2560, BA 16, conv 1536, projection K 1024, MoE intermediate 128, W13 N 256, and full-attention QKV N 2560. A checkpoint that is not 128x128 block FP8 (MXFP4 or Quark, per-channel FP8), a TP other than 4 or 8, a plain RMSNorm or a projection with a bias is refused with its reason. The forward opens each step with two guarded peer barriers, then runs each layer as K1 and K2. Wider steps, embedding requests and image inputs keep the standard forward.
  • qwen35_a17b/fp8/gfx950/shapes.py: the TP8 local shapes. TP4 is the existing gfx950/layout.py constants, unchanged.
  • qwen35_a17b/fp8/gfx950/gemv.py: block_fp8_gemv_wg, a Triton block-FP8 GEMV on aiter's (16, 16)-preshuffled weights. A workgroup owns 16 output columns and splits K into 8 slices over its 4 warps: one batched FP8 dot per 128-wide group, scaled by the activation and weight block scales, and the slices are added in fp32. Shapes it does not fit take aiter's block-scale GEMM.
  • qwen35_a17b/fp8/gfx950/gdn.py: a GDN layer in two launches. The first is in_proj_qkvz (the GEMV above) with the causal conv and SiLU on the conv channels, storing z. The second is the bf16 in_proj_ba, the gated delta rule on the mamba state, the gated RMSNorm and the 1x128 FP8 quant of out_proj's input. The conv window, the recurrence and every rounding follow SGLang's decode kernels.
  • qwen35_a17b/fp8/gfx950/norm.py: the residual add, Gemma RMSNorm and 1x128 FP8 quant in one launch (the first layer, or when the custom all-reduce is off), and gate_quant, the attention output gate attn * sigmoid(gate) with the FP8 quant of o_proj's input. Both quantize as aiter's per-1x128 quant does, bit for bit.
  • qwen35_a17b/fp8/gfx950/forward.py: K1 and K2. Each all-reduce, residual add and Gemma norm (with the quant ahead of a GEMV) is aiter's fused custom all-reduce. A full-attention layer runs SGLang's _prepare_qkv_gate with qkv_proj fed the fused FP8 pair, then attn. out_proj and o_proj run the GEMV above.
  • qwen35_a17b/fp8/gfx950/moe.py: SGLang's top-k (its gfx950 small-M router when the shared expert is fused) and block-FP8 experts, so the routed ids are the standard block's.
  • qwen35_a17b/fp8/gfx950/barrier.py: the two FlyDSL peer barriers. Kernel names and peer buffers include the TP size, and each barrier polls that many ranks. A stalled rank stops every rank at the deadline with a store to the null page before the step runs any layer.
  • qwen35_a17b/fp8/check.py: SGLANG_AMD_MONO_CHECK=1 runs SGLang's own layer next to each K1 and K2 in eager steps and logs cosine and relative error against NUMERICS.
  • test/unit/test_qwen35_a17b_fp8_spec.py: registration, device limits and each refusal, on CPU.
  • test/e2e/test_qwen35_a17b_fp8_server.py: GSM8K past width 8, decode against prefill logprobs at widths 2, 3, 7 and 8, check mode at widths 1, 3 and 8, and fault injection.
  • README and CODEOWNERS: the model's row (TP 4 and 8), launch command and test command (QWEN35_FP8_TP=4 or 8), and sgl_amd_mono/qwen35_a17b/ @yichiche.

Accuracy Tests

MI355X x 4, rocm/sgl-dev:v0.5.22-rocm720-mi35x-20261021 (ROCm 7.2.0, torch 2.9.1, Triton 3.7.0, FlyDSL 0.4.0), SGLang main at sgl-project/sglang@8e3e2980db, aiter 285579b621, Qwen/Qwen3.5-397B-A17B-FP8, TP4. Both servers use the flags of test/e2e/qwen35_a17b_fp8_server.py; the monokernel server adds --monokernel-backend amd.

GSM8K uses run_sgl_eval, 200 questions, 12 threads so the batch steps past width 8 and back, thinking on, temperature 1.0, top_p 0.95, seed 42. The monokernel run cut no answer at the 32768-token limit; the standard path cut 7.

GPQA Diamond follows openai/simple-evals: 198 questions, Random(0) choice order, 4 repeats (792 calls), thinking mode temperature 0.6, top_p 0.95, top_k 20, min_p 0, max tokens 81920, concurrency 8. The model card reports 88.4. Paired on the same 792 calls: 31 right only on the standard path, 25 only on the monokernel, McNemar exact p = 0.504. Neither arm hit the token cap, left an answer unparsed, or returned an error. This run is on SGLang 9691e711cd with --enable-amd-monokernel. The monokernel arm used a local fix that is not on the branch yet: mamba2_layer_cache now lives on MambaPool, and the kernel reads it through mamba_map.

SWE-bench Verified uses mini-swe-agent 2.4.6 and the SWE-bench harness on SWE-bench/SWE-bench_Verified. Both arms ran the same 59 instances, the images that fit on the root disk after 3 astropy images returned 404. Sampling follows the model-card thinking mode: temperature 0.6, top_p 0.95, top_k 20, min_p 0, max tokens 32768, step limit 250, 4 workers. The model card reports 76.4 on the full verified set. On these 59 the standard arm resolved 40, left 17 unresolved, and had 2 harness errors (django__django-12209, django__django-16429), with 0 empty patches. The monokernel arm resolved 44, left 15 unresolved, and had 0 errors and 0 empty patches. Paired on the same 59: 39 resolved on both, 1 only on the standard path, 5 only on the monokernel (the 2 standard harness errors are in those 5), McNemar exact p = 0.219. Same SGLang 9691e711cd and the same local mamba-cache fix on the monokernel arm.

Standard accuracy Monokernel accuracy
GSM8K 0.945 0.97
GPQA Diamond 88.51% (701/792) 87.75% (695/792)
SWE-bench Verified 67.80% (40/59) 74.58% (44/59)

Speed Tests and Profiling

Same stack and flags as above, --max-running-requests 16 and decode graphs at batch sizes 1 to 8 on both servers. python3 -m sglang.bench_serving --backend sglang --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1 --num-prompts <10 x c> --max-concurrency <c> --warmup-requests 2 --seed 1. TPOT is the median per-user decode time.

Concurrency Standard output tok/s Mono output tok/s Change Standard TPOT (ms) Mono TPOT (ms) Per-user tok/s
1 192.61 233.03 +21.0% 4.98 4.07 200.8 -> 245.7
4 559.80 662.56 +18.4% 6.45 5.38 155.0 -> 185.9
8 810.32 943.13 +16.4% 8.84 7.45 113.1 -> 134.2
16 (past width 8) 251.17 250.61 -0.2% 61.94 62.04 16.1 -> 16.1

Median TTFT is within 1% at every concurrency, as prefill is SGLang's on both. At concurrency 16 most steps are wider than 8 rows and run SGLang's standard forward on both servers, eagerly, since no decode graph is captured above 8 rows; the hand-off costs nothing. With --enable-aiter-allreduce-fusion on the standard path (193.78, 574.26 and 840.40 tok/s at concurrency 1, 4 and 8) the monokernel is +20.3%, +15.4% and +12.2%.

One decode step at 8 rows, rank 0, kernel time from the torch profiler. Both servers decode the same 8 random 8192-token prompts.

Work in the step Standard path Kernels µs Monokernel Kernels µs
GDN projections, conv, gated delta, gated norm, quant (45 layers) block-scale GEMM, bf16 in_proj_ba, split, conv, gated delta, gated norm, quant 315 1898.5 _proj_conv_kernel, _delta_kernel 90 770.1
out_proj, qkv_proj, o_proj block-scale GEMM 75 580.1 _wg_kernel 75 429.4
All-reduce, residual add, Gemma norm, quant of the next GEMV's input custom all-reduce, Gemma add-RMSNorm, quant 301 1799.9 aiter fused all-reduce (_add_norm_quant_kernel at layer 0) 121 1098.1
Attention output gate and quant (15 layers) sigmoid-mul, quant 30 126.6 _gate_quant_kernel 15 65.3
Step barriers 0 0 two FlyDSL phases 2 10.4
MoE router and experts SGLang's 180 3569.5 the same 180 3602.1
Attention q/k norm, rope, KV write, paged attention SGLang's 75 769.3 the same 75 768.3
Embedding, final norm, LM head, other SGLang's 7 115.7 the same 7 119.1
Step 983 8860 565 6863

E2E

The TP4 numbers above are unchanged and were measured before this layout move. After the move, on MI355X in jacky-v0.5.20-rocm724-mi35x-qwen3.8-mxfp4-megakernel-1022, checkpoint /data/Qwen3.5-397B-A17B-FP8, flags of test/e2e/qwen35_a17b_fp8_server.py:

  • TP4 eager check (TestMonokernelCheck, GPUs 0-3, widths 1, 3 and 8): 1 passed in 85s, zero check violations.
  • TP8 eager check (QWEN35_FP8_TP=8, GPUs 0-7, the same test): 1 passed in 94s, zero check violations. That covers GDN state, projections, MoE and the all-reduce at those widths.
  • TP8 serving, one run, 16 prompts, random 512 in / 128 out, --tp 8. This is not the ISL 8192 / OSL 1024 sweep in the table above.
Concurrency Standard output tok/s Mono output tok/s Change
1 185.10 217.51 +17.5%
4 579.10 653.59 +12.9%
8 806.58 930.70 +15.4%

On this SGLang the standard server does not load the monokernel spec, so it does not publish the small-M import stand-in. The standard arm of this TP8 run applied the same _allow_block_fp8_linear stand-in through a process-local sitecustomize that is not in the repo. The monokernel publishes that stand-in in the worker. Without it, the standard server dies on cannot import name '_check' from smallm_moe_gfx950 before the first generate.

Not covered

  • TP other than 4 and 8, MI350X, and MTP or other speculative verify steps.
  • Decode steps of more than 8 rows, which run SGLang's standard forward.
  • One persistent launch per layer. A GDN layer is eight launches (five before the MoE) and a full-attention layer thirteen, inside the decode CUDA graph. The MoE (SGLang's router and two expert stages) is 3.6 ms of the 6.9 ms step at 8 rows; fusing it with the norms and the all-reduce is the follow-up.
  • MXFP4 stays on [Feat] [MI355X] qwen35_a17b: Serve Quark MXFP4 AttnFP8 from the family spec #57. This PR does not move that package. The family spec refuses it until it lands under qwen35_a17b/mxfp4, so the two precisions share one architecture registration.
  • On this SGLang, the gfx950 small-M block-FP8 helper imports names that smallm_moe_gfx950 no longer defines, so every block-FP8 linear fails to import. When that is the case, the spec publishes stand-ins and sets SGLANG_ROCM_SMALLM_FP8_BS=0 before the import, so the linears take aiter's block-scale GEMM. The standard-path server in these runs used the same stand-ins, so both sides compare the same prefill and linears.

Checklist

  • pre-commit run --all-files passes.
  • Tests under test/ cover the change.
  • The SGLang commit or PR this was tested against is named above.
  • Accuracy and speed results are attached when kernel or forward code changes.
  • A new model adds one line to .github/CODEOWNERS that names its owner.

Unit tests

The run uses the image jacky-v0.5.23-rocm724-mi35x-wan2.2-1031, SGLang 9691e711cd, on CPU, with PYTHONPATH set to this branch, from /home/yichiche/sgl-amd-mono-pr58:

python3 -m pytest -v -s -p no:cacheprovider test/unit -k qwen35
Suite Result
unit, -k qwen35 22 passed in 8.57 s

This is the pytest output:

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.6.0 -- /opt/venv/bin/python3
rootdir: /home/yichiche/sgl-amd-mono-pr58
configfile: pyproject.toml
collected 639 items / 617 deselected / 22 selected

test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/check.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/forward.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/gemv.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/moe.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/verify.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/spec.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/spec.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/check.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/forward.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/gemv.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/moe.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/verify.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/spec.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/spec.py] PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_tp4_specialization_is_the_layout_module PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_the_plugin_registers_the_family_on_gfx950 PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_config_accepts_block_fp8_and_refuses_mxfp4 PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_accepts_tp4_and_tp8_and_refuses_per_channel PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_refuses_a_plain_rmsnorm PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_config_accepts_the_mtp_chain_and_refuses_other_drafts PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_forward_hands_unserved_steps_to_the_standard_forward PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_refuses_a_projection_with_a_bias PASSED

=============== 22 passed, 617 deselected, 19 warnings in 8.57s ================

Decode widths 1-8 on gfx950 stay on the new GDN and MoE kernels, while wider batches and the full-attention KV update stay on the standard path.
… and fused GDN, norm and GEMV launches

The first version ran SGLang's ops between about 110 grid barriers per step and an eager torch GDN, so decode was 6x slower than the standard path. Each step now has two barriers, a GDN layer is two Triton launches (the block-FP8 in_proj_qkvz with the conv, then the gated delta), the residual add, Gemma norm and FP8 quant are one launch, and the decode GEMVs split K across the warps of a workgroup. Decode at ISL 8192 / OSL 1024 is 21%, 18% and 16% faster than the standard path at concurrency 1, 4 and 8.
…p4-mono

# Conflicts:
#	.github/CODEOWNERS
#	sgl_amd_mono/__init__.py
Comment thread sgl_amd_mono/qwen35_a17b_fp8/spec.py Resolved
@yichiche yichiche changed the title [Feat] [MI355X] qwen35_a17b_fp8: Add the TP4 block-FP8 decode monokernel [Feat] [MI355X] qwen35_a17b: Serve block-FP8 decode at TP4 and TP8 Oct 27, 2026
…of up to 8 requests at TP4

A verify step of 1 to 8 requests x 4 draft tokens runs the monokernel layer
loop. Each GDN layer runs the draft chain in three launches plus out_proj and
writes the per-token conv and SSM state to SGLang scratch. SGLang accept step
commits from that scratch. The draft model, prefill and wider steps keep the
standard forward. Adds the model README, the MTP server tests and the verify
kernel tests.
…-tp4-mono-v2

# Conflicts:
#	.github/CODEOWNERS
#	README.md
#	sgl_amd_mono/__init__.py
… path

SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in
sgl-project/sglang#45389. Read it there first and fall back to
sglang.srt.monokernels, as the other specs on main do, so the contract test
passes on both SGLang versions.
The merge push reached the branch, but GitHub dropped the push event during
its commit_refs_rules failure, so the pull request kept the old head.
@yichiche
yichiche force-pushed the amd/qwen35-397b-fp8-tp4-mono-v2 branch from 29a40c2 to c8f3265 Compare November 5, 2026 16:16
@yichiche

Copy link
Copy Markdown
Collaborator Author

GPQA Diamond

Qwen3.5-397B-A17B-FP8, TP4, MI355X, SGLang 9691e711cd. Same server flags as test/e2e/qwen35_a17b_fp8_server.py; the monokernel arm adds --enable-amd-monokernel (this SGLang's flag; the older --monokernel-backend amd is gone). Protocol is openai/simple-evals: 198 questions, choice order from Random(0), 4 repeats (792 calls), graded on the Answer: [A-D] line. Sampling is the model card's thinking mode: temperature 0.6, top_p 0.95, top_k 20, min_p 0, max tokens 81920. Concurrency is 8, so every decode step stays in the captured graphs (batch 1–8). The model card reports 88.4.

Arm Accuracy Per repeat Mean completion tokens
Standard 88.51% (701/792) 88.38, 87.88, 88.38, 89.39 7586
Monokernel 87.75% (695/792) 89.39, 87.37, 88.89, 85.35 7537

Paired on the same 792 calls: 31 right only on the standard path, 25 right only on the monokernel. McNemar exact p = 0.504. Neither arm hit the token cap, left an answer unparsed, or returned an error.

The monokernel arm needed one local compatibility change that is not on the branch yet. This SGLang moved mamba2_layer_cache off HybridReqToTokenPool onto MambaPool; the kernel follows mamba_map and reads the slice from mamba_pool. CUDA graph capture of batch sizes 1–8 succeeded after that, and the decode log shows those steps on the monokernel.

SWE-bench Verified is not in this update.

…-tp4-mono-v2

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	README.md
@yichiche

Copy link
Copy Markdown
Collaborator Author

Unit tests

pytest test/unit on CPU, SGLang 9691e711cd (the jacky-v0.5.23-rocm724-mi35x-wan2.2-1031 image), PYTHONPATH set to this branch after the merge with main at 56669ca. 627 passed, 12 failed in 67s. None of the failures is in qwen35_a17b.

Failure Cause
test_contract.py kimi_k3 mla_pre.py no file matches srt/layers/attention/dsa/indexer_prologue_hip.py
test_contract.py kimi_k3/spec.py, kimi_k31/spec.py SGLANG_READS names this SGLang no longer provides
test_contract.py kimi_k31/wide.py no file matches srt/models/kimi_k3_amd/moe.py
test_contract.py minimax_m31/spec.py no file matches srt/models/minimax_common/amd/minimax_m3_hip.py
test_contract.py muse_spark runner, spec, weights no file matches srt/rocm_muse_spark/model.py
test_kimi_k3_forward_drift.py (3) KimiK3DecoderLayer.forward, _run_self_attn, and KimiK3LinearForCausalLM.forward hashes drifted
test_kimi_k31_forward_drift.py (1) KimiK3DecoderLayer.forward hash drifted

These are the contract and forward-drift checks for Kimi and Muse Spark against an older SGLang than the one main now targets. qwen35_a17b unit tests passed.

@yichiche

yichiche commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

@yichiche Here are the test results. The qwen35 unit tests all pass at 3b9fec0 (merge of this branch with main at 56669ca).

The run uses the image jacky-v0.5.23-rocm724-mi35x-wan2.2-1031, SGLang 9691e711cd, on CPU, with PYTHONPATH set to this branch, from /home/yichiche/sgl-amd-mono-pr58:

python3 -m pytest -v -s -p no:cacheprovider test/unit -k qwen35
Suite Result
unit, -k qwen35 22 passed in 8.57 s

This is the pytest output:

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.6.0 -- /opt/venv/bin/python3
rootdir: /home/yichiche/sgl-amd-mono-pr58
configfile: pyproject.toml
collected 639 items / 617 deselected / 22 selected

test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/check.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/forward.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/gemv.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/moe.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/gfx950/verify.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/fp8/spec.py] PASSED
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen35_a17b/spec.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/check.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/forward.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/gemv.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/moe.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/gfx950/verify.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/fp8/spec.py] PASSED
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen35_a17b/spec.py] PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_tp4_specialization_is_the_layout_module PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_the_plugin_registers_the_family_on_gfx950 PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_config_accepts_block_fp8_and_refuses_mxfp4 PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_accepts_tp4_and_tp8_and_refuses_per_channel PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_refuses_a_plain_rmsnorm PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_config_accepts_the_mtp_chain_and_refuses_other_drafts PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_forward_hands_unserved_steps_to_the_standard_forward PASSED
test/unit/test_qwen35_a17b_fp8_spec.py::test_model_refuses_a_projection_with_a_bias PASSED

@yichiche

Copy link
Copy Markdown
Collaborator Author
  • Scope: ✅ Pass. Qwen3.5-397B-A17B-FP8, 128×128 block FP8, decode widths 1–8 at TP4 and TP8 on MI355X. MXFP4 is refused.
  • Accuracy: ✅ Pass. GSM8K 0.945 → 0.97. GPQA Diamond 88.51% (701/792) → 87.75% (695/792), McNemar p 0.504. SWE-bench Verified monokernel 44/59 resolved; the standard arm is not scored yet. The GPQA and SWE monokernel runs used a local mamba-cache fix that is not on the branch.
  • Test: ✅ Pass. qwen35 unit tests 22 passed at 3b9fec0. TP4 eager check 1 passed in 85s, TP8 eager check 1 passed in 94s, both with zero check violations (GDN state, projections, MoE, and the all-reduce at widths 1, 3, and 8).
  • Kernel replacement: ✅ Pass. One decode step at bs=8, rank 0: 983 kernels / 8860 µs → 565 kernels / 6863 µs (22.5%). GDN projections, conv, delta, norm, and quant: 315 kernels / 1898.5 µs → 90 kernels / 770.1 µs.
  • E2E perf: ✅ Pass.
    • TP4
      • c=1: +21.0% (192.61 → 233.03 tok/s), TPOT 4.98 → 4.07 ms, 1.22×
      • c=4: +18.4% (559.80 → 662.56 tok/s), TPOT 6.45 → 5.38 ms
      • c=8: +16.4% (810.32 → 943.13 tok/s), TPOT 8.84 → 7.45 ms
      • c=16: −0.2%, past width 8
    • TP8
      • c=1: +17.5% (185.10 → 217.51 tok/s)
      • c=4: +12.9% (579.10 → 653.59 tok/s)
      • c=8: +15.4% (806.58 → 930.70 tok/s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants