Repository navigation
[Fix] QSA: Keep two groups in the pending index-key ring - #47051
amd-cgilli wants to merge 1 commit into
Conversation
A verify step writes the index-key ring slot of every draft row, keyed by position % ratio. When the step does not start on a group boundary, its later rows overwrite members of the group still pending before them: within the step, the group is read back after those stores, and across steps, rows that verify rejects leave their keys in those slots, so the group compresses with another group's or a rejected token's key. A ring of two groups (position % (2 * ratio)) keeps every member of the pending group in place.
QSA: Keep two groups in the pending index-key ring -- THE CHOSEN REMEDY
for the TARGET_VERIFY ring overwrite (stack entries 118-125/184 lineage).
A verify step writes the index-key ring slot of every draft row. When the
window does not start on a group boundary, its later rows overwrite members
of the group still pending before them: within the step the group is read
back after those stores, and across steps NOTHING restores the ring after
acceptance -- a rejected row's key is read back as a pending group member
on a later step, and the corrupted compressed key persists for the whole
request and skews indexer block selection. The two-group ring
(slot = position % (2 * ratio)) makes every pending group member survive
the window: a rejected row's key only sits in the slot of its own position,
which that position rewrites before any group reads it (the post-acceptance
"restore" is that no aliasing needs restoring).
Ported onto OUR ring layout (entry 118-125 parameterized num_groups ring +
entry 184 read-before-store + entries 200/212/213 sharding/launch state),
not the upstream one-group layout the PR edits. The slot arithmetic is
equal where the two layouts meet: at num_groups == 2 our
((p // ratio) % 2) * ratio + p % ratio is exactly the residue decomposition
of p % (2 * ratio), so the PR's remedy transfers as the two-group FLOOR
plus the shared constant, with every addressing site tied to the pool's
qsa_num_groups (single source of truth for the buffer and all readers).
UNIONS / DEVIATIONS vs the PR diff:
- qsa/metadata.py: PR defines QSA_RING_GROUPS = 2 here and hardcodes
ring = 2*ratio into build_pending_ring_slots/build_group_ring_slots.
Our metadata.py IMPORTS qsa_kv_pool (entry 200 sharding types), so the
PR's constant home would create a module cycle with the pool importing
it back; the constant is DEFINED in mem_cache/qsa_kv_pool.py and
re-exported by qsa/metadata.py (+ __all__) -- the PR's import path
qsa.metadata.QSA_RING_GROUPS resolves unchanged. Builders keep our
num_groups parameter with the default 1 -> QSA_RING_GROUPS (the
widening). Our layout keeps its inert extend-dump at rows [0, ratio)
(PR moves it to [0, ring)); both are inside never-allocated request
slot 0's territory -- ours kept per "keep OUR layout".
- qsa/graph_metadata.py: the PR's RING constexpr hunks on the row/draft
kernels are a NO-OP against our tree -- entries 184/200 already thread a
NUM_GROUPS constexpr with slot = req*ring_span + group*RATIO + pos%RATIO
(identical at 2, generalized wider). Only the launchers'
getattr(pool, "qsa_num_groups", 1) fallbacks move to QSA_RING_GROUPS so
a pool without the attribute cannot under-widen versus the builders'
default.
- qwen_sparse_attn_backend.py: the PR only rewords the guard
comment/message (its bound stays draft <= ratio). We keep our derived
window draft <= ratio*(groups-1)+1 (entries 118-125) -- the PR body's
own safety statement ("a window of up to ratio + 1 rows never shares a
slot") is the groups == 2 instance of our formula; prod 4/1/4 (4 draft
rows at ratio 4) is inside both. Guard None-pool fallback keeps the
historical ratio bound. _pending_ring_groups and the graph host-replay
builders take the QSA_RING_GROUPS fallback.
- mem_cache/qsa_kv_pool.py: PR grows ring_slots by the hardcoded factor
QSA_RING_GROUPS. Our pool already multiplies by qsa_num_groups; the PR
semantics land as (a) the __init__ default 1 -> QSA_RING_GROUPS, (b)
pending_ring_num_groups() floors at QSA_RING_GROUPS even without
speculation (PR ring is unconditional; our draft-derived widening of
ceil((draft + ratio - 1)/ratio) stays above the floor), and (c) the
stale "ceil(draft_tokens / ratio)" init comment corrected (it described
the one-group formula that caused the collision).
- qsa/qsa_indexer.py (not in the PR): entry-184 snapshot docstring
rationale updated for the widened ring (read-before-store stays needed:
a forward spanning three ring groups lands back on the pending group's
slots), and the getattr fallbacks sync to QSA_RING_GROUPS.
- utils/triton_prewarm_ops.py (not in the PR; entry 210/211 file,
one-line disjoint sync): the recipe geometry's fallback follows the
launchers' new QSA_RING_GROUPS default, so the prewarm cannot warm a
stale one-group specialization class.
- test_qsa.py: the PR's test_verify_rows_never_overwrite_a_pending_group
ports VERBATIM (start 0..7) -- our two-group default reproduces the PR
arithmetic exactly. Tooth-check on CPU here: 6 of 8 starts fail under
num_groups=1 ([1,2,3,5,6,7]), 0 of 8 under the port -- matches the PR's
"fails for 6 of the 8 start positions".
- Contract tests updated where the PR changes the contract:
test_qsa_ring_slots.test_defaults_to_single_group ->
test_defaults_to_the_two_group_ring (the default is now the PR layout;
the explicit num_groups=1 legacy oracle test is untouched), and
test_qsa_kv_pool.pending_ring_num_groups cases (0,1),(1,1) -> (0,2),(1,2)
for the floor.
Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv, per file):
test_qsa_ring_slots 12P/22sub, test_qsa_kv_pool 8P,
test_qsa_retraction_cpu_copy 3P, test_qsa 84P/140S (76P baseline +8 new;
skips = documented device-cuda debt), test_qsa_fused_kv_prepare 12P/146S,
test_disaggregation_wire 79P/70sub, qsa_fp8_store_kv 7P,
qsa_sparse_prefill_config 12P/40sub, kv_source 16P/5sub, pack_guard 27P/5sub,
profile_field_refs 2P, triton_load_watch_prewarm 30P, unit/spec 335P/9S/180sub
(identical to the stashed baseline), kernels/ops/qsa collect 536 tests,
check_registered_tests rc=0, ruff (v0.15.1 repo ruleset) check+format clean.
GPU-window debt unchanged: the paged-vs-packed torch.equal oracle and all
device-cuda qsa tests only collect here.
(cherry picked from commit f78ee79)
(cherry picked from commit a4e5e25ac2ca0d080ef8e47961f5cfb9e7627cdf)
PR: sgl-project#46125 (open, ported) Port ONLY the 497-line verify-ring test suite of the mutually-exclusive alternative remedy; its production fix (power-of-two qsa_ring_size sized by the draft count, "stop speculative verify from overwriting") is deliberately NOT taken -- sgl-project#47051's two-group ring (previous commit) is the chosen remedy. This suite is the cross-validation oracle for that choice. Test classification (the adaptation rule): * OBSERVABLE invariant -- 32 of 35 collected tests (24 test_full_acceptance_verify_matches_decode [widths 2,3,4 x prompt 61-64 x eager/graph], 6 test_partial_acceptance_batched_verify_matches_prefill [widths 4,5,8 -- the post-acceptance rejected-row case sgl-project#47051 documents], 2 test_mtp_draft_extend_matches_prefill): left ACTIVE, adapted only mechanically, expected bit-equal-PASS under the two-group remedy on the GPU window. No observable case here is genuinely unpassable under sgl-project#47051: the ring widths both remedies give every parametrized window are collision-free by the same residue argument (ratio=4: width 2/3/4/5 -> 8 slots = sgl-project#46125's pow2 width exactly; width 8 -> 12 here vs 16 there, both >= the required span of draft + ratio - 1 = 11 and inside this tree's guard draft <= ratio*(groups-1)+1 = 9). * INTERNALS of the untaken remedy -- 3 of 35 (test_pool_allocates_the_ring, max_rows/ring 1/4, 4/8, 8/16): skipped INDIVIDUALLY with reason. It asserts the pool attribute qsa_ring_size and sgl-project#46125's power-of-two rounding, including a ONE-group ring at plain decode -- sgl-project#47051 instead keeps the pool's qsa_num_groups derivation with a two-group floor (8 slots at max_rows 1 and 4 -- equal at the prod 4/1/4 geometry -- 12 at max_rows 8). These are remedy-choice layout facts, not key-corruption facts. Adaptations of the harness to THIS tree (mechanical, asserted behavior kept): - _pool: qsa_max_rows_per_request= (the untaken remedy's keyword) -> qsa_num_groups=QSATokenToKVPool.pending_ring_num_groups(...) -- this tree's sizing entry point; same window-fits guarantee. - _graph_metadata: also supplies write_owner_mask (bool) and decode_block_positions (-1-int32, mirrors the backend capture buffers), which THIS tree's row-metadata kernel writes on every row (entry-200 cache-sharding port; sgl-project#46125's upstream base kernel has neither). - Added the repo-convention module-level skipif(not cuda) so the CPU batteries collect this file cleanly (paired test_qsa_indexer.py carries the same mark); upstream's file relied on being CI-routed only. - launch_graph_metadata / QSAIndexerMetadata / QSAIndexer / backend calls: verified signature-compatible with this tree (entry 184 read-before-store and entries 212/213 paged-launch/owner-bound ports included) -- no test logic changed for them. Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv): 35 tests collected, 35 skipped here (module CUDA gate) -- this file and every device-cuda qsa suite are GPU-window DEBT until a CUDA host runs them; the skip reason of the 3 internalized tests distinguishes them on that run. Ruff v0.15.1 repo ruleset + format clean; check_registered_tests rc=0; the untouched test_qsa_pending_ring.py (179 CPU lines of the PR, pinning the sgl-project#46125-internal ring_size keyword of the slot builders and rings that "lose members") is deliberately NOT ported -- it is internals of the untaken remedy end to end, superseded for CPU purposes by this tree's test_qsa_ring_slots.py plus the sgl-project#47051 test ported in the previous commit. (cherry picked from commit 85cb8d0) (cherry picked from commit fc1b2182e9d5f9ca6a5a9eb7b83c35d47903e7cc)
qsa: Carry QSA indexer state through cpu_tensor retraction QSATokenToKVPool's retraction backup moved the compressed-K rows with the KV but NOT the per-request pending index-K ring: Req.offload_kv_cache snapshots through get_cpu_copy and load_kv_cache restores into a DIFFERENT request-pool row (verified in schedule_batch.py: offload/load pass req_pool_index=self.kv.req_pool_idx on both ends), so a resumed-after-retraction request runs with ANOTHER request's compressed index keys + pending ring -- the same stale-sidecar defect class as sgl-project#39830, on the retraction path. INTERACTION WITH THE sgl-project#47051 RING WIDENING (previous-but-one commit) -- the union this port had to resolve: the PR sizes one request's ring as ``req * ratio + position % ratio`` and transfers ``ratio`` rows. This tree's ring is ``ratio * qsa_num_groups`` rows with the two-group floor, so _qsa_ring_rows() takes the request's WHOLE ring, tracking get_qsa_pending_state_buf_infos' rows_per_item (the PD-transfer contract) rather than one group. Restoring only the first group would leave the second ring group holding the previous row owner's keys -- with the widened ring a verify-pending group's members may live in EITHER ring group, so a one-group carry would reintroduce exactly the cross-step key corruption sgl-project#47051 fixes, for every post-retraction request. UNIONS / DEVIATIONS vs the PR diff: - The PR (against upstream, where QSATokenToKVPool had NO override) adds get_cpu_copy/load_cpu_copy carrying a (parent, (compressed, ring)) tuple with its own _qsa_compressed_slots dedup gather. Our entry-25 port already shipped the compressed-K leg in the {"kv", "qsa_compressed_k"} dict form with one-row-per-token gathers and a legacy-tuple tolerance path; that shape is kept (it pairs with itself through RetractionBackup.cpu_tensors) and the PR's NEW content -- the ring gather/restore, the req_pool_index + local-QSA-layer gating, and the per-axis test coverage -- is grafted onto it as "qsa_pending_ring": (per-layer keys, rope) | None. - Sync plumbing: our compressed leg already used blocking .to("cpu") / index_copy_; the ring leg keeps the PR's non_blocking gathers with one current_platform.synchronize() per direction (the PR head c022f8a had already dropped two of its four syncs as redundant; ours are placed inside the ring gates, where the only async copies are). - _qsa_ring_rows widened to ratio * qsa_num_groups (the interaction above; the PR's single-group form). - The stale entry-25 NOTE on get_qsa_key_state_buffer ("deliberately not carried ... rewrites the ring from scratch") is REPLACED by the PR's finding -- its premise fails because the resume lands on a new request row. - Tests: the PR's new suite (test_qsa_retraction_backup.py, head c022f8a: CustomTestCase, partial-last-group with reordered pages and a new request row, PP-stage-without-QSA-layers, empty indices) is ported into OUR existing CPU file test_qsa_retraction_cpu_copy.py (register_cpu_ci base-a-test-cpu already there; CustomTestCase already used) rather than adding a second retraction suite. Payload shape adapted tuple-index -> dict-key (backup[1][1] -> backup["qsa_pending_ring"]); ring row arithmetic widened to RING_ROWS = RATIO * NUM_GROUPS with every ring row, group and RoPE axis carrying its own value; the PR's compressed-dedup gather stays our one-row-per-token form. The PR's first test (resumed request keeps its qsa state) already exists here for the compressed leg; its ring half ports as test_resumed_request_keeps_its_pending_ring covering BOTH ring groups. - PP-stage gate kept verbatim in logic: with no local QSA layer the ring leg is None and the shared RoPE ring is neither read nor written (matches the PD-registration rationale in get_qsa_pending_state_buf_infos). Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv, PYTHONPATH=this worktree; no GPU code run): test_qsa_retraction_cpu_copy 3P -> 7P (3 baseline +4; ring tests go RED (2F) with the ring gather gated off -- teeth proven, then restored). Battery re-run green on this tree: test_qsa_kv_pool 8P, test_qsa_ring_slots 12P/22sub, test_qsa 84P/140S (device-cuda skips), test_qsa_fused_kv_prepare 12P/146S, test_disaggregation_wire 79P/70sub, triton_load_watch_prewarm 30P, verify_ring oracle 35S (CUDA gate); combined six-file run 202P/286S/92sub; unit/mem_cache+unit/spec sweep 2384P/3037S/4149sub; kernels/ops/qsa collect 571 tests; check_registered_tests rc=0; ruff v0.15.1 ruleset+format clean. GPU-window debt (unchanged): every device-cuda qsa suite including the sgl-project#46125 oracle and the paged-vs-packed torch.equal oracle only COLLECTS on this box. (cherry picked from commit f3de461 + review follow-up c022f8a) (cherry picked from commit 01d6cdfdbd3e634caee0ffbb5e09386b987c1693)
With SGLang's 4-slot index-key ring, a verify row past the pending group overwrites one of its members, and a rejected row's key stays there; the 8-slot ring of sgl-project/sglang#47051 removes it. Say so at the ring-width check and in the README, which still described a 4-entry ring.
…nokernel (#29) ## Motivation Qwen3.8-Flash-Next (SGLang's Qwen4-Exp, `amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8`) decodes with its MTP head (NEXTN) at small batch sizes, where launch gaps dominate. This PR adds a FlyDSL monokernel for it to sgl-amd-mono, behind SGLang's monokernel interface (RFC #43949). `--monokernel-backend amd` selects it, and SGLang needs no model hooks. The server runs SGLang's AITER path (`SGLANG_USE_AITER=1`; the full launch command is in the README). - A verify step of 1-8 requests x 4 tokens (embedding, PLE, 48 GDN / QSA layers and the final mixer) runs in 13 persistent launches. - Each MTP draft step of 1-8 requests runs in one launch: a draft extend (R x 4 rows) or a draft decode step (R x 1 rows). Monokernel API v1 routes every runner of a speculative server by the target's verify key, which the NEXTN draft never runs. The draft spec therefore routes its own steps: a wrapper in front of the draft's `model.forward`. It can move to SGLang if a later monokernel API routes a draft's own step modes. The kernel itself (launch plan, stages, LDS and scratch layout, synchronization and fail-stop, weights, numerics) is described in [`sgl_amd_mono/qwen38_flash_next/README.md`](https://github.com/amd-cgilli/sgl-amd-mono/blob/qwen38-flash-next-monokernel/sgl_amd_mono/qwen38_flash_next/README.md). Below is a comparison of the before and after: <img width="1612" height="727" alt="b4" src="https://github.com/user-attachments/assets/85b581aa-d309-4fd5-8774-24fcb2d810dc" /> <img width="1272" height="669" alt="after" src="https://github.com/user-attachments/assets/2834498c-02a2-4546-b9c4-25d51567dfdb" /> ## Modifications - `sgl_amd_mono/qwen38_flash_next/`: - `spec.py`: the target spec, registered for `Qwen4ExpForConditionalGeneration` on gfx950, TP1, 1-8 requests x 4 tokens. It refuses unsupported configurations with reasons: - NEXTN/EAGLE with topk 1, 3 steps and 4 draft tokens only; - no ReplaySSM, no PP speculative scratch remapping; - bf16 KV, no HiSparse; - `SGLANG_USE_AITER`, the AITER/auto MoE runner, the fused shared expert, no paged experts or MegaMoE, no expert simulation or recording; - the Quark MXFP4 checkpoint; - `--attention-backend` triton or aiter; - plus per-layer config and weight-layout checks. - Image and video requests: a verify step is rotated by SGLang's MRoPE positions (`forward_batch.mrope_positions`, as the model's own forward does), and each request's sequence start comes from `seq_lens`. With images or video the two differ, and the QSA index tiles, ring slots and selection count from the sequence position. - QSA index-key ring: the launches index SGLang's QSA key ring by the width its pool allocates (`ring_width` in `gfx950/runner.py`: 4 or 8 slots a request). With today's 4-slot ring (one group), a verify row past the pending group overwrites one of its members, and a rejected row's key stays in the slot, so the group can later compress with it; SGLang's own verify has the same issue. sgl-project/sglang#47051 makes the ring two groups (8 slots), which keeps every pending member. The launches work with either layout. - `draft.py`: the MTP draft spec, registered for `Qwen4ExpForCausalLMMTP`. - `_build` installs a wrapper in front of the draft's `model.forward`. - The wrapper sends draft extend (`DRAFT_EXTEND_V2`, R x 4 rows) and draft decode (`DECODE`, R x 1 rows) steps of R <= 8 requests to the draft launch, and every other step to SGLang's forward. - The decision depends only on the step's shape, so a captured graph replays it. - It requires the draft's QSA attention backends (`"qsa"`), logs each draft step at its first launch, and refuses during capture a draft-shaped step the launch cannot read. - `check.py`: `SGLANG_AMD_MONO_CHECK=1` checks the target only: - at build, it compares the layer weights against the checkpoint; - on each eager verify step, it compares the logits and hidden states against SGLang's forward of the same step (`NUMERICS`). - `gfx950/`: - the kernel (`kernel.py`, the guarded wait in `device.py`), its layout, packing and config; - the layer views read from SGLang's params in place (`weights.py`); - the host-side runners (`runner.py`). The draft runner copies `input_ids` and new hidden states into buffers it owns, so its pointer tables stay valid under graph capture. - `README.md`: the kernel documentation. - The specs import SGLang's interface from `sglang.kernels.plugin_kernels.amd` and fall back to `sglang.srt.monokernels` (SGLang before sgl-project/sglang#45389), as the other specs do. - `sgl_amd_mono/__init__.py`: SPECS entries for the target and the MTP draft. - `README.md`: Supported Models, Usage, Structure, Testing. - Tests: - `test/unit/test_qwen38_flash_next_{spec,layout}.py`, CPU. These cover the specs, refusals, layout, the RoPE positions a routed verify passes, and the draft wrapper: - routing of draft extend and decode steps; - the fallback to SGLang's forward; - stacking under the runtime's forward switch; - the per-mode backend check; - the capture-time refusal; - the fixed-address input buffers. - `test/kernels/test_qwen38_flash_next_compile.py`, GPU. - `test/e2e/{qwen38_flash_next_server,test_qwen38_flash_next_server}.py`, GPU: - GSM8K at 1/4/16 requests, accept length; - the target and draft launch lines; - KL decode vs prefill; - 126k passkey; - check mode; - fault injection (target launches). ## Accuracy Tests 1 x MI355X, TP1, SGLang upstream main 2229345dff, NEXTN 3 steps / 4 draft tokens / topk 1, real acceptance. "Off" is the same server without `--monokernel-backend amd`. Measured before the rebase onto current main. The rebase and the cleanup leave the kernels' LLVM IR unchanged, and the MRoPE change reads the same sequence start for text requests. | Test | Monokernel on | Off | |---|---|---| | GSM8K, 400 q | 0.970 (accept length 3.52) | 0.975 (accept length 3.52) | | GSM8K 5-shot chat, 200 q, at 1 / 4 / 16 requests | 0.98 / 0.98 / 0.97 | 0.98 / 0.985 / 0.975 | | AIME25, 30 q x 4 seeds, pass@1 (thinking on, T 0.6, top-p 0.95) | 96.67% (116 of 120) | 98.33% (118 of 120) | | AIME25, 30 q x 4 seeds, pass@4 | 100% | 100% | | KL decode vs prefill, mean over fresh servers (bound 0.1) | 0.062 (5 runs, 0.041 to 0.085) | 0.059 (4 runs, 0.051 to 0.069) | | RULER NIAH at ~261K tokens (niah_single_1/2/3, niah_multikey_1, 25 each) | 100 | 100 | | GPQA-Diamond pass@1 (same settings) | 0.855 (sem .006) | 0.836 (sem .004) | | GPQA-Diamond majority@4 / pass@4 | 0.871 / 0.909 | 0.884 / 0.899 | <details> <summary>Unit + contract tests (CPU): 102 passed</summary> ```text date: 2026-10-29 09:50 UTC sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 sglang from: /sgl-workspace/sglang-main-latest/python/sglang versions: torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0 GPU: 1 x AMD Instinct MI355X (gfx950) $ python -m pytest -v -p no:cacheprovider test/unit/test_qwen38_flash_next_spec.py test/unit/test_qwen38_flash_next_layout.py test/unit/test_contract.py -k qwen38 ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python rootdir: /sgl-workspace/sgl-amd-mono configfile: pyproject.toml plugins: platformdirs-4.13.5, anyio-4.15.1 collecting ... collected 136 items / 34 deselected / 102 selected test/unit/test_qwen38_flash_next_spec.py::test_the_plugin_registers_the_target_and_the_draft PASSED [ 0%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_spec_declares_the_drafts_own_backend PASSED [ 1%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_keeps_the_forward_signature PASSED [ 2%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-1-4] PASSED [ 3%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-8-32] PASSED [ 4%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-1-1] PASSED [ 5%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-8-8] PASSED [ 6%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-1-4] PASSED [ 7%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-8-32] PASSED [ 8%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-1-1] PASSED [ 9%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-8-8] PASSED [ 10%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests] PASSED [ 11%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests extend] PASSED [ 12%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-verify] PASSED [ 13%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-extend] PASSED [ 14%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests] PASSED [ 15%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests extend] PASSED [ 16%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-verify] PASSED [ 17%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-extend] PASSED [ 18%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_with_input_embeds_is_refused PASSED [ 19%] test/unit/test_qwen38_flash_next_spec.py::test_the_runtimes_forward_wraps_the_draft_wrapper PASSED [ 20%] test/unit/test_qwen38_flash_next_spec.py::test_the_wrapper_checks_the_backend_once_a_step_mode PASSED [ 21%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[three rows a request] PASSED [ 22%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[fp32 hidden] PASSED [ 23%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[hidden of other rows] PASSED [ 24%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[strided hidden] PASSED [ 25%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[no hidden states] PASSED [ 26%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[qsa] PASSED [ 27%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[hybrid] PASSED [ 28%] test/unit/test_qwen38_flash_next_spec.py::test_every_draft_call_site_calls_the_model_forward_attribute PASSED [ 29%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_writes_the_buffer_its_input_does_not_overlap PASSED [ 30%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_reads_new_hidden_states_from_a_fixed_address PASSED [ 31%] test/unit/test_qwen38_flash_next_spec.py::test_build_installs_the_wrapper PASSED [ 32%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_worker_reads_index_sharing_off_after_the_build PASSED [ 33%] test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_is_accepted PASSED [ 34%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[hidden size] PASSED [ 35%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[experts] PASSED [ 36%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[index budget] PASSED [ 37%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[layer pattern] PASSED [ 38%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[quantization] PASSED [ 39%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[ssm dtype] PASSED [ 40%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[context] PASSED [ 41%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[no flydsl] PASSED [ 42%] test/unit/test_qwen38_flash_next_spec.py::test_the_ple_layer_is_one_reason PASSED [ 43%] test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_routes_verify_batches_of_one_to_eight PASSED [ 44%] test/unit/test_qwen38_flash_next_spec.py::test_the_requirements_are_declared_sglang_reads PASSED [ 45%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[two tokens] PASSED [ 46%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[nine requests] PASSED [ 47%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[decode] PASSED [ 48%] test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[mrope] PASSED [ 49%] test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[text positions] PASSED [ 50%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_reaches_the_jit_cache_key PASSED [ 50%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[kind] PASSED [ 51%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[empty] PASSED [ 52%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[requests] PASSED [ 53%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[tokens] PASSED [ 54%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[first source] PASSED [ 55%] test/unit/test_qwen38_flash_next_spec.py::test_a_verify_is_thirteen_launches_of_the_48_layers PASSED [ 56%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[1] PASSED [ 57%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[2] PASSED [ 58%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[3] PASSED [ 59%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[4] PASSED [ 60%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[5] PASSED [ 61%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[6] PASSED [ 62%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[7] PASSED [ 63%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[8] PASSED [ 64%] test/unit/test_qwen38_flash_next_spec.py::test_the_layout_matches_the_spec PASSED [ 65%] test/unit/test_qwen38_flash_next_spec.py::test_the_views_accept_the_layers PASSED [ 66%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[no fused shared expert] PASSED [ 67%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[unshuffled experts] PASSED [ 68%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[interleaved gate and up] PASSED [ 69%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[2-byte scales] PASSED [ 70%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[dequantized projection] PASSED [ 71%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[wide b / a projection] PASSED [ 72%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[bf16 projection] PASSED [ 73%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[conv weight] PASSED [ 74%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[f32 norm] PASSED [ 75%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[index projection] PASSED [ 76%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[q norm] PASSED [ 77%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[router] PASSED [ 78%] test/unit/test_qwen38_flash_next_spec.py::test_an_offloaded_ngram_table_is_refused PASSED [ 79%] test/unit/test_qwen38_flash_next_spec.py::test_the_model_is_accepted PASSED [ 80%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_model_is_one_reason PASSED [ 81%] test/unit/test_qwen38_flash_next_spec.py::test_the_mtp_draft_views_are_checked PASSED [ 82%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[2560-6144] PASSED [ 83%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[16384-2560] PASSED [ 84%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[128-2560] PASSED [ 85%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[640-2560] PASSED [ 86%] test/unit/test_qwen38_flash_next_layout.py::test_moe_padding_matches_fusedmoe PASSED [ 87%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle_moe_expert_inverts_fusedmoe_layout PASSED [ 88%] test/unit/test_qwen38_flash_next_layout.py::test_check_moe_params_accepts_fusedmoe_shapes PASSED [ 89%] test/unit/test_qwen38_flash_next_layout.py::test_pack_bf16_tiles_16_rows_by_64_k PASSED [ 90%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 91%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 92%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 93%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 94%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [ 95%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 96%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 97%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 98%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 99%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [100%] =============================== warnings summary =============================== <frozen importlib._bootstrap>:488 <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute <frozen importlib._bootstrap>:488 <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead. from distutils.dep_util import newer, newer_group test/unit/test_qwen38_flash_next_spec.py: 14 warnings /opt/venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =============== 102 passed, 34 deselected, 18 warnings in 8.10s ================ exit code: 0 ``` </details> <details> <summary>Kernel tests: 7 passed</summary> ```text date: 2026-10-29 09:50 UTC sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 sglang from: /sgl-workspace/sglang-main-latest/python/sglang versions: torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0 GPU: 1 x AMD Instinct MI355X (gfx950) $ python -m pytest -v -p no:cacheprovider test/kernels/test_qwen38_flash_next_launches.py ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python rootdir: /sgl-workspace/sgl-amd-mono configfile: pyproject.toml plugins: platformdirs-4.13.5, anyio-4.15.1 collecting ... collected 7 items test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=1] PASSED [ 14%] test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=8] PASSED [ 28%] test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=7+1 pad] PASSED [ 42%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-extend] PASSED [ 57%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-decode] PASSED [ 71%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-extend] PASSED [ 85%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-decode] PASSED [100%] =============================== warnings summary =============================== ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead. from distutils.dep_util import newer, newer_group -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================== 7 passed, 2 warnings in 24.13s ======================== exit code: 0 ``` </details> <details> <summary>Synthetic benchmark C1 8K input / 1K output</summary> ```text date: 2026-10-29 15:31 UTC sgl-amd-mono: 24c72df3930a4c6269df157565b83f1b194387ac Turn off crash dumps in the Qwen3.8-Flash-Next fault test SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 GPU: 1 x AMD Instinct MI355X (gfx950) MTP acceptance: measured (0 SGLANG_SIMULATE_ACC_* in the server env) # server $ SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 AITER_FLYDSL_FORCE=1 SGLANG_MAMBA_SSM_DTYPE=bfloat16 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \ python -m sglang.launch_server --model-path amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8 --port 31060 --tp 1 --attention-backend triton --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-max-bs-decode 8 --max-running-requests 16 --enable-amd-monokernel # client $ python -m sglang.bench_serving --backend sglang --port 31060 --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 --num-prompts 20 --max-concurrency 1 --warmup-requests 2 --seed 1 /sgl-workspace/sglang-main-latest/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`). warnings.warn( benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) Waiting up to 60s for http://0.0.0.0:31060/v1/models to become ready... Server ready in 0.0s. Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8', served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) #Input tokens: 163840 #Output tokens: 20480 Starting warmup with 2 sequences... Warmup completed with 2 sequences. Starting main benchmark run... 0%| | 0/20 [00:00<?, ?it/s] 5%|▌ | 1/20 [00:02<00:38, 2.03s/it] 10%|█ | 2/20 [00:03<00:35, 1.95s/it] 15%|█▌ | 3/20 [00:05<00:31, 1.85s/it] 20%|██ | 4/20 [00:11<00:55, 3.46s/it] 25%|██▌ | 5/20 [00:13<00:42, 2.87s/it] 30%|███ | 6/20 [00:15<00:38, 2.73s/it] 35%|███▌ | 7/20 [00:17<00:31, 2.41s/it] 40%|████ | 8/20 [00:19<00:27, 2.33s/it] 45%|████▌ | 9/20 [00:21<00:24, 2.23s/it] 50%|█████ | 10/20 [00:23<00:21, 2.16s/it] 55%|█████▌ | 11/20 [00:25<00:19, 2.14s/it] 60%|██████ | 12/20 [00:27<00:16, 2.02s/it] 65%|██████▌ | 13/20 [00:29<00:13, 1.94s/it] 70%|███████ | 14/20 [00:31<00:12, 2.02s/it] 75%|███████▌ | 15/20 [00:33<00:09, 1.94s/it] 80%|████████ | 16/20 [00:35<00:08, 2.00s/it] 85%|████████▌ | 17/20 [00:37<00:05, 1.93s/it] 90%|█████████ | 18/20 [00:38<00:03, 1.88s/it] 95%|█████████▌| 19/20 [00:40<00:01, 1.85s/it] 100%|██████████| 20/20 [00:42<00:00, 1.83s/it] 100%|██████████| 20/20 [00:42<00:00, 2.13s/it] ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 20 Benchmark duration (s): 42.57 Total input tokens: 163840 Total input text tokens: 163840 Total generated tokens: 20480 Total generated tokens (retokenized): 20422 Request throughput (req/s): 0.47 Input token throughput (tok/s): 3849.17 Output token throughput (tok/s): 481.15 Peak output token throughput (tok/s): 686.00 Peak concurrent requests: 2 Total token throughput (tok/s): 4330.31 Concurrency: 1.00 Accept length: 3.57 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2127.22 Median E2E Latency (ms): 1851.48 P90 E2E Latency (ms): 2236.08 P95 E2E Latency (ms): 2638.79 P99 E2E Latency (ms): 5264.40 ---------------Time to First Token---------------- Mean TTFT (ms): 456.02 Median TTFT (ms): 245.55 P90 TTFT (ms): 291.58 P95 TTFT (ms): 720.12 P99 TTFT (ms): 3441.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.63 Median TPOT (ms): 1.53 P90 TPOT (ms): 1.87 P95 TPOT (ms): 1.93 P99 TPOT (ms): 2.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.63 Median ITL (ms): 1.46 P90 ITL (ms): 1.92 P95 ITL (ms): 2.88 P99 ITL (ms): 5.79 Max ITL (ms): 6.31 ================================================== # monokernel lines from the server log [2026-10-29 15:31:03] monokernel qwen38_flash_next_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 2436.7 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: final mixer packed (2 tensors, 12.8 MiB), gdn layer packed (360 tens [2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: SGLang's QSA MTP index sharing off (index_share_for_mtp_iteration True -> False), so every draft decode selects its own blocks [2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: draft extend steps of up to 8 x 4 rows and draft decode steps of up to 8 x 1 are routed by the spec's wrapper of the draft's forward, and one it cannot launch raises; each key logs 'MTP draft runs' at its first launch [2026-10-29 15:31:09] monokernel qwen38_flash_next_mtp_draft_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 543.0 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: mtp packed (9 tensors, 66.0 MiB) [2026-10-29 15:31:24] monokernel qwen38_flash_next_hip on gfx950 runs target_verify steps of batch size [1, 2, 3, 4, 5, 6, 7, 8] x 4 tokens (8 of 8 captured decode graphs); every other step runs the standard path [2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 7, 7) [2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 8, 8) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 4, 4) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 5, 5) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 6, 6) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 1, 1) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 2, 2) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 3, 3) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 5, 20) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 6, 24) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 7, 28) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 8, 32) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 2, 8) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 3, 12) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 4, 16) [2026-10-29 15:31:31] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 1, 4) ``` </details> ## Speed Tests and Profiling 1 x MI355X, TP1, the same server flags with and without `--monokernel-backend amd`, CUDA graphs up to 8 requests. Verify and draft are both on the monokernel. Client: ```bash python -m sglang.bench_serving --backend sglang --dataset-name random \ --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \ --num-prompts <max(10 x conc, 20)> --max-concurrency <conc> --warmup-requests 2 --seed 1 ``` | C | Off: median TPOT / output tok/s | On: median TPOT / output tok/s | Speedup: TPOT [95% CI] / tok/s | Accept length off / on | |---|---|---|---|---| | 1 | 4.04 ms / 220 | 1.60 ms / 517 | **2.53x** [2.28, 2.84] / 2.35x | 3.36 / 3.38 | | 2 | 4.58 ms / 400 | 2.17 ms / 812 | **2.12x** [1.89, 2.26] / 2.03x | 3.39 / 3.43 | | 4 | 4.99 ms / 669 | 3.21 ms / 1063 | **1.56x** [1.51, 1.71] / 1.59x | 3.39 / 3.40 | | 8 | 6.22 ms / 1088 | 5.86 ms / 1180 | **1.06x** [1.02, 1.09] / 1.08x | 3.38 / 3.40 | ## Checklist - [x] `pre-commit run --all-files` passes. - [x] Tests under `test/` cover the change. - [x] The SGLang commit tested against: upstream main 2229345dff (accuracy and speed), f3f0f7b68c (C=1 re-check), f3f0f7b68c and af21e2b989 (unit tests). - [x] Accuracy and speed results. - [x] `.github/CODEOWNERS`: `**/*qwen38_flash_next*` is already in upstream's CODEOWNERS.
|
Here are NVIDIA results for this PR: SM120 and SM100, to go with your MI355X numbers. We ran the PR head
We measured a similar change in a downstream build on a Qwen3.8-Flash-Next-based checkpoint with NEXTN 3/1/4, in real serving on RTX PRO 6000. The share of decode-formed compressed keys that differed from a prefill of the same tokens went from about 82% to 0 of 1.78M. Step time and acceptance were unchanged (95.26 vs 94.98 ms/step at 96 concurrent requests, accept length 3.161 vs 3.151). For reviewers comparing with #46125, which fixes the same bug: at ratio 4 with up to 5 verify rows, both PRs use an 8-slot ring per request ( |
Motivation
The QSA indexer keeps each request's not-yet-compressed index keys in a per-request ring, at slot
req_pool_idx * ratio + position % ratio, so the ring holds one group (ratio = 4). A speculative verify step writes the ring key of every draft row. When the step does not start on a group boundary, its later rows land in the slots of the group still pending before them, which corrupts that group's compressed key in two ways.build_pending_ring_slots) and only then reads the completed group back from the ring. Takepos0 = 5, rows 5-8: group 4-7 completes at row 7, but row 8 has already overwritten key 4 in slot 0. The group is compressed from keys 8, 5, 6, 7.In both cases the stored compressed key is not the one the QSA definition specifies: RMSNorm(AvgPool(k[p_b : p_b + r - 1])) over the block's own keys (Qwen3.8-Next tech report, Eq. (13)). It stays in the compressed cache for the rest of the request and skews the indexer's block selection. Attention values and the KV cache are not affected. The ring is hit by any verify step that does not start on a multiple of 4, roughly 3 steps in 4.
Modifications
The ring holds two groups per request: slot
req_pool_idx * 2 * ratio + position % (2 * ratio). A verify window of up toratio + 1rows then never shares a slot with a member of the group pending before it. A rejected row's key only sits in the slot of its own position, which is rewritten when that position is processed again, before any group reads it.qsa/metadata.py:QSA_RING_GROUPS = 2.build_pending_ring_slotsandbuild_group_ring_slotsindex the ring byring = QSA_RING_GROUPS * ratio. Extend's dump rows become[0, ring).qsa/graph_metadata.py: the graph row-metadata kernel and the draft graph-metadata kernel take aRINGconstexpr forstate_slotsandring_locs. Compression still usesRATIO.mem_cache/qsa_kv_pool.py: the key-state and RoPE-position buffers havenum_request_slots * QSA_RING_GROUPS * ratiorows. That is 4 extra rows per request slot: one bf16 128-dim key per row in each QSA layer, plus the RoPE rows.qwen_sparse_attn_backend.py: the existing limitspeculative_num_draft_tokens <= ratiois unchanged; only its comment and error text are updated.test_verify_rows_never_overwrite_a_pending_group(test_qsa.py) checks, for every start position 0-7, that a 4-row verify window never writes a slot that holds a member of the pending group. It fails for 6 of the 8 start positions without this change.Accuracy Tests
test/registered/kernels/ops/attention/qsa/test_qsa.py(156 passed, 1 skipped),test_qsa_rocm.py(4 passed),test_qsa_fused_kv_prepare.py(174 passed) on MI355X.test_qsa_indexer.pyandtest_qsa_determinism.pyfail the same tests with and without this change on this machine.End to end, Qwen3.8-Flash-Next (
amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8), 1 x MI355X, TP1, NEXTN 3 steps / topk 1 / 4 draft tokens, with the sgl-amd-mono monokernel (--enable-amd-monokernel), which reads the ring layout from the pool: 10/10 requests at C=1, measured accept length 3.26, the same as before the change.SGLang's own path (no monokernel), same model and NEXTN setup,
--attention-backend triton, 16 concurrent requests (sgl-eval), main af21e2b without and with this change:AIME25's outputs average about 10k tokens, so the indexer selects blocks over the request's own generated tokens. GSM8K mostly stays within the 2048-token budget, where every block is selected.
Speed Tests and Profiling
The change only moves ring addresses; no kernel does more work.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #40668243157
Latest PR Test (Extra): ❌ Run #40668242990
Latest PR Test (AMD ROCm 10): ❌ Run #40668243190