Skip to content

[Fix] QSA: Keep two groups in the pending index-key ring - #47051

Open
amd-cgilli wants to merge 1 commit into
sgl-project:mainfrom
amd-cgilli:qsa-two-group-ring
Open

amd-cgilli wants to merge 1 commit into
sgl-project:mainfrom
amd-cgilli:qsa-two-group-ring

Conversation

@amd-cgilli

@amd-cgilli amd-cgilli commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

The QSA indexer keeps each request's not-yet-compressed index keys in a per-request ring, at slot req_pool_idx * ratio + position % ratio, so the ring holds one group (ratio = 4). A speculative verify step writes the ring key of every draft row. When the step does not start on a group boundary, its later rows land in the slots of the group still pending before them, which corrupts that group's compressed key in two ways.

  • Within a step: verify stores all rows' keys first (build_pending_ring_slots) and only then reads the completed group back from the ring. Take pos0 = 5, rows 5-8: group 4-7 completes at row 7, but row 8 has already overwritten key 4 in slot 0. The group is compressed from keys 8, 5, 6, 7.
  • Across steps: nothing restores the ring after acceptance. With rows 7 and 8 rejected, the next step starts at position 7, completes group 4-7, and reads key 4's slot, which still holds rejected key 8.

In both cases the stored compressed key is not the one the QSA definition specifies: RMSNorm(AvgPool(k[p_b : p_b + r - 1])) over the block's own keys (Qwen3.8-Next tech report, Eq. (13)). It stays in the compressed cache for the rest of the request and skews the indexer's block selection. Attention values and the KV cache are not affected. The ring is hit by any verify step that does not start on a multiple of 4, roughly 3 steps in 4.

Modifications

The ring holds two groups per request: slot req_pool_idx * 2 * ratio + position % (2 * ratio). A verify window of up to ratio + 1 rows then never shares a slot with a member of the group pending before it. A rejected row's key only sits in the slot of its own position, which is rewritten when that position is processed again, before any group reads it.

  • qsa/metadata.py: QSA_RING_GROUPS = 2. build_pending_ring_slots and build_group_ring_slots index the ring by ring = QSA_RING_GROUPS * ratio. Extend's dump rows become [0, ring).
  • qsa/graph_metadata.py: the graph row-metadata kernel and the draft graph-metadata kernel take a RING constexpr for state_slots and ring_locs. Compression still uses RATIO.
  • mem_cache/qsa_kv_pool.py: the key-state and RoPE-position buffers have num_request_slots * QSA_RING_GROUPS * ratio rows. That is 4 extra rows per request slot: one bf16 128-dim key per row in each QSA layer, plus the RoPE rows.
  • qwen_sparse_attn_backend.py: the existing limit speculative_num_draft_tokens <= ratio is unchanged; only its comment and error text are updated.
  • Test: test_verify_rows_never_overwrite_a_pending_group (test_qsa.py) checks, for every start position 0-7, that a 4-row verify window never writes a slot that holds a member of the pending group. It fails for 6 of the 8 start positions without this change.

Accuracy Tests

  • test/registered/kernels/ops/attention/qsa/test_qsa.py (156 passed, 1 skipped), test_qsa_rocm.py (4 passed), test_qsa_fused_kv_prepare.py (174 passed) on MI355X. test_qsa_indexer.py and test_qsa_determinism.py fail the same tests with and without this change on this machine.

  • End to end, Qwen3.8-Flash-Next (amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8), 1 x MI355X, TP1, NEXTN 3 steps / topk 1 / 4 draft tokens, with the sgl-amd-mono monokernel (--enable-amd-monokernel), which reads the ring layout from the pool: 10/10 requests at C=1, measured accept length 3.26, the same as before the change.

  • SGLang's own path (no monokernel), same model and NEXTN setup, --attention-backend triton, 16 concurrent requests (sgl-eval), main af21e2b without and with this change:

    main main + this PR
    GSM8K, 1319 q, thinking off, greedy 95.00% 95.15%
    AIME25, 30 q x 4, thinking, T 0.6, top-p 0.95, max 32k tokens: pass@1 / pass@4 / maj@4 90.00% / 93.33% / 93.33% 90.00% / 93.33% / 93.33%
    Accept length (server average after GSM8K / after AIME25) 2.99 / 2.89 2.99 / 2.87

    AIME25's outputs average about 10k tokens, so the indexer selects blocks over the request's own generated tokens. GSM8K mostly stays within the 2048-token budget, where every block is selected.

Speed Tests and Profiling

The change only moves ring addresses; no kernel does more work.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #40668243157
Latest PR Test (Extra): ❌ Run #40668242990
Latest PR Test (AMD ROCm 10): ❌ Run #40668243190

A verify step writes the index-key ring slot of every draft row, keyed by
position % ratio. When the step does not start on a group boundary, its later
rows overwrite members of the group still pending before them: within the step,
the group is read back after those stores, and across steps, rows that verify
rejects leave their keys in those slots, so the group compresses with another
group's or a rejected token's key. A ring of two groups (position % (2 * ratio))
keeps every member of the pending group in place.
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
QSA: Keep two groups in the pending index-key ring -- THE CHOSEN REMEDY
for the TARGET_VERIFY ring overwrite (stack entries 118-125/184 lineage).

A verify step writes the index-key ring slot of every draft row. When the
window does not start on a group boundary, its later rows overwrite members
of the group still pending before them: within the step the group is read
back after those stores, and across steps NOTHING restores the ring after
acceptance -- a rejected row's key is read back as a pending group member
on a later step, and the corrupted compressed key persists for the whole
request and skews indexer block selection. The two-group ring
(slot = position % (2 * ratio)) makes every pending group member survive
the window: a rejected row's key only sits in the slot of its own position,
which that position rewrites before any group reads it (the post-acceptance
"restore" is that no aliasing needs restoring).

Ported onto OUR ring layout (entry 118-125 parameterized num_groups ring +
entry 184 read-before-store + entries 200/212/213 sharding/launch state),
not the upstream one-group layout the PR edits. The slot arithmetic is
equal where the two layouts meet: at num_groups == 2 our
((p // ratio) % 2) * ratio + p % ratio is exactly the residue decomposition
of p % (2 * ratio), so the PR's remedy transfers as the two-group FLOOR
plus the shared constant, with every addressing site tied to the pool's
qsa_num_groups (single source of truth for the buffer and all readers).

UNIONS / DEVIATIONS vs the PR diff:
- qsa/metadata.py: PR defines QSA_RING_GROUPS = 2 here and hardcodes
  ring = 2*ratio into build_pending_ring_slots/build_group_ring_slots.
  Our metadata.py IMPORTS qsa_kv_pool (entry 200 sharding types), so the
  PR's constant home would create a module cycle with the pool importing
  it back; the constant is DEFINED in mem_cache/qsa_kv_pool.py and
  re-exported by qsa/metadata.py (+ __all__) -- the PR's import path
  qsa.metadata.QSA_RING_GROUPS resolves unchanged. Builders keep our
  num_groups parameter with the default 1 -> QSA_RING_GROUPS (the
  widening). Our layout keeps its inert extend-dump at rows [0, ratio)
  (PR moves it to [0, ring)); both are inside never-allocated request
  slot 0's territory -- ours kept per "keep OUR layout".
- qsa/graph_metadata.py: the PR's RING constexpr hunks on the row/draft
  kernels are a NO-OP against our tree -- entries 184/200 already thread a
  NUM_GROUPS constexpr with slot = req*ring_span + group*RATIO + pos%RATIO
  (identical at 2, generalized wider). Only the launchers'
  getattr(pool, "qsa_num_groups", 1) fallbacks move to QSA_RING_GROUPS so
  a pool without the attribute cannot under-widen versus the builders'
  default.
- qwen_sparse_attn_backend.py: the PR only rewords the guard
  comment/message (its bound stays draft <= ratio). We keep our derived
  window draft <= ratio*(groups-1)+1 (entries 118-125) -- the PR body's
  own safety statement ("a window of up to ratio + 1 rows never shares a
  slot") is the groups == 2 instance of our formula; prod 4/1/4 (4 draft
  rows at ratio 4) is inside both. Guard None-pool fallback keeps the
  historical ratio bound. _pending_ring_groups and the graph host-replay
  builders take the QSA_RING_GROUPS fallback.
- mem_cache/qsa_kv_pool.py: PR grows ring_slots by the hardcoded factor
  QSA_RING_GROUPS. Our pool already multiplies by qsa_num_groups; the PR
  semantics land as (a) the __init__ default 1 -> QSA_RING_GROUPS, (b)
  pending_ring_num_groups() floors at QSA_RING_GROUPS even without
  speculation (PR ring is unconditional; our draft-derived widening of
  ceil((draft + ratio - 1)/ratio) stays above the floor), and (c) the
  stale "ceil(draft_tokens / ratio)" init comment corrected (it described
  the one-group formula that caused the collision).
- qsa/qsa_indexer.py (not in the PR): entry-184 snapshot docstring
  rationale updated for the widened ring (read-before-store stays needed:
  a forward spanning three ring groups lands back on the pending group's
  slots), and the getattr fallbacks sync to QSA_RING_GROUPS.
- utils/triton_prewarm_ops.py (not in the PR; entry 210/211 file,
  one-line disjoint sync): the recipe geometry's fallback follows the
  launchers' new QSA_RING_GROUPS default, so the prewarm cannot warm a
  stale one-group specialization class.
- test_qsa.py: the PR's test_verify_rows_never_overwrite_a_pending_group
  ports VERBATIM (start 0..7) -- our two-group default reproduces the PR
  arithmetic exactly. Tooth-check on CPU here: 6 of 8 starts fail under
  num_groups=1 ([1,2,3,5,6,7]), 0 of 8 under the port -- matches the PR's
  "fails for 6 of the 8 start positions".
- Contract tests updated where the PR changes the contract:
  test_qsa_ring_slots.test_defaults_to_single_group ->
  test_defaults_to_the_two_group_ring (the default is now the PR layout;
  the explicit num_groups=1 legacy oracle test is untouched), and
  test_qsa_kv_pool.pending_ring_num_groups cases (0,1),(1,1) -> (0,2),(1,2)
  for the floor.

Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv, per file):
test_qsa_ring_slots 12P/22sub, test_qsa_kv_pool 8P,
test_qsa_retraction_cpu_copy 3P, test_qsa 84P/140S (76P baseline +8 new;
skips = documented device-cuda debt), test_qsa_fused_kv_prepare 12P/146S,
test_disaggregation_wire 79P/70sub, qsa_fp8_store_kv 7P,
qsa_sparse_prefill_config 12P/40sub, kv_source 16P/5sub, pack_guard 27P/5sub,
profile_field_refs 2P, triton_load_watch_prewarm 30P, unit/spec 335P/9S/180sub
(identical to the stashed baseline), kernels/ops/qsa collect 536 tests,
check_registered_tests rc=0, ruff (v0.15.1 repo ruleset) check+format clean.
GPU-window debt unchanged: the paged-vs-packed torch.equal oracle and all
device-cuda qsa tests only collect here.

(cherry picked from commit f78ee79)
(cherry picked from commit a4e5e25ac2ca0d080ef8e47961f5cfb9e7627cdf)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
PR: sgl-project#46125 (open, ported)

Port ONLY the 497-line verify-ring test suite of the mutually-exclusive
alternative remedy; its production fix (power-of-two qsa_ring_size sized by
the draft count, "stop speculative verify from overwriting") is deliberately
NOT taken -- sgl-project#47051's two-group ring (previous commit) is the chosen remedy.
This suite is the cross-validation oracle for that choice.

Test classification (the adaptation rule):
* OBSERVABLE invariant -- 32 of 35 collected tests (24
  test_full_acceptance_verify_matches_decode [widths 2,3,4 x prompt 61-64 x
  eager/graph], 6 test_partial_acceptance_batched_verify_matches_prefill
  [widths 4,5,8 -- the post-acceptance rejected-row case sgl-project#47051 documents],
  2 test_mtp_draft_extend_matches_prefill): left ACTIVE, adapted only
  mechanically, expected bit-equal-PASS under the two-group remedy on the
  GPU window. No observable case here is genuinely unpassable under
  sgl-project#47051: the ring widths both remedies give every parametrized window are
  collision-free by the same residue argument (ratio=4: width 2/3/4/5 -> 8
  slots = sgl-project#46125's pow2 width exactly; width 8 -> 12 here vs 16 there, both
  >= the required span of draft + ratio - 1 = 11 and inside this tree's
  guard draft <= ratio*(groups-1)+1 = 9).
* INTERNALS of the untaken remedy -- 3 of 35 (test_pool_allocates_the_ring,
  max_rows/ring 1/4, 4/8, 8/16): skipped INDIVIDUALLY with reason. It
  asserts the pool attribute qsa_ring_size and sgl-project#46125's power-of-two
  rounding, including a ONE-group ring at plain decode -- sgl-project#47051 instead
  keeps the pool's qsa_num_groups derivation with a two-group floor (8
  slots at max_rows 1 and 4 -- equal at the prod 4/1/4 geometry -- 12 at
  max_rows 8). These are remedy-choice layout facts, not key-corruption
  facts.

Adaptations of the harness to THIS tree (mechanical, asserted behavior kept):
- _pool: qsa_max_rows_per_request= (the untaken remedy's keyword) ->
  qsa_num_groups=QSATokenToKVPool.pending_ring_num_groups(...) -- this
  tree's sizing entry point; same window-fits guarantee.
- _graph_metadata: also supplies write_owner_mask (bool) and
  decode_block_positions (-1-int32, mirrors the backend capture buffers),
  which THIS tree's row-metadata kernel writes on every row (entry-200
  cache-sharding port; sgl-project#46125's upstream base kernel has neither).
- Added the repo-convention module-level skipif(not cuda) so the CPU
  batteries collect this file cleanly (paired test_qsa_indexer.py carries
  the same mark); upstream's file relied on being CI-routed only.
- launch_graph_metadata / QSAIndexerMetadata / QSAIndexer / backend calls:
  verified signature-compatible with this tree (entry 184 read-before-store
  and entries 212/213 paged-launch/owner-bound ports included) -- no test
  logic changed for them.

Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv): 35 tests collected, 35
skipped here (module CUDA gate) -- this file and every device-cuda qsa
suite are GPU-window DEBT until a CUDA host runs them; the skip reason of
the 3 internalized tests distinguishes them on that run. Ruff v0.15.1 repo
ruleset + format clean; check_registered_tests rc=0; the untouched
test_qsa_pending_ring.py (179 CPU lines of the PR, pinning the
sgl-project#46125-internal ring_size keyword of the slot builders and rings that
"lose members") is deliberately NOT ported -- it is internals of the untaken
remedy end to end, superseded for CPU purposes by this tree's
test_qsa_ring_slots.py plus the sgl-project#47051 test ported in the previous commit.

(cherry picked from commit 85cb8d0)
(cherry picked from commit fc1b2182e9d5f9ca6a5a9eb7b83c35d47903e7cc)
detain added a commit to detain/sglang that referenced this pull request Oct 29, 2026
qsa: Carry QSA indexer state through cpu_tensor retraction

QSATokenToKVPool's retraction backup moved the compressed-K rows with the
KV but NOT the per-request pending index-K ring: Req.offload_kv_cache
snapshots through get_cpu_copy and load_kv_cache restores into a
DIFFERENT request-pool row (verified in schedule_batch.py:
offload/load pass req_pool_index=self.kv.req_pool_idx on both ends), so a
resumed-after-retraction request runs with ANOTHER request's compressed
index keys + pending ring -- the same stale-sidecar defect class as
sgl-project#39830, on the retraction path.

INTERACTION WITH THE sgl-project#47051 RING WIDENING (previous-but-one commit) --
the union this port had to resolve: the PR sizes one request's ring as
``req * ratio + position % ratio`` and transfers ``ratio`` rows. This
tree's ring is ``ratio * qsa_num_groups`` rows with the two-group floor,
so _qsa_ring_rows() takes the request's WHOLE ring, tracking
get_qsa_pending_state_buf_infos' rows_per_item (the PD-transfer contract)
rather than one group. Restoring only the first group would leave the
second ring group holding the previous row owner's keys -- with the
widened ring a verify-pending group's members may live in EITHER ring
group, so a one-group carry would reintroduce exactly the cross-step key
corruption sgl-project#47051 fixes, for every post-retraction request.

UNIONS / DEVIATIONS vs the PR diff:
- The PR (against upstream, where QSATokenToKVPool had NO override) adds
  get_cpu_copy/load_cpu_copy carrying a (parent, (compressed, ring))
  tuple with its own _qsa_compressed_slots dedup gather. Our entry-25
  port already shipped the compressed-K leg in the
  {"kv", "qsa_compressed_k"} dict form with one-row-per-token gathers and
  a legacy-tuple tolerance path; that shape is kept (it pairs with itself
  through RetractionBackup.cpu_tensors) and the PR's NEW content -- the
  ring gather/restore, the req_pool_index + local-QSA-layer gating, and
  the per-axis test coverage -- is grafted onto it as
  "qsa_pending_ring": (per-layer keys, rope) | None.
- Sync plumbing: our compressed leg already used blocking .to("cpu") /
  index_copy_; the ring leg keeps the PR's non_blocking gathers with one
  current_platform.synchronize() per direction (the PR head c022f8a
  had already dropped two of its four syncs as redundant; ours are placed
  inside the ring gates, where the only async copies are).
- _qsa_ring_rows widened to ratio * qsa_num_groups (the interaction
  above; the PR's single-group form).
- The stale entry-25 NOTE on get_qsa_key_state_buffer ("deliberately not
  carried ... rewrites the ring from scratch") is REPLACED by the PR's
  finding -- its premise fails because the resume lands on a new request
  row.
- Tests: the PR's new suite (test_qsa_retraction_backup.py, head
  c022f8a: CustomTestCase, partial-last-group with reordered pages and
  a new request row, PP-stage-without-QSA-layers, empty indices) is
  ported into OUR existing CPU file test_qsa_retraction_cpu_copy.py
  (register_cpu_ci base-a-test-cpu already there; CustomTestCase
  already used) rather than adding a second retraction suite. Payload
  shape adapted tuple-index -> dict-key (backup[1][1] ->
  backup["qsa_pending_ring"]); ring row arithmetic widened to
  RING_ROWS = RATIO * NUM_GROUPS with every ring row, group and RoPE axis
  carrying its own value; the PR's compressed-dedup gather stays our
  one-row-per-token form. The PR's first test (resumed request keeps its
  qsa state) already exists here for the compressed leg; its ring half
  ports as test_resumed_request_keeps_its_pending_ring covering BOTH
  ring groups.
- PP-stage gate kept verbatim in logic: with no local QSA layer the ring
  leg is None and the shared RoPE ring is neither read nor written
  (matches the PD-registration rationale in
  get_qsa_pending_state_buf_infos).

Verified CPU (CUDA_VISIBLE_DEVICES=9, prod venv, PYTHONPATH=this
worktree; no GPU code run):
test_qsa_retraction_cpu_copy 3P -> 7P (3 baseline +4; ring tests go RED
(2F) with the ring gather gated off -- teeth proven, then restored).
Battery re-run green on this tree: test_qsa_kv_pool 8P,
test_qsa_ring_slots 12P/22sub, test_qsa 84P/140S (device-cuda skips),
test_qsa_fused_kv_prepare 12P/146S, test_disaggregation_wire 79P/70sub,
triton_load_watch_prewarm 30P, verify_ring oracle 35S (CUDA gate);
combined six-file run 202P/286S/92sub; unit/mem_cache+unit/spec sweep
2384P/3037S/4149sub; kernels/ops/qsa collect 571 tests;
check_registered_tests rc=0; ruff v0.15.1 ruleset+format clean.
GPU-window debt (unchanged): every device-cuda qsa suite including the
sgl-project#46125 oracle and the paged-vs-packed torch.equal oracle only COLLECTS
on this box.

(cherry picked from commit f3de461 +
review follow-up c022f8a)
(cherry picked from commit 01d6cdfdbd3e634caee0ffbb5e09386b987c1693)
amd-cgilli added a commit to amd-cgilli/sgl-amd-mono that referenced this pull request Oct 29, 2026
With SGLang's 4-slot index-key ring, a verify row past the pending group
overwrites one of its members, and a rejected row's key stays there; the
8-slot ring of sgl-project/sglang#47051 removes it. Say so at the ring-width
check and in the README, which still described a 4-entry ring.
chuyeh pushed a commit to sgl-project/sgl-amd-mono that referenced this pull request Oct 30, 2026
…nokernel (#29)

## Motivation

Qwen3.8-Flash-Next (SGLang's Qwen4-Exp,
`amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8`) decodes with its
MTP head (NEXTN) at small batch sizes, where launch gaps dominate. This
PR adds a FlyDSL monokernel for it to sgl-amd-mono, behind SGLang's
monokernel interface (RFC #43949). `--monokernel-backend amd` selects
it, and SGLang needs no model hooks. The server runs SGLang's AITER path
(`SGLANG_USE_AITER=1`; the full launch command is in the README).

- A verify step of 1-8 requests x 4 tokens (embedding, PLE, 48 GDN / QSA
layers and the final mixer) runs in 13 persistent launches.
- Each MTP draft step of 1-8 requests runs in one launch: a draft extend
(R x 4 rows) or a draft decode step (R x 1 rows).

Monokernel API v1 routes every runner of a speculative server by the
target's verify key, which the NEXTN draft never runs. The draft spec
therefore routes its own steps: a wrapper in front of the draft's
`model.forward`. It can move to SGLang if a later monokernel API routes
a draft's own step modes.

The kernel itself (launch plan, stages, LDS and scratch layout,
synchronization and fail-stop, weights, numerics) is described in
[`sgl_amd_mono/qwen38_flash_next/README.md`](https://github.com/amd-cgilli/sgl-amd-mono/blob/qwen38-flash-next-monokernel/sgl_amd_mono/qwen38_flash_next/README.md).

Below is a comparison of the before and after:
<img width="1612" height="727" alt="b4"
src="https://github.com/user-attachments/assets/85b581aa-d309-4fd5-8774-24fcb2d810dc"
/>
<img width="1272" height="669" alt="after"
src="https://github.com/user-attachments/assets/2834498c-02a2-4546-b9c4-25d51567dfdb"
/>





## Modifications

- `sgl_amd_mono/qwen38_flash_next/`:
- `spec.py`: the target spec, registered for
`Qwen4ExpForConditionalGeneration` on gfx950, TP1, 1-8 requests x 4
tokens. It refuses unsupported configurations with reasons:
    - NEXTN/EAGLE with topk 1, 3 steps and 4 draft tokens only;
    - no ReplaySSM, no PP speculative scratch remapping;
    - bf16 KV, no HiSparse;
- `SGLANG_USE_AITER`, the AITER/auto MoE runner, the fused shared
expert, no paged experts or MegaMoE, no expert simulation or recording;
    - the Quark MXFP4 checkpoint;
    - `--attention-backend` triton or aiter;
    - plus per-layer config and weight-layout checks.
- Image and video requests: a verify step is rotated by SGLang's MRoPE
positions (`forward_batch.mrope_positions`, as the model's own forward
does), and each request's sequence start comes from `seq_lens`. With
images or video the two differ, and the QSA index tiles, ring slots and
selection count from the sequence position.
- QSA index-key ring: the launches index SGLang's QSA key ring by the
width its pool allocates
(`ring_width` in `gfx950/runner.py`: 4 or 8 slots a request). With
today's 4-slot ring (one group),
a verify row past the pending group overwrites one of its members, and a
rejected row's key stays
in the slot, so the group can later compress with it; SGLang's own
verify has the same issue.
sgl-project/sglang#47051 makes the ring two groups (8 slots), which
keeps every pending member. The launches work with either layout.
- `draft.py`: the MTP draft spec, registered for
`Qwen4ExpForCausalLMMTP`.
- `_build` installs a wrapper in front of the draft's `model.forward`.
- The wrapper sends draft extend (`DRAFT_EXTEND_V2`, R x 4 rows) and
draft decode (`DECODE`, R x 1 rows) steps of R <= 8 requests to the
draft launch, and every other step to SGLang's forward.
- The decision depends only on the step's shape, so a captured graph
replays it.
- It requires the draft's QSA attention backends (`"qsa"`), logs each
draft step at its first launch, and refuses during capture a
draft-shaped step the launch cannot read.
  - `check.py`: `SGLANG_AMD_MONO_CHECK=1` checks the target only:
    - at build, it compares the layer weights against the checkpoint;
- on each eager verify step, it compares the logits and hidden states
against SGLang's forward of the same step (`NUMERICS`).
  - `gfx950/`:
- the kernel (`kernel.py`, the guarded wait in `device.py`), its layout,
packing and config;
    - the layer views read from SGLang's params in place (`weights.py`);
- the host-side runners (`runner.py`). The draft runner copies
`input_ids` and new hidden states into buffers it owns, so its pointer
tables stay valid under graph capture.
  - `README.md`: the kernel documentation.
- The specs import SGLang's interface from
`sglang.kernels.plugin_kernels.amd` and fall back to
`sglang.srt.monokernels` (SGLang before sgl-project/sglang#45389), as
the other specs do.
- `sgl_amd_mono/__init__.py`: SPECS entries for the target and the MTP
draft.
- `README.md`: Supported Models, Usage, Structure, Testing.
- Tests:
- `test/unit/test_qwen38_flash_next_{spec,layout}.py`, CPU. These cover
the specs, refusals, layout, the RoPE positions a routed verify passes,
and the draft wrapper:
    - routing of draft extend and decode steps;
    - the fallback to SGLang's forward;
    - stacking under the runtime's forward switch;
    - the per-mode backend check;
    - the capture-time refusal;
    - the fixed-address input buffers.
  - `test/kernels/test_qwen38_flash_next_compile.py`, GPU.
-
`test/e2e/{qwen38_flash_next_server,test_qwen38_flash_next_server}.py`,
GPU:
    - GSM8K at 1/4/16 requests, accept length;
    - the target and draft launch lines;
    - KL decode vs prefill;
    - 126k passkey;
    - check mode;
    - fault injection (target launches).

## Accuracy Tests

1 x MI355X, TP1, SGLang upstream main 2229345dff, NEXTN 3 steps / 4
draft tokens / topk 1, real acceptance. "Off" is the same server without
`--monokernel-backend amd`. Measured before the rebase onto current
main. The rebase and the cleanup leave the kernels' LLVM IR unchanged,
and the MRoPE change reads the same sequence start for text requests.

| Test | Monokernel on | Off |
|---|---|---|
| GSM8K, 400 q | 0.970 (accept length 3.52) | 0.975 (accept length 3.52)
|
| GSM8K 5-shot chat, 200 q, at 1 / 4 / 16 requests | 0.98 / 0.98 / 0.97
| 0.98 / 0.985 / 0.975 |
| AIME25, 30 q x 4 seeds, pass@1 (thinking on, T 0.6, top-p 0.95) |
96.67% (116 of 120) | 98.33% (118 of 120) |
| AIME25, 30 q x 4 seeds, pass@4 | 100% | 100% |
| KL decode vs prefill, mean over fresh servers (bound 0.1) | 0.062 (5
runs, 0.041 to 0.085) | 0.059 (4 runs, 0.051 to 0.069) |
| RULER NIAH at ~261K tokens (niah_single_1/2/3, niah_multikey_1, 25
each) | 100 | 100 |
| GPQA-Diamond pass@1 (same settings) | 0.855 (sem .006) | 0.836 (sem
.004) |
| GPQA-Diamond majority@4 / pass@4 | 0.871 / 0.909 | 0.884 / 0.899 |

<details>
<summary>Unit + contract tests (CPU): 102 passed</summary>

```text
date:        2026-10-29 09:50 UTC
sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
sglang from:  /sgl-workspace/sglang-main-latest/python/sglang
versions:     torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0
GPU:          1 x AMD Instinct MI355X (gfx950)

$ python -m pytest -v -p no:cacheprovider test/unit/test_qwen38_flash_next_spec.py test/unit/test_qwen38_flash_next_layout.py test/unit/test_contract.py -k qwen38

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python
rootdir: /sgl-workspace/sgl-amd-mono
configfile: pyproject.toml
plugins: platformdirs-4.13.5, anyio-4.15.1
collecting ... collected 136 items / 34 deselected / 102 selected

test/unit/test_qwen38_flash_next_spec.py::test_the_plugin_registers_the_target_and_the_draft PASSED [  0%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_spec_declares_the_drafts_own_backend PASSED [  1%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_keeps_the_forward_signature PASSED [  2%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-1-4] PASSED [  3%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-8-32] PASSED [  4%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-1-1] PASSED [  5%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-8-8] PASSED [  6%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-1-4] PASSED [  7%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-8-32] PASSED [  8%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-1-1] PASSED [  9%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-8-8] PASSED [ 10%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests] PASSED [ 11%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests extend] PASSED [ 12%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-verify] PASSED [ 13%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-extend] PASSED [ 14%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests] PASSED [ 15%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests extend] PASSED [ 16%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-verify] PASSED [ 17%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-extend] PASSED [ 18%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_with_input_embeds_is_refused PASSED [ 19%]
test/unit/test_qwen38_flash_next_spec.py::test_the_runtimes_forward_wraps_the_draft_wrapper PASSED [ 20%]
test/unit/test_qwen38_flash_next_spec.py::test_the_wrapper_checks_the_backend_once_a_step_mode PASSED [ 21%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[three rows a request] PASSED [ 22%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[fp32 hidden] PASSED [ 23%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[hidden of other rows] PASSED [ 24%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[strided hidden] PASSED [ 25%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[no hidden states] PASSED [ 26%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[qsa] PASSED [ 27%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[hybrid] PASSED [ 28%]
test/unit/test_qwen38_flash_next_spec.py::test_every_draft_call_site_calls_the_model_forward_attribute PASSED [ 29%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_writes_the_buffer_its_input_does_not_overlap PASSED [ 30%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_reads_new_hidden_states_from_a_fixed_address PASSED [ 31%]
test/unit/test_qwen38_flash_next_spec.py::test_build_installs_the_wrapper PASSED [ 32%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_worker_reads_index_sharing_off_after_the_build PASSED [ 33%]
test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_is_accepted PASSED [ 34%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[hidden size] PASSED [ 35%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[experts] PASSED [ 36%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[index budget] PASSED [ 37%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[layer pattern] PASSED [ 38%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[quantization] PASSED [ 39%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[ssm dtype] PASSED [ 40%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[context] PASSED [ 41%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[no flydsl] PASSED [ 42%]
test/unit/test_qwen38_flash_next_spec.py::test_the_ple_layer_is_one_reason PASSED [ 43%]
test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_routes_verify_batches_of_one_to_eight PASSED [ 44%]
test/unit/test_qwen38_flash_next_spec.py::test_the_requirements_are_declared_sglang_reads PASSED [ 45%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[two tokens] PASSED [ 46%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[nine requests] PASSED [ 47%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[decode] PASSED [ 48%]
test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[mrope] PASSED [ 49%]
test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[text positions] PASSED [ 50%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_reaches_the_jit_cache_key PASSED [ 50%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[kind] PASSED [ 51%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[empty] PASSED [ 52%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[requests] PASSED [ 53%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[tokens] PASSED [ 54%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[first source] PASSED [ 55%]
test/unit/test_qwen38_flash_next_spec.py::test_a_verify_is_thirteen_launches_of_the_48_layers PASSED [ 56%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[1] PASSED [ 57%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[2] PASSED [ 58%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[3] PASSED [ 59%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[4] PASSED [ 60%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[5] PASSED [ 61%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[6] PASSED [ 62%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[7] PASSED [ 63%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[8] PASSED [ 64%]
test/unit/test_qwen38_flash_next_spec.py::test_the_layout_matches_the_spec PASSED [ 65%]
test/unit/test_qwen38_flash_next_spec.py::test_the_views_accept_the_layers PASSED [ 66%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[no fused shared expert] PASSED [ 67%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[unshuffled experts] PASSED [ 68%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[interleaved gate and up] PASSED [ 69%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[2-byte scales] PASSED [ 70%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[dequantized projection] PASSED [ 71%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[wide b / a projection] PASSED [ 72%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[bf16 projection] PASSED [ 73%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[conv weight] PASSED [ 74%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[f32 norm] PASSED [ 75%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[index projection] PASSED [ 76%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[q norm] PASSED [ 77%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[router] PASSED [ 78%]
test/unit/test_qwen38_flash_next_spec.py::test_an_offloaded_ngram_table_is_refused PASSED [ 79%]
test/unit/test_qwen38_flash_next_spec.py::test_the_model_is_accepted PASSED [ 80%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_model_is_one_reason PASSED [ 81%]
test/unit/test_qwen38_flash_next_spec.py::test_the_mtp_draft_views_are_checked PASSED [ 82%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[2560-6144] PASSED [ 83%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[16384-2560] PASSED [ 84%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[128-2560] PASSED [ 85%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[640-2560] PASSED [ 86%]
test/unit/test_qwen38_flash_next_layout.py::test_moe_padding_matches_fusedmoe PASSED [ 87%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle_moe_expert_inverts_fusedmoe_layout PASSED [ 88%]
test/unit/test_qwen38_flash_next_layout.py::test_check_moe_params_accepts_fusedmoe_shapes PASSED [ 89%]
test/unit/test_qwen38_flash_next_layout.py::test_pack_bf16_tiles_16_rows_by_64_k PASSED [ 90%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 91%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 92%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 93%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 94%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [ 95%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 96%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 97%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 98%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 99%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [100%]

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
  /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead.
    from distutils.dep_util import newer, newer_group

test/unit/test_qwen38_flash_next_spec.py: 14 warnings
  /opt/venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============== 102 passed, 34 deselected, 18 warnings in 8.10s ================
exit code: 0
```

</details>

<details>
<summary>Kernel tests: 7 passed</summary>

```text
date:        2026-10-29 09:50 UTC
sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
sglang from:  /sgl-workspace/sglang-main-latest/python/sglang
versions:     torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0
GPU:          1 x AMD Instinct MI355X (gfx950)

$ python -m pytest -v -p no:cacheprovider test/kernels/test_qwen38_flash_next_launches.py

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python
rootdir: /sgl-workspace/sgl-amd-mono
configfile: pyproject.toml
plugins: platformdirs-4.13.5, anyio-4.15.1
collecting ... collected 7 items

test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=1] PASSED [ 14%]
test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=8] PASSED [ 28%]
test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=7+1 pad] PASSED [ 42%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-extend] PASSED [ 57%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-decode] PASSED [ 71%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-extend] PASSED [ 85%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-decode] PASSED [100%]

=============================== warnings summary ===============================
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
  /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead.
    from distutils.dep_util import newer, newer_group

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================== 7 passed, 2 warnings in 24.13s ========================
exit code: 0
```

</details>

<details>

<summary>Synthetic benchmark C1 8K input / 1K output</summary>

```text
date:        2026-10-29 15:31 UTC
sgl-amd-mono: 24c72df3930a4c6269df157565b83f1b194387ac Turn off crash dumps in the Qwen3.8-Flash-Next fault test
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
GPU:          1 x AMD Instinct MI355X (gfx950)
MTP acceptance: measured (0 SGLANG_SIMULATE_ACC_* in the server env)

# server
$ SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 AITER_FLYDSL_FORCE=1 SGLANG_MAMBA_SSM_DTYPE=bfloat16 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \
  python -m sglang.launch_server --model-path amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8 --port 31060 --tp 1 --attention-backend triton --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-max-bs-decode 8 --max-running-requests 16 --enable-amd-monokernel

# client
$ python -m sglang.bench_serving --backend sglang --port 31060 --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 --num-prompts 20 --max-concurrency 1 --warmup-requests 2 --seed 1

/sgl-workspace/sglang-main-latest/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
  warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None)
Waiting up to 60s for http://0.0.0.0:31060/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8', served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None)

#Input tokens: 163840
#Output tokens: 20480
Starting warmup with 2 sequences...
Warmup completed with 2 sequences. Starting main benchmark run...

  0%|          | 0/20 [00:00<?, ?it/s]
  5%|▌         | 1/20 [00:02<00:38,  2.03s/it]
 10%|█         | 2/20 [00:03<00:35,  1.95s/it]
 15%|█▌        | 3/20 [00:05<00:31,  1.85s/it]
 20%|██        | 4/20 [00:11<00:55,  3.46s/it]
 25%|██▌       | 5/20 [00:13<00:42,  2.87s/it]
 30%|███       | 6/20 [00:15<00:38,  2.73s/it]
 35%|███▌      | 7/20 [00:17<00:31,  2.41s/it]
 40%|████      | 8/20 [00:19<00:27,  2.33s/it]
 45%|████▌     | 9/20 [00:21<00:24,  2.23s/it]
 50%|█████     | 10/20 [00:23<00:21,  2.16s/it]
 55%|█████▌    | 11/20 [00:25<00:19,  2.14s/it]
 60%|██████    | 12/20 [00:27<00:16,  2.02s/it]
 65%|██████▌   | 13/20 [00:29<00:13,  1.94s/it]
 70%|███████   | 14/20 [00:31<00:12,  2.02s/it]
 75%|███████▌  | 15/20 [00:33<00:09,  1.94s/it]
 80%|████████  | 16/20 [00:35<00:08,  2.00s/it]
 85%|████████▌ | 17/20 [00:37<00:05,  1.93s/it]
 90%|█████████ | 18/20 [00:38<00:03,  1.88s/it]
 95%|█████████▌| 19/20 [00:40<00:01,  1.85s/it]
100%|██████████| 20/20 [00:42<00:00,  1.83s/it]
100%|██████████| 20/20 [00:42<00:00,  2.13s/it]

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    inf       
Max request concurrency:                 1         
Successful requests:                     20        
Benchmark duration (s):                  42.57     
Total input tokens:                      163840    
Total input text tokens:                 163840    
Total generated tokens:                  20480     
Total generated tokens (retokenized):    20422     
Request throughput (req/s):              0.47      
Input token throughput (tok/s):          3849.17   
Output token throughput (tok/s):         481.15    
Peak output token throughput (tok/s):    686.00    
Peak concurrent requests:                2         
Total token throughput (tok/s):          4330.31   
Concurrency:                             1.00      
Accept length:                           3.57      
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2127.22   
Median E2E Latency (ms):                 1851.48   
P90 E2E Latency (ms):                    2236.08   
P95 E2E Latency (ms):                    2638.79   
P99 E2E Latency (ms):                    5264.40   
---------------Time to First Token----------------
Mean TTFT (ms):                          456.02    
Median TTFT (ms):                        245.55    
P90 TTFT (ms):                           291.58    
P95 TTFT (ms):                           720.12    
P99 TTFT (ms):                           3441.69   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.63      
Median TPOT (ms):                        1.53      
P90 TPOT (ms):                           1.87      
P95 TPOT (ms):                           1.93      
P99 TPOT (ms):                           2.12      
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.63      
Median ITL (ms):                         1.46      
P90 ITL (ms):                            1.92      
P95 ITL (ms):                            2.88      
P99 ITL (ms):                            5.79      
Max ITL (ms):                            6.31      
==================================================

# monokernel lines from the server log
[2026-10-29 15:31:03] monokernel qwen38_flash_next_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 2436.7 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: final mixer packed (2 tensors, 12.8 MiB), gdn layer packed (360 tens
[2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: SGLang's QSA MTP index sharing off (index_share_for_mtp_iteration True -> False), so every draft decode selects its own blocks
[2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: draft extend steps of up to 8 x 4 rows and draft decode steps of up to 8 x 1 are routed by the spec's wrapper of the draft's forward, and one it cannot launch raises; each key logs 'MTP draft runs' at its first launch
[2026-10-29 15:31:09] monokernel qwen38_flash_next_mtp_draft_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 543.0 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: mtp packed (9 tensors, 66.0 MiB)
[2026-10-29 15:31:24] monokernel qwen38_flash_next_hip on gfx950 runs target_verify steps of batch size [1, 2, 3, 4, 5, 6, 7, 8] x 4 tokens (8 of 8 captured decode graphs); every other step runs the standard path
[2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 7, 7)
[2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 8, 8)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 4, 4)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 5, 5)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 6, 6)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 1, 1)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 2, 2)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 3, 3)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 5, 20)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 6, 24)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 7, 28)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 8, 32)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 2, 8)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 3, 12)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 4, 16)
[2026-10-29 15:31:31] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 1, 4)
```
</details>

## Speed Tests and Profiling

1 x MI355X, TP1, the same server flags with and without
`--monokernel-backend amd`, CUDA graphs up to 8 requests. Verify and
draft are both on the monokernel.

Client:

```bash
python -m sglang.bench_serving --backend sglang --dataset-name random \
  --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \
  --num-prompts <max(10 x conc, 20)> --max-concurrency <conc> --warmup-requests 2 --seed 1
```

| C | Off: median TPOT / output tok/s | On: median TPOT / output tok/s |
Speedup: TPOT [95% CI] / tok/s | Accept length off / on |
|---|---|---|---|---|
| 1 | 4.04 ms / 220 | 1.60 ms / 517 | **2.53x** [2.28, 2.84] / 2.35x |
3.36 / 3.38 |
| 2 | 4.58 ms / 400 | 2.17 ms / 812 | **2.12x** [1.89, 2.26] / 2.03x |
3.39 / 3.43 |
| 4 | 4.99 ms / 669 | 3.21 ms / 1063 | **1.56x** [1.51, 1.71] / 1.59x |
3.39 / 3.40 |
| 8 | 6.22 ms / 1088 | 5.86 ms / 1180 | **1.06x** [1.02, 1.09] / 1.08x |
3.38 / 3.40 |

## Checklist

- [x] `pre-commit run --all-files` passes.
- [x] Tests under `test/` cover the change.
- [x] The SGLang commit tested against: upstream main 2229345dff
(accuracy and speed), f3f0f7b68c (C=1 re-check), f3f0f7b68c and
af21e2b989 (unit tests).
- [x] Accuracy and speed results.
- [x] `.github/CODEOWNERS`: `**/*qwen38_flash_next*` is already in
upstream's CODEOWNERS.
@esarafian

Copy link
Copy Markdown
Contributor

Here are NVIDIA results for this PR: SM120 and SM100, to go with your MI355X numbers.

We ran the PR head f78ee79014 as authored, with image lmsysorg/sglang:nightly-dev-20261103-797af717:

GPU test_qsa.py of which never_overwrite (new) test_qsa_fused_kv_prepare.py
RTX PRO 6000 Blackwell Server (SM120) 164 passed, 1 skipped 8 passed 174 passed
GB200 (SM100) 164 passed, 1 skipped 8 passed 174 passed

We measured a similar change in a downstream build on a Qwen3.8-Flash-Next-based checkpoint with NEXTN 3/1/4, in real serving on RTX PRO 6000. The share of decode-formed compressed keys that differed from a prefill of the same tokens went from about 82% to 0 of 1.78M. Step time and acceptance were unchanged (95.26 vs 94.98 ms/step at 96 concurrent requests, accept length 3.161 vs 3.151).

For reviewers comparing with #46125, which fixes the same bug: at ratio 4 with up to 5 verify rows, both PRs use an 8-slot ring per request (2 × ratio here; the next power of two ≥ ratio + rows − 1 in #46125). #46125 also adds a 35-case GPU harness (test_qsa_verify_ring.py) that checks the compressed cache bit-for-bit against prefill. That harness also passes on SM120 and SM100. It might be worth running it against this layout too, before one of the two is picked.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants