Skip to content

[Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration - #45389

Merged
BBuf merged 21 commits into
sgl-project:mainfrom
BBuf:bbuf/low-latency-kernels
Oct 24, 2026
Merged

BBuf merged 21 commits into
sgl-project:mainfrom
BBuf:bbuf/low-latency-kernels

Conversation

@BBuf

@BBuf BBuf commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Some kernels are specialized for one model, configuration and platform, and replace a model's whole step. They come from external libraries that load as SGLang plugins:

Today these plugins serve small-batch decode and verify steps. Nothing ties them to low latency or to "mega" fusion; prefill or high-throughput specializations would plug in the same way.

SGLang's side of each integration lived in srt/megakernels and srt/monokernels. This PR splits that code at the kernel/runtime boundary:

  • what a plugin implements and registers moves to kernels/plugin_kernels/<hardware>/, recorded by a new KernelBackend.PLUGIN (as with kda_kernels/ and cake_kernels/)
  • SGLang's serving integration stays in srt/, as srt/kernel_plugins/<hardware>/

The kernels themselves stay in their libraries.

Modifications

python/sglang/kernels/plugin_kernels/   # the contract a plugin implements; imports nothing from srt
  nvidia/spec.py                        #   registration, API version, requirements, capabilities,
  amd/spec.py                           #   widths, build/forward contract
                                        #   (<hardware>/<model>/ for thin kernel wrappers later)
python/sglang/srt/kernel_plugins/       # SGLang's integration, one binding.py per hardware
  nvidia/binding.py                     #   startup checks on the resolved config, build, TP agreement,
  amd/binding.py                        #   CUDA graph coverage, forward dispatch with fallback,
                                        #   weight observation (AMD), the runtime state handed to a plugin
  • Move. The code moves unchanged; only imports are combined.
    • srt/{megakernels,monokernels}/spec.py → kernels/plugin_kernels/{nvidia,amd}/spec.py.
    • Each hardware's guards.py + runtime.py (and, on NVIDIA, binding.py) → srt/kernel_plugins/<hardware>/binding.py.
    • Unit tests → test/registered/unit/kernel_plugins/{nvidia,amd}/. They now patch the one binding module.
  • No aliases. sglang.srt.megakernels and sglang.srt.monokernels are removed.
  • KernelBackend.PLUGIN. Adds the enum value and a forward_plugin hook on BaseFusedOp, placed after KDA/CAKE in DEFAULT_PRIORITY. The namespace test pairs plugin_kernels/ with this backend.
  • --monokernel-backend amd → --enable-amd-monokernel. The flag becomes a boolean next to --enable-nvidia-megakernel.
    • --monokernel-backend amd stays as a deprecated spelling: it warns and sets the new flag. Any other value is rejected.
    • The field keeps its slot in POSITIONAL_FIELD_ORDER.
    • Both spellings also work in --config YAML files. Previously the merger accepted only store/store_true actions, which rejected the deprecated alias and, through the shared dest, the new key too.
      • The merger now passes single-value actions as --flag value.
      • A key stays usable when a supported flag also reads it.
      • A regression test covers monokernel-backend: amd, enable-amd-monokernel: true / false, and both CLI spellings. Before the fix, the three YAML cases raise ValueError.
    • The resolution hook is renamed handle_amd_monokernel.
    • Error messages, the server-args doc, the DeepSeek-V4.1 cookbook and the ROCm Dockerfile comment are updated.
  • Docs. kernels/README.md, plugin_kernels/README.md and the kernel-organization skill describe the split.

No runtime behavior changes: the default path is untouched, and the megakernel and monokernel code is the same code at new paths.

Not covered: the MSA plugin. The MiniMax-M3.1 cookbook (#45924) serves the MSA plugin with --monokernel-backend msa. On this PR that value is rejected, and the plugin's sglang.srt.monokernels imports fail. That plugin is moving off the monokernel name separately.

Accuracy Tests

All runs: 4x GB300 (sm_103), lmsysorg/sglang:dev-cu13, plugins at the pinned commits (sgl-nvidia-mega 6fd7ac3, sgl-amd-mono 05e0bb0).

The PR content was tested at 7311b92f02, which has main 6e667c4310 merged. The head has main 3ea7e210d2 merged instead; that merge changed nothing in this PR except resolving the NVIDIA pin conflict.

Tests and plugin loading

  • 43 CPU-relevant test files, each run in its own process: unit/kernels/**, unit/server_args/**, unit/model_loader/**, the moved tests, test_dspark_draft_block, test_runtime_context and others.
    • Main 6e667c4310 and this PR fail the same single case.
    • The 5 moved test files pass.
  • Plugins through their real entry points.
    • On this PR, sglang.srt.megakernels is gone, and both plugins register from the new locations.
    • On main, they register from the old ones.
    • Either way, the NVIDIA DeepSeek-V4.1 target and draft specs resolve to MegakernelSpec, and all six AMD specs resolve on gfx950.
  • The plugins' own test/unit.

The real megakernel, end to end

DeepSeek-V4.1-Flash, served with the plugin README's command (TP4, DSpark block 5, --enable-nvidia-megakernel and the three required env vars):

main b453996b7d this PR on b453996b7d (48b0063ff6)
NVIDIA megakernel: active (target and DSpark draft specs, 4 ranks) 8 lines 8 lines
draft proposal engaged during graph capture 108 108
GSM8K, sgl-eval 0.1.3, 200 examples, temperature 1.0, top_p 0.95, no thinking, seed 0 0.985 0.990

On main 6e667c4310, the same command fails with or without this PR, identically:

  • Both activate (8 lines).
  • Both then fail during the first verify-graph capture, also with --max-running-requests 8 --cuda-graph-max-bs-decode 8. SGLang's DeepSeek-V4 indexer, reached through view.low_ratio_sources, hits assert A.shape[:-1] == As.shape[:-1] in the fp8 wq_b GEMM.

A partial bisect puts that regression between bb9e9d5632 (good) and b588aa39e9 (bad, #45921). The head after the latest main merge was not run on GPU.

Not run: monokernel serving on gfx950. The AMD side was checked statically and on CPU only.

Speed Tests and Profiling

N/A: no kernel or forward change.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #40090470519
Latest PR Test (Extra): ❌ Run #40090470482
Latest PR Test (AMD ROCm 10): ❌ Run #40090470545

BBuf and others added 5 commits October 20, 2026 20:11
…o low_latency_kernels

Move sglang.srt.megakernels and sglang.srt.monokernels, with their unit tests,
to sglang.kernels.low_latency_kernels; only import paths change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pinned sgl-nvidia-mega and sgl-amd-mono import sglang.srt.megakernels and
sglang.srt.monokernels; alias the modules they import to the moved ones so the
installed plugins keep loading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KernelBackend.LOW_LATENCY records kernels in low_latency_kernels, with a
forward_low_latency hook on BaseFusedOp; the namespace test pairs the directory
with the backend as it does for kda_kernels and cake_kernels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ernel

amd is the only monokernel backend, so the flag becomes a boolean beside
--enable-nvidia-megakernel; --monokernel-backend amd stays as a deprecated
spelling of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docker/Dockerfile
#	docker/Dockerfile.cu134
@BBuf

BBuf commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

Reverted the contract rename: back to plugin_kernels/{nvidia,amd} and --enable-amd-monokernel, which the merged plugin pins (#21, #25) already import. The MSA plugin will move off the monokernel name on its side. Also merged main; the only conflict was main's NVIDIA pin (3333e4c), which I kept at 6fd7ac3 (it contains 3333e4c).

@BBuf BBuf added the run-ci CI: run the baseline test suite on this PR label Oct 24, 2026
@BBuf

BBuf commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test -c test_dspark_draft_block.py test_dspark_step_inputs_once.py test_dspark_draft_graph.py test_startup_weight_load.py

@BBuf

BBuf commented Oct 24, 2026

Copy link
Copy Markdown
Collaborator Author

/run-cpu-ci

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

🚀 ubuntu-latest (8 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dspark_draft_block.py
cd test/ && python3 registered/unit/kernel_plugins/amd/test_mono_contract.py
cd test/ && python3 registered/unit/kernel_plugins/amd/test_mono_framework.py
cd test/ && python3 registered/unit/kernel_plugins/nvidia/test_megakernel_binding.py
cd test/ && python3 registered/unit/kernel_plugins/nvidia/test_megakernel_guards.py
cd test/ && python3 registered/unit/kernel_plugins/nvidia/test_megakernel_runtime.py
cd test/ && python3 registered/unit/kernels/test_kernels_namespace.py
cd test/ && python3 registered/unit/server_args/test_server_args.py

🚀 1-gpu-5090 (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dspark_step_inputs_once.py
cd test/ && python3 registered/spec/dspark/test_dspark_draft_graph.py

⛔ test_startup_weight_load.py: Ambiguous filename test_startup_weight_load.py — matched 2 files:

  • test/registered/model_loading/test_startup_weight_load.py
  • test/registered/unit/model_executor/model_runner_components/test_startup_weight_load.py

Please provide the full path, e.g. /rerun-test test/registered/unit/model_executor/model_runner_components/test_startup_weight_load.py

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Running base-a-test-cpu on bb5df5520e52: View workflow run

@BBuf
BBuf merged commit b86d294 into sgl-project:main Oct 24, 2026
133 of 149 checks passed
@BBuf
BBuf deleted the bbuf/low-latency-kernels branch October 24, 2026 12:29
nvrohanv added a commit to nvrohanv/sgl-nvidia-mega that referenced this pull request Oct 25, 2026
…arget verify and NextN draft on B300 TP8

Add sgl_nvidia_mega/glm5_3: the GLM-5.3 target spec for SGLang's megakernel
layer (MTP verify steps of 1 to 4 requests, 4 to 16 rows, on 8 x B300 at
TP 8), its step functions with stock paths adapted from SGLang's model code,
and the CuTe DSL attention, small-M NVFP4 MoE and all-reduce hop kernels.
With --enable-nvidia-megakernel every target slot runs (attention, moe,
hops, qkv_a_small_m_shared, dnorm); SGLANG_NVIDIA_MEGAKERNEL_SLOTS is a test
and bisect override. Needs SGLang's DSA binding view.

Add the GlmMoeDsaForCausalLMNextN draft spec (slot draft, on by default) and
its CuTe DSL kernels: the embedding gather and eh norms in one kernel, the
column-sharded eh projection gathered by the target's all-reduce hop, the
distributed top-1, and the draft layer's MLA prologue, in every captured
draft graph. Needs embedding replication (with the flag, SGLang's megakernel
layer sets SGLANG_ENABLE_EMBED_REPLICATION when it is unset), greedy draft
acceptance and SGLang's EAGLE draft graph capture check; the contract test
skips the check's marker, naming it, on an SGLang without it.

At startup rank 0 logs the covered verify widths (1-4 requests) and the
derived weight copies' memory per GPU, and warns when
--max-running-requests is set above 8, where the megakernel helps little.

Also the README section, a NOTICE paragraph, a CODEOWNERS line and the
contract test's checks of the DSA binding view reads and the mirrored SGLang
code; the GLM-5.3 tests are filed separately.

The same specs serve GLM-5.2 NVFP4, which has GLM-5.3's architecture
(GlmMoeDsaForCausalLM) and the same value of every config field the target
and draft specs pin. The spec names and the plugin's log, refusal and rank
agreement texts name both models, and the module path stays glm5_3.
test/unit/test_glm5_3_glm5_2_config.py runs the published config of
nvidia/GLM-5.2-NVFP4 (revision 53e0691e21895a3863a606dfd12910c69eba94ab)
through SGLang's requirement check for both specs, and checks that a config
that differs in any pinned value, or lacks one, is still refused.

The package imports SGLang's megakernel interface from its location since
sgl-project/sglang#45389 (sglang.kernels.plugin_kernels.nvidia,
sglang.srt.kernel_plugins.nvidia) and falls back to sglang.srt.megakernels
on an older SGLang, as deepseek_v41 does.
amd-cgilli added a commit to amd-cgilli/sgl-amd-mono that referenced this pull request Oct 26, 2026
Drop unused helpers, parameters and constants and the provenance comments,
trim comments that restated the code, and import SGLang's monokernel interface
from its new location first (sgl-project/sglang#45389), as the other specs do.
The kernels compile to the same IR.
jhinpan pushed a commit to sgl-project/sgl-amd-mono that referenced this pull request Oct 27, 2026
…s wait deadline (#61)

## Motivation

Since #23, a routed decode step starts with layer 0's launch, which
takes the step's tokens and sums the embedding rows over the ranks. So
no RCCL collective, which has no deadline, meets the ranks before the
step's first wait on a peer. A captured graph runs a step's launches
back to back, so any host work between steps lands in that first launch.
Its waits have the same deadline as every other wait: half of
`--watchdog-timeout`, at most 60 s.

A cold SGLang JIT cache creates such host work. The first real decode
builds `clamp_position` (`ForwardBatch.init_new` → `load_jit`) under an
`fcntl.flock` in `~/.cache/sglang/jit`. When that directory is on NFSv3
with `local_lock=none`, a rank blocked on the lock wakes only on its
next 30 s poll. The ranks then reach their first routed step 0, 30 or 60
s apart. Past 60 s, the early ranks' first wait on a peer times out and
the guard stops every GPU: a false fail-stop.

On main, 3 of 5 cold boots died this way during the server's own warmup
request. The GPUs faulted at 0x0 in layer 0's launch
(`glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1`). A host stall of more
than 60 s between steps, on any one rank, does the same.

## Modifications

- `spec.py`: the runtime that runs layer 0 gets `min(max(ctx.deadline_s,
0.5 * watchdog_timeout), guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ)` as
its wait deadline. That runtime is the one with `key.tokens`, and layer
0 is always a step's first launch, since `_build` refuses a layer 0 that
is not dense. Every other runtime keeps `ctx.deadline_s`.
- The new deadline is half the watchdog, SGLang's own bound for a wait,
without its 60 s cap. So a wait that never ends still stops every GPU
before the watchdog fires.
- It is at most 5,497 s, the longest whole second that a wait's 32-bit
deadline check holds (`guard.MAX_DEADLINE_TICKS`, 5,497.56 s). So from
`--watchdog-timeout 10994` on, and at `inf`, layer 0's deadline is 5,497
s, and `guard.deadline_ticks` takes it. Whole seconds keep the cap's
conversion to ticks exact.

  | `--watchdog-timeout` | every other launch | layer 0's launch |
  | --- | --- | --- |
  | 300 s (SGLang's default) | 60 s | 150 s |
  | 600 s (the e2e test's) | 60 s | 300 s |
  | 20 s (the e2e fault test's) | 10 s | 10 s |
  | 11,100 s | 60 s | 5,497 s |
  | `inf` | 60 s | 5,497 s |

- `_build` passes the cap to `_deadline`, so the deadline test runs on a
CPU-only host: `common/guard.py` imports AITER, which needs a GPU to
import.
- The spec reads `get_device().watchdog_timeout` directly and declares
`get_device` and `watchdog_timeout` in `SGLANG_READS`. SGLang computes
`ctx.deadline_s` from the same read: `monokernels/runtime.py` on
ab580b2e06, `kernel_plugins/amd/binding.py` after
sgl-project/sglang#45389.
- No kernel change. The deadline is a word of the launch's guard block,
which the host writes at build (`FlashRuntime.set_deadline`).
- `test/unit/test_glm53_flash_spec.py`:
`test_a_step_s_first_launch_waits_up_to_half_the_watchdog` checks both
deadlines at watchdogs of 300, 600, 20 and 11,100 s and `inf`. It takes
the framework's deadline from SGLang's own `deadline_s`.
- Second commit, test only, in the same file, promised in #27's review:
`test_the_spec_needs_aiters_grouped_topk` re-implemented SGLang's env
comparison, and `test_sglang_refuses_its_jit_grouped_topk` replaces it.
- The new test calls SGLang's `monokernel_disable_reasons` with and
without `SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK=1`. Setting the variable
must add exactly that refusal.
  - Like #25, it imports from #45389's location first.

What this does not change:
- A real stall in layer 0's launch now stops after half the watchdog
(150 s at the default) instead of 60 s. With `--watchdog-timeout inf`,
SGLang's watchdog never fires, and such a stall stops after 5,497 s.
- With `--disable-cuda-graph`, each launch is enqueued by its host, so
host skew can reach any launch, and those launches keep 60 s.
- The stall itself remains: this fix only removes the false stop. A
SGLang JIT cache on local disk (`SGLANG_JIT_CACHE_DIR`) avoids the 30–60
s wait.

## Accuracy Tests

No numerics change: there is no kernel or forward change.

Setup: MI355X, `rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014`
(`sha256:e4498f5d595f`), SGLang main ab580b2e06,
`amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25`. Main is at da5faf9 and this PR
at ac9f232. The boots run at TP4 with the e2e test's server flags, each
a fresh server: its own warmup request makes the first routed step, then
one 32-token greedy request follows. Rank 3's host stalls before its
first routed decode replay, using the out-of-tree hook in the first
comment below.

**Around the guard's limit** (5 s stall on rank 3). Layer 0's deadline
is 60 s on main at every watchdog here.

| `--watchdog-timeout` | Layer 0's deadline with this PR | main | this
PR |
| --- | --- | --- | --- |
| 10,900 s | 5,450 s | serves | serves |
| 11,100 s | 5,497 s (the cap) | serves | serves |
| `inf` | 5,497 s (the cap) | serves | serves |

With `inf`, each scheduler's watchdog thread exits at start, on main as
here (`time.sleep(inf)` raises `OverflowError` in SGLang's
`watchdog.py`), so SGLang's watchdog never fires.

**Host stall into the first routed step:**

| Case | Watchdog | Layer 0's deadline (main / this PR) | main | this PR
|
| --- | --- | --- | --- | --- |
| 90 s stall | 600 s | 60 s / 300 s | GPUs 0 to 2 fault at 0x0 in layer
0's launch 60.0 s after the stall starts; the server never comes up |
the first routed step waits out the stall; the server is up 90.1 s after
it starts, and the request completes |
| 400 s stall: a peer that never arrives | 300 s | 60 s / 150 s | GPUs 0
to 2 fault at 0x0 in layer 0's launch at 60.1 s | GPUs 0 to 2 fault at
0x0 in layer 0's launch at 150.0 s, before SGLang's watchdog |

**A stall past the first launch** uses the repo's fault injection:
watchdog 300 s and `SGLANG_AMD_MONO_FAULT_INJECT=1:22:20`, with the e2e
fault test's request (2 × 1,024 tokens).
- Rank 1 stalls at guarded wait 22 once 20 steps have run. Layer 0's
launch and the dense launches have at most 21 guarded waits, and the KDA
and sparse-attention launches have 22 to 51, so only a later launch can
stall.
- All four GPUs fault at 0x0 in a KDA launch
(`glm53_flash_layer_s2_tp4_k1_d0_q1_h1`) 60.2 s after the request. No
tokens are returned, and the server exits.
- So the later launches keep the 60 s deadline.

**Cold SGLang JIT cache** (watchdog 600 s, the cache removed before each
boot, `HOME` on NFSv3 with `local_lock=none`; main and this PR
alternate, 5 boots each). "Entry skew" is the last rank's entry into its
first routed decode replay, measured from the first rank's. It comes
from a per-rank host timeline that is out of tree.

| | Passed | Entry skew (s) |
| --- | --- | --- |
| main | 2 of 5 | failed: three boots with a rank still waiting on the
lock when the GPUs stopped; passed: 30.2, 58.7 |
| this PR | 5 of 5 | 58.2, 60.6, 60.6, 30.1, 60.3 |

- On main, the GPUs stop 60.1, 60.0 and 60.0 s after the first rank
enters, and the server never comes up.
- With this PR, the early ranks' first step waits up to 60.6 s on the
GPU, and the request completes.
- One more boot of main is not counted: its server failed to open its
device at start (`hipErrorLaunchFailure` in `current_device()`), before
loading the model. Its rerun is the 30.2 s pass.

**Other checks on this head:**
- **E2E:** `test_glm53_flash_server.py` at TP4 with
`HSA_DISABLE_COREDUMP_ON_EXCEPTION=1`, 5 of 5 pass: GSM8K 0.975, mean KL
against prefill 0.012 to 0.033 (bound 0.06), 21 of 21 returned
hidden-state rows give the sampled token, and the fault case, where both
deadlines are 10 s.
- **Code objects:** compile-only builds of every kind (FFN-only, KDA,
sparse attention, dense, layer 0, tail), at every width and at TP 2, 4
and 8, preshuffled and row-major FP8 where they apply: 72 a tree.
  - In one build of each tree, 70 of 72 are byte-identical to main's.
- The other two vary from build to build on main itself. Over 7 builds
of main and 6 of this PR, the TP4 sparse-attention builds at 2 and 8
rows (both layouts) each give the same 2 or 3 variants on both trees,
and every other build never varies.
- Every build of both trees has the same VGPRs, SGPRs, spills, LDS and
scratch as main's.
- **Unit tests**, run with `--continue-on-collection-errors`. Two
MiniMax test files fail at collection on a CPU-only host, on main as
here.
- CPU only, SGLang ab580b2e06: 261 passed (main 260). On both, 3
contract cases fail: deepseek_v41's two and minimax_m31's.
- CPU only, SGLang main 39b54ae0ee: 262 passed (main 261). On both, the
minimax and minimax_m31 contract cases fail.
- With a GPU visible, SGLang ab580b2e06: 293 passed (main 292), with the
same 3 contract failures.
- Negative controls, CPU only: without the `min`, the deadline test
fails at 11,100 s (5,550.0 against 5,497), and with that case removed,
at `inf` (inf against 5,497). If `_deadline` returns the framework's
deadline for every key, it fails at 300 s (60.0 against 150.0). Without
the variable in `requires.env`, the grouped top-k test fails.

## Speed Tests and Profiling

Not measured. The code objects are byte-identical to main's. At run
time, the only difference is the value of one guard word in layer 0's
runtime, and a wait reads that word only on its periodic check while it
is still pending.

## Checklist

- [x] `pre-commit run --all-files` passes.
- [x] Tests under `test/` cover the change.
- [x] The SGLang commit or PR this was tested against is named above
(SGLang main ab580b2e06; the CPU unit tests also on 39b54ae0ee).
- [x] Accuracy and speed results are attached when kernel or forward
code changes (no kernel or forward change).

---------

Co-authored-by: Hassan Eslami <hassan@marlowe.ai>
chuyeh pushed a commit to sgl-project/sgl-amd-mono that referenced this pull request Oct 30, 2026
…nokernel (#29)

## Motivation

Qwen3.8-Flash-Next (SGLang's Qwen4-Exp,
`amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8`) decodes with its
MTP head (NEXTN) at small batch sizes, where launch gaps dominate. This
PR adds a FlyDSL monokernel for it to sgl-amd-mono, behind SGLang's
monokernel interface (RFC #43949). `--monokernel-backend amd` selects
it, and SGLang needs no model hooks. The server runs SGLang's AITER path
(`SGLANG_USE_AITER=1`; the full launch command is in the README).

- A verify step of 1-8 requests x 4 tokens (embedding, PLE, 48 GDN / QSA
layers and the final mixer) runs in 13 persistent launches.
- Each MTP draft step of 1-8 requests runs in one launch: a draft extend
(R x 4 rows) or a draft decode step (R x 1 rows).

Monokernel API v1 routes every runner of a speculative server by the
target's verify key, which the NEXTN draft never runs. The draft spec
therefore routes its own steps: a wrapper in front of the draft's
`model.forward`. It can move to SGLang if a later monokernel API routes
a draft's own step modes.

The kernel itself (launch plan, stages, LDS and scratch layout,
synchronization and fail-stop, weights, numerics) is described in
[`sgl_amd_mono/qwen38_flash_next/README.md`](https://github.com/amd-cgilli/sgl-amd-mono/blob/qwen38-flash-next-monokernel/sgl_amd_mono/qwen38_flash_next/README.md).

Below is a comparison of the before and after:
<img width="1612" height="727" alt="b4"
src="https://github.com/user-attachments/assets/85b581aa-d309-4fd5-8774-24fcb2d810dc"
/>
<img width="1272" height="669" alt="after"
src="https://github.com/user-attachments/assets/2834498c-02a2-4546-b9c4-25d51567dfdb"
/>





## Modifications

- `sgl_amd_mono/qwen38_flash_next/`:
- `spec.py`: the target spec, registered for
`Qwen4ExpForConditionalGeneration` on gfx950, TP1, 1-8 requests x 4
tokens. It refuses unsupported configurations with reasons:
    - NEXTN/EAGLE with topk 1, 3 steps and 4 draft tokens only;
    - no ReplaySSM, no PP speculative scratch remapping;
    - bf16 KV, no HiSparse;
- `SGLANG_USE_AITER`, the AITER/auto MoE runner, the fused shared
expert, no paged experts or MegaMoE, no expert simulation or recording;
    - the Quark MXFP4 checkpoint;
    - `--attention-backend` triton or aiter;
    - plus per-layer config and weight-layout checks.
- Image and video requests: a verify step is rotated by SGLang's MRoPE
positions (`forward_batch.mrope_positions`, as the model's own forward
does), and each request's sequence start comes from `seq_lens`. With
images or video the two differ, and the QSA index tiles, ring slots and
selection count from the sequence position.
- QSA index-key ring: the launches index SGLang's QSA key ring by the
width its pool allocates
(`ring_width` in `gfx950/runner.py`: 4 or 8 slots a request). With
today's 4-slot ring (one group),
a verify row past the pending group overwrites one of its members, and a
rejected row's key stays
in the slot, so the group can later compress with it; SGLang's own
verify has the same issue.
sgl-project/sglang#47051 makes the ring two groups (8 slots), which
keeps every pending member. The launches work with either layout.
- `draft.py`: the MTP draft spec, registered for
`Qwen4ExpForCausalLMMTP`.
- `_build` installs a wrapper in front of the draft's `model.forward`.
- The wrapper sends draft extend (`DRAFT_EXTEND_V2`, R x 4 rows) and
draft decode (`DECODE`, R x 1 rows) steps of R <= 8 requests to the
draft launch, and every other step to SGLang's forward.
- The decision depends only on the step's shape, so a captured graph
replays it.
- It requires the draft's QSA attention backends (`"qsa"`), logs each
draft step at its first launch, and refuses during capture a
draft-shaped step the launch cannot read.
  - `check.py`: `SGLANG_AMD_MONO_CHECK=1` checks the target only:
    - at build, it compares the layer weights against the checkpoint;
- on each eager verify step, it compares the logits and hidden states
against SGLang's forward of the same step (`NUMERICS`).
  - `gfx950/`:
- the kernel (`kernel.py`, the guarded wait in `device.py`), its layout,
packing and config;
    - the layer views read from SGLang's params in place (`weights.py`);
- the host-side runners (`runner.py`). The draft runner copies
`input_ids` and new hidden states into buffers it owns, so its pointer
tables stay valid under graph capture.
  - `README.md`: the kernel documentation.
- The specs import SGLang's interface from
`sglang.kernels.plugin_kernels.amd` and fall back to
`sglang.srt.monokernels` (SGLang before sgl-project/sglang#45389), as
the other specs do.
- `sgl_amd_mono/__init__.py`: SPECS entries for the target and the MTP
draft.
- `README.md`: Supported Models, Usage, Structure, Testing.
- Tests:
- `test/unit/test_qwen38_flash_next_{spec,layout}.py`, CPU. These cover
the specs, refusals, layout, the RoPE positions a routed verify passes,
and the draft wrapper:
    - routing of draft extend and decode steps;
    - the fallback to SGLang's forward;
    - stacking under the runtime's forward switch;
    - the per-mode backend check;
    - the capture-time refusal;
    - the fixed-address input buffers.
  - `test/kernels/test_qwen38_flash_next_compile.py`, GPU.
-
`test/e2e/{qwen38_flash_next_server,test_qwen38_flash_next_server}.py`,
GPU:
    - GSM8K at 1/4/16 requests, accept length;
    - the target and draft launch lines;
    - KL decode vs prefill;
    - 126k passkey;
    - check mode;
    - fault injection (target launches).

## Accuracy Tests

1 x MI355X, TP1, SGLang upstream main 2229345dff, NEXTN 3 steps / 4
draft tokens / topk 1, real acceptance. "Off" is the same server without
`--monokernel-backend amd`. Measured before the rebase onto current
main. The rebase and the cleanup leave the kernels' LLVM IR unchanged,
and the MRoPE change reads the same sequence start for text requests.

| Test | Monokernel on | Off |
|---|---|---|
| GSM8K, 400 q | 0.970 (accept length 3.52) | 0.975 (accept length 3.52)
|
| GSM8K 5-shot chat, 200 q, at 1 / 4 / 16 requests | 0.98 / 0.98 / 0.97
| 0.98 / 0.985 / 0.975 |
| AIME25, 30 q x 4 seeds, pass@1 (thinking on, T 0.6, top-p 0.95) |
96.67% (116 of 120) | 98.33% (118 of 120) |
| AIME25, 30 q x 4 seeds, pass@4 | 100% | 100% |
| KL decode vs prefill, mean over fresh servers (bound 0.1) | 0.062 (5
runs, 0.041 to 0.085) | 0.059 (4 runs, 0.051 to 0.069) |
| RULER NIAH at ~261K tokens (niah_single_1/2/3, niah_multikey_1, 25
each) | 100 | 100 |
| GPQA-Diamond pass@1 (same settings) | 0.855 (sem .006) | 0.836 (sem
.004) |
| GPQA-Diamond majority@4 / pass@4 | 0.871 / 0.909 | 0.884 / 0.899 |

<details>
<summary>Unit + contract tests (CPU): 102 passed</summary>

```text
date:        2026-10-29 09:50 UTC
sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
sglang from:  /sgl-workspace/sglang-main-latest/python/sglang
versions:     torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0
GPU:          1 x AMD Instinct MI355X (gfx950)

$ python -m pytest -v -p no:cacheprovider test/unit/test_qwen38_flash_next_spec.py test/unit/test_qwen38_flash_next_layout.py test/unit/test_contract.py -k qwen38

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python
rootdir: /sgl-workspace/sgl-amd-mono
configfile: pyproject.toml
plugins: platformdirs-4.13.5, anyio-4.15.1
collecting ... collected 136 items / 34 deselected / 102 selected

test/unit/test_qwen38_flash_next_spec.py::test_the_plugin_registers_the_target_and_the_draft PASSED [  0%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_spec_declares_the_drafts_own_backend PASSED [  1%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_keeps_the_forward_signature PASSED [  2%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-1-4] PASSED [  3%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-8-32] PASSED [  4%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-1-1] PASSED [  5%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-8-8] PASSED [  6%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-1-4] PASSED [  7%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-8-32] PASSED [  8%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-1-1] PASSED [  9%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-8-8] PASSED [ 10%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests] PASSED [ 11%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests extend] PASSED [ 12%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-verify] PASSED [ 13%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-extend] PASSED [ 14%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests] PASSED [ 15%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests extend] PASSED [ 16%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-verify] PASSED [ 17%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-extend] PASSED [ 18%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_with_input_embeds_is_refused PASSED [ 19%]
test/unit/test_qwen38_flash_next_spec.py::test_the_runtimes_forward_wraps_the_draft_wrapper PASSED [ 20%]
test/unit/test_qwen38_flash_next_spec.py::test_the_wrapper_checks_the_backend_once_a_step_mode PASSED [ 21%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[three rows a request] PASSED [ 22%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[fp32 hidden] PASSED [ 23%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[hidden of other rows] PASSED [ 24%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[strided hidden] PASSED [ 25%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[no hidden states] PASSED [ 26%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[qsa] PASSED [ 27%]
test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[hybrid] PASSED [ 28%]
test/unit/test_qwen38_flash_next_spec.py::test_every_draft_call_site_calls_the_model_forward_attribute PASSED [ 29%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_writes_the_buffer_its_input_does_not_overlap PASSED [ 30%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_reads_new_hidden_states_from_a_fixed_address PASSED [ 31%]
test/unit/test_qwen38_flash_next_spec.py::test_build_installs_the_wrapper PASSED [ 32%]
test/unit/test_qwen38_flash_next_spec.py::test_the_draft_worker_reads_index_sharing_off_after_the_build PASSED [ 33%]
test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_is_accepted PASSED [ 34%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[hidden size] PASSED [ 35%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[experts] PASSED [ 36%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[index budget] PASSED [ 37%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[layer pattern] PASSED [ 38%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[quantization] PASSED [ 39%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[ssm dtype] PASSED [ 40%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[context] PASSED [ 41%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[no flydsl] PASSED [ 42%]
test/unit/test_qwen38_flash_next_spec.py::test_the_ple_layer_is_one_reason PASSED [ 43%]
test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_routes_verify_batches_of_one_to_eight PASSED [ 44%]
test/unit/test_qwen38_flash_next_spec.py::test_the_requirements_are_declared_sglang_reads PASSED [ 45%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[two tokens] PASSED [ 46%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[nine requests] PASSED [ 47%]
test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[decode] PASSED [ 48%]
test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[mrope] PASSED [ 49%]
test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[text positions] PASSED [ 50%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_reaches_the_jit_cache_key PASSED [ 50%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[kind] PASSED [ 51%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[empty] PASSED [ 52%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[requests] PASSED [ 53%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[tokens] PASSED [ 54%]
test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[first source] PASSED [ 55%]
test/unit/test_qwen38_flash_next_spec.py::test_a_verify_is_thirteen_launches_of_the_48_layers PASSED [ 56%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[1] PASSED [ 57%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[2] PASSED [ 58%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[3] PASSED [ 59%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[4] PASSED [ 60%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[5] PASSED [ 61%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[6] PASSED [ 62%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[7] PASSED [ 63%]
test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[8] PASSED [ 64%]
test/unit/test_qwen38_flash_next_spec.py::test_the_layout_matches_the_spec PASSED [ 65%]
test/unit/test_qwen38_flash_next_spec.py::test_the_views_accept_the_layers PASSED [ 66%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[no fused shared expert] PASSED [ 67%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[unshuffled experts] PASSED [ 68%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[interleaved gate and up] PASSED [ 69%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[2-byte scales] PASSED [ 70%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[dequantized projection] PASSED [ 71%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[wide b / a projection] PASSED [ 72%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[bf16 projection] PASSED [ 73%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[conv weight] PASSED [ 74%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[f32 norm] PASSED [ 75%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[index projection] PASSED [ 76%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[q norm] PASSED [ 77%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[router] PASSED [ 78%]
test/unit/test_qwen38_flash_next_spec.py::test_an_offloaded_ngram_table_is_refused PASSED [ 79%]
test/unit/test_qwen38_flash_next_spec.py::test_the_model_is_accepted PASSED [ 80%]
test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_model_is_one_reason PASSED [ 81%]
test/unit/test_qwen38_flash_next_spec.py::test_the_mtp_draft_views_are_checked PASSED [ 82%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[2560-6144] PASSED [ 83%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[16384-2560] PASSED [ 84%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[128-2560] PASSED [ 85%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[640-2560] PASSED [ 86%]
test/unit/test_qwen38_flash_next_layout.py::test_moe_padding_matches_fusedmoe PASSED [ 87%]
test/unit/test_qwen38_flash_next_layout.py::test_unshuffle_moe_expert_inverts_fusedmoe_layout PASSED [ 88%]
test/unit/test_qwen38_flash_next_layout.py::test_check_moe_params_accepts_fusedmoe_shapes PASSED [ 89%]
test/unit/test_qwen38_flash_next_layout.py::test_pack_bf16_tiles_16_rows_by_64_k PASSED [ 90%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 91%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 92%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 93%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 94%]
test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [ 95%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 96%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 97%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 98%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 99%]
test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [100%]

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
  /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead.
    from distutils.dep_util import newer, newer_group

test/unit/test_qwen38_flash_next_spec.py: 14 warnings
  /opt/venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============== 102 passed, 34 deselected, 18 warnings in 8.10s ================
exit code: 0
```

</details>

<details>
<summary>Kernel tests: 7 passed</summary>

```text
date:        2026-10-29 09:50 UTC
sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
sglang from:  /sgl-workspace/sglang-main-latest/python/sglang
versions:     torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0
GPU:          1 x AMD Instinct MI355X (gfx950)

$ python -m pytest -v -p no:cacheprovider test/kernels/test_qwen38_flash_next_launches.py

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python
rootdir: /sgl-workspace/sgl-amd-mono
configfile: pyproject.toml
plugins: platformdirs-4.13.5, anyio-4.15.1
collecting ... collected 7 items

test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=1] PASSED [ 14%]
test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=8] PASSED [ 28%]
test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=7+1 pad] PASSED [ 42%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-extend] PASSED [ 57%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-decode] PASSED [ 71%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-extend] PASSED [ 85%]
test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-decode] PASSED [100%]

=============================== warnings summary ===============================
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15
  /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead.
    from distutils.dep_util import newer, newer_group

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================== 7 passed, 2 warnings in 24.13s ========================
exit code: 0
```

</details>

<details>

<summary>Synthetic benchmark C1 8K input / 1K output</summary>

```text
date:        2026-10-29 15:31 UTC
sgl-amd-mono: 24c72df3930a4c6269df157565b83f1b194387ac Turn off crash dumps in the Qwen3.8-Flash-Next fault test
SGLang:       7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main)
aiter:        ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560
GPU:          1 x AMD Instinct MI355X (gfx950)
MTP acceptance: measured (0 SGLANG_SIMULATE_ACC_* in the server env)

# server
$ SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 AITER_FLYDSL_FORCE=1 SGLANG_MAMBA_SSM_DTYPE=bfloat16 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \
  python -m sglang.launch_server --model-path amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8 --port 31060 --tp 1 --attention-backend triton --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-max-bs-decode 8 --max-running-requests 16 --enable-amd-monokernel

# client
$ python -m sglang.bench_serving --backend sglang --port 31060 --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 --num-prompts 20 --max-concurrency 1 --warmup-requests 2 --seed 1

/sgl-workspace/sglang-main-latest/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
  warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None)
Waiting up to 60s for http://0.0.0.0:31060/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8', served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None)

#Input tokens: 163840
#Output tokens: 20480
Starting warmup with 2 sequences...
Warmup completed with 2 sequences. Starting main benchmark run...

  0%|          | 0/20 [00:00<?, ?it/s]
  5%|▌         | 1/20 [00:02<00:38,  2.03s/it]
 10%|█         | 2/20 [00:03<00:35,  1.95s/it]
 15%|█▌        | 3/20 [00:05<00:31,  1.85s/it]
 20%|██        | 4/20 [00:11<00:55,  3.46s/it]
 25%|██▌       | 5/20 [00:13<00:42,  2.87s/it]
 30%|███       | 6/20 [00:15<00:38,  2.73s/it]
 35%|███▌      | 7/20 [00:17<00:31,  2.41s/it]
 40%|████      | 8/20 [00:19<00:27,  2.33s/it]
 45%|████▌     | 9/20 [00:21<00:24,  2.23s/it]
 50%|█████     | 10/20 [00:23<00:21,  2.16s/it]
 55%|█████▌    | 11/20 [00:25<00:19,  2.14s/it]
 60%|██████    | 12/20 [00:27<00:16,  2.02s/it]
 65%|██████▌   | 13/20 [00:29<00:13,  1.94s/it]
 70%|███████   | 14/20 [00:31<00:12,  2.02s/it]
 75%|███████▌  | 15/20 [00:33<00:09,  1.94s/it]
 80%|████████  | 16/20 [00:35<00:08,  2.00s/it]
 85%|████████▌ | 17/20 [00:37<00:05,  1.93s/it]
 90%|█████████ | 18/20 [00:38<00:03,  1.88s/it]
 95%|█████████▌| 19/20 [00:40<00:01,  1.85s/it]
100%|██████████| 20/20 [00:42<00:00,  1.83s/it]
100%|██████████| 20/20 [00:42<00:00,  2.13s/it]

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    inf       
Max request concurrency:                 1         
Successful requests:                     20        
Benchmark duration (s):                  42.57     
Total input tokens:                      163840    
Total input text tokens:                 163840    
Total generated tokens:                  20480     
Total generated tokens (retokenized):    20422     
Request throughput (req/s):              0.47      
Input token throughput (tok/s):          3849.17   
Output token throughput (tok/s):         481.15    
Peak output token throughput (tok/s):    686.00    
Peak concurrent requests:                2         
Total token throughput (tok/s):          4330.31   
Concurrency:                             1.00      
Accept length:                           3.57      
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2127.22   
Median E2E Latency (ms):                 1851.48   
P90 E2E Latency (ms):                    2236.08   
P95 E2E Latency (ms):                    2638.79   
P99 E2E Latency (ms):                    5264.40   
---------------Time to First Token----------------
Mean TTFT (ms):                          456.02    
Median TTFT (ms):                        245.55    
P90 TTFT (ms):                           291.58    
P95 TTFT (ms):                           720.12    
P99 TTFT (ms):                           3441.69   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.63      
Median TPOT (ms):                        1.53      
P90 TPOT (ms):                           1.87      
P95 TPOT (ms):                           1.93      
P99 TPOT (ms):                           2.12      
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.63      
Median ITL (ms):                         1.46      
P90 ITL (ms):                            1.92      
P95 ITL (ms):                            2.88      
P99 ITL (ms):                            5.79      
Max ITL (ms):                            6.31      
==================================================

# monokernel lines from the server log
[2026-10-29 15:31:03] monokernel qwen38_flash_next_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 2436.7 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: final mixer packed (2 tensors, 12.8 MiB), gdn layer packed (360 tens
[2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: SGLang's QSA MTP index sharing off (index_share_for_mtp_iteration True -> False), so every draft decode selects its own blocks
[2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: draft extend steps of up to 8 x 4 rows and draft decode steps of up to 8 x 1 are routed by the spec's wrapper of the draft's forward, and one it cannot launch raises; each key logs 'MTP draft runs' at its first launch
[2026-10-29 15:31:09] monokernel qwen38_flash_next_mtp_draft_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 543.0 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: mtp packed (9 tensors, 66.0 MiB)
[2026-10-29 15:31:24] monokernel qwen38_flash_next_hip on gfx950 runs target_verify steps of batch size [1, 2, 3, 4, 5, 6, 7, 8] x 4 tokens (8 of 8 captured decode graphs); every other step runs the standard path
[2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 7, 7)
[2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 8, 8)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 4, 4)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 5, 5)
[2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 6, 6)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 1, 1)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 2, 2)
[2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 3, 3)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 5, 20)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 6, 24)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 7, 28)
[2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 8, 32)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 2, 8)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 3, 12)
[2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 4, 16)
[2026-10-29 15:31:31] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 1, 4)
```
</details>

## Speed Tests and Profiling

1 x MI355X, TP1, the same server flags with and without
`--monokernel-backend amd`, CUDA graphs up to 8 requests. Verify and
draft are both on the monokernel.

Client:

```bash
python -m sglang.bench_serving --backend sglang --dataset-name random \
  --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \
  --num-prompts <max(10 x conc, 20)> --max-concurrency <conc> --warmup-requests 2 --seed 1
```

| C | Off: median TPOT / output tok/s | On: median TPOT / output tok/s |
Speedup: TPOT [95% CI] / tok/s | Accept length off / on |
|---|---|---|---|---|
| 1 | 4.04 ms / 220 | 1.60 ms / 517 | **2.53x** [2.28, 2.84] / 2.35x |
3.36 / 3.38 |
| 2 | 4.58 ms / 400 | 2.17 ms / 812 | **2.12x** [1.89, 2.26] / 2.03x |
3.39 / 3.43 |
| 4 | 4.99 ms / 669 | 3.21 ms / 1063 | **1.56x** [1.51, 1.71] / 1.59x |
3.39 / 3.40 |
| 8 | 6.22 ms / 1088 | 5.86 ms / 1180 | **1.06x** [1.02, 1.09] / 1.08x |
3.38 / 3.40 |

## Checklist

- [x] `pre-commit run --all-files` passes.
- [x] Tests under `test/` cover the change.
- [x] The SGLang commit tested against: upstream main 2229345dff
(accuracy and speed), f3f0f7b68c (C=1 re-check), f3f0f7b68c and
af21e2b989 (unit tests).
- [x] Accuracy and speed results.
- [x] `.github/CODEOWNERS`: `**/*qwen38_flash_next*` is already in
upstream's CODEOWNERS.
kevin-mii added a commit to sgl-project/sgl-amd-mono that referenced this pull request Nov 4, 2026
…104)

- kimi_k3: the standalone MoE launch hands the shared stages the timeline keys
  (tl_on, tls) the router and route stamps read; check_moe raised KeyError at trace.
- minimax, minimax_m31: the spec types, features_in_use and attention_backends_of
  from where SGLang defines them (sgl-project/sglang#45389), the old paths as the
  fallback; srt/monokernels is a shim SGLang plans to drop.
- glm53, glm53_flash, qwen38_flash_next, muse_spark: SGLANG_READS names ForwardMode's
  and CaptureHiddenMode's methods in forward_mode.py, forward_batch_info.py before.
- test_glm53_flash_spec: the JIT grouped top-k test checks the spec's declared env
  only; re-running SGLang's guard there broke on its signature, and SGLang tests
  the guard itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yichiche added a commit to yichiche/sgl-amd-mono that referenced this pull request Nov 5, 2026
… path

SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in
sgl-project/sglang#45389. Read it there first and fall back to
sglang.srt.monokernels, as the other specs on main do, so the contract test
passes on both SGLang versions.

Co-authored-by: Cursor <cursoragent@cursor.com>
yichiche added a commit to yichiche/sgl-amd-mono that referenced this pull request Nov 5, 2026
… path

SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in
sgl-project/sglang#45389. Read it there first and fall back to
sglang.srt.monokernels, as the other specs on main do, so the contract test
passes on both SGLang versions.
YAMY1234 added a commit to sgl-project/sgl-nvidia-mega that referenced this pull request Nov 10, 2026
… NVFP4 MTP target verify and NextN draft (#24)

* [Feat] [B300] glm5_3: NVIDIA megakernel specs for GLM-5.3 NVFP4 MTP target verify and NextN draft on B300 TP8

Add sgl_nvidia_mega/glm5_3: the GLM-5.3 target spec for SGLang's megakernel
layer (MTP verify steps of 1 to 4 requests, 4 to 16 rows, on 8 x B300 at
TP 8), its step functions with stock paths adapted from SGLang's model code,
and the CuTe DSL attention, small-M NVFP4 MoE and all-reduce hop kernels.
With --enable-nvidia-megakernel every target slot runs (attention, moe,
hops, qkv_a_small_m_shared, dnorm); SGLANG_NVIDIA_MEGAKERNEL_SLOTS is a test
and bisect override. Needs SGLang's DSA binding view.

Add the GlmMoeDsaForCausalLMNextN draft spec (slot draft, on by default) and
its CuTe DSL kernels: the embedding gather and eh norms in one kernel, the
column-sharded eh projection gathered by the target's all-reduce hop, the
distributed top-1, and the draft layer's MLA prologue, in every captured
draft graph. Needs embedding replication (with the flag, SGLang's megakernel
layer sets SGLANG_ENABLE_EMBED_REPLICATION when it is unset), greedy draft
acceptance and SGLang's EAGLE draft graph capture check; the contract test
skips the check's marker, naming it, on an SGLang without it.

At startup rank 0 logs the covered verify widths (1-4 requests) and the
derived weight copies' memory per GPU, and warns when
--max-running-requests is set above 8, where the megakernel helps little.

Also the README section, a NOTICE paragraph, a CODEOWNERS line and the
contract test's checks of the DSA binding view reads and the mirrored SGLang
code; the GLM-5.3 tests are filed separately.

The same specs serve GLM-5.2 NVFP4, which has GLM-5.3's architecture
(GlmMoeDsaForCausalLM) and the same value of every config field the target
and draft specs pin. The spec names and the plugin's log, refusal and rank
agreement texts name both models, and the module path stays glm5_3.
test/unit/test_glm5_3_glm5_2_config.py runs the published config of
nvidia/GLM-5.2-NVFP4 (revision 53e0691e21895a3863a606dfd12910c69eba94ab)
through SGLang's requirement check for both specs, and checks that a config
that differs in any pinned value, or lacks one, is still refused.

The package imports SGLang's megakernel interface from its location since
sgl-project/sglang#45389 (sglang.kernels.plugin_kernels.nvidia,
sglang.srt.kernel_plugins.nvidia) and falls back to sglang.srt.megakernels
on an older SGLang, as deepseek_v41 does.

* docs: correct GLM GPU test availability

* [Feat] [B300] glm5_3: serve MTP with 5 draft steps (6 verify rows a request)

The GLM-5.3 / GLM-5.2 specs served MTP / EAGLE with 3 draft steps only, so
the 5 / 1 / 6 setup of SGLang's GLM-5.3 and GLM-5.2 B300 NVFP4 low-latency
recipes stopped at startup with --enable-nvidia-megakernel.

The verify widths, the MoE row map and the draft eh rows are now derived
from the verify rows a request T (speculative_num_draft_tokens =
speculative_num_steps + 1, top-1 drafts), and the specs serve the tested
values 4 and 6 (plan.SERVED_TOKENS_PER_REQUEST); any other value is refused
at startup, naming them.

- Widths: every whole number of requests up to the kernels' 16 rows
  (plan.supported_rows): 4 / 8 / 12 / 16 rows at T 4 as before, 6 / 12 rows
  (1-2 requests) at T 6. Steps of more requests run SGLang's path.
- MoE: the engine takes at most 4 row groups of at most 4 rows, token sets
  over one expert list rather than requests. moe_split groups the rows by 4
  with the remainder last: T 4 keeps (4,) to (4, 4, 4, 4); 6 rows run on the
  two-group engine as (4, 2) and 12 rows on the four-group engine as
  (4, 4, 4). The one-group engine holds 4 rows, so it cannot take 6.
- Draft: the eh_proj shard serves the draft decode's 1-4 rows and the draft
  extend at the covered widths, so T 4 keeps its rows and T 6 adds 6;
  remove the unused max_requests.
- Attention, the hops and the draft attention already take any row count
  up to 16 on their 8- and 16-row forms; no kernel changes.

At T 4 every width, row map, eh row set and kernel form is unchanged.

* [Perf] [B300] glm5_3: the one-request fast forms at 6 verify rows

With 5 draft steps a one-request verify step has 6 rows, past the 4-row
limit of the forms that serve one request of 3 draft steps. Serve them at up
to 6 rows:

- the split MoE fold hop (SPLIT_MAX_ROWS 6; still taken only where the
  cooperative occupancy holds 2 x rows + 1 clusters of 8 CTAs, probed at
  build);
- the plain all-reduce hops on the dense layers (DENSE_MAX_ROWS 6);
- the 16-warp sparse MLA (one CTA per SM; its kernel already takes any row
  count and gives the 8-warp form's bytes).

No width of 4 verify rows a request has 5 or 6 rows, so those steps keep
their forms. The kernels are unchanged.

* docs: glm5_3 MoE groups and eh_proj rows with 6 draft tokens

The README's GLM-5.3 slot table named the MoE engine forms by requests
and listed the eh_proj rows of 4 draft tokens only. Name the forms by row
groups (6 rows run as 4 + 2, 12 as 4 + 4 + 4) and add the eh_proj rows of
6 draft tokens (6 and 12).

* [Fix] glm5_3: pin the SGLang mirrors with main's mirror digest

Main's SGLANG_MIRRORS check (#20) digests a
function's tokens in another encoding than this branch's copy, which the
merge replaced with main's. Pin the 18 GLM-5.3 mirrors with main's
mirror_digest of the same SGLang code: no mirrored body changes, only how
its digest is written.

* [Fix] glm5_3: re-pin four SGLang mirrors for current SGLang main

Four mirrored SGLang bodies changed on SGLang main. Each pin is now a
tuple of the digests before and after the change, so the check passes on
either SGLang:

- DeepseekMLAForwardMixin.forward_absorb_prepare (sgl-project/sglang#46359):
  every changed line is on the --dcp-replicate-q-proj path, which runs only
  with decode context parallelism; the specs refuse it, and the stock
  attention step does not copy those lines.
- DeepseekSparseAttnBackend._forward_trtllm (sgl-project/sglang#46990): the
  KV store and read go through the pool's layer router; on the DSA pool, a
  plain pool, that is the same set_mla_kv_buffer and get_key_buffer call
  with the same arguments, which the stock step makes.
- DeepseekSparseAttnBackend.forward_decode: sgl-project/sglang#46990 as
  above, and sgl-project/sglang#44322, a triton branch for ROCm with
  decode context parallelism.
- LogitsProcessor.forward (sgl-project/sglang#43449): the new lines run only
  with --enable-return-aux-hidden-states, which turns speculative decoding
  off, so never with the NextN draft; the output's new last_hidden_states
  stays None.

The copied steps stay as they are. The other 14 mirrored bodies are
unchanged on SGLang main.

* [Fix] glm5_3: adapt the specs to SGLang's megakernel API version 2

SGLang main's megakernel contract is version 2: a built object may also
offer the EAGLE entries draft_loop, draft_extend and begin_draft_round and
attention graph_variants, which SGLang reads from every built object. The
GLM-5.3 / GLM-5.2 target and draft built objects offer none of them (None:
SGLang's own path, as before), and the specs declare API version 2, as
main's DeepSeek-V4.1 and Qwen3.5 specs do.

* docs: glm5_3 boundary docstring says plain dense hops up to 6 rows

DENSE_MAX_ROWS is 6 since the 5-draft-step change; the module docstring
still said 4.

* test: glm5_3 pins the forward_mla mirrors after sglang#45946

sgl-project/sglang#45946 adds SAIL-only fp8 conditions to
forward_absorb_prepare / forward_absorb_core and a FlashMLA-backend-only
llama_4_scaling step to forward_absorb_prepare; the DSA path the spec
serves is unchanged. Pin the new digests next to the old ones.

* [Feat] [Rubin] glm5_3: serve SM107, TP8 across two trays, and nodes without NVLink multicast

Port the Rubin pieces of #19 (glm_dsa) to the
GLM-5.3 / GLM-5.2 specs. The kernels are the same; only the admission changes.

- spec, draft: admit sm_107 (Rubin / VR NVL72) next to sm_103. The draft
  takes the target's ISAs.
- spec, draft: declare Feature.MULTI_NODE_TP, so TP8 can span the two 4-GPU
  trays of an NVL72 rack (GB300 NVL72, Rubin VR NVL72), as the Qwen3.5 and
  Kimi K3 specs do.
- runtime: SGLANG_NVIDIA_MEGAKERNEL_GLM_ALLOW_NO_AR_FUSION=1 (off by default)
  lets the model check and the per-step coverage pass where SGLang's FlashInfer
  all-reduce + RMSNorm fusion does not apply (no NVLink multicast). The stock
  residual steps then run SGLang's unfused all-reduce, which is what its
  boundaries run there, and slot hops runs its own symmetric-memory all-reduce.
  Without the variable nothing changes.
- README: the GPUs and the two-tray launch. test/unit/test_glm5_3_rubin.py
  covers the three changes on CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: Run the NextN draft-extend MLP on the select_index rows only

The draft extend runs the NextN layer on every verify row of each request, but
the next draft step reads only the select_index row (one per request); the
logits processor already prunes to those rows before the LM head. After the
post-attention norm the draft step now keeps those rows of the MLP input and of
the residual SGLang's prepare wrote, so the MoE, its all-reduce and the final
norm run on B rows instead of B * tokens (6 -> 1 and 12 -> 2 rows at EAGLE
5/1/6). The attention still writes every row's KV and indexer key. The hidden
states handed to the logits then hold only the selected rows, with a
LogitsMetadata that does not select again.

- Slot draft_rowselect (shipped, a draft slot, needs slot draft).
- Gate (draft_step.select_index_rows): DRAFT_EXTEND_V2, no logprobs, hidden
  capture not FULL, no padded-row mask, a 1-D select_index with 0 < B < N
  rows; the NextN MLP is DeepseekV2MoE without an all-to-all. Host-only facts,
  fixed per captured shape. draft_attention.select_rows raises on any other
  stream layout.
- draft_attention.decoder_layer_forward takes the rows to keep (ffn_rows) and,
  at a width the draft attention family does not serve, SGLang's own attention
  call.
- Rank 0 logs once per captured draft graph its attention and MLP rows.

Numerics: the selection itself is exact (the kept rows equal the full-row
stream's rows bit for bit through SGLang's residual stream and final add +
RMSNorm). The draft MoE and its router pick their kernels by the row count, so
the MLP output on B rows is not bit-identical to the same rows of the N-row run
(draft.rowselect.mlp.* in the draft NUMERICS). Draft-only: target verification
decides every emitted token.

Tests: test/unit/test_glm5_3_draft_rowselect.py (CPU; the logits hand-off
needs CUDA), test/kernels/test_glm5_3_draft_rowselect.py (4 GPUs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Feat] glm5_3: TP4 profiles, verify steps composed per width from kernel units

The spec serves TP8 at 4 or 6 tokens per request up to 16 rows. It now also
accepts TP4 at 3 to 6 tokens per request up to 48 rows (one GB300 tray: the
InferenceX AgentX TP4 aggregate and PD decode-role configurations), and serves
those widths with kernel units composed per step; the TP8 path
(plan.build_providers, the executor, the families) is unchanged.

- profile.py: the facts a build is decided on (TP, heads per rank, tokens per
  request, widths, device, features) and the accepted table: the TP8 envelope
  (plan.py) plus TP4 at 3-6 tokens, 48 rows. Profile.units says which build
  runs.
- registry.py: candidates claim steps of a group atomically, with coverage per
  (profile, rows, mode) and contract tokens; per width the first preferred,
  switched-on, covering unit takes each step, the rest run the stock body. A
  decline (coverage) runs stock and is logged once per width; a family that
  covers a width but fails to build raises on every rank. A mixed group is a
  subclass of the stock group with the claimed steps bound to their units, so
  the stock step bodies and Providers.describe see them.
- units.py: the TP4 preference table. Every unit is a slot (runtime.LL_SLOTS,
  shipped; OPT_IN_SLOTS only when listed), selected by
  SGLANG_NVIDIA_MEGAKERNEL_SLOTS as the TP8 families are; "" is parity (every
  accepted width on the stock bodies).
- runtime/composed.py: the executor (GlmMegakernelVerify: its layer loop,
  first-capture checks and parity switch) on the composed step groups; its
  logits take ll.gemm's lm_head role and share the model processor's gatherer.
  runtime.build hands a TP4 profile to it.
- spec.py: tp_sizes {4, 8}, the (TP, tokens) pairing checked against the
  profile table (profile_reasons), max_rows 48, the units' named tolerances.
- ll/__init__.py, ll/jit.py: the units' package (optional modules, the
  ll.linear GEMM entry point, named tolerances) and the JIT loader for its .cuh
  kernels (SGLang's load_jit; refuses to compile under capture; failures
  raise).

Tests: test/unit/test_glm5_3_{profile,registry,spec,ll_jit,verify_logits}.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 small-M BF16 GEMMs on PDL split-K and TGV (ll.gemm)

The TP4 roles' small-M BF16 GEMMs that SGLang leaves on cuBLAS launch without
programmatic dependent launch. ll.gemm moves them, at the row counts where a
GB300 crossover bench measured a win in the GEMM's model context, to a PDL
kernel (slot ll.gemm.<role> each):

- router (FP32 logits): split-K at 17-96 rows (the tiny GEMM keeps <= 16);
- indexer wk|weights_proj: split-K at 1-64 rows;
- o_proj TP4: TGV 1-72; dense_down TP4: TGV 1-48; NextN eh_proj: TGV;
- lm_head TP4: TGV 1-60; measured but deferred (run only when listed): the
  TP8 o_proj / q_b shapes.

gemm_plan.py holds the plan and the named tolerances. The plan applies only on
SM103 with 152 SMs; elsewhere, and for quantized or biased weights or unplanned
rows, every role runs SGLang's own GEMM for the module, bit for bit.

Split-K is a vendored copy of FlashInfer v0.7.2's cluster split-K kernel with
FP32 output / any M and synchronization fixes (owner wait at .acquire.cluster,
phase-0 wait before the TMEM release, tcgen05 thread-sync fences; no arithmetic
change). FlashInfer's own kernel is used only when it carries those fixes
(SplitKTactic.SYNC_FIXED); SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK pins
auto / flashinfer / vendored. TGV is SGLang's cutedsl_bf16_gemm.

Tests: test/unit/test_glm5_3_ll_gemm_plan.py (CPU: tactics, engagement only on
SM103 / 152 SMs, slots, FlashInfer detection, stock dispatch);
test/kernels/test_glm5_3_ll_gemm.py (1 GPU: path per role and row, tolerances
vs cuBLAS and FP64, graph replay, PDL edges, launch stress).
benchmark/bench_glm5_3_ll_gemm.py re-checks the crossover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 sparse MLA flash decode as unit ll.sparse_mla_fd

The sparse_mla and uv_proj steps of the TP4 verify attention and of the NextN
draft layer's attention on a flash-decoding CUDA kernel over the top-k gathered
FP8 KV rows, instead of TRTLLM-Gen's sparse MLA decode + torch.bmm. The step
reads the running forward's ForwardView (view.dsa, view.kv_pool,
view.metadata), as the stock step does.

- Kernels (ll/csrc/attention/glm_sparse_mla_fd_*.cuh, adapted from
  TensorRT-LLM's sparse MLA flash decode): one 16-CTA cluster per row, 8 or 16
  heads; a v_b fold instance (8 heads, bitwise cuBLAS's BMM); a multi-tile mode
  (clusters of 2-8 CTAs, 8 heads).
- Launch plan (GB300, 152 SMs): one-tile up to 7 rows; built at compile; the
  co-resident cluster counts are agreed over the TP group (MIN); a build
  failure on any rank raises on every rank.
- Covers sm_103 with 152 SMs, 16 heads (TP4), <= 7 rows; declines with
  TRTLLM-Gen skip-softmax decode on.
- Tolerance ll.sparse_mla_fd.max_abs = 0.0625 against TRTLLM-Gen (measured
  0.052 / 0.055 at 8 / 16 heads).

16 heads: -2.1 to -2.8 us per layer at 5 / 6 rows
(benchmark/bench_glm5_3_sparse_mla.py).

Tests: numerics vs TRTLLM-Gen and an FP64 oracle, the provider steps vs the
stock steps, CUDA-graph replay and the PDL chain (1 GPU); the launch plan and
rank agreement (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 attention front on low-latency fused-A and a one-launch MLA KV half

The unit ll.attention (groups attention and draft_attention) serves the
attention front of a TP4 rank (16 heads, SM103, 1-48 rows, every step mode) by
claiming qkv_a / prologue / index_k / rope_cache of AttentionStep:

- qkv_a: SGLang's min-latency fused-A kernel with a larger stage ring and a
  pipelined MMA loop from 9 rows, up to 16 rows (bit for bit SGLang's kernel); a
  split-K form at 17-24 rows when one wave fits (ll.attention.qkv_a_wide.max_rel
  of fp32); stock above 24 rows.
- prologue: stock q_a norm; q_b on the pipelined form at 9-16 rows (bitwise);
  the kv_a norm deferred into rope_cache when folded.
- index_k: the stock body with its key / head-gate projection through
  ll.linear("wk_weights_proj").
- rope_cache: one PDL launch for the kv_a norm (folded), RoPE and FP8 of q and
  k and the MLA cache append, bit for bit FlashInfer's norm and
  mla_rope_quantize_fp8 and SGLang's set_mla_kv_buffer (the cache through the
  step's ForwardView).

The fold (slot ll.attention.kv_norm_fold) engages only when FlashInfer's norm
backend, read from its public surface, is CuTe DSL and a probe at build gives
the stock kv_a norm's bits; the family logs which way it went. fused_a_ll.cuh is
adapted from SGLang's dsv3_fused_a_gemm.cuh.

Tests (1 GPU): fused-A forms bitwise at 1-16 rows and within tolerance at
17-24, graph replay; the KV half bitwise (q and every cache byte); the stock
AttentionStep vs ll.attention on one TP4 layer at 5-48 rows, eager and
replayed. CPU: the norm-backend decision, coverage and composition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 all-reduce + residual + RMSNorm hops on a plugin-owned Lamport buffer

The unit ll.hops serves the three hop steps (HopSteps) of the TP4 verify step:

- csrc/distributed/glm_ll_mnnvl_oneshot.cuh: one-shot / two-shot MNNVL
  all-reduce + residual + RMSNorm with a MoE-finalize prologue and an NVFP4
  epilogue (builds with nvcc 13.0 and 13.4). Every wait on a peer is bounded:
  past the deadline it writes a host-mapped error record and traps (fail-stop).
- ll/transport.py: the Lamport buffer and flags are the plugin's own (torch
  symmetric memory, or FlashInfer's McastGPUBuffer) in FlashInfer's MNNVL
  layout; no FlashInfer workspace or flags are shared. Modules load through
  ll.jit; a build failure raises on every rank.
- ll/hops.py: the GB300 launch plan per world size and row count (W=4: one-shot
  up to 36 rows, two-shot 37-48), the family (deadline min(20 s, the build
  context's), o_proj through ll.linear) and its step group. Covers the TP4
  verify step on sm_103 up to 48 rows; provides hops:finalize and
  mlp_input:nvfp4 (written only when a taken MLP unit consumes it).

Tests: the plan tables (CPU); at W=4 every row 1-48 bit-identical to
FlashInfer's one-shot / two-shot, the finalize hop bit-identical to
moe_finalize_fuse_shared + the hop, a protocol mix eager and in a CUDA graph;
fail-stop of every cross-GPU wait; a two-tray multicast probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 MoE and dense MLP steps with the deferred finalize (ll.moe)

The units ll.moe.nvfp4 and ll.moe serve the MLP step (MlpSteps) of the TP4
verify step and leave the MoE finalize to the next hop:

- MoE layers: the router GEMM (ll.linear("router"), FP32 logits), then
  FlashInfer's public trtllm_fp4_block_scale_moe with do_finalize=False and the
  layer's processed NVFP4 weights (SGLang's
  fused_experts_none_to_flashinfer_trtllm_fp4 call for this configuration), and
  the shared expert on the layer's alternate stream; returns a deferred
  finalize for the next hop.
- ll.moe.nvfp4 takes the routed experts' FC1 input from the post-attention
  hop's NVFP4 epilogue; ll.moe quantizes its own input.
- Dense layers: the stock gate_up_proj and SiLU-and-mul, then
  ll.linear("dense_down"), un-reduced.
- Opt-in slot ll.moe.alt_stream_priority: the shared expert on a stream of the
  greatest priority at <= 6 rows. It gains only with FlashInfer's small-batch
  routing kernel (TP4 5/1/6: -1.82% step at 6 rows); on stock FlashInfer 0.7.2
  it loses (+1.2% at 5 rows, TP4 4/1/5, n = 2; +2.4% at 6 rows), so it runs only
  when listed in SGLANG_NVIDIA_MEGAKERNEL_SLOTS.
- Both require hops:finalize. A layer that does not fit declines the family
  with the layer and the reason.

Tests (4 GPUs): the copied call equals SGLang's runner bit for bit at 5-48 rows;
the layer (hop + MoE + finalize hop) equals the stock layer bit for bit on the
hop's FP4 input inside FlashInfer's one-shot band; CUDA-graph replays equal
eager; the priority stream keeps the bits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Perf] [GB300] glm5_3: TP4 NextN draft composed from units (P6a eh_proj, LL attention, row select)

The draft step (draft_step.FusedDraft) serves TP8. At a TP4 profile the draft
spec now builds draft_units.ComposedBuiltDraft: each draft graph (draft decode
at one row per request, draft extend at the verify rows) is composed through
the registry from the draft step groups (draft_front, draft_eh,
draft_attention, draft_mlp, draft_top1):

- draft.front (slot draft): the draft step's fused embedding + eh norms (the
  kernel does not depend on the TP size);
- ll.draft_eh (P6a, ll/draft.py): eh_proj row-parallel, each rank's K window
  times its weight columns. With the target's LL hops (found through the
  shared-table registry, runtime.register_target_hops) the partials are summed
  and the draft layer's input norm applied in one hop launch, and the layer
  starts from that (draft_attention.PreparedInput: the attention prepare is
  skipped); otherwise SGLang's all-reduce and the stock norm. Per draft forward
  at TP4, L2 flushed: 20-26 us vs 39-41 us for the replicated GEMM;
- the draft layer's attention on ll.attention and ll.sparse_mla_fd (the stock
  DraftAttention on the step's ForwardView elsewhere);
- draft_rowselect in the draft extend (draft_step.select_index_rows);
- the LM head on ll.gemm's lm_head role; the top-1 stays SGLang's (at TP4 its
  one-tray multimem gather is as fast as the distributed argmax or faster).

The composed draft is built when the target shares its embedding and LM head
(set_embed_and_head, before the KV pool is sized), so the LM-head role is built
on the target's head. Draft graphs outside the profile's widths run SGLang's
draft forward. The TP8 draft keeps the TP8 widths (<= 16 rows).

Tests: test/unit/test_glm5_3_draft_units.py and test_glm5_3_draft_eh_prepared.py
(CPU: widths, compositions, slots, the hand-off to the logits processor),
test/kernels/test_glm5_3_draft_eh_proj.py (4 GPUs: numerics at 1-48 rows, the
hop sum + norm, graph replay).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Test] glm5_3: Serving tests and decode A/B configuration at the AgentX TP4 / TP8 points

- test/e2e/glm5_3_server.py, test_glm5_3_server.py: servers at the InferenceX
  AgentX configurations (tp4-416: aggregated TP4, EAGLE 4/1/5, HiCache;
  tp8-516: aggregated TP8 on two trays, EAGLE 5/1/6, HiCache; tp4-516: the
  disaggregated decode role, TP4, EAGLE 5/1/6, standalone): on / off (the
  active line, covered decode graphs, GSM8K-200 within 3 points), parity
  (SGLANG_NVIDIA_MEGAKERNEL_SLOTS="" vs flag off, greedy ids and logprobs
  bitwise) and HiCache load-back (load-back output equals the device-hit
  output).
- benchmark/README.md: the GLM decode A/B protocol at those configurations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Feat] [GB300] glm5_3: Serve the TP4 decode role of a PD-disaggregated deployment

A PD deployment of GLM-5.2 on GB300 (the InferenceX AgentX 1p5d-c30 / 1p3d-c40
points) runs a TP1 PP4 prefill role and TP4 EAGLE 5/1/6 decode roles. The spec now
lists Feature.PD_DISAGGREGATION, so a decode role serves its verify steps on the
TP4 kernel units: the KV arrives through the transfer backend (NIXL) into pool
slots before a request's first verify step, outside the covered step, and the
decode radix cache is not a framework feature. The TP8 envelope keeps PD refused
(pd_reasons). The prefill role runs without the flag (TP1 and PP are refused
anyway).

Tests: test/unit/test_glm5_3_spec.py (the decode role accepted at TP4, refused at
TP8); test/e2e/test_glm5_3_server.py TestGlm53PDDecode (GSM8K through a running
1P1D router, decode engaged, prefill not, no NIXL transfer failure).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Test] glm5_3: Remove the tests this PR added (repo policy: no tests in sgl-nvidia-mega)

Per review, sgl-nvidia-mega carries no tests. Remove the 28 test files this
PR added under test/e2e, test/kernels and test/unit (the GLM-5.3 TP4, draft
row select and PD decode tests and their helpers glm5_3_server.py and
glm5_3_slots.py), and drop the benchmark README's pointer to the removed
test/e2e/glm5_3_server.py. Tests that already exist on the base branch are
left untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* glm5_3: Keep SGLang's copyright notice in the headers of files adapted from SGLang

Apache-2.0 requires derivative files to retain the copyright notices of the
work they adapt. Following the OSRB guidance, add
"Portions Copyright 2023-2024 SGLang Team (Apache-2.0)" after the NVIDIA SPDX
lines of the glm5_3 files adapted from SGLang, and name the origin of
ll/moe.py, whose routed-expert call copies SGLang's. fused_a_ll.cuh copies
most of SGLang's dsv3_fused_a_gemm.cuh, so it keeps that file's header
unchanged and adds the NVIDIA and SGLang lines below it. Header lines only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Fix] glm5_3: pin the SGLang mirrors for main's SGLang (cf6133a)

main's unit tests check out SGLang cf6133a, where eight glm5_3 mirrors no
longer matched. Each change keeps the covered steps as mirrored, so the new
digest is added beside the old ones (both SGLangs pass):

- forward_absorb_prepare / forward_absorb_core (#44503, #38943): the indexer,
  the unfused RoPE and the W_UV projection moved into the run_mla_indexer,
  prepare_mla_qkv and project_mla_values hooks, whose defaults are the old
  calls (no SGLang class overrides them); the DCP extend gather is out of the
  spec's setups.
- Indexer.forward_cuda, DeepseekSparseAttnBackend._forward_trtllm (#46890):
  prefill-CP only.
- DeepseekV2DecoderLayer.forward (#46857), in the hops, the MoE step and the
  draft attention: an assert under CP decode attention TP, which the spec
  refuses.
- LogitsProcessor.forward (#47200): vocab-sharded logits for a target verify
  only; the draft steps the mirror covers take the gather as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* [Doc] glm5_3: Document the TP4 split-K GEMM source switch in the README

SGLANG_NVIDIA_MEGAKERNEL_GLM_GEMM_SPLITK picks the split-K source of the TP4
small-M GEMMs (auto, flashinfer or vendored); document it next to the TP4
units, as the README documents Qwen3.5's SGLANG_NVIDIA_MEGAKERNEL_MOE_TACTICS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: shuw <shuw@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kaixi Matteo Chen <kaiximatteoc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

amd deepseek dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation jit-kernel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants