Repository navigation
Conversation
Registers the backend nvidia with SGLang's monokernel framework through the sglang.srt.plugins entry point. Its specs serve DeepSeek-V4.1-Flash DSpark verify steps of 1 to 48 rows on SM103 at TP4, and the DSpark draft stages, with the flashmla attention backend and an FP8 KV cache. Every other step runs SGLang's standard path, and nothing beyond the entry point is imported without the flag. A routed step rebuilds SGLang's own step from SGLang's modules and stage boundaries, and fills three slots with this package's kernels: - at 1 to 8 rows, wo_b's GEMM epilogue publishes into a Lamport mailbox on every rank, and a Lamport consumer runs the attention all-reduce with the mHC post, combine and RMSNorm as the attention post's fused reduction; - at 1 to 48 rows, the routed experts run on a route kernel and a pair of 8-row tcgen05 block-scaled GEMMs inside SGLang's dual-stream MoE; - at 1 to 8 and 48 rows, the router's top-k runs in one CTA per row, bit for bit SGLang's fused gate. Each slot is probed at build and stays off, with the reason logged, on an SGLang without its seam. The mailbox, the kernels for every routed width and their tactics are built and compiled at startup, before the KV cache is sized. Every wait on another GPU has a deadline below SGLang's watchdog and faults at address 0x0 when missed. Tests: CPU contract tests pin every SGLang name the package reads; kernel tests on GB300 check each kernel against SGLang's or FlashInfer's op and the fail-stop deadline.
test/e2e launches DeepSeek-V4.1-Flash servers with and without the monokernel backend: GSM8K across the largest covered width, KL of covered steps and of long prompts against the normal prefill path, a passkey past the indexer's candidate span, and fail-stop when one rank skips a hop, stops or dies. benchmark/ holds the static ISL 8K / OSL 1K client behind the speed table and its protocol.
With SGLang running MXFP8 linears at 1-8 rows on FlashInfer's low-latency GEMM, its wo_b takes the autotuner's cached tactic for the shape. The publishing GEMM looks that tactic up on eager steps (graph warm-up included) and publishes with it, so its partial equals wo_b's; before tuning it keeps the tactic precompile fixed. A capture replays the last eager choice, so nothing compiles inside it.
SGLang serves DeepSeek-V4.1-Flash's block-FP8 wo_b on FlashInfer's persistent CuTe DSL MXFP8 GEMM with a pinned tactic, not on the low-latency GEMM, so there is no autotuned low-latency wo_b tactic to follow. The lookup only read whatever FlashInfer's autotune cache held for the publish GEMM's shape. Keep the tactic precompile fixed per row count. This reverts commit da109a8.
SGLang now declares enable_decoder_swa_bounded_replay as Optional[bool] and resolves None at startup. The slots serve with replay on or off, so the spec accepts None as well.
…ck step stubs
The covered step now runs the plugin's executor (runtime/) instead of SGLang's
decoder layers: the embedding, the Engram hashes and the mHC broadcast, then per
layer Engram and the capture on the materialized streams and
boundary.pre -> attention.step -> boundary.post("attn") -> moe.step ->
boundary.post("ffn"), then the head and the logits. StepState carries the step
between the step functions.
Each package (boundary/, attention/, moe/, engram/, logits/) has its interface
and a stock implementation. engram/ and logits/ call SGLang's lower-level parts;
boundary/, attention/ and moe/ still call SGLang's stage boundaries, attention and
MoE modules in the order the decoder layer does, until their ports land. Every
kernel slot is off (plan.Facts() builds nothing); the kernels stay in sm103/. The
DSpark draft runs SGLang's own path, so its spec is no longer registered.
…es in the runtime SGLANG_NVIDIA_MEGA_PARITY_SORT_TOPK (test only) moves with the executor from the removed forward.py to runtime/parity.py, unchanged: build installs it.
…d by SGLANG_ENABLE_NVIDIA_MEGAKERNEL The megakernel needs no SGLang interface of its own. plugin.register() (the sglang.srt.plugins entry point) registers two hooks when SGLANG_ENABLE_NVIDIA_MEGAKERNEL=1, and nothing otherwise: - after ModelRunner.load_model: for the DeepSeek-V4.1 target, check the setup on every rank (requirements.py; an unsupported setup stops the server listing every reason) and build the executor before the KV cache is sized; - around DeepseekV4ForCausalLM.forward: MegakernelDecode.plan covers a DSpark verify step of a built width from host state alone, and runs it; every other step runs SGLang's forward. The slot plan routes a width when the MoE step built its GEMM pair. spec.py and draft.py go with the framework they plugged into; the DSpark draft runs SGLang's own path. The contract test resolves every hook target. Prose, logs and variables say megakernel (SGLANG_NVIDIA_MEGAKERNEL_SLOTS, _PARITY, _PARITY_SORT_TOPK).
…the SGLANG_NVIDIA_MEGAKERNEL_ prefix The fault injection (test only) and the serving tests' variables follow the switch's naming: SGLANG_NVIDIA_MEGAKERNEL_FAULT_INJECT and SGLANG_NVIDIA_MEGAKERNEL_E2E_*.
moe.step(layer, m_in, st) runs a covered decode step's FFN the way DeepseekV2MoE.forward does for the covered configuration, calling the block's own gate, top-k, routed experts and shared expert: the dual-stream order at CUDA graph capture, the eager order otherwise. It returns what the decoder layer hands the FFN post (the unmerged pieces, or the deferred finalize). A CPU test runs SGLang's forward and the step on the same stubbed block and compares every call, its stream and its arguments, and the output.
…e MoE step The MoE package builds PR sgl-project#1's routed-expert GEMM pair for every MoE layer (slot moe) and its fast top-k (slot gate), reading SGLang's loaded MXFP4 experts in place; every rank agrees or none is built. At widths whose plan routes, the step runs the dual-stream order with the pair: the fast gate takes the top-k on SGLang's router logits, the pair replaces the routed call and returns the deferred-finalize output, and the gate, the routed-input quant stream and the shared expert on the alternate stream stay as SGLang issues them.
…nature A CPU test calls the slot against an autospec of RoutedExpertGemm: with the fast top-k the launcher gets the routed input's event and the row count, and joins the input itself; with SGLang's top-k the slot joins it before the pair. A change in the launcher's signature fails on CPU, not at the first routed step on a GPU. Public functions of the MoE package gain type hints.
…'s resolved spec config The server argument speculative_num_draft_tokens is unset under DSpark until SGLang resolves it (block size + 1); build reads get_spec(), as SGLang's runners do. The kernel runner's messages say "NVIDIA megakernel".
…y covered width A GB300 serving test: the megakernel on with every slot off, eager verify steps at 1, 2, 4 and 8 requests (6 to 48 rows), the parity switch running SGLang's forward and the megakernel's on each; every logged step must show no difference in the logits and the hidden states.
…ce the forward per instance From the design review, each checked against SGLang 5cbe2371a53: - The one hook is AFTER ModelRunner.initialize, not load_model: the post-load transforms, LoRA, batch-invariant mode and the KV dtype run after load_model; both points come before the scheduler sizes the KV pool. - The megakernel's forward is installed on the target model instance (functools.wraps keeps the signature SGLang reads), not as an AROUND hook on DeepseekV4ForCausalLM.forward, so no other instance or subclass is touched; an instance whose forward is already replaced is refused. - Every rank logs "NVIDIA megakernel: active (...)" once installed (SGLang logs and swallows a plugin's load errors, so tests assert this line). - Refused: deterministic inference, hierarchical sparse attention, decode CUDA graphs other than FULL or disabled, SGLANG_RAGGED_VERIFY_MODE other than static. - The model declares _derived_weight_cache_error, so online weight updates refuse instead of serving stale derived state. - The build's memory is measured from free device memory (symmetric memory bypasses the caching allocator).
…_a; type the step's entry points
boundary/stock.py runs the boundaries between a covered step's sublayers as SGLang's stage boundary does for DeepSeek-V4.1 with the pre-mix carried from the previous sublayer: pre forks the mHC stats stream and collapses the streams into the sublayer's input (or takes the one the previous post built); post reduces the sublayer's TP-partial output in SGLang's fused all-reduce + mHC post with the same next boundary (the FFN norm after attention; the next layer's input norm after the MoE, none before an Engram layer or after the last one). Past the push plane the attention rows are summed on the attention TP group before a plain post, as SGLang's transport does.
…blished wo_b boundary/hop_a.py owns PR sgl-project#1's hop-A unit for the executor: the Lamport mailbox, the publishing wo_b GEMM precompiled for each width up to 8 rows, and the consumer. The attention step publishes through Boundaries.publish_wo_b; the attention post then collects a published partial with the triplet SGLang's post computes and the FFN norm, in place of SGLang's fused collective. Off unless the hop_a slot is on; any other partial takes SGLang's post. The boundary module is renamed boundaries.py.
… compiled it for Slots.hop_a follows the boundary's built hop: on at its compiled widths (1-8 rows, those the publishing GEMM takes), off with the boundary's reason otherwise.
…tion step At hop-A widths the attention step runs wo_b through the boundary's publishing GEMM, which stores the TP-partial rows into the Lamport mailbox, and returns the partial the attention post then collects. Where the GEMM declines the rows, wo_b stays SGLang's with the deferred all-reduce. Adds the attention package's CPU tests: the prepare path is SGLang's choice, and the publish contract.
… settings; review fixes
From the PR review, each checked against SGLang 5cbe2371a53:
- With the megakernel on, every slot this package ships runs (hop_a, moe, gate).
SGLANG_NVIDIA_MEGAKERNEL_SLOTS is an override for tests and bisecting; its names
are stripped and validated (gate needs moe), a requested slot that builds at no
width stops the server, and the startup line reports the slots built and their
widths. The serving tests launch the default and assert the built slots.
- The setup checks and the build read SGLang's resolved settings (the published
config namespaces, else resolution's view), not raw ServerArgs: the raw decode
CUDA graph config is still unset when the model runner is prepared.
- The test-only sorted sparse indices are an AROUND hook registered through SGLang's
HookRegistry (plugin.TEST_HOOKS), not an assignment on SGLang's class.
- Files adapted from SGLang's model code name their origin; NOTICE lists SGLang.
- StepState drops unused fields; one log prefix ("NVIDIA megakernel: "); the
serving tests read SGLANG_NVIDIA_MEGAKERNEL_E2E_MODEL_PATH.
…fuse a late build; drop dead step state - The startup line's 'gate' slot comes from the MoE build's own result, not only from the plan. - The build refuses a model runner that already sized its KV cache or captured its decode graphs (the hook must run inside ModelRunner.initialize). - StepState drops 'hidden' and 'layer', which nothing read.
…e-nvidia-megakernel The switch is SGLang's server flag (ServerArgs.enable_nvidia_megakernel), not an environment variable. plugin.register() always registers its hook after ModelRunner.initialize; the hook returns at once unless the server was started with the flag, so a server without it, or an SGLang that has no such flag, runs untouched. Its name lives in plugin.FLAG / plugin.FLAG_CLI. The startup refusals, the weight-update error, the serving tests, the README and the benchmark README say --enable-nvidia-megakernel; SGLANG_ENABLE_NVIDIA_MEGAKERNEL is gone. The SGLANG_NVIDIA_MEGAKERNEL_* test variables stay.
…framework SGLang now runs the megakernel through sglang.srt.megakernels: with --enable-nvidia-megakernel it checks the setup at argument resolution, builds the registered spec after ModelRunner.initialize and installs its forward. The plugin no longer hooks SGLang's code: plugin.register() registers DeepseekV4ForCausalLM's spec (deepseek_v41/spec.py), and only the parity test keeps its hook. deepseek_v41/spec.py holds what requirements.py held as data for SGLang's checks (TP 4, SM103, 152 SMs, bfloat16, the server arguments, the HF config, the declared numerics) plus the toolchain and FlashInfer GEMM checks; the generic feature probes, the resolved-config reader and the wait deadline are SGLang's now. runtime.build takes SGLang's BuildContext, and the executor returns SGLang's Step and reports its slots through summary(). The step code and kernels are unchanged.
…refusals at launch The verify test now counts SGLang's "NVIDIA megakernel: active (...)" line, once per TP rank with every shipped slot and widths 6 to 48, and the capture line for the decode graphs of 1 to 8 requests. A new e2e test launches with the flag and checks that argument resolution stops the server before the weights load: when SGLANG_PLUGINS leaves the plugin out, and with an unsupported setup, whose reasons are all listed. The benchmark README's startup check follows the new line.
…esolution - spec.API_VERSION pins the sglang.srt.megakernels contract the spec is written against; the spec took SGLang's own constant, so SGLang's version check could never refuse it. - The requirements state the decode CUDA graph backends the covered step is tested with (FULL or none) instead of relying on SGLang's default. - SGLANG_NVIDIA_MEGAKERNEL_SLOTS is checked with the toolchains at argument resolution (config_disable_reasons), before the weights load; the model checks stay after the load. - The spec tests run SGLang's resolution step on the spec the plugin registers and also name: no DSpark, overlapped weight loading, breakable decode graphs, ragged verify, the dtype, the GPU and a bad slot list; the plugin tests check the entry point against SGLang's name and an SGLang without the framework. - The contract pins the entry point name and the BuiltMegakernel members SGLang reads; the package docstring and README say when the checks run.
…per rank All draft steps' Markov sampling in one cooperative NVRTC kernel per rank: each rank keeps its quarter of the Markov head in registers, and ranks agree on every step's token through a symmetric-memory mailbox (a direct exchange at 1 row, a rank-local fold at 2-8 rows, a wide kernel at 9-64 rows). Request state and temperatures are read once per launch. Every cross-rank wait faults at the megakernel deadline; a chain built without logit staging refuses a logits cache. Tests (GB300): tokens and kept logits bitwise vs SGLang's MarkovSampleStep with exactly-summing weights at 1 and 4 ranks, 1-64 rows, CUDA graphs and counter wrap; with real-valued weights, tokens equal and kept logits within one BF16 ulp of the step's bias; a missing rank faults its peers.
…ft spec The DSpark draft gets its own megakernel spec (DeepseekV4ForCausalLMDSpark, registered in plugin.SPECS). Its build covers no step width, so the draft's forward stays SGLang's, and its built object offers draft_sampler: SGLang's folded proposal calls it once with its buffers and, for each batch size the chain serves, runs the step (the last stage's norm, this rank's LM-head shard on FlashInfer's low-M BF16 kernel at 1-16 rows, the Markov chain) in place of compute_base_logits and draft_markov_block. The slots markov and draft_lm_head ship (draft_lm_head needs markov); with markov off, draft_sampler is None and SGLang's proposal runs. A rank that cannot run the chain, a buffer layout the chain does not write, or ranks that built different batch sizes or LM heads refuse on every rank with every rank's reason. The build logs its device memory (mem_get_info). Test-only SGLANG_NVIDIA_MEGAKERNEL_MARKOV_PARITY counts proposals that differ from SGLang's own.
YAMY1234
force-pushed
the
deepseek-v41-draft-sampler
branch
from
October 18, 2026 05:20
692d8bd to
dd798e9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #1; this PR's own changes are its two commits on #1's head
f2b69d2:42b485d(the Markov kernel) anddd798e9(the draft spec and its proposal).It attaches through SGLang's megakernels layer (
sglang.srt.megakernels, sgl-project/sglang#44945, which this PR needs): the plugin registers a draft spec there, and SGLang runs it only on a server started with--enable-nvidia-megakernel. There is noHookRegistryhook and no environment switch.Motivation
Each DSpark step on GB300 ends its draft graph with the proposal: the draft's hidden states go through the LM head, then a Markov chain picks the 5 draft tokens one after another. In SGLang the chain runs per draft step, as separate kernels with a vocab all-gather and a TP sync per step, so at 1 to 8 requests it is a chain of small, latency-bound launches. This PR runs the whole chain in one cooperative kernel per rank, vocab-sharded: each rank keeps its quarter of the Markov head, and the ranks agree on every step's token inside the kernel through a symmetric-memory mailbox. The draft's LM head runs FlashInfer's low-M BF16 kernel at up to 16 rows.
Decode step on GB300, TP4, megakernel off → on with this PR's slots alone, the target on SGLang's own step in both arms (same node, fresh launches, two nodes in opposite orders): −74 to −145 µs at 1–4 requests, −112 / −222 µs at 8, −386 / −439 µs at 64; per-user decode at 1 request +2.5 % on both nodes (741.5 → 760.3 and 733.0 → 751.0 tok/s).
Modifications
How the proposal runs
The DSpark draft gets its own megakernel spec (
deepseek_v41/draft.py, registered forDeepseekV4ForCausalLMDSparkthrough the plugin's entry point next to the target's). With--enable-nvidia-megakernel, SGLang's megakernels framework builds it after the draft model loads:draft_sampler. SGLang's folded DSpark proposal (DsparkDraftSampler, the tail of the draft CUDA graph) calls it once on every rank, with its own buffers (draft tokens, step logits, step temperatures, the draft noise), before any graph is captured.compute_base_logits+draft_markov_block, followed by SGLang's confidence head as before. Every other size runs SGLang's proposal.Per proposal:
mm_bf16(cute-dsl, low-M split-K) at 1–16 rows, tuned at build on every rank; SGLang's projection above 16 rows;It writes the tokens and the kept step logits into SGLang's buffers and advances the noise counter once per block, as SGLang's block does.
Build and failure behavior
markovslot on, a setup the chain cannot serve (another TP size, an FP32 Markov head, an uneven LM-head vocab split, a vocabulary or draft length outside the kernel's limits) refuses at startup on every rank, listing every rank's reason. Nothing falls back silently.draft_samplerserves the same batch sizes, over the group its per-step token sync uses, and refuses on every rank if not: a rank on SGLang's proposal would otherwise wait in a sync its peers skip.BuildContext.deadline_s: half of--watchdog-timeout, at most 60 s). A rank that misses it faults at address 0x0.markov(the chain) anddraft_lm_head(the low-M LM head, which needsmarkov).SGLANG_NVIDIA_MEGAKERNEL_SLOTSoverrides the set for tests and bisecting. Withoutmarkov, the draft spec'sdraft_samplerisNoneand every proposal runs SGLang's.SGLang changes
None in this PR. The draft sampler's call into the framework (
megakernel_draft_sampler) is part of SGLang's megakernels framework PR (sgl-project/sglang#44945).Accuracy Tests
Measured on the earlier plugin-hook route, before SGLang's megakernels layer existed: plugin heads
ce543db(first row) and6a5ec68(last two rows) on plain SGLang5cbe2371a53, the same step and kernel code reached through SGLang's general plugin hooks. On the framework route (this PR's head), the proposal test below shows SGLang's own sampler with the draft spec draws exactly SGLang's proposal; the end-to-end gates on the pushed heads are to follow. GB300, TP4, DSpark block 5, FP8 KV cache,--dsv4-attn-backend flashmla,--watchdog-timeout 600. Every launch logs the proposal built and engaged on all 4 ranks and has 0 server errors.End to end, with a test-only switch (
SGLANG_NVIDIA_MEGAKERNEL_MARKOV_PARITY=1) that runs SGLang's own proposal (compute_base_logits+draft_markov_block) beside the megakernel's inside every captured draft graph, on the same inputs and noise, and counts the differences on the GPU. Traffic: GSM8K-200 (greedy, real acceptance), then sampled requests (temperature 0.8) at 1, 2, 4, 8, 16 and 32 concurrent, so both the narrow (1–8) and the wide (9–64) kernels run.markovmarkovmarkov,draft_lm_headGSM8K over the three launches: 197 / 196 / 196; with both shipped slots, 196 / 200 and accept length 4.283. The draft tokens were equal to SGLang's in all 2,833 compared blocks. The kept step logits are not bit for bit: SGLang computes the Markov bias with a BF16
F.linearand the kernel with MMAs, so the FP32 sums run in a different order and a BF16-rounded bias can differ by one ulp.To follow: needle, KL and #1 vs #1 + this PR at the same accuracy gates.
Kernel tests (
test/kernels, GB300):test_deepseek_v41_markov.py), with Markov weights whose products sum exactly in any order: draft tokens and kept step logits equal SGLang'sMarkovSampleStepbit for bit at 1 and 4 ranks; rows 1, 2, 4, 8 (narrow kernel) and 9, 16, 32, 64 (wide kernel); BF16 and FP32 base logits; this rank's shard or the whole vocabulary; temperatures all 0, all 1 and mixed; a greedy-only build. Under CUDA graph replay: 70,400 launches at 1 rank and 14,000 interleaved launches over all three kernels at 4 ranks, noise offsets up to 2³³, and the launch counters wrapping past 2³².F.linearbias +MarkovSampleStepat 1, 2, 4 and 8 rows: the same draft tokens, and every kept logit within one BF16 ulp of the step's bias (measured: at most 3.3 × 10⁻³ of the step's largest bias, bound 2⁻⁷).test_deepseek_v41_draft_proposal.py, one GPU; plugin2fdf1cc, the same step and kernel code asdd798e9, on SGLang's megakernels branch43690fc3c32): SGLang'smaybe_build_draft_sampler, given the draft spec'sdraft_sampler, draws the same tokens, keeps the same step logits and advances the noise counter as one without it, at 1, 2, 4 and 8 requests, eagerly and under CUDA graph replay.test_deepseek_v41_draft_lm_head.py): against the BF16 projection on the TP4 shard (N 32,320, K 5,120) at 5–16 rows, relative error ≤ 4.7 × 10⁻³ (bound 2⁻⁷), no argmax change; CUDA graph replay.429ba47withdraft_lm_head_mxfp8off, i.e. the BF16 low-M head; SGLangbdb221116de; job 4600679. That plugin is FU2's tree (30e80e6) plus an opt-in MXFP8 head; it differs from this PR's head insampler/__init__.py(that opt-in branch, off here, and test-only parity counters; the BF16 low-M path is unchanged) anddraft.py(the draft-forward slot).CPU tests (
test/unitondd798e9, against sgl-project/sglang#44945's head58fb997cb59): 125 passed, including the draft spec (draft_samplerisNonewithoutmarkov; with it, SGLang's buffers reach the chain as given), every refusal, and the draft noise counter advancing as SGLang's does.Speed Tests and Profiling
Kernels (GB300, TP4, cold L2, CUDA graph replay):
End to end, on the earlier plugin-hook route (plugin
591adbd, plain SGLang5cbe2371a53; the step and kernel code are this PR's): static ISL 8K / OSL 1K, DSpark with synthetic acceptance 3.51. Off = SGLang alone; on = the same launch with the megakernel switched on (at the time theSGLANG_ENABLE_NVIDIA_MEGAKERNEL=1environment variable, which--enable-nvidia-megakernelhas since replaced) andSGLANG_NVIDIA_MEGAKERNEL_SLOTS=markov,draft_lm_head, so only the draft proposal differs (the target runs SGLang's step in both arms). Two nodes, one launch per arm each, in opposite orders (node A off → on, node B on → off); each launch runs the batch sizes forward and then reversed, reported as the mean of the two passes. The step time is the steadier number at fixed acceptance; per-user throughput moves with it within its run-to-run noise.The step is shorter at every batch size on both nodes. To follow on the pushed heads: #1 + this PR against #1, with #1's target slots on in both arms.
Fault behavior
A rank that never joins the chain's exchange (kernel test, 2 s deadline, 4 ranks): its 3 peers fault at 2.1 s, for the direct, fold and wide exchanges (9 of 9 waiting ranks). In serving the deadline is half of
--watchdog-timeout, at most 60 s.Scope and limits
SGLANG_DSPARK_OPT_MARKOV_W2_BF16, SGLang's default) and an LM head split evenly over the vocab.--mem-fraction-static 0.8.Checklist
pre-commit run --all-filespasses (ondd798e9).test/cover the change.5cbe2371a53for the plugin-hook route,43690fc3c32/58fb997cb59for the framework route).