Skip to content

[Perf] [GB300] deepseek_v41: Megakernel DSpark draft proposal (sharded Markov chain + low-M draft LM head) - #3

Closed
YAMY1234 wants to merge 32 commits into
sgl-project:mainfrom
YAMY1234:deepseek-v41-draft-sampler
Closed

YAMY1234 wants to merge 32 commits into
sgl-project:mainfrom
YAMY1234:deepseek-v41-draft-sampler

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Based on #1; this PR's own changes are its two commits on #1's head f2b69d2: 42b485d (the Markov kernel) and dd798e9 (the draft spec and its proposal).

It attaches through SGLang's megakernels layer (sglang.srt.megakernels, sgl-project/sglang#44945, which this PR needs): the plugin registers a draft spec there, and SGLang runs it only on a server started with --enable-nvidia-megakernel. There is no HookRegistry hook and no environment switch.

Motivation

Each DSpark step on GB300 ends its draft graph with the proposal: the draft's hidden states go through the LM head, then a Markov chain picks the 5 draft tokens one after another. In SGLang the chain runs per draft step, as separate kernels with a vocab all-gather and a TP sync per step, so at 1 to 8 requests it is a chain of small, latency-bound launches. This PR runs the whole chain in one cooperative kernel per rank, vocab-sharded: each rank keeps its quarter of the Markov head, and the ranks agree on every step's token inside the kernel through a symmetric-memory mailbox. The draft's LM head runs FlashInfer's low-M BF16 kernel at up to 16 rows.

Decode step on GB300, TP4, megakernel off → on with this PR's slots alone, the target on SGLang's own step in both arms (same node, fresh launches, two nodes in opposite orders): −74 to −145 µs at 1–4 requests, −112 / −222 µs at 8, −386 / −439 µs at 64; per-user decode at 1 request +2.5 % on both nodes (741.5 → 760.3 and 733.0 → 751.0 tok/s).

Modifications

How the proposal runs

The DSpark draft gets its own megakernel spec (deepseek_v41/draft.py, registered for DeepseekV4ForCausalLMDSpark through the plugin's entry point next to the target's). With --enable-nvidia-megakernel, SGLang's megakernels framework builds it after the draft model loads:

  • Its build covers no step width, so the draft's forward stays SGLang's on every step.
  • Its built object offers draft_sampler. SGLang's folded DSpark proposal (DsparkDraftSampler, the tail of the draft CUDA graph) calls it once on every rank, with its own buffers (draft tokens, step logits, step temperatures, the draft noise), before any graph is captured.
  • For each batch size the chain serves (1 to 64 requests), the returned step runs in place of compute_base_logits + draft_markov_block, followed by SGLang's confidence head as before. Every other size runs SGLang's proposal.

Per proposal:

  1. the last stage's norm (SGLang's module);
  2. this rank's LM-head shard: FlashInfer mm_bf16 (cute-dsl, low-M split-K) at 1–16 rows, tuned at build on every rank; SGLang's projection above 16 rows;
  3. the draft noise for this rank's vocab shard (SGLang's Philox key, counters and offset), drawn on a side stream while the LM head runs;
  4. the chain: one kernel per rank for all 5 steps. 1 request: a direct exchange; 2–8 requests: a rank-local fold; 9–64 requests: the wide kernel. Request state and temperatures are read once per launch.

It writes the tokens and the kept step logits into SGLang's buffers and advances the noise counter once per block, as SGLang's block does.

Build and failure behavior

  • With the markov slot on, a setup the chain cannot serve (another TP size, an FP32 Markov head, an uneven LM-head vocab split, a vocabulary or draft length outside the kernel's limits) refuses at startup on every rank, listing every rank's reason. Nothing falls back silently.
  • The chain's kernels are compiled (NVRTC) and their co-residency checked before the collective set-up; a failure on any rank raises on every rank. After the build, the ranks compare the batch sizes they serve and the LM head they use, and refuse if they differ (their captured graphs would wait on each other).
  • Peer buffers are symmetric memory, allocated when the draft graphs are built: ≈ 395 MiB per rank for a sampled draft at TP4 (staging for the kept step logits of up to 64 requests), under 1 MiB for a greedy draft.
  • SGLang's draft sampler ([Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins sglang#44945) also checks that every rank's draft_sampler serves the same batch sizes, over the group its per-step token sync uses, and refuses on every rank if not: a rank on SGLang's proposal would otherwise wait in a sync its peers skip.
  • Every wait on another GPU has the framework's deadline (BuildContext.deadline_s: half of --watchdog-timeout, at most 60 s). A rank that misses it faults at address 0x0.
  • Two slots ship and run by default with the megakernel on: markov (the chain) and draft_lm_head (the low-M LM head, which needs markov). SGLANG_NVIDIA_MEGAKERNEL_SLOTS overrides the set for tests and bisecting. Without markov, the draft spec's draft_sampler is None and every proposal runs SGLang's.

SGLang changes

None in this PR. The draft sampler's call into the framework (megakernel_draft_sampler) is part of SGLang's megakernels framework PR (sgl-project/sglang#44945).

Accuracy Tests

Measured on the earlier plugin-hook route, before SGLang's megakernels layer existed: plugin heads ce543db (first row) and 6a5ec68 (last two rows) on plain SGLang 5cbe2371a53, the same step and kernel code reached through SGLang's general plugin hooks. On the framework route (this PR's head), the proposal test below shows SGLang's own sampler with the draft spec draws exactly SGLang's proposal; the end-to-end gates on the pushed heads are to follow. GB300, TP4, DSpark block 5, FP8 KV cache, --dsv4-attn-backend flashmla, --watchdog-timeout 600. Every launch logs the proposal built and engaged on all 4 ranks and has 0 server errors.

End to end, with a test-only switch (SGLANG_NVIDIA_MEGAKERNEL_MARKOV_PARITY=1) that runs SGLang's own proposal (compute_base_logits + draft_markov_block) beside the megakernel's inside every captured draft graph, on the same inputs and noise, and counts the differences on the GPU. Traffic: GSM8K-200 (greedy, real acceptance), then sampled requests (temperature 0.8) at 1, 2, 4, 8, 16 and 32 concurrent, so both the narrow (1–8) and the wide (9–64) kernels run.

Slots GSM8K-200 Accept length Draft blocks compared Draft tokens that differ Kept step logits: largest difference
markov 197 / 200 4.294 908 0 one BF16 ulp of the bias
markov 196 / 200 4.286 835 0 1.0 at |logit| ≤ 255 (one BF16 ulp)
markov,draft_lm_head 196 / 200 4.283 1,090 0 1.0 at |logit| ≤ 257 (one BF16 ulp)

GSM8K over the three launches: 197 / 196 / 196; with both shipped slots, 196 / 200 and accept length 4.283. The draft tokens were equal to SGLang's in all 2,833 compared blocks. The kept step logits are not bit for bit: SGLang computes the Markov bias with a BF16 F.linear and the kernel with MMAs, so the FP32 sums run in a different order and a BF16-rounded bias can differ by one ulp.

To follow: needle, KL and #1 vs #1 + this PR at the same accuracy gates.

Kernel tests (test/kernels, GB300):

  • The chain (test_deepseek_v41_markov.py), with Markov weights whose products sum exactly in any order: draft tokens and kept step logits equal SGLang's MarkovSampleStep bit for bit at 1 and 4 ranks; rows 1, 2, 4, 8 (narrow kernel) and 9, 16, 32, 64 (wide kernel); BF16 and FP32 base logits; this rank's shard or the whole vocabulary; temperatures all 0, all 1 and mixed; a greedy-only build. Under CUDA graph replay: 70,400 launches at 1 rank and 14,000 interleaved launches over all three kernels at 4 ranks, noise offsets up to 2³³, and the launch counters wrapping past 2³².
  • The chain with real-valued weights, against SGLang's F.linear bias + MarkovSampleStep at 1, 2, 4 and 8 rows: the same draft tokens, and every kept logit within one BF16 ulp of the step's bias (measured: at most 3.3 × 10⁻³ of the step's largest bias, bound 2⁻⁷).
  • A rank that never joins: its 3 peers fault at 2.1 s with a 2 s deadline, for the direct, fold and wide exchanges.
  • The proposal on SGLang's own sampler (test_deepseek_v41_draft_proposal.py, one GPU; plugin 2fdf1cc, the same step and kernel code as dd798e9, on SGLang's megakernels branch 43690fc3c32): SGLang's maybe_build_draft_sampler, given the draft spec's draft_sampler, draws the same tokens, keeps the same step logits and advances the noise counter as one without it, at 1, 2, 4 and 8 requests, eagerly and under CUDA graph replay.
  • The draft LM head (test_deepseek_v41_draft_lm_head.py): against the BF16 projection on the TP4 shard (N 32,320, K 5,120) at 5–16 rows, relative error ≤ 4.7 × 10⁻³ (bound 2⁻⁷), no argmax change; CUDA graph replay.
  • The draft LM head on the full model: over the GSM8K test set (1,319 prompts, greedy) served at concurrency 1 and then 8, with the parity check running SGLang's own proposal on the same hidden states and random draws, the low-M head's proposed tokens equal SGLang's at all 213,280 draft positions of the batch-1 blocks, on 4 / 4 ranks (0 mismatches). Run: plugin 429ba47 with draft_lm_head_mxfp8 off, i.e. the BF16 low-M head; SGLang bdb221116de; job 4600679. That plugin is FU2's tree (30e80e6) plus an opt-in MXFP8 head; it differs from this PR's head in sampler/__init__.py (that opt-in branch, off here, and test-only parity counters; the BF16 low-M path is unchanged) and draft.py (the draft-forward slot).

CPU tests (test/unit on dd798e9, against sgl-project/sglang#44945's head 58fb997cb59): 125 passed, including the draft spec (draft_sampler is None without markov; with it, SGLang's buffers reach the chain as given), every refusal, and the draft noise counter advancing as SGLang's does.

Speed Tests and Profiling

Kernels (GB300, TP4, cold L2, CUDA graph replay):

Requests Chain, µs per draft block (greedy)
1 36.2
2 41.4
4 42.0
8 40.1
16 61.4
32 90.2
64 150.8
LM-head rows Low-M (µs) cuBLAS (µs)
5 47.4 55.3
10 48.9 51.2
15 49.3 51.3

End to end, on the earlier plugin-hook route (plugin 591adbd, plain SGLang 5cbe2371a53; the step and kernel code are this PR's): static ISL 8K / OSL 1K, DSpark with synthetic acceptance 3.51. Off = SGLang alone; on = the same launch with the megakernel switched on (at the time the SGLANG_ENABLE_NVIDIA_MEGAKERNEL=1 environment variable, which --enable-nvidia-megakernel has since replaced) and SGLANG_NVIDIA_MEGAKERNEL_SLOTS=markov,draft_lm_head, so only the draft proposal differs (the target runs SGLang's step in both arms). Two nodes, one launch per arm each, in opposite orders (node A off → on, node B on → off); each launch runs the batch sizes forward and then reversed, reported as the mean of the two passes. The step time is the steadier number at fixed acceptance; per-user throughput moves with it within its run-to-run noise.

Requests Node B step off → on (ms) Change Node B tok/s/user off → on Node A step change Node A tok/s/user off → on
1 4.759 → 4.621 −138 µs 741.5 → 760.3 (+2.5 %) −103 µs 733.0 → 751.0 (+2.5 %)
2 5.553 → 5.408 −145 µs 633.5 → 647.7 (+2.2 %) −74 µs 632.9 → 645.2 (+2.0 %)
4 6.405 → 6.261 −144 µs 531.2 → 528.6 (−0.5 %) −186 µs 524.5 → 528.4 (+0.7 %)
8 8.299 → 8.077 −222 µs 360.6 → 365.4 (+1.4 %) −112 µs 355.9 → 358.7 (+0.8 %)
16 10.569 → 10.377 −192 µs 244.8 → 246.1 (+0.5 %) −186 µs 244.8 → 246.8 (+0.8 %)
32 13.139 → 12.793 −347 µs 159.1 → 160.5 (+0.9 %) −146 µs 161.4 → 158.4 (−1.9 %)
64 17.846 → 17.460 −386 µs 95.8 → 96.0 (+0.2 %) −439 µs 95.8 → 96.6 (+0.7 %)

The step is shorter at every batch size on both nodes. To follow on the pushed heads: #1 + this PR against #1, with #1's target slots on in both arms.

Fault behavior

A rank that never joins the chain's exchange (kernel test, 2 s deadline, 4 ranks): its 3 peers fault at 2.1 s, for the direct, fold and wide exchanges (9 of 9 waiting ranks). In serving the deadline is half of --watchdog-timeout, at most 60 s.

Scope and limits

  • GB300 (SM103), TP 4; DeepSeek-V4.1-Flash's DSpark draft with a BF16 rank-256 Markov head (SGLANG_DSPARK_OPT_MARKOV_W2_BF16, SGLang's default) and an LM head split evenly over the vocab.
  • The chain serves 1–64 requests; the low-M LM head 1–16 rows (1–3 requests).
  • The chain's peer buffers (≈ 395 MiB per rank for a sampled draft) are allocated when SGLang builds the draft graphs, after it sizes the KV cache; every launch above booted at --mem-fraction-static 0.8.

Checklist

  • pre-commit run --all-files passes (on dd798e9).
  • Tests under test/ cover the change.
  • The SGLang commits these were tested against are named above (5cbe2371a53 for the plugin-hook route, 43690fc3c32 / 58fb997cb59 for the framework route).
  • Accuracy and speed results are attached; the framework-route end-to-end gates are to follow.

Registers the backend nvidia with SGLang's monokernel framework through the
sglang.srt.plugins entry point. Its specs serve DeepSeek-V4.1-Flash DSpark verify
steps of 1 to 48 rows on SM103 at TP4, and the DSpark draft stages, with the
flashmla attention backend and an FP8 KV cache. Every other step runs SGLang's
standard path, and nothing beyond the entry point is imported without the flag.

A routed step rebuilds SGLang's own step from SGLang's modules and stage
boundaries, and fills three slots with this package's kernels:
- at 1 to 8 rows, wo_b's GEMM epilogue publishes into a Lamport mailbox on every
  rank, and a Lamport consumer runs the attention all-reduce with the mHC post,
  combine and RMSNorm as the attention post's fused reduction;
- at 1 to 48 rows, the routed experts run on a route kernel and a pair of 8-row
  tcgen05 block-scaled GEMMs inside SGLang's dual-stream MoE;
- at 1 to 8 and 48 rows, the router's top-k runs in one CTA per row, bit for
  bit SGLang's fused gate.
Each slot is probed at build and stays off, with the reason logged, on an SGLang
without its seam.

The mailbox, the kernels for every routed width and their tactics are built and
compiled at startup, before the KV cache is sized. Every wait on another GPU has
a deadline below SGLang's watchdog and faults at address 0x0 when missed.

Tests: CPU contract tests pin every SGLang name the package reads; kernel tests
on GB300 check each kernel against SGLang's or FlashInfer's op and the fail-stop
deadline.
test/e2e launches DeepSeek-V4.1-Flash servers with and without the monokernel
backend: GSM8K across the largest covered width, KL of covered steps and of long
prompts against the normal prefill path, a passkey past the indexer's candidate
span, and fail-stop when one rank skips a hop, stops or dies. benchmark/ holds the
static ISL 8K / OSL 1K client behind the speed table and its protocol.
With SGLang running MXFP8 linears at 1-8 rows on FlashInfer's low-latency GEMM,
its wo_b takes the autotuner's cached tactic for the shape. The publishing GEMM
looks that tactic up on eager steps (graph warm-up included) and publishes with
it, so its partial equals wo_b's; before tuning it keeps the tactic precompile
fixed. A capture replays the last eager choice, so nothing compiles inside it.
SGLang serves DeepSeek-V4.1-Flash's block-FP8 wo_b on FlashInfer's
persistent CuTe DSL MXFP8 GEMM with a pinned tactic, not on the
low-latency GEMM, so there is no autotuned low-latency wo_b tactic to
follow. The lookup only read whatever FlashInfer's autotune cache held for
the publish GEMM's shape. Keep the tactic precompile fixed per row count.

This reverts commit da109a8.
SGLang now declares enable_decoder_swa_bounded_replay as Optional[bool] and
resolves None at startup. The slots serve with replay on or off, so the spec
accepts None as well.
…ck step stubs

The covered step now runs the plugin's executor (runtime/) instead of SGLang's
decoder layers: the embedding, the Engram hashes and the mHC broadcast, then per
layer Engram and the capture on the materialized streams and
boundary.pre -> attention.step -> boundary.post("attn") -> moe.step ->
boundary.post("ffn"), then the head and the logits. StepState carries the step
between the step functions.

Each package (boundary/, attention/, moe/, engram/, logits/) has its interface
and a stock implementation. engram/ and logits/ call SGLang's lower-level parts;
boundary/, attention/ and moe/ still call SGLang's stage boundaries, attention and
MoE modules in the order the decoder layer does, until their ports land. Every
kernel slot is off (plan.Facts() builds nothing); the kernels stay in sm103/. The
DSpark draft runs SGLang's own path, so its spec is no longer registered.
…es in the runtime

SGLANG_NVIDIA_MEGA_PARITY_SORT_TOPK (test only) moves with the executor from the
removed forward.py to runtime/parity.py, unchanged: build installs it.
…d by SGLANG_ENABLE_NVIDIA_MEGAKERNEL

The megakernel needs no SGLang interface of its own. plugin.register() (the
sglang.srt.plugins entry point) registers two hooks when
SGLANG_ENABLE_NVIDIA_MEGAKERNEL=1, and nothing otherwise:
- after ModelRunner.load_model: for the DeepSeek-V4.1 target, check the setup on
  every rank (requirements.py; an unsupported setup stops the server listing every
  reason) and build the executor before the KV cache is sized;
- around DeepseekV4ForCausalLM.forward: MegakernelDecode.plan covers a DSpark
  verify step of a built width from host state alone, and runs it; every other
  step runs SGLang's forward.

The slot plan routes a width when the MoE step built its GEMM pair. spec.py and
draft.py go with the framework they plugged into; the DSpark draft runs SGLang's
own path. The contract test resolves every hook target. Prose, logs and variables
say megakernel (SGLANG_NVIDIA_MEGAKERNEL_SLOTS, _PARITY, _PARITY_SORT_TOPK).
…the SGLANG_NVIDIA_MEGAKERNEL_ prefix

The fault injection (test only) and the serving tests' variables follow the
switch's naming: SGLANG_NVIDIA_MEGAKERNEL_FAULT_INJECT and
SGLANG_NVIDIA_MEGAKERNEL_E2E_*.
moe.step(layer, m_in, st) runs a covered decode step's FFN the way
DeepseekV2MoE.forward does for the covered configuration, calling the block's
own gate, top-k, routed experts and shared expert: the dual-stream order at
CUDA graph capture, the eager order otherwise. It returns what the decoder
layer hands the FFN post (the unmerged pieces, or the deferred finalize). A
CPU test runs SGLang's forward and the step on the same stubbed block and
compares every call, its stream and its arguments, and the output.
…e MoE step

The MoE package builds PR sgl-project#1's routed-expert GEMM pair for every MoE layer
(slot moe) and its fast top-k (slot gate), reading SGLang's loaded MXFP4
experts in place; every rank agrees or none is built. At widths whose plan
routes, the step runs the dual-stream order with the pair: the fast gate takes
the top-k on SGLang's router logits, the pair replaces the routed call and
returns the deferred-finalize output, and the gate, the routed-input quant
stream and the shared expert on the alternate stream stay as SGLang issues
them.
…nature

A CPU test calls the slot against an autospec of RoutedExpertGemm: with the
fast top-k the launcher gets the routed input's event and the row count, and
joins the input itself; with SGLang's top-k the slot joins it before the pair.
A change in the launcher's signature fails on CPU, not at the first routed
step on a GPU. Public functions of the MoE package gain type hints.
…'s resolved spec config

The server argument speculative_num_draft_tokens is unset under DSpark until
SGLang resolves it (block size + 1); build reads get_spec(), as SGLang's runners
do. The kernel runner's messages say "NVIDIA megakernel".
…y covered width

A GB300 serving test: the megakernel on with every slot off, eager verify steps at
1, 2, 4 and 8 requests (6 to 48 rows), the parity switch running SGLang's forward
and the megakernel's on each; every logged step must show no difference in the
logits and the hidden states.
…ce the forward per instance

From the design review, each checked against SGLang 5cbe2371a53:
- The one hook is AFTER ModelRunner.initialize, not load_model: the post-load
  transforms, LoRA, batch-invariant mode and the KV dtype run after load_model;
  both points come before the scheduler sizes the KV pool.
- The megakernel's forward is installed on the target model instance
  (functools.wraps keeps the signature SGLang reads), not as an AROUND hook on
  DeepseekV4ForCausalLM.forward, so no other instance or subclass is touched; an
  instance whose forward is already replaced is refused.
- Every rank logs "NVIDIA megakernel: active (...)" once installed (SGLang logs and
  swallows a plugin's load errors, so tests assert this line).
- Refused: deterministic inference, hierarchical sparse attention, decode CUDA
  graphs other than FULL or disabled, SGLANG_RAGGED_VERIFY_MODE other than static.
- The model declares _derived_weight_cache_error, so online weight updates refuse
  instead of serving stale derived state.
- The build's memory is measured from free device memory (symmetric memory
  bypasses the caching allocator).
boundary/stock.py runs the boundaries between a covered step's sublayers as
SGLang's stage boundary does for DeepSeek-V4.1 with the pre-mix carried from
the previous sublayer: pre forks the mHC stats stream and collapses the
streams into the sublayer's input (or takes the one the previous post built);
post reduces the sublayer's TP-partial output in SGLang's fused all-reduce +
mHC post with the same next boundary (the FFN norm after attention; the next
layer's input norm after the MoE, none before an Engram layer or after the
last one). Past the push plane the attention rows are summed on the attention
TP group before a plain post, as SGLang's transport does.
…blished wo_b

boundary/hop_a.py owns PR sgl-project#1's hop-A unit for the executor: the Lamport
mailbox, the publishing wo_b GEMM precompiled for each width up to 8 rows,
and the consumer. The attention step publishes through
Boundaries.publish_wo_b; the attention post then collects a published partial
with the triplet SGLang's post computes and the FFN norm, in place of SGLang's
fused collective. Off unless the hop_a slot is on; any other partial takes
SGLang's post. The boundary module is renamed boundaries.py.
… compiled it for

Slots.hop_a follows the boundary's built hop: on at its compiled widths (1-8 rows,
those the publishing GEMM takes), off with the boundary's reason otherwise.
…tion step

At hop-A widths the attention step runs wo_b through the boundary's publishing
GEMM, which stores the TP-partial rows into the Lamport mailbox, and returns the
partial the attention post then collects. Where the GEMM declines the rows, wo_b
stays SGLang's with the deferred all-reduce. Adds the attention package's CPU
tests: the prepare path is SGLang's choice, and the publish contract.
… settings; review fixes

From the PR review, each checked against SGLang 5cbe2371a53:
- With the megakernel on, every slot this package ships runs (hop_a, moe, gate).
  SGLANG_NVIDIA_MEGAKERNEL_SLOTS is an override for tests and bisecting; its names
  are stripped and validated (gate needs moe), a requested slot that builds at no
  width stops the server, and the startup line reports the slots built and their
  widths. The serving tests launch the default and assert the built slots.
- The setup checks and the build read SGLang's resolved settings (the published
  config namespaces, else resolution's view), not raw ServerArgs: the raw decode
  CUDA graph config is still unset when the model runner is prepared.
- The test-only sorted sparse indices are an AROUND hook registered through SGLang's
  HookRegistry (plugin.TEST_HOOKS), not an assignment on SGLang's class.
- Files adapted from SGLang's model code name their origin; NOTICE lists SGLang.
- StepState drops unused fields; one log prefix ("NVIDIA megakernel: "); the
  serving tests read SGLANG_NVIDIA_MEGAKERNEL_E2E_MODEL_PATH.
…fuse a late build; drop dead step state

- The startup line's 'gate' slot comes from the MoE build's own result, not only
  from the plan.
- The build refuses a model runner that already sized its KV cache or captured its
  decode graphs (the hook must run inside ModelRunner.initialize).
- StepState drops 'hidden' and 'layer', which nothing read.
…e-nvidia-megakernel

The switch is SGLang's server flag (ServerArgs.enable_nvidia_megakernel), not an
environment variable. plugin.register() always registers its hook after
ModelRunner.initialize; the hook returns at once unless the server was started with
the flag, so a server without it, or an SGLang that has no such flag, runs
untouched. Its name lives in plugin.FLAG / plugin.FLAG_CLI. The startup refusals,
the weight-update error, the serving tests, the README and the benchmark README say
--enable-nvidia-megakernel; SGLANG_ENABLE_NVIDIA_MEGAKERNEL is gone. The
SGLANG_NVIDIA_MEGAKERNEL_* test variables stay.
…framework

SGLang now runs the megakernel through sglang.srt.megakernels: with
--enable-nvidia-megakernel it checks the setup at argument resolution, builds
the registered spec after ModelRunner.initialize and installs its forward. The
plugin no longer hooks SGLang's code: plugin.register() registers
DeepseekV4ForCausalLM's spec (deepseek_v41/spec.py), and only the parity test
keeps its hook.

deepseek_v41/spec.py holds what requirements.py held as data for SGLang's
checks (TP 4, SM103, 152 SMs, bfloat16, the server arguments, the HF config,
the declared numerics) plus the toolchain and FlashInfer GEMM checks; the
generic feature probes, the resolved-config reader and the wait deadline are
SGLang's now. runtime.build takes SGLang's BuildContext, and the executor
returns SGLang's Step and reports its slots through summary(). The step code
and kernels are unchanged.
…refusals at launch

The verify test now counts SGLang's "NVIDIA megakernel: active (...)" line, once per
TP rank with every shipped slot and widths 6 to 48, and the capture line for the
decode graphs of 1 to 8 requests. A new e2e test launches with the flag and checks
that argument resolution stops the server before the weights load: when
SGLANG_PLUGINS leaves the plugin out, and with an unsupported setup, whose reasons
are all listed. The benchmark README's startup check follows the new line.
…esolution

- spec.API_VERSION pins the sglang.srt.megakernels contract the spec is written
  against; the spec took SGLang's own constant, so SGLang's version check could
  never refuse it.
- The requirements state the decode CUDA graph backends the covered step is
  tested with (FULL or none) instead of relying on SGLang's default.
- SGLANG_NVIDIA_MEGAKERNEL_SLOTS is checked with the toolchains at argument
  resolution (config_disable_reasons), before the weights load; the model checks
  stay after the load.
- The spec tests run SGLang's resolution step on the spec the plugin registers
  and also name: no DSpark, overlapped weight loading, breakable decode graphs,
  ragged verify, the dtype, the GPU and a bad slot list; the plugin tests check
  the entry point against SGLang's name and an SGLang without the framework.
- The contract pins the entry point name and the BuiltMegakernel members SGLang
  reads; the package docstring and README say when the checks run.
…per rank

All draft steps' Markov sampling in one cooperative NVRTC kernel per rank:
each rank keeps its quarter of the Markov head in registers, and ranks
agree on every step's token through a symmetric-memory mailbox (a direct
exchange at 1 row, a rank-local fold at 2-8 rows, a wide kernel at 9-64
rows). Request state and temperatures are read once per launch. Every
cross-rank wait faults at the megakernel deadline; a chain built without
logit staging refuses a logits cache.

Tests (GB300): tokens and kept logits bitwise vs SGLang's MarkovSampleStep
with exactly-summing weights at 1 and 4 ranks, 1-64 rows, CUDA graphs and
counter wrap; with real-valued weights, tokens equal and kept logits
within one BF16 ulp of the step's bias; a missing rank faults its peers.
…ft spec

The DSpark draft gets its own megakernel spec (DeepseekV4ForCausalLMDSpark,
registered in plugin.SPECS). Its build covers no step width, so the draft's
forward stays SGLang's, and its built object offers draft_sampler: SGLang's
folded proposal calls it once with its buffers and, for each batch size the
chain serves, runs the step (the last stage's norm, this rank's LM-head shard
on FlashInfer's low-M BF16 kernel at 1-16 rows, the Markov chain) in place of
compute_base_logits and draft_markov_block.

The slots markov and draft_lm_head ship (draft_lm_head needs markov); with
markov off, draft_sampler is None and SGLang's proposal runs. A rank that
cannot run the chain, a buffer layout the chain does not write, or ranks that
built different batch sizes or LM heads refuse on every rank with every
rank's reason. The build logs its device memory (mem_get_info). Test-only
SGLANG_NVIDIA_MEGAKERNEL_MARKOV_PARITY counts proposals that differ from
SGLang's own.
@YAMY1234
YAMY1234 force-pushed the deepseek-v41-draft-sampler branch from 692d8bd to dd798e9 Compare October 18, 2026 05:20
@YAMY1234 YAMY1234 closed this Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant