Repository navigation
Conversation
3 of 5 tasks
…ecture, and agree on the entries it offers - install_megakernel builds the binding SGLang registers for the model's architecture (runtime._BINDINGS), and MegakernelBinding for every other one. It hands the binding over as BuildContext.binding, as before. - spec.ENTRIES names the optional entries of a built object: draft_sampler and the EAGLE entries (draft_loop, draft_extend, begin_draft_round, verify_sample and gdn_extend). Every rank must offer the same ones, or every rank refuses. megakernel_entry returns an entry, or None for SGLang's own path. - The startup line of a spec that covers no step names the entries it offers. The API version is unchanged: a built object written without the EAGLE entries offers none of them, and the DSpark draft sampler path is untouched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
megakernels/eagle_binding.py is what binding.py is for DeepSeek-V4.1: the one place the megakernel of a Qwen3.5 target, whose GatedDeltaNet and full-attention layers run under HybridLinearAttnBackend, and of its MTP draft read SGLang's runtime state. runtime._BINDINGS hands EagleBinding to the Qwen3.5 architectures. - A covered step gets EagleForwardView as Step.view: the full-attention metadata (page table, KV lengths, cumulative lengths), the GDN metadata (mamba slots, query offsets, intermediate rows, track slots), the KV pool per full-attention layer with its FP8 scales, and the mamba pool with its ReplaySSM ring. A model without GDN layers, such as the draft, gets no GDN metadata and no mamba pool. - The pools are read once and sealed; assert_live, after the decode graphs are captured, raises once one was reallocated. - The module defines the structs of the EAGLE entries (draft loop, draft extend, draft round, verify sample, GDN extend) and documents what each entry may write. Its TOLERATED lists name what a megakernel may call or read beyond that data: the TP process groups, SGLang's kernel ops, configuration, and the covered forward's own batch fields. test_eagle_binding.py checks that the view holds the backends' own tensors, the sealing, that every struct carries data only, and that every listed name exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nel's draft_loop and draft_extend When the draft model's megakernel offers draft_loop or draft_extend, the EAGLE draft and draft-extend graph runners call the factory once, before capture, with their own outputs: the draft tokens, and the extend's top-1, its probability and its hidden state. For a batch size whose step the factory returns, the graph captures that step in place of SGLang's draft loop, or in place of the draft model step and its top-1. Every rank must serve the same batch sizes, and every other size runs SGLang's code. SGLang keeps the graphs, their replays and the outputs. - begin_draft_round, when offered, runs on the host before a served draft graph replays. It says whether a served draft extend of the same requests produced the round's input. - Draft entries serve graph replays only; an eager draft step runs SGLang's loop. - The worker refuses offered entries it cannot serve: a draft other than a topk-1 chain, rejection sampling, or backends other than GDN with trtllm_mha. speculative/eagle_megakernel.py holds SGLang's side of the entries: the served batch sizes, the outputs, and the provenance of each draft round's input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rget megakernel's verify_sample in them When a megakernel offers EAGLE entries, SGLang captures the verify's post-section in its own graphs: sampling and acceptance, the commit outputs and, on a side stream, the GDN state commit. When the target's megakernel offers verify_sample, the worker calls the factory once after the target graphs are captured. A batch size whose step it returns runs that step there in place of SGLang's sampling, with EagleVerifyIO and EagleVerifyOutputs; every rank must serve the same sizes. - Per key (batch size, sampling mode, forced acceptance, tracking, the verify logits buffer), two slots of staged inputs and outputs alternate, so one round's outputs stay valid while the host consumes them. Batches a graph cannot reproduce run SGLang's eager path: grammar, penalties, seeds, min-p, watermarks, and an eager verify. - The side-stream commit is joined once the draft extend is enqueued. - Every TP rank accepts the same tokens. The forced acceptance length (SGLANG_SIMULATE_ACC_LEN) comes from a generator the graphs own: it is seeded from the server seed and drawn once per round on every rank, whatever else uses the global generator. It reaches the device through a pinned ring, and eagle_sample(simulated_acc_len=) applies it with generate_simulated_accept_index_device, which writes what the host-length version writes. - verify_sample gets a Philox key and a device counter it advances in place (noise_key, noise_offset), as DSpark's draft gets DraftNoise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bitwise up to bs 64 A GPU test runs SGLang's post-verify graphs at batch sizes 1, 3, 16 and 64, greedy and with forced acceptance (real draft tokens). Every round must equal SGLang's eager sampling on the same inputs bitwise: the eager warm-ups, the capture of each slot, and the replays of both alternating slots. A round's outputs must also stay unchanged through the next round, since the host may still be reading them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aptured prefill buckets The FLA chunk and chunk_o kernels take optional caller-built chunk tables, and the Triton GDN extend passes them through; without them the eager path is unchanged. gdn_prefill_graph.py holds a captured bucket's static tables, refilled before each replay by one H2D copy, and gdn_prefill_graph_track.py takes the prefix-cache snapshots of conv windows and SSM states inside the graph, converting a BF16 chunk state before the store. Nothing calls them yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e prefill graphs, the megakernel's when offered When a megakernel offers EAGLE entries, GDNAttnBackend implements the linear-backend prefill-graph protocol, as KDA does. Breakable prefill CUDA graphs then capture each GDN layer's extend instead of breaking at it: the extend launches the eager extend's Triton kernels on the bucket's static tables, which are refilled before every replay. Without an offer nothing changes. - Before any capture, EAGLEWorkerV2 calls the target backend's enable_prefill_graph_extend. The hybrid backend then re-resolves whether breakable prefill graphs capture the linear extend, a decision it first makes at construction, before the entries are known. - When the target's megakernel offers gdn_extend, the captured extend calls it with EagleGdnExtendIO: the bucket's tables, the layer's inputs, the mamba pool and an SGLang-owned zeroed output. The entry also updates the layer's conv and SSM states and track slots, as SGLang's extend does. - A setup the graph-safe chain does not support keeps breaking at every layer and never calls the megakernel's extend. - A GPU test captures one bucket and replays batches of several shapes. It compares the output, the conv and SSM pools and the track slots with the eager forward_extend, bit for bit, for FP32 and BF16 SSM states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he post-verify graphs and the eager verify With --enable-nvidia-megakernel the EAGLE worker owns one generator for SGLANG_SIMULATE_ACC_LEN, seeded from the server seed. The post-verify graphs and the eager verify draw from it once per round, so the forced lengths of a run no longer depend on which rounds a graph served. Without the flag the eager verify draws from the global CPU RNG as before. - EagleMegakernel takes the worker's generator instead of seeding its own. - run_eagle_verify, eagle_sample and generate_simulated_accept_index pass it through; None keeps the global CPU RNG. - Tests: rounds that mix graph and eager paths force the lengths all-graph and all-eager rounds force, on every rank; without the flag the eager draws equal the global RNG's and leave it in the same state. The GPU post-verify test compares against eagle_sample's own host-length draw, with eager rounds in between, which also covers the CPU device-length test this drops. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
force-pushed
the
nv-mega/qwen35-megakernel-binding
branch
from
October 19, 2026 06:03
576dc8a to
adfc015
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
sgl-project#44945 adds
--enable-nvidia-megakernel, the framework for NVIDIA megakernel plugins, and the binding through which the DeepSeek-V4.1 megakernel reads SGLang's runtime state.This PR adds what a Qwen3.5 megakernel needs on the SGLang side. It uses the same shape:
megakernels/eagle_binding.py). It hands the plugin data only:SGLang keeps every runner, worker, schedule batch and CUDA graph; a plugin never gets one. With the flag off, nothing changes.
One framework change is needed:
MegakernelBindingfor every model, and itsForwardViewreads the attention backend'sforward_metadata.HybridLinearAttnBackendkeeps its metadata on its two child backends, full attention and GDN. The view would therefore carrymetadata=None.MegakernelBinding.Modifications
1. Framework (
megakernels/runtime.py,megakernels/spec.py)runtime._BINDINGSmaps an HF architecture to the binding its megakernel gets. Every other architecture getsMegakernelBinding.spec.ENTRIESnames the optional entries a built object may offer:draft_sampler;draft_loop,draft_extend,begin_draft_round,verify_sampleandgdn_extend.megakernel_entry(model, name)returns an offered entry, or None for SGLang's own path.megakernel_agreechecks this at install. That also covers a draft sampler that only some ranks would offer.MEGAKERNEL_API_VERSIONis unchanged. DeepSeek-V4.1's binding, its spec path and its draft sampler are untouched.2. The Qwen3.5 binding (
megakernels/eagle_binding.py, new)It covers the Qwen3.5 target, whose GatedDeltaNet and full-attention layers run under
HybridLinearAttnBackend, and its MTP draft.EagleBindingis handed to the build asBuildContext.binding. Itsforward_view()returns theEagleForwardViewthat a covered step gets asStep.view:attn: the page table, KV lengths and cumulative lengths;gdn: mamba slots, query offsets, intermediate rows and track slots. It is None for the draft.kv: each full-attention layer's K/V buffers and FP8 scales;mamba: the conv and SSM states, the intermediate buffers, and the ReplaySSM ring with its cursors. It is None for the draft.Each pool is sealed at its first read.
assert_liveruns after the decode graphs are captured.The entries' structs carry tensors and numbers only:
EagleDraftLoopIO,EagleDraftExtendIO,EagleDraftRoundIO,EagleVerifyIO/EagleVerifyOutputs, andEagleGdnExtendIO/GdnExtendTables. The module docstring states what each entry may write.What crosses besides data is listed as literal tuples, so a plugin's contract test can pin them:
3. EAGLE draft graphs (
eagle_draft_cuda_graph_runner.py,eagle_draft_extend_cuda_graph_runner.py,speculative/eagle_megakernel.py)draft_loopordraft_extend, the graph runner calls the factory once, before capture, with its own outputs.draft_loopreplaces SGLang's draft loop;draft_extendreplaces the draft model step and its top-1.begin_draft_roundruns on the host before a served draft graph replays.trtllm_mha.4. Post-verify graphs (
speculative/eagle_megakernel.py,eagle_utils.py,eagle_worker_v2.py,eagle_worker_common.py,spec_utils.py)verify_sampleserves runs that step there, in place of SGLang's sampling.verify_samplegets a Philox key and a device counter, as DSpark's draft getsDraftNoise.5. GDN extend in breakable prefill graphs (
linear/gdn_backend.py,linear/gdn_prefill_graph.py,fla/chunk.py,fla/chunk_o.py,fla/gdn_prefill_graph_track.py,hybrid_linear_attn_backend.py)GDNAttnBackendimplements the linear-backend prefill-graph protocol, as KDA does. Breakable prefill graphs then capture each GDN layer's extend instead of breaking at it, using static tables that are refilled before each replay.gdn_extend, the captured extend calls it.6. One forced acceptance length stream (
[Fix])--enable-nvidia-megakernel, the EAGLE worker owns one generator for the forced acceptance length (SGLANG_SIMULATE_ACC_LEN), seeded from the server seed.Known limits
assert_liveruns at the capture check only.Accuracy Tests
With the flag off, SGLang's behavior is unchanged: every new path runs only when a megakernel offers EAGLE entries, and the forced-length generator exists only with the flag.
Tests are behaviour-level, in sgl-project#44945's style: one CPU test per behaviour this PR guarantees, on sgl-project#44945's shared fakes, 14 CPU tests in all, plus two GPU test files. The behaviours covered:
assert_live;CPU, on this head:
pre-commit, with the repo's pinned hooks, passes on every changed file and modifies nothing.Unit suites. Each was run as
PYTHONPATH=python python3 -m pytest -q -rfE -p no:cacheprovider --continue-on-collection-errors <suite>, on this PR and on its base ([Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins sgl-project/sglang#44945's head). The sorted failing test IDs are identical on both heads.test/registered/unit/megakernels/test/registered/unit/spec/test/registered/unit/layers/attention/test/registered/unit/model_executor/runner/spec/are this PR's GPU tests.tvm_ffiorflashinfer.GPU, GB300, one GPU, all bitwise:
test_eagle_post_verify_graphs.py: 2 passed, 96 subtests. At batch sizes 1, 3, 16 and 64, greedy and with forced acceptance, SGLang's post-verify graphs equal SGLang's eager sampling bitwise in every round. That covers the eager warm-ups, the capture of each slot, and the replays of both slots. The test also checks two more things:test_gdn_prefill_graph.py: 1 passed, 10 subtests. A captured bucket, replayed at several shapes, equals the eagerforward_extendbitwise, for FP32 and BF16 SSM states. The comparison covers the output, the conv and SSM pools, and the track slots.With a Qwen3.5 megakernel plugin, at TP8 on GB300, on this PR's head:
d5bc21f). Every slot is on:SGLANG_NVIDIA_MEGAKERNEL_SLOTS=route,mkm,gdn_verify,gdn_pdl,hop. The runs "with the entry" addverify_sample, a factory that returns no step at any batch size, so SGLang's post-verify graphs sample and accept with SGLang's own code.nvidia/Qwen3.5-397B-A17B-NVFP4-V2, revision8f590eae8f10bf55d9a46f79ea0280bde435c9f8.SGLANG_SIMULATE_ACC_LEN=4.8,SGLANG_SIMULATE_ACC_LEN=5, and real acceptance.entries:NVIDIA megakernel: active (Qwen3.5: slots ['gdn_pdl', 'gdn_verify', 'hop', 'mkm', 'route']; entries ['verify_sample']; widths [7]; bitwise numerics; built in 3.4 s, 636.0 MiB; wait deadline 60 s)NVIDIA megakernel: Qwen3.5 runs the decode graphs of batch sizes [1] (1 of 1); every other step runs the standard forwardNVIDIA megakernel: captured the EAGLE post-verify graphs for batch size 1. No rank logs it in a run without the entry.SGLANG_SIMULATE_ACC_LEN=4.8, with both arms on--random-seed 20261019: 7 of 7 requests token-identical (672 tokens). Verify steps are equal (140 vs 140).SGLANG_SIMULATE_ACC_LEN=5: 7 of 7 token-identical (672 tokens); verify steps equal (133 vs 133).The plugin does not offer the draft entries or
gdn_extendyet. Those paths are covered by the GPU tests above.Speed Tests and Profiling
With the flag off nothing changes. The plugin's end-to-end speed will be reported with the plugin.
Checklist
🤖 Generated with Claude Code