Skip to content

[Feat] [NVIDIA] megakernels: Qwen3.5 binding and EAGLE entries for NVIDIA megakernel plugins - #41

Closed
YAMY1234 wants to merge 8 commits into
nv-mega/enable-nvidia-megakernel-prfrom
nv-mega/qwen35-megakernel-binding
Closed

YAMY1234 wants to merge 8 commits into
nv-mega/enable-nvidia-megakernel-prfrom
nv-mega/qwen35-megakernel-binding

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Stacked on sgl-project#44945. This PR's base is sgl-project#44945's branch, so the diff shows only the 8 commits on top of it.

Motivation

sgl-project#44945 adds --enable-nvidia-megakernel, the framework for NVIDIA megakernel plugins, and the binding through which the DeepSeek-V4.1 megakernel reads SGLang's runtime state.

This PR adds what a Qwen3.5 megakernel needs on the SGLang side. It uses the same shape:

  • One binding for Qwen3.5 (megakernels/eagle_binding.py). It hands the plugin data only:
    • the running forward's full-attention and GDN metadata;
    • the KV pool and the mamba pool;
    • the buffers of the EAGLE entries.
  • EAGLE entries in the style of [Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins sgl-project/sglang#44945's DSpark draft sampler. An SGLang component calls the plugin's factory once, with its own buffers. It then runs the returned step at the batch sizes the plugin serves; every other size runs SGLang's code.
  • SGLang-owned CUDA graphs around the entries:
    • the verify's post-section: sampling, acceptance and the commit;
    • the GDN extend, inside breakable prefill graphs.

SGLang keeps every runner, worker, schedule batch and CUDA graph; a plugin never gets one. With the flag off, nothing changes.

One framework change is needed:

Modifications

1. Framework (megakernels/runtime.py, megakernels/spec.py)

  • runtime._BINDINGS maps an HF architecture to the binding its megakernel gets. Every other architecture gets MegakernelBinding.
  • spec.ENTRIES names the optional entries a built object may offer:
    • draft_sampler;
    • the EAGLE entries draft_loop, draft_extend, begin_draft_round, verify_sample and gdn_extend.
  • megakernel_entry(model, name) returns an offered entry, or None for SGLang's own path.
  • Every rank must offer the same entries; megakernel_agree checks this at install. That also covers a draft sampler that only some ranks would offer.
  • MEGAKERNEL_API_VERSION is unchanged. DeepSeek-V4.1's binding, its spec path and its draft sampler are untouched.

2. The Qwen3.5 binding (megakernels/eagle_binding.py, new)

It covers the Qwen3.5 target, whose GatedDeltaNet and full-attention layers run under HybridLinearAttnBackend, and its MTP draft.

  • EagleBinding is handed to the build as BuildContext.binding. Its forward_view() returns the EagleForwardView that a covered step gets as Step.view:

    • attn: the page table, KV lengths and cumulative lengths;
    • gdn: mamba slots, query offsets, intermediate rows and track slots. It is None for the draft.
    • kv: each full-attention layer's K/V buffers and FP8 scales;
    • mamba: the conv and SSM states, the intermediate buffers, and the ReplaySSM ring with its cursors. It is None for the draft.
  • Each pool is sealed at its first read. assert_live runs after the decode graphs are captured.

  • The entries' structs carry tensors and numbers only: EagleDraftLoopIO, EagleDraftExtendIO, EagleDraftRoundIO, EagleVerifyIO / EagleVerifyOutputs, and EagleGdnExtendIO / GdnExtendTables. The module docstring states what each entry may write.

  • What crosses besides data is listed as literal tuples, so a plugin's contract test can pin them:

    TOLERATED_CALLABLES = (
        # the TP group's process groups: a megakernel builds its exchange buffers over
        # them once, never per step
        "EagleBinding.tp_device_group",
        "EagleBinding.tp_cpu_group",
        "sglang.kernels.ops",  # SGLang's kernel ops
        "the replaced model's modules, weights and quant methods",
    )
    TOLERATED_WRITES = ()
    TOLERATED_READS = (
        "runtime_context getters (get_parallel, get_exec, get_spec, get_platform)",
        "capture-mode queries (get_is_capture_mode, is_in_breakable_cuda_graph)",
        "ForwardBatch.batch_size",
        "ForwardBatch.forward_mode",
        "ForwardBatch.out_cache_loc",
        "ForwardBatch.req_pool_indices",
        "ForwardBatch.seq_lens",
        "ForwardBatch.mrope_positions",
    )

3. EAGLE draft graphs (eagle_draft_cuda_graph_runner.py, eagle_draft_extend_cuda_graph_runner.py, speculative/eagle_megakernel.py)

  • When the draft model's built object offers draft_loop or draft_extend, the graph runner calls the factory once, before capture, with its own outputs.
  • For each batch size whose step the factory returns, that size's graph captures the step. draft_loop replaces SGLang's draft loop; draft_extend replaces the draft model step and its top-1.
  • begin_draft_round runs on the host before a served draft graph replays.
  • Draft entries serve graph replays only. An eager draft step runs SGLang's loop.
  • The worker refuses offered entries it cannot serve: a draft other than a topk-1 chain, rejection sampling, or attention backends other than GDN with trtllm_mha.

4. Post-verify graphs (speculative/eagle_megakernel.py, eagle_utils.py, eagle_worker_v2.py, eagle_worker_common.py, spec_utils.py)

  • When a megakernel offers EAGLE entries, SGLang captures the verify's post-section in its own graphs: sampling and acceptance, the commit outputs and, on a side stream, the GDN state commit.
  • A batch size that the target's verify_sample serves runs that step there, in place of SGLang's sampling.
  • Two slots alternate per key, so one round's outputs stay valid while the host reads them. The key is the batch size, sampling mode, forced acceptance, tracking and the logits buffer.
  • A batch a graph cannot reproduce runs SGLang's eager path: grammar, penalties, seeds, min-p, watermarks, or an eager verify.
  • verify_sample gets a Philox key and a device counter, as DSpark's draft gets DraftNoise.

5. GDN extend in breakable prefill graphs (linear/gdn_backend.py, linear/gdn_prefill_graph.py, fla/chunk.py, fla/chunk_o.py, fla/gdn_prefill_graph_track.py, hybrid_linear_attn_backend.py)

  • The FLA chunk kernels take optional, caller-built chunk tables. Without them the eager path is unchanged.
  • When a megakernel offers EAGLE entries, GDNAttnBackend implements the linear-backend prefill-graph protocol, as KDA does. Breakable prefill graphs then capture each GDN layer's extend instead of breaking at it, using static tables that are refilled before each replay.
  • When the target offers gdn_extend, the captured extend calls it.

6. One forced acceptance length stream ([Fix])

  • With --enable-nvidia-megakernel, the EAGLE worker owns one generator for the forced acceptance length (SGLANG_SIMULATE_ACC_LEN), seeded from the server seed.
  • The post-verify graphs and the eager verify both draw from it, once per round. Every rank therefore draws the same length, and a run whose rounds mix graph and eager paths draws the same lengths as an all-eager run.
  • Without the flag, the draw comes from the global CPU RNG, as before.

Known limits

Accuracy Tests

With the flag off, SGLang's behavior is unchanged: every new path runs only when a megakernel offers EAGLE entries, and the forced-length generator exists only with the flag.

Tests are behaviour-level, in sgl-project#44945's style: one CPU test per behaviour this PR guarantees, on sgl-project#44945's shared fakes, 14 CPU tests in all, plus two GPU test files. The behaviours covered:

  • the binding chosen by architecture;
  • the view holds the backends' own tensors, and every tolerated name exists;
  • sealing and assert_live;
  • entry structs carry data only;
  • offered entries, and the served batch sizes, agree across ranks;
  • unsupported setups are refused before capture;
  • without an offered entry, SGLang keeps its own path;
  • the forced-length draw is one stream with the flag, and unchanged without it.

CPU, on this head:

  • pre-commit, with the repo's pinned hooks, passes on every changed file and modifies nothing.

  • Unit suites. Each was run as PYTHONPATH=python python3 -m pytest -q -rfE -p no:cacheprovider --continue-on-collection-errors <suite>, on this PR and on its base ([Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins sgl-project/sglang#44945's head). The sorted failing test IDs are identical on both heads.

    suite this PR base
    test/registered/unit/megakernels/ 22 passed 10 passed
    test/registered/unit/spec/ 252 passed, 5 skipped 252 passed, 3 skipped
    test/registered/unit/layers/attention/ 258 passed 256 passed
    test/registered/unit/model_executor/runner/ 80 passed 80 passed
    • The two extra skips in spec/ are this PR's GPU tests.
    • The same 12 tests fail on both heads on this CPU-only machine, all for environment reasons:
      • 7 need CUDA in torch;
      • 1 needs the Triton JIT;
      • 4 files cannot be collected without tvm_ffi or flashinfer.

GPU, GB300, one GPU, all bitwise:

  • test_eagle_post_verify_graphs.py: 2 passed, 96 subtests. At batch sizes 1, 3, 16 and 64, greedy and with forced acceptance, SGLang's post-verify graphs equal SGLang's eager sampling bitwise in every round. That covers the eager warm-ups, the capture of each slot, and the replays of both slots. The test also checks two more things:
    • a round's outputs stay unchanged through the next round;
    • rounds that mix graph and eager paths, drawing from the worker's generator, match an all-eager reference.
  • test_gdn_prefill_graph.py: 1 passed, 10 subtests. A captured bucket, replayed at several shapes, equals the eager forward_extend bitwise, for FP32 and BF16 SSM states. The comparison covers the output, the conv and SSM pools, and the track slots.

With a Qwen3.5 megakernel plugin, at TP8 on GB300, on this PR's head:

  • The plugin is an in-development Qwen3.5 package for sgl-nvidia-mega. It is not published yet (commit d5bc21f). Every slot is on: SGLANG_NVIDIA_MEGAKERNEL_SLOTS=route,mkm,gdn_verify,gdn_pdl,hop. The runs "with the entry" add verify_sample, a factory that returns no step at any batch size, so SGLang's post-verify graphs sample and accept with SGLang's own code.
  • Hardware: 8 × GB300 (SM103), TP8 across two 4-GPU nodes in one NVLink domain.
  • Model: nvidia/Qwen3.5-397B-A17B-NVFP4-V2, revision 8f590eae8f10bf55d9a46f79ea0280bde435c9f8.
  • Serving setup:
    • MTP speculative decoding (NEXTN, 6 steps, top-k 1, 7 draft tokens), batch size 1;
    • FP8 KV cache, TRT-LLM MHA attention, the FlashInfer TRT-LLM MoE runner, FlashInfer's MNNVL CuTe DSL all-reduce fusion;
    • ReplaySSM state for the GatedDeltaNet layers.
  • Runs: one server boot per run; the client sends 7 greedy requests of up to 96 tokens, one at a time.
    • Three pairs, each with and without the entry: SGLANG_SIMULATE_ACC_LEN=4.8, SGLANG_SIMULATE_ACC_LEN=5, and real acceptance.
    • One eager run, with decode CUDA graphs off.
  • Install line on all 8 ranks; runs without the entry print the same line without entries:
    NVIDIA megakernel: active (Qwen3.5: slots ['gdn_pdl', 'gdn_verify', 'hop', 'mkm', 'route']; entries ['verify_sample']; widths [7]; bitwise numerics; built in 3.4 s, 636.0 MiB; wait deadline 60 s)
  • Capture check, on all 8 ranks in the graph runs:
    NVIDIA megakernel: Qwen3.5 runs the decode graphs of batch sizes [1] (1 of 1); every other step runs the standard forward
  • Post-verify graphs: in every run with the entry, all 8 ranks log NVIDIA megakernel: captured the EAGLE post-verify graphs for batch size 1. No rank logs it in a run without the entry.
  • Per-step parity: on every covered target-verify step, SGLang's own forward and the megakernel's forward run on the same inputs. The logits and the step's 7 hidden rows are compared bit for bit; for graph steps, the comparison runs inside the CUDA graph. 0 differ in every run:
    • decode CUDA graphs, six runs: 135 / 135 (4.8), 129 / 129 (5) and 142 / 142 (real acceptance) steps × 8 ranks, with / without the entry. That is 11,291,607,040 logits and 186,253,312 hidden values compared.
    • eager, one run: 156 steps × 8 ranks, 2,169,323,520 logits and 35,782,656 hidden values compared.
  • Outputs with the entry, against the same build without it:
    • SGLANG_SIMULATE_ACC_LEN=4.8, with both arms on --random-seed 20261019: 7 of 7 requests token-identical (672 tokens). Verify steps are equal (140 vs 140).
    • SGLANG_SIMULATE_ACC_LEN=5: 7 of 7 token-identical (672 tokens); verify steps equal (133 vs 133).
    • Real acceptance: 7 of 7 token-identical (672 tokens); verify steps equal (154 vs 154).
    • Neither the forced-5 pair nor the real-acceptance pair draws anything from the random seed, so those boots kept their own seeds.
  • No hang and no rank divergence: every run completed all 7 requests, and no rank logged an error or a timeout.

The plugin does not offer the draft entries or gdn_extend yet. Those paths are covered by the GPU tests above.

Speed Tests and Profiling

With the flag off nothing changes. The plugin's end-to-end speed will be reported with the plugin.

Checklist

🤖 Generated with Claude Code

YAMY1234 and others added 8 commits October 18, 2026 22:29
…ecture, and agree on the entries it offers

- install_megakernel builds the binding SGLang registers for the model's
  architecture (runtime._BINDINGS), and MegakernelBinding for every other one. It
  hands the binding over as BuildContext.binding, as before.
- spec.ENTRIES names the optional entries of a built object: draft_sampler and
  the EAGLE entries (draft_loop, draft_extend, begin_draft_round, verify_sample
  and gdn_extend). Every rank must offer the same ones, or every rank refuses.
  megakernel_entry returns an entry, or None for SGLang's own path.
- The startup line of a spec that covers no step names the entries it offers.

The API version is unchanged: a built object written without the EAGLE entries
offers none of them, and the DSpark draft sampler path is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
megakernels/eagle_binding.py is what binding.py is for DeepSeek-V4.1: the one place
the megakernel of a Qwen3.5 target, whose GatedDeltaNet and full-attention layers
run under HybridLinearAttnBackend, and of its MTP draft read SGLang's runtime state.
runtime._BINDINGS hands EagleBinding to the Qwen3.5 architectures.

- A covered step gets EagleForwardView as Step.view: the full-attention metadata
  (page table, KV lengths, cumulative lengths), the GDN metadata (mamba slots,
  query offsets, intermediate rows, track slots), the KV pool per full-attention
  layer with its FP8 scales, and the mamba pool with its ReplaySSM ring. A model
  without GDN layers, such as the draft, gets no GDN metadata and no mamba pool.
- The pools are read once and sealed; assert_live, after the decode graphs are
  captured, raises once one was reallocated.
- The module defines the structs of the EAGLE entries (draft loop, draft extend,
  draft round, verify sample, GDN extend) and documents what each entry may
  write. Its TOLERATED lists name what a megakernel may call or read beyond that
  data: the TP process groups, SGLang's kernel ops, configuration, and the covered
  forward's own batch fields.

test_eagle_binding.py checks that the view holds the backends' own tensors, the
sealing, that every struct carries data only, and that every listed name exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nel's draft_loop and draft_extend

When the draft model's megakernel offers draft_loop or draft_extend, the EAGLE
draft and draft-extend graph runners call the factory once, before capture, with
their own outputs: the draft tokens, and the extend's top-1, its probability and
its hidden state. For a batch size whose step the factory returns, the graph
captures that step in place of SGLang's draft loop, or in place of the draft
model step and its top-1. Every rank must serve the same batch sizes, and every
other size runs SGLang's code. SGLang keeps the graphs, their replays and the
outputs.

- begin_draft_round, when offered, runs on the host before a served draft graph
  replays. It says whether a served draft extend of the same requests produced
  the round's input.
- Draft entries serve graph replays only; an eager draft step runs SGLang's loop.
- The worker refuses offered entries it cannot serve: a draft other than a topk-1
  chain, rejection sampling, or backends other than GDN with trtllm_mha.

speculative/eagle_megakernel.py holds SGLang's side of the entries: the served
batch sizes, the outputs, and the provenance of each draft round's input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rget megakernel's verify_sample in them

When a megakernel offers EAGLE entries, SGLang captures the verify's
post-section in its own graphs: sampling and acceptance, the commit outputs and,
on a side stream, the GDN state commit. When the target's megakernel offers
verify_sample, the worker calls the factory once after the target graphs are
captured. A batch size whose step it returns runs that step there in place of
SGLang's sampling, with EagleVerifyIO and EagleVerifyOutputs; every rank must
serve the same sizes.

- Per key (batch size, sampling mode, forced acceptance, tracking, the verify
  logits buffer), two slots of staged inputs and outputs alternate, so one
  round's outputs stay valid while the host consumes them. Batches a graph cannot
  reproduce run SGLang's eager path: grammar, penalties, seeds, min-p,
  watermarks, and an eager verify.
- The side-stream commit is joined once the draft extend is enqueued.
- Every TP rank accepts the same tokens. The forced acceptance length
  (SGLANG_SIMULATE_ACC_LEN) comes from a generator the graphs own: it is seeded
  from the server seed and drawn once per round on every rank, whatever else
  uses the global generator. It reaches the device through a pinned ring, and
  eagle_sample(simulated_acc_len=) applies it with
  generate_simulated_accept_index_device, which writes what the host-length
  version writes.
- verify_sample gets a Philox key and a device counter it advances in place
  (noise_key, noise_offset), as DSpark's draft gets DraftNoise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bitwise up to bs 64

A GPU test runs SGLang's post-verify graphs at batch sizes 1, 3, 16 and 64,
greedy and with forced acceptance (real draft tokens). Every round must equal
SGLang's eager sampling on the same inputs bitwise: the eager warm-ups, the
capture of each slot, and the replays of both alternating slots. A round's
outputs must also stay unchanged through the next round, since the host may
still be reading them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aptured prefill buckets

The FLA chunk and chunk_o kernels take optional caller-built chunk tables,
and the Triton GDN extend passes them through; without them the eager path
is unchanged. gdn_prefill_graph.py holds a captured bucket's static tables,
refilled before each replay by one H2D copy, and gdn_prefill_graph_track.py
takes the prefix-cache snapshots of conv windows and SSM states inside the
graph, converting a BF16 chunk state before the store. Nothing calls them yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e prefill graphs, the megakernel's when offered

When a megakernel offers EAGLE entries, GDNAttnBackend implements the
linear-backend prefill-graph protocol, as KDA does. Breakable prefill CUDA graphs
then capture each GDN layer's extend instead of breaking at it: the extend
launches the eager extend's Triton kernels on the bucket's static tables, which
are refilled before every replay. Without an offer nothing changes.

- Before any capture, EAGLEWorkerV2 calls the target backend's
  enable_prefill_graph_extend. The hybrid backend then re-resolves whether
  breakable prefill graphs capture the linear extend, a decision it first makes
  at construction, before the entries are known.
- When the target's megakernel offers gdn_extend, the captured extend calls it
  with EagleGdnExtendIO: the bucket's tables, the layer's inputs, the mamba pool
  and an SGLang-owned zeroed output. The entry also updates the layer's conv and
  SSM states and track slots, as SGLang's extend does.
- A setup the graph-safe chain does not support keeps breaking at every layer and
  never calls the megakernel's extend.
- A GPU test captures one bucket and replays batches of several shapes. It
  compares the output, the conv and SSM pools and the track slots with the eager
  forward_extend, bit for bit, for FP32 and BF16 SSM states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he post-verify graphs and the eager verify

With --enable-nvidia-megakernel the EAGLE worker owns one generator for
SGLANG_SIMULATE_ACC_LEN, seeded from the server seed. The post-verify graphs
and the eager verify draw from it once per round, so the forced lengths of a
run no longer depend on which rounds a graph served. Without the flag the eager
verify draws from the global CPU RNG as before.

- EagleMegakernel takes the worker's generator instead of seeding its own.
- run_eagle_verify, eagle_sample and generate_simulated_accept_index pass it
  through; None keeps the global CPU RNG.
- Tests: rounds that mix graph and eager paths force the lengths all-graph and
  all-eager rounds force, on every rank; without the flag the eager draws equal
  the global RNG's and leave it in the same state. The GPU post-verify test
  compares against eagle_sample's own host-length draw, with eager rounds in
  between, which also covers the CPU device-length test this drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@YAMY1234
YAMY1234 force-pushed the nv-mega/qwen35-megakernel-binding branch from 576dc8a to adfc015 Compare October 19, 2026 06:03
@YAMY1234 YAMY1234 closed this Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant