Skip to content

[Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins - #44945

Merged
Oasis-Git merged 10 commits into
sgl-project:mainfrom
YAMY1234:nv-mega/enable-nvidia-megakernel-pr
Oct 19, 2026
Merged

Oasis-Git merged 10 commits into
sgl-project:mainfrom
YAMY1234:nv-mega/enable-nvidia-megakernel-pr

Conversation

@YAMY1234

@YAMY1234 YAMY1234 commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The NVIDIA megakernel for DeepSeek-V4.1-Flash decode on GB300 ships as an out-of-tree package, sgl-project/sgl-nvidia-mega. It runs the decode steps it covers with its own step code over the weights SGLang loaded.

This PR adds the SGLang side such a plugin needs, and nothing else:

  • one launch flag and a small framework that checks the setup, replaces the model's forward for the steps the plugin covers, and offers a DSpark draft-proposal entry;
  • one binding file through which SGLang hands the plugin's build, forward and draft entries the runtime state they need (the KV pool, the running forward's attention metadata, the Engram tables), as data.

Everything model-specific stays in the plugin. SGLang's own decode flow (the DSpark worker included) is unchanged.

This PR previously also carried DSpark decode hooks and runtime changes; they are no longer part of it.

# with the flag and the NVIDIA megakernel plugin, every rank logs one line per installed model:
NVIDIA megakernel: active (<spec>: <summary>; widths [...]; <numerics> numerics; built in N s, M MiB; wait deadline D s)
# a DSpark draft model with its own spec logs its own line (an "active" line when it covers draft steps)

Modifications

Flag. --enable-nvidia-megakernel (bool, default off), in the exec argument group. With it off, nothing in the megakernel layer runs.

python/sglang/srt/megakernels/

  • spec.py: the contract a plugin implements, plus a lazy per-architecture registry that the plugin's sglang.srt.plugins entry point fills. The contract is:
    • MegakernelSpec;
    • Requirements and Numerics, given as data;
    • BuildContext (now with binding) and Step (now with view);
    • the BuiltMegakernel protocol, with an optional draft-sampler factory.
  • guards.py: the argument-resolution step handle_nvidia_megakernel. It runs before any weight loads. With the flag set, it refuses the launch and lists every reason together:
    • the plugin's entry point is not installed (with an install hint), or SGLANG_PLUGINS is set and does not list it;
    • the model has no spec for this GPU;
    • an SGLang feature in use that the spec does not accept (one probe each, on the resolved settings);
    • tensor parallel size, dtype, decode CUDA graph backend, server arguments, HF config fields or environment variables that the spec does not accept;
    • the watchdog's wait deadline;
    • the spec's own configuration checks;
    • another whole-step kernel backend selected at the same time.
  • runtime.py: install_megakernel, the last step of ModelRunner.initialize. It runs after every post-load change to the model and before the KV cache is sized. With the flag set:
    1. On every TP rank, it finds the spec and the reasons it cannot serve this model, gathers them, and raises on every rank if any rank has one.
    2. It builds the spec at the covered step widths, handing it this model's binding (below).
    3. It replaces the forward on that model instance only. A step the built object plans runs on the plugin, with the running forward's view; every other step runs the model's own forward. The plan reads host state only, so all ranks agree and a captured graph replays the decision made at capture.
    4. It makes weight updates refuse, and logs one line per rank saying what is active.
    5. It adds a capture check that refuses a runner none of whose decode graphs run the plugin, and checks the binding's sealed sources there.
    6. A target spec that builds no width is refused on every rank. A draft model's spec is checked the same way; it may cover draft steps (its own forward and binding) and/or offer a draft sampler.
  • megakernel_agree (in runtime.py): the one cross-rank check every decision above uses. It gathers a value over the TP CPU group and refuses on every rank, listing each rank's value, when they differ.
  • binding.py (new): the binding, the one place a megakernel reads SGLang's runtime state (next section).

The binding (megakernels/binding.py).

  • MegakernelBinding, one per installed model, handed to the spec's build as BuildContext.binding. The target model and the DSpark draft model each get their own (each is installed on its own runner).

    • forward_view(): the ForwardView of the forward running now, read from SGLang's forward context. A draft entry calls it inside the draft step and gets the draft runner's view.
    • engram_history(): the model's Engram hash history tensor, or None.
    • engram_host_table(engram): the host table of one of the model's Engram layers, or None.
    • assert_live(): raises, naming the source, once a sealed source was reallocated.
  • ForwardView, the running forward's attention data. A covered step gets it as Step.view; SGLang builds it only when the plan returned a step, so at capture for a graph step, and its tensors are the ones the graph replays.

    • metadata: the attention backend's forward metadata;
    • kv_pool: a KvPoolView;
    • trtllm_attn;
    • the backend's own attention, store_cache, swa_out_cache_loc, low_ratio_sources and core_compressor calls (tolerated, below).
  • KvPoolView, the running forward's KV pool under the pool's own names: uniform_fp8, request_window, get_swa_key_buffer_radix(layer), get_swa_key_page_size(), get_swa_key_bytes_per_token(), get_extra_key_buffer(layer), get_extra_key_page_size(layer), get_extra_key_bytes_per_token(layer), set_swa_key_buffer_radix_fused_norm_rope (tolerated), and pool (tolerated: only passed back into SGLang's own calls). A pool offers the members of its kind; each is read on access.

  • What crosses besides data, listed as literal tuples in the file so a plugin's contract test can pin them:

    TOLERATED_CALLABLES = (
        "ForwardView.attention",  # the attention backend's own forward
        "ForwardView.store_cache",  # the KV write
        "ForwardView.swa_out_cache_loc",
        "ForwardView.low_ratio_sources",  # compressor / indexer inputs
        "ForwardView.core_compressor",
        "KvPoolView.set_swa_key_buffer_radix_fused_norm_rope",  # the fused SWA key write
        "KvPoolView.pool",  # passed back to SGLang's own draft KV store only
        "sglang.kernels.ops",  # SGLang's kernel ops
        "the replaced model's modules, weights and quant methods",
    )
    TOLERATED_WRITES = (
        "ForwardView.metadata.q_pad_buffer",
        "ForwardView.metadata.low_ratio_req_indices",
        "ForwardView.metadata.low_ratio_pos_i64",
    )
    TOLERATED_READS = (
        "runtime_context getters (get_parallel, get_exec, get_spec, get_platform)",
        "capture-mode queries (get_is_capture_mode, is_in_breakable_cuda_graph)",
        "MegakernelBinding.engram_host_table",  # the per-layer host table object
    )
  • Lifetime. The binding is built at install, before the KV pools exist, so it reads lazily. The KV pool and the Engram history are each sealed (data pointer, shape and dtype, or the object) at the first read that finds them. assert_live runs in the capture check, once every decode graph of the runner is captured, and raises naming a source that was reallocated since.

Call sites:

  • the resolution step in arg_groups/pipeline.py and resolution_hooks.py;
  • one call at the end of ModelRunner.initialize;
  • maybe_build_draft_sampler. It passes the draft model's factory, when one is registered, to DsparkDraftSampler, which then runs the plugin's step for the batch sizes it serves. Every rank must serve the same batch sizes, checked with megakernel_agree before graph capture; the BuiltMegakernel docstring states what a step must do (identical tokens on every rank, the buffers it writes, the noise counter).

Docs: a row in the server-arguments documentation.

Images: docker/Dockerfile and docker/Dockerfile.cu134 install the plugin at a pinned commit (SGL_NVIDIA_MEGA_COMMIT, sgl-nvidia-mega main d47deb3) and check that its entry point is registered, as docker/rocm.Dockerfile does for the AMD plugin. Images with CUDA below 12.9 skip it; opt out with --build-arg ENABLE_SGL_NVIDIA_MEGA=0. The plugin does nothing unless a server passes --enable-nvidia-megakernel.

Known limits

  • Re-capture. assert_live runs at the capture check only. A later SGLang re-capture would have to check again; the binding does not support re-capture yet. With the flag set, weight updates already refuse.
  • Tolerated items. The three lists above are what is still handed over beyond data: the backend's own attention / KV-write / compressor calls, three writes into the forward metadata, and configuration reads. Each can move to data later without changing the plugin's other code.

Accuracy Tests

With the flag off, SGLang's behavior is unchanged: nothing in the megakernel layer runs, and no binding is built.

CPU, on this head:

  • pre-commit passes on every changed file, on a first run.
  • SGLang unit tests (test/registered/unit/megakernels/, model_executor/runner/, model_executor/test_cuda_graph_buffer_registry.py, spec/test_dspark*.py, layers/test_dsv41*.py): 154 passed, 1 skipped, 0 failed.
    • The one skip is test/registered/unit/model_executor/runner/test_flashinfer_autotune_sync.py:365 (skipUnless(torch.cuda.is_available(), "FlashInfer requires CUDA")), so it skips on a CPU run; this PR does not touch that file.
  • The PR's 9 tests, one per failure mode:
    • the binding's view is the running forward's attention backend and KV pool;
    • a source replaced after capture fails assert_live by name;
    • a disabled flag looks nothing up and leaves the model's forward untouched;
    • argument resolution refuses an unmet tp_size, and an empty tp_sizes accepts any size;
    • a covered step runs the megakernel with the running forward's view, and an uncovered step runs the model's own forward;
    • a target without a usable spec (none in the workers, or an unmet tp_size after load) is refused, an empty tp_sizes is accepted, and a draft without a spec keeps its forward;
    • ranks that disagree on the spec or the built widths are refused on every rank;
    • the capture check refuses when no captured decode graph runs the megakernel;
    • the draft sampler hands the megakernel step SGLang's buffers only on batch sizes every rank serves, and falls back to SGLang's block on the others.
  • The NVIDIA megakernel plugin's CPU suite passes against this head (0 skipped); the plugin constructs and imports on it (51 modules), its contract test pins the three lists, and a test drives its real build path to check every package receives the binding.

GB300, TP4, DeepSeek-V4.1-Flash, DSpark, with the plugin (sgl-project/sgl-nvidia-mega#1) at its default slot set:

  • boot: passes. Both models install, with two "active" lines per rank (the target DeepSeek-V4.1 and the draft deepseek_v41_dspark_draft_nvidia) on all 4 ranks; the capture check passes; 0 server errors over the run;
  • GSM8K-200 with real DSpark acceptance: 197 / 200 at 8 and 197 / 200 at 64 concurrent requests;
  • needle and the long-context sets at 1 request: needle 128 / 135 over 5 passes; the 16K set 96 / 96 (accept length 4.22) and the 24-32K set 32 / 32 (accept length 4.27);
  • the per-rank Engram huge-page coverage of the run: 100 % on every rank and Engram layer (4 ranks x layers 1 and 14), on a node gated for free huge pages with the page cache evicted and the huge pages warmed before launch.

Speed Tests and Profiling

With the flag off nothing changes. The plugin's end-to-end speed is in sgl-project/sgl-nvidia-mega#1.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ⏳ Run #39302053082
Latest PR Test (Extra): ⏳ Run #39302053139
Latest PR Test (AMD ROCm 10): ⏳ Run #39302053159

A launch flag for the NVIDIA megakernel plugin (the sgl-nvidia-mega package),
in the exec argument group. SGLang only carries the flag and a startup check:
the handle_nvidia_megakernel resolution step, in the argument-resolution
pipeline, raises when the flag is set and the plugin's
sglang.srt.plugins entry point (nvidia_megakernel) is not installed, with an
install hint, or when SGLANG_PLUGINS is set and does not list it, since the
plugin would then never load. Everything else, including the checks of the
supported setup, stays in the plugin. Off by default; with it off nothing
changes.
The sglang.srt.megakernels package for --enable-nvidia-megakernel. spec.py holds
the contract a plugin implements (MegakernelSpec, Requirements, Numerics,
BuildContext, Step, the BuiltMegakernel protocol with an optional draft_sampler
factory) and the lazy per-architecture registry the plugin's entry point fills.
guards.py and runtime.py are interface stubs, filled in by the next commits.
…unner.initialize

install_megakernel runs as the last step of ModelRunner.initialize, after every
post-load change to the model and before the KV cache is sized. With
--enable-nvidia-megakernel it finds the spec for the model's architecture on
every TP rank, gathers every rank's refusal reasons and raises on all ranks if
any rank has one, builds the spec at the step widths of whole requests up to its
max_rows, and replaces the forward on that model instance only: a step the built
object plans runs it, every other step the model's forward. Weight updates then
refuse, one line logs what is active, and a capture check refuses a runner none
of whose decode graphs run the megakernel. A model without a registered spec,
such as a draft, keeps its forward; a spec that builds no width only offers its
draft sampler (megakernel_draft_sampler).
guards.handle_nvidia_megakernel, the resolution step in the argument-resolution
pipeline, now owns every startup check of --enable-nvidia-megakernel,
before any weight loads:
- the sglang.srt.plugins entry point is installed (with an install hint), and
  SGLANG_PLUGINS, when set, lists it;
- the model has a spec for this GPU, and every reason it cannot serve the
  resolved setup is listed together: SGLang features in use that the spec does
  not accept (one probe each, on the resolved settings), tensor parallel size,
  dtype, decode CUDA graph backend, server arguments, HF config fields,
  environment variables, the watchdog's wait deadline, and the spec's own
  configuration checks.
deadline_s and device_isa serve the runtime. The plugin-loader check from the
flag commit moves here, so sglang.srt.plugins is unchanged.
…t proposal

maybe_build_draft_sampler asks the megakernel framework for the draft
model's draft-sampler factory (megakernel_draft_sampler) and passes it to
DsparkDraftSampler, which calls it once with its own buffers (draft
tokens, step logits, step temperatures, the draft noise). For a batch size
the factory serves, the returned step replaces compute_base_logits and
draft_markov_block; every other size, and every server without the
megakernel, runs today's proposal. A sampled draft's step logits are
zeroed up front when the factory is set.
…orkers; an empty TP set accepts any size

- install_megakernel takes the model's role: a target model whose spec no rank
  found (the plugin failed to load in the workers after argument resolution
  found it) is refused on every rank, naming the cause, instead of running its
  own forward; a draft model without a spec still keeps its own forward.
- The post-load requirement check treats an empty tp_sizes as any size, as
  argument resolution and the Requirements contract do.
One test per behaviour the framework guarantees: the binding's view and
tolerated names, a replaced source caught after capture; every refusal reason at
resolution and the resolution step; install routing covered and uncovered steps;
a target without a spec refused and the draft fallback; the post-load checks;
ranks that disagree refused; the capture check; the draft sampler only on the
batch sizes every rank serves.

@Oasis-Git Oasis-Git left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Codex]: Please also add the image-installation integration for sgl-nvidia-mega, following the AMD monokernel setup. In docker/rocm.Dockerfile, AMD pins SGL_AMD_MONO_COMMIT; the installation block checks out that revision, installs the plugin for the supported image, and verifies its entry point.

This PR adds the NVIDIA runtime integration but no equivalent plugin installation or commit pin in the CUDA image recipes. An image built from this change therefore still requires a separate manual plugin installation before --enable-nvidia-megakernel can work. Please install a compatible, pinned plugin revision in the applicable CUDA image(s), with an explicit opt-out/platform scope and an entry-point smoke check. This would make the SGLang/plugin pairing reproducible and provide the same ready-to-use experience as AMD. The gap is in image provisioning; building and installing the plugin wheel manually already passed a local smoke check.

…the CUDA images

Install sgl-nvidia-mega, the plugin behind --enable-nvidia-megakernel, in the
framework_final stage of docker/Dockerfile and docker/Dockerfile.cu134, as
docker/rocm.Dockerfile does for sgl-amd-mono: after SGLang, clone it, check out
the pinned SGL_NVIDIA_MEGA_COMMIT (sgl-nvidia-mega main, d47deb3), install it
editable with --no-deps, and assert that its nvidia_megakernel entry point is
registered. The runtime stage copies site-packages and /sgl-workspace from
framework_final, so it ships the plugin too.

Scope: the plugin's SM103 kernels compile at runtime, so the step skips with a
message when the image's CUDA_VERSION is below 12.9 (both recipes build CUDA
13.x today). It is not limited by architecture: GB300 is arm64, B300 is x86_64.
The plugin does nothing unless a server passes the flag.

Opt out with --build-arg ENABLE_SGL_NVIDIA_MEGA=0.
@Oasis-Git

Copy link
Copy Markdown
Collaborator

[Codex]: Use /tag-and-rerun-ci for UT-only CI, as requested; monitor the actual Stage A CPU and single-GPU small partitions and cancel this base workflow after all required Stage A jobs pass. This does not constitute a full CI pass.

@Oasis-Git

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Oct 19, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator

[Codex]: UT CI passed for 5ba9bb2dc23ff27696aa209e932f53f40ba76c4a: base run, attempt 2. All 16 required Stage A partitions passed: 15 CPU partitions (0–14) and 1 single-GPU-small partition, with no Stage A skips. Stage A passed; the workflow was intentionally cancelled, and GitHub confirms completed / cancelled. No Stage B/C test jobs were observed before cancellation. This is Stage A coverage only, not a full CI pass; sibling workflows were not cancelled.

@BBuf

BBuf commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Could you simplify the tests in this PR, especially the “1 + 1 = 2” style of AI-generated tests that mostly restate the implementation or check names we just defined? Each test should protect a concrete behavior that could regress, with as little test scaffolding as practical.

In particular:

  • Please consider removing test_megakernel_contract.py. Its AST scan checks whether a name or string appears somewhere in a source file, which does not establish that the runtime object actually provides the required interface. An unrelated occurrence can satisfy it, while moving an implementation between files can break it without changing behavior.
  • In the binding tests, the TOLERATED_CALLABLES / hasattr enumeration seems less useful than the existing checks that the view exposes the actual backend/pool and that replacing a sealed source is detected. Keep those behavioral checks and trim the interface-name bookkeeping.
  • Keep the meaningful guard/dispatch coverage: disabled flag, covered-step vs. fallback routing, missing target vs. optional draft spec, rank disagreement, no captured graph using the plugin, and draft-sampler buffer handoff/fallback. These cover distinct failure modes; they do not need a larger matrix of trivial assertions.

The goal is a small, readable suite that catches real regressions, rather than more assertions or elaborate mocks for their own sake.

Drop the AST name scan of the SGLang members the framework reads and the binding's
tolerated-name enumeration; neither shows the runtime object provides the interface.
Keep one test each for the disabled flag, covered-step vs fallback routing, a missing
or unusable target spec vs an optional draft spec (including an empty tp_sizes at
resolution and after load), rank disagreement, and no captured graph using the
megakernel; the binding keeps the view-is-the-running-backend and sealed-source checks.
@YAMY1234

Copy link
Copy Markdown
Collaborator Author

@BBuf Simplified in db39573 (test-only): removed test_megakernel_contract.py and the binding name checks. One test per failure mode remains, 9 in total.

@Oasis-Git All three points are addressed:

  • 06f3ea7: a target without a spec is refused on every rank, and an empty tp_sizes accepts any size at both checks.
  • 5ba9bb2: the CUDA images install sgl-nvidia-mega at a pinned commit for CUDA >= 12.9, with an opt-out build arg and an entry-point check.

@Oasis-Git
Oasis-Git merged commit 722669e into sgl-project:main Oct 19, 2026
117 of 169 checks passed
YAMY1234 added a commit to YAMY1234/sglang that referenced this pull request Oct 20, 2026
…ble-nvidia-megakernel and a layer for NVIDIA megakernel plugins
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deepseek documentation Improvements or additions to documentation run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants