Repository navigation
[Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins - #44945
Conversation
A launch flag for the NVIDIA megakernel plugin (the sgl-nvidia-mega package), in the exec argument group. SGLang only carries the flag and a startup check: the handle_nvidia_megakernel resolution step, in the argument-resolution pipeline, raises when the flag is set and the plugin's sglang.srt.plugins entry point (nvidia_megakernel) is not installed, with an install hint, or when SGLANG_PLUGINS is set and does not list it, since the plugin would then never load. Everything else, including the checks of the supported setup, stays in the plugin. Off by default; with it off nothing changes.
The sglang.srt.megakernels package for --enable-nvidia-megakernel. spec.py holds the contract a plugin implements (MegakernelSpec, Requirements, Numerics, BuildContext, Step, the BuiltMegakernel protocol with an optional draft_sampler factory) and the lazy per-architecture registry the plugin's entry point fills. guards.py and runtime.py are interface stubs, filled in by the next commits.
…unner.initialize install_megakernel runs as the last step of ModelRunner.initialize, after every post-load change to the model and before the KV cache is sized. With --enable-nvidia-megakernel it finds the spec for the model's architecture on every TP rank, gathers every rank's refusal reasons and raises on all ranks if any rank has one, builds the spec at the step widths of whole requests up to its max_rows, and replaces the forward on that model instance only: a step the built object plans runs it, every other step the model's forward. Weight updates then refuse, one line logs what is active, and a capture check refuses a runner none of whose decode graphs run the megakernel. A model without a registered spec, such as a draft, keeps its forward; a spec that builds no width only offers its draft sampler (megakernel_draft_sampler).
guards.handle_nvidia_megakernel, the resolution step in the argument-resolution pipeline, now owns every startup check of --enable-nvidia-megakernel, before any weight loads: - the sglang.srt.plugins entry point is installed (with an install hint), and SGLANG_PLUGINS, when set, lists it; - the model has a spec for this GPU, and every reason it cannot serve the resolved setup is listed together: SGLang features in use that the spec does not accept (one probe each, on the resolved settings), tensor parallel size, dtype, decode CUDA graph backend, server arguments, HF config fields, environment variables, the watchdog's wait deadline, and the spec's own configuration checks. deadline_s and device_isa serve the runtime. The plugin-loader check from the flag commit moves here, so sglang.srt.plugins is unchanged.
…t proposal maybe_build_draft_sampler asks the megakernel framework for the draft model's draft-sampler factory (megakernel_draft_sampler) and passes it to DsparkDraftSampler, which calls it once with its own buffers (draft tokens, step logits, step temperatures, the draft noise). For a batch size the factory serves, the returned step replaces compute_base_logits and draft_markov_block; every other size, and every server without the megakernel, runs today's proposal. A sampled draft's step logits are zeroed up front when the factory is set.
08c92b2 to
58fb997
Compare
…orkers; an empty TP set accepts any size - install_megakernel takes the model's role: a target model whose spec no rank found (the plugin failed to load in the workers after argument resolution found it) is refused on every rank, naming the cause, instead of running its own forward; a draft model without a spec still keeps its own forward. - The post-load requirement check treats an empty tp_sizes as any size, as argument resolution and the Requirements contract do.
One test per behaviour the framework guarantees: the binding's view and tolerated names, a replaced source caught after capture; every refusal reason at resolution and the resolution step; install routing covered and uncovered steps; a target without a spec refused and the draft fallback; the post-load checks; ranks that disagree refused; the capture check; the draft sampler only on the batch sizes every rank serves.
Oasis-Git
left a comment
There was a problem hiding this comment.
[Codex]: Please also add the image-installation integration for sgl-nvidia-mega, following the AMD monokernel setup. In docker/rocm.Dockerfile, AMD pins SGL_AMD_MONO_COMMIT; the installation block checks out that revision, installs the plugin for the supported image, and verifies its entry point.
This PR adds the NVIDIA runtime integration but no equivalent plugin installation or commit pin in the CUDA image recipes. An image built from this change therefore still requires a separate manual plugin installation before --enable-nvidia-megakernel can work. Please install a compatible, pinned plugin revision in the applicable CUDA image(s), with an explicit opt-out/platform scope and an entry-point smoke check. This would make the SGLang/plugin pairing reproducible and provide the same ready-to-use experience as AMD. The gap is in image provisioning; building and installing the plugin wheel manually already passed a local smoke check.
…the CUDA images Install sgl-nvidia-mega, the plugin behind --enable-nvidia-megakernel, in the framework_final stage of docker/Dockerfile and docker/Dockerfile.cu134, as docker/rocm.Dockerfile does for sgl-amd-mono: after SGLang, clone it, check out the pinned SGL_NVIDIA_MEGA_COMMIT (sgl-nvidia-mega main, d47deb3), install it editable with --no-deps, and assert that its nvidia_megakernel entry point is registered. The runtime stage copies site-packages and /sgl-workspace from framework_final, so it ships the plugin too. Scope: the plugin's SM103 kernels compile at runtime, so the step skips with a message when the image's CUDA_VERSION is below 12.9 (both recipes build CUDA 13.x today). It is not limited by architecture: GB300 is arm64, B300 is x86_64. The plugin does nothing unless a server passes the flag. Opt out with --build-arg ENABLE_SGL_NVIDIA_MEGA=0.
|
[Codex]: Use /tag-and-rerun-ci for UT-only CI, as requested; monitor the actual Stage A CPU and single-GPU small partitions and cancel this base workflow after all required Stage A jobs pass. This does not constitute a full CI pass. |
|
/tag-and-rerun-ci |
|
[Codex]: UT CI passed for |
|
Could you simplify the tests in this PR, especially the “1 + 1 = 2” style of AI-generated tests that mostly restate the implementation or check names we just defined? Each test should protect a concrete behavior that could regress, with as little test scaffolding as practical. In particular:
The goal is a small, readable suite that catches real regressions, rather than more assertions or elaborate mocks for their own sake. |
Drop the AST name scan of the SGLang members the framework reads and the binding's tolerated-name enumeration; neither shows the runtime object provides the interface. Keep one test each for the disabled flag, covered-step vs fallback routing, a missing or unusable target spec vs an optional draft spec (including an empty tp_sizes at resolution and after load), rank disagreement, and no captured graph using the megakernel; the binding keeps the view-is-the-running-backend and sealed-source checks.
|
@BBuf Simplified in db39573 (test-only): removed @Oasis-Git All three points are addressed: |
…ble-nvidia-megakernel and a layer for NVIDIA megakernel plugins
Motivation
The NVIDIA megakernel for DeepSeek-V4.1-Flash decode on GB300 ships as an out-of-tree package, sgl-project/sgl-nvidia-mega. It runs the decode steps it covers with its own step code over the weights SGLang loaded.
This PR adds the SGLang side such a plugin needs, and nothing else:
Everything model-specific stays in the plugin. SGLang's own decode flow (the DSpark worker included) is unchanged.
This PR previously also carried DSpark decode hooks and runtime changes; they are no longer part of it.
Modifications
Flag.
--enable-nvidia-megakernel(bool, default off), in the exec argument group. With it off, nothing in the megakernel layer runs.python/sglang/srt/megakernels/spec.py: the contract a plugin implements, plus a lazy per-architecture registry that the plugin'ssglang.srt.pluginsentry point fills. The contract is:MegakernelSpec;RequirementsandNumerics, given as data;BuildContext(now withbinding) andStep(now withview);BuiltMegakernelprotocol, with an optional draft-sampler factory.guards.py: the argument-resolution stephandle_nvidia_megakernel. It runs before any weight loads. With the flag set, it refuses the launch and lists every reason together:SGLANG_PLUGINSis set and does not list it;runtime.py:install_megakernel, the last step ofModelRunner.initialize. It runs after every post-load change to the model and before the KV cache is sized. With the flag set:megakernel_agree(inruntime.py): the one cross-rank check every decision above uses. It gathers a value over the TP CPU group and refuses on every rank, listing each rank's value, when they differ.binding.py(new): the binding, the one place a megakernel reads SGLang's runtime state (next section).The binding (
megakernels/binding.py).MegakernelBinding, one per installed model, handed to the spec's build asBuildContext.binding. The target model and the DSpark draft model each get their own (each is installed on its own runner).forward_view(): theForwardViewof the forward running now, read from SGLang's forward context. A draft entry calls it inside the draft step and gets the draft runner's view.engram_history(): the model's Engram hash history tensor, or None.engram_host_table(engram): the host table of one of the model's Engram layers, or None.assert_live(): raises, naming the source, once a sealed source was reallocated.ForwardView, the running forward's attention data. A covered step gets it asStep.view; SGLang builds it only when the plan returned a step, so at capture for a graph step, and its tensors are the ones the graph replays.metadata: the attention backend's forward metadata;kv_pool: aKvPoolView;trtllm_attn;attention,store_cache,swa_out_cache_loc,low_ratio_sourcesandcore_compressorcalls (tolerated, below).KvPoolView, the running forward's KV pool under the pool's own names:uniform_fp8,request_window,get_swa_key_buffer_radix(layer),get_swa_key_page_size(),get_swa_key_bytes_per_token(),get_extra_key_buffer(layer),get_extra_key_page_size(layer),get_extra_key_bytes_per_token(layer),set_swa_key_buffer_radix_fused_norm_rope(tolerated), andpool(tolerated: only passed back into SGLang's own calls). A pool offers the members of its kind; each is read on access.What crosses besides data, listed as literal tuples in the file so a plugin's contract test can pin them:
Lifetime. The binding is built at install, before the KV pools exist, so it reads lazily. The KV pool and the Engram history are each sealed (data pointer, shape and dtype, or the object) at the first read that finds them.
assert_liveruns in the capture check, once every decode graph of the runner is captured, and raises naming a source that was reallocated since.Call sites:
arg_groups/pipeline.pyandresolution_hooks.py;ModelRunner.initialize;maybe_build_draft_sampler. It passes the draft model's factory, when one is registered, toDsparkDraftSampler, which then runs the plugin's step for the batch sizes it serves. Every rank must serve the same batch sizes, checked withmegakernel_agreebefore graph capture; theBuiltMegakerneldocstring states what a step must do (identical tokens on every rank, the buffers it writes, the noise counter).Docs: a row in the server-arguments documentation.
Images:
docker/Dockerfileanddocker/Dockerfile.cu134install the plugin at a pinned commit (SGL_NVIDIA_MEGA_COMMIT, sgl-nvidia-mega maind47deb3) and check that its entry point is registered, asdocker/rocm.Dockerfiledoes for the AMD plugin. Images with CUDA below 12.9 skip it; opt out with--build-arg ENABLE_SGL_NVIDIA_MEGA=0. The plugin does nothing unless a server passes--enable-nvidia-megakernel.Known limits
assert_liveruns at the capture check only. A later SGLang re-capture would have to check again; the binding does not support re-capture yet. With the flag set, weight updates already refuse.Accuracy Tests
With the flag off, SGLang's behavior is unchanged: nothing in the megakernel layer runs, and no binding is built.
CPU, on this head:
pre-commitpasses on every changed file, on a first run.test/registered/unit/megakernels/,model_executor/runner/,model_executor/test_cuda_graph_buffer_registry.py,spec/test_dspark*.py,layers/test_dsv41*.py): 154 passed, 1 skipped, 0 failed.test/registered/unit/model_executor/runner/test_flashinfer_autotune_sync.py:365(skipUnless(torch.cuda.is_available(), "FlashInfer requires CUDA")), so it skips on a CPU run; this PR does not touch that file.assert_liveby name;tp_size, and an emptytp_sizesaccepts any size;tp_sizeafter load) is refused, an emptytp_sizesis accepted, and a draft without a spec keeps its forward;GB300, TP4, DeepSeek-V4.1-Flash, DSpark, with the plugin (sgl-project/sgl-nvidia-mega#1) at its default slot set:
DeepSeek-V4.1and the draftdeepseek_v41_dspark_draft_nvidia) on all 4 ranks; the capture check passes; 0 server errors over the run;Speed Tests and Profiling
With the flag off nothing changes. The plugin's end-to-end speed is in sgl-project/sgl-nvidia-mega#1.
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ⏳ Run #39302053082
Latest PR Test (Extra): ⏳ Run #39302053139
Latest PR Test (AMD ROCm 10): ⏳ Run #39302053159