Repository navigation
Conversation
…ure checks, early draft embedding sharing A megakernel that replaces the decode or verify step of a DSA (sparse MLA) model reads the DSA backend's layout and decode buffers and writes the MLA and index-K rows of its KV pool. Offer them through the binding: KvPoolView gains the MLA key and index-K buffers, the index page size and the MLA key write; ForwardView.dsa is a DsaView of the running DSA backend (MLA dimensions, decode implementation, top-k and DCP settings, the decode workspace and the multi-CTA KV counter as sealed data, and the indexer metadata). TOLERATED_DSA_CALLABLES / TOLERATED_DSA_WRITES list what a megakernel still calls or writes; the existing TOLERATED_* lists are unchanged. The EAGLE draft-decode and draft-extend graph runners now call the capture tail hooks at the end of their captured function, as the decode graph runner does, so a megakernel spec installed on a draft model is checked after capture. The draft-decode runner runs them under its first per-step attention backend and checks that the per-step backends share one KV pool. spec.EAGLE_DRAFT_CAPTURE_CHECK marks an SGLang that does this; Step documents the draft modes. Under SGLANG_ENABLE_EMBED_REPLICATION the NextN draft loads its own full embedding table on every rank. EagleDraftWorker replaced it with the target's in alloc_memory_pool, which the scheduler calls after it has sized the target KV pool, so the draft's copy (and its lm_head shard) was still allocated when the pool was sized. Under replication the draft worker now binds the target's embedding and lm_head at the end of its construction, so the draft's own copies are freed before the pool is sized and the pool can use that memory. Without replication the calls stay where they were. With --enable-nvidia-megakernel, argument resolution now sets a variable of spec.FLAG_SETS_ENV (SGLANG_ENABLE_EMBED_REPLICATION) that the served spec requires and the user left unset, to the spec's value, and logs it. SGLang reads that variable only when a model is built, in the model processes started after resolution, so every rank sees the value. A value the user set is kept and checked as before, and a refused setup leaves the environment as it was.
c39924b to
465a9dd
Compare
|
Validated the combined GLM-5.3 NVFP4 static-table stack on 8×B300 TP8 with hooks Full table and protocol: sgl-project/sgl-nvidia-mega#24 (comment). This measures the combined stack, not the hooks' isolated contribution. The DSA test consolidation is now in |
One conflict, in the spec import block of python/sglang/srt/kernel_plugins/nvidia/binding.py: this branch imports FLAG_SETS_ENV and main (sgl-project#45266) imports GRAPH_VARIANT_RUNNERS; both are kept.
sgl-project#45266 made the megakernel contract version 2: the binding reads draft_loop, draft_extend, begin_draft_round and graph_variants from every built object. The draft capture test's built object offers none of them, as main's FakeBuilt does.
One conflict, in handle_nvidia_megakernel of python/sglang/srt/kernel_plugins/nvidia/binding.py: main (sgl-project#46625) returns early when the MSA plugin serves the architectures, and this branch starts its FLAG_SETS_ENV bookkeeping (env_set = []) at the same place. Both are kept: main's return first, then env_set, so the flag sets variables only for an sgl-nvidia-mega spec.
- spec, draft_spec: API_VERSION 2; the built objects offer no EAGLE entries or graph variants (draft_loop, draft_extend, begin_draft_round, graph_variants are None). Restore the target's _Built (widths, summary, plan, forward), which the package split had dropped, so build returns a BuiltMegakernel again. - spec: accept Feature.MULTI_NODE_TP, so TP8 can span the two 4-GPU trays of an NVL72 rack, as the draft spec already does. - spec: drop the DsaView binding-member and EAGLE_DRAFT_CAPTURE_CHECK gates, which only an unmerged SGLang (sgl-project/sglang#46210) passes; the covered step reads the DSA backend through get_attn_backend() and never through those views. - attention: build the AttentionGlue directly (it has no build classmethod). - draft_spec: import the megakernel interface from its new location first. - glm_dsa, spec_step: declare every SGLang import in SGLANG_READS (the stricter contract test); import modules as modules (import a.b.c as c); name reads by the file that defines them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with main's draft runtime and binding seal refactor: - binding.py: seal the DSA buffers through main's _seal(name, value, current) in forward_view (non-hybrid only); take main's assert_live; drop the PR's duplicate KvPoolView.get_key_buffer that shadowed main's sealing one. - eagle_draft_cuda_graph_runner.py: keep main's draft_runtime.run_draft_loop; call the capture tail hooks at the end of run_stock, as the extend runner does. - spec.py: keep main's Step mode list and add the draft modes and notes. - test_megakernel_draft_capture.py: follow the hook call into run_stock and give the patched forward_view a view main's seal_capture_view accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YAMY1234
left a comment
There was a problem hiding this comment.
Merged main and resolved the conflicts; LGTM.
|
/tag-and-rerun-ci |
|
/tag-and-rerun-ci |
# Conflicts: # python/sglang/kernels/plugin_kernels/nvidia/spec.py # python/sglang/srt/speculative/eagle_worker_v2.py
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
…mbed_share The merge of main added `_init_draft_probs_vocab_scatter()` to `EagleDraftWorker.alloc_memory_pool`. That method reads `target_worker`, which the test's bare worker does not have. Stub it out like the other init calls, since the test only checks the order of the sharing calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] @nvrohanv, the relevant CI failures appear to be caused by this PR. Both also failed the same way on the previous head
Could you please fix them? Thanks! (A third failure, |
…ure tests - KvPoolView.set_mla_kv_buffer is now a method that forwards its arguments to the pool, not a property returning the pool's bound method. test_physical_write_loc checks every class's set_mla_kv_buffer as a pool write door, and a door must forward its loc untouched. Callers see the same behavior. - test_inkling_checkpoint_indices: give the fake draft model runner an empty capture_tail_hooks list. EAGLEDraftExtendCudaGraphRunner now runs those hooks after each captured draft-extend forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] @nvrohanv, follow-up to my previous comment: I pushed 7c0ede9 to fix both failures, so no action is needed from you unless you'd prefer a different fix.
|
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
…apture tail hooks EAGLEDraftCudaGraphRunner and EAGLEDraftExtendCudaGraphRunner now run model_runner.capture_tail_hooks after each captured forward. The attention-unittest mock model runners skip ModelRunner.__init__, so they lack the attribute, and test_fa4.py's EAGLE draft runner cases failed with AttributeError. The draft worker harnesses now default it to an empty list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] @nvrohanv, one more of the same kind: |
|
On current main this crashes GLM-5.2 megakernel decode at the first CUDA graph capture:
return self._backend.attn_dcp_sizeSeen with this PR's head ( |
sgl-project#48721 renamed the DSA backend's dcp_size attribute to attn_dcp_size, so DsaView.dcp_size raised AttributeError at the first CUDA graph capture of a megakernel decode on DSA models (reported for GLM-5.2). Update the unit test's fake backend to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] @mattteochen, thanks for the report! I pushed cc01bc7 with your suggested fix: |
|
[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest Anyone can stop me at any time: just post a comment asking me to stop (for example, |
|
A second failure on current main, with the Stack: The draft runs one Same setup as above: Nov-15 SGLang nightly image, GB300, GLM-5.2 NVFP4 PD decode TP4 with |
|
@nvrohanv suggested fixes for the two failures above. With both applied on top of this PR's head (
1. --- a/python/sglang/srt/kernel_plugins/nvidia/binding.py
+++ b/python/sglang/srt/kernel_plugins/nvidia/binding.py
@@ class DsaView:
@property
def dcp_size(self) -> int:
- return self._backend.dcp_size
+ return self._backend.attn_dcp_size2. The draft runs one --- a/python/sglang/srt/kernel_plugins/nvidia/binding.py
+++ b/python/sglang/srt/kernel_plugins/nvidia/binding.py
@@ class MegakernelBinding:
self._capture_aux_buffers: dict[str, torch.Tensor] = {}
+ # A draft model runs one DSA backend per EAGLE draft step, each with its own
+ # workspace / counter buffers, so a name-keyed DSA seal would trip at step 1.
+ self._skip_dsa_seal = False
@@ def forward_view(...):
- dsa = None if hybrid_family else view.dsa
+ dsa = None if hybrid_family or self._skip_dsa_seal else view.dsa
@@ def install_megakernel(
binding = _binding_class(hf_config.architectures)(model=model)
+ if is_draft:
+ binding._skip_dsa_seal = TrueA stricter alternative would key the DSA seal by backend (e.g. |
|
Non-megakernel A/B for the early draft embed share ( This piece is gated only on
Weight footprint is the same in both arms. The difference is that the draft's replicated embedding (and lm_head) copy is released before KV pool sizing instead of after, so the pool picks up about 2.1 GB/GPU. Without the env var this code path is a no-op. ** real-weight A/B (same setup as above,
Accuracy and accept length match, so the draft is still using the target's embedding/lm_head after the share moved earlier. The only effect on the non-mega path is the larger KV pool. |
…ition test_init_ends_with_the_early_share required the share to be the last statement of EagleDraftWorker.__init__, so any unrelated line appended to __init__ failed it. The behavioral tests call the share directly (running __init__ needs a GPU), so a source check is still needed to catch the call being dropped. Assert instead that __init__ calls it exactly once, after self.draft_runner is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
| view = ForwardView( | ||
| backend, self._seal_cache, hybrid=hybrid, legacy=not hybrid_family | ||
| ) | ||
| dsa = None if hybrid_family else view.dsa |
There was a problem hiding this comment.
[by Claude Code] Severity: functional | Confidence: High
Issue
The DSA buffers are sealed by name only (dsa_workspace, dsa_kv_counter). An EAGLE draft builds one DeepseekSparseAttnBackend per draft step (draft_utils.py), and each backend allocates its own _multi_ctas_kv_counter_buffer in __init__. So when the draft binding's forward_view reaches step 1, it sees a different counter under the same name, and _seal raises dsa_kv_counter was reallocated after its first read. The head still has this path: only the dcp_size fix landed.
Example
GLM-5.2 NVFP4 with EAGLE and --enable-nvidia-megakernel fails at draft CUDA graph capture, as reported in #46210 (comment).
Fix
Key the DSA seal by backend, for example (id(backend), name) with a getter for that backend. Or apply the tested minimal fix from #46210 (comment) and skip the DSA seal when install_megakernel runs with is_draft=True. Add a CPU test with two backends that own different counters.
Motivation
SGLang-side support for the GLM-5.3 / GLM-5.2 NVFP4 MTP megakernel specs in sgl-project/sgl-nvidia-mega#24 (merged). Those specs refuse at startup without this PR. It closes four gaps in the megakernel layer (#44945):
SGLANG_ENABLE_EMBED_REPLICATION, the EAGLE v2 draft worker binds the target's embedding / lm_head only after the KV pool is sized, so the draft's own copies still count against the KV cache.SGLANG_ENABLE_EMBED_REPLICATION=1refuses--enable-nvidia-megakernelunless the user also sets the variable by hand.Companion PR: #46211 (EAGLE draft step in one CUDA graph). The two are meant to land together.
Modifications
srt/kernel_plugins/nvidia/binding.py):ForwardView.dsareturns aDsaViewof the running DSA backend (Noneotherwise): MLA dims, decode impl, KV dtype, top-k / DCP settings, decode workspace, multi-CTA KV counter andindexer_metadata. The two buffers are sealed at first read and checked byassert_live.KvPoolViewgainsget_index_k_with_scale_buffer,index_page_sizeandset_mla_kv_buffer. NewTOLERATED_DSA_CALLABLES/TOLERATED_DSA_WRITESlist what a megakernel still calls or writes; existingTOLERATED_*lists are unchanged.eagle_draft_cuda_graph_runner.py,eagle_draft_extend_cuda_graph_runner.py): both runners call_run_capture_tail_hooksat the end of their stock captured forward. The draft-decode runner runs the hooks under its first per-step backend and checks that all per-step backends share one KV pool. Drafts served by a megakernel's owndraft_loop/draft_extenddo not take this path.spec.pyadds theEAGLE_DRAFT_CAPTURE_CHECKmarker and documents the draft step modes.eagle_worker_v2.py): under embedding replication,EagleDraftWorkerrunsinit_token_map/init_lm_headat the end of__init__instead of inalloc_memory_pool, so the draft's copies are freed before the KV pool is sized. Without replication nothing moves.spec.py,binding.py):FLAG_SETS_ENV = {SGLANG_ENABLE_EMBED_REPLICATION}.handle_nvidia_megakernelsets it to the spec's value when the spec requires it and the user left it unset, logs it, and restores it if the setup is refused. A user-set value is kept and checked as before (0is still refused).TestMegakernelBindingDsaintest_megakernel_binding.py,TestFlagSetsEnvintest_megakernel_guards.py, and newtest_megakernel_draft_capture.pyandtest/registered/unit/spec/test_eagle_draft_embed_share.py.Accuracy Tests
No output change is expected: the view only reads backend state, the hooks run only at capture, and the embedding change binds the same target tensors earlier. Measured on 8 x B300, TP 8, MTP, GLM-5.3 NVFP4, with this PR + #46211 + #45795 + sgl-nvidia-mega#24:
--enable-nvidia-megakernelalone (no replication env): the launcher logsset SGLANG_ENABLE_EMBED_REPLICATION=Trueonce and target + draft specs are active on 8 / 8 ranks.Speed Tests and Profiling
No per-step change in SGLang. The hooks run once per captured batch size at startup.
Checklist
CI States
Latest PR Test (Base): ✅ Run #43941300624
Latest PR Test (Extra): ❌ Run #43941300494
Latest PR Test (AMD ROCm 10): ✅ Run #43941300449
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: