Skip to content

[Feat] [NVIDIA] megakernels: DSA / MLA binding view, EAGLE draft capture checks, early draft embedding sharing - #46210

Open
nvrohanv wants to merge 14 commits into
sgl-project:mainfrom
nvrohanv:megakernel-dsa-view-draft-capture
Open

nvrohanv wants to merge 14 commits into
sgl-project:mainfrom
nvrohanv:megakernel-dsa-view-draft-capture

Conversation

@nvrohanv

@nvrohanv nvrohanv commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

SGLang-side support for the GLM-5.3 / GLM-5.2 NVFP4 MTP megakernel specs in sgl-project/sgl-nvidia-mega#24 (merged). Those specs refuse at startup without this PR. It closes four gaps in the megakernel layer (#44945):

  1. A megakernel replacing the decode / verify step of a DSA (sparse MLA) model has no binding view of the DSA backend's layout and buffers, so it would have to read private backend attributes.
  2. The EAGLE draft-decode and draft-extend CUDA graph runners do not call the model runner's capture tail hooks, so a draft-model spec is never checked after capture.
  3. With SGLANG_ENABLE_EMBED_REPLICATION, the EAGLE v2 draft worker binds the target's embedding / lm_head only after the KV pool is sized, so the draft's own copies still count against the KV cache.
  4. A spec that requires SGLANG_ENABLE_EMBED_REPLICATION=1 refuses --enable-nvidia-megakernel unless the user also sets the variable by hand.

Companion PR: #46211 (EAGLE draft step in one CUDA graph). The two are meant to land together.

Modifications

  • DSA / MLA binding view (srt/kernel_plugins/nvidia/binding.py): ForwardView.dsa returns a DsaView of the running DSA backend (None otherwise): MLA dims, decode impl, KV dtype, top-k / DCP settings, decode workspace, multi-CTA KV counter and indexer_metadata. The two buffers are sealed at first read and checked by assert_live. KvPoolView gains get_index_k_with_scale_buffer, index_page_size and set_mla_kv_buffer. New TOLERATED_DSA_CALLABLES / TOLERATED_DSA_WRITES list what a megakernel still calls or writes; existing TOLERATED_* lists are unchanged.
  • Draft capture checks (eagle_draft_cuda_graph_runner.py, eagle_draft_extend_cuda_graph_runner.py): both runners call _run_capture_tail_hooks at the end of their stock captured forward. The draft-decode runner runs the hooks under its first per-step backend and checks that all per-step backends share one KV pool. Drafts served by a megakernel's own draft_loop / draft_extend do not take this path. spec.py adds the EAGLE_DRAFT_CAPTURE_CHECK marker and documents the draft step modes.
  • Early draft embedding sharing (eagle_worker_v2.py): under embedding replication, EagleDraftWorker runs init_token_map / init_lm_head at the end of __init__ instead of in alloc_memory_pool, so the draft's copies are freed before the KV pool is sized. Without replication nothing moves.
  • Flag sets the replication env (spec.py, binding.py): FLAG_SETS_ENV = {SGLANG_ENABLE_EMBED_REPLICATION}. handle_nvidia_megakernel sets it to the spec's value when the spec requires it and the user left it unset, logs it, and restores it if the setup is refused. A user-set value is kept and checked as before (0 is still refused).
  • Tests (CPU): TestMegakernelBindingDsa in test_megakernel_binding.py, TestFlagSetsEnv in test_megakernel_guards.py, and new test_megakernel_draft_capture.py and test/registered/unit/spec/test_eagle_draft_embed_share.py.

Accuracy Tests

No output change is expected: the view only reads backend state, the hooks run only at capture, and the embedding change binds the same target tensors earlier. Measured on 8 x B300, TP 8, MTP, GLM-5.3 NVFP4, with this PR + #46211 + #45795 + sgl-nvidia-mega#24:

  • GSM8K-200 (5-shot, greedy, 4 concurrent): 0.995 with 3 draft steps, 0.990 with 5.
  • The plugin's test-only parity mode: stock target-verify and draft steps equal SGLang's (max abs diff 0; 560 verify / 1,680 draft steps at 4 draft tokens, 336 / 2,760 at 6).
  • Greedy token IDs before vs after the embedding change: identical on 20 / 20 requests (5,120 tokens).
  • --enable-nvidia-megakernel alone (no replication env): the launcher logs set SGLANG_ENABLE_EMBED_REPLICATION=True once and target + draft specs are active on 8 / 8 ranks.

Speed Tests and Profiling

No per-step change in SGLang. The hooks run once per captured batch size at startup.

Checklist


CI States

Latest PR Test (Base): ✅ Run #43941300624
Latest PR Test (Extra): ❌ Run #43941300494
Latest PR Test (AMD ROCm 10): ✅ Run #43941300449

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

…ure checks, early draft embedding sharing

A megakernel that replaces the decode or verify step of a DSA (sparse MLA)
model reads the DSA backend's layout and decode buffers and writes the MLA and
index-K rows of its KV pool. Offer them through the binding: KvPoolView gains
the MLA key and index-K buffers, the index page size and the MLA key write;
ForwardView.dsa is a DsaView of the running DSA backend (MLA dimensions,
decode implementation, top-k and DCP settings, the decode workspace and the
multi-CTA KV counter as sealed data, and the indexer metadata).
TOLERATED_DSA_CALLABLES / TOLERATED_DSA_WRITES list what a megakernel still
calls or writes; the existing TOLERATED_* lists are unchanged.

The EAGLE draft-decode and draft-extend graph runners now call the capture
tail hooks at the end of their captured function, as the decode graph runner
does, so a megakernel spec installed on a draft model is checked after
capture. The draft-decode runner runs them under its first per-step attention
backend and checks that the per-step backends share one KV pool.
spec.EAGLE_DRAFT_CAPTURE_CHECK marks an SGLang that does this; Step documents
the draft modes.

Under SGLANG_ENABLE_EMBED_REPLICATION the NextN draft loads its own full
embedding table on every rank. EagleDraftWorker replaced it with the
target's in alloc_memory_pool, which the scheduler calls after it has
sized the target KV pool, so the draft's copy (and its lm_head shard) was
still allocated when the pool was sized. Under replication the draft
worker now binds the target's embedding and lm_head at the end of its
construction, so the draft's own copies are freed before the pool is
sized and the pool can use that memory. Without replication the calls
stay where they were.

With --enable-nvidia-megakernel, argument resolution now sets a variable
of spec.FLAG_SETS_ENV (SGLANG_ENABLE_EMBED_REPLICATION) that the served
spec requires and the user left unset, to the spec's value, and logs it.
SGLang reads that variable only when a model is built, in the model
processes started after resolution, so every rank sees the value. A
value the user set is kept and checked as before, and a refused setup
leaves the environment as it was.
@nvrohanv
nvrohanv force-pushed the megakernel-dsa-view-draft-capture branch from c39924b to 465a9dd Compare October 25, 2026 18:41
@YAMY1234

Copy link
Copy Markdown
Collaborator

Validated the combined GLM-5.3 NVFP4 static-table stack on 8×B300 TP8 with hooks 465a9dd5b7, step 3ba3fd1d2e and mega ad0bf9103d on base dba22e9b68. Target+draft reaches 595.8 tok/s/user at c1 (1.520× the same-session baseline), with two fresh launches per configuration and two passes per launch. Relative gains are consistent with the original table; both baseline and enabled throughput are higher in absolute terms.

Full table and protocol: sgl-project/sgl-nvidia-mega#24 (comment). This measures the combined stack, not the hooks' isolated contribution. The DSA test consolidation is now in 1574eb89dc (16 tests + 4 subtests passed).

One conflict, in the spec import block of
python/sglang/srt/kernel_plugins/nvidia/binding.py: this branch imports
FLAG_SETS_ENV and main (sgl-project#45266) imports
GRAPH_VARIANT_RUNNERS; both are kept.
sgl-project#45266 made the megakernel contract version 2: the
binding reads draft_loop, draft_extend, begin_draft_round and
graph_variants from every built object. The draft capture test's built
object offers none of them, as main's FakeBuilt does.
One conflict, in handle_nvidia_megakernel of
python/sglang/srt/kernel_plugins/nvidia/binding.py: main
(sgl-project#46625) returns early when the MSA plugin serves the
architectures, and this branch starts its FLAG_SETS_ENV bookkeeping
(env_set = []) at the same place. Both are kept: main's return first,
then env_set, so the flag sets variables only for an sgl-nvidia-mega
spec.
wenscarl added a commit to wenscarl/sgl-nvidia-mega that referenced this pull request Nov 3, 2026
- spec, draft_spec: API_VERSION 2; the built objects offer no EAGLE entries or
  graph variants (draft_loop, draft_extend, begin_draft_round, graph_variants are
  None). Restore the target's _Built (widths, summary, plan, forward), which the
  package split had dropped, so build returns a BuiltMegakernel again.
- spec: accept Feature.MULTI_NODE_TP, so TP8 can span the two 4-GPU trays of an
  NVL72 rack, as the draft spec already does.
- spec: drop the DsaView binding-member and EAGLE_DRAFT_CAPTURE_CHECK gates, which
  only an unmerged SGLang (sgl-project/sglang#46210) passes; the covered step reads
  the DSA backend through get_attn_backend() and never through those views.
- attention: build the AttentionGlue directly (it has no build classmethod).
- draft_spec: import the megakernel interface from its new location first.
- glm_dsa, spec_step: declare every SGLang import in SGLANG_READS (the stricter
  contract test); import modules as modules (import a.b.c as c); name reads by the
  file that defines them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with main's draft runtime and binding seal refactor:
- binding.py: seal the DSA buffers through main's _seal(name, value, current)
  in forward_view (non-hybrid only); take main's assert_live; drop the PR's
  duplicate KvPoolView.get_key_buffer that shadowed main's sealing one.
- eagle_draft_cuda_graph_runner.py: keep main's draft_runtime.run_draft_loop;
  call the capture tail hooks at the end of run_stock, as the extend runner does.
- spec.py: keep main's Step mode list and add the draft modes and notes.
- test_megakernel_draft_capture.py: follow the hook call into run_stock and
  give the patched forward_view a view main's seal_capture_view accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@YAMY1234 YAMY1234 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged main and resolved the conflicts; LGTM.

@YAMY1234

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 11, 2026
@nvrohanv
nvrohanv marked this pull request as ready for review November 14, 2026 01:54

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@wenscarl

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

# Conflicts:
#	python/sglang/kernels/plugin_kernels/nvidia/spec.py
#	python/sglang/srt/speculative/eagle_worker_v2.py
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

…mbed_share

The merge of main added `_init_draft_probs_vocab_scatter()` to
`EagleDraftWorker.alloc_memory_pool`. That method reads `target_worker`,
which the test's bare worker does not have. Stub it out like the other
init calls, since the test only checks the order of the sharing calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] @nvrohanv, the relevant CI failures appear to be caused by this PR. Both also failed the same way on the previous head ade8dd41e7 (run 43557226298, both attempts), before the latest merge of main:

  • https://github.com/sgl-project/sglang/actions/runs/43804853739/job/133093439859: test/registered/unit/mem_cache/test_physical_write_loc.py::TestEveryPoolDoorTakesAWriteLoc flags kernel_plugins/nvidia/binding.py:KvPoolView.set_mla_kv_buffer. The AST scan treats any method named set_mla_kv_buffer as a pool write door that must unwrap or forward its loc, and the new KvPoolView.set_mla_kv_buffer property (which returns the pool's bound method) does not match that shape.
  • https://github.com/sgl-project/sglang/actions/runs/43804853739/job/133093439783: test/registered/kernels/ops/attention/test_inkling_checkpoint_indices.py::test_draft_runners_capture_and_replay_checkpoint_destinations fails with AttributeError: 'types.SimpleNamespace' object has no attribute 'capture_tail_hooks'. The PR's new EAGLEDraftExtendCudaGraphRunner._run_capture_tail_hooks reads self.model_runner.capture_tail_hooks, and the test's fake model runner does not define it.

Could you please fix them? Thanks!

(A third failure, test_eagle_draft_embed_share.py, came from my conflict resolution, and I pushed 2758ef7 to fix it.)

…ure tests

- KvPoolView.set_mla_kv_buffer is now a method that forwards its
  arguments to the pool, not a property returning the pool's bound
  method. test_physical_write_loc checks every class's
  set_mla_kv_buffer as a pool write door, and a door must forward its
  loc untouched. Callers see the same behavior.
- test_inkling_checkpoint_indices: give the fake draft model runner an
  empty capture_tail_hooks list. EAGLEDraftExtendCudaGraphRunner now
  runs those hooks after each captured draft-extend forward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] @nvrohanv, follow-up to my previous comment: I pushed 7c0ede9 to fix both failures, so no action is needed from you unless you'd prefer a different fix.

  • KvPoolView.set_mla_kv_buffer is now a method that forwards *args, **kwargs to the pool, instead of a property that returns the pool's bound method. Callers see the same behavior, and it now passes the write-door check in test_physical_write_loc.py.
  • test_inkling_checkpoint_indices.py: the fake draft model runner now has capture_tail_hooks=[].

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

…apture tail hooks

EAGLEDraftCudaGraphRunner and EAGLEDraftExtendCudaGraphRunner now run
model_runner.capture_tail_hooks after each captured forward. The
attention-unittest mock model runners skip ModelRunner.__init__, so they
lack the attribute, and test_fa4.py's EAGLE draft runner cases failed
with AttributeError. The draft worker harnesses now default it to an
empty list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] @nvrohanv, one more of the same kind: test/registered/attention/unittests/dense/test_fa4.py (job) failed with AttributeError: 'MockModelRunner' object has no attribute 'capture_tail_hooks', because the PR's EAGLEDraftCudaGraphRunner._run_capture_tail_hooks now reads it. I pushed 0828975, so the EAGLE draft harnesses in python/sglang/test/kits/attention_unittest/runner_modes/ now give their mock runners an empty capture_tail_hooks list. That covers every attention mock model runner, not just the dense one. I couldn't run the test locally; CI will confirm.

@mattteochen

Copy link
Copy Markdown
Contributor

On current main this crashes GLM-5.2 megakernel decode at the first CUDA graph capture:

AttributeError: 'DeepseekSparseAttnBackend' object has no attribute 'dcp_size'

DsaView.dcp_size in binding.py returns self._backend.dcp_size, but #48721 renamed that attribute to attn_dcp_size. Suggested fix:

return self._backend.attn_dcp_size

Seen with this PR's head (0828975449) on the Nov-15 SGLang nightly image (lmsysorg/sglang:nightly-cu134-20261115-d33d81f): GB300, GLM-5.2 NVFP4, PD decode at TP4 with --enable-nvidia-megakernel.

sgl-project#48721 renamed the DSA backend's dcp_size attribute to attn_dcp_size, so
DsaView.dcp_size raised AttributeError at the first CUDA graph capture
of a megakernel decode on DSA models (reported for GLM-5.2). Update the
unit test's fake backend to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] @mattteochen, thanks for the report! I pushed cc01bc7 with your suggested fix: DsaView.dcp_size now returns self._backend.attn_dcp_size. The fake backend in test_megakernel_binding.py now uses attn_dcp_size as well. I also checked every other backend attribute that DsaView reads against the current DSA backend, and none of the others were renamed.

@nvpohanh

Copy link
Copy Markdown
Collaborator

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

@mattteochen

Copy link
Copy Markdown
Contributor

A second failure on current main, with the dcp_size fix above applied locally. Target capture now passes, but the EAGLE draft CUDA graph capture fails:

RuntimeError: megakernel binding: dsa_kv_counter was reallocated after its first read

Stack: eagle_draft_cuda_graph_runner.capture_one_shape → run_draft_loop → draft_forward → binding.forward → forward_view → _seal.

The draft runs one DeepseekSparseAttnBackend per draft step (attn_backends), and each makes its own multi_ctas_kv_counter_buffer in __init__. The DSA seal is keyed by name, so it sees a different counter tensor at step 1. The counter can also be grown by _ensure_multi_ctas_kv_counter_capacity during graph init. Possible fixes: seal per backend, skip the DSA seal for draft-step backends, or share one counter across the draft backends.

Same setup as above: Nov-15 SGLang nightly image, GB300, GLM-5.2 NVFP4 PD decode TP4 with --enable-nvidia-megakernel.

@mattteochen

Copy link
Copy Markdown
Contributor

@nvrohanv suggested fixes for the two failures above. With both applied on top of this PR's head (0828975449), the Nov-15 SGLang nightly image (lmsysorg/sglang:nightly-cu134-20261115-d33d81f) served a full GLM-5.2 NVFP4 PD run on GB300:

  • 1 prefill + 3 TP4 decode workers with --enable-nvidia-megakernel, Dynamo 1.7;
  • full megakernel coverage on decode (target bs 1–8 and the NextN draft);
  • 0 errors over a 60-min AgentX benchmark.

1. DsaView.dcp_size after the attn_dcp_size rename on main (#48721)

--- a/python/sglang/srt/kernel_plugins/nvidia/binding.py
+++ b/python/sglang/srt/kernel_plugins/nvidia/binding.py
@@ class DsaView:
     @property
     def dcp_size(self) -> int:
-        return self._backend.dcp_size
+        return self._backend.attn_dcp_size

2. dsa_kv_counter was reallocated after its first read in the EAGLE draft capture

The draft runs one DeepseekSparseAttnBackend per draft step, each with its own multi_ctas_kv_counter_buffer, so the name-keyed DSA seal sees a new tensor at step 1. The minimal fix we tested skips the DSA seal in a draft model's binding. The target / verify seal and the kv_pool checks are unchanged.

--- a/python/sglang/srt/kernel_plugins/nvidia/binding.py
+++ b/python/sglang/srt/kernel_plugins/nvidia/binding.py
@@ class MegakernelBinding:
         self._capture_aux_buffers: dict[str, torch.Tensor] = {}
+        # A draft model runs one DSA backend per EAGLE draft step, each with its own
+        # workspace / counter buffers, so a name-keyed DSA seal would trip at step 1.
+        self._skip_dsa_seal = False
@@ def forward_view(...):
-        dsa = None if hybrid_family else view.dsa
+        dsa = None if hybrid_family or self._skip_dsa_seal else view.dsa
@@ def install_megakernel(
     binding = _binding_class(hf_config.architectures)(model=model)
+    if is_draft:
+        binding._skip_dsa_seal = True

A stricter alternative would key the DSA seal by backend (e.g. (id(backend), name)), so each draft step's buffers stay checked. A CPU check confirms that a replaced dsa_kv_counter on the target binding still raises with fix 2.

@wenscarl

wenscarl commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Non-megakernel A/B for the early draft embed share (eagle_worker_v2.py)

This piece is gated only on SGLANG_ENABLE_EMBED_REPLICATION, not on --enable-nvidia-megakernel, so I measured it on the plain (non-mega) EAGLE v2 path.

  • Setup: DeepSeek-V3.2 (FP8) + NextN MTP, TP4 on 4× VR200, EAGLE (steps 3, topk 1, draft tokens 4), --mem-fraction-static 0.85, SGLANG_ENABLE_EMBED_REPLICATION=1, megakernel off, --load-format dummy.
  • Arms: base = merge-base 3981de0, PR = cc01bc7.
base PR Δ
Avail mem after draft weight load (per GPU) 110.44 GB 110.44 GB 0
KV cache size (per GPU) 67.80 GB 69.92 GB +2.12 GB
max_total_num_tokens 1,685,504 1,738,304 +52,800 (+3.1%)

Weight footprint is the same in both arms. The difference is that the draft's replicated embedding (and lm_head) copy is released before KV pool sizing instead of after, so the pool picks up about 2.1 GB/GPU. Without the env var this code path is a no-op.

** real-weight A/B (same setup as above, --load-format default)**

base 3981de0 PR cc01bc7
gsm8k accuracy (1319 q, 5-shot) 0.946 0.947
avg spec accept length 3.088 3.087
KV cache size (per GPU) 67.86 GB 69.98 GB (+2.12 GB)
max_total_num_tokens 1,687,104 1,739,904 (+3.1%)

Accuracy and accept length match, so the draft is still using the target's embedding/lm_head after the share moved earlier. The only effect on the non-mega path is the larger KV pool.
cc. @nvrohanv

…ition

test_init_ends_with_the_early_share required the share to be the last
statement of EagleDraftWorker.__init__, so any unrelated line appended to
__init__ failed it. The behavioral tests call the share directly (running
__init__ needs a GPU), so a source check is still needed to catch the call
being dropped. Assert instead that __init__ calls it exactly once, after
self.draft_runner is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

view = ForwardView(
backend, self._seal_cache, hybrid=hybrid, legacy=not hybrid_family
)
dsa = None if hybrid_family else view.dsa

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: functional | Confidence: High

Issue

The DSA buffers are sealed by name only (dsa_workspace, dsa_kv_counter). An EAGLE draft builds one DeepseekSparseAttnBackend per draft step (draft_utils.py), and each backend allocates its own _multi_ctas_kv_counter_buffer in __init__. So when the draft binding's forward_view reaches step 1, it sees a different counter under the same name, and _seal raises dsa_kv_counter was reallocated after its first read. The head still has this path: only the dcp_size fix landed.

Example

GLM-5.2 NVFP4 with EAGLE and --enable-nvidia-megakernel fails at draft CUDA graph capture, as reported in #46210 (comment).

Fix

Key the DSA seal by backend, for example (id(backend), name) with a getter for that backend. Or apply the tested minimal fix from #46210 (comment) and skip the DSA seal when install_megakernel runs with is_draft=True. Add a CPU test with two backends that own different counters.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jit-kernel run-ci CI: run the baseline test suite on this PR speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants