nemotron_h: load in-checkpoint MTP head for self-speculation#1533
Open
pierre427 wants to merge 8 commits into
Open
nemotron_h: load in-checkpoint MTP head for self-speculation#1533pierre427 wants to merge 8 commits into
pierre427 wants to merge 8 commits into
Conversation
Speculative decoding currently raises for models with recurrent
ArraysCache layers ("requires a trimmable prompt cache", ml-explore#980): the
recurrent state summarizes the whole past and cannot be trimmed.
This adds an exact, cheap rollback instead of forbidding the trim:
- ArraysCache: start/stop_speculation + record_rollback(T, fn, snapshot);
while speculating the cache is trimmable, and trim(n) restores the
entries via the recorded fn (or the snapshot for a full rollback).
- GDN layers (qwen3_5, qwen3_next): while speculating, record a rollback
closure over the exact per-token tensors the kernel consumed. Replaying
the recurrence from the pre-forward state over the first m stashed
inputs reproduces the post-m state bit-for-bit (causality: rows of a
chunk do not depend on later rows), and the conv state is a slice of
conv_input. No deepcopy, no re-forward, no extra weight streaming —
the rollback costs one tiny kernel launch per GDN layer.
- speculative_generate_step: allow models declaring
supports_speculative_rollback, and start/stop recording around the
decode loop (recording is off during prefill so no prompt-sized
tensors are retained).
Tests: tail-independence bit-exactness (same m-prefix, different tails
=> identical caches and logits after rollback), greedy speculative ==
greedy vanilla end-to-end with a mismatched draft (rollback every
round), and the unsupported-model error path.
Addresses findings from an adversarial audit of the first version: - ArraysCache keeps a small stack of rollback records (bounded by a 64-token window) instead of a single record, so multi-forward windows can be rewound — enabling HYBRID DRAFT models: a Qwen3.5 draft records one entry per T=1 draft step and its cache is now rewound exactly (the setup reported in ml-explore#1446 uses a hybrid draft). Partial rollbacks re-record the still-valid prefix. - trim() raises when asked to trim beyond the recorded window instead of silently clamping (a clamp would desync ArraysCache layers from KVCache layers, which trim the full amount). - speculative_generate_step: per-cache capability check (trimmable OR rollback-capable with model support) instead of a model-wide bypass; start/stop speculation on the draft cache too; n is initialized to num_draft at the top of each round so a mid-round exception or generator close makes the finally-block rewind a no-op rather than trimming with stale values — previously the rollback RuntimeError could mask the original exception. Tests: hybrid draft == vanilla greedy (rollback across stacked T=1 records), mid-round draft exception surfaces unmasked, trim beyond the window raises. 8/8 in this file; full suite failure set identical to main (pre-existing failures only). Real-weights e2e: dense draft 1.37x identical; hybrid draft pair 1.33x identical.
…d caches Fix: converted '-mtp' checkpoints (which keep mtp.* tensors with already-shifted norms) were double-shifted at load because mtp presence was treated as a raw-checkpoint signal; the signal is now the unsanitized conv1d layout only. Feature: mtp_num_hidden_layers builds an MTPModule (fc/pre_fc_norms/ full-attn decoder layer/norm, shared lm_head), self-dropped when a checkpoint has no mtp tensors. TextModel gains mtp_step() and rollback_speculative_cache(): KV caches trim; GatedDeltaNet states are rebuilt by replaying gdn_sink-captured verify inputs on the accepted prefix in one batched gated_delta_update (single-sequence scope). Replay pattern after mlx-vlm qwen3_5 / MTPLX (Apache-2.0). Measured on Qwen3.5-122B-A10B-oQ4-mtp (M5 Max): MTP chain k=4 = 1.36-2.11x vs plain greedy, output lossless modulo sub-ULP kernel ties. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oth) Combines two independently-built halves into one story: - MTP head loading + norm-shift fix (our PR ml-explore#1456): un-strips mtp.* weights, builds MTPModule, mtp_step/make_mtp_cache; converted -mtp checkpoints load. - GDN exact-replay speculative rollback (lBroth/gdn-exact-replay): model- agnostic ArraysCache protocol (start/stop_speculation, record_rollback, trim applies the recorded replay) wired into speculative_generate_step, so --draft-model speculation works on hybrid (GatedDeltaNet) caches today, incl. hybrid draft models; covers qwen3_5* and qwen3_next. Our model-file-local gdn_sink + rollback_speculative_cache is DROPPED in favor of lBroth's cache-level protocol (cleaner: model files stay agnostic, the generation loop's existing trim_prompt_cache just works). Our single- kernel batched replay remains a credited follow-up optimization for the per-layer trim path. Tests: 8 exact-replay (incl. bit-exact tail-independence) + 3 MTP loading + norm-shift contract — all pass. Co-Authored-By: lBroth <noreply@github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pierre Lamy <pierre@userid.org>
Extends the exact-rollback protocol to the OTHER non-trimmable cache: sliding-window RotatingKVCache. Once it wraps (offset >= max_size) the tokens a verify step pushes out of the window are gone, so stock trim can't rewind. Same protocol as ArraysCache: start/stop_speculation + record_rollback; update_and_fetch stashes the pre-forward window + new K/V while speculating, trim restores + re-appends the accepted prefix. No model-file hook (KV is not recurrent). is_trimmable() is True while speculating; gpt_oss declares supports_speculative_rollback=True, so speculative_generate_step accepts it unchanged -> --draft-model spec now works on gpt-oss-family sliding-window models, not just GDN hybrids. 4 tests incl. bit-exact tail-independence (two verify chunks sharing an m-prefix with different tails leave identical windows after rollback-to-m) + stacked draft records. 37/37 across prompt_cache + all rollback + MTP. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pierre Lamy <pierre@userid.org>
The MTP (nextn) head lives on the inner language_model/TextModel, but self_mtp_generate_step reads mtp/logits/mtp_step/make_mtp_cache from the top-level model. For wrapped checkpoints (qwen3_5_moe) that left the head unreachable. Delegate the four members to language_model so self-spec works on the top-level model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nemotron 3 Super checkpoints ship a DeepSeek-style multi-token-prediction head. Wire it into nemotron_h.py so it loads from the checkpoint and drives self-speculation through the same head interface ml-explore#1456 established for qwen3_5 (logits / make_mtp_cache / mtp_step; sanitize drops the module when a checkpoint carries no mtp.* tensors). Because the backbone is a hybrid Mamba-2/attention stack, the recurrent SSM layers cannot be trimmed like a KV cache. Add ssm_sink rollback support: an optional list threaded through the Mamba forwards records the exact inputs to replay each SSM update, and Model.rollback_speculative_cache trims the KV caches while rebuilding each Mamba ArraysCache on the accepted prefix. No new cache hook is needed; this path is self-contained and does not use ArraysCache.record_rollback. Stacks on ml-explore#1456. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
nemotron_h: load in-checkpoint MTP head for self-speculation
Stacked on #1456 (MTP self-speculation machinery + qwen3_5 MTP-head
template). Please review/merge #1456 first.
What
Nemotron 3 Super checkpoints ship a DeepSeek-style multi-token-prediction
(MTP) head alongside the main backbone. This wires that head into
nemotron_h.pyso it loads from the checkpoint and drives self-speculativedecoding through the same interface #1456 established for
qwen3_5(
logits/make_mtp_cache/mtp_step, plussanitizethat drops themodule when a checkpoint carries no
mtp.*tensors).Because the backbone is a hybrid Mamba-2/attention stack, the recurrent
(SSM) layers cannot be trimmed the way a KV cache can. Rolling back a
rejected speculative block therefore needs to replay the accepted prefix
through the SSM update rather than truncate it. This PR adds that support:
ssm_sink— an optional list threaded through the Mamba mixer / block/ model forwards. When present, each Mamba layer appends the exact inputs
(
hidden_states,B,C,dt, per-layerA_log/D/dt_bias, thepre-update state, mask, and padded conv input) needed to replay its update.
When
None(the default, i.e. normal decoding) there is zero overhead.Model.rollback_speculative_cache(caches, ssm_states, keep, block_size)— trims the KV caches normally and rebuilds each Mamba
ArraysCachebyre-running
ssm_updateon the kept prefix (or restoring the capturedpre-verify state verbatim when
keep == 0). Single-sequence (B=1,unpadded) only; raises on padded/batched caches.
NemotronHMTPBlock/NemotronHMTP— the head itself. The first MTPlayer carries the embed/hidden fusion (
eh_proj/enorm/hnorm), thelast carries
final_layernorm; each block is otherwise a standardNemotronHBlock. The shipped head is transformer (attention) blocks, somtp_stepbuilds the attention mask.ModelArgsgainsnum_nextn_predict_layers,mtp_layers_block_type,and
mtp_hybrid_override_pattern(the block-type list is normalized tosingle-char pattern codes in
__post_init__, mirroring the backbone'shybrid_override_pattern). The MTP module is built only when the configdeclares MTP layers and dropped again in
sanitizeif the checkpoint hasno
mtp.*tensors, so strict loading stays consistent for plain(non-MTP) Nemotron-H checkpoints.
How it wires to the self-spec machinery
Identical head interface to #1456's
qwen3_5template:logits(hidden)projects a hidden throughlm_head.make_mtp_cache()builds the head's own per-layer cache (KV forattention layers).
mtp_step(hidden, tokens, mtp_cache)runs one MTP forward: fuses thebackbone hidden with the next token's embedding via
enorm/hnorm/eh_proj, runs the head layers, appliesfinal_layernorm,and returns
(logits, post_hidden)so draft depth can be chained.sanitizeuses the same "drop the module when the checkpoint has nomtp.*tensors" logic (and only stacks experts for prefixes actuallypresent, covering both backbone and MTP layers).
The one Nemotron-specific addition over the qwen template is the SSM
rollback path (
ssm_sink+rollback_speculative_cache). Nemotron'srecurrent state does not use the base's
ArraysCache.record_rollbackhook — it is self-contained: the verify forward captures replay inputs via
ssm_sink, androllback_speculative_cacherebuilds the state directly. Nonew cache hook was required; all helpers it needs (
ssm_update,create_attention_mask,ArraysCache,KVCache) already exist on the#1456 base.
Test
tests/test_nemotron_h_mtp.py(mirrorstests/test_qwen3_5_mtp.py):test_mtp_step_shapes_and_chaining— tiny hybrid config with a2-layer attention MTP head; real backbone forward, then
mtp_stepproduces a valid next-token prediction with correct shapes and cache
offsets, and chaining on the head's own hidden advances the cache.
test_mtp_module_dropped_without_weights— a checkpoint with nomtp.*tensors drops the module (model.mtp is None).test_rollback_speculative_cache_matches_fresh_forward— exercisesssm_sinkcapture +rollback_speculative_cache: after ablock_size-4verify forward with
keep-2 accepted, both the KV caches and the Mambarecurrent states match a fresh forward over just the committed prefix plus
kept tokens.
All pass;
test_qwen3_5_mtp.pyand the existingnemotron_hmodel test areunaffected.
black+isort --profile blackclean.