Skip to content

nemotron_h: load in-checkpoint MTP head for self-speculation#1533

Open
pierre427 wants to merge 8 commits into
ml-explore:mainfrom
pierre427:pr/nemotron-h-mtp-head
Open

nemotron_h: load in-checkpoint MTP head for self-speculation#1533
pierre427 wants to merge 8 commits into
ml-explore:mainfrom
pierre427:pr/nemotron-h-mtp-head

Conversation

@pierre427

Copy link
Copy Markdown

nemotron_h: load in-checkpoint MTP head for self-speculation

Stacked on #1456 (MTP self-speculation machinery + qwen3_5 MTP-head
template). Please review/merge #1456 first.

What

Nemotron 3 Super checkpoints ship a DeepSeek-style multi-token-prediction
(MTP) head alongside the main backbone. This wires that head into
nemotron_h.py so it loads from the checkpoint and drives self-speculative
decoding through the same interface #1456 established for qwen3_5
(logits / make_mtp_cache / mtp_step, plus sanitize that drops the
module when a checkpoint carries no mtp.* tensors).

Because the backbone is a hybrid Mamba-2/attention stack, the recurrent
(SSM) layers cannot be trimmed the way a KV cache can. Rolling back a
rejected speculative block therefore needs to replay the accepted prefix
through the SSM update rather than truncate it. This PR adds that support:

  • ssm_sink — an optional list threaded through the Mamba mixer / block
    / model forwards. When present, each Mamba layer appends the exact inputs
    (hidden_states, B, C, dt, per-layer A_log/D/dt_bias, the
    pre-update state, mask, and padded conv input) needed to replay its update.
    When None (the default, i.e. normal decoding) there is zero overhead.
  • Model.rollback_speculative_cache(caches, ssm_states, keep, block_size)
    — trims the KV caches normally and rebuilds each Mamba ArraysCache by
    re-running ssm_update on the kept prefix (or restoring the captured
    pre-verify state verbatim when keep == 0). Single-sequence (B=1,
    unpadded) only; raises on padded/batched caches.
  • NemotronHMTPBlock / NemotronHMTP — the head itself. The first MTP
    layer carries the embed/hidden fusion (eh_proj / enorm / hnorm), the
    last carries final_layernorm; each block is otherwise a standard
    NemotronHBlock. The shipped head is transformer (attention) blocks, so
    mtp_step builds the attention mask.
  • ModelArgs gains num_nextn_predict_layers, mtp_layers_block_type,
    and mtp_hybrid_override_pattern (the block-type list is normalized to
    single-char pattern codes in __post_init__, mirroring the backbone's
    hybrid_override_pattern). The MTP module is built only when the config
    declares MTP layers and dropped again in sanitize if the checkpoint has
    no mtp.* tensors, so strict loading stays consistent for plain
    (non-MTP) Nemotron-H checkpoints.

How it wires to the self-spec machinery

Identical head interface to #1456's qwen3_5 template:

  • logits(hidden) projects a hidden through lm_head.
  • make_mtp_cache() builds the head's own per-layer cache (KV for
    attention layers).
  • mtp_step(hidden, tokens, mtp_cache) runs one MTP forward: fuses the
    backbone hidden with the next token's embedding via
    enorm/hnorm/eh_proj, runs the head layers, applies final_layernorm,
    and returns (logits, post_hidden) so draft depth can be chained.
  • sanitize uses the same "drop the module when the checkpoint has no
    mtp.* tensors" logic (and only stacks experts for prefixes actually
    present, covering both backbone and MTP layers).

The one Nemotron-specific addition over the qwen template is the SSM
rollback path (ssm_sink + rollback_speculative_cache). Nemotron's
recurrent state does not use the base's ArraysCache.record_rollback
hook — it is self-contained: the verify forward captures replay inputs via
ssm_sink, and rollback_speculative_cache rebuilds the state directly. No
new cache hook was required; all helpers it needs (ssm_update,
create_attention_mask, ArraysCache, KVCache) already exist on the
#1456 base.

Test

tests/test_nemotron_h_mtp.py (mirrors tests/test_qwen3_5_mtp.py):

  • test_mtp_step_shapes_and_chaining — tiny hybrid config with a
    2-layer attention MTP head; real backbone forward, then mtp_step
    produces a valid next-token prediction with correct shapes and cache
    offsets, and chaining on the head's own hidden advances the cache.
  • test_mtp_module_dropped_without_weights — a checkpoint with no
    mtp.* tensors drops the module (model.mtp is None).
  • test_rollback_speculative_cache_matches_fresh_forward — exercises
    ssm_sink capture + rollback_speculative_cache: after a block_size-4
    verify forward with keep-2 accepted, both the KV caches and the Mamba
    recurrent states match a fresh forward over just the committed prefix plus
    kept tokens.

All pass; test_qwen3_5_mtp.py and the existing nemotron_h model test are
unaffected. black + isort --profile black clean.

lBroth and others added 8 commits July 2, 2026 08:41
Speculative decoding currently raises for models with recurrent
ArraysCache layers ("requires a trimmable prompt cache", ml-explore#980): the
recurrent state summarizes the whole past and cannot be trimmed.

This adds an exact, cheap rollback instead of forbidding the trim:

- ArraysCache: start/stop_speculation + record_rollback(T, fn, snapshot);
  while speculating the cache is trimmable, and trim(n) restores the
  entries via the recorded fn (or the snapshot for a full rollback).
- GDN layers (qwen3_5, qwen3_next): while speculating, record a rollback
  closure over the exact per-token tensors the kernel consumed. Replaying
  the recurrence from the pre-forward state over the first m stashed
  inputs reproduces the post-m state bit-for-bit (causality: rows of a
  chunk do not depend on later rows), and the conv state is a slice of
  conv_input. No deepcopy, no re-forward, no extra weight streaming —
  the rollback costs one tiny kernel launch per GDN layer.
- speculative_generate_step: allow models declaring
  supports_speculative_rollback, and start/stop recording around the
  decode loop (recording is off during prefill so no prompt-sized
  tensors are retained).

Tests: tail-independence bit-exactness (same m-prefix, different tails
=> identical caches and logits after rollback), greedy speculative ==
greedy vanilla end-to-end with a mismatched draft (rollback every
round), and the unsupported-model error path.
Addresses findings from an adversarial audit of the first version:

- ArraysCache keeps a small stack of rollback records (bounded by a
  64-token window) instead of a single record, so multi-forward
  windows can be rewound — enabling HYBRID DRAFT models: a Qwen3.5
  draft records one entry per T=1 draft step and its cache is now
  rewound exactly (the setup reported in ml-explore#1446 uses a hybrid draft).
  Partial rollbacks re-record the still-valid prefix.
- trim() raises when asked to trim beyond the recorded window instead
  of silently clamping (a clamp would desync ArraysCache layers from
  KVCache layers, which trim the full amount).
- speculative_generate_step: per-cache capability check (trimmable OR
  rollback-capable with model support) instead of a model-wide bypass;
  start/stop speculation on the draft cache too; n is initialized to
  num_draft at the top of each round so a mid-round exception or
  generator close makes the finally-block rewind a no-op rather than
  trimming with stale values — previously the rollback RuntimeError
  could mask the original exception.

Tests: hybrid draft == vanilla greedy (rollback across stacked T=1
records), mid-round draft exception surfaces unmasked, trim beyond the
window raises. 8/8 in this file; full suite failure set identical to
main (pre-existing failures only). Real-weights e2e: dense draft
1.37x identical; hybrid draft pair 1.33x identical.
…d caches

Fix: converted '-mtp' checkpoints (which keep mtp.* tensors with
already-shifted norms) were double-shifted at load because mtp presence
was treated as a raw-checkpoint signal; the signal is now the
unsanitized conv1d layout only.

Feature: mtp_num_hidden_layers builds an MTPModule (fc/pre_fc_norms/
full-attn decoder layer/norm, shared lm_head), self-dropped when a
checkpoint has no mtp tensors. TextModel gains mtp_step() and
rollback_speculative_cache(): KV caches trim; GatedDeltaNet states are
rebuilt by replaying gdn_sink-captured verify inputs on the accepted
prefix in one batched gated_delta_update (single-sequence scope).
Replay pattern after mlx-vlm qwen3_5 / MTPLX (Apache-2.0).

Measured on Qwen3.5-122B-A10B-oQ4-mtp (M5 Max): MTP chain k=4 =
1.36-2.11x vs plain greedy, output lossless modulo sub-ULP kernel ties.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oth)

Combines two independently-built halves into one story:
- MTP head loading + norm-shift fix (our PR ml-explore#1456): un-strips mtp.* weights,
  builds MTPModule, mtp_step/make_mtp_cache; converted -mtp checkpoints load.
- GDN exact-replay speculative rollback (lBroth/gdn-exact-replay): model-
  agnostic ArraysCache protocol (start/stop_speculation, record_rollback,
  trim applies the recorded replay) wired into speculative_generate_step, so
  --draft-model speculation works on hybrid (GatedDeltaNet) caches today,
  incl. hybrid draft models; covers qwen3_5* and qwen3_next.

Our model-file-local gdn_sink + rollback_speculative_cache is DROPPED in
favor of lBroth's cache-level protocol (cleaner: model files stay agnostic,
the generation loop's existing trim_prompt_cache just works). Our single-
kernel batched replay remains a credited follow-up optimization for the
per-layer trim path.

Tests: 8 exact-replay (incl. bit-exact tail-independence) + 3 MTP loading +
norm-shift contract — all pass.

Co-Authored-By: lBroth <noreply@github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pierre Lamy <pierre@userid.org>
Extends the exact-rollback protocol to the OTHER non-trimmable cache:
sliding-window RotatingKVCache. Once it wraps (offset >= max_size) the
tokens a verify step pushes out of the window are gone, so stock trim
can't rewind. Same protocol as ArraysCache: start/stop_speculation +
record_rollback; update_and_fetch stashes the pre-forward window + new K/V
while speculating, trim restores + re-appends the accepted prefix. No
model-file hook (KV is not recurrent). is_trimmable() is True while
speculating; gpt_oss declares supports_speculative_rollback=True, so
speculative_generate_step accepts it unchanged -> --draft-model spec now
works on gpt-oss-family sliding-window models, not just GDN hybrids.

4 tests incl. bit-exact tail-independence (two verify chunks sharing an
m-prefix with different tails leave identical windows after rollback-to-m)
+ stacked draft records. 37/37 across prompt_cache + all rollback + MTP.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Pierre Lamy <pierre@userid.org>
The MTP (nextn) head lives on the inner language_model/TextModel, but
self_mtp_generate_step reads mtp/logits/mtp_step/make_mtp_cache from the
top-level model. For wrapped checkpoints (qwen3_5_moe) that left the head
unreachable. Delegate the four members to language_model so self-spec
works on the top-level model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nemotron 3 Super checkpoints ship a DeepSeek-style multi-token-prediction
head. Wire it into nemotron_h.py so it loads from the checkpoint and drives
self-speculation through the same head interface ml-explore#1456 established for
qwen3_5 (logits / make_mtp_cache / mtp_step; sanitize drops the module when
a checkpoint carries no mtp.* tensors).

Because the backbone is a hybrid Mamba-2/attention stack, the recurrent SSM
layers cannot be trimmed like a KV cache. Add ssm_sink rollback support: an
optional list threaded through the Mamba forwards records the exact inputs
to replay each SSM update, and Model.rollback_speculative_cache trims the KV
caches while rebuilding each Mamba ArraysCache on the accepted prefix. No new
cache hook is needed; this path is self-contained and does not use
ArraysCache.record_rollback.

Stacks on ml-explore#1456.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants